java.lang.NoClassDefFoundError: org/tensorflow/Tensor

[aarandag]

I'm trying to run this example locally (machine translation with MarianMT), and I'm finding this error:

sentence_detector_dl download started this may take some time.
Approximate size to download 514.9 KB
Download done! Loading the resource.
[ / ]
An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel.
: java.lang.NoClassDefFoundError: org/tensorflow/Tensor
at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.read(TensorflowWrapper.scala:453)
at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel.readTensorflowModel(TensorflowSerializeModel.scala:154)
at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel.readTensorflowModel$(TensorflowSerializeModel.scala:123)
at com.johnsnowlabs.nlp.annotators.sentence_detector_dl.SentenceDetectorDLModel$.readTensorflowModel(SentenceDetectorDLModel.scala:648)
at com.johnsnowlabs.nlp.annotators.sentence_detector_dl.ReadsSentenceDetectorDLGraph.readSentenceDetectorDLGraph(SentenceDetectorDLModel.scala:621)
...

I'm using this artifact: com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3, configuring it in spark.jars.packages, so all dependencies should be installed. Things I tried with no success:

• install tensorflow with pip.
• install the org.tensorflow.tensorflow:1.15.0

What is strange to me is that Tensorflow should be bundled in spark-nlp, right?

Any help would be greatly appreciated. Thank you!

EDIT: I encountered this error in an Apple Silicon (M1) laptop. When running on an arm64 laptop, I didn't find this issue. So I think it must be related.

[DevinTDHa]

Hi @aarandag,

Thanks for reporting. Indeed, Tensorflow should be bundled with Spark NLP.

However, if you are running on Apple Silicon, you need to consider the following:

1. Your Java installation needs to be compiled for Apple Silicon, see Installation for M1 Macs
2. Start Spark NLP with the right flags to enable Apple Silicon support:

import sparknlp

spark = sparknlp.start(apple_silicon=True)

Can you try this and let us know if it works?