TypeError: can't pickle fasttext_pybind.fasttext objects

[lafi...@gmail.com]

Hello,

I'm runing Fastext  pretrained model Model with spark, when i broadcast the model like that:

sc = SparkContext.getOrCreate()

binary_model_brodcast = sc.broadcast(binary_model)

i get this error:

Error Log:

Traceback (most recent call last):
  File "/opt/spark/python/lib/pyspark.zip/pyspark/broadcast.py", line 110, in dump
    pickle.dump(value, f, 2)
TypeError: can't pickle fasttext_pybind.fasttext objects
Traceback (most recent call last):
  File "/opt/spark/python/lib/pyspark.zip/pyspark/broadcast.py", line 110, in dump
TypeError: can't pickle fasttext_pybind.fasttext objects


Traceback (most recent call last):
  File "/opt/spark/examples/src/main/python/SparkTweet.py", line 249, in <module>
    binary_classification_model_brodcast = sc.broadcast(binary_classification_model)
  File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 848, in broadcast
  File "/opt/spark/python/lib/pyspark.zip/pyspark/broadcast.py", line 90, in __init__
  File "/opt/spark/python/lib/pyspark.zip/pyspark/broadcast.py", line 117, in dump
_pickle.PicklingError: Could not serialize broadcast: TypeError: can't pickle fasttext_pybind.fasttext objects

what would be the solution to fix this?

Thank you so much

Lafi Raed