sparkdl 1.3.0 for Spark 2.4 makes Pipeline not MLWritable

Dave Jaffe

unread,

Nov 16, 2018, 11:54:12 PM11/16/18

to DL-Pipelines-users

When I switched to the new sparkdl package (databricks:spark-deep-learning:1.3.0-spark2.4-s_2.11) I was no longer able to save a pipeline model.

For example, I added a save command to this sample code from https://docs.databricks.com/applications/deep-learning/deep-learning-pipelines.html#deep-learning-pipelines-1:

from pyspark.ml.classification import LogisticRegression
from pyspark.ml import Pipeline
from sparkdl import DeepImageFeaturizer 

featurizer = DeepImageFeaturizer(inputCol="image", outputCol="features", modelName="InceptionV3")
lr = LogisticRegression(maxIter=20, regParam=0.05, elasticNetParam=0.3, labelCol="label")
p = Pipeline(stages=[featurizer, lr])

p_model = p.fit(train_df)

p_model.save('flower_model')

I received the following error. This worked with the previous sparkdl package (databricks:spark-deep-learning:1.2.0-spark2.3-s_2.11).

ValueError: ('Pipeline write will fail on this pipeline because stage %s of type %s is not MLWritable', 'DeepImageFeaturizer_d5f21d7be757', <class 'sparkdl.transformers.named_image.DeepImageFeaturizer'>)

Can somebody verify I am doing this correctly and possibly offer a workaround?

Xiangrui Meng

unread,

Nov 17, 2018, 12:08:58 AM11/17/18

to Dave Jaffe, DL-Pipelines-users

See https://github.com/databricks/spark-deep-learning/pull/161.

--
You received this message because you are subscribed to the Google Groups "DL-Pipelines-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dl-pipelines-us...@googlegroups.com.
To post to this group, send email to dl-pipeli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dl-pipelines-users/23c1f7ac-ffed-4094-9539-e8bdc6f888fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Xiangrui Meng

Software Engineer

Databricks Inc.