How can I use BigDL with PyCharm?

395 views
Skip to first unread message

marioru...@gmail.com

unread,
Jun 5, 2017, 8:25:40 AM6/5/17
to BigDL User Group
Hello,

I'am new to the Spark and bigDL world. I have installed both thanks to the Intel tutorial but I don't know how to get them working on PyCharm.

Can you help me?

Thank you!

zhichao

unread,
Jun 6, 2017, 2:12:17 AM6/6/17
to marioru...@gmail.com, BigDL User Group
+ group

On Tue, Jun 6, 2017 at 2:08 PM, zhichao <lisur...@gmail.com> wrote:
There are some variables need to be set in Pycharm:

PYTHONPATH=SPARK_HOME/python/lib/pyspark.zip:SPARK_HOME/python/lib/py4j-0.9-src.zip:BigDL_HOME/spark/dl/src/main/resources/spark-bigdl.conf
spark.master=local[2]
SPARK_CLASSPATH=BigDL_HOME/spark/dl/target/bigdl-0.2.0-SNAPSHOT-jar-with-dependencies.jar
SPARK_HOME=SPARK_HOME

and also 
$ source PATH_To_BigDL/dist/bin/bigdl.sh
need to be run before starting pycharm which essentially just set some environment variables.

Thanks,
Zhichao

--
You received this message because you are subscribed to the Google Groups "BigDL User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/5ef9b860-8db7-4d22-807e-4a4afb699a72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Mario Ruggieri

unread,
Jun 7, 2017, 6:48:42 AM6/7/17
to BigDL User Group
Thank you for answering my question.
After adding variables and the path into the project interpreter, it runs.
Now the issue is: "TypeError: 'JavaPackage' object is not callable"
I think the problem is that I'am not using spark-submit with the options in https://github.com/intel-analytics/BigDL/wiki/Python-Support
How can I use it in PyCharm?

Thanks

Il giorno martedì 6 giugno 2017 08:12:17 UTC+2, Chao Li ha scritto:
+ group

On Tue, Jun 6, 2017 at 2:08 PM, zhichao <lisur...@gmail.com> wrote:
There are some variables need to be set in Pycharm:

PYTHONPATH=SPARK_HOME/python/lib/pyspark.zip:SPARK_HOME/python/lib/py4j-0.9-src.zip:BigDL_HOME/spark/dl/src/main/resources/spark-bigdl.conf
spark.master=local[2]
SPARK_CLASSPATH=BigDL_HOME/spark/dl/target/bigdl-0.2.0-SNAPSHOT-jar-with-dependencies.jar
SPARK_HOME=SPARK_HOME

and also 
$ source PATH_To_BigDL/dist/bin/bigdl.sh
need to be run before starting pycharm which essentially just set some environment variables.

Thanks,
Zhichao
On Mon, Jun 5, 2017 at 8:25 PM, <marioru...@gmail.com> wrote:
Hello,

I'am new to the Spark and bigDL world. I have installed both thanks to the Intel tutorial but I don't know how to get them working on PyCharm.

Can you help me?

Thank you!

--
You received this message because you are subscribed to the Google Groups "BigDL User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-gro...@googlegroups.com.
To post to this group, send email to bigdl-us...@googlegroups.com.

Li, Zhichao

unread,
Jun 7, 2017, 8:46:23 PM6/7/17
to Mario Ruggieri, BigDL User Group

Mario Ruggieri

unread,
Jun 8, 2017, 8:56:02 AM6/8/17
to BigDL User Group
Ok thanks I solved! Now the problem is that I run bigdl.sh before starting PyCharm but I still have this problem:

"Cannot find MKL_DISABLE_FAST_MM in environment variables"

Why? 

To post to this group, send email to bigdl-u...@googlegroups.com.

Li, Zhichao

unread,
Jun 8, 2017, 9:54:05 PM6/8/17
to Mario Ruggieri, BigDL User Group

It’s due to pycharm didn’t inherit those environments successfully.

 

One thing you can try is put the sourcing script at the beginning of the pycharm starting script.

# PyCharm startup script.

# ---------------------------------------------------------------------

#

. $BIGDLHOME/dist/bin/bigdl.sh

 

Or you can set those environments directly into pycharm the same way as you set PYTHONPATH

Yiheng Wang

unread,
Jun 8, 2017, 9:57:39 PM6/8/17
to Li, Zhichao, Mario Ruggieri, BigDL User Group
The ENV variable is annoying. We're trying to eliminate this. There's an issue for this https://github.com/intel-analytics/BigDL/issues/943

To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "BigDL User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigdl-user-group+unsubscribe@googlegroups.com.
To post to this group, send email to bigdl-user-group@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigdl-user-group/F03D1005CC201743BE1B3E1C1E14668B6ABF5E3E%40SHSMSX101.ccr.corp.intel.com.
Reply all
Reply to author
Forward
0 new messages