jupyter pyspark issue

[krishnachai...@gmail.com]

after activating the anaconda parcel, i’m getting below error while executing the script
[root@hostname~]# PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python spark-submit pi.py 1000
WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark) overrides detected (/opt/cloudera/parcels/CDH/lib/spark/).
WARNING: Running spark-class from user-defined location.
jupyter: '/home/admin/pi.py' is not a Jupyter command

[Kristopher Overholt]

Aside from the warnings (which don't appear to be causing any errors), I am unable to determine why you are getting an error about Jupyter when using spark-submit. Is there something in pi.py that calls Jupyter? Can you post the contents of pi.py?

[ben.z...@continuum.io]

It would also be great if you could post other env variables set:

For example:

mrmr@asdb:~$ env
TERM_PROGRAM=iTerm.app
TERM=xterm-color
SHELL=/bin/bash
HADOOP_HOME=/usr/local/Cellar/hadoop/2.6.0/
TMPDIR=/var/folders/1t/t94brwgx7sjcn8jgz4gr3_c00000gq/T/
mrmr@asdb:~$

[ss.sha...@gmail.com]

hi i have the same problem with running spark...it keeps saying that .py is not a jupiter command...any help is appreciated.