pyspark not working due to python module not found, path to module set on PYTHONPATH env var

[Gael Abadin]

Hi guys,

I've got a configuration issue running pyspark from hue notebooks examples.

Python spark jobs sent to yarn are failing with an ImportError: No module named pyspark.shellerror.

Googling about it I just see that SPARK_HOME and PYTHONPATH must be set and contain the path to pyspark.

I think I have done this right, by adding the required exports to spark-env template on ambari spark service configuration and restarting the service. The exports are now present on spark-env.sh on every node, but the issue remains.

I don't know if this is relevant, but I am sending the task using livy from hue's notebook. Both services are running on one of the nodes of the cluster. I have also try adding the exports to /etc/profiles.d/spark-env.sh on each node and restarting both services.

Python spark jobs sent to yarn are failing with an ImportError: No module named pyspark.shellerror. Looking at other answers I just see that SPARK_HOME and PYTHONPATH must be set and contain the path to pyspark. I think I have done this right, by adding the required exports to spark-env template on ambari spark service configuration and restarting the service. The exports are now present on spark-env.sh on every node, but the issue remains. I don't know if this is relevant, but I am sending the task using livy from hue's notebook. Both services are running on one of the nodes of the cluster. I have also try adding the exports to /etc/profiles.d/spark-env.sh on each node and restarting both services. Any ideas on how to fix it?

--

	. Alberto Gael Abadin Martinez Analista Programador - Programmer Analyst www.imatia.com Tel: +34 986 342 774 ext 4537 Email: gael....@imatia.com Edificio CITEXVI Fonte das Abelleiras, s/n - Local 27 36310 Vigo (Pontevedra) España .
	. Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener información confidencial, siendo para uso exclusivo del destinatario. Queda prohibida su divulgación copia o distribución a terceros sin la autorización expresa del remitente. Si usted ha recibido este mensaje erróneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboración. This message, and in the case of any file annexed to it, can have confidential information, and it is exclusively for the use of the addressee of the message. It is strictly forbidden to spread a copy or distribute to third parties, without the express order of the sender. If you have received this message mistakenly, we request you to notify to the sender, and please be sure to erase it. Thank you for your collaboration. .

[Romain Rigaux]

I would recommend https://groups.google.com/a/cloudera.org/group/livy-user for this question. You might have an environment problem.

On the Hue side I use the 0.2 version of http://livy.io/ with Spark 1.6 and it worked for me.

--
You received this message because you are subscribed to the Google Groups "Hue-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

[Gael Abadin]

Thank you, Romain. I'll send the question to them.