How to change Python version for Pyspark Action Plugin

[shreyasi misra]

I am running cdap sandbox locally on my system. 

My system (Fedora 36) has Python3.10 by default which the pyspark action plugin is utilizing.

However, due to updates in this new version of python, collections package has been affected.

When I try to run the default code in Pyspark plugin, i.e:

from pyspark import *
from pyspark.sql import *
from cdap.pyspark import SparkExecutionContext

sec = SparkExecutionContext()
sc = SparkContext()

I get the error : 

ImportError: cannot import name 'MutableMapping' from 'collections' (/usr/lib64/python3.10/collections/__init__.py)

(I have attached the log from this error below)

If I try to correct that by importing packages as a workaround, I get stuck at:

ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib64/python3.10/collections/__init__.py)

I have installed Python3.9 and wanted to direct the python path to this in cdap.

How could I direct the Python path to 3.9 version?

Thank you

[Christophe DIEDHIOU]

Hi how are you,

I think the python plugin in cdap has no version 3.xx see the last python version of the plugin and adapt your script

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/ca5a96d8-59a5-425d-84f3-c1f56e5e2fd9n%40googlegroups.com.

[shreyasi misra]

Hi, I am good, thanks. Hope you are well too.

This is the source code for the plugin I deployed from the Hub

https://github.com/cdapio/hub/tree/deploy/packages/hydrator-plugin-dynamic-spark/2.2.3 

Could you direct me a little on how to check the python version and make the necessary changes to make my code work?

Thanks

[Christophe DIEDHIOU]

Check in cdap documentation if it is there.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/4d0fd928-31f0-4e63-a0ea-f795cbc6f234n%40googlegroups.com.

[shreyasi misra]

I did check there but couldn't find anything related to this. Any other leads?

[Christophe DIEDHIOU]

Do this to find the jar you have

ls cdap-home/data/namespaces/the_name_of_namespace/artifacts/python-transform/

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/9f84dd8f-a930-4b6a-801f-5822124e675en%40googlegroups.com.

[shreyasi misra]

Hi, Thanks for the help.

However, I had to go through a different route for it. Thus, posting it here in case someone else is also stuck.

-Install the required python version on your system

-Install a virtual environment library like virtualenv

-Create a virtual env referencing the python version to create one with

-Add the required python packages for pyspark action plugin

-Activate virtual env, start cdap sandbox

Now, the processes will use the python and packages made available in the virtual environment.