[JIRA] (PLUGIN-946) Python plugin fails when when used in Native mode.

Sagar Kapare (Jira)

unread,

Oct 15, 2021, 2:03:50 PM10/15/21

to cdap-...@googlegroups.com

Sagar Kapare created an issue

CDAP Plugins /

PLUGIN-946

Python plugin fails when when used in Native mode.

Issue Type:	Bug
Assignee:	Unassigned
Created:	15/Oct/21 11:03 AM
Priority:	Major
Reporter:	Sagar Kapare

Python plugin fails when used in the Native mode. Following error is seen in the logs

 
                              2021-09-30 19:12:38,164 - ERROR [Executor task launch worker for task 0.0 in stage 0.0 (TID 0):i.c.p.p.t.Py4jPythonExecutor@219] - An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 977, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1115, in start
    self.socket.connect((self.address, self.port))
  File "/usr/lib/python3.7/ssl.py", line 1150, in connect
    self._real_connect(addr, False)
  File "/usr/lib/python3.7/ssl.py", line 1137, in _real_connect
    super().connect(addr)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:
 
                              

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS

This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100183-sha1:24053be)

Sagar Kapare (Jira)

unread,

Oct 15, 2021, 2:07:00 PM10/15/21

to cdap-...@googlegroups.com

Sagar Kapare commented on

PLUGIN-946

Re: Python plugin fails when when used in Native mode.

For Native mode we have to provide PYTHONPATH config containing path to the py4j-<version>-src.zip For example on my Hadoop cluster zip was located at following path: /usr/lib/spark/python:/usr/lib/spark/python/lib/py4j-0.10.9-src.zip:$PYTHONPATH

Even after giving PYTHONPATH, plugin now fails with exception mentioned in the bug.

We tried different memory configurations for the job based on some online research but didnt help.