Connection error in client streaming with python

[manoj92...@gmail.com]

Hi,

 

I am using CDAP version 4.1.1 with Docker.

I have written a python script for ingesting data to my data pipeline, using CDAP streaming client for python. My item size is of 489 Bytes. I am ingesting  data item to CDAP in a while loop.

But after some time it will report a connection problem for  different threads. 

one of the thread -- 

Exception in thread Thread-16285:

Traceback (most recent call last):

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\connection.py", line 138, in _new_conn

    (self.host, self.port), self.timeout, **extra_kw)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\util\connection.py", line 98, in create_connection

    raise err

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\util\connection.py", line 88, in create_connection

    sock.connect(sa)

OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 594, in urlopen

    chunked=chunked)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 361, in _make_request

    conn.request(method, url, **httplib_request_kw)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\http\client.py", line 1239, in request

    self._send_request(method, url, body, headers, encode_chunked)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\http\client.py", line 1285, in _send_request

    self.endheaders(body, encode_chunked=encode_chunked)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\http\client.py", line 1234, in endheaders

    self._send_output(message_body, encode_chunked=encode_chunked)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\http\client.py", line 1026, in _send_output

    self.send(msg)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\http\client.py", line 964, in send

    self.connect()

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\connection.py", line 163, in connect

    conn = self._new_conn()

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\connection.py", line 147, in _new_conn

    self, "Failed to establish a new connection: %s" % e)

requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.HTTPConnection object at 0x000001DD3CE3F1D0>: Failed to establish a new connection: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\adapters.py", line 423, in send

    timeout=timeout

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 643, in urlopen

    _stacktrace=sys.exc_info()[2])

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\packages\urllib3\util\retry.py", line 363, in increment

    raise MaxRetryError(_pool, url, error or ResponseError(cause))

requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.99.100', port=32768): Max retries exceeded with url: /v3/namespaces/default/streams/Live2 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x000001DD3CE3F1D0>: Failed to establish a new connection: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\threading.py", line 916, in _bootstrap_inner

    self.run()

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\threading.py", line 864, in run

    self._target(*self._args, **self._kwargs)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\cdap_stream_client-1.3.0_snapshot-py3.6.egg\cdap_stream_client\streampromise.py", line 130, in __worker_target

    u'POST', uri, dataToSend, headersToSend)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\cdap_stream_client-1.3.0_snapshot-py3.6.egg\cdap_stream_client\serviceconnector.py", line 89, in request

    verify=self.__connectionConfig.ssl_cert_check)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\api.py", line 56, in request

    return session.request(method=method, url=url, **kwargs)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\sessions.py", line 488, in request

    resp = self.send(prep, **send_kwargs)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\sessions.py", line 609, in send

    r = adapter.send(request, **kwargs)

  File "C:\Users\manoj8\AppData\Local\Continuum\Anaconda3\lib\site-packages\requests\adapters.py", line 487, in send

    raise ConnectionError(e, request=request)

requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.99.100', port=32768): Max retries exceeded with url: /v3/namespaces/default/streams/Live2 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x000001DD3CE3F1D0>: Failed to establish a new connection: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted',))

Can any one please help me to get out of this situation ?

Thanks,

Manoj

[Ali Anwar]

Hi Manoj.

Based upon the logs, it seems like you're hitting an issue of TCP/IP port exhaustion. This is due to the fact that the python stream client is not efficient in reusing connections in all cases, for which I filed CDAP-11443.
The python stream client could be improved to reuse connections in all cases, but in the meantime, there are two possible workarounds:
1. Leverage the Java StreamClient, which is more robust.
2. Increase the upper range of ephemeral ports or reduce the TCP/IP socket connection timeout value, as documented here.

Regards,

Ali Anwar

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/fa0ffadc-eaa7-4636-8caf-60e5a3aa3770%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[manoj92...@gmail.com]

Hi Ali,

Thank you very much for this information, I will go for JAVA Stream Client.

Regards,

Manoj