SSL Error

[Ethan Lyon]

Using the requests library with Google Cloud APIs, I'm receiving the following error:

ERROR - ('Connection aborted.', OSError("(32, 'EPIPE')",))
Traceback (most recent call last):
  File "/opt/python3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 317, in _send_until_done
    return self.connection.send(data)
  File "/opt/python3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1737, in send
    self._raise_ssl_error(self._ssl, result)
  File "/opt/python3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (32, 'EPIPE')

The code works locally, only when it's in Composer does it have an SSL error.

Is there a way to update OpenSSL in Composer?

[Feng Lu]

You can upgrade the openssl version by using the managed python dependency installation feature in Composer. 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAHSbcs9CognXS8J4xzoz3R5aOjteja_Z74oQ%3DxTj-6%2BjWaNEpg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Ethan Lyon]

Hi Feng,

Thank you for the tip. It looks like upgrading requests solved the OpenSSL issue, but created a weird bug with Composer. I'm hoping you can tell me if the new issue is related to upgrading packages in Composer.

Here are the environment packages:

    pypiPackages:
      google-cloud-bigquery: ==1.9.0
      google-cloud-datastore: ==1.7.3
      requests: ==2.22.0
      sendgrid: ==6.0.5
    pythonVersion: '3'

The weird thing is I'm getting a specific task error:

And when I go to check the error task I see this:

The error isn't displayed in the logs. It doesn't show the task has having been completed or stopped by an error.

The odd thing is the beginning of each workflow uses the requests library. Why would a task fail without logging the error? All of the tasks work except that one. I've tried it locally and it functions as expected. It seems to be an issue with the Composer environment failing a task and not showing the logs. 

Is this new issue related to upgrading packages or is it unrelated? Any suggestions for trouble shooting? Without the logs, I'm lost on how to fix the issue.

Thank you in advance,

- Ethan

[Ethan Lyon]

I've continued to investigate the issue and it appears to be a memory issue. A single task can't handle more than a couple MBs of memory. I'm storing 1.56 MBs in a variable and it seems like Composer task instance fails but doesn't convey why to the database, therefore, there isn't an error. 

Is that the right way of looking at it Feng? Would I increase the size to handle data larger than a couple MBs?

- Ethan

[Ethan Lyon]

Any thoughts here Feng?

[Ethan Lyon]

I thought this might be a memory issue so I stopped storing large amounts of data and limited it to 100kb. It worked locally but failed again when uploading to Composer. Without logs in the UI, I went to Stackdriver. There is a log with "detected as zombie" and based on this post, I added 3 retries with the same result: the task fails without any error.

How can I troubleshoot this error on Composer?

[David Sabater Dinter]

Hi Ethan,

We recommend to follow our documentation to test your DAGs before deploying in your production environment.

One important area to look for is about python package conflicts, you can again check our recommended validations as per the official documentation to troubleshoot your DAGs.

Hope that helps to identify your issue.

Kind regards.

Any thoughts here Feng?

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Sergei Guschin]

Ethan,

I struggled with debugging tasks as well. I saw in another post how to get into composer worker pod and see what's going on down there. I found slightly easier way: Go to your Composer Environment -> GKE Cluster Details (view cluster details) - remember cluster name - in kubernetes engine go to Workloads - filter on cluster name - and click on 'airflow-worker' - in managed pod you will see you workers - click on one - at top above charts you will see KUBECTL drop down - Attach>airflow-worker - at the bottom screen you will see system console and everything outputted will be here, as well errors which failed to make to Stackdriver.

Sergei