ERROR - Task received SIGTERM signal

[Thuy Le]

Hi all, I meet this issue ERROR - Task received SIGTERM signal

 
[2019-08-25 15:15:36,136] {base_task_runner.py:101} INFO - Job 293: Subtask wait_for_staging_bq_act_card_acct_map [2019-08-25 15:15:36,136] {models.py:1641} ERROR - Received SIGTERM. Terminating subprocesses.
[2019-08-25 15:15:36,269] {models.py:1796} ERROR - Task received SIGTERM signal
Traceback (most recent call last)
  File "/usr/local/lib/airflow/airflow/models.py", line 1664, in _run_raw_tas
    result = task_copy.execute(context=context
  File "/usr/local/lib/airflow/airflow/sensors/base_sensor_operator.py", line 112, in execut
    sleep(self.poke_interval
  File "/usr/local/lib/airflow/airflow/models.py", line 1643, in signal_handle
    raise AirflowException("Task received SIGTERM signal"
airflow.exceptions.AirflowException: Task received SIGTERM signa
[2019-08-25 15:15:36,270] {base_task_runner.py:101} INFO - Job 293: Subtask wait_for_staging_bq_act_card_acct_map [2019-08-25 15:15:36,269] {models.py:1796} ERROR - Task received SIGTERM signal

Do I need some special configuration?
Any suggestion to solved that issue?

many thanks

[Ethan Lyon]

I ran into the same issue, increased the Composer environment and it was resolved. Have you tried that?

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/ff228254-6612-48c0-89e3-6718e7fd4e9e%40googlegroups.com.

[Thuy Le]

Hi Ethan,

thanks for your suggestion, can you specify how did you increase the composer environment?

many thanks

[Ethan Lyon]

I created a new environment with a higher memory and nodes and killed the old one. Does that make sense?

[Thuy Le]

My environment has 3 node (each node 30GB) so total 90GB,

Machine type

n1-standard-8

Do you think it is enough?

[Kiran Arshewar]

Getting the same error on three node, my config are as below

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAE4G%2B9VCPuvn9YhrfArJAhL5pJfQTEQt5raPv-WHV8DXuU7uxA%40mail.gmail.com.

Privileged/Confidential Information may be contained in this message. If you are
not the addressee indicated in this message (or responsible for delivery of the
message to such person), you may not copy or deliver this message to anyone. In
such case, you should destroy this message and kindly notify the sender by reply
email. Please advise immediately if you or your employer does not consent to email
for messages of this kind. Opinions, conclusions and other information in this
message that do not relate to the official business of Group M Worldwide LLC and/or
other members of the GroupM group of companies shall be understood as neither given
nor endorsed by it. GroupM is the global media investment management arm of WPP.
For more information on our business ethical standards and Corporate Responsibility
policies please refer to WPP's website at http://www.wpp.com/WPP/About/

[Jiawei Zhang]

Hi,

I'm getting the same error 'ERROR - Received SIGTERM. Terminating subprocesses.' when running an auto-sklearn model on composer. I'm using the following configurations:

Anyone knows how to solve this? How should I update my environment?

Thanks!

[Ethan Lyon]

We ran into this issue because of a single process consuming all the resources. We moved to an autoscaling environment with 5 max nodes and haven't had an issue yet.

Have you tried autoscaling? 

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/a5ce4d1d-77bf-48cc-9127-e37910914897n%40googlegroups.com.

[Ethan Lyon]

Sorry for the double email. You mentioned using auto-sklearn. Have you tried moving your ML to another service? Something like ML Engine.

That might prevent your Composer environment from doing the heavy lifting. 

On Wed, Jun 24, 2020 at 3:54 PM Jiawei Zhang <jiawei...@mail.dfci.harvard.edu> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/a5ce4d1d-77bf-48cc-9127-e37910914897n%40googlegroups.com.