Task complete successfully but log contain SIGTERM

735 views
Skip to first unread message

Sergei Guschin

unread,
May 18, 2020, 11:18:47 AM5/18/20
to cloud-composer-discuss
With some regularity my tasks have SIGTERM at the end, while all work is completed without any issue, SUCCESS in database and after last line - do anyone having same issues?

My first take is GCS client need force exit

[2020-05-17 07:00:34,255] {helpers.py:308} INFO - Sending Signals.SIGTERM to GPID 152278
[2020-05-17 07:00:34,256] {taskinstance.py:905} ERROR - Received SIGTERM. Terminating subprocesses.
[2020-05-17 07:00:34,364] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu [2020-05-17 07:00:34,256] {taskinstance.py:905} ERROR - Received SIGTERM. Terminating subprocesses.
[2020-05-17 07:00:34,365] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu Traceback (most recent call last):
[2020-05-17 07:00:34,365] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu   File "/opt/python3.6/lib/python3.6/logging/__init__.py", line 1048, in close
[2020-05-17 07:00:34,366] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu     stream.close()
[2020-05-17 07:00:34,366] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu OSError: [Errno 5] Input/output error
[2020-05-17 07:00:34,367] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu 
[2020-05-17 07:00:34,377] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu During handling of the above exception, another exception occurred:
[2020-05-17 07:00:34,386] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu 
[2020-05-17 07:00:34,386] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu Traceback (most recent call last):
[2020-05-17 07:00:34,386] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu   File "/usr/local/bin/airflow", line 7, in <module>
[2020-05-17 07:00:34,386] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu     exec(compile(f.read(), __file__, 'exec'))
[2020-05-17 07:00:34,386] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu   File "/usr/local/lib/airflow/airflow/bin/airflow", line 37, in <module>
[2020-05-17 07:00:34,386] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu     args.func(args)
[2020-05-17 07:00:34,387] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu   File "/usr/local/lib/airflow/airflow/utils/cli.py", line 74, in wrapper
[2020-05-17 07:00:34,387] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu     return f(*args, **kwargs)
[2020-05-17 07:00:34,387] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu   File "/usr/local/lib/airflow/airflow/bin/cli.py", line 552, in run
[2020-05-17 07:00:34,387] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu     logging.shutdown()
[2020-05-17 07:00:34,387] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu   File "/opt/python3.6/lib/python3.6/logging/__init__.py", line 1946, in shutdown
[2020-05-17 07:00:34,387] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu     h.close()
[2020-05-17 07:00:34,387] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu   File "/opt/python3.6/lib/python3.6/logging/__init__.py", line 1048, in close
[2020-05-17 07:00:34,388] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu     stream.close()
[2020-05-17 07:00:34,398] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu   File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 907, in signal_handler
[2020-05-17 07:00:34,408] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu     raise AirflowException("Task received SIGTERM signal")
[2020-05-17 07:00:34,418] {base_task_runner.py:115} INFO - Job 938164: Subtask tsmc_wli_map_processing_dlfpu airflow.exceptions.AirflowException: Task received SIGTERM signal
[2020-05-17 07:00:35,191] {helpers.py:286} INFO - Process psutil.Process(pid=152278, status='terminated') (152278) terminated with exit code 1

Ethan Lyon

unread,
May 18, 2020, 11:52:44 AM5/18/20
to Sergei Guschin, cloud-composer-discuss
I used to have this issue and spent hours trying different things out. The issue went away when I increased the size of my environment. 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/ebdbd270-733c-462d-91b9-61770a7d665e%40googlegroups.com.

Sergei Guschin

unread,
Jun 1, 2020, 4:37:11 PM6/1/20
to cloud-composer-discuss
Ethan,

From what size to what size you changed.

Note: my tasks is very lightweight and mostly observed if i use GCS Operator/Hook.

Sergei
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

Ethan Lyon

unread,
Jun 1, 2020, 4:46:11 PM6/1/20
to Sergei Guschin, cloud-composer-discuss
We were on the smallest machines and increased the size to the next one up. We had thousands of tasks running, which I'm guessing jammed up the scheduler. If I were to do it again, I'd set up an auto-scaler.

- Ethan

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/c0532a60-6b26-4c33-a587-45e885558e01%40googlegroups.com.

Sergei Guschin

unread,
Jun 3, 2020, 12:05:47 PM6/3/20
to cloud-composer-discuss
hm, in my case i am already not on the smallest machines (n1-highmem-8) 

Sergei
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

Ethan Lyon

unread,
Jun 3, 2020, 12:07:28 PM6/3/20
to Sergei Guschin, cloud-composer-discuss
Have you tried auto-scaling your nodes?

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/bc1a292e-ca2f-4920-b2c6-8fdbfb8640a8%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages