Composer v2 webserver unstable

[Tobias Kaymak]

Hello,

we are migrating to Composer v2 from Composer v1 (1.10.15).

After creating a medium sized environment, adjusting our DAGs, and copying them and their plugins we see that the webserver is marked unstable and being restarted a couple of times. The webserver has enough resources in terms of RAM, CPU and disk space.

Is there anything we should look at? We already tried creating a new environment from scratch so far.

Best,
Tobi

[Uri Goldstein]

Hi Tobi,

I'm no expert on Airflow/Composer but perhaps you can try looking at the health check that precedes the restarts?

From the log you posted it looks like something is "curling" /_ah/Health successfully (?) but 2 seconds later a hang-up signal is sent to make the webserver restart.

Maybe a closer look at Airflow's health check feature could show you why it think the webserver is unstable?

Cheers,

Uri

[raphael auv]

the timeout of gunicorn is too short on composer , and not customizable
https://stackoverflow.com/questions/56810208/cloud-composer-airflow-web-server-worker-timeout-environmental-variable/69829924#69829924

[Shyam Ashar]

Try to increase the webserver workers. By default it is 2 and if you have a lot of DAGs the webserver becomes very unstable. The guincorn documentation suggests that the workers should be 2x + 1 the number of cores. As a managed service Cloud Composer should ideally come with these settings but looks like they dont do it out of the box which is disappointing.

https://docs.gunicorn.org/en/0.16.1/design.html#:~:text=Gunicorn%20should%20only%20need%204,workers%20to%20start%20off%20with.

Regards,

Shyam

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/32f112dd-0d9f-4e4f-83d5-57dbf047925an%40googlegroups.com.

[Tobias Kaymak]

Thank you for your responses.

We have only 39 DAGs and I just tried setting the number of webserver workers to 3 and 4 - with no effect on the webserver health.

I have the feeling that this is a bug in the Composer v2 setup itself but I am out of ideas where to look.

Best,
Tobias

[Tobias Kaymak]

What I also found is that for the webserver proxy there seems to be a misconfiguration that I can't change:

[Tobias Kaymak]

I digged deeper and I am pretty sure this is a misconfiguration of the v2 build:
I disabled the healthyness and readyness probes on the webserver pod in the GKE cluster running composer v2.
Then I still saw the following messages in the logs:

This IP (10.63.129.6) is belonging to the airflow-monitoring pod in the composer-system namespace. It's poking and then 2 seconds later the HUP signal is being received by gunicorn.

Investigating on the monitoring pod is not so easy as its secured through GKEAutopilot's authz.

[Tobias Kaymak]

Further finding: The gcs-sync sidecar pod seems to cause the webserver to error-out when it's cleaning the pyc files, which looks like a race-condition:

[The example file here is a plugin from the plugin folder that's being cleaned by the gcs-sync sidecar, then the webserver pod can't find it and shuts down]

Removing file:///home/airflow/gcs/plugins/operators/__pycache__/trigger_emarsys_event_operator.cpython-38.pyc
{webserver_command.py:217} ERROR - [Errno 2] No such file or directory: '/home/airflow/gcs/plugins/operators/__pycache__/trigger_emarsys_event_operator.cpython-38.pyc'
{webserver_command.py:218} ERROR - Shutting down webserver

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/db28b8ed-61d1-4ad0-8a5c-726b8c1f1ea4n%40googlegroups.com.