Prod Incident : Airflow environment health - unhealthy

[Debodirno Chandra]

Hello everyone,

Need some urgent guidance how to avoid this in future for such a long duration.

Airflow worker died because of high disk usage, however, no new worker came up for a very long time and it affected our production.

Any idea why this could have happened? How can we mitigate this? As in, if one worker dies, the other comes up real fast.

We had to manually change the environment config (worker memory) and saw that a new worker came up with that change.

Any pointers?