Issue when deploying airflow container on kubernetes.

[Ashish r]

We currently are containerizing the airflow application Below is the configuration for the same

Master Pod

1. Airflow schedular

2. Airflow webserver

3. Airflow flower

Worker pod

1. Airflow worker

Redis pod

1. Redis

Mariadb pod

1. Mariadb

We have a default airflow.cfg that has the broker and sql endpoint set to 8.8.8.8 incorrect entries. We build the Docker Images using the Default.cfg . We update the configuration file using config map in kubernetes, but we don't directly update the default.cfg but create a symbolic link from the corrected configuration file to the default.cfg in the container. (We do this because we can not change the config file directly as it is on the read-only filesytem)

Issue:

1. When the container is started it is pointing to the incorrect endpoint for broker and sql as 8.8.8.8, but when we check the airflow.cfg file it does point to the correct endpoint. So the symbolic link works, but the problem is the service doesn't pick up the configuration file that was mapped using config map. This does not happen reguarly, but happen intermittently as it sometimes pick the correct value if not it restart the service and then correct the service to use correct config file.

I think the service is not being restarted after the correct config file is mounted to the container. 

We are trying to find the best way to resolve this issue.