Accessing kube_client in DAG file code

[Amir Amangeldi]

Hi all,

What is the recommended way to initiate kube_client from the scheduler, specifically for the purposes of extracting values of k8s Secrets? I am programmatically generating tasks and would like to pass in values of associated k8s Secrets to each task.

The following code works as part of a PythonOperator callable, but not outside of the scope of the operator:

from airflow.contrib.kubernetes import kube_client

client = kube_client.get_kube_client(

in_cluster=False,

cluster_context=None,

config_file='/home/airflow/composer_kube_config',

)

secrets = client.list_secret_for_all_namespaces()

The error I'm receiving is: Broken DAG: [/home/airflow/gcs/dags/test_amir.py] [Errno 2] No such file or directory: '/home/airflow/composer_kube_config'.

I dropped a shell onto Airflow scheduler and worker pods, and found out that /home/airflow/composer_kube_config file only exists on the workers (not the scheduler). 

Is it safe to re-create the same file on the scheduler?

I am using composer-1.4.2-airflow-1.10.0.

Any advice is much appreciated.

Thank you,
Amir Amangeldi

EVERQUOTE  |  Software Engineer

[Tanjin Panna]

Hey Amir,

I ran into this issue recently but in a slightly different manner. For me I was missing the `composer_kube_config` file in some of my workers. I did your suggestion (recreate the file in my other workers) and now it is no longer an issue.

I still don't know why this had occurred, I am on CC 1.6.1 w/ Airflow 1.10.1. I know now to check for the presence of that file in the worker pods moving forward if I were to get this error again.

Curious to hear if you've reached any resolution for your particular use case.

[Amir Amangeldi]

Hi Tanjin,

We ended using an operator since the workers have the ability to use kube_client similarly to KubernetesPodOperator. Our operator simply extends PythonOperator; it reads k8s ConfigMaps and Secrets, encrypts appropriately, and shares with the other tasks via xcom.

This worked fine for us but can be a problem for generating tasks dynamically using the data in k8s Secrets and ConfigMaps.

Cheers,

Amir

--
You received this message because you are subscribed to a topic in the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cloud-composer-discuss/q-YzQ9CmZ-k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/1b3d2d6f-1813-4934-9ce1-ee64cb34c1b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.