Error in KubernetesPodOperator

[Kadu Vido]

Hello,

I have use KubernetesPodOperator launching docker images in all my DAGs, but my latest deployment has been raising some odd warnings and errors that I have never seen.

I'm running composer-1.16.7-airflow-1.10.15, with secret manager as the secrets backend. No other DAGs are having this issue.

My DAG consists of a BranchOperator launching one of 3 different KubernetesPodOperators on specific occasions, all run the same docker image with different env vars (so, essentially, my branch operator just picks the env vars for the next task).

All of the failed tasks have this warning, which I've never seen before:

WARNING - /usr/local/lib/airflow/airflow/kubernetes/pod_launcher.py:331: DeprecationWarning: Using `airflow.contrib.kubernetes.pod.Pod` is deprecated. Please use `k8s.V1Pod`

After the warning, either the task takes a while and fails with this error:

ERROR - Pod Launching failed: Pod Launching failed: Pod took too long to start

Or it fails immediately with this one:

ERROR - Exception when attempting to create Namespaced Pod
(...)
kubernetes.client.rest.ApiException: (400 Reason: Bad Reques HTTP response headers: HTTPHeaderDict({'Audit-Id': '85d06cfd-abb4-4fb2-88d5-ca55926a52ed', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Thu, 01 Jul 2021 00:01:31 GMT', 'Content-Length': '520'} HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod:(...)

The only change between these two is the env vars dictionary. I have no idea how this would cause this type issue, but for what it's worth, here's what they look like respectively:

{"ACTION": "SELL","GOOGLE_CREDENTIALS": {{var.value.commitment_manager_credentials}}",}

{"ACTION": "BUY", "COMMITMENT_SIZE": 400, "GOOGLE_CREDENTIALS": "{{var.value.commitment_manager_credentials}}",}

I checked and the credentials are rendering correctly, so that's not it. I'm completely stumped, if anyone has any ideas I'm open to suggestions.

[cole...@google.com]

Oops - I think I just did reply to author the first time, but I wanted to post my reply publicly too for other folks as well

Hey! I have a theory. There was a bug (https://github.com/apache/airflow/issues/14287) in the KubernetesPodOperator that had to do with environment variable templating, which was fixed in the `apache-airflow-backport-providers-cncf-kubernetes` March release. This is installed in the version of Composer you're using. However, I think that based on the warning that you're seeing, the WARNING - /usr/local/lib/airflow/airflow/kubernetes/pod_launcher.py:331: DeprecationWarning: Using `airflow.contrib.kubernetes.pod.Pod` is deprecated. Please use `k8s.V1Pod`, it might not be the version you're importing. To use the KubernetesPodOperator from the backport providers package, you would want to change the import in your DAG from

`from airflow.contrib.operators import kubernetes_pod_operator` to ` from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator`. This should at least resolve the first warning (If you're still seeing other warnings like that, it might be because you have other imports you need to adjust similarly.)

Regarding pods taking too long to start, this can be caused by a number of issues, and if you take a look at your cluster workloads in GKE, you might get more detailed error messages in the pods themselves rather than just the "Pod took too long to start" which gets surfaced. I think that those underlying errors will be different than the immediate environment variable failure you're seeing, because I think that the cause of that environment variable failure is the issue I linked earlier. But again, if not, please let me know. :) 

[Kadu Vido]

Hello,

Thanks for your reply, I've tried using airflow.providers.cncf.kubernetes.operators.kubernetes_pod to no effect - still getting the same deprecation warning and the same set of errors. I started seeing the deprecation warning on other errors too, at least, but they do what they're supposed to do.