Forbidden opperation for "default" user in KubernetesPodOperator

[abor...@slb.com]

I am trying to run a simple KubernetesPodOperator in my Composer environment as per the documentation here.

The airflow runtime is failing due to lack of permission for the user "default".

I can see people using this operator, so it may be related to how I created the environment...

That said, how to properly create an environment or to set up default user permissions in order for this code to work?

Creation command:

gcloud composer environments create spwarehouse-composer \
 --location $REGION --zone $ZONE \
 --machine-type n1-standard-1 --disk-size 50 \
 --image-version composer-latest-airflow-1.10 \
 --python-version 3 \
 --labels env=worker

DAG:

    price_analysis = KubernetesPodOperator(

        task_id='price-analysis',
        name='price-analysis',
        namespace='default',
        image='bash',
        image_pull_policy='Always',
        cmds=['echo'],
        arguments=['something'],
        env_vars={
            'EXPOSURE_THRESHOLD': '5',
            'ESTIMATE_WINDOW': '3,7',
        },
        in_cluster=True,
    )

Logs:

-------------------------------------------------------------------------------

Starting attempt 1 of
-------------------------------------------------------------------------------


[2019-04-03 14:54:15,611] {models.py:1595} INFO - Executing <Task(KubernetesPodOperator): price-analysis> on 2019-04-03T14:53:59.658367+00:00
[2019-04-03 14:54:15,612] {base_task_runner.py:118} INFO - Running: ['bash', '-c', u'airflow run vat-analysis price-analysis 2019-04-03T14:53:59.658367+00:00 --job_id 54 --raw -sd DAGS_FOLDER/vat_analysis_dag.py --cfg_path /tmp/tmp3RdZOV']
[2019-04-03 14:54:18,375] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:18,374] {settings.py:176} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2019-04-03 14:54:19,652] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,651] {default_celery.py:80} WARNING - You have configured a result_backend of redis://airflow-redis-service.default.svc.cluster.local:6379/0, it is highly recommended to use an alternative result_backend (i.e. a database).
[2019-04-03 14:54:19,659] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,659] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-04-03 14:54:19,826] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,825] {app.py:51} WARNING - Using default Composer Environment Variables. Overrides have not been applied.
[2019-04-03 14:54:19,842] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,842] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-04-03 14:54:19,868] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,867] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-04-03 14:54:20,380] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:20,378] {models.py:271} INFO - Filling up the DagBag from /home/airflow/gcs/dags/vat_analysis_dag.py
[2019-04-03 14:54:21,490] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:21,490] {cli.py:484} INFO - Running <TaskInstance: vat-analysis.price-analysis 2019-04-03T14:53:59.658367+00:00 [running]> on host airflow-worker-5b6d7c75c9-w6995
[2019-04-03 14:54:22,093] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:21,822] {pod_launcher.py:58} ERROR - Exception when attempting to create Namespaced Pod.
[2019-04-03 14:54:22,103] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis Traceback (most recent call last):
[2019-04-03 14:54:22,107] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 55, in run_pod_async
[2019-04-03 14:54:22,113] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
[2019-04-03 14:54:22,116] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6115, in create_namespaced_pod
[2019-04-03 14:54:22,122] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
[2019-04-03 14:54:22,126] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6206, in create_namespaced_pod_with_http_info
[2019-04-03 14:54:22,129] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     collection_formats=collection_formats)
[2019-04-03 14:54:22,134] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 321, in call_api
[2019-04-03 14:54:22,150] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     _return_http_data_only, collection_formats, _preload_content, _request_timeout)
[2019-04-03 14:54:22,155] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 155, in __call_api
[2019-04-03 14:54:22,159] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     _request_timeout=_request_timeout)
[2019-04-03 14:54:22,138] {models.py:1760} ERROR - (403
Reason: Forbidde
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 03 Apr 2019 14:54:21 GMT', 'Audit-Id': 'c027d4cb-5186-498a-a9b5-0e6c4420b816', 'Content-Length': '284', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:composer-1-6-0-airflow-1-10-1-ea0745b4:default\" cannot create pods in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403

[abor...@slb.com]

I tried GKEPodOperator as well, same results (403)...

[Nidhin Pattaniyil]

I had a similar issue, but I "solved" it by running 

CLUSTER_NAME=.....
NAMESPACE=k8s-tasks
kubectl create ns ${NAMESPACE}

kubectl create clusterrolebinding default-admin \
    --clusterrole cluster-admin \
    --serviceaccount=default:default \
    --namespace ${NAMESPACE}

Note: I am creating/using a new namespace "k8s-tasks"

hope that helps

[abor...@slb.com]

It worked indeed! Thank you.

I hope the next version of the Google Composer will make it clear on how to configure that.

Regards.

[Mayank Kumar]

Composer 2 now provides a dedicated namespace `composer-user-workloads` where these task pods/workloads should be run. The `default` service account for this namespace already has the correct binding (Workload Identity) set up with the Composer SA, so no additional steps need to be performed by the user.

In our case, however, we had to additionally specify the `config_file` parameter and set it to `/home/airflow/composer_kube_config` to get past the permissions error we experienced (even though the Google documentation says this is not a mandatory parameter).

reference: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/composer/workflows/kubernetes_pod_operator_c2.py#L66

[Dan Ladd]

Thanks for that tip. 

I was getting the following error before adding the config_file argument:

kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:composer-2-1-11-airflow-2-4-3-2d1b795a:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"composer-user-workloads\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

[Boris Shilov]

Thank you for the fix and the error description everyone. We just hit this and it would have taken us a very long time to find the root cause is this default parameter that explicitly contradicts the documentation. We were able to get everything working again in an hour thanks to the help here.

[Mayank Kumar]

You're welcome! (I've been a heavy consumer of the open-source community for a long time, and now it feels great that my small contribution helped someone else, so thank you for your message :) )

Btw, anyone has an idea on how this can be reported to the GCP team, so this is fixed? (either in the code or in the documentation).

thank you,

[cole...@google.com]

Please use the "Send Feedback" page directly on the documentation!

[Nahid Oulmi]

Thank you for the information & fix ! 

Adding the parameters

namespace='composer-user-workloads',

config_file="/home/airflow/composer_kube_config"

to my Pod Operators also fixed the issue after upgrading to composer-2.4.1-airflow-2.5.3. Now I just need to update all my dags !

[Ashwani Kumar]

Thanks for this. 
It works and as I understand, The `default` service account for  `composer-user-workloads`  namespace already has the correct binding (Workload Identity) set up with the Composer SA. Which means all the pipelines would run with the same single Composer SA. 
this is not helpful in cases when we want to segregate the BigQuery cost per SA. On way to work around is to create custom namespace and KSA, and binding ourselves right. But in our org , we dont have kubectl access to the underlying GKE cluster as we can't create VMs to get that access.
Is there any other  work around? Any one else encountered this situation., How did you deal with it please?