Using xcom_pull from KubernetesPodOperator

[MH]

Hi,

How do I use xcom_pull inside KubernetesPodOperator?

Any example is appreciated.

TIA

[Anthony Brown]

You can only use it in the templated fields - which in the KubernetesPodOperator are ('cmds', 'arguments', 'env_vars', 'config_file')

You should be able to do something like

    RUN_POD = kubernetes_pod_operator.KubernetesPodOperator(
        task_id='run_pod',
        name='run-pod',
        env_vars={
            'VAR1_FROM_XCOM': '{{ task_instance.xcom_pull(task_ids="task_xcom_set", key="xcom_key_name") }}',
        },
        image='MY_IMAGE')

In your container image, the environment variable VAR1_FROM_XCOM is set to the value of the xcom at run time

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/a2431ebb-7f73-41ac-94d3-443d13c12c56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

-- 

Anthony Brown

Data Engineer BI Team - John Lewis

Tel : 0787 215 7305

**********************************************************************
This email is confidential and may contain copyright material of the John Lewis Partnership.
If you are not the intended recipient, please notify us immediately and delete all copies of this message.
(Please note that it is your responsibility to scan this message for viruses). Email to and from the
John Lewis Partnership is automatically monitored for operational and lawful business reasons.
**********************************************************************

John Lewis plc
Registered in England 233462
Registered office 171 Victoria Street London SW1E 5NN

Websites: https://www.johnlewis.com
http://www.waitrose.com
https://www.johnlewisfinance.com
http://www.johnlewispartnership.co.uk

**********************************************************************

[hd...@toreta.in]

If you are using the KubernetesPodOperator, it will try to read from a file located at /airflow/xcom/return.json in a sidecar container.

https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py#L147-L162

If you are using the GKEPodOperator, I believe there is a bug preventing it from returning the result.

https://issues.apache.org/jira/browse/AIRFLOW-3113

Anyway, we ended up with backporting the GKEPodOperator as a custom plugin.

[Feng Lu]

fyi- GkePodOperator is already supported out-of-the-box since Airflow 1.1.0.

--

You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/f9467012-5108-485d-93ad-7237fe6001a8%40googlegroups.com.

[jasper...@gmail.com]

Halo, 

I have tried to use GKEPodOperator but I can't get the xcom_push from the operator. Is there any difference of xcom_push between GKEPodOperator and KubernetesPodOperator?

Thanks a lot.

Feng Lu於 2018年10月11日星期四 UTC+8下午12時32分52秒寫道：

fyi- GkePodOperator is already supported out-of-the-box since Airflow 1.1.0.

On Mon, Oct 8, 2018 at 10:02 PM <hd...@toreta.in> wrote:

If you are using the KubernetesPodOperator, it will try to read from a file located at /airflow/xcom/return.json in a sidecar container.

https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py#L147-L162

If you are using the GKEPodOperator, I believe there is a bug preventing it from returning the result.

https://issues.apache.org/jira/browse/AIRFLOW-3113

Anyway, we ended up with backporting the GKEPodOperator as a custom plugin.

On Monday, October 8, 2018 at 5:01:16 PM UTC+9, MH wrote:

Hi,

How do I use xcom_pull inside KubernetesPodOperator?

Any example is appreciated.

TIA

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Shehzan Devani]

Looks like there was a very recent PR to enable xcom on GKEPodOperator: https://github.com/apache/airflow/pull/4905.