ImportError with BigQueryCreateEmptyTableOperator

[nir...@merklescience.com]

from airflow.contrib.operators.bigquery_operator import BigQueryOperator, BigQueryCreateEmptyTableOperator

My DAG which contained the line above is not getting parsed and airflow shows me an error like this

cannot import name BigQueryCreateEmptyTableOperator

Has anyone else here faced this issue on Cloud Composer? This does not seem to be happpening on my local-machine

[Crystal Qian]

What happens when you ssh into the Kubernetes worker pods and try importing in Python?

[Cameron Moberg]

Hello,

Unfortunately I believe this is due to the versioning mis-match between your local branch and Cloud Composer's Airflow version. We currently use Apache Airflow 1.9.0. Due to this, the Operator isn't currently in our Airflow. 

This is the current bigquery_operator.py we are using: 

https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/contrib/operators/bigquery_operator.py

Once we get Cloud Composer stable with version 1.10.0 it will be supported. 

Thanks,

Cameron Moberg

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/5096f46c-d183-4f25-b251-d6270d778f84%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[nir...@merklescience.com]

Awesome! I've found a work-around for that either ways, so it was not much of an issue.

Waiting for the 1.10.0 release, In addition to the point you've mentioned above, I think the BigQueryCheckOperator supports standard-sql in that version, whereas until now that operator has supported only legacy-sql.

BigQueryCheckOperator does not work on time-partitioned tables in cloud-composer for now.

On Tuesday, September 11, 2018 at 7:10:28 AM UTC+8, Cameron Moberg wrote:

Hello,

Unfortunately I believe this is due to the versioning mis-match between your local branch and Cloud Composer's Airflow version. We currently use Apache Airflow 1.9.0. Due to this, the Operator isn't currently in our Airflow. 

This is the current bigquery_operator.py we are using: 

https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/contrib/operators/bigquery_operator.py

Once we get Cloud Composer stable with version 1.10.0 it will be supported. 

Thanks,

Cameron Moberg

On Mon, Sep 10, 2018 at 4:43 PM 'Crystal Qian' via cloud-composer-discuss <cloud-compo...@googlegroups.com> wrote:

What happens when you ssh into the Kubernetes worker pods and try importing in Python?

On Friday, September 7, 2018 at 9:38:03 PM UTC-7, nir...@merklescience.com wrote:

from airflow.contrib.operators.bigquery_operator import BigQueryOperator, BigQueryCreateEmptyTableOperator

My DAG which contained the line above is not getting parsed and airflow shows me an error like this

cannot import name BigQueryCreateEmptyTableOperator

Has anyone else here faced this issue on Cloud Composer? This does not seem to be happpening on my local-machine

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[alakh...@freedomdebtrelief.com]

can you share us the workaround you found for this issue?

Thank you