cloud storage to BQ with autodetect schema

[zu...@google.com]

hi,

I need load some .CSV files to BQ and I want to set the "autodetect schema" option in GoogleCloudStorageToBigQueryOperator 

what should I pass to "schema_fileds", "schema_objects" 

thanks

Antonio

[Wilson Lian]

Hi,

This is a known issue (https://issues.apache.org/jira/browse/AIRFLOW-461). You could work around this with the BashOperator and "bq load --autodetect".

best,

Wilson

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.
To post to this group, send email to cloud-composer-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/38fce52b-98eb-4e40-8bff-a1dc5df6cae5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Antonio Zurlo]

thanks,

that's what I did and it works fine.

how can I trigger a dag from a command line? something equivalent of  airflow trigger_dag mydag

thanks again

Antonio

On Mon, Jun 4, 2018 at 4:36 PM Wilson Lian <wwl...@google.com> wrote:

Hi,

This is a known issue (https://issues.apache.org/jira/browse/AIRFLOW-461). You could work around this with the BashOperator and "bq load --autodetect".

best,

Wilson

On Mon, Jun 4, 2018 at 8:07 AM, zurlo via cloud-composer-discuss <cloud-compo...@googlegroups.com> wrote:

hi,

I need load some .CSV files to BQ and I want to set the "autodetect schema" option in GoogleCloudStorageToBigQueryOperator 

what should I pass to "schema_fileds", "schema_objects" 

thanks

Antonio

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.

[Feng Lu]

Hi Antonio,

Here you go:

gcloud beta composer environments run {your-environment-name} trigger_dag -- {your-dag-id}

LMK if you run into any issues.

Feng 

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/CAJcv0%3D2a%2B5sDir9Cqd6JyfpFZ4nnioo0dN_q63auHOpX0-aKgg%40mail.gmail.com.