Create external table with BigQueryInsertJobOperator does not work on Google Cloud Composer

[Antony Wan]

I am currently working on updating Composer, based on the "composer-2.1.9-airflow-2.3.4" image, to the latest version "composer-2.6.3-airflow-2.6.3". When running the script with exactly the same code, all operators function correctly, except for the BigQueryInsertJobOperator with the tableDefinitions parameter. This parameter is supposed to create intermediate tables but is not working as expected.

I got the code:

and getting the following exception:

To conduct tests, I deployed a Composer instance based on the same version as our current Composer, and I encountered the same error. Has anyone else experienced this issue and successfully resolved it? I would greatly appreciate any assistance.

[Matias Coca]

Hi Antony,

At first glance it seems a `key` error, because is expecting `dataset_id=input_table[“datasetId”]` and in your code you have `destinationTable[“datasetId”]` 

But the log you provided is truncated and will be very helpful to have the complete log to see it.

I hope this help.

Matias Coca

Cloud Brainy | www.cloudbrainy.com

On 11 Mar 2024, at 13:56, Antony Wan <anto...@gmail.com> wrote:

I am currently working on updating Composer, based on the "composer-2.1.9-airflow-2.3.4" image, to the latest version "composer-2.6.3-airflow-2.6.3". When running the script with exactly the same code, all operators function correctly, except for the BigQueryInsertJobOperator with the tableDefinitions parameter. This parameter is supposed to create intermediate tables but is not working as expected.

I got the code:

<script_bug.png>

and getting the following exception:

<exception_bug.png>

To conduct tests, I deployed a Composer instance based on the same version as our current Composer, and I encountered the same error. Has anyone else experienced this issue and successfully resolved it? I would greatly appreciate any assistance.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/931c835f-1dab-4824-ae30-d1041891700bn%40googlegroups.com.
<script_bug.png><exception_bug.png>

[Augusto Hidalgo]

We are taking a look at this error in the Composer team, meanwhile you can disable Cloud Lineage Integration in the environment and that should resolve the issue for now.

Augusto Hidalgo

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/588A58CC-5BC9-4C2F-B852-1370769A9B99%40gmail.com.

[Augusto Hidalgo]

The fix for this issue should be available in Composer 2.6.5 when it's released.