[JIRA] (CDAP-19294) Create Table DDL in BQExecute will fail with CMEK enabled CDF instance

[Sean Zhou (Jira)]

Sean Zhou created an issue

CDAP / CDAP-19294 Create Table DDL in BQExecute will fail with CMEK enabled CDF instance Issue Type: Bug Assignee: Prerna Bellara Attachments: test_v1-cdap-data-pipeline.json Components: Pipelines Created: 13/May/22 6:03 PM Priority: Major Reporter: Sean Zhou For CMEK enabled CDF instance, we will pass a runtime argument gcp.cmek.key.name` by default. and the attache d pipeline will fail with com.google.cloud.bigquery.BigQueryException: Cannot set kms key name in jobs with DDL statements. For CREATE TABLE statement, use OPTIONS(kms_key_name=...) instead it’s because BQExecute will set the kms key name in the BQ job if this runtime argument is set. Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100198-sha1:8950bff)

[Ankit Jain (Jira)]

Ankit Jain commented on CDAP-19294

Re: Create Table DDL in BQExecute will fail with CMEK enabled CDF instance This is a known issue, we cannot set kms_key_name & destination_table in the BQ job created by the plugin. https://stackoverflow.com/questions/51089598/google-cloud-bigquery-library-error Since, the purpose of Encryption Key Name in BQ Execute plugin is to create table and dataset with cmek if they do not exist when user wants to store the query results in a destination table. So, a possible solution could be to set cmek in the BQ job config only in that case instead of passing it in every case. https://github.com/data-integrations/google-cloud/blob/e970af07cd113958e5378e02997c7304cb8bd500/src/main/java/io/cdap/plugin/gcp/bigquery/action/BigQueryExecute.java#L90 Albert Shau & Sean Zhou plz correct me if I am wrong.

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS

This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100198-sha1:f46a033)

[Ankit Jain (Jira)]

Ankit Jain edited a comment on CDAP-19294

Re: Create Table DDL in BQExecute will fail with CMEK enabled CDF instance

This is a known issue, we cannot set kms_key_name & destination_table in the BQ job created by the plugin with DDL statements . [https://stackoverflow.com/questions/51089598/google-cloud-bigquery-library-error|https://stackoverflow.com/questions/51089598/google-cloud-bigquery-library-error|smart-link] Since, the purpose of {{Encryption Key Name}} in BQ Execute plugin is to create table and dataset with cmek if they do not exist when user wants to store the query results in a destination table. So, a possible solution could be to set cmek in the BQ job config only in that case instead of passing it in every case. [https://github.com/data-integrations/google-cloud/blob/e970af07cd113958e5378e02997c7304cb8bd500/src/main/java/io/cdap/plugin/gcp/bigquery/action/BigQueryExecute.java#L90|https://github.com/data-integrations/google-cloud/blob/e970af07cd113958e5378e02997c7304cb8bd500/src/main/java/io/cdap/plugin/gcp/bigquery/action/BigQueryExecute.java#L90|smart-link] [~accountid:5d0bd1ddbc0eae0bbc6c0644] & [~accountid:5f2858118fd8ab001bfd9dfa] plz correct me if I am wrong.

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100198-sha1:f46a033)

[Ankit Jain (Jira)]

Ankit Jain edited a comment on CDAP-19294

Re: Create Table DDL in BQExecute will fail with CMEK enabled CDF instance

This is a known issue, we cannot set kms_key_name & destination_table in the BQ job with DDL statements. [https://stackoverflow.com/questions/51089598/google-cloud-bigquery-library-error|https://stackoverflow.com/questions/51089598/google-cloud-bigquery-library-error|smart-link] [https://sql.info/d/solved-cannot-set-kms-key-name-in-jobs-with-ddl-statements-bigquery|https://sql.info/d/solved-cannot-set-kms-key-name-in-jobs-with-ddl-statements-bigquery|smart-link]

Since, the purpose of {{Encryption Key Name}} in BQ Execute plugin is to create table and dataset with cmek if they do not exist when user wants to store the query results in a destination table. So, a possible solution could be to set cmek in the BQ job config only in that case instead of passing it in every case. [https://github.com/data-integrations/google-cloud/blob/e970af07cd113958e5378e02997c7304cb8bd500/src/main/java/io/cdap/plugin/gcp/bigquery/action/BigQueryExecute.java#L90|https://github.com/data-integrations/google-cloud/blob/e970af07cd113958e5378e02997c7304cb8bd500/src/main/java/io/cdap/plugin/gcp/bigquery/action/BigQueryExecute.java#L90|smart-link] [~accountid:5d0bd1ddbc0eae0bbc6c0644] & [~accountid:5f2858118fd8ab001bfd9dfa] plz correct me if I am wrong.

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100198-sha1:f46a033)

[Sean Zhou (Jira)]

Sean Zhou commented on CDAP-19294

Re: Create Table DDL in BQExecute will fail with CMEK enabled CDF instance

I agree. if user do want to enable CMEK for the DDL , they can add below at the end of the DDL OPTIONS(kms_key_name="XXXX")

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS

This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100198-sha1:70dbdd1)