[JIRA] (PLUGIN-894) BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console"

[raginic (Jira)]

raginic created an issue

CDAP Plugins / PLUGIN-894 BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console" Issue Type: Bug Assignee: Unassigned Attachments: Incorrect_projectID_v1-cdap-data-pipeline.json, Screen Shot 2021-09-15 at 7.45.30 PM.png, Screen Shot 2021-09-15 at 7.45.38 PM.png Components: E2E Test Created: 15/Sep/21 7:16 AM Labels: e2e_test_bug Priority: Major Reporter: raginic BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console" Steps to reproduce:- 1. Launch CDAP local host url:- http://localhost:11011 2. Select BigQuery as source and sink it with GCS 3. Browse any file using the established connection 4.Enter any invalid project ID in the GCS sink 5.Deploy and run the pipeline Actual result:- Pipeline gets deployed successfully without any error Expected result:- Pipeline should fail as the project ID is incorrect and data should not be transferred Version 6.5.0-SNAPSHOT Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100177-sha1:6b64421)

[Sumit Nigam (Jira)]

Sumit Nigam commented on PLUGIN-894

Re: BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console" Though wrong project-id is provided in the GCSSink properties, but ServiceAccountCredential.json file has correct project-id. As per the code, it's noticed that when new Storage is being created, ServiceAccountCredentials.json is used, because of which it doesn't show any error. Attaching screenshots for reference

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS

This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100206-sha1:4ed3dc7)

[Sumit Nigam (Jira)]

Sumit Nigam edited a comment on PLUGIN-894

Re: BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console"

Though wrong project-id is provided in the GCSSink properties, but ServiceAccountCredential.json file has correct project-id and other details.

As per the code, it's noticed that when new Storage is being created, ServiceAccountCredentials.json is used, because of which it doesn't show any error. Attaching screenshots for reference

!Screen Shot 2022-08-26 at 5.09.33 PM.png|width=1774,height=662! !Screen Shot 2022-08-26 at 5.26.08 PM.png|width=1774,height=943!

Add Comment

[Sumit Nigam (Jira)]

Sumit Nigam commented on PLUGIN-894

Re: BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console"

Prashant Jaikumar Ankit Jain Please suggest if this ticket can be closed

Add Comment

[Prashant Jaikumar (Jira)]

Prashant Jaikumar commented on PLUGIN-894

Re: BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console"

What happens when you run the pipeline? Does it succeed? We might be missing some validation logic in configurePipeline

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS

This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100206-sha1:1e36365)

[Sumit Nigam (Jira)]

Sumit Nigam commented on PLUGIN-894

Re: BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console"

Prashant Jaikumar Pipeline runs fine without any failure.

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS

This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100206-sha1:41d1dbf)

[Prashant Jaikumar (Jira)]

Prashant Jaikumar commented on PLUGIN-894

Re: BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console"

I see. Looks like storage.get() works correctly because the correct project name gets picked up from Credentials. The projectId is only used by the storage implementation when Credentials are not specified, i.e. they are picked up from the environment. If you can find find a way to extract the projectId from Credentials, you can look into adding some validation. Otherwise I would just close this ticket as the pipeline works as expected.

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS

This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100206-sha1:2202fee)

[Sumit Nigam (Jira)]

Sumit Nigam commented on PLUGIN-894

Re: BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console"

Hi Prashant Jaikumar Changes have been made for this issue and jar/json can be taken from this link for testing the change: gs://cdf-athena-sumit-test/bugfix jars/PLUGIN-894-1071/ Attaching the changes patch, will raise PR upon confirmation from QA, please assign

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS

This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100209-sha1:392b984)

[Sumit Nigam (Jira)]

Sumit Nigam edited a comment on PLUGIN-894

Re: BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console"

Hi [~accountid:5deef88106eaff0e5452418a] Changes have been made for this issue and jar/json can be taken from this link for testing the change:

gs://cdf-athena-sumit-test/bugfix jars/PLUGIN-894-1071/ Attaching the changes patch, will raise PR upon confirmation from QA, please assign

Note/ Confirmation needed [~accountid:5deef88106eaff0e5452418a] : I have assumed that: 1) Service account file/json will always have only one project id 2) Project Id property in GCS sink will be case insensitive

Add Comment

[Sumit Nigam (Jira)]

Sumit Nigam edited a comment on PLUGIN-894

Re: BigQuery as source and GCS as sink:- The pipeline gets deployed successfully even after entering incorrect project ID which does not exists in "Google Cloud Platform Console"

Hi [~accountid:5deef88106eaff0e5452418a] Changes have been made for this issue and jar/json can be taken from this link GCS, ping me for testing the change: same

gs://cdf-athena-sumit-test/bugfix jars/PLUGIN-894-1071/ Attaching the changes patch, will raise PR upon confirmation from QA, please assign Note/ Confirmation needed [~accountid:5deef88106eaff0e5452418a] : I have assumed that: 1) Service account file/json will always have only one project id 2) Project Id property in GCS sink will be case insensitive

Add Comment