Training pipeline failed with error message: Internal error occurred. Please retry in a few minutes. If you still experience errors, contact Vertex AI.

[aishwarya badlani]

Training pipeline failed with error message: Internal error occurred. Please retry in a few minutes. If you still experience errors, contact Vertex AI.

How to identify why it failed ?

raining pipeline ID6730485607081967616

[Darko Pevec]

Hi!

I am experiencing the same problem:

Training pipeline failed with error message: Internal error occurred. Please retry in a few minutes. If you still experience errors, contact Vertex AI.

Additional Details:
Operation State: Failed with errors
Resource Name: 
projects/397581031430/locations/us-central1/trainingPipelines/5824283516631777280
Error Messages: Internal error occurred. Please retry in a few minutes. If 

you still experience errors, contact Vertex AI.

I cannot find any useful error messages.

Regards,

Darko

[Dawei Jia]

These two are separate issues.

For Aishwarya's issue Training pipeline ID6730485607081967616, Do you mind reducing the number of columns? The pipeline took too long for the feature engineering steps and the job service time it out.

For Darko's issue, the pipeline don't have permission to pull the data from bigquery. Do you mind to let aiplatform service account to have access to your bq tables or view? https://cloud.google.com/vertex-ai/docs/datasets/prepare-tabular#bq

--
You received this message because you are subscribed to the Google Groups "cloud-automl-tables-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-automl-tables...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-automl-tables-discuss/c4dfadd5-9256-4960-a790-60dcb2d53945n%40googlegroups.com.

[Yang Yang]

Hi Darko,

Thanks for reaching out. Sorry for the inconvenience.

We checked the error log, the error shows:

> Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials.

It seems that the underlying input data is in Google Drive or Google Sheets, which by default doesn't grant the read permission to Vertex AI service agent.

To solve this issue, you can find the Vertex AI service agent(note the custom code service agent) from your GCP IAM page, and grant the read permission(share as viewer) of the Google Driver documents to the Vertex AI service agent.

We will continue to improve our product and documents.

Please let us know if you have any questions.

Thanks,

Yang

On Tue, Jan 25, 2022 at 11:20 PM Darko Pevec <darko...@login5.org> wrote:

--

You received this message because you are subscribed to the Google Groups "cloud-automl-tables-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-automl-tables...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-automl-tables-discuss/c4dfadd5-9256-4960-a790-60dcb2d53945n%40googlegroups.com.

--

YangY

[Alex Martin]

+Cloud AutoML Tables Feedback 

Darko, does the problem still persist?

On Tue, Jan 25, 2022 at 11:20 PM Darko Pevec <darko...@login5.org> wrote:

--

[Darko Pevec]

Hi Alex,

I believe the problem still persists. I have shared my spreadsheet with our service account service-39...@gcp-sa-aiplatform.iam.gserviceaccount.com and given it Viewer access, but when i run anything from VertexAI, ie. generate dataset statistics, i still get the internal error message.

My workaround was to make a hard copy of the dataset in BQ.

Regards,

Darko

[Yang Yang]

I just took a deep dive into the issue, the root cause turns out to be that the Google Drive scope needs to be specified from the underlying Dataflow job which is not yet supported in Apache Beam SDK. We need to contact the Apache Beam team to add the support.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-automl-tables-discuss/CADQuPWj4ww9wuMn9igEN_vWxZ-ZK00jH%3DRBH7uAZKc1XqmS3LQ%40mail.gmail.com.

--

YangY

[Raghavendra Vasari]

How to identify why it failed?

Training pipeline ID6758227659840290816

How are you able to view and tell us the problem?
where can we find the root cause?

thanks & regards,

Raghavendra