Re: Internal error occurred. Please retry in a few minutes. If you still experience errors, contact Vertex AI.

[Helin Wang]

Hi ARNAUD,

Could you please share your project number and the failed job id? We will look up what internal error happened.

Thanks,
Helin

On Wed, Apr 27, 2022 at 5:23 AM 'ARNAUD LEJEUNE' via cloud-automl-tables-discuss <cloud-automl-...@googlegroups.com> wrote:

Hello,

I'm trying to make prediction with a model I trained with forecasting tabular.

I have this error when I try to do it :

"Internal error occurred. Please retry in a few minutes. If you still experience errors, contact Vertex AI."

Do you have any idea how can I understand and correct it ? 

Thanks.

Regards,

Arnaud

--
You received this message because you are subscribed to the Google Groups "cloud-automl-tables-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-automl-tables...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-automl-tables-discuss/38c37295-2e4d-4943-bfe1-705fa0b1d93bn%40googlegroups.com.

[ARNAUD LEJEUNE]

Hi,

I succeeded in making forecasts, the problem seems to be some missing columns on the prediction table.

However, I train a new model with differents parameters and the same training table :

- Granularity : Day

- Forecast horizon : 365

- Forecast window : 365

When i did it with :

- Forecast horizon : 1

- Forecast window : 1

The model trained and I can make predictions.

With new parameters (horizon 365), I have the same error :

"Training pipeline failed with error message:Internal error occurred. Please retry in a few minutes. If you still experience errors, contact Vertex AI."

Thanks for your response and your help.

Project_id = 683460410774

Job_id = 3949476553296969728

Regards,

Arnaud

--

Arnaud Lejeune   Data Engineer   EDD Supply

[Chenyu Zhao]

Hi Arnaud,

It'll be difficult to diagnose without more detailed information. The best way for us to help debug these internal errors is if you can file a GCP Support Case: https://cloud.google.com/support-hub

Thanks,

Chenyu

[Helin Wang]

+Dawei Jia our oncall to help take a look. Dawei, the corresponding bug is b/230672958.

Thanks,

Helin

[Dawei Jia]

Hi ARNAUD,

The failure is because your dataset is too large. Do you mind to downsample it to 20% or so dataset and try it again?

[ARNAUD LEJEUNE]

Hi,

I try with a smaller dataset and with more node hours. But I still have an error, do you have any idea ?

Thanks a lot.

Project_id = 683460410774

Job_id = 3498736159536709632

Regards,

[Yang Yang]

Hi ARNAUD,

The issue is caused by a new feature we recently introduced in the training. We have root caused the issue, and are working on the fix, which will be rolled out within 2 weeks.

Currently, to workaround the issue, you can use a small context window and a small horizon, e.g. 28. Once the fix is rolled out, this workaround will no longer be needed.

Sorry for the inconvenience. Please let us know if you have any other questions or concerns.

Thanks,

Yang

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-automl-tables-discuss/9a6acd60-b1ac-4dea-a791-6c9b46c1ee88n%40googlegroups.com.

--

YangY