Google Cloud Platform - Vertex AI Image classification training fails with no specific error message

[Weilin Meng]

I'm doing an image classification task using Vertex AI and after about 3 hours of training it fails. The error message is nondescript. It says "Training pipeline failed with error message: Internal error occurred. Please retry in a few minutes. If you still experience errors, contact Vertex AI."

It's done that for three of my models using the same image dataset (about 45k large). What could be the error here? How can I find out?

[Jun (Cloud Platform Support)]

Hi, 

It is unclear what had happened for your image classification training failing based on the error message you're seeing. Thus I'll suggest that you can report this with more details to the public Issue Tracker (for product bugs or unexpected behaviors) so that the support can work with you to dig into this to have more insights on the behavior.

Therefore, please notice that Google Groups are reserved for general product discussion, and StackOverflow for technical questions. To get a better support you should post to the relevant forum, thus please read the Community Support article for better understanding. 

[1] "Training pipeline failed with error message: Internal error occurred. Please retry in a few minutes. If you still experience errors, contact Vertex AI."

[Weilin Meng]

Thanks for the reply. I made a report but I don't really have anything to add and I doubt someone will come looking for me to help. This is all the information I get and I'm looking to resolve it soon. Is there anyone live I can talk to?

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-vision-discuss/f4330f4b-913b-45e8-9a80-292816b28611n%40googlegroups.com.

[Tanner Davis]

I experienced this exact same error yesterday as well. The error message is not helpful and no logs are given for me to try and figure out the issue. But I tried 3 different models and each "failed" at the 2.5 hour mark.

[Tanner Davis]

My dataset/model is image classification as well but only about 3k images total in the dataset

[Weilin Meng]

I also failed at the 2.5-3.5 hours mark

IMPORTANT NOTICE: This message is intended for the individual or entity to which it is addressed and may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that copying, forwarding or other dissemination or distribution of this message is prohibited and that taking any action in reliance on the content of this message is to be avoided. Should you receive this e-mail in error, please notify the sender immediately via e-mail or call 855.688.6398 and delete this message from your system. Thank you.

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-vision-discuss/16fa1a7f-298e-4675-8427-39cfb8ba958cn%40googlegroups.com.

[Javier Azcoiti Yerro]

There is an IssueTracker [1] created requesting a more interpretable and detailed error message corresponding to the error you have obtained. 

I recommend following it and possibly sharing your case there in order to increase its visibility. 

While there is no better error message, I would recommend creating a support case should you want to have your resources manually investigated in order to troubleshoot this circumstance.

[1] - https://issuetracker.google.com/208989810