Poor Entity Extraction performance when training through Python API

[Sebastien Naus]

Hello,

In my Vertex Pipeline, I am running an AutoML Entity Extraction training job in Vertex AI using the google-cloud-aiplatform Python package (google.cloud.aiplatform.AutoMLTextTrainingJob method). I am using a random training/validation/test split of 0.8/0.1/0.1 on a managed dataset imported in Vertex AI containing >1000 items.

The resulting model has very low recall (around 11%) and good precision (>80%). This is the case after running the same training jobs 3 times.

However, when I train an AutoML Entity Extraction on the same managed dataset directly through the Vertex AI UI with the same random 0.8/0.1/0.1 split, the resulting model has much better precision/recall trade-off (both above 75%). This is also verified on multiple training runs.

Why could this be?

[Kailong (Cloud Platform Support)]

Hi Sebastien,

This seems to be an issue with the AI Platform Python client library. We need to gather more information in order to proceed with the troubleshooting process, such as:

• Project ID

• Pipeline Job ID

• SDK(KFP/TFX) Version

• Any screenshot

However, for privacy and security reasons, I would like to ask you to create a private issue on Issue Track site [1]. And provide the information needed for troubleshooting in the private issue, so that our team can provide more dedicated help to your issue. Please remember to include a link to your original thread/question in the new issue.

Please keep in mind that Google Groups are reserved for general product discussion, StackOverflow for technical questions whereas Issue Tracker for product bugs (unexpected behaviors) and feature requests. 

Best Regards,

Kailong

Google Cloud Platform Support

[1] https://issuetracker.google.com/issues/new?component=491162&template=1161151