Is TFLite export broken for AutoML Object Detection? Localization great but confidence maxes out at 50%

[Eric Forkosh]

Problem
I think TFLite conversion is broken - localization works well but object confidence scores max out at 50% when running on mobile device.

Background
I have been using Google AutoML Vision for over a year now to develop Object Detection models. I export them to Edge devices using TFLite format. 

About 30 days ago, the models were exported using the TensorFlow 1.14 runtime and TOCO Converted. I see that metadata via Netron.

Yesterday, when I exported my model, I noticed the TFLite model was being exported using the TensorFlow 2.5 runtime and MLIR converted. This happens when I export from Vertex AI or from AutoML Vision directly.

Problem In Detail

These new TFLite models seem to have a bug with confidence score. The bounding box localization seems to perfectly localize my objects but the confidence score maxes out at 50% always. The score numbers change between 0% and 50%, but never ever goes above 50%.

Why I think its the export that's broken
Curiously, on the Vertex AI evaluation you can see that the confidence definitely goes beyond 50% (attached). I believe it is an issue with the TFLite export itself.

I'm using the exact same code as my previous models, so I know I'm pretty sure its not the code but the model itself. Same dataset, different exported models - old one works and new one has this 50% maximum bug.


Has anyone else experienced this issue?

[Charles Harring]

Hey I noticed something weird too. My problem is that my program no longer runs when exported with 2.5 (https://groups.google.com/g/cloud-vision-discuss/c/9dyvQbISjUk) so I cant verify your bug, but will be following along as this is a huge problem for me.