imageProxy.image is Experimental.

[Deepu George Jacob]

Hi,

I am trying to get text using camera x and ml kit. When I am analysing something I notice that imagProxy.image is experimental. Why this is experimental and what should be alternate for that?

Thanks and Regards

Deepu George Jacob

 

[Gerry Fan]

this one is hot topic: Last time seeing the debate(github and other places), the scripts (if write on paper) could run from the moon to the earth :-)

The experimental tag is kind of guarding off image leaking, I think.  I did not see the alternate direct alternate; mlkit doc page recommends to use ImputImage.fromMediaImage() and snippet directly use it(imageProxy.image). It is a disconnection, and previous debate on github turned to a debate about the meaning of word "experimental"..., which is super interesting too :-)

Now back to topic about this usage: the model input has the final saying, it is said that most models only take

  - square

  - rgb (not alpha) in a butter (compact, no padding either -- need to double check this one)

The output from CameraX:

   - rectangle(you could set to target rectangle, but you may not get a square out)

   - yuv (media.image) or rgba (byte buffer saved in the first plane pointer)

In order to have them match, someone will have to do the color conversion and scaling: if you give media.image to mlkit, it does these 2 things internally(with its image processing lib, I think); or you could do it yourself or do portion of them.  The path of directly passing image objects to mlkit is questionable(because it is experimental, you saw that and created this thread). So you probably would try other paths. One sample tried with the Bitmap path:

- turn on RGBA output format with CameraX imageAnalysis (setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)

- create bitmap from the RGBA output(note: it is a copy here, copy is not free in the sense you give up some frame rate)

- send it to mlkit (probably it will do those alpha removal, and scaling internally, unless the mode could take non-square tensor)

It is a detour, but works; did not see the performance hit(it actually is a boost comparing to previous renderscript way). Comparing to imageProxy.image, you would need a few extra lines, but performance wise, you would need to compare both paths.

As for this "experimental" word, I will study the image path internally, if it could be removed, that would be great and will let you know. Kind of sorry for the boring long email, it is touchy one though.

[Deepu George Jacob]

Thank you Gerry Fan,

It is very helpful information.

Thanks and Regards

Deepu George Jacob