How to use RenderScript surface in CameraX

[Jun Tung Liew]

Is there a way to use RenderScript Allocation surfaces to do image processing in CameraX? It is useful for doing YUV -> RGB conversion or to rotate the image. Sure, I can combine 3 YUV buffers to one and feed into RenderScript but the process of combining buffers can cause performance issues in certain phones.

Here's how I modify the ImageAnalysis use case to utilize RenderScript and I managed to get it working.

https://gist.github.com/LiewJunTung/ca88953af56301551009d2d3b06c524d

[Franklin Wu]

At the moment CameraX does not provide a way to use RenderScript allocation. It is intentional that creating of the consumers (i.e. ImageReader, SurfaceTexture, etc) is handled by CameraX in order to make sure that a successful CameraCaptureSession can be created.

Is there something specific that you are trying to achieve with Renderscript that is difficult to do with a YUV buffer or is this more of a general optimization you are hoping for?

[Jun Tung Liew]

It's for optimization. Currently, YUV is split into 3 separate ByteBuffer in different planes. Joining them to one big ByteBuffer for processing is slow on some phones (Such as Singapore's Samsung S8 - ~200ms average). 

It's possible to counter this via using NDK but working with RenderScript in NDK has a more involved process if we want to use hand-written RenderScript.

[Franklin Wu]

Would it be correct to understand that you actually want a ByteBuffer that is in RGB format instead of YUV (based on the Renderscript call)? With continuous interleaved RGBA for processing and it really doesn't matter how you get it as long as it is fast. 

Or do you actually want to have the ability to process using renderscript.

Note that YUV_420_888 is the recommended format for camera2 which is why CameraX provides this by default.

[Jun Tung Liew]

Yes, having the ability to use RenderScript Allocation Surface will be nice in ImageAnalysis (Capture if that's possible also). It is the fastest YUV -> RGB solution, that I can get on certain Samsung phones such as S8.  Besides YUV to RGB, we can rotate the buffer immediately after doing YUV to RGB also.

I understand that YUV_420_888 is the recommended format. Having a way to do the processing fast is crucial in some application.

[Jun Tung Liew]

Actually, scratch that. I have found a solution by passing Image to ImageWriter.  https://developer.android.com/reference/android/media/ImageWriter 

If the application already has an Image from ImageReader, the application can directly queue this Image into the ImageWriter (via queueInputImage(Image)), potentially with zero buffer copies. 

Here's how I did it. https://gist.github.com/LiewJunTung/b9b7f8444037fbcbe5b30a915850d494

Thanks for CameraX, love it so far!