Use case binding failed (VideoCapture and ImageAnalysis)

[Tatiana]

Hello
Thank you for your great support for all camerax developers.

I want to bind 4 use cases simulteniously on our custom device.
CameraX application is bounded to Preview, VideoCapture, ImageCapture and ImageAnalysis use cases simultaneously.
Preview, ImageCapture and ImageAnalysis use cases set up without any resolutions strategy
VideoCapture Use case is set to UHD (3840x2160) resolution
For any other video qualities it works fine, problem appears only for UHD.

CameraX version: 1.4.1
Hardware Level for our custom device: LIMITED
We've got error on the start:
java.lang.IllegalArgumentException: androidx.camera.core.internal.CameraUseCaseAdapter$CameraException: java.lang.IllegalArgumentException: No supported surface combination is found for camera device - Id : 0.
May be attempting to bind too many use cases. Existing surfaces: [] New configs: [androidx.camera.core.impl.ImageCaptureConfig@6d57320, androidx.camera.core.streamsharing.StreamSharingConfig@2e98e9e, androidx.camera.core.impl.ImageAnalysisConfig@cb5f4d9]

Seems that it is not supported when video and image analysis use cases are bound together (this test without image capture use case)
java.lang.IllegalArgumentException: androidx.camera.core.internal.CameraUseCaseAdapter$CameraException: java.lang.IllegalArgumentException:
 No supported surface combination is found for camera device - Id : 0.  May be attempting to bind too many use cases. Existing surfaces: [] New configs: [androidx.camera.core.impl.ImageAnalysisConfig@dc11e62, androidx.camera.core.streamsharing.StreamSharingConfig@447fbf3]
 
As device manufacturer we know that 4k yuv stream is supported. All required stream combinations are worked on camera2 application api.
Could you clarify how CameraХ gets available stream combinations? Is it based only on hardware level?
Please advise how to make described scenario work

Thank you

[Charcoal Chen]

Hi,

Thanks for reaching out to clarify the issue.

Yes, CameraX determines the available stream combinations on the guaranteed configurations tables which are defined here for each different hardware level device.

Please see the below table for LIMITED level devices. When ImageAnalysis using YUV type stream is bound, VideoCapture using PRIV type stream size is limited under PREVIEW size (<=1080p).

Actually, CameraX has an internal stream sharing mechanism to share the images from one PRIV stream to Preview and VideoCapture streams. Otherwise, only three UseCases can be bound according to the guaranteed configurations table.

LIMITED-level additional guaranteed configurations

Limited-level (CameraCharacteristics.INFO_SUPPORTED_HARDWARE_LEVEL == LIMITED) devices support at least the following stream combinations in addition to those for LEGACY devices:

Target 1		Target 2		Target 3		Sample use case(s)
Type	Max size	Type	Max size	Type	Max size
PRIV	PREVIEW	PRIV	RECORD			High-resolution video recording with preview.
PRIV	PREVIEW	YUV	RECORD			High-resolution in-app video processing with preview.
YUV	PREVIEW	YUV	RECORD			Two-input in-app video processing.
PRIV	PREVIEW	PRIV	RECORD	JPEG	RECORD	High-resolution recording with video snapshot.
PRIV	PREVIEW	YUV	RECORD	JPEG	RECORD	High-resolution in-app processing with video snapshot.
YUV	PREVIEW	YUV	PREVIEW	JPEG	MAXIMUM	Two-input in-app processing with still capture.

About "All required stream combinations are worked on camera2 application api",  this can be true because the guaranteed configurations tables define the lower bound capability when the devices declare their hardware level. The devices can actually support more. CameraX can't know the extra supported combinations info and CameraX does have an API to allow apps to set extra supported combinations for specific devices.

For your app's requirements, one workaround is that your app can retrieve the images from Preview to do the analysis if the VideoCapture with UHD resolution is a hard requirement. Then, your app only needs to bind Preview + VideoCapture + ImageCapture. In this condition, VideoCapture can have RECORD resolution. (Please also note that ImageCapture resolution will also be limited under RECORD size and the RECORD size can vary on different devices. RECORD refers to the camera device's maximum supported recording resolution, as determined by CamcorderProfile.)

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/bbd30b6b-902b-432b-8bb6-3869060ad5een%40android.com.

[Tatiana]

Thank you for your answer.

UHD Video is a hard requirement for our camera app. Could you get more information about correct way to retrieve images from Preview in CameraX API?

[Charcoal Chen]

Hi,

Please refer to the below info provided by gemini.google.com.

(CameraX might have some related code but it is for analysis purposes. For example, set OnFrameAvailableListener here and read pixels here. The info provided by Gemini should be more clear and easy to understand.)

========================================================================

When using a TextureView to display camera preview images and you've set an OnFrameAvailableListener, you can access the frames for analysis by following these steps:

1. Obtain the SurfaceTexture:

   • Get the SurfaceTexture associated with the TextureView using getTextureView().getSurfaceTexture().

2. Update the SurfaceTexture:

   • In the onFrameAvailable() callback of your OnFrameAvailableListener, call surfaceTexture.updateTexImage() to update the SurfaceTexture with the latest frame from the camera.

3. Acquire a Frame Buffer:

   • Create a frame buffer using GLES20.glGenFramebuffers(1, fbo, 0).

4. Bind the Frame Buffer and SurfaceTexture:

   • Bind the frame buffer with GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, fbo[0]).
   • Attach the SurfaceTexture to the frame buffer using GLES20.glFramebufferTexture2D(GLES20.GL_FRAMEBUFFER, GLES20.GL_COLOR_ATTACHMENT0, GLES20.GL_TEXTURE_2D, textureId, 0).

5. Read Pixels:

   • Use GLES20.glReadPixels() to read the pixel data from the frame buffer into a buffer.

Image Format:

The format of the image data obtained will depend on the camera's output format, which you can configure using Camera.Parameters. Common formats include:

• NV21 (YUV420SP): A common format where the Y (luma) component is stored as a plane, followed by interleaved U and V (chroma) components.
• YV12 (YUV420P): Similar to NV21, but with the U and V planes reversed.
• RGB: Each pixel is represented by three bytes for red, green, and blue components.

Example Code Snippet:

// In your OnFrameAvailableListener
@Override
public void onFrameAvailable(SurfaceTexture surfaceTexture) {
    surfaceTexture.updateTexImage();

    // ... (Bind frame buffer and read pixels as described above)

    // Assuming the camera output format is NV21
    ByteBuffer buffer = ByteBuffer.allocateDirect(width * height * 3 / 2); 
    GLES20.glReadPixels(0, 0, width, height, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, buffer);

    // ... (Process the image data in 'buffer')
}

Important Considerations:

• Performance: Reading pixel data from the GPU can be expensive. Consider optimizing your image analysis algorithms or performing the analysis on a separate thread to avoid blocking the UI thread.
• Synchronization: Ensure proper synchronization between the camera frames and your analysis to avoid race conditions.
• Error Handling: Check for errors during frame buffer binding and pixel reading using GLES20.glGetError().