Incorrect resolutions for VideoCapture + ImageCapture use cases

[Tatiana Zalesskaya]

Hello CameraX developers

Our CameraX application binded to Preview, VideoCapture, ImageCapture and ImageAnalysis use cases simultaneously. We want to set different captured image and video resolutions, but it cause unpredictable resolution results

CameraX version: 1.4.0-rc01
Hardware Level for our custom device: LIMITED

Available output sizes are the same for Image capturing JPEG, Image capturing YUV_420_888, Preview

4224x3136
3840x2160
3264x2448
2592x1944
2112x1568
1920x1080
1280x960
1280x720
640x480
352x288
320x240
176x144

Available Video qualities:
[ConstantQuality{value=8, name=UHD, typicalSizes=[3840x2160]},
 ConstantQuality{value=6, name=FHD, typicalSizes=[1920x1080]},
 ConstantQuality{value=5, name=HD, typicalSizes=[1280x720]},
 ConstantQuality{value=4, name=SD, typicalSizes=[720x480, 640x480]}]


Example:
Image capture use case resolutionSelector size has been set to 4224x3136 //could be changed to any other available resolutions
Preview Use case is always set to 1920x1080
Video Quality set to FHD, typicalSizes=[1920x1080] //could be changed to other availbale quality

Image Capture resolution selector code:
setResolutionStrategy(
            ResolutionStrategy(Size(4224, 3136),
                ResolutionStrategy.FALLBACK_RULE_CLOSEST_HIGHER_THEN_LOWER)
        )


Preview resolutions selector code:
setResolutionSelector(ResolutionSelector.Builder()
            .setResolutionFilter { supportedSizes, _ ->
                val resultList = mutableListOf<Size>()
                if (supportedSizes.contains(Size(1920, 1080))) {
                    resultList.add(Size(1920, 1080))
                }
                resultList
            }


Result:
Video is captured with resolution 1900x1070
Image is captured with 3264x2448

From logs:

VideoCapture: onSuggestedStreamSpecUpdated: StreamSpec{resolution=1900x1070,
VideoCapture: suggested resolution 1900x1070 is not in custom ordered resolutions [1920x1080]
ImageCapture: createPipeline  StreamSpec{resolution=3264x2448

Is it possible to configure video and capture use cases with the correct resolutions?
Thank you

[Charcoal Chen]

Hi,

According to the info you provided, the 1900x1070 resolution for VideoCapture might be caused by the ViewPort settings.

When four UseCases are bound, CameraX internally use a stream to produce (crop/scale) the output images to VideoCapture and Preview. If a ViewPort is set, it will be applied to the VideoCapture's output resolution. Then, it might be the reason that your app gets the 1900x1070 output resolution for VideoCapture that the resolution is not any one of the supported output sizes.

To know whether this is the root cause, does your app have ViewPort setting when binding the UseCases and what is the setting value?

About the ImageCapture's output resolution, please refer to the guaranteed supported configurations tables in the javadoc of CameraDevice's Regular capture section. Looks like the following configuration will be used.

Could you let us know what is the resolution selector setting for ImageAnalysis and which resolution is selected? So that we can clarify whether 3264x2448 is selected for ImageCapture because your app requests a RECORD level size resolution for ImageAnalysis.

   PRIV / PREVIEW YUV / RECORD JPEG / RECORD         High-resolution in-app processing with video snapshot.

Also, is it possible to provide us the following info? We may need to know more detailed information to think why those resolutions are selected for each UseCase.

 - bugreport

 - The device info (brand, device name, model name, display size)

Thanks.

[Tatiana]

Hi,
Thank you for your answer. I was able to reproduce problems with device
Samsung Galaxy Tab S9 5G

Device Level: INFO_SUPPORTED_HARDWARE_LEVEL_LIMITED
Display size width=1600, height=2560

Available JPEG output sizes:
4128x3096
4128x2580
4128x2322
3088x3088
3840x2160
1920x1440
1920x1080
1728x1080
1920x824
1440x1080
1088x1088
1280x720
960x720
720x480
640x480
640x360
352x288
320x240
176x144

Available Video Qualities:
name=UHD, typicalSizes=[3840x2160]},
name=FHD, typicalSizes=[1920x1080]},
name=HD, typicalSizes=[1280x720]},
name=SD, typicalSizes=[720x480, 640x480]

1. Video Capture output size problem
Note: This issue is reproducible only with our custom device (screen size 1200x2000), not reprodusble with Samsung device
ViewPort is set in our application to the value:
setViewPort(
ViewPort.Builder(
sessionSettings.aspectRatio.ratio, // 16x9
previewUseCase.targetRotation // ROTATION_90
).build()

When I removed this code problem with VideoCapture dissapeared. Does it mean that we are not able to change viewport size without affecting Video Capture output size?

Problems reproducible on Samsung Galaxy Tab:
2. When I've changed video quality to the UHD (3840x2160) application crashed with the following error:
 AndroidRuntime: java.lang.IllegalArgumentException: androidx.camera.core.internal.CameraUseCaseAdapter$CameraException:
 java.lang.IllegalArgumentException: No supported surface combination is found for camera device - Id : 0.  May be attempting to bind too many use cases.

Setup is the same (Preview and ImageAnalysis set to 1920x1080, ImageCapture to the highest size (4128x3096))
Please refer to the attached bugreport. When I commented ImageAnalysis or ImageCapture use case binding - application is started and able to capture video in UHD quality.

3. Image Capture 16x9 sizes problem
When I shoose image size with the aspect ration 4x3 it's captured with the correct result, but for 16x9 sizes - it's never set correctly. Captured image always has 4:3 resolution
Example
1280x720 set to 1440x1080
3840x2160 set to 4128x3096
4128x2322 set to 4128x3096

Please refer to the attached log (samsung_16x9_image_capture)

4. Setting image capture aspect ratio behavior
Observed following behavior:
I'm not setting resolution selector image size in this case, only aspect ratio strategy. When ImageCapture resolution aspect ratio strategy set to RATIO_4_3_FALLBACK_AUTO_STRATEGY, all image output size has the highest size (4128x3096)
When ImageCapture resolution strategy set to setAspectRatioStrategy(AspectRatioStrategy.RATIO_16_9_FALLBACK_AUTO_STRATEGY):
Captured image size is set to RECORDER output = 3840x2160 for Video Quality UHD and 4128x2322 for FHD. Is it expected behavior?
But it is not the highest image captured resolution. If I want to set the highest image capture output size should I set image size directly without any fallback resolution strategy (only with resolution filter), is it correct?

Thank you.

[Charcoal Chen]

Hi,

Please see the answers below:

1. Video Capture output size problem

"When I removed this code problem with VideoCapture dissapeared. Does it mean that we are not able to change viewport size without affecting Video Capture output size?"

=> Yes. Please see the javadoc of ViewPort - 

Usually ViewPort is configured to optimize for Preview so that ImageAnalysis and ImageCapture produce the same crop rect in a WYSIWYG way.

WYSIWYG means "What You See Is What You Get". If you don't want to crop the output video, please do not set ViewPort.

2. When I've changed video quality to the UHD (3840x2160) application crashed with the following error:

When I commented ImageAnalysis or ImageCapture use case binding - application is started and able to capture video in UHD quality.

=> Camera capability is limited. Although CameraX has an internal stream sharing mechanism to allow apps to bind four UseCases on almost all devices, the resolution of each UseCases might be restricted.

When you force VideoCapture to have the UHD quality setting without fallback options, one of the following configurations will be used to select resolutions for each UseCase (LIMITED level device):

PRIV / PREVIEW     PRIV / RECORD                                      High-resolution video recording with preview.

PRIV / PREVIEW     PRIV / RECORD     JPEG / RECORD     High-resolution recording with video snapshot.

 Maybe ImageCapture can be bound together, but the additional ImageAnalysis (YUV format stream) is not allowed.

3. Image Capture 16x9 sizes problem

When I shoose image size with the aspect ration 4x3 it's captured with the correct result, but for 16x9 sizes - it's never set correctly. Captured image always has 4:3 resolution

=> ResolutionSelector has default 4:3 aspect ratio setting. When setting a 16:9 bound size ResolutionStrategy and you also want to select a 16:9 output size, please set the corresponding 

RATIO_16_9_FALLBACK_AUTO_STRATEGY to the ResolutionSelector. This should resolve the problem.

4. Setting image capture aspect ratio behavior

When ImageCapture resolution strategy set to setAspectRatioStrategy(AspectRatioStrategy.RATIO_16_9_FALLBACK_AUTO_STRATEGY):

Captured image size is set to RECORDER output = 3840x2160 for Video Quality UHD and 4128x2322 for FHD. Is it expected behavior?

=> IIUC, you mean that

When VideoCapture is set to have UHD Quality output, ImageCapture can only have 3840x2160 output size

When VideoCapture is set to have FHD Quality output, ImageCapture can have 4128x2322 output size

It is expected behavior. It is also caused by the camera device capability restrictions. When VideoCapture is set to have UHD Quality output, it will impact ImageCapture to only support RECORD size output.

PRIV / PREVIEW     PRIV / RECORD     JPEG / RECORD     High-resolution recording with video snapshot.

When VideoCapture is set to have FHD Quality output, ImageCapture might be able to support a resolution higher than RECORD size.

But it is not the highest image captured resolution. If I want to set the highest image capture output size should I set image size directly without any fallback resolution strategy (only with resolution filter), is it correct?

=> As I mentioned above, camera capability is limited. If the ImageCapture's resolution is most important to your app, you may be able to set it as the highest resolution without strategy. The trade-off is that you will need to set fallback strategy to the other UseCases. Otherwise, your app might receive the IllegalArgumentException you mentioned above on many devices.

The resolution selection logic depends on many factors, the camera device hardware level, the display size, the maximum recorder profile size, the bound UseCase combination. Therefore, I will recommend apps to have fallback settings to ResolutionSelector as possible. Then, CameraX will help apps to find the best workable resolutions on all devices.