setAspectRatioStrategy to ImageCapture and Preview not working

[Jaydeep sinroja]

I use below code to get aspect ratio by 16:9, 4:3 and 1:1 respectively

cameraExecutor = Executors.newSingleThreadExecutor()

imgCaptureExecutor = Executors.newSingleThreadExecutor()
cameraProviderFuture = ProcessCameraProvider.getInstance(this)

cameraProviderFuture.addListener({
val cameraProvider: ProcessCameraProvider = cameraProviderFuture.get()

val preview = Preview.Builder()
.apply {
val resolutionSelectorBuilder = ResolutionSelector.Builder().apply {
if (sharedPrefs.ratio == StoreUserData.RATIO_16_9) {
setAspectRatioStrategy(AspectRatioStrategy.RATIO_16_9_FALLBACK_AUTO_STRATEGY)
} else if (sharedPrefs.ratio == StoreUserData.RATIO_4_3) {
setAspectRatioStrategy(AspectRatioStrategy.RATIO_4_3_FALLBACK_AUTO_STRATEGY)
} else {
setResolutionStrategy(
ResolutionStrategy(
Size(
binding.cameraViewer.width,
binding.cameraViewer.width
), ResolutionStrategy.FALLBACK_RULE_CLOSEST_HIGHER_THEN_LOWER
)
)
}
}
this.setResolutionSelector(resolutionSelectorBuilder.build())
}
.build()
.also {
it.setSurfaceProvider(binding.cameraViewer.surfaceProvider)
}

imageCapture = ImageCapture.Builder()
.apply {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
setTargetRotation(display?.rotation ?: Surface.ROTATION_0)
}else{
setTargetRotation(Surface.ROTATION_0)
}
val resolutionSelectorBuilder = ResolutionSelector.Builder().apply {
if (sharedPrefs.ratio == StoreUserData.RATIO_16_9){
setAspectRatioStrategy(AspectRatioStrategy.RATIO_16_9_FALLBACK_AUTO_STRATEGY)
} else if (sharedPrefs.ratio == StoreUserData.RATIO_4_3){
setAspectRatioStrategy(AspectRatioStrategy.RATIO_4_3_FALLBACK_AUTO_STRATEGY)
} else {
setResolutionStrategy(
ResolutionStrategy(
Size(
binding.cameraViewer.width,
binding.cameraViewer.width
), ResolutionStrategy.FALLBACK_RULE_CLOSEST_HIGHER_THEN_LOWER
)
)
}
}
this.setResolutionSelector(resolutionSelectorBuilder.build())
}
.build()

cameraProvider.unbindAll()
val camera = cameraProvider.bindToLifecycle(
this, cameraSelector, imageCapture,preview
)

}, ContextCompat.getMainExecutor(this))

I use viewPort also but not work as well like this

val viewPort = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
ViewPort.Builder(
getAspectRatio(), display?.rotation ?: Surface.ROTATION_0
).build()
} else {
ViewPort.Builder(
getAspectRatio(), Surface.ROTATION_0
).build()
}

private fun getAspectRatio(): Rational {
return when (sharedPrefs.ratio) {
StoreUserData.RATIO_1_1 -> Rational(1, 1)
StoreUserData.RATIO_4_3 -> Rational(3, 4)
else -> Rational(9, 16)
}
}

but surprisingly, when i bind videoCapture into lifecycle, viewport works fine except one device of Vivo V30 Pro. into that device 4:3 and 1:1 works but 16:9 not works as expected.

My question is
why viewport not work without videoCapture?
why setAspectRatioStrategy not work into preview or imageCapture?
why 16:9 not work as expected on one of the device only?
Or where i went wrong? 

[Charcoal Chen]

Hi,

I modified the CameraXBasic sample app to try the setAspectRatioStrategy API with CameraX library 1.4.0-beta01. Looks like it can work well on my testing.

For the following else-block, an corresponding AspectRatioStrategy setting is needed. Otherwise, the default 4:3 AspectRatioStrategy will be used and 4:3 output sizes will be selected in priority. Please see the ResolutionSelector's javadoc description I append in the end

} else {

setResolutionStrategy(

ResolutionStrategy(

Size(

binding.cameraViewer.width,

binding.cameraViewer.width

), ResolutionStrategy.FALLBACK_RULE_CLOSEST_HIGHER_THEN_LOWER

)

)

}

ResolutionSelector's javadoc description:

When creating a ResolutionSelector instance, the RATIO_4_3_FALLBACK_AUTO_STRATEGY will be the default AspectRatioStrategy if it is not set.

I'm not sure whether the issue is caused by the corresponding AspectRatioStrategy setting not being set. If it is not the case,  I might need to get more info from you to know what might be the possible root causes.

Q1: why viewport not work without videoCapture?

Please refer to the ViewPort's documentation. 

  For ImageAnalysis and in-memory ImageCapture, the output crop rect is getCropRect; for on-disk ImageCapture, the image is cropped before saving; for Preview, the crop rect is getCropRect.

Could you describe more details about what does "viewport not work without videoCapture" mean? For example, what is your setting and what are the incorrect output sizes or the resolutions selected for the Preview/ImageCapture?

Q2: why setAspectRatioStrategy not work into preview or imageCapture?

CameraX's selection logic is to select the best resolution from the supported output sizes retrieved from StreamConfigurationMap#getOutputSizes(). For 16:9 or 4:3 aspect ratio settings, most devices should support output sizes of the aspect ratios. But for 1:1, you will need to implement a ResolutionFilter to put the 1:1 output sizes to the higher priority position. Because most devices might not support 1:1 output sizes and sometimes the supported 1:1 size might not match your requirement (maybe too small). Please also describe more details about what is the final AspectRatioStrategy setting and what are the incorrect output sizes or the resolutions selected for the Preview/ImageCapture for this question?

Q3: why 16:9 not work as expected on one of the device only?

Which UseCase's output size or selected resolution is not 16:9? What is the incorrect output size or selected resolution?

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/41e697cb-6e20-4d4d-bd4a-58c47f0f8ac3n%40android.com.

[Jaydeep sinroja]

I found that, on high resolution device viewport not giving perfect preview view.
I attached ss below, please see this screenshots of Pixel 8 pro device.

and i use stable version 1.3.3 cameraX library.
into that, on xml preview view height width setting match_parent and without viewport 
setAspectRatioStrategy or setTargetAspectRatio(androidx.camera.core.AspectRatio.RATIO_16_9) not works preview view getting math_parents means fullscreen and not cropped by 16:9

[Charcoal Chen]

Hi,

According to the screenshot, it looks like a layout related issue, and you used an emulator to test it.

Please see the attached screenshot that I ran the CameraXBasic sample app built with CameraX 1.3.3 on the API 34 Pixel 8 Pro emulator. The PreviewView's setting is also match_parent (see here). The preview images can be full screen.

To clarify the issue, you can set PreviewView's background to a specific color and do not call "bindToLifecycle" to check whether the PreviewView's position matches your expectation or not. Maybe you also need to check what the PreviewView's parent's layout position is and adjust it to be full screen.

If it is not the layout problem after checking, please let me know your emulator's settings so that I can check the issue on the emulator with the correct API level settings.

[Jaydeep sinroja]

Your screenshot have full screen (match_parent) preview view and not cropped with aspect ratio.

In my testing, the preview behaves as expected on devices with a resolution of 1080x2400 pixels when using a wrap_content preview view. The preview perfectly matches the viewport and is correctly cropped to the specified aspect ratio.

However, this behavior is not consistent on higher resolution devices, such as the Pixel 8 Pro. For instance, I've tested on an emulator configured with the Pixel 8 Pro, running API level 34 (Android version UpsideDownCake) with a resolution of 1344x2992 pixels. On this device, the preview does not appear cropped to the desired aspect ratio.

And also I've encountered an issue with the setAspectRatioStrategy method in my application, particularly with how the preview is displayed on different devices. Specifically, I'm aiming to achieve a cropped preview with aspect ratios of 1:1, 4:3, or 16:9.

Here are the key points of my issue:

Goal: Achieve a cropped preview with aspect ratios of 1:1, 4:3, or 16:9 using the setAspectRatioStrategy method.

Working Scenario: On devices with a resolution of 1080x2400 pixels, the preview is correctly cropped and matches the viewport using wrap_content (without setAspectRatioStrategy).

Problem Scenario: On higher resolution devices like the Pixel 8 Pro (1344x2992 pixels), the preview is not cropped as expected.
Could you provide insights into why this discrepancy occurs and how to ensure consistent cropped previews across different device resolutions, including high-resolution devices like the Pixel 8 Pro?

[Charcoal Chen]

Hi,

I’m wondering whether there are some misunderstandings about what ResolutionSelector (ResolutionStrategy, AspectRatioStrategy) is used for? For an app to show the preview on the display, it needs to obtain the images from the camera device, and then resize/scale/rotate the images to correctly show on an Android View on the display.

The ResolutionSelector (ResolutionStrategy, AspectRatioStrategy) is used to select the best suitable resolution for your app from the device supported output sizes, but not for cropping the output images. After launching the your camera app, you can use the command “adb shell dumpsys media.camera | grep Dims” to know what resolutions are selected to create the camera capture session to output images for Preview/ImageCapture. It looks like below. 1920x1080 is the output size for Preview and 4032x2268 is for ImageCapture.

Dims: 1920 x 1080, format 0x22, dataspace 0x8c20000      Dims: 4032 x 2268, format 0x21, dataspace 0x8c20000

The PreviewView is used to show the images on the display. It helps you to resize/scale/rotate to show the images in the correct direction/aspect ratio (FILL_CENTER by default).

If you want to make the preview be 1:1, 4:3, 16:9, you will need to manually adjust the PreviewView's layout setting in your app to the desired aspect ratio. Then, no matter which aspect ratio of output size is selected to create the capture session for the Preview UseCase, the images shown in the PreviewView should be correct.

setAspectRatioStrategy can’t help the app to directly crop to output images. ViewPort can help to crop the images in the image or video saved by CameraX. But for preview, in-memory still capture or ImageAnalysis images, only a crop rect info is provided for the app to know the area mapping to the ViewPort.

Please let me know whether this can explain the issue you encountered or not. Thanks.

[Jaydeep sinroja]

Thank you
Just tell me that what AspectRatioStrategy should we use for 1:1 output size?
i use 

setAspectRatioStrategy(AspectRatioStrategy(androidx.camera.core.AspectRatio.RATIO_16_9, AspectRatioStrategy.FALLBACK_RULE_AUTO))

for 16:9, and below for 4:3

setAspectRatioStrategy(AspectRatioStrategy(androidx.camera.core.AspectRatio.RATIO_4_3, AspectRatioStrategy.FALLBACK_RULE_AUTO))

both works fine. 

[Charcoal Chen]

1. If you only need the preview to be 1:1, you can set the AspectRatioStrategy#RATIO_4_3_FALLBACK_AUTO_STRATEGY and adjust the PreviewView layout setting to be 1:1.
2. If you need the saved still images or video to be 1:1, you will need to use ViewPort with ratio 1:1 along with the settings in #1.
3. If you need the camera to output 1:1 images, you will need to implement and set a ResolutionFilter to filter the 1:1 output sizes. But, unless your app only runs on a device that you can make sure there are 1:1 output sizes supported, otherwise, you will still need to have code logic to handle the case that the device doesn't support 1:1 output sizes.