implementing own resolution selection mechanics

[Christian Günther]

Hi there,

i am part of an development team, where we do a special kind of image analysis of high-quality scans in android. To get sufficient results we need low noise, good sharpness and high resolution. Since we need more than 1080p (1920x1080px) we do the analysis on image captures instead of analysis use-case.

Overall it works pretty good, but we have troubles to widen-up our supported device fleet. For most devices we want 3000x3000 pixels, since this resolution is sufficient for our use-case and is the sweetspot of the most sensors (like 12mp after binning having 3000px on the shorter edge). But some more budget devices have still some noise-problems on this resolution, where we want to override the desired resolution for a single device specifically.

The main technical problem for us with camerax is at the moment how we can handle fine-grained resolution control mechanism. Especially the behavior of `setTargetResolution` with
```
The target resolution attempts to establish a minimum bound for the image resolution. The actual image resolution will be the closest available resolution in size that is not smaller than the target resolution, as determined by the Camera implementation. However, if no resolution exists that is equal to or larger than the target resolution, the nearest available resolution smaller than the target resolution will be chosen. Resolutions with the same aspect ratio of the provided Size will be considered in higher priority before resolutions of different aspect ratios.
```
does not fit our use-case since it prefers the higher resolutions, but we need instead a resolution which is as close as possible to the desired one, even when it is smaller than 3000x3000px. Additionally our use-case can handle different aspect-ratios for the analysis, but we lack the possibility to say:
Give use the best matching camera resolution for still (yuv) images, where the shorter edge is as close as possible to 3000px. I think this behavior for resolution selection is way out of the general camerax usage and we have to implement our own resolution matching algorithm. So it comes down the the question:

Does exists there any method / api of implementing own resolution selection mechanics?

[Charcoal Chen]

Hi,

1. Does your application only need one use case and ImageCapture is used in order to obtain an image larger than 1080P?

2. The ImageCapture basically should return images of JPEG format. But you mentioned "Give use the best matching camera resolution for still (yuv) images". Does it mean you use the hidden ImageCapture.Builder#setBufferFormat(int) to change the default format of ImageCapture?

There is no API to set customized resolution selection mechanics now. CameraX allows ImageCapture use case to use any size in the supported sizes list. If your application only needs an ImageCapture use case, a workaround is that you can get all supported JPEG sizes by StreamConfigurationMap#getOutputSizes(int) for the target camera device. Implementing your mechanics to select your preferred size. And then set the target resolution as your preferred size. On most devices, the preferred size will be selected for the ImageCapture use case. (This might not be true on few specific devices which some resolutions might cause some specific issues. So those resolutions won't be selected to use.)

If your application needs more than one use case and the ImageCapture use case's default format is changed as YUV_420_888, then, the above approach might not work. Please refer to the guaranteed configurations tables listed in CameraDevice's Regular capture section. The above approach works only if we can find a configuration which matches your needed use cases combination and the ImageCapture can still use a MAXIMUM size.

[Christian Günther]

1. Currently we use every use-case camerax can give us :D We use:

  - a preview use-case

  - a image-analysis (yuv) use-case to search with tensorflow-lite for the object we want to analyze (really low res here, like 96x96)

    - we use the bounding box of the object detection result to configure the camera --> focus point, ae-point and whitebalance (metering points in general)

    - if everything is metered, and our quality assumptions from image analysis are met, we assume to have a good capture situation and triggere the capture use-case in minimal latency mode (since everything is already metered)

  - a image capture (yuv or jpeg fallback) use-case (without saving) in minimal latency mode, since we expect that everything is already correctly metered, when the capture is triggered. This result is passed into a yuv-rgb convertion and analyzed in a c++ core implementation. This is the use-case were we need more fine-grained resolution control. The yuv config is achieved by `captureBuilder.setBufferFormat(ImageFormat.YUV_420_888);`

For highest quality we try to get rid of jpeg compression artifacts, by using the use-cases in a yuv-yuv-prev mode. But some devices are not able to provide render backends with 2 yuv buffers, as discussed here: https://groups.google.com/a/android.com/g/camerax-developers/c/-MQDoM41GGk/m/4hDNZeswAgAJ. For those phones, we fallback to yuv-jpeg-prev use-cases and hope for low jpeg artifacts.

Our core image scan pipeline is able to handle arbitrary aspect ratios. The object we scan has only 1:1 aspect ratio, so everything above is no problem, but involves unnecessary copy operations. So we do not need more than 1:1 aspect ratio and we try to spare some ressources by getting already cropped images. This is achieved by putting all use-cases into a group and applying there the desired aspect ratio:

```

UseCaseGroup.Builder()

.addUseCase(mPreviewUseCase)

.addUseCase(mImageAnalysisUseCase)

.addUseCase(mImageCaptureUseCase)

.setViewPort(new ViewPort.Builder(new Rational(1,1),

activity.getWindowManager().getDefaultDisplay().getRotation()).build())

.build();

```

2. yes we do use this interface --> see answer in part 1. - a image capture use-case

When i try to summarize on what we really need here is:

 1. set explicitly the image format --> already achievable by ImageCapture.Builder#setBufferFormat(int)

 2. set explicitly the capture resolution --> some kind of writing own resolution selector --> can we implement some own selector or bypass the camerax setTargetResolution mechanics here? It seems we should also handle the jpeg fallback here ourself, if the camera hardware level does not provide sufficient resolution on yuv-capture. 
 3. handle explicitly the cropping , like crop all use-cases to 1:1 aspect ratio, keeping the shorter axis on maximum resolution --> does this already the use-case group achieve?

[Charcoal Chen]

Hi,

Thanks for providing the detailed information. If I understand correctly, it seems that providing a way to set your customized resolution selector won't solve your problem.

CameraX's resolution selection algorithm was implemented based on the guaranteed configurations tables in  the CameraDevice's Regular capture section. The resolution selection is not for only one use case but need to consider all the bound use cases to find a workable configuration from the tables. Even an API is provided to set customized resolution selection algorithm, it can only allow you to determine the priority order of each resolution and then return a result list. The resolutions put in the front of the list which meet your requirements might not be used when considering other use cases which are bound together.

Just as you have mentioned in the previous comment. Your design will dynamically determine to use yuv-yuv-priv or yuv-jpeg-priv combination. By checking the guaranteed configuration tables, it seems you can also consider to use the camera device hardware level to determine that.

If the device is LEGACY or LIMITED level, then the yuv-jpeg-priv should be used. Because the following configuration which can support LEGACY and above devices will allow you to use any supported size for the JPEG ImageCapture use case.

PRIV PREVIEW YUV PREVIEW JPEG MAXIMUM Still capture plus in-app processing.

If the device is FULL-level or LEVEL-3, then the yuv-yuv-priv combination can be used. The following configuration which can support for FULL and above devices will allow you to use any supported size for YUV ImageCapture use case.

YUV 640x480 PRIV PREVIEW YUV MAXIMUM Standard video recording plus maximum-resolution in-app processing.

If your application uses the camera device hardware level to determine the used combination, the ImageCapture can always use the MAXIMUM size. And then, just as I have mentioned in the previous comment, you can get all supported sizes by StreamConfigurationMap#getOutputSizes(int) and determine which size you will use by calling setTargetResolution().

————————————————————
authentic.network
INDUSTRY365 UG (haftungsbeschränkt) 
Beckerstraße 13
09120 Chemnitz
Deutschland

Telefon:   +49 371 40 08 450
E-Mail: he...@authentic.network

[Christian Günther]

Here there,

I am now able to put some time into this topic again. Currently i try to implement exactly the desired solution of querying myself the possible resolutions, selecting the exact desired one and putting this one into the setTargetResolution.

The overall implementation looks like this:

```
Camera2CameraInfo camera2Info = Camera2CameraInfo.from(camerax.getCameraInfo());

StreamConfigurationMap map = camera2Info.getCameraCharacteristic(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
Size desiredResolution = OptimalSizeComparator.sorted(map.getOutputSizes(mJpgFallback ? ImageFormat.JPEG : ImageFormat.YUV_420_888))[0];

[...]

captureBuilder.setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
   .setTargetRotation(activity.getWindowManager().getDefaultDisplay().getRotation())
   .setTargetResolution(desiredResolution);

```

Running this on my Redmi Note 10 Pro and other android devices i get often times other resolutions in the capture use-case selected by camerax than i put into it. For the specific example of the Redmi, i put the `Size(4000,3000)`, as determined from the getOutputSizes into setTargetResolution. Camerax then uses instead `3280x2464` as resolution with 90° rotation for the capture use-case. I discovered that when i switch the width and height of the size in the `setTargetResolution(new Size(3000,4000))` i get the desired resolution and camerax does report to use `4000x3000` for the capture use-case with the same 90° rotation.

So i am able to get the correct resolution for the Redmi by switching it in the following way:

```
captureBuilder.setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
   .setTargetRotation(activity.getWindowManager().getDefaultDisplay().getRotation())
   .setTargetResolution(new Size(desiredResolution.getHeight(), desiredResolution.getWidth()));

```

So my Question here is, what is going on? Is this a bug or a feature? Is the orientation of the size in setTargetResolution depended on the sensor orientation and we have to some-time switch the width and height depending on the orientation or do we have a constant switched behavior for all devices, that i have to switch width and height for all devices? Why does that happen overall, since it would be nice to have camerax providing an consistently oriented resolution size handling on its interfaces.

[Charcoal Chen]

Hi,

Please refer to the following javadoc description of ImageCapture.Builder#setTargetResolution(Size)API.

• The resolution Size should be expressed in the coordinate frame after rotating the supported sizes by the target rotation. For example, a device with portrait natural orientation in natural target rotation requesting a portrait image may specify 480x640, and the same device, rotated 90 degrees and targeting landscape orientation may specify 640x480.