CameraX with Tensorflow camera preview overlay aspect ratio

158 views
Skip to first unread message

Talal Shah

unread,
Apr 17, 2024, 7:24:31 AMApr 17
to Android CameraX Discussion Group
When utilizing CameraX with TensorFlow for object detection, I encountered an issue related to the positioning of rectangular detection boxes within the camera preview. Specifically, the overlay functionality functions correctly, accurately delineating objects within the specified rectangular region, when the preview resolution is set to `setTargetAspectRatio(AspectRatio.RATIO_16_9)`.

However, in order to obtain cropped images of the detected objects at higher resolutions, I utilize the `.setResolutionSelector()` method with a `ResolutionStrategy.HIGHEST_AVAILABLE_STRATEGY` parameter. This ensures that the preview resolution is set to the highest available option, which is essential for obtaining clear images of the detected objects. Maintaining a default resolution results in low-quality images, which, when cropped, further deteriorate in clarity.

Upon implementing the high-resolution settings within both the `ImageAnalyzer` and the Camera Preview, I encountered an unexpected behavior where the bounding boxes are drawn on the preview no longer aligned with the actual objects. This misalignment adversely affects the subsequent cropping process, leading to inaccuracies in the resultant images. Conversely, when utilizing the `AspectRatio.RATIO_16_9`, the bounding boxes are correctly positioned, and the cropping process functions as expected.

It is imperative to resolve this discrepancy to ensure accurate object detection and cropping at higher resolutions, in line with project requirements.

Below mentioned code is working when ImageAnalyzer and CameraPreview target resolution is set to .setTargetAspectRatio(AspectRatio.RATIO_16_9

This is my cropping method of rectangular box which is not working according to the expectation
public Bitmap croppedBitmap(Bitmap rotated) {

//calculate aspect ratio
float koefX = (float) rotated.getWidth() / (floatpreviewView.getWidth();
float koefY = (float) rotated.getHeight() / (floatpreviewView.getHeight();
//get viewfinder border size and position on the screen
int x1 = borderCamera.getLeft();
int y1 = borderCamera.getTop();

int x2 = borderCamera.getWidth();
int y2 = borderCamera.getHeight();

//calculate position and size for cropping
int cropStartX = Math.round(x1 * koefX);
int cropStartY = Math.round(y1 * koefY);

int cropWidthX = Math.round(x2 * koefX);
int cropHeightY = Math.round(y2 * koefY);

//check limits and make crop
if (cropStartX + cropWidthX <= rotated.getWidth() && cropStartY + cropHeightY <= rotated.getHeight()) {
/*
This is the cropped bitmap image we will pass it into create image file function
* */
croppedBitmap = Bitmap.createBitmap(rotatedcropStartXcropStartY cropWidthXcropHeightY );
else {
croppedBitmap null;
}
return croppedBitmap;
}


This is the bounding boxes code 
private var scaleFactor: Float = 1f
val boundingBox = result.boundingBox
val top = boundingBox.top scaleFactor
val bottom = boundingBox.bottom scaleFactor
val left = boundingBox.left scaleFactor
val right = boundingBox.right scaleFactor

// Draw bounding box around detected objects
val drawableRect = RectF(lefttoprightbottom)
canvas.drawRect(drawableRectboxPaint)

Xi Zhang (张熹)

unread,
Apr 17, 2024, 11:03:25 AMApr 17
to Talal Shah, Android CameraX Discussion Group
The best way to transform coordinates between UseCases is using the sensor-to-buffer value that provided with the output.  In your case, you want to transform the rect in Preview coordinate system to ImageAnalysis coordinate system. The Transform output topic in the CameraX document has a code sample for this. 

Note that the API in that document is going to be deprecated soon and replaced by this new API in the next alpha version: https://github.com/androidx/androidx/blob/26452388e174a6511bcfacb372259b6ebfd3cfd3/camera/camera-view/src/main/java/androidx/camera/view/PreviewView.java#L1059

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/2e2358ba-8d3c-4362-be64-3992c5acc997n%40android.com.

Talal Shah

unread,
Apr 18, 2024, 3:41:19 AMApr 18
to Android CameraX Discussion Group, xi...@google.com, Android CameraX Discussion Group, Talal Shah
In my current workflow, I am utilizing bitmap cropping to match the dimensions of the frame displayed in my camera preview before its transmission to TensorFlow. The frame is specified within an XML file, from which I retrieve the top, left, right, and bottom values for the cropping process I'm getting aspect ratio by  dividing (Image.width/previewView.width) and (Image.height/previewView.height). While this methodology performs adequately when the frame aspect ratio is set to .setTargetAspectRatio(AspectRatio.RATIO_16_9), I encounter difficulties when working with aspect ratios higher than this.

Given your previous suggestion, I find myself at a juncture of uncertainty regarding the viability of implementing the suggested approach within my specific context. Can you share an example?
Screenshot_20240417_162745_Keenu.jpg
imageAnalysis.setAnalyzer(Executors.newSingleThreadExecutor(), image -> {

int rotationDegrees = image.getImageInfo().getRotationDegrees();

final List<Detector.Recognition> results = detector.recognizeImage(croppedFrame, rotationDegrees);

});

Xi Zhang (张熹)

unread,
Apr 18, 2024, 10:51:34 AMApr 18
to Talal Shah, Android CameraX Discussion Group
I don't think using a high resolution preview helps your case. I assume that you are using the ImageAnlaysis output for calculation instead of preview. The preview resolution, no matter how high it is, is still limited by the resolution of the screen. So even if we use a super high resolution for preview, the user won't see the benefit of it. 

Also PreviewView might crop the preview stream if the aspect ratio of preview doesn't match the aspect ratio of the PreviewView, which is another factor why the coordinates don't align.

To calculate the transformation between PreviewView and ImageAnalysis, you can used the code sample in this document
// imageProxy is the output of an ImageAnalysis.
OutputTransform target = ImageProxyTransformFactory().getOutputTransform(imageProxy);
OutputTransform source = previewView.getOutputTransform();

// Build the transform from ImageAnalysis to PreviewView
CoordinateTransform coordinateTransform = new CoordinateTransform(source, target);

// Transform the bounding box.
coordinateTransform
.mapRect(boundingBox);
Then you can use the value of boundingBox to crop the ImageAnalysis output.

BTW I wonder how you get the Bitmap. Are you using ImageProxy#toBitmap? Cropping the Bitmap before sending to your ML model is inefficient because it requires an extra memory allocation and buffer copy. A better way would be sending the bounding box to your ML model and applying it there.

Talal Shah

unread,
Apr 18, 2024, 11:22:54 AMApr 18
to Xi Zhang (张熹), Android CameraX Discussion Group
Using a high resolution in imageAnalysis and Preview gives me higher resolution image. when cropping applied to the higher resolution image the coordinates don’t aligns. When AspectRatio is 16:9 I get an image of lets say (640x540) and cropping the image perfectly on the bounding boxes when the same code run with a higher resolution I get an image (1024x540) and bounding boxes don’t align the detected object and cropping result are not as expected. This is the main issue I have applied the code as per your suggestion it helps to draw the bounding boxes at perfect locations but cropping an image by using bounding box doesn't give the desired result

Xi Zhang (张熹)

unread,
Apr 18, 2024, 1:13:12 PMApr 18
to Talal Shah, Android CameraX Discussion Group
Using a high resolution in imageAnalysis and Preview gives me higher resolution image. when cropping applied to the higher resolution image the coordinates don’t aligns.

Only the ImageAnalysis needs the high resolution. The preview can stay at the screen resolution.
 
When AspectRatio is 16:9 I get an image of lets say (640x540) and cropping the image perfectly on the bounding boxes when the same code run with a higher resolution I get an image (1024x540) and bounding boxes don’t align the detected object and cropping result are not as expected. This is the main issue I have applied the code as per your suggestion it helps to draw the bounding boxes at perfect locations but cropping an image by using bounding box doesn't give the desired result

The code is for transforming PreviewView coordinates to ImageAnalysis coordinates. I assume that you specify the bounding box in PreviewView coordinates, and you want the ImageAnalysis output to match the bounding box in PreviewView, right?

Talal Shah

unread,
Apr 18, 2024, 1:49:04 PMApr 18
to Xi Zhang (张熹), Android CameraX Discussion Group
Yes this is exactly the same I want. I’ll keep the preview resolution same of screen resolution. I’ve transformed previewView coordinates to imageAnalysis using Transform class code snippet but I’m not getting the desired output. Can you please share the example for this

Xi Zhang (张熹)

unread,
Apr 18, 2024, 2:10:40 PMApr 18
to Talal Shah, Android CameraX Discussion Group
We don't have a code sample other than the java doc of CoordinateTransform.  If you can create a minimal reproducible example on GitHub, e.g. with only the cropping not the ML model, I might be able to debug it for you.

Talal Shah

unread,
Apr 18, 2024, 2:11:55 PMApr 18
to Xi Zhang (张熹), Android CameraX Discussion Group
Sure I’ll share it withyou soon

Talal Shah

unread,
Apr 19, 2024, 10:31:10 AMApr 19
to Android CameraX Discussion Group, Talal Shah, Android CameraX Discussion Group, xi...@google.com
Hi I've created a git repo you can run this I've added a comment in ImageAnalysis you can see the difference in doing comment/uncomment this line 
ImageAnalysis imageAnalysis = new ImageAnalysis.Builder()
//if you comment below line cropping works fine
.setTargetResolution(TargetResolutionHelper.getCalibratedTargetResolution(getApplicationContext(),getCameraId(getApplicationContext()), Surface.ROTATION_90, outputSizes[0]))
.setTargetRotation(this.getWindowManager().getDefaultDisplay().getRotation())
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.build();

Xi Zhang (张熹)

unread,
Apr 19, 2024, 1:58:43 PMApr 19
to Talal Shah, Android CameraX Discussion Group
Thanks for the code sample. It's really helpful. The missing part is in the MainActivity.java.

Instead of:
Camera camera = cameraProvider.bindToLifecycle(this, cameraSelector, preview, imageAnalysis);
Do:
UseCaseGroup useCaseGroup = new UseCaseGroup.Builder().addUseCase(preview).addUseCase(imageAnalysis).setViewPort(previewView.getViewPort()).build();
Camera camera = cameraProvider.bindToLifecycle(this, cameraSelector, useCaseGroup);
Basically, the deprecated transformation API requires all the UseCases in the same group with the ViewPort set. 

Talal Shah

unread,
Apr 19, 2024, 2:34:49 PMApr 19
to Xi Zhang (张熹), Android CameraX Discussion Group
Thanks for your help I’ll implement it. What should I use if I don’t use deprecated transformation API?

Xi Zhang (张熹)

unread,
Apr 19, 2024, 2:37:03 PMApr 19
to Talal Shah, Android CameraX Discussion Group
It's "soon to be" deprecated.:) The new API arrives in the next version.

As for the deprecated API, you can continue to use this unless there is an issue.

Talal Shah

unread,
Apr 19, 2024, 2:40:41 PMApr 19
to Xi Zhang (张熹), Android CameraX Discussion Group
Thanks for the clarification you’re really helpful

Talal Shah

unread,
Apr 24, 2024, 11:03:31 AMApr 24
to Android CameraX Discussion Group, Talal Shah, Android CameraX Discussion Group, xi...@google.com
I have implemented UseCases and it is working fine the image coordinate issue has been resolved now thanks for your quick response I'm able to crop the image to the perfect size. I have implemented TensorFlow lite in the project, While detecting objects I'm facing issues in the camera preview the bounding boxes of the object are not aligned to the object they are moving towards the right side as I have used the same Coordinate conversion previously used in image cropping also the preview is hanging on high resolution and the bounding boxes are not drawing smoothly in the camera preview I have created another branch in the project you can simple run to check the behavior https://github.com/Talal0/camerax_image_crop/tree/deveopment

Xi Zhang (张熹)

unread,
Apr 24, 2024, 12:17:43 PMApr 24
to Talal Shah, Android CameraX Discussion Group
My work laptop broke down and I am unable to check the code sample right now. But I think the issue is caused by using the wrong coordinate system. By cropping the image analysis output, you are adding a translation to the transformation, which you will need to compensate for. Assume your detected objects' coordinates are (x, y). Instead of transforming (x,y), you need to transform (x+cropRect.left, y +cropRect.top). Please let me know if that works.

Talal Shah

unread,
Apr 24, 2024, 1:48:28 PMApr 24
to Xi Zhang (张熹), Android CameraX Discussion Group
Im cropping image on button click but before this Im just drawing bounding boxes over object will this work on this? I’m using the same coordinate transformation to draw bounding boxes as I did at the time of cropping and what about the intermittent preview and drawing bounding boxes? 

Talal Shah

unread,
Apr 25, 2024, 5:32:31 AMApr 25
to Android CameraX Discussion Group, Talal Shah, Android CameraX Discussion Group, xi...@google.com
The issue is resolved now it was PreviewView scaleType issue I didn't set it in XML. The only issue I'm facing is the jerkiness of the camera preview when I set high resolution on ImageAnalysis bounding boxes are drawing very slow can you explain what I'm missing?

Xi Zhang (张熹)

unread,
Apr 25, 2024, 1:37:54 PMApr 25
to Talal Shah, Android CameraX Discussion Group
It would be helpful if you can provide a screen recording to show your definition of jerkiness. My guess is that the time spent analyzing a frame is too high. To optimize it, you could first profile the code and see where the bottleneck is. I don't know the details of your solution but two things come to mind for you to try:

1. Do not crop the image before sending it to the CV algorithm. Instead, pass the crop rect to the CV and use it to ignore the invalid area.
2. Configure ImageAnalysis to use a lower resolution. Usually ML models do not require high resolution input but your use case might be different.


Talal Shah

unread,
Apr 25, 2024, 1:45:43 PMApr 25
to Xi Zhang (张熹), Android CameraX Discussion Group
I’ll share you the screen recording. I’m not cropping the image just sending as it is to the ML model for the detection, cropping logic is working on the button only. If i use the lower resolution this will lead me to get a pixelated image because on default resolution I get 640x520 image after the object detection (identity card) I’m cropping the image of the object but due to low resolution cropped image get pixelated 
Reply all
Reply to author
Forward
0 new messages