CameraX with Tensorflow camera preview overlay aspect ratio

[Talal Shah]

When utilizing CameraX with TensorFlow for object detection, I encountered an issue related to the positioning of rectangular detection boxes within the camera preview. Specifically, the overlay functionality functions correctly, accurately delineating objects within the specified rectangular region, when the preview resolution is set to `setTargetAspectRatio(AspectRatio.RATIO_16_9)`.

However, in order to obtain cropped images of the detected objects at higher resolutions, I utilize the `.setResolutionSelector()` method with a `ResolutionStrategy.HIGHEST_AVAILABLE_STRATEGY` parameter. This ensures that the preview resolution is set to the highest available option, which is essential for obtaining clear images of the detected objects. Maintaining a default resolution results in low-quality images, which, when cropped, further deteriorate in clarity.

Upon implementing the high-resolution settings within both the `ImageAnalyzer` and the Camera Preview, I encountered an unexpected behavior where the bounding boxes are drawn on the preview no longer aligned with the actual objects. This misalignment adversely affects the subsequent cropping process, leading to inaccuracies in the resultant images. Conversely, when utilizing the `AspectRatio.RATIO_16_9`, the bounding boxes are correctly positioned, and the cropping process functions as expected.

It is imperative to resolve this discrepancy to ensure accurate object detection and cropping at higher resolutions, in line with project requirements.

Below mentioned code is working when ImageAnalyzer and CameraPreview target resolution is set to .setTargetAspectRatio(AspectRatio.RATIO_16_9) 

This is my cropping method of rectangular box which is not working according to the expectation

public Bitmap croppedBitmap(Bitmap rotated) {

//calculate aspect ratio
float koefX = (float) rotated.getWidth() / (float) previewView.getWidth();
float koefY = (float) rotated.getHeight() / (float) previewView.getHeight();
//get viewfinder border size and position on the screen
int x1 = borderCamera.getLeft();
int y1 = borderCamera.getTop();

int x2 = borderCamera.getWidth();
int y2 = borderCamera.getHeight();

//calculate position and size for cropping
int cropStartX = Math.round(x1 * koefX);
int cropStartY = Math.round(y1 * koefY);

int cropWidthX = Math.round(x2 * koefX);
int cropHeightY = Math.round(y2 * koefY);

//check limits and make crop
if (cropStartX + cropWidthX <= rotated.getWidth() && cropStartY + cropHeightY <= rotated.getHeight()) {
/*
This is the cropped bitmap image we will pass it into create image file function
* */
croppedBitmap = Bitmap.createBitmap(rotated, cropStartX, cropStartY , cropWidthX, cropHeightY );
} else {
croppedBitmap = null;
}
return croppedBitmap;
}

This is the bounding boxes code 

private var scaleFactor: Float = 1f

val boundingBox = result.boundingBox
val top = boundingBox.top * scaleFactor
val bottom = boundingBox.bottom * scaleFactor
val left = boundingBox.left * scaleFactor
val right = boundingBox.right * scaleFactor

// Draw bounding box around detected objects
val drawableRect = RectF(left, top, right, bottom)
canvas.drawRect(drawableRect, boxPaint)

[Xi Zhang (张熹)]

The best way to transform coordinates between UseCases is using the sensor-to-buffer value that provided with the output.  In your case, you want to transform the rect in Preview coordinate system to ImageAnalysis coordinate system. The Transform output topic in the CameraX document has a code sample for this. 

Note that the API in that document is going to be deprecated soon and replaced by this new API in the next alpha version: https://github.com/androidx/androidx/blob/26452388e174a6511bcfacb372259b6ebfd3cfd3/camera/camera-view/src/main/java/androidx/camera/view/PreviewView.java#L1059

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/2e2358ba-8d3c-4362-be64-3992c5acc997n%40android.com.

[Talal Shah]

In my current workflow, I am utilizing bitmap cropping to match the dimensions of the frame displayed in my camera preview before its transmission to TensorFlow. The frame is specified within an XML file, from which I retrieve the top, left, right, and bottom values for the cropping process I'm getting aspect ratio by  dividing (Image.width/previewView.width) and (Image.height/previewView.height). While this methodology performs adequately when the frame aspect ratio is set to .setTargetAspectRatio(AspectRatio.RATIO_16_9), I encounter difficulties when working with aspect ratios higher than this.

Given your previous suggestion, I find myself at a juncture of uncertainty regarding the viability of implementing the suggested approach within my specific context. Can you share an example?

imageAnalysis.setAnalyzer(Executors.newSingleThreadExecutor(), image -> {

int rotationDegrees = image.getImageInfo().getRotationDegrees();

final List<Detector.Recognition> results = detector.recognizeImage(croppedFrame, rotationDegrees);

});

[Xi Zhang (张熹)]

I don't think using a high resolution preview helps your case. I assume that you are using the ImageAnlaysis output for calculation instead of preview. The preview resolution, no matter how high it is, is still limited by the resolution of the screen. So even if we use a super high resolution for preview, the user won't see the benefit of it. 

Also PreviewView might crop the preview stream if the aspect ratio of preview doesn't match the aspect ratio of the PreviewView, which is another factor why the coordinates don't align.

To calculate the transformation between PreviewView and ImageAnalysis, you can used the code sample in this document: 

// imageProxy is the output of an ImageAnalysis.
OutputTransform target = ImageProxyTransformFactory().getOutputTransform(imageProxy);
OutputTransform source = previewView.getOutputTransform();

// Build the transform from ImageAnalysis to PreviewView
CoordinateTransform coordinateTransform = new CoordinateTransform(source, target);

// Transform the bounding box.
coordinateTransform.mapRect(boundingBox);

Then you can use the value of boundingBox to crop the ImageAnalysis output.

BTW I wonder how you get the Bitmap. Are you using ImageProxy#toBitmap? Cropping the Bitmap before sending to your ML model is inefficient because it requires an extra memory allocation and buffer copy. A better way would be sending the bounding box to your ML model and applying it there.

[Talal Shah]

Using a high resolution in imageAnalysis and Preview gives me higher resolution image. when cropping applied to the higher resolution image the coordinates don’t aligns. When AspectRatio is 16:9 I get an image of lets say (640x540) and cropping the image perfectly on the bounding boxes when the same code run with a higher resolution I get an image (1024x540) and bounding boxes don’t align the detected object and cropping result are not as expected. This is the main issue I have applied the code as per your suggestion it helps to draw the bounding boxes at perfect locations but cropping an image by using bounding box doesn't give the desired result

[Xi Zhang (张熹)]

Using a high resolution in imageAnalysis and Preview gives me higher resolution image. when cropping applied to the higher resolution image the coordinates don’t aligns.

Only the ImageAnalysis needs the high resolution. The preview can stay at the screen resolution.

 

When AspectRatio is 16:9 I get an image of lets say (640x540) and cropping the image perfectly on the bounding boxes when the same code run with a higher resolution I get an image (1024x540) and bounding boxes don’t align the detected object and cropping result are not as expected. This is the main issue I have applied the code as per your suggestion it helps to draw the bounding boxes at perfect locations but cropping an image by using bounding box doesn't give the desired result

The code is for transforming PreviewView coordinates to ImageAnalysis coordinates. I assume that you specify the bounding box in PreviewView coordinates, and you want the ImageAnalysis output to match the bounding box in PreviewView, right?

[Talal Shah]

Yes this is exactly the same I want. I’ll keep the preview resolution same of screen resolution. I’ve transformed previewView coordinates to imageAnalysis using Transform class code snippet but I’m not getting the desired output. Can you please share the example for this

[Xi Zhang (张熹)]

We don't have a code sample other than the java doc of CoordinateTransform.  If you can create a minimal reproducible example on GitHub, e.g. with only the cropping not the ML model, I might be able to debug it for you.

[Talal Shah]

Sure I’ll share it withyou soon

[Talal Shah]

Hi I've created a git repo you can run this I've added a comment in ImageAnalysis you can see the difference in doing comment/uncomment this line 

GIt repo: https://github.com/Talal0/camerax_image_crop.git

ImageAnalysis imageAnalysis = new ImageAnalysis.Builder()
//if you comment below line cropping works fine
.setTargetResolution(TargetResolutionHelper.getCalibratedTargetResolution(getApplicationContext(),getCameraId(getApplicationContext()), Surface.ROTATION_90, outputSizes[0]))
.setTargetRotation(this.getWindowManager().getDefaultDisplay().getRotation())
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.build();

[Xi Zhang (张熹)]

Thanks for the code sample. It's really helpful. The missing part is in the MainActivity.java.

Instead of:

Camera camera = cameraProvider.bindToLifecycle(this, cameraSelector, preview, imageAnalysis);

Do:

UseCaseGroup useCaseGroup = new UseCaseGroup.Builder().addUseCase(preview).addUseCase(imageAnalysis).setViewPort(previewView.getViewPort()).build();
Camera camera = cameraProvider.bindToLifecycle(this, cameraSelector, useCaseGroup);

Basically, the deprecated transformation API requires all the UseCases in the same group with the ViewPort set. 

[Talal Shah]

Thanks for your help I’ll implement it. What should I use if I don’t use deprecated transformation API?

[Xi Zhang (张熹)]

It's "soon to be" deprecated.:) The new API arrives in the next version.

As for the deprecated API, you can continue to use this unless there is an issue.

[Talal Shah]

Thanks for the clarification you’re really helpful

[Talal Shah]

I have implemented UseCases and it is working fine the image coordinate issue has been resolved now thanks for your quick response I'm able to crop the image to the perfect size. I have implemented TensorFlow lite in the project, While detecting objects I'm facing issues in the camera preview the bounding boxes of the object are not aligned to the object they are moving towards the right side as I have used the same Coordinate conversion previously used in image cropping also the preview is hanging on high resolution and the bounding boxes are not drawing smoothly in the camera preview I have created another branch in the project you can simple run to check the behavior https://github.com/Talal0/camerax_image_crop/tree/deveopment

[Xi Zhang (张熹)]

My work laptop broke down and I am unable to check the code sample right now. But I think the issue is caused by using the wrong coordinate system. By cropping the image analysis output, you are adding a translation to the transformation, which you will need to compensate for. Assume your detected objects' coordinates are (x, y). Instead of transforming (x,y), you need to transform (x+cropRect.left, y +cropRect.top). Please let me know if that works.

[Talal Shah]

Im cropping image on button click but before this Im just drawing bounding boxes over object will this work on this? I’m using the same coordinate transformation to draw bounding boxes as I did at the time of cropping and what about the intermittent preview and drawing bounding boxes? 

[Talal Shah]

The issue is resolved now it was PreviewView scaleType issue I didn't set it in XML. The only issue I'm facing is the jerkiness of the camera preview when I set high resolution on ImageAnalysis bounding boxes are drawing very slow can you explain what I'm missing?

[Xi Zhang (张熹)]

It would be helpful if you can provide a screen recording to show your definition of jerkiness. My guess is that the time spent analyzing a frame is too high. To optimize it, you could first profile the code and see where the bottleneck is. I don't know the details of your solution but two things come to mind for you to try:

1. Do not crop the image before sending it to the CV algorithm. Instead, pass the crop rect to the CV and use it to ignore the invalid area.

2. Configure ImageAnalysis to use a lower resolution. Usually ML models do not require high resolution input but your use case might be different.

[Talal Shah]

I’ll share you the screen recording. I’m not cropping the image just sending as it is to the ML model for the detection, cropping logic is working on the button only. If i use the lower resolution this will lead me to get a pixelated image because on default resolution I get 640x520 image after the object detection (identity card) I’m cropping the image of the object but due to low resolution cropped image get pixelated