Camerax ImageAnalysis image blurr issue

[Talal Shah]

I am currently utilizing Camerax in conjunction with TensorFlow Lite for the purpose of detecting identity cards of users, followed by text extraction using MLKit. However, I am encountering two significant challenges outlined below:

1. Image Blurr Issue: Upon processing frames obtained through Camerax and passing them to TensorFlow Lite for object detection, a prevalent issue arises wherein a considerable portion of the acquired images exhibit blurriness, significantly impacting data reading accuracy through MLKit.

2. Camera Focus: Despite implementing camera focus functionality within the application code, the camera's continuous frame capturing during the focusing process presents a hurdle. This presents difficulty in distinguishing frames post-focusing, leading to inconsistent results. Below is a snippet of the implemented code for focusing the camera:

ListenableFuture<FocusMeteringResult> getCam = camera.getCameraControl().startFocusAndMetering(focusMeteringAction); FocusMeteringResult camResult = getCam.get(); if (camResult.isFocusSuccessful()) { System.out.println("isFocusSuccessful"); } else { // Focus has failed System.out.println("Focus has failed"); }

Despite executing the above code to initiate camera focus, the captured frames following the isFocusSuccessful() evaluation often display blurriness.

In light of these challenges, I am seeking guidance and potential solutions to enhance the image clarity and focus consistency within the Camerax and MLKit integration framework. Any insights or recommendations from the community would be greatly appreciated.

[Xi Zhang (张熹)]

Is the blurriness caused by being out-of-focus? Do you have an example of a blurry image? You can use the Camera2Interop API to check if a frame is out of focus, and discard them if necessary. 

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/3d80ac20-c5ae-41ad-bd84-4909fe619e1fn%40android.com.

[Scott Nien]

hi , 

Can you give more details regarding how you construct the FocusMeteringAction and MeteringPoint ?  
Typically,  startFocusAndMetering is used to implement tap-to-focus to focus on some area.  And if you just need to get a good focus image,  you don't have to do anything,  By default the continuous AF is enabled. 
Have you tried checking the image blurriness without invoking startFocusAndMetering ?  

--

[Xi Zhang (张熹)]

You are right that ImageAnalysis provides continuous frames regardless of whether the frames are focused. The 2~3 blurry frames after the focus is completed may be due to those frames being buffered up by CameraX thus delivered with a delay. Either way, you can use  Camera2Interop to get the frame metadata and check the CaptureResult.CONTROL_AF_STATE field.

On Mon, Apr 1, 2024 at 2:47 AM Talal Shah <talal...@gmail.com> wrote:

ImageAnalysis returns continuous frames while camera focus, when camera focus completed it also returns blurry image then after 2-3 frames it provides correct image

Sample Blurr Image:

[Talal Shah]

can you provide an example to achieve this I'm using ImageAnalysis I'm unable to get frame metadata 

[Xi Zhang (张熹)]

ImageAnalysis.Builder builder = new ImageAnalysis.Builder();

new Camera2Interop.Extender<>(builder).setSessionCaptureCallback(
        new CameraCaptureSession.CaptureCallback() {
            @Override
            public void onCaptureCompleted(@NonNull CameraCaptureSession session,
                    @NonNull CaptureRequest request, @NonNull TotalCaptureResult result) {
                super.onCaptureCompleted(session, request, result);
                int afState = result.get(TotalCaptureResult.CONTROL_AF_STATE);
                long timestamp = result.get(CaptureResult.SENSOR_TIMESTAMP);
            }
        });

The code above is an example of how to get the capture result of a frame with Camera2Interop. Then you need to match the timestamp to the timestamp of the ImageProxy so that you will know the AF state of the ImageProxy.

[Talal Shah]

I've implemented this code but timestamp never match, am I doing any wrong?

CameraCaptureSession.CaptureCallback sessionCaptureCallback = new CameraCaptureSession.CaptureCallback() {

@Override
public void onCaptureCompleted(@NonNull CameraCaptureSession session, @NonNull CaptureRequest request, @NonNull TotalCaptureResult result) {
super.onCaptureCompleted(session, request, result);

// boolean areWeFocused = false;
int afState = result.get(TotalCaptureResult.CONTROL_AF_STATE);
timestamp = result.get(CaptureResult.SENSOR_TIMESTAMP);

imageAnalysis.setAnalyzer(Executors.newSingleThreadExecutor(), image -> {
if(image.getImageInfo().getTimestamp() == timestamp){
areWeFocused = true;
}
});

[Xi Zhang (张熹)]

The timestamp should match. The ImageProxy usually arrives later than the CaptureResult, so you need to put the CaptureResult in a cache then try to match incoming ImageProxy with all the CaptureResult in the cache. 

[Talal Shah]

I did the same as you suggest im caching timestamp in a global variable and comparing it with imageproxy but what i see onCaptureCompleted is calling every time and the cached timestamp is replaced with the new one. I’ve shared my 

onCaptureCompleted implementatIon is this correct? 

[Xi Zhang (张熹)]

You need to put the CaptureResult in a Map:

Map<Long, Integer> afStateMap = new HashMap<>();

new Camera2Interop.Extender<>(builder).setSessionCaptureCallback(
        new CameraCaptureSession.CaptureCallback() {
            @Override
            public void onCaptureCompleted(@NonNull CameraCaptureSession session,
                    @NonNull CaptureRequest request, @NonNull TotalCaptureResult result) {
                super.onCaptureCompleted(session, request, result);
                int afState = result.get(TotalCaptureResult.CONTROL_AF_STATE);
                long timestamp = result.get(CaptureResult.SENSOR_TIMESTAMP);

                afStateMap.put(timestamp, afState);
            }
        });

Then, you can get the focus state of an ImageProxy by:

imageAnalysis.setAnalyzer(Executors.newSingleThreadExecutor(), image -> {
    int afState = afStateMap.remove(image.getImageInfo().getTimestamp());
});

Hope this helps.

[Talal Shah]

I used this code but imageProxy afState is getting 0 everytime even after camera focus.

This is my Camera2Intop

Camera2Interop.Extender ext = new Camera2Interop.Extender<>(builder1);
ext.setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_AUTO)
.setCaptureRequestOption(CaptureRequest.CONTROL_AWB_MODE, CaptureRequest.CONTROL_AWB_MODE_AUTO)
.setCaptureRequestOption(CaptureRequest.SENSOR_SENSITIVITY, 100)
.setSessionCaptureCallback(sessionCaptureCallback);

CameraCaptureSession.CaptureCallback sessionCaptureCallback = new CameraCaptureSession.CaptureCallback() {
@Override
public void onCaptureCompleted(@NonNull CameraCaptureSession session, @NonNull CaptureRequest request, @NonNull TotalCaptureResult result) {
super.onCaptureCompleted(session, request, result);

Integer afState = result.get(TotalCaptureResult.CONTROL_AF_STATE);

long timestamp = result.get(CaptureResult.SENSOR_TIMESTAMP);

afStateMap.put(timestamp, afState);

}

}

imageAnalysis.setAnalyzer(Executors.newSingleThreadExecutor(), image -> {

int afState =  afStateMap.remove(image.getImageInfo().getTimestamp());

})

[Talal Shah]

Another issue I have observed is that I set autofocus in Camera2Introp but the camera is not focusing I see blurry preview when I tap on screen then it focus because I have used setOnTouchListner() event on previewView 

  Camera2Interop.Extender ext = new Camera2Interop.Extender<>(builder1);
ext.setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_AUTO)
.setCaptureRequestOption(CaptureRequest.CONTROL_AWB_MODE, CaptureRequest.CONTROL_AWB_MODE_AUTO)
.setCaptureRequestOption(CaptureRequest.SENSOR_SENSITIVITY, 100)
.setSessionCaptureCallback(sessionCaptureCallback);

[Tahsin Masrur]

Another issue I have observed is that I set autofocus in Camera2Introp but the camera is not focusing I see blurry preview when I tap on screen then it focus because I have used setOnTouchListner() event on previewView 
  Camera2Interop.Extender ext = new Camera2Interop.Extender<>(builder1);
ext.setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_AUTO)

Hi, just wanted to add in - the behavior you are seeing with CONTROL_AF_MODE_AUTO works as intended according to the API documentation for that AF mode.

Basically, with AF_MODE_AUTO, an explicit AF trigger request has to be submitted to the camera for the lens to move and apply auto-focus based on the metering/focus regions set to the camera. Once it is done (i.e. AF state is in FOCUSED/NOT_FOCUSED), the AF is locked to that state until a new AF trigger request is submitted.

This is why, unless you are tapping on the screen (CameraX triggers an AF trigger request for that), the focus is not working as you intend and the AF state is shown as 0.

If you want the AF to continuously keep working without focusing on a specific area by tapping, you can use CONTROL_AF_MODE_CONTINUOUS_PICTURE instead (or AF_MODE_CONTINUOUS_VIDEO based on your specific requirements). Actually, CameraX already does this for you by default which you are probably overriding by setting AF_MODE_AUTO in the Camera2Interop request. So maybe you can simply try with removing the following line first.

ext.setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_AUTO)

To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/b26ddab0-e571-4655-a3aa-74921834b09dn%40android.com.

[Talal Shah]

Thanks for your response it helped me a lot but sometimes I'm getting blurred images after getting AF status of the frame. This is the rectangular box in which I want to focus the camera. 

// Configure focus and metering action with autofocus point and duration
SurfaceOrientedMeteringPointFactory factory = new SurfaceOrientedMeteringPointFactory(1.0f,1.0f);
MeteringPoint autoFocusPoint = factory.createPoint(0.5f, 0.5f);

FocusMeteringAction focusMeteringAction = new FocusMeteringAction.Builder(
autoFocusPoint)
.build();

try {

ListenableFuture<FocusMeteringResult> getCam = camera.getCameraControl().startFocusAndMetering(focusMeteringAction);
FocusMeteringResult camResult = getCam.get();
if (camResult.isFocusSuccessful()) {
System.out.println("isFocusSuccessful");
} else {
// Focus has failed
System.out.println("Focus has failed");
}

} catch (ExecutionException e) // Thrown exceptions
{
e.printStackTrace();
image.close();
}