Camerax ImageAnalysis randomly stop working

[Talal Shah]

I’ve integrated CameraX ImageAnalysis in my application along with the TensorFlow Lite library. The detection works properly, and I’m receiving images from the analyzer as expected. According to the application flow, after detecting an object, the app should navigate to another screen to display it and then return to the main detection screen. However, after returning, the camera sometimes stops detecting, and the image analyzer stops providing images. Additionally, my detection process is running too slowly

ListenableFuture<ProcessCameraProvider> cameraProviderFuture = ProcessCameraProvider.getInstance(this);

cameraProviderFuture.addListener(() -> {
try {
ProcessCameraProvider cameraProvider = cameraProviderFuture.get();

// Set up the preview use case
Preview preview = new Preview.Builder()
.setTargetAspectRatio(AspectRatio.RATIO_4_3)
.build();
ImageAnalysis.Builder builder1 = new ImageAnalysis.Builder();

//Images are processed by passing an executor in which the image analysis is run
//
Camera2Interop.Extender ext = new Camera2Interop.Extender<>(builder1);
ext.setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, TotalCaptureResult.CONTROL_AF_MODE_AUTO)
.setCaptureRequestOption(CaptureRequest.CONTROL_AWB_MODE, CaptureRequest.CONTROL_AWB_MODE_AUTO)
.setCaptureRequestOption(CaptureRequest.SENSOR_SENSITIVITY, 100)
.setSessionCaptureCallback(sessionCaptureCallback);
// Set up the image analysis use case
imageAnalysis = builder1
.setResolutionSelector(new ResolutionSelector.Builder()
.setAspectRatioStrategy(AspectRatioStrategy.RATIO_4_3_FALLBACK_AUTO_STRATEGY)
.setResolutionStrategy(new ResolutionStrategy(new Size(1920, 1080),
ResolutionStrategy.FALLBACK_RULE_CLOSEST_LOWER)).build())
.setTargetName("ImageAnalysis")
.setTargetRotation(this.getWindowManager().getDefaultDisplay().getRotation())
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.build();

initListener();

preview.setSurfaceProvider(previewView.getSurfaceProvider());
imageAnalysis.setAnalyzer(ContextCompat.getMainExecutor(this), new ImageAnalyzer(this ,bitmapAnalyzedListener, previewView, borderCamera,afStateMap,belowPreviewTextView,overlayView));
// Select the camera and bind to lifecycle
CameraSelector cameraSelector = new CameraSelector.Builder()
.requireLensFacing(CameraSelector.LENS_FACING_BACK)
.build();

UseCaseGroup useCaseGroup = new UseCaseGroup.Builder().addUseCase(preview).addUseCase(imageAnalysis).setViewPort(previewView.getViewPort()).build();
camera = cameraProvider.bindToLifecycle(this, cameraSelector, useCaseGroup);
setUpTapToFocus();
} catch (InterruptedException | ExecutionException e) {
Log.e("CameraXApp", "Error binding camera use cases", e);
}
}, ContextCompat.getMainExecutor(this));

[Charcoal Chen]

Hi,

Thanks for reporting the issue.

Can you help to confirm that the images provided by the ImageAnalysis have been closed by your ImageAnalyzer implementation. If the images are not correctly closed, it might block the analyzer from receiving the new images. Please see the doc of ImageAnalysis#Analyzer.

• It is the responsibility of the application to close the image once done with it. If the images are not closed then it may block further images from being produced (causing the preview to stall) or drop images as determined by the configured backpressure strategy.

If it is not the root cause, could you explain more about the following situation:

• after returning, the camera sometimes stops detecting, and the image analyzer stops providing images

When the ImageAnalysis gets the problem, does the Preview still work normally?

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/596ffae8-ce9f-430f-9192-9c731830bba2n%40android.com.

[Talal Shah]

Below is the analyzer() code. I'm closing the image after processing, but my main issue is that the preview is jerky and is lagging and appearing choppy., and detected objects are drawn slowly. This occurs on devices like the Samsung A54 5G and earlier models (Samsung A31, Samsung A33, and Redmi 12 Pro). However, it works fine on newer devices like the Redmi 13 Pro and Samsung Galaxy S24. Sometimes image analysis stops working without any errors in Logcat. Other times, it shows the following error: Failed to get service from broker. java.lang.SecurityException: Unknown calling package name 'com.google.android.gms'.

@Override
@ExperimentalGetImage
@TransformExperimental
public void analyze(ImageProxy image) {
if(isProcessing){
image.close();
return;
}
int afState = afStateMap.size() != 0 ? afStateMap.containsKey(image.getImageInfo().getTimestamp()) ? afStateMap.remove(image.getImageInfo().getTimestamp()) : 0: 0;
// Your image analysis logic here
// Access the image data using image.getImage() method
objectDetectorHelper.detect(image, image.getImageInfo().getRotationDegrees());
// Call the listener with the bitmap
if (listener != null) {
if (TotalCaptureResult.CONTROL_AF_MODE_CONTINUOUS_PICTURE == afState || TotalCaptureResult.CONTROL_AF_STATE_PASSIVE_FOCUSED == afState || TotalCaptureResult.CONTROL_AF_STATE_ACTIVE_SCAN == afState || TotalCaptureResult.CONTROL_AF_STATE_FOCUSED_LOCKED == afState) {
if(isCardInsideFrame){
if (!isCameraFocusStarted) {
isCameraFocusStarted = true;
mainActivity.autoFocusInFrame();
}
String cardSide = determineDetectedCardType(detectedResults);
if(cardSide !=null) {
if (cardSide.equals(Constants.cnicCardSide)) {
processCardSide(cardSide, image, detectedResults, listener);
}
}else{
isCameraFocusStarted = false;
isProcessing = false;
isTextRecognitionCompleted = false;
mainActivity.setFocusStatus(false);
}
}else{
isCameraFocusStarted = false;
isProcessing = false;
isTextRecognitionCompleted = false;
mainActivity.setFocusStatus(false);
}
}
}
image.close();
}

[Charcoal Chen]

Hi,

This might be an image analyzing performance related issue. Because the Redmi 13 Pro and Samsung Galaxy S24 devices have more powerful hardware, the issue doesn't occur on them.

For general case, the FPS might need to be closer to 30 to make the preview look smoothly. Since your app need to analyze the ImageAnalysis images and then apply some effects on the preview. You will need to make sure that it won't take too much time to impact the preview FPS. From the provided code, I can't estimate how long the image-analyzing + preview-drawing flow will take. Could you have some testing in your side to get more info about this issue?

1. Directly close the images when your app receives them via the Analyzer and see whether the issue still occurs (jerky preview, analysis stop working). This might be able to quickly clarify whether the issue is caused by the device/CameraX library or by the image-analyzing + preview-drawing flow in your app.
2. Print some log to know whether the  image-analyzing + preview-drawing flow in your app can reach 30 FPS  on Samsung A54 or the other earlier modes?
3. Use try-catch-finally and invoke image.close in the finally code block to make sure the images are all correctly closed. In case some exceptions might occur in your implementation to make the image not be closed. Then, it might occupy the available image proxies and block the image pipeline.

If possible, it will be great to provide us a minimal workable project that can reproduce the issue. Then, we can use it to clarify the issue root cause. 

[Hacking with sinistervibranium]

Hi sorry but who are you how you know me

To view this discussion visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/40517e3d-6055-4427-bc38-2246ba274f00n%40android.com.

[Scott Nien]

>Hi sorry but who are you how you know me

I think you join the CameraX developers groups https://groups.google.com/a/android.com/g/camerax-developers. That's the reason you receive the update.  You can unsubscribe by the instruction attached at the end of email.