ImageAnalysis: detecting when frames are dropped

[Josh Hight]

I'm currently using an ImageAnalysis instance with a STRATEGY_KEEP_ONLY_LATEST backpressure strategy. The documentation for this strategy mentions:

If more images are produced while that image is being analyzed, they will be dropped and not queued for delivery.

I'm trying to detect when frames are dropped. If I implement a CameraCaptureSession.CaptureCallback instance on the Camera2InteropExtender, will its onCaptureBufferLost handler method fire in this scenario? If not, is there another approach I can take?

[Scott Nien]

There is no way to detect the frames drop in STRATEGY_KEEP_ONLY_LATEST mode in current design, unfortunately. 

Could you tell us what the purpose of detecting frame drop is ?  That will help us rethink the design. 

Using STRATEGY_BLOCK_PRODUCER instead may work.. since app will get every frame but it now becomes app's responsibility to close the image in time otherwise the preview will be stalled

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/c1efd28b-fd5d-4bbb-8988-bc62760a29ffn%40android.com.

[Josh Hight]

Thanks for your reply, Scott. We're performing a measurement that requires observing certain data in sequence, at the fastest rate possible. Ideally, we'd the data to be clean, so in the event of a frame being dropped we would restart the measurement.

Given these requirements, would STRATEGY_KEEP_ONLY_LATEST or STRATEGY_BLOCK_PRODUCER be the recommended backpressure strategy? Does using STRATEGY_BLOCK_PRODUCER allow us to be notified when a frame is dropped?

[Scott Nien]

Using STRATEGY_BLOCK_PRODUCER means you will get EVERY frame so basically no frame will be dropped.   But this also means you must carefully close the Image quickly otherwise it will stall the preview. 

You can also call setImageQueueDepth  to enlarge the image queue which mitigates the preview stall but generally you should process the frame as quickly as possible. 

[Josh Hight]

Is there a recommended approach in CameraX for catching the IllegalStateException that occurs when the maximum number of images have been acquired when using STRATEGY_BLOCK_PRODUCER?

[Scott Nien]

That is the issue CameraX should fix. The RC03 release which will be released today or tomorrow will contain the fix that catches the IllegalStateException of maxImage.  

[Patrick Liao]

Hi Josh and Scott,

I do have an idea regarding drop frame detection.

Say the shutter speed is 1/60s (Which is typically used for shooting 30fps videos), then you would expect frames coming in every 16ms (1000ms / 60 = 16ms). If a frame comes in around 16ms, I would say that it is probably fine. By contrast, if the next frame took 30ms or even 40ms to come in, that's when I can tell that something is going on...

This is what I am currently using for debugging my analyzer code, not sure if this helps (For video, I lock AE_TARGET_FPS_RANGE to 50 so I expect the shutter speed to be 1/50s~1/100s and the frames should come in every 20ms).

Best,

Patrick Liao