Noise on Pixel 5 when displaying preview and performing analysis

[Josh Hight]

I'm noticing some unusual flickering noise on the Pixel 5 when attempting to perform analysis on frames at 60 fps while also displaying a preview. This doesn't seem to happen when capturing frames at 30 fps, or when there is no preview.

I've created a simple demo project to reproduce the issue, and have captured the results in a spreadsheet, showing an example analysis output at 60 fps with and without preview and 30 fps with and without preview. I also ran the demo against a Samsung Galaxy S20 FE for comparison. These scenarios are represented by different tabs in the spreadsheet.

To reproduce the issue, please check out the project and modify SHOW_PREVIEW and FRAMERATE constants in MainActivity.kt. The demo project simply captures frame luma for 10 seconds, optionally displaying a preview, and when finished prints out each frame's total luma. I tested both the Pixel 5 and Galaxy S20 FE by resting them a few millimeters off a dark mat. If your results are like mine, the Pixel 5's preview will be very noisy when displaying a preview and requesting a framerate of 60 fps.

To better illustrate the point, here's a few graphs.

My questions:

1. Is this a bug in CameraX's support for the Pixel 5, or am I using one of the Camera2 request options incorrectly?
   
2. I'm reading CameraCharacteristics.CONTROL_AE_AVAILBLE_TARGET_FPS_RANGES to determine the best (highest) framerate to capture frame data. Is that a valid approach?
   
3. If this isn't a bug, is there a recommended approach for avoiding this kind of noise?
   

[Eino-Ville Talvala]

Hi Josh -

A few thoughts to narrow down possible causes:

1. Have you tried this with outdoor lighting, or in general with any scene that isn't very dark (which it sounds like you're doing, pointing at a dark mat)?
2. It doesn't look like you're actually getting 60fps output for either case on Pixel 5.  60fps isn't guaranteed for all output sizes and combinations, unfortunately, but you'd probably have to use Camera2 directly to query the minimum frame durations supported for each size/format output.  
3. In the live preview, this looks like an occasional burst of noise, not a change in overall brightness? Or do you have sample images/video of what it looks like?

Thanks!

- Eino-Ville

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/2e2fc24d-4f32-43db-b9e3-19d7bf69937fn%40android.com.

[Josh Hight]

I realize my test sounds a bit contrived, but our typical use case will involve the possibility of low light. I did record an example using the demo app outdoors and pointing at a bright screen, and the flickering was present in both.

[Josh Hight]

I should probably clarify that the greenish hue and the lack of focus in the videos aren't an issue. And it's not present in the demo code, but I am querying Camera2 to determine that 60 fps is possible on the Pixel 5. I am able to achieve this framerate sometimes.

The primary issue for us is just the flickering brightness, which is only present when requesting 60 fps and displaying a preview.

[Eino-Ville Talvala]

My best guess, on no real evidence, is that this is an instability in the auto-exposure algorithm.  And that adding in preview is changing what sensor mode or use case the camera implementation believes you're aiming for, which is changing the AE algorithm behavior.

You could try to observe the capture result values for the exposure and sensitivity values the AE is selecting, to see if they have a jump that matches the noise you see.  Should be feasible via https://developer.android.com/reference/kotlin/androidx/camera/camera2/interop/Camera2Interop.Extender#setSessionCaptureCallback(android.hardware.camera2.CameraCaptureSession.CaptureCallback) and https://developer.android.com/reference/android/hardware/camera2/CameraCaptureSession.CaptureCallback.html#onCaptureCompleted(android.hardware.camera2.CameraCaptureSession,%20android.hardware.camera2.CaptureRequest,%20android.hardware.camera2.TotalCaptureResult) to get a https://developer.android.com/reference/android/hardware/camera2/CaptureResult.  I'd check the following outputs:

• https://developer.android.com/reference/android/hardware/camera2/CaptureResult#SENSOR_EXPOSURE_TIME
  
• https://developer.android.com/reference/android/hardware/camera2/CaptureResult#SENSOR_FRAME_DURATION
  
• https://developer.android.com/reference/android/hardware/camera2/CaptureResult#SENSOR_SENSITIVITY
  
• https://developer.android.com/reference/android/hardware/camera2/CaptureResult#CONTROL_POST_RAW_SENSITIVITY_BOOST
  

If they're all stable, then it's probably not the AE acting up, and it's a bit more mysterious.

[Josh Hight]

Thanks for the help, Eino-Ville. It seems I was able to remove the flickering by adding a NOISE_REDUCTION_MODE capture request option with a value of NOISE_REDUCTION_MODE_HIGH_QUALITY.