Real time filters with capture support

[Astha Garg]

Hi Team,

I'm using camerax 1.1.0 and my requirement is to support real time filters like grayscale and Black & white, in which the user should be able to see grayscale ( or B&W ) applied on Preview and he should be able to capture that filter applied to the preview as well.

I tried to show the grayscale applied Preview using ImageAnalysis but on capturing the image, it is not the grayscale one.

Also, I tried using setCaptureRequestOption to set CONTROL_EFFECT_MODE as CONTROL_EFFECT_MODE_MONO to get greyscale results but it also did not work for me.

Can you guide me on how can I achieve this, I’m not finding much information about this.

Thanks,

Astha

[Charcoal Chen]

Hi,

I modified the CameraXBasic sample app to run on an Galaxy Note10+ device:

1. Updated to use CameraX 1.1.0 here

2. Updated to apply CONTROL_EFFECT_MODE_MONO on either Preview, ImageCapture or ImageAnalysis here

    val builder = ImageAnalysis.Builder()
    val camera2InterOp = Camera2Interop.Extender(builder)

    camera2InterOp.setCaptureRequestOption(CaptureRequest.CONTROL_EFFECT_MODE,
    CameraMetadata.CONTROL_EFFECT_MODE_MONO)

    // ImageAnalysis
    imageAnalyzer = builder

        ...

        .build()

I can see the preview and the captured still images are in mono mode. Not all devices support CONTROL_EFFECT_MODE_MONO. Could you check whether the device you tested on supports the mono mode by retrieving CameraCharacteristics.CONTROL_AVAILABLE_EFFECTS from CameraCharacteristics?

[Xi Zhang (张熹)]

Thank you for reaching out Astha. 

As I mentioned on my personal GitHub and StackOverflow, the team is actively working on a feature that applies post-processing effects to output. Currently in the latest alpha release, you can apply a GPU effect to preview with the CameraEffect API.  An example of a tone mapping effect(e.g. grayscale): https://github.com/androidx/androidx/blob/androidx-main/camera/integration-tests/viewtestapp/src/main/java/androidx/camera/integration/view/ToneMappingPreviewEffect.kt 

Using the output of ImageAnalysis for preview is inefficient. We do not recommend that.

In the coming alpha release, there will be an API for applying effects to ImageCapture too. You can find a tone mapping code sample here: https://github.com/androidx/androidx/blob/androidx-main/camera/integration-tests/viewtestapp/src/main/java/androidx/camera/integration/view/ToneMappingImageEffect.kt

Please let me know if you have more questions. I would also like to hear more about your use case. e.g. what are you building? What are some other effects that you want to apply?

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/7dc5904b-0c5c-46a8-8de6-f09a4b4c95d0n%40android.com.

[Astha Garg]

Thanks, Xi & Charcoal for your reply.

Charcoal, 

If CONTROL_EFFECT_MODE_MONO is not supported by all the devices, I think I shouldn't use this API otherwise all users will have a different experience, rather I referred your reply https://medium.com/p/d73c09b4b729

Xi

Glad that the google team is working on Effects API. But since it's in alpha I was thinking to wait for a stable release before using it in my production Application.

For now, I have referred to this reply from Charcoal (Point 2 
https://medium.com/p/d73c09b4b729). So, I created an ImageView over Preview of CameraX and used the output of ImageAnalysis(converting the ImageProxy to Grayscale bitmap) to show in that ImageView and that's how I'm able to show grayscale Preview. And to support Capturing the grayscale preview, I'm using the output file in onImageSaved() and converting to grayscale and saving it. 

With the above implementation, I'm able to implement the requirement I have but wanted to know any more optimised & direct way to support it.

[Xi Zhang (张熹)]

As I mentioned, using ImageAnalysis output for preview is inefficient. It requires an extra download/uploading image from/to GPU, as well as a YUV->RGB conversion in CPU. This adds a latency and puts a burden on the system health such as battery, heat and memory. Depending on your use case, you may run into more stability issues on lower end devices.

However, since you do not want to depend on alpha releases, you will have to continue with the current path.  

[Astha Garg]

Hi Xi,

Do we have any tentative date for the release of CameraEffect API?

And meanwhile, do we have any codelab or sample code for using the CameraEffect API to support showing the grayscale preview and capturing the same?

[Xi Zhang (张熹)]

The effects API for Preview is already in alpha release. The API for ImageCapture is scheduled to release soon in the next alpha version. As for the stable version, it's undecided at the moment.

You can take a look at the source code of our test app for code samples:

A preview tone mapping effect: https://github.com/androidx/androidx/blob/androidx-main/camera/integration-tests/viewtestapp/src/main/java/androidx/camera/integration/view/ToneMappingPreviewEffect.kt 

A ImageCapture tone mapping effect: https://github.com/androidx/androidx/blob/androidx-main/camera/integration-tests/viewtestapp/src/main/java/androidx/camera/integration/view/ToneMappingImageEffect.kt

[Astha Garg]

Hi Charcoal, Xi

Thanks for the code of the test App.

Until the CameraEffectAPI is stable, I would like to know if there is any way to update the Preview of camerax ?

Since creating bitmap repetitively and setting in an overlay ImageView is not efficient(from analyze method which is called very frequently ), I was thinking I can update the preview with the selected effect somehow?

[Xi Zhang (张熹)]

Without the effects API, you will have to basically implement the effects API yourself. Our core test app provides an example of using OpenGL to process the preview. You need to do the same in your app, then modify the shader here to provide a grayscale effect. It