CameraX with Selfie segmentation | ML Kit

[Yogendra Singh]

Hi,

I want to replace background with image or color using Selfie segmentation.
How can i do recording with background image or color?

[Xi Zhang (张熹)]

Hi Yogendra:

It's possible to MLKit's segmentation features replace the video background, but it will require some work. There is one idea:

1. Use ImageAnalysis to send the ImageProxy to MLKit and get the mask. Note that the mask is in the ImageAnalysis' coordinate system, so it has to be transformed to the video output coordinate system before applying. 

2. The CameraEffect API is for adding special effects to CameraX output. You can use OpenGL to process the input. One example can be found here: link

3. Then, in your OpenGL shader, blend the mask, the input image, and the background image to replace the background.

Thanks,

Xi

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/bbe152b9-f3fd-402e-a72a-adf56fd1e18fn%40android.com.

[Yogendra Singh]

Hi Xi,
Thanks for the guidance.

3. Then, in your OpenGL shader, blend the mask, the input image, and the background image to replace the background.

Can you guide for that? How i can use that class OpenGlRenderer  to blend the detected mask and background image?

[Xi Zhang (张熹)]

Here are some code samples:

- Upload image as GL texture.

- Blend texture on top of camera input.

The folder contains code of laying over an image on top of camera input. Your case is more complicated, but it should be similar.