CameraX with Selfie segmentation | ML Kit

341 views
Skip to first unread message

Yogendra Singh

unread,
Sep 28, 2023, 2:48:56 PM9/28/23
to Android CameraX Discussion Group
Hi,
I want to replace background with image or color using Selfie segmentation.
How can i do recording with background image or color?

Xi Zhang (张熹)

unread,
Sep 28, 2023, 3:27:36 PM9/28/23
to Yogendra Singh, Android CameraX Discussion Group
Hi Yogendra:

It's possible to MLKit's segmentation features replace the video background, but it will require some work. There is one idea:

1. Use ImageAnalysis to send the ImageProxy to MLKit and get the mask. Note that the mask is in the ImageAnalysis' coordinate system, so it has to be transformed to the video output coordinate system before applying. 
2. The CameraEffect API is for adding special effects to CameraX output. You can use OpenGL to process the input. One example can be found here: link
3. Then, in your OpenGL shader, blend the mask, the input image, and the background image to replace the background.

Thanks,
Xi

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/bbe152b9-f3fd-402e-a72a-adf56fd1e18fn%40android.com.

Yogendra Singh

unread,
Oct 15, 2023, 1:28:05 PM10/15/23
to Android CameraX Discussion Group, xi...@google.com, Android CameraX Discussion Group, Yogendra Singh
Hi Xi,
Thanks for the guidance.

3. Then, in your OpenGL shader, blend the mask, the input image, and the background image to replace the background.
Can you guide for that? How i can use that class OpenGlRenderer  to blend the detected mask and background image?

Xi Zhang (张熹)

unread,
Oct 15, 2023, 2:27:35 PM10/15/23
to Yogendra Singh, Android CameraX Discussion Group
Here are some code samples:

The folder contains code of laying over an image on top of camera input. Your case is more complicated, but it should be similar.
Reply all
Reply to author
Forward
0 new messages