Video record by cameraX with segmentation feature | ML Kit

Skip to first unread message

ajay sharma

Feb 20, 2024, 6:28:35 AMFeb 20
to Android CameraX Discussion Group
I want to record a video by replacing the background with an image or color.
How can I achieve this feature?

Thanks & Regards

Xi Zhang (张熹)

Feb 20, 2024, 12:56:32 PMFeb 20
to ajay sharma, Android CameraX Discussion Group
Hi Ajay, there are a couple of ways to do this. However, there are trade-offs between the performance and the complexity. If you are looking for something simple, I would recommend the following:

1. Use ML Kit to get the segmentation mask.

Usen the MLKitAnalyzer API, and set the targetCoordinateSystem param to be COORDINATE_SYSTEM_ORIGINAL

2. Use the OverlayEffect to draw an overlay

Set up the effect to target PREVIEW | VIDEO_CAPTUREThis is for setting up a listener for frame drawing.

3. Transform the segmentation mask, and draw a different background on top of the camera frame. 

The mask you get from ML Kit is in a different coordinate system with the video stream. The transformation from camera sensor to ML Kit output is ImageProxy#getImageInfo#getSensorToBufferTransformMatrix. The transformation from camera sensor to the video frame is Frame#getSensorToBufferTransform. You need to combine those two values to get the transformation from ML Kit output to the video frame, then use it to transform the segmentation mask. After that, you can draw the mask on top of the frame using the canvas API.

You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
To view this discussion on the web visit
Reply all
Reply to author
0 new messages