Video record by cameraX with segmentation feature | ML Kit

[ajay sharma]

Hi,

I want to record a video by replacing the background with an image or color.
How can I achieve this feature?

Thanks & Regards
Ajay 

[Xi Zhang (张熹)]

Hi Ajay, there are a couple of ways to do this. However, there are trade-offs between the performance and the complexity. If you are looking for something simple, I would recommend the following:

1. Use ML Kit to get the segmentation mask.

Usen the MLKitAnalyzer API, and set the targetCoordinateSystem param to be COORDINATE_SYSTEM_ORIGINAL. 

2. Use the OverlayEffect to draw an overlay

Set up the effect to target PREVIEW | VIDEO_CAPTURE. This is for setting up a listener for frame drawing.

3. Transform the segmentation mask, and draw a different background on top of the camera frame. 

The mask you get from ML Kit is in a different coordinate system with the video stream. The transformation from camera sensor to ML Kit output is ImageProxy#getImageInfo#getSensorToBufferTransformMatrix. The transformation from camera sensor to the video frame is Frame#getSensorToBufferTransform. You need to combine those two values to get the transformation from ML Kit output to the video frame, then use it to transform the segmentation mask. After that, you can draw the mask on top of the frame using the canvas API.

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/bdf36484-9888-4171-b693-4524c790bbf9n%40android.com.