CameraX Preview overlay and saved Video Capture Overlay

[Aaron Egger]

I am working an application that uses camera X in real time to display supplemental information on top of the preview. We are getting speed data from an external bluetooth radar device. 

During video capture I save that supplemental data to the meta data track. 
For in app view playback we access that meta data track and re-render the supplemental data on top of the video. 

We also would like to export this video to share as a fully render mp4, and product wants multiple formats of overlays on top of the video.  For example maybe the data is in the top right for facebook, but the bottom left for instagram. (Dumb example, but you get the idea) 

In app video playback is not a problem and quick because we aren't re-rendering frame by frame, but exportation is taking a very long time.

Any suggestions on how we would implement this? 
We've considered in real time creating multiple videos with the overlay in the right position, but ideally we would generate the correct video post capture so as to not eat up the user's storage. 

-aaron

[Xi Zhang (张熹)]

How do you "re-render the supplemental data on top of the video" currently?

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/3e929106-1865-48ff-b9a9-df6e8c6633b4n%40android.com.

[Aaron Egger]

Right now I'm reading the meta data track data and animating it in and out to text view on top of the video view. 

[Xi Zhang (张熹)]

I see. In that case, this is a bit out of the scope of CameraX. It's more about video editing and processing.

CameraX is planning a feature for argumenting the camera output. However it's not ready and it probably won't help in your case. I am not sure if generating multiple augmented videos on the fly is a good practice. The way I see it, there are 2 options:

1. After the video is captured, run a post-processing job then generate all the augmented videos. Like you said, the drawback is that it will eat up the disk space.

2. Store the original video on disk, Only generate the augmented video at the time of sharing. But this could introduce a major latency for sharing.

[Luis Mendieta]

Hi to all,

About the same line, i want to render video with image/text (ideal animation image) how i can do that? the only way is use the ImageAnalysis? or only with Preview i can do that? Which are the steps?

e.g. Put a title of the person that i see in the video, like a TV show.

[Xi Zhang (张熹)]

Hi Luis:

In the latest alpha version, we provide a new API called the Effects API. You can apply any effects using GPU. This requires writing GPU code like OpenGL. The drawback is that the implementation will be complicated. Or, you can use ImageAnalysis to get a CPU buffer, convert it to a Bitmap and draw on it with Canvas before writing it to the MediaCodec Surface.  This drawback is that the performance will suffer.

If you let me know what you prefer, I can provide you with more code samples.

Thanks,

Xi

[Luis Mendieta]

Thanks for the info.

Yes please could you show you one example please. Thank you 

[Mrpicchio77]

No hablo ingles ,

Alessandro Mazzotta

To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/CAMQRvAVU7-7WG0V41sXFig7TNoq8TkXERfeycOo2v5FAhdV2BA%40mail.gmail.com.

[Luis Mendieta]

Dear, my objective is to put a png with transparent and put inside letters and numbers 

[Xi Zhang (张熹)]

Our test app has code samples about the Effects API. However the complete solution will be rather complicated, unfortunately. I can provide suggestions on how to start.

1. First create an OpenGL render that copies the camera input to the output Surface. You can take a look at the code here as your starting point. Then you need to update the render to take in the png, use it as a texture, and blend it with the input frames. This is the most challenging part. 

2. Then wrap the renderer with a SurfaceProcessor interface. You can copy-paste most  of the code here.

3. Build a CameraEffect with the SurfaceProcessor, and set it with UseCaseGroup.Builder#addEffect. This one should be straightforward.

For this to work, you will have to use the latest CameraX alpha release. Please let me know if you have more questions if you decide to go down this route.

[Luis Mendieta]

Dear,

Thank for your orientation. I progress with my project. I can fix the landscape orientation and the fit screen. Now i try to to the overlay but maybe the camerax lib can´t help because i want to do the overlay and then record the video, so i understand that it is not possible do this:

this as LifecycleOwner, cameraSelector, preview, imageAnalysis, videoCapture)

I can do this in camera2 lib?

[Luis Mendieta]

Dear,

About these steps:

1. First create an OpenGL render that copies the camera input to the output Surface. You can take a look at the code here as your starting point. Then you need to update the render to take in the png, use it as a texture, and blend it with the input frames. This is the most challenging part. 

2. Then wrap the renderer with a SurfaceProcessor interface. You can copy-paste most  of the code here.

3. Build a CameraEffect with the SurfaceProcessor, and set it with UseCaseGroup.Builder#addEffect. This one should be straightforward.

I understand that this type of project is not Compose project and dont use CameraX if not Camera2, that correct?

Saludos,

Luis Mendieta

[Xi Zhang (张熹)]

Compose or not, we will need to implement the CameraEffect and a SurfaceProcessor to overlay the VideoCapture. The editing happens before the camera stream reaches the UI.

I don't think using camera2 will make it easier. You still need the OpenGL code to edit the stream with camera2. 

[Luis Mendieta]

Dear,

I progress step by step. 

Now i know do a OpenGL app (triangle moving) in Kotlin/Compose. I put the GLSurfaceView in AndroidView, like that:

AndroidView(
    factory = {
        GLSurfaceView(it).apply {
            setEGLContextClientVersion(2)
            setRenderer(MyGLRenderer())
        }
    }
)

But i don't know how i render the camera preview in this surface? My goal is that see the camera with triangle overlay.

Saludos,

Luis Mendieta

[Xi Zhang (张熹)]

Thanks for reporting back your experience. I understand that GL has a steep learning curve. :) We plan to put a code sample on GitHub that overlays bitmap on VideoCapture. Please stay tuned. ETA is unclear but I will try to get it sooner than later.

[Luis Mendieta]

Thanks for your answer. I am very grateful for the support with the sample. But just to move forward and learn, what is the development strategy? Step 1. Generate an OpenGL surface (GLSurfaceView) Step 2. Render Case use Preview in this surface? Step 3. In the GLSurfaceView put all overlay that i need (image/text) Step 4. With Case Use VideoCapture generate the output final?

Saludos,

Luis Mendieta

[Xi Zhang (张熹)]

Hi:

I forked CameraXVideo and added a sample here: https://github.com/android/camera-samples/compare/main...xizhang:camera-samples:main

In this sample, I use CameraX with Jetpack media3 to overlay a Bitmap at the center of the stream. I didn't create a pull request just yet because it has some issues. e.g. the stream freezes after a while. I am working with the media3 team to resolve this but in the meantime, hopefully this will shed some light on your project.

Xi

[Luis Mendieta]

Thank you, i review that

Saludos,

Luis Mendieta

[Luis Mendieta]

Dear Xi,

First, thank you for your support in my project. I have some question:

1. It is a high level program for me.
2. I try to start the app in Landscape, but the app colapse. I change in MainActivity in onCreate (ActivityInfo.SCREEN_ORIENTATION_PORTRAIT to ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE). I see in the res/layout/fragment_capture, there are 2 model
3. In fragments/CaptureFragment.kt/createCameraEffect, I see that in this function is the overlay and put the ic_launcher, but i dont know where do I custom the position of this image?
4. if i can put font in the overlay, do I need to work in createCameraEffect?

[Xi Zhang (张熹)]

Uploaded a new commit that should have fixed some of the issues: https://github.com/android/camera-samples/commit/4834b9a83b517c948a1f05c4472d34f413fd6901

2. I try to start the app in Landscape, but the app colapse. I change in MainActivity in onCreate (ActivityInfo.SCREEN_ORIENTATION_PORTRAIT to ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE). I see in the res/layout/fragment_capture, there are 2 model

- This is fixed by checking if the effect is created.

3. In fragments/CaptureFragment.kt/createCameraEffect, I see that in this function is the overlay and put the ic_launcher, but i dont know where do I custom the position of this image?

- You can use a matrix to transform the overlay. However I realized that there is a bug in CameraX, so that if the stream is shared between video and preview.  The transformation is wrong. I will investigate this issue but for the time being, you need to target your effect to CameraEffect.PREVIEW or CameraEffect.VIDEO_CAPTURE, but not both.

4. if i can put font in the overlay, do I need to work in createCameraEffect?

- You need to create a Bitmap with the text you want.