CameraX Effect API to overlay webview in real time?

[Lukas]

I would like to capture the video feed from camera, add a webview overlay then encode the feed with overlay to mp4 in real time?
I am not an android developer, I know this is very complicated, still I want to know if it is doable.

I am thinking two ways:

1. CameraX with effect API. Here is a post: https://groups.google.com/a/android.com/g/camerax-developers/c/64eahzvdY4U here is the details:

1. First create an OpenGL render that copies the camera input to the output Surface. You can take a look at the code here as your starting point. Then you need to update the render to take in the png, use it as a texture, and blend it with the input frames. This is the most challenging part. 2. Then wrap the renderer with a SurfaceProcessor interface. You can copy-paste most of the code here. 3. Build a CameraEffect with the SurfaceProcessor, and set it with UseCaseGroup.Builder#addEffect. This one should be straightforward.

4. Use camerax's image proxy, obtain the frame, convert webview to bitmap, overlay the bitmap to the camera feed frames, and send the frames to mediacodec for encoding. I have tried to see if this can be done in real time, unfortunately the process takes longer than 33ms per frame, which means i am unable to do it in real time.

So is option 1 doable and able to get everything done in less than 33ms per frame?

thanks

[Lukas]

I think this is definitely, since prism live is doing the exactly the same.  So how did they do that?

[Xi Zhang (张熹)]

The bottleneck in your solutions is converting the webview to an image. After that, solution 1(blending with OpenGL) is more performant than solution 2, but taking a snapshot of the webview at 30fps is going to be expensive.

May I ask why do you want to blend the webview? The prism live app you mentioned seems like a streaming app. If you are implementing live streaming features, have you considered using the MediaProjection API?

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/ee7f9fb7-c1d2-4e03-95fa-233dede1e3a9n%40android.com.

[Lukas]

Thanks!  The reason is to integrate the scoreboard for sports games.  The scoreboard is dynamic web page, (keep changing, such as timer etc)

So if converting webview to image is slow, any other options?

also, I would like either to stream the merged video, or have an option to save the entire merged video to file

thanks!

[Lukas]

I just looked at MediaProjection API, per the documentation, everything on the screen is recorded, however, I will have other controls or ui elements that I don't want to be part of the saved video.  Is this possible with MediaProjection API?

[Xi Zhang (张熹)]

MediaProjection API is for recording a single app or the entire screen, without finer controls. 

If it's not too difficult, I would suggest you manually render the score board and blend with your stream. Otherwise, you can always take snapshots of the webview then blend it with the camera feed. In the case of a score board, you don't have to snapshot it at 30fps since it doesn't change that frequently. So the performance might not be so bad.

If you implement this using the CameraX effect, then you will automatically have it available via Preview and VideoCapture use case. For streaming though, you might need to manually download the merged frame with OpenGL then stream it.

[Lukas]

Thanks Xi.  MediaProjection has almost everything I need except I don't want other UI elements to be visible in the recording.  I asked AI. it says possible.  So is it possible?

[Andrew Lewis]

It looks like MediaProjection is flexible enough to allow recording some 'off-screen' views. Here is a code snippet I was able to get to work (editing Media3 Transformer here, but the exact same approach should work elsewhere):

virtualDisplay = mediaProjection.createVirtualDisplay(
  VIRTUAL_DISPLAY_NAME,
  checkNotNull(captureFormat).width,
  captureFormat.height,
  densityDpi,
  VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY | VIRTUAL_DISPLAY_FLAG_PUBLIC,
  surface,
  /* callback= */ null,
  /* handler= */ null);

Context displayContext = context.createDisplayContext(virtualDisplay.getDisplay());
Presentation presentation = new Presentation(displayContext, virtualDisplay.getDisplay());
WebView myWebView = new WebView(displayContext);
myWebView.loadUrl("https://www.google.com");
presentation.setContentView(myWebView, new ViewGroup.LayoutParams(captureFormat.width, captureFormat.height));
presentation.show();

Once you've got the content into a Surface like this you should be able to do arbitrary things with it, eg. reading from it in a GL external texture and compositing it with other content.

Hope this helps!

Andrew

[Lukas]

Thanks!  I will give it a try!

[Xi Zhang (张熹)]

It's not possible at the moment. MediaProjection records everything on the screen/app. You could remove the unwanted UI elements in post-processing, but I imagine it will be harder than rendering the score board yourself.