What format is provided via the VideoSink interface, onFrame?

[Neil Young]

- Android screen capturer

- The internal looper thread calls the VideoSink "onFrame" implementation of my capturer

- I forward the frame to a given capturerObserver.onFrameCaptured

In earlier versions (lower Android SDK, WebRTC M105) I could forward the frame "as is". Now I'm using SDK 31/33 and M114. The format of the VideoFrame offered via onFrame seems to not be I420 anymore, since I have to explicitly convert it (otherwise a black image is encoded).

This seems to be required now:

public void onFrame(VideoFrame frame) {

  VideoFrame i420Frame = new VideoFrame(frame.getBuffer().toI420(),  frame.getRotation(), frame.getTimestampNs());

  this.capturerObserver.onFrameCapturered(i420Frame);

   i420Frame.release();

}

Can anybody elaborate about the format provided in onFrame?

[V I]

On Android it's either NV12 or Native (https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video/video_frame_buffer.h;l=54;drc=f5bdc89c7395ed24f1b8d196a3bdd6232d5bf771). The latter is a GPU memory handle which you can use to pass to the HW encoder, but if you need access to the pixel data for some processing and/or SW encoding, you have to transfer it to the system memory with toI420().

Did you try calling type() on the buffers you get?

--
This list falls under the WebRTC Code of Conduct - https://webrtc.org/support/code-of-conduct.
---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/2e55d337-5c24-4093-9c02-a82306a618den%40googlegroups.com.

[Neil Young]

Thanks V I. I double checked my two realisations. Both are using SDK 31, but different WebRTC versions. The one which works w/o explicit "toI420()" conversion is using M105, the other, which requires "toI420()" is using M114. In both versions the returned buffer type is 0 (native). I'm sure my M105 based solution is also supporting VP8 encoding, which is only available as software encoder on the device in question. So if there was no change in buffer content between 105 and 114, then it could probably have to do with different EglBase handling in both apps....

BTW, I'm not sure with the final "i420Frame.release()" call either. But nobody complaints and if I don't do it the system dies on memory shortage somewhen.