Custom video source for WebRTC on Android

1,922 views
Skip to first unread message

Jernej Jerin

unread,
Apr 20, 2020, 6:33:40 AM4/20/20
to discuss-webrtc
Hello!

I've already posted this question on Stackoverflow hoping to get some answers. I even put out a bounty for +100 points reputation, but until now no luck. I'm reposting that question from SO here, hopefully to get some additional exposure directly from official WebRTC community. Feel free to also answer question on StackOverflow to receive bounty.

Overview
I would like to use a custom video source to live stream video via WebRTC Android implementation. If I understand correctly, existing implementation only supports front and back facing cameras on Android phones. The following classes are relevant in this scenario:

Currently for using front facing camera on Android phone, I'm doing the following steps:
CameraEnumerator enumerator = new Camera1Enumerator(false);
VideoCapturer videoCapturer = enumerator.createCapturer(deviceName, null);
VideoSource videoSource = peerConnectionFactory.createVideoSource(false);

videoCapturer.initialize(surfaceTextureHelper, this.getApplicationContext(), videoSource.getCapturerObserver());
VideoTrack localVideoTrack = peerConnectionFactory.createVideoTrack(VideoTrackID, videoSource);

My scenario

I've a callback handler that receives video buffer in byte array from a custom video source:

public void onReceive(byte[] videoBuffer, int size) {}


How would I be able to send this byte array buffer via WebRTC? I'm not sure about the solution, but I think I would have to implement custom VideoCapturer?

Existing questions

This question might be relevant, though I'm not using libjingle library, only native WebRTC Android package.

Similar questions/articles:

Byoungchan Lee

unread,
Apr 21, 2020, 9:03:24 PM4/21/20
to discuss-webrtc
How would I be able to send this byte array buffer via WebRTC? I'm not sure about the solution, but I think I would have to implement custom VideoCapturer?

Yes, you will need to implement VideoCapturer and create VideoFrame using your []byte buffer.

Jernej Jerin

unread,
Apr 22, 2020, 7:55:25 AM4/22/20
to discuss-webrtc


On Wednesday, April 22, 2020 at 3:03:24 AM UTC+2, Byoungchan Lee wrote:
How would I be able to send this byte array buffer via WebRTC? I'm not sure about the solution, but I think I would have to implement custom VideoCapturer?

Yes, you will need to implement VideoCapturer and create VideoFrame using your []byte buffer.

I thought so, do you maybe have some code examples on how to do that? 

Byoungchan Lee

unread,
Apr 22, 2020, 7:12:29 PM4/22/20
to discuss-webrtc
FileVideoCapturer is one of the concise implementations of VideoCapturer.

Jernej Jerin

unread,
Apr 23, 2020, 2:41:35 PM4/23/20
to discuss-webrtc
Great, totally missed this implementation. This helps a lot, will report if I managed to implement it.

Jernej Jerin

unread,
Apr 27, 2020, 8:44:23 AM4/27/20
to discuss-webrtc
Hello!

I managed to construct VideoFrame and send it out:

public void onYuvDataReceived(MediaFormat mediaFormat, ByteBuffer yuvFrame, int dataSize, int width, int height) {
    long timestampNS = TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime());
    Log.d("DJIFRAME", "Got new Yuv frame " + yuvFrame + ", dataSize:  " + dataSize + ", width: " + width + ", height: " + height + " at time: " + timestampNS);
    NV21Buffer buffer = new NV21Buffer(yuvFrame.array(), width, height, null);
    Log.d("DJIFRAME", "Start constructing video frame");
    VideoFrame videoFrame = new VideoFrame(buffer, 0, timestampNS);
    videoSource.getCapturerObserver().onFrameCaptured(videoFrame);
    videoFrame.release();
}

In the end I didn't use custom VideoCapturer as I'm just using onFrameCaptured to register receiving frame on VideoSource. I'm receiving frames in yuvFrame, but when I construct a NV21Buffer from received yuvFrame byte array, I can see in the logs the following:

2020-04-27 14:27:13.466 29437-29890/com.amazonaws.kinesisvideo.demoapp D/DJIFRAME: Got new Yuv frame java.nio.DirectByteBuffer[pos=0 lim=3110400 cap=3110400], dataSize:  3110400, width: 1920, height: 1080 at time: 827459643000000
2020-04-27 14:27:13.466 29437-29890/com.amazonaws.kinesisvideo.demoapp D/DJIDecodeServer: releaseDecoder() start
2020-04-27 14:27:13.488 29437-29890/com.amazonaws.kinesisvideo.demoapp D/DJIDecodeServer: releaseDecoder() end
2020-04-27 14:27:13.607 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodec: allocate(c2.qti.avc.decoder)
2020-04-27 14:27:13.607 29437-30146/com.amazonaws.kinesisvideo.demoapp I/Codec2Client: Creating a Codec2 client to service "default"
2020-04-27 14:27:13.611 29437-30146/com.amazonaws.kinesisvideo.demoapp I/Codec2Client: Client to Codec2 service "default" created
2020-04-27 14:27:13.612 29437-30146/com.amazonaws.kinesisvideo.demoapp I/CCodec: setting up 'default' as default (vendor) store
2020-04-27 14:27:13.615 29437-30146/com.amazonaws.kinesisvideo.demoapp I/CCodec: Created component [c2.qti.avc.decoder]
2020-04-27 14:27:13.615 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecConfig: read media type: video/avc
2020-04-27 14:27:13.616 29437-30146/com.amazonaws.kinesisvideo.demoapp D/ReflectedParamUpdater: extent() != 1 for single value type: output.buffers.pool-ids.values
2020-04-27 14:27:13.621 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecConfig: ignoring local param raw.size (0xd2001800) as it is already supported
2020-04-27 14:27:13.621 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecConfig: ignoring local param raw.color (0xd2001809) as it is already supported
2020-04-27 14:27:13.621 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecConfig: ignoring local param raw.hdr-static-info (0xd200180a) as it is already supported
2020-04-27 14:27:13.622 29437-30146/com.amazonaws.kinesisvideo.demoapp I/CCodecConfig: query failed after returning 17 values (BAD_INDEX)
2020-04-27 14:27:13.623 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecConfig: c2 config diff is Dict {
      c2::i32 algo.priority.value = -1
      c2::float algo.rate.value = 4.2039e-44
      c2::u32 algo.secure-mode.value = 0
      c2::float coded.frame-rate.value = 30
      c2::u32 coded.pl.level = 20480
      c2::u32 coded.pl.profile = 20480
      c2::u32 coded.vui.color.matrix = 0
      c2::u32 coded.vui.color.primaries = 0
      c2::u32 coded.vui.color.range = 0
      c2::u32 coded.vui.color.transfer = 0
      c2::u32 default.color.matrix = 0
      c2::u32 default.color.primaries = 3
      c2::u32 default.color.range = 2
      c2::u32 default.color.transfer = 0
      c2::u32 input.buffers.max-size.value = 13271040
      c2::u32 input.delay.value = 4
      string input.media-type.value = "video/avc"
      c2::u32 output.delay.value = 18
      string output.media-type.value = "video/raw"
      c2::u32 raw.color.matrix = 0
      c2::u32 raw.color.primaries = 0
      c2::u32 raw.color.range = 0
      c2::u32 raw.color.transfer = 0
      c2::float raw.hdr-static-info.mastering.blue.x = 1.4013e-45
      c2::float raw.hdr-static-info.mastering.blue.y = 1.4013e-45
      c2::float raw.hdr-
2020-04-27 14:27:13.623 29437-30146/com.amazonaws.kinesisvideo.demoapp W/ColorUtils: expected specified color aspects (0:0:0:0)
2020-04-27 14:27:13.625 29437-29890/com.amazonaws.kinesisvideo.demoapp E/VideoDecoder: initVideoDecoder create
2020-04-27 14:27:13.626 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecConfig: no c2 equivalents for color-format
2020-04-27 14:27:13.626 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecConfig: no c2 equivalents for flags
2020-04-27 14:27:13.628 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecConfig: c2 config diff is   c2::u32 raw.pixel-format.value = 842094169
      c2::u32 raw.size.height = 1088
2020-04-27 14:27:13.628 29437-30146/com.amazonaws.kinesisvideo.demoapp W/ColorUtils: expected specified color aspects (0:0:0:0)
2020-04-27 14:27:13.629 29437-30146/com.amazonaws.kinesisvideo.demoapp W/Codec2Client: query -- param skipped: index = 1107298332.
2020-04-27 14:27:13.629 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodec: setup formats input: AMessage(what = 0x00000000) = {
      int32_t feature-secure-playback = 0
      int32_t frame-rate = 30
      int32_t height = 1088
      int32_t level = 1
      int32_t max-input-size = 3317760
      string mime = "video/avc"
      int32_t priority = 1
      int32_t profile = 1
      int32_t width = 1920
      Rect crop(0, 0, 1919, 1087)
    } and output: AMessage(what = 0x00000000) = {
      int32_t android._video-scaling = 1
      Rect crop(0, 0, 1919, 1087)
      int32_t color-standard = 0
      int32_t color-range = 0
      int32_t color-transfer = 0
      int32_t android._dataspace = 0
      int32_t width = 1920
      int32_t feature-secure-playback = 0
      int32_t frame-rate = 30
      int32_t height = 1088
      int32_t max-height = 1080
      int32_t max-width = 1920
      string mime = "video/raw"
      int32_t priority = 1
      int32_t rotation-degrees = 0
      Buffer hdr-static-info = {
        00000000:  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
        00000010:  00 00 00 00 00 00 00 00  00                       .........
      }
      int32_t android._color-format =
2020-04-27 14:27:13.630 29437-29890/com.amazonaws.kinesisvideo.demoapp E/VideoDecoder: initVideoDecoder configure
2020-04-27 14:27:13.630 29437-30145/com.amazonaws.kinesisvideo.demoapp I/MediaCodec: MediaCodec will operate in async mode
2020-04-27 14:27:13.652 29437-30146/com.amazonaws.kinesisvideo.demoapp W/Codec2Client: query -- param skipped: index = 1342179345.
2020-04-27 14:27:13.653 29437-30146/com.amazonaws.kinesisvideo.demoapp W/Codec2Client: query -- param skipped: index = 2415921170.
2020-04-27 14:27:13.653 29437-30146/com.amazonaws.kinesisvideo.demoapp W/Codec2Client: query -- param skipped: index = 1610614798.
2020-04-27 14:27:13.653 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecBufferChannel: [c2.qti.avc.decoder#355] Query input allocators returned 0 params => BAD_INDEX (6)
2020-04-27 14:27:13.654 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecBufferChannel: [c2.qti.avc.decoder#355] Created input block pool with allocatorID 16 => poolID 42 - OK (0)
2020-04-27 14:27:13.655 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecBufferChannel: [c2.qti.avc.decoder#355] Query output allocators returned 0 params => BAD_INDEX (6)
2020-04-27 14:27:13.655 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecBufferChannel: [c2.qti.avc.decoder#355] Configured output block pool ids 1 => OK
2020-04-27 14:27:13.715 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecConfig: c2 config diff is   c2::u32 raw.color.matrix = 1
      c2::u32 raw.color.primaries = 1
      c2::u32 raw.color.range = 2
      c2::u32 raw.color.transfer = 3
      c2::u32 raw.crop.height = 1080
      c2::u32 raw.crop.left = 0
      c2::u32 raw.crop.top = 0
      c2::u32 raw.crop.width = 1920
      c2::u32 raw.size.height = 1080
2020-04-27 14:27:13.715 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecBufferChannel: [c2.qti.avc.decoder#355] onWorkDone: output format changed to AMessage(what = 0x00000000) = {
      int32_t android._video-scaling = 1
      Rect crop(0, 0, 1919, 1079)
      int32_t color-standard = 1
      int32_t color-range = 2
      int32_t color-transfer = 3
      int32_t android._dataspace = 260
      int32_t width = 1920
      int32_t feature-secure-playback = 0
      int32_t frame-rate = 30
      int32_t height = 1080
      int32_t max-height = 1080
      int32_t max-width = 1920
      string mime = "video/raw"
      int32_t priority = 1
      int32_t rotation-degrees = 0
      Buffer hdr-static-info = {
        00000000:  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
        00000010:  00 00 00 00 00 00 00 00  00                       .........
      }
      int32_t android._color-format = 2135033992
      int32_t color-format = 19
    }
2020-04-27 14:27:13.743 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecBuffers: [c2.qti.avc.decoder#355:2D-BB-Output] updating image-data
2020-04-27 14:27:13.743 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecBuffers: [c2.qti.avc.decoder#355:2D-BB-Output] updating stride = 1920
2020-04-27 14:27:13.743 29437-30146/com.amazonaws.kinesisvideo.demoapp D/CCodecBuffers: [c2.qti.avc.decoder#355:2D-BB-Output] updating vstride = 1080
2020-04-27 14:27:13.746 29437-29890/com.amazonaws.kinesisvideo.demoapp E/DJIDecodeServer: dequeueOutputBuffer INFO_OUTPUT_FORMAT_CHANGED

There is never a log for starting constructing a frame, so I reckon this instruction to construct NV21Buffer must be failing? Output is pretty cryptic what's actually wrong.

Jernej Jerin

unread,
Apr 27, 2020, 1:59:41 PM4/27/20
to discuss-webrtc
I figured it out, above issue wasn't connected to WebRTC. It now works as expected!

rajus...@gmail.com

unread,
Oct 1, 2020, 5:58:32 AM10/1/20
to discuss-webrtc
Hello,

Can you please the reference implementation ..
And for what file type you implemented ?

Thanks,
Nagaraju Samudrala

Jernej Jerin

unread,
Oct 7, 2020, 3:52:42 AM10/7/20
to discuss-webrtc
Hello!

Please check out my answer on the SO: https://stackoverflow.com/a/61465517/535661

Amit Ramdath

unread,
Jun 27, 2021, 5:43:20 AM6/27/21
to discuss-webrtc
This is super helpful but how did you setup the VideoSource without a VideoCapturer?
Reply all
Reply to author
Forward
0 new messages