custom MediaStream from native client

[shty...@gmail.com]

I have been reading various online sources and code examples to try and understand if the following is possible and what the best way to go about it would be:

I would like to create a native (c/c++) webrtc peer that will capture video frames from ffmpeg desktop capture (this part I know how to do) in raw h264 NAL frames and send these NAL frames via webrtc for decode and display in the browser as a MediaStream (so that I can attach directly to a video tag for best rendering performance). Is this even remotely possible?

Looking at the peerconnection_client examples it seems that I might want to start there, but any tips or pointers in the right direction would be appreciated. One of the biggest unanswered questions I have (after looking at the example) is which class(es) or functions should I implement/overload which I need to feed the data I receive from ffmpeg so that it is properly feed to a MediaStream and received on the other side. Additionally, I am wondering if the MediaStream decode (browser side) can handle frame loss as well as support for both I-Frames and P-Frames.

I am aware of the MediaSource APIS (mediaSource.addSourceBuffer and sourcebuffer.appendBuffer) however would prefer to leverage MediaStream since I believe that will avoid any javascript code intercepting the frame data/buffers and can leverage hardware decoding of h264 if available and provide the lowest latency and lowest CPU usage experience.

[Kaiduan Xie]

This is possible, please contact me offline.

/Kaiduan

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "discuss-webrtc" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to discuss-webrt...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/discuss-webrtc/efd6a2a7-4699-464c-9813-fbe22a82e923%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[shty...@gmail.com]

Adding a reply by Kaiduan off list for reference:

You basically need to implement your own video capture to use ffmpeg
desktop capture, I assume the output is already H.264 encoded, then
you have to bypass the encoding path to deliver the H.264 encoded
frame to RTP directly. 

Please see the following code about video capture in
https://chromium.googlesource.com/external/webrtc/+/master/webrtc/modules/video_capture/
and the way to deliver encoded frame to RTP in
https://chromium.googlesource.com/external/webrtc/+/master/webrtc/modules/video_coding/codecs/vp8/vp8_impl.cc
encoded_complete_callback_->Encoded(encoded_images_[encoder_idx],
&codec_specific, NULL);

Seems like a start. If any other folks have advice on the question or the above notes I would be interested in learning more about the pipeline.

[William Cheung]

Hi, were you were successful in sending a custom video stream via c++ client? If so, did you use webrtc/video_capture and vp8_impl as described?

[Roman Shtylman]

Yes, I was successful. Actually ended up using the h264 patchset (https://codereview.webrtc.org/1306813009/#ps530001) and changing the h264_encoder_impl.

The other gotchas were to also make sure the SDP offer contained the protocol in a way the receiving browser could decode it and that the receiving browser selected this protocol and not another one.

--

---
You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/rAuuGGiAkiM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e9c1b3da-fd55-46bb-a351-150e2aad0ff6%40googlegroups.com.

[Oren Shir]

Roman, could you share more details about your implementation? Did you implement only an encoder or more?