Multi-participants and RTP forwarding

[Avital Yachin]

Hi All,

Our existing flow consists of a single participant:

Browser/WebRTC --> Janus videoroom --> RTP Forwarding --> ffmpeg --> RTMP Endpoint

We are looking to add multi-participants capability, i.e. enabling the "Host" to add/remove "Guests".

There are few requirements:

1. The output RTMP stream needs to be "merged", i.e. include a single video stream with all participants.

2. The participants needs to see each other at low-latency (i.e. using WebRTC).

3. The Host remains active through the entire session, but Guests can come and go.

Our thought for a possible architecture/flow is this:

- The Host receives all Guests streams and merge them in the browser (including his own camera track) using HTML5 Canvas to a single merged stream.

- Guests are sending their own camera track and show the host's merged stream

- RTP forwarding (to ffmpeg) is done only to the merge stream

Happy to hear any thoughts on the above flow or suggestions.

Thanks!

[Lorenzo Miniero]

We personally don't do any merging in HTML5. We created our own videomixer (based on libavcodec/libavformat) that can receive separate RTP-forwarded streams of different participant, and performs the mixing itself before encoding and sending to a CDN via RTMP. Unlike Janus, this component is not open source, though. The same functionality can very likely be achieved using open source tools alone, e.g., by building a similar tool around ffmpeg or gstreamer yourself.

L.

[Avital Yachin]

Thanks Lorenzo.

Our main concern with such approach of server-side merging is layout flexibility, i.e. ability to switch between single and multiple participants, screenshare, PiP, side-by-side, handling of different resolutions etc. 

We have an experience with merging graphic overlays with the video stream at server side (using ffmpeg), and the flexibility is quite limited.

Did you encounter similar issues? Or, are you using a fixed layout for the merged stream?

Thanks much!

-Avital

[Lorenzo Miniero]

We implemented our videomixer with dynamic layouts and the ability to add overlays, so we don't feel like we're impacted by such a limitation. That said, using a canvas element would definitely give you more flexibility: I think it's what companies like Streamyard & co do too.

L.