WebRTC --> FFmpeg --> WebRTC

[Avital Yachin]

Hi All,

We are looking at the following scenario: capture user's webcam (from browser) --> send to server --> process with ffmpeg/opencv --> return back to the user's browser

In order to capture the stream in the server by ffmpeg, we are considering the use of the video_room plugin with the rtp_forwarding option.

Now comes the question of how to return the post-processed stream back to the client's browser with minimum latency (i.e. via WebRTC). 

A possible option would be to create a "shadow user" on the server side (using node and janus.js) with its video source set to the ffmpeg/opencv output, and have it join the room. This would broadcast the ffmpeg output stream to all (2) room participants including the client side. However, there would be a "penalty" of 2 streams sent back to the client (the original remote stream, and the post-processed stream).

Another option would be to try use the streaming plugin, but we're not sure if it's possible to stream to a client that is already using another plugin.

Couple questions that will surely help us on this matter:

1. What should be the general approach for this use case?

2. Is it possible to tell the video_room plugin not to send certain participants streams back to a specific participant? 

3. Is it possible to change the remote stream of a room's participant? (other words: forward the incoming stream of a participant AND set its "outdoing" stream to a different source). This would enable to achieve our goal without adding a "shadow user"

4. Is it possible to use the streaming plugin to send video to a client that is already using the video_room plugin? 

5. Is it possible to capture a client's webcam stream on the server with ffmpeg in any way other than using the video_room w/rtp_forwarding?

Thanks in advance,

-Avital

[Lorenzo Miniero]

If you want in/out on the same PeerConnection, you'll need to write your own plugin, something that combines the RTP forwarding of the VideoRoom and the RTP restreaming of the Streaming plugin. If you want to use the stock plugins, the only approach right now is using the VideoRoom for injection, and the Streaming plugin for watching the result. Not sure what you meant by questions 2, 3 and 4 instead.

Lorenzo

[Avital Yachin]

Thanks much Lorenzo.

The approach we were considering was to use a single PeerConnection on the client, hence the idea of the adding the "shadow user" to the video_room.

If I understand you correctly, then you are suggesting to use one PeerConnection with the video_room plugin in order to "hijack" the stream on the server, and then stream it back using the streaming plugin, which will require to open another PeerConnection on the client. Is that correct?

Also, if we are going to use the streaming plugin to stream the client's video back to himself, then we'll need a way to dynamically set the video source (per user). I noticed that the video source is defined in the janus.plugin.streaming.cfg file. Is there any recommended approach to set the source of the streaming plugin dynamically and support multiple sources/users simultaneously?  

 

Thanks again,

Avital

[Lorenzo Miniero]

Il giorno mercoledì 15 novembre 2017 09:51:28 UTC+1, Avital Yachin ha scritto:

Thanks much Lorenzo.

The approach we were considering was to use a single PeerConnection on the client, hence the idea of the adding the "shadow user" to the video_room.

The shadow user wouldn't help, as you can never use a single PeerConnection to send *and* receive with the VideoRoom. All VideoRoom PCs are monodirectional: either publishers, or subscribers.

 

If I understand you correctly, then you are suggesting to use one PeerConnection with the video_room plugin in order to "hijack" the stream on the server, and then stream it back using the streaming plugin, which will require to open another PeerConnection on the client. Is that correct?

Yes, this is the easiest way to do it with the existing plugins. As I said, though, you can create a new plugin that does what you need.

 

Also, if we are going to use the streaming plugin to stream the client's video back to himself, then we'll need a way to dynamically set the video source (per user). I noticed that the video source is defined in the janus.plugin.streaming.cfg file. Is there any recommended approach to set the source of the streaming plugin dynamically and support multiple sources/users simultaneously?  

Mountpoints can be created dynamically via API, check older posts for examples.

L.

[Avital Yachin]

Just in case someone find it useful, here are some curl commands to generate RTP mountpoints with the Streaming plugin dynamically:

Needless to say that you'll need the session_id and handle_id of the specific streaming plugin instance.

List existing mountpoints:

curl -H "Content-Type: application/json" -X POST -d '{"body":{"request":"list","secret":"adminpwd"},"janus":"message","transaction":"some_random_text"}' http://myserver.com/janus/5047052667785047/558499164148112

Create new mountpoint:

curl -H "Content-Type: application/json" -X POST -d '{"body":{"request":"create","type":"rtp","id":99,"description":"stream_description","audio":true,"video":true,"audioport":1111,"audiopt":111,"audiortpmap":"opus/48000/2","videoport":2222,"videopt":100,"videortpmap":"VP8/90000","permanent":false,"secret":"adminpwd"},"janus":"message","transaction":"some_random_text"}' http://myserver.com/janus/5047052667785047/558499164148112

Note: if "id" isn't provided, a random one will be generated. Keep it as you'll need it for deleting the mountpoint.

Delete a mountpoint:

curl -H "Content-Type: application/json" -X POST -d '{"body":{"request":"destroy","id":99,"secret":"adminpwd"},"janus":"message","transaction":"some_random_text"}' http://myserver.com/janus/5047052667785047/558499164148112

References: 

https://github.com/meetecho/janus-gateway/blob/master/plugins/janus_streaming.c

https://groups.google.com/forum/#!searchin/meetecho-janus/streaming$20mountpoints|sort:date/meetecho-janus/wlk7GOdscug/tp1QNMy9AQAJ