Adding multiple video streams to the same Peer Connection object

[vishnu]

I was wondering, would it be possible to add more than one stream to a
Peer Connection object using the addStream() function and to receive
it at the other end using the onAddStream function?

I'm trying to forward all the remote streams available to a particular
peer to all other connected peers. Essentially setting up a Video
Conferencing system where one peer acts as the 'Hub'.

[Punyabrata Ray]

While this might be possible with multiple PeerConnection objects, we are verifying on our end whether this 

has been implemented end to end.

-pr

[Bryan Donnovan]

I have tested offer SDP containing multiple streams, and only the first of them triggers an onAddStream callback. After applying the offer, if you examine the pc.remoteDescription, you find that the second stream has been removed entirely.

On Tuesday, May 29, 2012 5:44:14 PM UTC-7, Punyabrata Ray wrote:

While this might be possible with multiple PeerConnection objects, we are verifying on our end whether this 

has been implemented end to end.

-pr

[vishnu]

When a local peer adds a second stream to the PeerConnection object,
his 'localStreams' array is updated but on the remote peer's end,
neither does this trigger the 'onaddstream' function, nor does the
'remoteStreams' array reflect the changes.

[Harald Alvestrand]

We have not implemented support for adding a remote stream as an outgoing stream to another PeerConnection.

There are some tricky issues with that (ie what happens if the two PeerConnections don't support the same data format), and we'd like to defer that until all the basic stuff is working.

            Harald

[vishnu]

So as of now, is there no way to send multiple streams from one peer
to the other?

On May 30, 2:59 pm, Harald Alvestrand <h...@google.com> wrote:

[Harald Alvestrand]

On Wed, May 30, 2012 at 12:26 PM, vishnu <vishnum...@gmail.com> wrote:

So as of now, is there no way to send multiple streams from one peer
to the other?

Only if you have multiple local cameras, and call GetUserMedia twice, selecting different ones.

(there was a bug in this area, which I can't find right now in the tracker. Might have been fixed.)

[vishnu]

Hmm... Somewhere down the lane, will we be able to attach and forward
incoming streams to other peers?

On May 30, 4:06 pm, Harald Alvestrand <h...@google.com> wrote:

[Bryan Donnovan]

Does the "basic stuff" include 2 separate streams on the same peer connection if there is no forwarding involved ?

[Ronghua Wu]

On Tue, May 29, 2012 at 7:23 PM, Bryan Donnovan <bryand...@gmail.com> wrote:

I have tested offer SDP containing multiple streams, and only the first of them triggers an onAddStream callback. After applying the offer, if you examine the pc.remoteDescription, you find that the second stream has been removed entirely.

Can you share more detail about this? Calling sequences, offer you used etc?

[Bryan Donnovan]

Here is a simple call sequence that illustrates my concern.  

After calling "foo" I expect to see console output for streams "s1" and "s2", however, there is output only for s1. If you then examine the state of the peer connection, you see that everything related to "s2" has been truncated from the remote description.

function foo(){ var pc = new webkitPeerConnection00(null,function(ice){}); pc.onaddstream = function(e){ console.log("added stream:" + e.stream.label); } pc.setRemoteDescription(pc.SDP_OFFER,new SessionDescription(sdp) ); }

where "sdp" is equal to:

v=0 o=- 1338394544093 1 IN IP4 192.168.4.184 s= t=0 0 m=audio 5049 RTP/SAVPF 103 a=ice-ufrag:USER a=ice-pwd:PASS c=IN IP4 192.168.4.184 a=candidate:0 1 udp 2130706432 192.168.4.184 5049 typ host generation 1 a=mid:audio a=rtcp-mux a=crypto:0 AES_CM_128_HMAC_SHA1_32 inline:0000000000000000000000000000000000000000 a=rtpmap:103 ISAC/16000 a=ssrc:1 cname:s1 a=ssrc:1 mslabel:s1 a=ssrc:1 label:s1 m=video 5049 RTP/SAVPF 100 a=ice-ufrag:USER a=ice-pwd:PASS a=ice-pwd:PASS c=IN IP4 192.168.4.184 a=candidate:0 1 udp 2130706432 192.168.4.184 5049 typ host generation 1 a=mid:video a=rtcp-mux a=crypto:0 AES_CM_128_HMAC_SHA1_80 inline:0000000000000000000000000000000000000000 a=rtpmap:100 VP8/90000 a=ssrc:2 cname:s1 a=ssrc:2 mslabel:s1 a=ssrc:2 label:s1 m=audio 1 RTP/SAVPF 103 a=ice-ufrag:USER a=ice-pwd:PASS c=IN IP4 192.168.4.184 a=candidate:0 1 udp 2130706432 192.168.4.184 5049 typ host generation 1 a=mid:audio2 a=rtcp-mux a=crypto:0 AES_CM_128_HMAC_SHA1_32 inline:0000000000000000000000000000000000000000 a=rtpmap:103 ISAC/16000 a=ssrc:3 cname:s2 a=ssrc:3 mslabel:s2 a=ssrc:3 label:s2 m=video 1 RTP/SAVPF 100 a=ice-ufrag:USER a=ice-pwd:PASS c=IN IP4 192.168.4.184 a=candidate:0 1 udp 2130706432 192.168.4.184 5049 typ host generation 1 a=mid:video2 a=rtcp-mux a=crypto:0 AES_CM_128_HMAC_SHA1_80 inline:0000000000000000000000000000000000000000 a=rtpmap:100 VP8/90000 a=ssrc:4 cname:s2 a=ssrc:4 mslabel:s2 a=ssrc:4 label:s2 

On Wednesday, May 30, 2012 8:03:17 AM UTC-7, Ronghua Wu wrote:

[Ronghua Wu]

Bryan,

Current implementation doesn't support multiple m lines with the same media type, in which case only the first one will be handled as you noticed. Can you try to put the second stream in the same m line see if that works for you?

[Bryan Donnovan]

That worked -- I now get the expected callbacks.  Thank you.  I did not realize SDP allowed this, but it is more straightforward than what I was trying.  Now, I will work on seeing if I can get the media rendering into 2 video elements.

Thanks again.

On Wednesday, May 30, 2012 11:21:19 AM UTC-7, Ronghua Wu wrote:

Bryan,

Current implementation doesn't support multiple m lines with the same media type, in which case only the first one will be handled as you noticed. Can you try to put the second stream in the same m line see if that works for you?

[Bryan Donnovan]

You just have to replicate the a=ssrc lines with different labels and SSRC ids.

v=0 o=- 1338394544093 1 IN IP4 192.168.4.184 s= t=0 0 m=audio 5049 RTP/SAVPF 103 a=ice-ufrag:USER a=ice-pwd:PASS c=IN IP4 192.168.4.184 a=candidate:0 1 udp 2130706432 192.168.4.184 5049 typ host generation 1 a=mid:audio a=rtcp-mux a=crypto:0 AES_CM_128_HMAC_SHA1_32 inline:0000000000000000000000000000000000000000 a=rtpmap:103 ISAC/16000 a=ssrc:1 cname:s1 a=ssrc:1 mslabel:s1 a=ssrc:1 label:s1

a=ssrc:3 cname:s2 a=ssrc:3 mslabel:s2 a=ssrc:3 label:s2

On Tuesday, June 12, 2012 5:15:17 AM UTC-7, Erez A. Korn wrote:

Hello Brian,

Can you please also post the updated SDP? I would like to see how the m line holds the two streams.

Thank you,

Erez

[Bryan Donnovan]

Yes, I was successful with this, however, as the peer connection is permuted with updated offer/answer/SSRCs there are random video stream freezes as seen by the receivers, as well as browser freezes.  

At one point, it seemed that I could blame this problem on packet loss, or RTCP messages not being forwarded correctly, however, those issues have been resolved, and video freezes persist.

Unfortunately, all of the sample applications create one or more peer connection(s) at startup and maintain the connection state for the life of the app.  I have not seen any samples that permute the connection by adding/removing streams or samples that deal in multiple streams on a single peer connection.  The problems that I see do not seem to appear with loopback networking, so submitting bug reports on the problems is a challenge, and the lack of sample apps means that test coverage of this use case may be weak.

In my use case, each client has a single peer connection to a server.  Each time one client begins speaking, the server assigns the client a new SSRC..  All clients then update the offer from the server to include that new SSRC in the list of media streams, and create/apply the answer.  When a client stops speaking it goes into "a=recvonly" mode (or a=inactive if nobody is speaking).  So, even if 100 clients are connected to the server, the offer SDP contains SSRC tracks only for the currently speaking clients.  

[mooniac]

Not sure how Bryan got this to work previously but it should no longer work as adding another media stream to a peer connection fails when PeerConnection::AddStream calls CanAddLocalMediaStream with the error:

LOG(LS_ERROR) << "AddStream - Currently only one audio track and one"

<< "video track is supported per PeerConnection.";

Reference issue: https://code.google.com/p/webrtc/issues/detail?id=794

Regards,
Adam

On 9/12/12 2:36 PM, Nick wrote:

Hi Brian,

What type of server are you using as your central server?  Software wise?

Thanks

Nick

--
 
 
 

[Bryan Donnovan]

I have a java server based on Netty.  They have built in WebSockets, and I rolled my own RTP. 

Due to problems I saw with multiple streams on a single connection and changing those streams over time, I've had to go with the more typical use of one stream per PC.  Multiple PCs per client add non-trivial overhead for the server, but that is an optimization for another day.

On Wednesday, September 12, 2012 2:36:54 PM UTC-7, Nick wrote:

Hi Brian,

What type of server are you using as your central server?  Software wise?

Thanks

Nick

On Tuesday, August 21, 2012 2:36:09 PM UTC-4, Bryan Donnovan wrote:

[bryand...@gmail.com]

@Adam - I use multiple Remote streams from a central server, not multiple local streams.

[Manvendra Pratap Singh]

i am trying to add multiple remote stream  on single peerconnction object .. so can any buddy help to overcome this problem ...

[Vikas]

What problem are you seeing, just FYI currently we only support relaying a remote video track. See issues 383 and 2192. 

[Sitaram Shastri]

I'm also trying to do what Bryan has outlined above. So I have many clients (Chrome 32 stable), each of which maintains only a single PeerConnection with my server. Whenever a participant joins/leaves the conference, their SSRCs are added/removed & new offers are sent by the server to all the clients, over their existing PeerConnections. Unfortunately, this works only for 2 participants.

There is a video element to display the local stream & a bunch of them for each remote stream. When there are only 2 participants, the remote stream works perfectly.

When a 3rd joins the conf, the server sends the new offers to the clients, the clients call setRemoteDescription on receiving them & the onaddstream event is also fired. However, the 3rd video element just stays black. Meanwhile, the 2nd video element (for the initial remote stream), now randomly freezes for a while & also randomly shows the 3rd participant!

Any clues on what can be going wrong?

PS: Since there is only 1 PeerConnection maintained by each client, the reoffer doesn't trigger the onnegotiationneeded event. Also, as there are no changes to the local stream in the client, so I'm not calling createAnswer or setLocalDescription again. Will this leave the signaling state in a condition to receive more (re)offers later on in the session?

[Philipp Hancke]

I've seen that working quite well with more than ten participants. You need to call createAnswer and setLocalDescription however, i.e. have a stable signalling state before adding/removing SSRCs. Also make sure that ice is connected.

--
 
---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Harald Alvestrand]

I've seen that exact symptom (in a completely different context, long before WebRTC) when the sender was sending both videos on the same SSRC, and the receiver was mixing up the packets and trying to display them in one video frame.

It's entirely possible that the PeerConnection hasn't "understood" that the new SSRCs are accepted, since you never alled setLocalDescription(answer); I recommend calling createAnswer ans setLocalDescription, and seeing if that helps.

If it still doesn't work, you may have to start looking at chrome://webrtc-internals to make sure the video is indeed coming in on different SSRCs.

On Tue, Jan 28, 2014 at 12:58 PM, Sitaram Shastri <sitaram...@gmail.com> wrote:

--

[Sitaram Shastri]

Oh yes, I'd forgotten to add that webrtc-internals shows only the original SSRC, the one received in the initial SDP. The new SSRCs just aren't shown. Any ideas when this can happen?

Also, setLocal & createAnswer haven't helped. An observation here is that the signaling state changes to stable only after setLocal is called in the createAnswer callback. But setLocal changes the IceConnection state to disconnected! Is this because it generates new candidates after this call? On the MCU side, the ICE connection remains in connected state. This also is confusing me a bit :-)

I assume if the signaling state stays in haveRemoteOffer, it won't accept any new reoffers subsequently?

[Harald Alvestrand]

On Wed, Jan 29, 2014 at 12:36 PM, Sitaram Shastri <sitaram...@gmail.com> wrote:

Oh yes, I'd forgotten to add that webrtc-internals shows only the original SSRC, the one received in the initial SDP. The new SSRCs just aren't shown. Any ideas when this can happen?

I think you'd better post the SDP from the setRemote and setLocal calls. this seems strange.

 

Also, setLocal & createAnswer haven't helped. An observation here is that the signaling state changes to stable only after setLocal is called in the createAnswer callback. But setLocal changes the IceConnection state to disconnected! Is this because it generates new candidates after this call? On the MCU side, the ICE connection remains in connected state. This also is confusing me a bit :-)

setLocal(answer) shouldn't disconnect iceConnections state unless the new offer causes an ICE restart. 

I assume if the signaling state stays in haveRemoteOffer, it won't accept any new reoffers subsequently?

Right.

[Philipp Hancke]

On Wed, Jan 29, 2014 at 2:20 PM, Harald Alvestrand <h...@google.com> wrote:

Also, setLocal & createAnswer haven't helped. An observation here is that the signaling state changes to stable only after setLocal is called in the createAnswer callback. But setLocal changes the IceConnection state to disconnected! Is this because it generates new candidates after this call? On the MCU side, the ICE connection remains in connected state. This also is confusing me a bit :-)

setLocal(answer) shouldn't disconnect iceConnections state unless the new offer causes an ICE restart. 

You currently need to wait for iceconnectionstate to change to connected though. See issue 2688.

https://github.com/ESTOS/colibri.js/blob/master/colibri.js#L735 shows one way to work around this.

 

[Sitaram Shastri]

Attaching the SDPs exchanged. The setLocal call on createAnswer doesn't trigger the onicecandidate callback, so ICE should continue to stay in connected state. Not sure what's going on. I guess all the issues must be in the SDP itself...

Hope you guys can find a bug in it :-)

[Sitaram Shastri]

SDPs attached, hope you guys can find something amiss in it :-)

[Sitaram Shastri]

Thanks, I'll check this. Meanwhile, the setLocal after createAnswer doesn't trigger the onicecandidate callback; I guess the ICE connection state should remain in the connected state?

[Harald Alvestrand]

Nothing immediately leaps to mind.... is your ICE password really the literal string "(null)"?

Are you sending real data on the bogus SSRC 123456? I see that the re-offer has three SSRCs, one of which is the same as before (123456), so the result should be two addstream events, resulting in two new streams, each with only a video track and no audio track. Is that what you're getting?

[Sitaram Shastri]

On Thursday, January 30, 2014 2:28:55 AM UTC+5:30, Harald Alvestrand wrote:

Nothing immediately leaps to mind.... is your ICE password really the literal string "(null)"?

It's a simple open TURN & STUN server with no users.

Are you sending real data on the bogus SSRC 123456? I see that the re-offer has three SSRCs, one of which is the same as before (123456), so the result should be two addstream events, resulting in two new streams, each with only a video track and no audio track. Is that what you're getting?

Yes, proper RTP data is being sent on that SSRC. Yes, 2 addstream events are fired for the 2 additional SSRCs, with only video tracks. I thought I'll see video working before bothering with audio...
 

[Sitaram Shastri]

Ok, I got this working after grappling a bit with our legacy MCU :-))

An observation tho. createAnswer after the reoffer creates an SDP without any ICE candidate lines. This was causing an ICE restart after setLocal, & that was failing because Chrome was sending STUN binding requests to itself! Any clues on what could cause this behaviour? The reoffer SDP only changed the msid-semantic line & added the new SSRCs.

Instead, I used peerConnection.localDescription.sdp for setLocal, & everything works fine...