Proper use of multi track/stream

[Yoshimasa Iwase]

There're 3 ways to send/receive multi mediastream, that is:

[1]. N peer connection

[2]. 1 peer connection / N mediastream

[3]. 1 peer connection, 1 mediastream, N mediastream track

What's the difference between 2 and 3?

Now I'm thinking that

'[2] 1 peer connection / N mediastream' is used for handling audio and video mediastream in the samestream,

and

'[3]. 1 peer connection, 1 mediastream, N mediastream track' is used for handling video and another video(like screensharing) in the same video stream.

 

Is this right?

[Alexandre GOUAILLARD]

Hi,

sorry I have not been clear last tuesday :-) My Japanese is too rusty.

THE API / MEDIA

- even though the W3C MediaStream specifications always indicated that 1 stream can have multiple tracks, for a very long time, firefox only supported a maximum of 1 video track per stream. 

- For a very long time, the consumers (<audio> object, <video> object, could only consume one track of a given type. Because of that, while it is ok by the specs to put many tracks in a stream, practically, almost no body does it, as no track but the first one of each media type would not be directly audible/viewable.

- because of this assumptions, a lot of the APIs of the original stream object were eventually moved to the track object (e.g. stop())

- nowadays, all browsers support media streams with multiple tracks.

- the MediStream is nowadays more a convenience container than anything else. It is the type of object produced by GetUserMedia, and it is the type of object consumed by peer connection, <audio> and <video> (and some more: web audio, .....).

- if you want to handle the media (mute, stop, ...) you need to manipulate tracks. For example, if you want to add an audio track to a screensharing video track, you sometimes need to make two calls to GetUserMedia (one for screensharing, one for audio), and get the tracks from one to the other, before you attach the result to peer connection.

so according to the MediStream spec, all three options you proposed are possible, but practically, not always appealing. The real problem is how to handle the corresponding signaling.

THE SIGNALING (a.k.a. JSEP, SDP and SDP O/A)

- the biggest questions for long was: if I have more than one stream in a peer connection, how do I let the remote side know?

- the first proposal was plan B which is AFAIK still in use by chrome (here)

- more recently Unified plan was proposed by Microsoft, Google and mozilla, and even more recently, it was implemented in firefox.

Hope this helps.

alex.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/cd3bd8e5-8ef0-4d91-b62b-b0f2918712a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

CTO - Temasys Communications, S'pore / Mountain View

President - CoSMo Software, Cambridge, MA

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

[Yoshimasa IWASE]

Hi Alex,

Sorry for late reply and thank you so much for you explanation. Before posting the first question based on your talk in Japan I understood 2 drafts about signalling: how to let remote side know about multi streams

This question is just for APIs, how do I choose proper API when I want to send/receive multiple streams. However because of your reply, now that I understand that every 3 way is possible when handling multiple streams/tracks.  

--

---
You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/vhKETlh4Jcw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAHgZEq5pANYU%2Bbz9ntnWuU3goS_TyL3gspV-o%2BY_4z1s%2BrbHTw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Yoshimasa IWASE

[Randell Jesup]

On 8/12/2015 11:34 PM, Yoshimasa IWASE wrote:

On Thu, Aug 6, 2015 at 10:50 AM, Alexandre GOUAILLARD <agoua...@gmail.com> wrote:

- because of this assumptions, a lot of the APIs of the original stream object were eventually moved to the track object (e.g. stop())

- nowadays, all browsers support media streams with multiple tracks.

- the MediStream is nowadays more a convenience container than anything else. It is the type of object produced by GetUserMedia, and it is the type of object consumed by peer connection, <audio> and <video> (and some more: web audio, .....).

Note: one big thing that's tied to MediaStreams currently is synchronization - the tracks in a single MediaStream are synced, which means video should be synced to audio.  (I doubt browsers fully handle the multiple audio tracks in a MediaStream and sync them to each other, but it should generally Just Work even if the sync isn't guaranteed - and with jitter buffers, it's hard to guarantee audio to audio sync).

THE SIGNALING (a.k.a. JSEP, SDP and SDP O/A)

- the biggest questions for long was: if I have more than one stream in a peer connection, how do I let the remote side know?

- the first proposal was plan B which is AFAIK still in use by chrome (here)

- more recently Unified plan was proposed by Microsoft, Google and mozilla, and even more recently, it was implemented in firefox.

More to the point, the WG agreed to do the Unified Plan quite a while ago.  It is possible (with some pain) to convert Plan B to Unified and vice-versa; see the article on the Hacks blog: https://hacks.mozilla.org/   Chrome has a bug for implementing it, but it'll be a while it appears.

-- 
Randell Jesup
randel...@jesup.org

[Yoshimasa Iwase]

Note: one big thing that's tied to MediaStreams currently is synchronization - the tracks in a single MediaStream are synced, which means video should be synced to audio.  (I doubt browsers fully handle the multiple audio tracks in a MediaStream and sync them to each other, but it should generally Just Work even if the sync isn't guaranteed - and with jitter buffers, it's hard to guarantee audio to audio sync).

You wrote about 'A single MediaStream with multiple tracks'. How about 'A single PeerConnection with multiple MediaStreams, like video x 2 and audio x 1'? Is there any issues about synchronisation?  

2015年8月15日土曜日 4時27分04秒 UTC+9 Randell Jesup:

[Alexandre GOUAILLARD]

On Sat, Aug 15, 2015 at 4:44 PM, Yoshimasa Iwase <iwase.y...@gmail.com> wrote:

Note: one big thing that's tied to MediaStreams currently is synchronization - the tracks in a single MediaStream are synced, which means video should be synced to audio.  (I doubt browsers fully handle the multiple audio tracks in a MediaStream and sync them to each other, but it should generally Just Work even if the sync isn't guaranteed - and with jitter buffers, it's hard to guarantee audio to audio sync).

You wrote about 'A single MediaStream with multiple tracks'. How about 'A single PeerConnection with multiple MediaStreams, like video x 2 and audio x 1'? Is there any issues about synchronisation?  

there is no issue: there is NO synchronization between media streams within a peerconnection. Only between tracks in a a MS. 

• 
  

[Yoshimasa IWASE]

I see, thanks! 

--

---
You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/vhKETlh4Jcw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAHgZEq5St5z-XzGZ8F42ou-WuGFWL8cs17cA2U91VhiaV0pb5w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Yoshimasa IWASE