PSA: Spec-compliant Simulcast using addTransceiver API

[ami...@webrtc.org]

Hi,

Following the intent-to-ship I'm happy to announce that spec-compliant Simulcast will be coming to chrome in M74 (assuming all approvals and due process).

Backwards Compatibility

If you are already achieving simulcast by using the legacy SDP munging technique, then rest assured, you do not have to make any changes and your scenario will not break. 

You are highly encouraged to change your code to use the new API surface. <meme>Munging the SDP? Just say "no!"</meme>

Summary

Users can now call addTransceiver on the peer connection and specify multiple send encodings to configure the platform to send multiple encoding layers of the same video source.

pc.addTransceiver('video', {

    sendEncodings: [ { "rid": "f" },

                                 { "rid": "h", "scaleResolutionDownBy": 2 },

                                 { "rid": "q", "scaleResolutionDownBy": 4 }]

  });

During negotiation, this will generate the appropriate SDP lines to signal simulcast:

a=rid:f send

a=rid:h send

a=rid:q send

a=simulcast:send f;h;q

The simulcast envelope (i.e. the number of layers and their RIDs) cannot be changed and is determined in the call to addTransceiver().

After the number of layers is determined, the RtpSenderInterface can be queried for the parameters using getParameters()

and the configuration for layers (not their number or RIDs) can change using setParameters().

Details

Simulcast must be negotiated by both parties. If the answer SDP does not contain the symmetric a=simulcast:recv line, then simulcast will not be sent. Instead the platform will fall back to only sending the first layer.

Layers can be rejected in the answer through omission. Omitted layers will not be sent and are removed from the sender's parameters.

A client indicating multiple send encoding layers without specifying RID values, will get RIDs that are generated by the platform.

Layers can be paused by setting the active attribute to false. This is different than layers being rejected because paused layers can be re-enabled in future calls to setParameters()

More details can be found in the simulcast section in the spec.

The Fine Print

PeerConnection does not support receiving simulcast. A server (Selective Forwarding Unit) must be used between PeerConnections.

Simulcast scenario is relevant for group calls where an SFU should be used, so no support is available for trying to negotiate simulcast between two peers without an SFU.

SSRCs will not be signaled in this scenario. 

RIDs are the identifiers that are used in simulcast and providing another source of identifiers without spec-compliant grammar to indicate how the two identifiers correlate introduces issues of conflicting ids.

SSRCs are used in all other unified plan scenarios for backwards compatibility until MID routing is more prevalent, however, since simulcast requires the SFU to signal RIDs, it is reasonable to assume that signaling RIDs indicates support for them.

An important difference between simulcast and the SDP munging technique is the negotiation aspect. The munging scenario does not consult the other party if simulcast should be sent and there is not way to reject it.

The new API requires that simulcast be 'accepted' during negotiation and multiple layers will not be sent if they are not acknowledged by the receiving party.

Looking forward to seeing this used in the wild and hearing community feedback,

Amit

[Lorenzo Miniero]

I've added support for this in an experimental branch of Janus and it seems to be working fine with M74. For people interested in the code, you can find it here:

    https://github.com/meetecho/janus-gateway/pull/1547

What I've noticed during the tests, though, is that the very first packets Chrome sends for video don't contain neither mid nor rid/repaired-rid extensions: I can tell they're a retransmission packet because of the payload type, but not which m-line or layer they refer to, and so have to drop those packets.

Hope this helps,

Lorenzo

[pablo]

The example shows how to specify the simulcast layers in the client offer.

Is it possible to specify layers in an answer assuming the SFU offer contains a=simulcast:recv?

[ami...@webrtc.org]

The answer should contain the same rid lines (or at least lines for accepted streams):

a=rid:f send

a=rid:h send

a=rid:q send

and should contain the symmetric response accepting the simulcast streams:

a=simulcast:recv f;h;q

of course, offer and answer should negotiate rid/mid/rrid extension headers as well.

if the answer does not indicate willingness to accept simulcast, simulcast will not be used (all layers except the first will be removed).

Note that it is not supported to send simulcast to a peer connection, so the SFU is mandated in this scenario and a peer connection should not receive a=simulcast:send (which indicates that the other party is sending simulcast).

[ami...@webrtc.org]

And if the SFU is creating the offer, then it should specify the streams it wants to receive:

a=rid:f send

a=rid:h send

a=rid:q send

a=simulcast:recv f;h;q

The accepting PeerConnection should respond with the symmetric

a=rid:f send

a=rid:h send

a=rid:q send

a=simulcast:send f;h;q

The application should set the parameters on the transceiver before calling SetLocalDescription().

I believe you also need to set the direction to SendReceive (or SendOnly) to enable the transceiver to send video.

Hope this is what you are looking for.

[pablo platt]

On Thu, Mar 21, 2019 at 8:00 PM <ami...@webrtc.org> wrote:

And if the SFU is creating the offer, then it should specify the streams it wants to receive:

a=rid:f send

a=rid:h send

a=rid:q send

a=simulcast:recv f;h;q

The accepting PeerConnection should respond with the symmetric

a=rid:f send

a=rid:h send

a=rid:q send

a=simulcast:send f;h;q

The application should set the parameters on the transceiver before calling SetLocalDescription().

I believe you also need to set the direction to SendReceive (or SendOnly) to enable the transceiver to send video.

How can the application set the parameters on the transceiver before calling setLocalDescription()?

If the SFU sends the first offer, the application won't have a transceiver yet. The application can add another transceiver but we'll have two transceivers instead of one.

This is what the spec says which seems very limiting:

Another implication is that the answerer cannot set the simulcast envelope directly. Upon calling the setRemoteDescription method of the RTCPeerConnection object, the simulcast envelope is configured on the RTCRtpTransceiver to contain the layers described by the specified RTCSessionDescription

 

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/834ece6b-5da3-421b-a581-909e4e2ce7a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Harald Alvestrand]

Yes, allowing the initiator to decide the number of simulcast layers is limiting.

It  is, however, not a very limiting limitation. If you have control over both ends, just send the max number of layers that is reasonable to use.

If you don't have control over both ends, staying within the restrictions proposed by the initiator seems like a good idea.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CANdLC8UyaeknGfQU9VSgi8%2Bobk%2BcD%2BopcWysv0603zdbEkbudw%40mail.gmail.com.

[ami...@webrtc.org]

1. SFU creates and sends an offer with a=simulcast:recv 1;2;3

2. PC client calls setRemoteDescription(offer)

at this point a transceiver will be created in PC for the simulcast media section with 3 layers with RIDs 1,2,3.

3. PC client calls getTransceivers(), finds the simulcast transceiver and uses getParameters() + setParameters() to configure the layers (or disable them)

4. PC client calls createAnswer() + setLocalDescription(answer)

so the envelope is only set by the initiator correct, but that isn't the real limitation. it appears that much of simulcast is left to the application layer logic (such as communicating what the actual layers are), it's not unreasonable to assume that applications can use some maximum or signal the requested number of layers off-band. the answerer can 'reduce' the envelope and reject unwanted layers.

[pablo platt]

Thanks Amit and Harald.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/b02fccc7-2e8f-4e88-a1b0-204fd3219b49%40googlegroups.com.

[pablo platt]

I'm not able to get simulcast RTP packets from Chrome 74 Beta on Ubuntu 18.04.

Audio track and video track without simulcast work fine.

The SFU sends an offer with:

a=rid:h send
a=rid:m send
a=rid:l send
a=simulcast:recv h;m;l

Chrome call pc.setRemoteDescription(offer). A transceiver is created and transceiver.getParameters() gives:

{

  encodings: [
    {rid: "h", active: true, networkPriority: "low", priority: "low"}
    {rid: "m", active: true, networkPriority: "low", priority: "low"}
    {rid: "l", active: true, networkPriority: "low", priority: "low"}

  ],

  codecs: [],

  headerExtensions: [],

  rtcp: {cname: "", reducedSize: false},

  transactionId: "b8641e8c-91a9-4840-a61a-0dc77e307275"

}

Chrome creates an answer with:

a=rid:h send
a=rid:m send
a=rid:l send
a=simulcast:send h;m;l

I'm not getting any error but the SFU doesn't receive RTP and RTCP packets from this track.

Am I missing a step?

[Lorenzo Miniero]

Pablo, you have 'send' in your SFU offer for the rid lines, it should be 'recv' I believe.

Lorenzo

[pablo platt]

I've tried both send and recv. The spec is a bit confusing [1]:

As defined in [JSEP] (section 3.7.), an offer from a user-agent will only contain a "send" description and no "recv" description on the "a=simulcast" line. Alternatives and restrictions (described in [MMUSIC-SIMULCAST]) are not supported.

 

[1] https://rawgit.com/w3c/webrtc-pc/master/webrtc.html#simulcast-functionality

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/cd281db8-aaaf-40f6-bc8a-1f9f3815451b%40googlegroups.com.

[ami...@webrtc.org]

That's not it. The direction in a=rid lines is arbitrary in webrtc, because it relates to the direction of restrictions, which are not in the webrtc spec.

Did you remember to set the direction on the transceiver to send or send/receive?

Because the offer is incoming, the transceiver is created as recv only.

[pablo platt]

Yes, I've set the transceiver.direction to 'sendonly'.

If I'm removing a=rid and a=simulcast lines Chrome sends rtp packets so my setup should be good.

Do I need to call sender.setParameters() between setRemoteDescription() and setLocalDescription() or should it send rtp packets with the default settings?

I'm only adding the track in the second negotiation, could this be the problem?

1. SFU sends offer with a=rid and a=simulcast:recv, a=recvonly lines.

2. Chrome call pc.setRemoteDescription(), pc.createAnswer() and pc.setLocalDescription()

3. Chrome call getUserMedia, let transceiver = pc.getTransceivers()[0], transceiver.sender.replaceTrack(track), transceiver.direction = 'sendonly'

4. SFU sends offer, Chrome call pc.setRemoteDescription(), pc.createAnswer() and pc.setLocalDescription()

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e1da8ea9-ae25-4f61-84a3-94b0565e9c8b%40googlegroups.com.

[ami...@webrtc.org]

Are you still negotiating it fully in the first negotiation? i.e. are you sending a=simulcast and a=rid lines in the initial answer?

Does it work if you add the track in the initial negotiation?

Make sure that SRD is the one that creates the transceiver for you. you should only add the track after SRD. check how many layers you have in getParameters()

It should send rtp packets without having to setParameters, but i guess you should call that anyway to set the layer characteristics.

[pablo platt]

On Mon, Mar 25, 2019 at 8:14 PM <ami...@webrtc.org> wrote:

Are you still negotiating it fully in the first negotiation? i.e. are you sending a=simulcast and a=rid lines in the initial answer?

Yes, I'm sending  a=simulcast and a=rid with recv or send direction in all offers and answers.

Does it work if you add the track in the initial negotiation?

I've tried it now and I'm still not getting rtp packets:

1. SFU send offer with a=simulcast, a=rid and a=recvonly lines.

2. Chrome call pc.setRemoteDescription(offer)

3. Chrome calls console.log(pc.getTransceivers()[1].sender.getParameters()). Result below.

4. Chrome call let transceiver = pc.getTransceivers()[1]; transceiver.sender.replaceTrack(localVideoTrack); transceiver.direction = 'sendonly';

5. Chrome call pc.createAnswer(); pc.setLocalDescription(answer);

pc.getTransceivers()[1].sender.getParameters() after pc.setRemoteDescription(offer) gives me:

{

  encodings: [
    {rid: "h", active: true, networkPriority: "low", priority: "low"}
    {rid: "m", active: true, networkPriority: "low", priority: "low"}
    {rid: "l", active: true, networkPriority: "low", priority: "low"}

  ],

  codecs: [],

  headerExtensions: [],

  rtcp: {cname: "", reducedSize: false},

  transactionId: "b8641e8c-91a9-4840-a61a-0dc77e307275"

}

Maybe unrelated small issue:

If the SFU sends only one a=rid line, Chrome doesn't include a=rid and a=simulcast lines in the answer. I know it doesn't really make sense to have only one rid line.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/f022a0d5-49f1-480d-904e-333d3f84d411%40googlegroups.com.

[ami...@webrtc.org]

I cannot think of something off the top of my head. i will try it on my playground.

On Monday, March 25, 2019 at 11:33:06 AM UTC-7, pablo wrote:

On Mon, Mar 25, 2019 at 8:14 PM <ami...@webrtc.org> wrote:

Are you still negotiating it fully in the first negotiation? i.e. are you sending a=simulcast and a=rid lines in the initial answer?

Yes, I'm sending  a=simulcast and a=rid with recv or send direction in all offers and answers.

Does it work if you add the track in the initial negotiation?

I've tried it now and I'm still not getting rtp packets:

1. SFU send offer with a=simulcast, a=rid and a=recvonly lines.

2. Chrome call pc.setRemoteDescription(offer)

3. Chrome calls console.log(pc.getTransceivers()[1].sender.getParameters()). Result below.

4. Chrome call let transceiver = pc.getTransceivers()[1]; transceiver.sender.replaceTrack(localVideoTrack); transceiver.direction = 'sendonly';

5. Chrome call pc.createAnswer(); pc.setLocalDescription(answer);

pc.getTransceivers()[1].sender.getParameters() after pc.setRemoteDescription(offer) gives me:

{

  encodings: [
    {rid: "h", active: true, networkPriority: "low", priority: "low"}
    {rid: "m", active: true, networkPriority: "low", priority: "low"}
    {rid: "l", active: true, networkPriority: "low", priority: "low"}

  ],

  codecs: [],

  headerExtensions: [],

  rtcp: {cname: "", reducedSize: false},

  transactionId: "b8641e8c-91a9-4840-a61a-0dc77e307275"

}

Maybe unrelated small issue:

If the SFU sends only one a=rid line, Chrome doesn't include a=rid and a=simulcast lines in the answer. I know it doesn't really make sense to have only one rid line.

The spec says that a single rid should be removed, so when there is only one, it does not trigger simulcast. 

[篠原俊一]

By following the discuttion in this thread, I also tried
server(SFU)-offer simucast signaling.
As pablo did, I confirmed no RTP packets came from Chrome.

What I noticed so far are:
- The sdp generated by createAnswer at Chrome **has** "a=rid" lines of recv, but
- the sdp obtained by PeerConnection.localDescription after
setLocalDescription does **NOT**.

The phenomena can be reproduced Chrome side only and the snippet is
at: https://gist.github.com/shino/c950790644d5dcdee80322e03ae2d6f6
The version of Chrome I tried are:
- Version 75.0.3752.0 (Official Build) canary (64-bit)
- Version 74.0.3729.40 (Official Build) beta (64-bit)

I'm not sure this has something with no RTP transmission... but I'm wondering
whether lack of a=rid lines is normal/proper thing or not.

Thanks,
Shino

2019年3月26日(火) 5:15 <ami...@webrtc.org>:

> To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/8b97c62a-5ad5-462e-9aa5-865f764cb70f%40googlegroups.com.

[ami...@webrtc.org]

Important clarification about the backwards compatibility statement in this PSA.

If you are currently using a=simulcast: in your SDP chrome behavior *will* change and possibly break your application.

If you are relying on chrome ignoring a=simulcast lines, then your application will undoubtedly break.

Chrome will no longer ignore a=simulcast: lines, but will treat them as simulcast.

As such:

If a=simulcast lines are not compliant to the spec (the format has been stable since version #04 in 2016), then chrome will:

1. Reject the SDP with error, or

2. Misinterpret the lines.

If the line is understood (or misunderstood without error) then:

1. If this is an incoming offer requesting the platform to send simulcast, then a transceiver will be created with the RIDs from the simulcast attribute

2. If this is an incoming answer, then layers may be removed for RIDs that do not appear in the simulcast attribute.

If you are currently using a=simulcast and your application relies on chrome ignoring these lines or you fear that your application may break, I ask that you please respond back (privately, or publicly, your choice) so that we can work out the details and resolve them to the best of the community.

Thanks,

Amit

[Alexander Abagian]

Sorry, what rrid extenstion do you mean ? Is it urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id ?

[ami...@webrtc.org]

Correct. rrid is the repaired-rid used for RTX packets. That is the header extension uri.

[pablo platt]

Is it possible to control or detect the number of temporal layers per rid?

Is it possible to get only one stream with 3 temporal layers?

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/b80f4b71-28d1-45ea-b86f-62d736f2df45%40googlegroups.com.

[Harald Alvestrand]

RIDs only work with simulcast (independent encodings).

You can get varying framerates from the simulcast streams, but I don't think it's correct to call these "temporal layers".

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CANdLC8Xa9drdoKMYiWtZs44%3D32ESVqxGmD_aZ0g1FNnp87BicA%40mail.gmail.com.

[Alexander Abagian]

And the RTX packets are really required for simulcast ?

[ami...@webrtc.org]

No, RTX is not required for simulcast. you can disable RTX by not signaling the codec for it or by indicating that you do not support rrid (not signaling the extension header).

The platform will not attempt to send them in these cases.

[pablo platt]

Did you reproduce the issue where the sfu is sending an offer with simulcast and Chrome doesn't send media packets?

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/eacadd9e-6409-4873-bbbf-35feb7bc441f%40googlegroups.com.

[ami...@webrtc.org]

I think i have something that reproduces it, we need to run more tests to make sure and find root cause.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.

[ami...@webrtc.org]

Track this bug through:

http://bugs.webrtc.org/10551

Currently planning on fixing this in M75.