Video freezes and Failed to unprotect SRTP packet, err=9

[Vasiliy Ganchev]

Hi, community!

I have a problem with freezes in video calls.

Server: kamailio+rtpengine+asterisk

Call participants:

- 2 Webrtc phones (Web), built on jssip library

Firstly making audio call, than add video on both sides

RTP video flow is going:

Web <----> rtpengine (DTLS-SRTP <-> RTP)  <----> asterisk <------> rtpengine ( RTP <-> DTLS-SRTP ) <-----------> Web

Problem is that time to time video freezes (on one of the participants, or both) after 2-10s video is recovered

It freezes more often in case if the system in a cloud.

Frequency of freezes is not constant, mainly once per minute (periodicity may differ)

Before freeze:

- checking tcpdump - RTP flow A->B, B->A - there is now drops, big delays etc, nothing that could point out - what may be wrong.

When freeze happens:

- In all my tests - I see such errors in Chrome logs:

[1:24:1103/103728:WARNING:srtpfilter.cc(583)] Failed to unprotect SRTP packet, err=9

[1:24:1103/103728:ERROR:channel.cc(813)] Failed to unprotect video RTP packet: size=1195, seqnum=40871, SSRC=1682432964

[1:24:1103/103728:WARNING:srtpfilter.cc(583)] Failed to unprotect SRTP packet, err=9

[1:24:1103/103728:ERROR:channel.cc(813)] Failed to unprotect video RTP packet: size=1195, seqnum=40872, SSRC=1682432964

[1:24:1103/103728:WARNING:srtpfilter.cc(583)] Failed to unprotect SRTP packet, err=9

[1:24:1103/103728:ERROR:channel.cc(813)] Failed to unprotect video RTP packet: size=1195, seqnum=40873, SSRC=1682432964

.....

- the number of such errors may be different - but always freeze is surrounded with such errors (at least, I did not found any more hints - on what could be wrong).

While freeze:

- in tcpdump I see RTCP PLI packets sent, remote side re-send key frame and picture is recovered

 

Looking for webrtc sources - found that err=9 -  replay check failed (bad index)

 what index could be wrong? where/how to check?

 

Any suggestions how to debug deeper would be highly appreciated!

Thanks in advance!

[Sergey Grigoryev]

Hi Vasiliy,

You can look on similar problems on my side:

http://stackoverflow.com/questions/37159178/native-c-webrtc-client-on-wifi-reduces-video-quality-but-fine-on-calls-from-c/39837375#39837375
https://groups.google.com/forum/#!topic/discuss-webrtc/wmYo7AU3evI

And one more man here:
https://groups.google.com/forum/#!topic/discuss-webrtc/WwoJYmxmnhk

I'm not monitoring this group so often, so feel free to contact me grigoryevsa at g m a i l .c o m

Br,
Sergey

пятница, 4 ноября 2016 г., 14:28:28 UTC+3 пользователь Vasiliy Ganchev написал:

[Vasiliy Ganchev]

Hi, Sergey!

Thanks for your reply!

seems to me, that our cases are different.

Maybe someone else have ideas regarding the issue?

[Jeremy Noring]

Licode and other SFUs have encountered similar errors; I think it may be SRTP's replay detection kicking in and preventing NACK'd packets from making it through.  This results in freezing video and ultimately a PLI, which then causes the video to snap back in until packet loss again causes things to freeze.  

In licode there's a patch to libsrtp specifically to deal with this issue.

[Philipp Hancke]

See https://twitter.com/juberti/status/699854177392992259

The SFU received the packet. Asking the sender to resend it because it was lost between the SFU and the receiver is... a replay attack.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/0195cb8a-5046-4baa-b00e-5564796bdfbd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dennis Dowhy]

you're probably not signaling or sending or handling the correct RTX packet format.   google's first implementation was not spec compliant (https://tools.ietf.org/html/rfc4588) , on receiving a NACK, they were just re-sending the exact same packet instead of sending the right format per https://tools.ietf.org/html/rfc4588 spec.  They fixed this somewhere between M32 and M45 I dont remember the exact version.   But assuming you are on the latest chrome builds, the webrtc stack is capable of sending either format depending on the sdp signalling.    If its not signalled right, webrtc will send the old (wrong) packet format. and if you send this packet to your srtp decrypter, it will fail with the Failed to unprotect SRTP packet, err=9 error.     the "right" way is to follow the spec now, siginaling the RTX stream and ssrc number and payload type, and on the wire the RTX packets are basically their own separate stream (multiplexed) with its on SSRC value (assigned in SDP) and its own sequence number space, and your middlebox needs to be capable of receiving this, decrypting it (this should work now once without failing), and restoring the original packet (sequence number and ssrc and shifting the payload and updating the rtp packet length) before sending it on.

to signal the spec-compliant format (rfc4588), make sure sdp contains at these lines related to the video:

m=video 9 UDP/TLS/RTP/SAVPF 100 96     // these are payload types, 100 for video and 96 for the RTX

a=rtpmap:100 VP8/90000

a=rtcp-fb:100 ccm fir

a=rtcp-fb:100 nack

a=rtcp-fb:100 nack pli

a=rtpmap:96 rtx/90000  // these are to signal the rfc4588 RTX packet transmission format

a=fmtp:96 apt=100        // this ties togetehr the two saying that packets with PT 96 are for the PT 100 stream

a=ssrc-group:FID 4069523937 476665172  // this is what says that  '476665172 ' is the ssrc repair identifier for the '4069523937'  stream

a=ssrc:4069523937 cname:mKSXF68N+X61D4OT

a=ssrc:4069523937 msid:zN9qFKoJrs4jTUm39bsSdA8KAI6jL7Q0gj4T 32db7cba-bd60-4022-8a20-b83c8b93be9f

a=ssrc:4069523937 mslabel:zN9qFKoJrs4jTUm39bsSdA8KAI6jL7Q0gj4T

a=ssrc:4069523937 label:32db7cba-bd60-4022-8a20-b83c8b93be9f

a=ssrc:476665172 cname:mKSXF68N+X61D4OT

a=ssrc:476665172 msid:zN9qFKoJrs4jTUm39bsSdA8KAI6jL7Q0gj4T 32db7cba-bd60-4022-8a20-b83c8b93be9f

a=ssrc:476665172 mslabel:zN9qFKoJrs4jTUm39bsSdA8KAI6jL7Q0gj4T

a=ssrc:476665172 label:32db7cba-bd60-4022-8a20-b83c8b93be9f

if you leave this out and your signalling looks like this instead:

m=video 9 UDP/TLS/RTP/SAVPF 100 

a=rtpmap:100 VP8/90000

a=rtcp-fb:100 ccm fir

a=rtcp-fb:100 nack

a=rtcp-fb:100 nack pli

a=ssrc:4069523937 cname:mKSXF68N+X61D4OT

a=ssrc:4069523937 msid:zN9qFKoJrs4jTUm39bsSdA8KAI6jL7Q0gj4T 32db7cba-bd60-4022-8a20-b83c8b93be9f

a=ssrc:4069523937 mslabel:zN9qFKoJrs4jTUm39bsSdA8KAI6jL7Q0gj4T

a=ssrc:4069523937 label:32db7cba-bd60-4022-8a20-b83c8b93be9f

then the stack is going to send the OLD (bad) format which your SRTP decrypter is going to fail with replay attack errors.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CADxkKi%2BU9LmRVt0wErRbwmkB8rHsUCWxfB0vXUC_EU4gRD0aFg%40mail.gmail.com.

[Dennis Dowhy]

oh and if it wasn't implied from the above, your middle boxes need to support this both ways.  so if your receiver is on the latest chrome and you update your signalling to the rfc4588 way, then your middleboxes better be sending the right RTX format and not just sending the exact same byte buffer that was sent originally.

[Jeremy Noring]

Even if an SFU caches packets, the SFU will still have to NACK packets lost between publishers and the SFU, and it's the same replay concern.

publisher ----> SFU -----> subscriber

....there are two network segments on which packets can be lost, each with its own separate peer connection.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

[Jeremy Noring]

Also, my version of licode caches packets.  ;)

[Dennis Dowhy]

NACK'ing is done at the transport level, therefore SFU needs to be caching outbound packets and handling the NACKs itself, not forwarding them back up to the publisher.   If you don't cache and instead just reissue a NACK back to the publisher, by the time the retransmitted packet is received and resent back end-to-end you've probably got enough total latency between the original transmission and the retransmit that it'll just be dropped by the decoder on the subscriber side anyway.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/99b6150a-9eff-407f-9945-34a2542b4891%40googlegroups.com.

[Jeremy Noring]

No, the *renderer* may drop that frame.  The video decoder still needs it.  If a packet isn't NACK'd, the stream will ultimately freeze or suffer corruption, and the subscriber will have to send a PLI to essentially request a whole new keyframe so rendering can recover.

So for an SFU, clients pushing video receive NACKs from the SFU.  Subscribers to that publisher have everything NACKed out of the packet cache.  That's how I built it into my version of licode and it works well.

[Vasiliy Ganchev]

Hi Dennis!

This explanation looks true for my case, we do not transmit RTX, and behaviour is just like you explained

Thanks for the answer!

Will try to make changes and come back here with results!

Philipp, Jeremy thank you too, for involving to the topic!

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/0195cb8a-5046-4baa-b00e-5564796bdfbd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

[Juan Navarro]

Hi,
a new experimental branch contains a tentative fix that probably solves this issue; please use it either by installing with this apt-get repository:

For Ubuntu 16.04 Xenial:

    deb [arch=amd64] http://ubuntu.openvidu.io/fix-srtp-unprotect xenial kms6

For Ubuntu 18.04 Bionic:

    deb [arch=amd64] http://ubuntu.openvidu.io/fix-srtp-unprotect bionic kms6

or with this Docker image (https://hub.docker.com/r/kurento/kurento-media-server-exp/tags):

    kurento/kurento-media-server-exp:fix-srtp-unprotect

For more information and discussion about this bugfix, check the announcement here: https://groups.google.com/d/topic/kurento/cKzgDVrAch4/discussion

--
Juan Navarro
Kurento maintainer & developer
j1elo @ Twitter / GitHub

[Abhay Gupta]

Hi can we know the change required in libsrtp since this is a issue with Asterisk server as well .

Regards 

Abhay 

Sent from my iPhone

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/133c0a83-d853-4798-b98e-e55c36dc6a2b%40googlegroups.com.