Different behavior between 13.4 and 14.1 - ICE candidates

[Gustavo Almeida]

After the attempt to bump the RTPEngine version to 14.1.1.13 (from 13.4.1.7), i saw a different between the two versions, in the RTPEngine, during the media allocation with WebRTC Client.

The scenario is when VPN is used by the Agent.

STUN Server returns more than one ICE Candidate, most priority one points to the real IP and the second one to the VPN one.

In the link provided, i have RTPEngine logs showing the difference between both versions.

From what i see, DTLS handshake is started soon as possible in the 13.4 version, and the media allocation seems to happens only after it.

On 14.1 version, we can see that media path logic happens before/between DTLS handshake.

In both versions i always use endpoint-learning = heuristic.

I have done some tests, building 14.1.1.13, with some reverts, like these two: 8317cc7 and 6ce23d, but the symptom persists.

MT#63317 seems to introduce a lot of changes, and multiple commits there could be related to the divergent behavior i got.

Do you have any clue regarding this topic, and there is a way to bump to >= 14.x, and avoid this behavior. Yes, updating endpoint-learning to delayed solves the problem, but could lead to problem that are hard to knew a-priori, and dependent on clients environment.

Link with Diff --> https://www.diffchecker.com/WjyKHMxW/ 

- 2.83.67.147 (real Public IP)

- 104.28.166.5 (VPN Public IP)

- 10.40.148.178 (RTPEngine)

Thank you

[Gustavo Almeida]

Some more interesting finds.

The problem also happens in 13.5. I have tested with 13.5.1.17, and the problem still happens.

So, same logic added between 13.4 and 13.5 that introduces this divergent behavior.

Looking into the log differences, the kernelizing logs only appear in 13.4 after the DTLS handshake, and in 13.5 we already have the same behaviour we saw in 14 release.

Logs --> https://www.diffchecker.com/EUp94lrD/

[Richard Fuchs]

I have trouble understanding what exactly it is that you're asking.

There are differences of course, but what exactly is the issue?

--
You received this message because you are subscribed to the Google Groups "Sipwise rtpengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rtpengine+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rtpengine/8a361b17-0d0c-4b1a-a234-1178b9932b2bn%40googlegroups.com.

[Gustavo Almeida]

The problem is in a WebRTC scenario. Where a client is using a VPN, and STUN Server returns 2 ice candidates, one (the most priority one) with its real IP, and the other one with VPN exposed IP.

What happens is: >= 13.5, RTPEngine chooses the wrong candidate (VPN one).

So, RTPEngine sends audio to VPN IP, and relays to the other peer (no audio).

This is really strange because the audio always reach RTPEngine via the real IP (2.x).

Some interesting points i have observed:

- This not happens always, happens testing with Google STUN Server, if i test with a self-hosted Coturn, does not happen anymore.

- This problem happens if STUN bind response (map-addr) from VPN IP (104.x) resolves before the DTLS handshake completes. If does not complete before DTLS handshake, the problem does not happen. This is the behavior i saw switching between Google STUN Server and Coturn;

Based on this, seems to be a timing/delay thing. But, in 13.4 this never happens, so any logic in code starts triggering this after 13.5 (happens, for 13.5, 14.1 and 26.0).

Same RTPEngine conf and Kamailio flags are used in both scenarios, the only change is from RTPEngine version.

The only important point in the rtpengine conf is endpoint-learning=heuristic.

The flags from kamailio:

", "ICE": "remove", "label": "offer", "direction": [ "pub", "priv" ], "flags": [ "trust-address", "strict-source" ], "replace": [ "origin" ], "transport-protocol": "RTP/AVP", "rtcp-mux": [ "demux" ], "call-id": "n95t64bgod09hts8oa22", "received-from": [ "IP4", "104.28.181.215" ], "from-tag": "b77uo32mea", "command": "offer" }

 ", "ICE": "force", "label": "answer", "direction": [ "priv", "pub" ], "flags": [ "trust-address", "strict-source" ], "replace": [ "origin" ], "transport-protocol": "UDP/TLS/RTP/SAVPF", "rtcp-mux": [ "offer" ], "SDES": [ "off" ], "call-id": "n95t64bgod09hts8oa22", "received-from": [ "IP4", "10.34.70.118" ], "from-tag": "b77uo32mea", "to-tag": "rKc6XcDB4XH4c", "command": "answer" }

INVITE SDP --> 

v=0
o=- 3025983496308153705 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0
a=extmap-allow-mixed
a=msid-semantic: WMS 7bc57e45-4954-495c-960f-ca2e4ea7bd5d
m=audio 49180 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126
c=IN IP4 2.83.67.147
a=rtcp:9 IN IP4 0.0.0.0
a=candidate:120520624 1 udp 1685987071 2.83.67.147 49180 typ srflx raddr 192.168.1.81 rport 49180 generation 0 network-id 1 network-cost 10
a=candidate:1819658633 1 udp 1685855999 104.28.181.215 28117 typ srflx raddr 172.16.0.2 rport 53863 generation 0 network-id 3 network-cost 50
a=ice-ufrag:LrjH
a=ice-pwd:5EziLE9744AcDR42YX1zV4Xs
a=ice-options:trickle
a=fingerprint:sha-256 D8:48:87:A5:F2:A7:B2:4C:28:F5:3D:49:0C:A1:AC:9F:0F:1A:B6:1E:84:AA:1F:8E:2A:3D:3C:08:12:F4:89:E5
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:7bc57e45-4954-495c-960f-ca2e4ea7bd5d 026487d4-c6b2-471f-b1f3-4e77810dbb79
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
a=ssrc:1382527953 cname:NlxN7bgjqFQ3Yclh
a=ssrc:1382527953 msid:7bc57e45-4954-495c-960f-ca2e4ea7bd5d 026487d4-c6b2-471f-b1f3-4e77810dbb79

Thank you

[Gustavo Almeida]

Diff between 13.4 and 13.5, if helps.

https://www.diffchecker.com/EUp94lrD/

[Richard Fuchs]

The logs would be a lot more helpful with debugging enabled.

ICE negotiation is explicit and the controlling side decides which candidate pair to use. You can use Wireshark to capture the STUN packets and inspect the details of the negotiation process.

To view this discussion visit https://groups.google.com/d/msgid/rtpengine/4ef620d8-2eed-404f-9297-49fa12ea69fdn%40googlegroups.com.

[Gustavo Almeida]

Ok, thank you.

I will give you the logs from 13.4, after a while.

This is the debug logs from 13.5 using Google Stun Server (left side) and Coturn (right side). --> https://www.diffchecker.com/pRoXetrn/

I can't give you a capture, but this is a screenshot from STUN/DTLS --> 

[Gustavo Almeida]

Ok, repeating the scenario again for 13.4, STUN bind response from VPN IP also resolves during the DTLS hanshake, so its not he issue.

RTPEngine sends the Client key exchange to VPN IP, and in the failure case sends to Real Public IP. Its the only change i can see.

This is the debug log between 13.4 (left, working fine) and 13.5(right, bad case), both using Google Stun Server.

Thank you

[Gustavo Almeida]

Ok, i already saw which is the issue.

A commit was introduced in 13.5 that causes this divergence, after 13.4.

In the next week i will do a PR to enable/disable that change via a new parameter.

Already test it, and solves it.

Thank you again.