Debugging ICE-TCP issues

[Lorenzo Miniero]

Hi there,

I'm trying to get our gateway (which uses libnice >= 0.1.8) to work with the ICE-TCP support in Chrome, but I'm getting some problems. The ICE setup works fine, and both the browser and the gateway report a successful completion of ICE. Anyway, the DTLS Hello messages sent by the gateway are ignored by Chrome, or at least it never answers, and so the handshake never completes. It doesn't look like a transport issues as I can see, using Wireshark, connectivity checks still being exchanged between the two.

Is there any way I can debug this issue using Chrome's logging? I tried the usual --enable-logging --v=1 --vmodule=*source*/talk/*=3 stuff and I got no useful info out of that. Just let me know if you need me to provide any additional information.

Thanks,

Lorenzo

[Vikas]

What version of Chrome you testing with and on what platform? On linux you can try:  google-chrome --enable-logging=stderr --vmodule=*/libjingle/*=3,*=0 to see if you get more logging.

/Vikas

[Lorenzo Miniero]

I'm on Linux, yes. Thanks, I'll try that.

L.

[Nazmus Shakeeb]

I have never seen Chrome allocating TCP relay port in the turnserver and Justin already informed that it will not be supported.

https://groups.google.com/forum/#!searchin/discuss-webrtc/turn$20tcp$20/discuss-webrtc/t5I7_umnoRk/d8TbYjPojowJ

So Chrome doesn't support ICE-TCP using turnserver. Please don't confuse with UDP allocation using TCP.  This is not related to ICE TCP. I think this may help you.

[Philipp Hancke]

Chrome supports connecting to a server that does ICE-TCP (and even SSLTCP) because Hangouts uses it (instead of TURN).

ICE-TCP is a required feature by https://tools.ietf.org/html/draft-ietf-rtcweb-transports-07

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lorenzo Miniero]

Yes that's what I assumed. I'm indeed trying to get TCP candidates to work with my gateway, not TURN over TCP.

That said, the debug line Vikas suggested didn't provide additional info, unfortunately.

Lorenzo

[Jeremy Noring]

libnice has a debug mode you can enable to get much more verbose logging (very, very chatty); have you done that?

[Justin Uberti]

Can you get a capture of just the ICE and DTLS traffic?

--

[Lorenzo Miniero]

Jeremy,

yes, I know, that's also how I could verify that the only traffic happening are connectivity checks, and that the Hello sent by my gateway is indeed sent (no send error). All of this was verified with Wireshark after that. That's why I wanted to check what was happening on the Chrome side.

L.

[Lorenzo Miniero]

Justin,

sure, I'll do a capture tomorrow and make it available.

Lorenzo

[Justin Uberti]

One thing I thought of - are you framing the DTLS packets in RFC4571, as required by ICE-TCP?

[Lorenzo Miniero]

Yes, that's handled automatically by the libnice stack when using TCP.

You can find a pcap dump, together with a couple of additional documents that provide some more info, here:

http://janus.conf.meetecho.com/ice-tcp/

janus-handle-info.txt is a dump of the info we collect in Janus for a handle (an abstraction of a PeerConnection attached to a feature, basically), so there you'll find both the SDPs, the local and remote candidates, and the pair that has been selected from Janus/libnice's perspective. The info also shows how the DTLS handshake was started but didn't complete, with some handshake data being sent but none received.

chrome-internals.txt is a snippet from webrtc-internals that shows how the selected pair and the transport were the same Janus came to for Chrome as well. I guess the important bit there is the "googReadable" still to false: not sure whether "googActiveConnection" just refers to the established connectivity or not, or whether "bytesReceived" just refers to audio/video/data bytes or handshake data as well.

Finally, janus-chrome-icetcp.pcap.pcapng is the actual dump of the messages being exchanged over the selected pair. You'll see how Chrome successfully connects to Janus, and sends connectivity checks that Janus (libnice) answers to. Intertwined with these exchanges, there are some DTLS Hello messages Janus is sending (219 bytes, which become 221 with the 2 bytes of framing header) and retransmitting as it's getting no response. Eventually, after about a minute I closed the PeerConnection and the TCP connection was shut down.

Not sure where the issue is: thinking about it, I only see libnice answering to connectivity checks over the TCP connection, and not generating any. DTLS is originated immediately after the first successful connectivity check. May it be that libnice assumes an ICE-TCP "channel" is ready as soon as the underlying TCP connection is established and a CC received, while Chrome expects to receive CCs as well which is never happening?

I hope this is enough data for helping you debug this. If not, let me know and I'll make sure I make more available. Anyway, the issue is easy enough to replicate, if you think it might be helpful: it's enough to install Janus somewhere (even locally on a laptop), disable UDP (e.g., via iptables) and have a browser connect to one of the demos (e.g., echotest).

Thanks!

Lorenzo

[Justin Uberti]

On Thursday, February 5, 2015 at 2:12:07 AM UTC-8, Lorenzo Miniero wrote:

Yes, that's handled automatically by the libnice stack when using TCP.

You can find a pcap dump, together with a couple of additional documents that provide some more info, here:

http://janus.conf.meetecho.com/ice-tcp/

janus-handle-info.txt is a dump of the info we collect in Janus for a handle (an abstraction of a PeerConnection attached to a feature, basically), so there you'll find both the SDPs, the local and remote candidates, and the pair that has been selected from Janus/libnice's perspective. The info also shows how the DTLS handshake was started but didn't complete, with some handshake data being sent but none received.

chrome-internals.txt is a snippet from webrtc-internals that shows how the selected pair and the transport were the same Janus came to for Chrome as well. I guess the important bit there is the "googReadable" still to false: not sure whether "googActiveConnection" just refers to the established connectivity or not, or whether "bytesReceived" just refers to audio/video/data bytes or handshake data as well.

Finally, janus-chrome-icetcp.pcap.pcapng is the actual dump of the messages being exchanged over the selected pair. You'll see how Chrome successfully connects to Janus, and sends connectivity checks that Janus (libnice) answers to. Intertwined with these exchanges, there are some DTLS Hello messages Janus is sending (219 bytes, which become 221 with the 2 bytes of framing header) and retransmitting as it's getting no response. Eventually, after about a minute I closed the PeerConnection and the TCP connection was shut down.

Not sure where the issue is: thinking about it, I only see libnice answering to connectivity checks over the TCP connection, and not generating any. DTLS is originated immediately after the first successful connectivity check. May it be that libnice assumes an ICE-TCP "channel" is ready as soon as the underlying TCP connection is established and a CC received, while Chrome expects to receive CCs as well which is never happening?

You need to generate CCs, even on ICE-TCP. See https://tools.ietf.org/html/rfc6544#section-7.1 

[Lorenzo Miniero]

Il giorno venerdì 6 febbraio 2015 03:28:23 UTC+1, Justin Uberti ha scritto:

You need to generate CCs, even on ICE-TCP. See https://tools.ietf.org/html/rfc6544#section-7.1 

 Thanks for the clarification, I'll notify the bug to the libnice developers.

Lorenzo

[Michael T]

I've been working on patching libnice to get it to interoperate with Chrome and I think I'm getting close.  I definitely needed to add an initial connectivity check to get Chrome to start receiving media.

However, I'm wondering if Chrome expects the connectivity checks to continue after the session is established.  It looks like the libnice code assumes that they aren't needed when using a reliable transport like TCP.  However, if they are, that might explain why media streams seem to stop after working fine for a bit.

On a related note, https://tools.ietf.org/html/rfc6544#section-4.5 seems to indicate that the port should be set to 9 for active TCP candidates, but Chrome is setting it to 0.

Thanks,

Mike

[Vikas]

Chrome does expect connectivity checks after connection establish.

/Vikas

[Lorenzo Miniero]

Michael,

thanks for working on this! If you need any beta-tester for this, just let me know.

Lorenzo

[Justin Uberti]

Chrome doesn't require connectivity checks after the session is established, but it does send them, and the remote side must respond to them.

[Iñaki Baz Castillo]

2015-02-03 16:41 GMT+01:00 Lorenzo Miniero <lmin...@gmail.com>:
> I'm trying to get our gateway (which uses libnice >= 0.1.8) to work with the
> ICE-TCP support in Chrome, but I'm getting some problems. The ICE setup
> works fine, and both the browser and the gateway report a successful
> completion of ICE. Anyway, the DTLS Hello messages sent by the gateway are
> ignored by Chrome, or at least it never answers, and so the handshake never
> completes. It doesn't look like a transport issues as I can see, using
> Wireshark, connectivity checks still being exchanged between the two.

Hi Lorenzo,

I've ICE+DTLS+RTP over TCP working in my server [*] (note however that
it is a ICE-Lite server).

I don't remember now the exact details but when handling with TCP it
may happen that the remote SDP has not been received yet, so a peer
may receive a TCP connection and DTLS messages even before it knows
the fingerprint of the remote peer and so on. In that case it cannot
just "discard" such a TCP packet since the sender won't send it again
(TCP rules).

If this is not your issue at all, then I also remember that Chrome
won't open a TCP connection from the localhost interface (127.0.0.1 or
::1). So you need to play with non-localhost IPs.


[*] https://github.com/ibc/MediaSoup


--
Iñaki Baz Castillo
<i...@aliax.net>

[Lorenzo Miniero]

Hi Iñaki,

all the steps you described should apply to Janus and what/how I tried as well, so I guess the only different there is the fact we're not advertising as an ICE-lite server, I'll try that too (although there are some known deployments where we cannot do that). Are you using libnice too in your project or something different?

Thanks,

Lorenzo

[Iñaki Baz Castillo]

2015-02-19 9:19 GMT+01:00 Lorenzo Miniero <lmin...@gmail.com>:
> all the steps you described should apply to Janus and what/how I tried as
> well, so I guess the only different there is the fact we're not advertising
> as an ICE-lite server, I'll try that too (although there are some known
> deployments where we cannot do that). Are you using libnice too in your
> project or something different?

I implemented STUN/ICE from scratch.

Are you forcing just TCP candidates in the SDP sent by the server?

[Lorenzo Miniero]

I disabled UDP via iptables thus forcing ICE-TCP candidates to be used. As explained above, that seems to be working until it comes to libnice sending a connectivity check after TCP is established. This isn't happening due to how libnice is implemented, which causes Chrome never to get to a working stage. Apart from that, TCP works fine, Chrome sends CCs and libnice answers, the first CC from libnice is the only thing that is missing. I assume your own stack takes care of that which is why it works for you.

Lorenzo

[Iñaki Baz Castillo]

2015-02-19 11:17 GMT+01:00 Lorenzo Miniero <lmin...@gmail.com>:
> Chrome sends CCs and libnice answers, the first CC from libnice is the only
> thing that is missing. I assume your own stack takes care of that which is
> why it works for you.

My stack does not send ICE connectivity checks since it's a Lite
implementation :)

Have you tried signaling "a=ice-lite" in the SDP so Chrome does not
expect a CC? I mean, just for testing.

[Lorenzo Miniero]

I actually haven't, I'll try that later, thanks!

Lorenzo

[Lorenzo Miniero]

Just to confirm that, as Iñaki suggested (thanks for that!), with ICE Lite it does indeed work as expected. This seems to also confirm the nature of the problem when libnice is working in Full mode instead, which should probably be fixed anyway.

Lorenzo

[Michael T]

To be clear, I'm not part of the libnice team; I'm working on this for our own product.  We are focussed only on interoperability with Chrome at this point, so YMMV.  However, I've tried to make a very limited set of changes so hopefully haven't changed compatibility with other endpoints.

With that said, my repo is at: https://github.com/quicklyfrozen/libnice.git   

Using this version, video over TCP seems to be working well for us (using licode for the server and Chrome clients).

Based on the answer I received to this thread, I've also changed it to initiate keep alive checks on TCP connections, and changed the interval to 2.5 seconds (much frequenter then the RFC calls for, but closer to what Chrome does).

I've also fixed another issue we ran in to when going through a NAT that changed port numbers.  Libnice would create a new candidate for the new port/IP combo, but it wouldn't copy the username/password so it would not work with Chrome.

And one change you may not want -- it dumps logging to stdout rather then using glib calls.  That was more convenient for us :-).

-Mike