Jitsi Videobridge high rate of ICE Failures

468 views
Skip to first unread message

David Ertel

unread,
Nov 2, 2015, 7:39:58 PM11/2/15
to discuss-webrtc
We are using Jitsi videobridge (version 554) and are experiencing a very high rate of ICE failures (somewhere around 80%). The other 20% seem to establish very quickly. The video bridge is deployed in Amazon EC2 in a VPC. We also have an xmpp server acting as a focus controller running in the same VPC. All incoming UDP ports are open to the video bridge server. It appears, from the jitsi videobridge logs that the bridge is not finding any candidates to establish streams with. It only finds it's own IPs to add to the candidate list.

This is the message that the bridge is sending to our XMPP server 
<iq id="5637db780000da5d243ede4b" to="myr...@fake.com" from="videobridge.fake.com"type="result">
<conference xmlns="http://jitsi.org/protocol/colibri" id="e8ba7b9446d40e12">
<content name="audio">
<channel endpoint="myr...@fake.com/emberchat-6617" expire="60" id="4c68932d6e69c45f" initiator="true" rtp-level-relay-type="translator">
<source xmlns="urn:xmpp:jingle:apps:rtp:ssma:0" ssrc="1754789699"/>
<transport xmlns="urn:xmpp:jingle:transports:ice-udp:1" pwd="3u99oh7tfst7neonupb7h7u6k8" ufrag="bv0l41a3532jm1">
<fingerprint xmlns="urn:xmpp:jingle:apps:dtls:0" hash="sha-1">
35:8E:DD:2B:22:10:F3:26:0D:02:16:A1:46:DE:68:1D:D4:96:D9:D8
</fingerprint>
<candidate component="1" foundation="1" generation="0" id="e8ba7b9446d40e127fe6921113f53cbc0496664e0" network="0" priority="2130706431" protocol="udp"type="host" ip="172.18.13.87" port="10096"/>
<candidate component="1" foundation="2" generation="0" id="e8ba7b9446d40e127fe6921113f53cbc04068f746" network="0" priority="1694498815" protocol="udp"type="srflx" ip="54.152.14.241" port="10096" rel-addr="172.18.13.87" rel-port="10096"/>
<candidate component="2" foundation="1" generation="0" id="e8ba7b9446d40e127fe6921113f53cbc06168d3c5" network="0" priority="2130706430" protocol="udp"type="host" ip="172.18.13.87" port="10097"/>
<candidate component="2" foundation="2" generation="0" id="e8ba7b9446d40e127fe6921113f53cbc0327b9698" network="0" priority="1694498814" protocol="udp"type="srflx" ip="54.152.14.241" port="10097" rel-addr="172.18.13.87" rel-port="10097"/>
</transport>
</channel>
</content>
<content name="video">
<channel endpoint="myr...@fake.com/emberchat-6617" expire="60" id="add991c30ee91f05" initiator="true" rtp-level-relay-type="translator">
<source xmlns="urn:xmpp:jingle:apps:rtp:ssma:0" ssrc="4166628451"/>
<transport xmlns="urn:xmpp:jingle:transports:ice-udp:1" pwd="4h13rim848c82ftr74n55r9oe5" ufrag="70si41a3532jnd">
<fingerprint xmlns="urn:xmpp:jingle:apps:dtls:0" hash="sha-1">
99:B3:93:53:69:A0:84:63:6C:FD:6A:83:7B:78:6F:53:77:3D:95:BC
</fingerprint>
<candidate component="1" foundation="1" generation="0" id="e8ba7b9446d40e122dd147da349e15b0676a1f0c" network="0" priority="2130706431" protocol="udp"type="host" ip="172.18.13.87" port="10098"/>
<candidate component="1" foundation="2" generation="0" id="e8ba7b9446d40e122dd147da349e15b034c66dfc" network="0" priority="1694498815" protocol="udp"type="srflx" ip="54.152.14.241" port="10098" rel-addr="172.18.13.87" rel-port="10098"/>
<candidate component="2" foundation="1" generation="0" id="e8ba7b9446d40e122dd147da349e15b0681c81de" network="0" priority="2130706430" protocol="udp"type="host" ip="172.18.13.87" port="10099"/>
<candidate component="2" foundation="2" generation="0" id="e8ba7b9446d40e122dd147da349e15b061231e7c" network="0" priority="1694498814" protocol="udp"type="srflx" ip="54.152.14.241" port="10099" rel-addr="172.18.13.87" rel-port="10099"/>
</transport>
</channel>
</content>
</conference>
</iq>


Here is what I see in the logs, notice that the only candidates are the local IPs of the server.

2015-11-02 22:20:01.070 INFO: [20] org.jitsi.videobridge.Videobridge.info() Created conference 2fc39ec860d30ff6. The total number of conferences is now 1, channels 0, video streams 0.

2015-11-02 22:20:01.071 INFO: [20] org.jitsi.videobridge.Conference.info() Created content audio of conference 2fc39ec860d30ff6. The total number of conferences is now 1, channels 0, video streams 0.

2015-11-02 22:20:01.170 INFO: [20] org.jitsi.videobridge.IceUdpTransportManager.info() Appending an AWS harvester to the ICE agent.

2015-11-02 22:20:01.170 INFO: [20] org.ice4j.ice.Agent.gatherCandidates() Gather candidates for component audio.RTP

2015-11-02 22:20:01.175 INFO: [20] org.ice4j.ice.Agent.createComponent() 172.18.13.87:10056/udp (host)

2015-11-02 22:20:01.175 INFO: [20] org.ice4j.ice.Agent.createComponent() 54.152.14.241:10056/udp (srflx)

2015-11-02 22:20:01.175 INFO: [20] org.ice4j.ice.Agent.gatherCandidates() Gather candidates for component audio.RTCP

2015-11-02 22:20:01.176 INFO: [20] org.ice4j.ice.Agent.createComponent() 172.18.13.87:10057/udp (host)

2015-11-02 22:20:01.176 INFO: [20] org.ice4j.ice.Agent.createComponent() 54.152.14.241:10057/udp (srflx)


Eventually the channels time out and the conference is torn down due to ICE never succeeding so we see a bunch of messages like the following

2015-11-02 21:55:27.786 INFO: [13] org.ice4j.ice.Agent.setState() ICE state changed from Waiting to Terminated

2015-11-02 21:55:27.788 INFO: [13] org.jitsi.videobridge.Channel.info() Expired channel 4c68932d6e69c45f of content audio of conference e8ba7b9446d40e12. The total number of conferences is now 1, channels 1, video streams 1.


I'm not sure where to go to further debug this issue, any advice would be greatly appreciated.

Boris Grozev

unread,
Nov 2, 2015, 8:19:02 PM11/2/15
to discuss...@googlegroups.com

Hi,

On Nov 2, 2015 6:39 PM, "David Ertel" <david...@gmail.com> wrote:
>
> We are using Jitsi videobridge (version 554) and are experiencing a very high rate of ICE failures (somewhere around 80%). The other 20% seem to establish very quickly. The video bridge is deployed in Amazon EC2 in a VPC. We also have an xmpp server acting as a focus controller running in the same VPC. All incoming UDP ports are open to the video bridge server. It appears, from the jitsi videobridge logs that the bridge is not finding any candidates to establish streams with. It only finds it's own IPs to add to the candidate list.

In the response below the bridge's public IP address is included. The bridge will discover the clients' candidates as peer reflexive once it receives STUN binding requests from them. So this part seems normal.

This doesn't look right at all. In ice4j the "Terminated" state is supposed to only be reached after a connection is established.

Can you include the full logs from videobridge? Are you using it with Jicofo or another "focus"? Have you tried different versions?

Regards,
Boris

PS this seems to be specific to jitsi-vedeobridge. Feel free to also raise the issue on d...@jitsi.org

>
> 2015-11-02 21:55:27.788 INFO: [13] org.jitsi.videobridge.Channel.info() Expired channel 4c68932d6e69c45f of content audio of conference e8ba7b9446d40e12. The total number of conferences is now 1, channels 1, video streams 1.
>
>
> I'm not sure where to go to further debug this issue, any advice would be greatly appreciated.
>

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/d4a5b0d3-ea6b-423a-9a94-edac68e4c74e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Jeremy Noring

unread,
Nov 2, 2015, 9:02:34 PM11/2/15
to discuss-webrtc
Any idea if it's failing ICE negotiation (in other words, failing to negotiate a network candidate) or failing DTLS negotiation (in other words, failing to secure the channel after a network candidate has been negotiated)?  That's probably the first thing I'd try to work out.  The reality is some percentages of your failures are probably going to fall in both of those categories.


On Monday, November 2, 2015 at 5:39:58 PM UTC-7, David Ertel wrote:
We are using Jitsi videobridge (version 554) and are experiencing a very high rate of ICE failures (somewhere around 80%). The other 20% seem to establish very quickly. The video bridge is deployed in Amazon EC2 in a VPC. We also have an xmpp server acting as a focus controller running in the same VPC. All incoming UDP ports are open to the video bridge server. It appears, from the jitsi videobridge logs that the bridge is not finding any candidates to establish streams with. It only finds it's own IPs to add to the candidate list.

This is the message that the bridge is sending to our XMPP server 
<iq id="5637db780000da5d243ede4b" to="myr...@fake.com" from="videobridge.fake.com"type="result">
<conference xmlns="http://jitsi.org/protocol/colibri" id="e8ba7b9446d40e12">
<content name="audio">
<channel endpoint="myroom@fake.com/emberchat-6617" expire="60" id="4c68932d6e69c45f" initiator="true" rtp-level-relay-type="translator">
<source xmlns="urn:xmpp:jingle:apps:rtp:ssma:0" ssrc="1754789699"/>
<transport xmlns="urn:xmpp:jingle:transports:ice-udp:1" pwd="3u99oh7tfst7neonupb7h7u6k8" ufrag="bv0l41a3532jm1">
<fingerprint xmlns="urn:xmpp:jingle:apps:dtls:0" hash="sha-1">
35:8E:DD:2B:22:10:F3:26:0D:02:16:A1:46:DE:68:1D:D4:96:D9:D8
</fingerprint>
<candidate component="1" foundation="1" generation="0" id="e8ba7b9446d40e127fe6921113f53cbc0496664e0" network="0" priority="2130706431" protocol="udp"type="host" ip="172.18.13.87" port="10096"/>
<candidate component="1" foundation="2" generation="0" id="e8ba7b9446d40e127fe6921113f53cbc04068f746" network="0" priority="1694498815" protocol="udp"type="srflx" ip="54.152.14.241" port="10096" rel-addr="172.18.13.87" rel-port="10096"/>
<candidate component="2" foundation="1" generation="0" id="e8ba7b9446d40e127fe6921113f53cbc06168d3c5" network="0" priority="2130706430" protocol="udp"type="host" ip="172.18.13.87" port="10097"/>
<candidate component="2" foundation="2" generation="0" id="e8ba7b9446d40e127fe6921113f53cbc0327b9698" network="0" priority="1694498814" protocol="udp"type="srflx" ip="54.152.14.241" port="10097" rel-addr="172.18.13.87" rel-port="10097"/>
</transport>
</channel>
</content>
<content name="video">
<channel endpoint="myroom@fake.com/emberchat-6617" expire="60" id="add991c30ee91f05" initiator="true" rtp-level-relay-type="translator">
</content>
</conference>
</iq>
Reply all
Reply to author
Forward
0 new messages