Coturn not working with BigBlueButton and CoTurn behind SAME(!) NAT

932 views
Skip to first unread message

Philipp Memmel

unread,
Aug 22, 2020, 5:31:03 AM8/22/20
to TURN Server (Open-Source project)
Hey guys,

I've been setting setting up a BigBlueButton system for a week in combination with a coturn STUN/TURN server. My current setup looks like that:
  • physical machine behind NAT/Firewall (with BigBlueButton)
  • virtual machine behind same NAT/Firewall (coturn server with working let's encrypt certificates)
  • both machines are in the same private subnet (172.16.x.0/24, also IPv6)
  • NAT reflection is enabled on the firewall
  • firewall is forwarding alle neccessary tcp/udp ports
  • Trickle WebRTC STUN and TURN server tests work fine
  • STUN service ist working perfectly, also with BigBlueButton
PROBLEM: BigBlueButton can't use coturn server

According to documentation and tutorials i tried the following turnserver.conf:

listening-port=3478
tls-listening-port=5349

external-ip=EXTERNAL_IP_OF_COTURN_SERVER_v4/INTERNAL_IP_OF_COTURN_SERVER_v4
external-ip=EXTERNAL_IP_OF_COTURN_SERVER_v6/INTERNAL_IP_OF_COTURN_SERVER_v6

fingerprint

use-auth-secret
static-auth-secret=SECRET

realm=OFFICIAL_HOSTNAME_OF_TURNSERVER

cert=/PATH/TO/fullchain.pem
pkey=/PATH/TO/privkey.pem
cipher-list="ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS"
dh2066
no-tlsv1
no-tlsv1_1

log-file=/var/log/coturn.log

simple-log
verbose

The effect was: Relay candidates are being sent to the BigBlueButton server, offering the external IP of the turn server. In the coturn logs the connection is being seen and everything looks fine at first. However, after a few seconds the client just closes the connection without any error and BigBlueButton audio session fails to connect.

According to comments in config file I was pretty sure it should work, but it didn't. After my server admin and me did a lot of trying and researching we finally concluded that the firewall was not the problem (no packets being dropped or anything else). My server admin then finally figured out a configuration where the TURN server worked:

He changed the "external-ip" entries in /etc/turnserver.conf to the IPs of the BigBlueButton server(!!!), which makes completely no sense to me. I now see the local IP of the turn server given as ICE candidate to the WebRTC client (which I'm not supposed to see, right?), but the BigBlueButton server can apparently deal with that und I can finally use the turnserver in this setup.

However, I'm pretty sure that's not how it is supposed to work. Do you guys have any explanation for that and maybe an idea how to make the configuration "correct"? Is there a bug or does the firewall/NAT reflection have to enable special features or something?

Thanks a lot in advance. Feel free to ask, if you need more information about my setup.

Regards

Philipp


David P

unread,
Aug 22, 2020, 6:16:18 PM8/22/20
to TURN Server (Open-Source project)
I'm not a coturn expert, but what I'd try is: Run the coturn turn client from the commandline of a client machine. The coturn turn client is part of the coturn repo.

On Sun, 23 Aug 2020, 9:57 am , <turn-server-project...@googlegroups.com> wrote:

Hey guys,
 
I've been setting setting up a BigBlueButton system for a week in
combination with a coturn STUN/TURN server. My current setup looks like
that:
 
- physical machine behind NAT/Firewall (with BigBlueButton)
- virtual machine behind same NAT/Firewall (coturn server with working
let's encrypt certificates)
- both machines are in the same private subnet (172.16.x.0/24, also IPv6)
- NAT reflection is enabled on the firewall
- firewall is forwarding alle neccessary tcp/udp ports
- Trickle WebRTC STUN and TURN server tests work fine
- STUN service ist working perfectly, also with BigBlueButton

*PROBLEM: BigBlueButton can't use coturn server*
*He changed the "external-ip" entries in /etc/turnserver.conf to the IPs of
the BigBlueButton server(!!!),* which makes completely no sense to me. I
now see the local IP of the turn server given as ICE candidate to the
WebRTC client (which I'm not supposed to see, right?), but the
BigBlueButton server can apparently deal with that und I can finally use
the turnserver in this setup.
 
However, I'm pretty sure that's not how it is supposed to work. Do you guys
have any explanation for that and maybe an idea how to make the
configuration "correct"? Is there a bug or does the firewall/NAT reflection
have to enable special features or something?
 
Thanks a lot in advance. Feel free to ask, if you need more information
about my setup.
 
Regards
 
Philipp
You received this digest because you're subscribed to updates for this group. You can change your settings on the group membership page.
To unsubscribe from this group and stop receiving emails from it send an email to turn-server-project-rfc57...@googlegroups.com.
Message has been deleted

bayden10

unread,
Aug 22, 2020, 10:44:25 PM8/22/20
to TURN Server (Open-Source project)
Hi, did you use the BBB install script with the TURN option? Are you running DNAT on your core router using hairpin NAT?

Did you setup the 

turn-stun-servers.xml


Should look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">




    <bean id="stun0" class="org.bigbluebutton.web.services.turn.StunServer">
        <constructor-arg index="0" value="stun:turn.domain.ca"/>
    </bean>








    <bean id="turn0" class="org.bigbluebutton.web.services.turn.TurnServer">
        <constructor-arg index="0" value="1534a29f6e2d27f34cb3f2d6f000d85d"/>
        <constructor-arg index="1" value="turns:turn.domain.ca:443?transport=tcp"/>
        <constructor-arg index="2" value="86400"/>
    </bean>




    <bean id="turn1" class="org.bigbluebutton.web.services.turn.TurnServer">
        <constructor-arg index="0" value="1534a29f6e2d27f34cb3f2d6f000d85d"/>
        <constructor-arg index="1" value="turn:turn.domain.ca:443?transport=tcp"/>
        <constructor-arg index="2" value="86400"/>
    </bean>




    <bean id="stunTurnService"
            class="org.bigbluebutton.web.services.turn.StunTurnService">
        <property name="stunServers">
            <set>
                <ref bean="stun0"/>
            </set>
        </property>
        <property name="turnServers">
            <set>
                <ref bean="turn0"/>
                <ref bean="turn1"/>
            </set>
        </property>
    </bean>
</beans>

 
You could also open a thread here:  bigbluebutton-setup

~b10

Philipp Memmel

unread,
Aug 23, 2020, 3:39:58 AM8/23/20
to TURN Server (Open-Source project)
Hey,
the firewall/router is allowing connections between both of the vm and the bbb server on every service via their external static ip addresses (tested with https, ssh etc.). I tested the stun server functionality by executing "stun EXTERNAL_HOSTNAME_OF_TURNSERVER" on the bbb machine which resulted in this:

STUN client version 0.97
Primary: Dependent Mapping, random port, no hairpin
Return value is 0x000018

Is this maybe part of the problem? Executing "stun INTERNAL_IP" leads to the following output:

STUN client version 0.97
Primary: Independent Mapping, Independent Filter, preserves ports, will hairpin
Return value is 0x000003


According to WebRTC client logs the stun server part of coturn is working flawlessly though.

@b10: I also tested hairpin nat functionality according to the bbb installation tutorial (https://docs.bigbluebutton.org/2.2/configure-firewall.html#configure-a-dummy-nic-if-required). Everything looks fine, the curl command there worked.

I haven't used the turn option of bbb-install.sh, but configured the turn-stun-servers.xml manually exactly like you posted. However, I substituted the according ports 443 to 3478 and 5349 to match my coturn config, of course.

The reason I didn't open a thread at "bigbluebutton-setup" at first is that my admin transferred the coturn server to a non-natted public host for testing. Result: Without changing any config on the BBB server (except exchanging the hostname of the coturn server of course) it could use the TURN server functionality of coturn perfectly, BBB was running as it should.

I attached the output of "turnutils_uclient -v EXTERNAL_HOSTNAME_OF_TURNSERVER" in both cases, running it on a test client from home (so completely externally from the server network). The only difference I can see ist that the given relay address in one case is the public static IP of the turnserver, in the other case it is the local IP of the turnserver in the natted network. Should I be worried about the not transferred packages?

Thanks in advance!

Philipp



With external ip mapping of turnserver in turnserver.conf:

root@HOSTNAME:~# turnutils_uclient -v -W SECRET EXTERNAL_TURNSERVER_HOSTNAME
0: IPv4. Connected from: LOCAL_IP_OF_TEST_CLIENT:45193
0: IPv4. Connected to: EXTERNAL_IP_OF_TURNSERVER:3478
0: allocate sent
0: allocate response received:
0: allocate sent
0: allocate response received:
0: success
0: IPv4. Received relay addr: EXTERNAL_IP_OF_TURNSERVER:55322
0: clnet_allocate: rtv=842173284937230865
0: refresh sent
0: refresh response received:
0: success
0: IPv4. Connected from: LOCAL_IP_OF_TEST_CLIENT:38827
0: IPv4. Connected to: EXTERNAL_IP_OF_TURNSERVER:3478
0: IPv4. Connected from: LOCAL_IP_OF_TEST_CLIENT:39758
0: IPv4. Connected to: EXTERNAL_IP_OF_TURNSERVER:3478
0: allocate sent
0: allocate response received:
0: allocate sent
0: allocate response received:
0: success
0: IPv4. Received relay addr: EXTERNAL_IP_OF_TURNSERVER:55323
0: clnet_allocate: rtv=0
0: refresh sent
0: refresh response received:
0: success
0: allocate sent
0: allocate response received:
0: allocate sent
0: allocate response received:
0: success
0: IPv4. Received relay addr: EXTERNAL_IP_OF_TURNSERVER:63550
0: clnet_allocate: rtv=1341045736506650642
0: refresh sent
0: refresh response received:
0: success
0: channel bind sent
0: cb response received:
0: success: 0x6615
0: channel bind sent
0: cb response received:
0: success: 0x6615
0: channel bind sent
0: cb response received:
0: success: 0x491e
0: channel bind sent
0: cb response received:
0: success: 0x491e
0: channel bind sent
0: cb response received:
0: success: 0x6dd5
0: Total connect time is 0
1: start_mclient: msz=2, tot_send_msgs=0, tot_recv_msgs=0, tot_send_bytes ~ 0, tot_recv_bytes ~ 0
2: start_mclient: msz=2, tot_send_msgs=5, tot_recv_msgs=0, tot_send_bytes ~ 500, tot_recv_bytes ~ 0
3: start_mclient: msz=2, tot_send_msgs=5, tot_recv_msgs=0, tot_send_bytes ~ 500, tot_recv_bytes ~ 0
4: start_mclient: msz=2, tot_send_msgs=5, tot_recv_msgs=0, tot_send_bytes ~ 500, tot_recv_bytes ~ 0
5: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
6: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
7: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
8: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
9: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
10: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
11: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
11: done, connection 0x7f2842950010 closed.
12: start_mclient: msz=1, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
12: start_mclient: msz=1, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
14: start_mclient: msz=1, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
14: done, connection 0x7f284292f010 closed.
14: start_mclient: tot_send_msgs=10, tot_recv_msgs=0
14: start_mclient: tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
14: Total transmit time is 14
14: Total lost packets 10 (100.000000%), total send dropped 0 (0.000000%)
14: Average round trip delay 0.000000 ms; min = 4294967295 ms, max = 0 ms
14: Average jitter -nan ms; min = 4294967295 ms, max = 0 ms




With external ip mapping of turnserver in turnserver.conf:

0: IPv4. Connected from: LOCAL_IP_OF_TEST_CLIENT:51239
0: IPv4. Connected to: EXTERNAL_IP_OF_TURNSERVER:3478
0: allocate sent
0: allocate response received:
0: allocate sent
0: allocate response received:
0: success
0: IPv4. Received relay addr: LOCAL_IP_OF_TURNSERVER_IN_NETWORK_BEHIND_NAT:55456
0: clnet_allocate: rtv=13778300869919701620
0: refresh sent
0: refresh response received:
0: success
0: IPv4. Connected from: 192.168.1.10:44659
0: IPv4. Connected to: EXTERNAL_IP_OF_TURNSERVER:3478
0: IPv4. Connected from: 192.168.1.10:53107
0: IPv4. Connected to: EXTERNAL_IP_OF_TURNSERVER:3478
0: allocate sent
0: allocate response received:
0: allocate sent
0: allocate response received:
0: success
0: IPv4. Received relay addr: LOCAL_IP_OF_TURNSERVER_IN_NETWORK_BEHIND_NAT:55457
0: clnet_allocate: rtv=0
0: refresh sent
0: refresh response received:
0: success
0: allocate sent
0: allocate response received:
0: allocate sent
0: allocate response received:
0: success
0: IPv4. Received relay addr: LOCAL_IP_OF_TURNSERVER_IN_NETWORK_BEHIND_NAT:64116
0: clnet_allocate: rtv=12497462629213989747
0: refresh sent
0: refresh response received:
0: success
0: channel bind sent
0: cb response received:
0: success: 0x48d2
0: channel bind sent
0: cb response received:
0: success: 0x48d2
0: channel bind sent
0: cb response received:
0: success: 0x4504
0: channel bind sent
0: cb response received:
0: success: 0x4504
0: channel bind sent
0: cb response received:
0: success: 0x6933
0: Total connect time is 0
1: start_mclient: msz=2, tot_send_msgs=0, tot_recv_msgs=0, tot_send_bytes ~ 0, tot_recv_bytes ~ 0
2: start_mclient: msz=2, tot_send_msgs=0, tot_recv_msgs=0, tot_send_bytes ~ 0, tot_recv_bytes ~ 0
3: start_mclient: msz=2, tot_send_msgs=0, tot_recv_msgs=0, tot_send_bytes ~ 0, tot_recv_bytes ~ 0
4: start_mclient: msz=2, tot_send_msgs=5, tot_recv_msgs=0, tot_send_bytes ~ 500, tot_recv_bytes ~ 0
5: start_mclient: msz=2, tot_send_msgs=5, tot_recv_msgs=0, tot_send_bytes ~ 500, tot_recv_bytes ~ 0
6: start_mclient: msz=2, tot_send_msgs=5, tot_recv_msgs=0, tot_send_bytes ~ 500, tot_recv_bytes ~ 0
7: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
8: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
9: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
10: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
11: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
12: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
13: start_mclient: msz=2, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
13: done, connection 0x7f636b9a4010 closed.
14: start_mclient: msz=1, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
15: start_mclient: msz=1, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
16: start_mclient: msz=1, tot_send_msgs=10, tot_recv_msgs=0, tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
16: done, connection 0x7f636b983010 closed.
16: start_mclient: tot_send_msgs=10, tot_recv_msgs=0
16: start_mclient: tot_send_bytes ~ 1000, tot_recv_bytes ~ 0
16: Total transmit time is 16
16: Total lost packets 10 (100.000000%), total send dropped 0 (0.000000%)
16: Average round trip delay 0.000000 ms; min = 4294967295 ms, max = 0 ms
16: Average jitter -nan ms; min = 4294967295 ms, max = 0 ms

bayden10

unread,
Aug 23, 2020, 10:32:48 AM8/23/20
to TURN Server (Open-Source project)
I would recommend using the bbb-install option switch for TURN as it should use port 443 when doing a hand off. 

This indicates it's not working using an internal IP
STUN client version 0.97
Primary: Independent Mapping, Independent Filter, preserves ports, will hairpin
Return value is 0x000003

STUN should be external, like use stun:stun.3cx.com or Google's. This can also be set in the WebRtcEndpoint.conf.ini but it requires an IP/Port if issue still resides.

~b10

David P

unread,
Aug 23, 2020, 6:35:17 PM8/23/20
to TURN Server (Open-Source project)
I believe 'independent mapping' is necessary.

There's also been discussion on the webrtc list about DNS names working better than IP numbers for TURN servers, and that it might be a bug in Chrome specifically.

On Mon, 24 Aug 2020, 9:57 am , <turn-server-project...@googlegroups.com> wrote:
David P <davidsw...@gmail.com>: Aug 23 10:16AM +1200

I'm not a coturn expert, but what I'd try is: Run the coturn turn client
from the commandline of a client machine. The coturn turn client is part of
the coturn repo.
 
On Sun, 23 Aug 2020, 9:57 am , <
bayden10 <jimira...@gmail.com>: Aug 22 07:44PM -0700

Hey,
the firewall/router is allowing connections between both of the vm and the
bbb server on every service via their external static ip addresses (tested
with https, ssh etc.). I tested the stun server functionality by executing
"stun EXTERNAL_HOSTNAME_OF_TURNSERVER" on the bbb machine which resulted in
this:
 
STUN client version 0.97

Primary: Dependent Mapping, random port, no hairpin
Return value is 0x000018
 
Is this maybe part of the problem? Executing "stun INTERNAL_IP" leads to
the following output:
 
STUN client version 0.97
Primary: Independent Mapping, Independent Filter, preserves ports, will
hairpin
Return value is 0x000003
 
 
bayden10 <jimira...@gmail.com>: Aug 23 07:32AM -0700
Reply all
Reply to author
Forward
0 new messages