Issues with bbb multi docker container in Swarm cluster

[Jerome Revillard]

Dear all,

I'm trying to install bbb using the following compose file: https://github.com/bigbluebutton/bigbluebutton/tree/master/labs/docker

I successfully installed it using a regular standalone docker system but I cannot make it working in a Swarm environment where Traefik is deployed independently (it's another stack)

The Swarm drawbacks are:

• depends_on is not recognised: this means that containers which cannot find the needed dependencies must exit so that swarm restart them. After some time, the all infrastructure should be stabilized.
• you cannot depend on an external env file so for the moment I hardcoded all the values inside my compose file.

Anyway, I managed to get the services started:

root@swarm-node1 ~ # docker stack ls
NAME           SERVICES
bigbluebutton  13
traefik        3

root@swarm-node1 ~ # docker stack services bigbluebutton
ID                  NAME                           MODE                REPLICAS            IMAGE                                   PORTS
7fujjacxv78u        bigbluebutton_bbb-html5        replicated          1/1                 fcecagno/bigbluebutton:bbb-html5        
8fpew1wp10zm        bigbluebutton_bbb-fsesl-akka   replicated          1/1                 fcecagno/bigbluebutton:bbb-fsesl-akka  
8vkp7hv9ifqk        bigbluebutton_redis            replicated          1/1                 redis:latest                            
epfghqnzfb49        bigbluebutton_bbb-webhooks     replicated          1/1                 fcecagno/bigbluebutton:bbb-webhooks    
htcxwvwvzr6i        bigbluebutton_bbb-web          replicated          1/1                 fcecagno/bigbluebutton:bbb-web          
jlhrh3pt6dy7        bigbluebutton_bbb-freeswitch   replicated          1/1                 fcecagno/bigbluebutton:bbb-freeswitch  
o9tf9lcytqas        bigbluebutton_kurento          replicated          1/1                 fcecagno/bigbluebutton:bbb-kurento      
t3hws6hh54g0        bigbluebutton_nginx            replicated          1/1                 fcecagno/bigbluebutton:bbb-nginx        
uibq5i20q442        bigbluebutton_bbb-webrtc-sfu   replicated          1/1                 fcecagno/bigbluebutton:bbb-webrtc-sfu  
v4ik9swqux52        bigbluebutton_bbb-apps-akka    replicated          1/1                 fcecagno/bigbluebutton:bbb-apps-akka    
xyhswotks2fu        bigbluebutton_coturn           replicated          1/1                 fcecagno/bigbluebutton:bbb-coturn       *:3478->3478/udp,*:3478->3478/tcp
y52b9dtenzbp        bigbluebutton_bbb-greenlight   replicated          1/1                 bigbluebutton/greenlight:v2            
y7onehzfn0oe        bigbluebutton_mongo            replicated          1/1                 mongo:3.4  

When I open the Greenlight interface, I can create a new user but I then get this error:

I tried to look at the different container logs but cannot figured out what the problem could be.

I found this in the greenlight_logs/production.log file:

F, [2018-10-29T16:08:27.568274 #1] FATAL -- : [e080600b-afbe-4b4a-b403-8f94052a80f0]  
F, [2018-10-29T16:08:27.568439 #1] FATAL -- : [e080600b-afbe-4b4a-b403-8f94052a80f0] BigBlueButton::BigBlueButtonException (Impossible to convert XML to hash. Error: File does not exist: Bad Gateway.):
F, [2018-10-29T16:08:27.568745 #1] FATAL -- : [e080600b-afbe-4b4a-b403-8f94052a80f0]  
F, [2018-10-29T16:08:27.568910 #1] FATAL -- : [e080600b-afbe-4b4a-b403-8f94052a80f0] app/models/room.rb:117:in `recordings'
[e080600b-afbe-4b4a-b403-8f94052a80f0] app/controllers/rooms_controller.rb:46:in `show'
I, [2018-10-29T16:08:27.569645 #1]  INFO -- : [e080600b-afbe-4b4a-b403-8f94052a80f0] Processing by ErrorsController#internal_error as HTML
I, [2018-10-29T16:08:27.569818 #1]  INFO -- : [e080600b-afbe-4b4a-b403-8f94052a80f0]   Parameters: {"room_uid"=>"jer-263-32j"}
I, [2018-10-29T16:08:27.570433 #1]  INFO -- : [e080600b-afbe-4b4a-b403-8f94052a80f0]   Rendering errors/internal_error.html.erb within layouts/application
I, [2018-10-29T16:08:27.570828 #1]  INFO -- : [e080600b-afbe-4b4a-b403-8f94052a80f0]   Rendered errors/internal_error.html.erb within layouts/application (0.2ms)
D, [2018-10-29T16:08:27.571818 #1] DEBUG -- : [e080600b-afbe-4b4a-b403-8f94052a80f0]   CACHE (0.0ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = ? LIMIT ?  [["id", 1], ["LIMIT", 1]]
D, [2018-10-29T16:08:27.576563 #1] DEBUG -- : [e080600b-afbe-4b4a-b403-8f94052a80f0]   Room Load (3.8ms)  SELECT  "rooms".* FROM "rooms" WHERE "rooms"."id" = ? LIMIT ?  [["id", 1], ["LIMIT", 1]]
I, [2018-10-29T16:08:27.577453 #1]  INFO -- : [e080600b-afbe-4b4a-b403-8f94052a80f0]   Rendered shared/_header.html.erb (5.9ms)
I, [2018-10-29T16:08:27.578441 #1]  INFO -- : [e080600b-afbe-4b4a-b403-8f94052a80f0]   Rendered shared/modals/_create_room_modal.html.erb (0.7ms)
I, [2018-10-29T16:08:27.578886 #1]  INFO -- : [e080600b-afbe-4b4a-b403-8f94052a80f0]   Rendered shared/_footer.html.erb (0.2ms)
I, [2018-10-29T16:08:27.579173 #1]  INFO -- : [e080600b-afbe-4b4a-b403-8f94052a80f0] Completed 500 Internal Server Error in 9ms (Views: 5.0ms | ActiveRecord: 3.9ms)

Could somebody help me solving it please ?

Thanks in advance,

Jérôme

[Chad Pilkey]

That docker stuff isn't really ready for any sort of end-user use. It's still in the very early stages and there's still big holes (no recordings at all) in the implementation. If you want to continue on even knowing that, then I would suggest just using Mconf's API Mate to test it first http://mconf.github.io/api-mate/ before bringing in another variable (Greenlight).

[Jerome Revillard]

OK, thanks for the information. I thought it was a bit more mature for sure.
Would you have an example of compose file which brings bbb and Greenlight together then (traefik/letsencrypt is managed externally) ?

Anyway, as I managed to get it working as a standard compose file, I would like to investigate a bit. So basically, I always obtain "Bad Gateway" as soon as I try to use the /bigbluebutton/api endpoint. Could you tell me where could I find some logs in order to progress ?

Best,
Jerome

[Chad Pilkey]

Most of the container work that we've done has been focused on running containers in Kubernetes rather than Docker Swarm so most of my knowledge won't be much help. The first thing I would do is to figure out if the API requests are even reaching the bbb-web container. Accessing "<server>/bigbluebutton/api" should be returning the bbb-web version and if you're getting an error then I would figure out where the error is actually coming from. I'm going to guess that the error is from the external traefik and it's not configured to route the requests. The docker-compose file in that directory that you linked to has traefik routes and the traefik image included in it, but I'm not sure how that would translate to an externally managed traefik. BigBlueButton has quite a few endpoints and I think you would need to replicate them in your own traefik deployment.

[Jerome Revillard]

Thanks for the quick answer; I managed to fix this first issue. I had an issue in my compose file for the bbb-web container where I had traefik.port=80 instead of 8080.

No, when I try to join the meeting audio, I get an error and see this into hte logs:

bigbluebutton_nginx.1.r1t2pb22xwzs@swarm-node2.gnubila.fr    | 2018/10/29 22:33:45 [error] 15#15: *21 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.6, server: xxxxx, request: "GET /ws HTTP/1.1", upstream: "https://10.0.0.31:7443/ws", host: "xxxxx".

This is certainly again a traefik issue. the nginx container cannot contact the bbb-freeswitch one.... This configuration is a bit strange to be by the way as the ngnix traefic backend has the name of the bbb-feeeswitch one.

Best,

Jérôme

[Jerome Revillard]

Ok I found the issue but not the slution yet. In fact the freeswitch container get 2 interfaces (eth0 and eth1):

root@062b11cdae35:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:0a:00:00:08  
          inet addr:10.0.0.8  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:28 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1661 (1.6 KB)  TX bytes:1370 (1.3 KB)

eth1      Link encap:Ethernet  HWaddr 02:42:ac:13:00:03  
          inet addr:172.19.0.3  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1002 errors:0 dropped:0 overruns:0 frame:0
          TX packets:920 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6172373 (6.1 MB)  TX bytes:67249 (67.2 KB)

eth1 seems to be a private swarm interface wherease eth0 is the one to use. Unfortunatly, freeswitch listen to the wron one:

root@062b11cdae35:/# netstat -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 172.19.0.3:5090         0.0.0.0:*               LISTEN      18/freeswitch  
tcp        0      0 172.19.0.3:5060         0.0.0.0:*               LISTEN      18/freeswitch  
tcp        0      0 172.19.0.3:5066         0.0.0.0:*               LISTEN      18/freeswitch  
tcp        0      0 172.19.0.3:8081         0.0.0.0:*               LISTEN      18/freeswitch  
tcp        0      0 172.19.0.3:8082         0.0.0.0:*               LISTEN      18/freeswitch  
tcp        0      0 172.19.0.3:7443         0.0.0.0:*               LISTEN      18/freeswitch  
tcp        0      0 127.0.0.11:41108        0.0.0.0:*               LISTEN      -              
tcp        0      0 0.0.0.0:8021            0.0.0.0:*               LISTEN      18/freeswitch  
udp        0      0 0.0.0.0:1337            0.0.0.0:*                           18/freeswitch  
udp        0      0 127.0.0.11:43979        0.0.0.0:*                           -              
udp        0      0 172.19.0.3:5060         0.0.0.0:*                           18/freeswitch  
udp        0      0 172.19.0.3:5090         0.0.0.0:*                           18/freeswitch  

Any advise on how to fix it ?

best.

[Felipe Cecagno]

You can try to modify this: https://github.com/bigbluebutton/bigbluebutton/blob/v2.0.x-release/labs/docker/freeswitch/docker-entrypoint.sh#L7

I don't know what the value of IP is in your setup, you can start by checking that.

We needed to use both nginx and traefik for FreeSWITCH because we couldn't configure traefik properly to proxy FreeSWITCH - there was something to do with the need of FreeSWITCH receiving the connection under 7443 (secured websocket), but we couldn't configure traefik to do HTTPS->HTTPS. That's why we do traefik (HTTPS) -> nginx (HTTP) -> FreeSWITCH (HTTPS), and it works.

Please keep sharing your results, we're very interested in the work you're doing.

Best regards,

--

   Felipe Cecagno

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To post to this group, send email to bigblueb...@googlegroups.com.
Visit this group at https://groups.google.com/group/bigbluebutton-dev.
For more options, visit https://groups.google.com/d/optout.

[Jerome Revillard]

I did some progress with freeswitch. In fact, you have a typo in your compose file. You have:

xmlstarlet edit --inplace --update '//X-PRE-PROCESS[@cmd="set" and starts-with(@data, "local_ip_v4=")]/@data' --value "${IP}" /opt/freeswitch/conf/vars.xml

whereas it must be:

xmlstarlet edit --inplace --update '//X-PRE-PROCESS[@cmd="set" and starts-with(@data, "local_ip_v4=")]/@data' --value "local_ip_v4=${IP}" /opt/freeswitch/conf/vars.xml

In the case of a simple docker environment it works because Freeswitch automatically update this variable with the first interface returned by the OS.. but in the case of swarm, or as soon as you have more than one network interface, Freeswitch can choose the wrong one.

So, now the next issue when I click on the microphone to join a meeting audio:

bigbluebutton_bbb-html5.1.6n6e6d0t9mb9@swarm-node2.gnubila.fr    | info: REDIS: UpdateRecordingTimerEvtMsg completed sync
bigbluebutton_bbb-web.1.2puphecrt2ys@swarm-node2.gnubila.fr    | Session token = [8jxzzrkkok8hxoad]
bigbluebutton_bbb-web.1.2puphecrt2ys@swarm-node2.gnubila.fr    | 2018-10-30T21:09:34.282Z INFO  org.bigbluebutton.web.services.turn.StunTurnService  -
bigbluebutton_bbb-web.1.2puphecrt2ys@swarm-node2.gnubila.fr    | StunTurnService::getStunServers
bigbluebutton_bbb-web.1.2puphecrt2ys@swarm-node2.gnubila.fr    |
bigbluebutton_bbb-web.1.2puphecrt2ys@swarm-node2.gnubila.fr    | 2018-10-30T21:09:34.282Z INFO  org.bigbluebutton.web.services.turn.StunTurnService  -
bigbluebutton_bbb-web.1.2puphecrt2ys@swarm-node2.gnubila.fr    | StunTurnService::getStunAndTurnServersFor w_9rguiiaeeeqo
bigbluebutton_bbb-web.1.2puphecrt2ys@swarm-node2.gnubila.fr    |
bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted
bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted
bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted
bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted
bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted
bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted
bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted
bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted
bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted
bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted
bigbluebutton_bbb-html5.1.6n6e6d0t9mb9@swarm-node2.gnubila.fr    | info: REDIS: CheckAlivePongSysMsg skipped
bigbluebutton_bbb-html5.1.6n6e6d0t9mb9@swarm-node2.gnubila.fr    | info: REDIS: {"header":{"name":"UpdateRecordingTimerEvtMsg","meetingId":"ddb6ce3ded348aaf915f64a1d2825f12e1f0814a-1540933677899","userId":"not-used"},"body":{"time":0}} emitted
bigbluebutton_bbb-html5.1.6n6e6d0t9mb9@swarm-node2.gnubila.fr    | info: REDIS: UpdateRecordingTimerEvtMsg completed sync
bigbluebutton_bbb-html5.1.6n6e6d0t9mb9@swarm-node2.gnubila.fr    | info: REDIS: CheckAlivePongSysMsg skipped
bigbluebutton_bbb-html5.1.6n6e6d0t9mb9@swarm-node2.gnubila.fr    | error: CLIENT LOG: Audio Error: ICE_NEGOTIATION_FAILED [object RTCPeerConnection]
bigbluebutton_bbb-html5.1.6n6e6d0t9mb9@swarm-node2.gnubila.fr    |  sessionToken=8jxzzrkkok8hxoad, meetingId=ddb6ce3ded348aaf915f64a1d2825f12e1f0814a-1540933677899, requesterUserId=w_9rguiiaeeeqo, fullname=Jerome Revillard, confname=Home Room, externUserID=gl-eoqkmibfgmpq, validUser=valid

Any advise ?

[Chad Pilkey]

It might be the UDP ports not being mapped. WebRTC requires a large range of UDP ports and a lot of the container management systems don't have that kind of mapping well (if at all). It looks like coturn is configured to use UDP 49152-65535 (https://github.com/bigbluebutton/bigbluebutton/blob/v2.0.x-release/labs/docker/coturn/turnserver.conf.tmpl#L2-L3) and by default FreeSWITCH uses UDP 16384-32768.

I tried to find documentation or examples for how to map large UDP port ranges with swarm, but I didn't find much that was useful. I think I found documentation on how to expose a range of ports (https://docs.docker.com/compose/compose-file/#short-syntax-1), but I'm not really sure if it directly translates to swarm.

[Felipe Cecagno]

Good catch! 

bigbluebutton_coturn.1.ri7kx6mam30x@swarm-node1.gnubila.fr    | udp send: Operation not permitted

It looks like something specific to Swarm, check if you need to explicitly allow a container to send/receive UDP. 

--
   Felipe Cecagno

(Sent from a mobile device)

[Felipe Cecagno]

We don't need a large range of UDP ports because coturn interfaces with the client through port 3478, and the large range is used for internal communication only between coturn and freeswitch (same thing for kurento). 

--
   Felipe Cecagno

(Sent from a mobile device)

[Felipe Cecagno]

Make sure freeswitch and coturn are in the same Swarm net. 

--
   Felipe Cecagno

(Sent from a mobile device)

[Jerome Revillard]

Yes they are... and port exposition is done the same way you did already in your original compose file.

[Jerome Revillard]

TCP and UDP is opened from everywhere:

[jerome:~] $ nc -v -u -z -w 3 meeting.gnubila.fr 3478
swarm-node1.gnubila.fr [46.105.47.47] 3478 (?) open

[jerome:~] $ telnet meeting.gnubila.fr 3478
Trying 46.105.47.47...
Connected to swarm-node1.gnubila.fr.
Escape character is '^]'.

Another idea ?

Le lundi 29 octobre 2018 17:32:02 UTC+1, Jerome Revillard a écrit :

Dear all,

I'm trying to install bbb using the following compose file: https://github.com/bigbluebutton/bigbluebutton/tree/master/labs/docker

I successfully installed it using a regular standalone docker system but I cannot make it working in a Swarm environment where Traefik is deployed independently (it's another stack)

The Swarm drawbacks are:

• depends_on is not recognised: this means that containers which cannot find the needed dependencies must exit so that swarm restart them. After some time, the all infrastructure should be stabilized.
• you cannot depend on an external env file so for the moment I hardcoded all the values inside my compose file.

Anyway, I managed to get the services started:

root@swarm-node1 ~ # docker stack ls
NAME           SERVICES
bigbluebutton  13
traefik        3

root@swarm-node1 ~ # docker stack services bigbluebutton
ID                  NAME                           MODE                REPLICAS            IMAGE                                   PORTS
7fujjacxv78u        bigbluebutton_bbb-html5        replicated          1/1                 fcecagno/bigbluebutton:bbb-html5        
8fpew1wp10zm        bigbluebutton_bbb-fsesl-akka   replicated          1/1                 fcecagno/bigbluebutton:bbb-fsesl-akka  
8vkp7hv9ifqk        bigbluebutton_redis            replicated          1/1                 redis:latest                           

[Jerome Revillard]

And from inside the Freeswitch container it's also ok:

root@27854368dd3b:/# nc -v -u -z -w 3 coturn 3478
Warning: inverse host lookup failed for 10.0.0.27: Unknown host
coturn [10.0.0.27] 3478 (?) open
root@27854368dd3b:/# telnet coturn 3478
Trying 10.0.0.27...
Connected to coturn.

Escape character is '^]'.

Best.

[Chad Pilkey]

I don't think FreeSWITCH wouldn't be communicating with coturn on 3478. I think 3478 is only for the client side. FreeSWITCH would be using one of the large UDP ports (49152-65535).

[Jerome Revillard]

Ok, anyway, I think that there is the same typo in the freeswitch entrypoint fro the stun url:

--value "stun:coturn" -> --value "external_rtp_ip=stun:coturn"

and

--value "stun:coturn" -> --value "external_sip_ip=stun:coturn"

Anyway, that does not fix my issue.. still:

bigbluebutton_coturn.1.c6ek4esb0teu@swarm-node1.gnubila.fr    | udp send: Operation not permitted

If I understand correctly, this is an issue for coturn not able to send UDP packet right ? but to whom ?

[Jerome Revillard]

Please find the full Freeswitch logs with some warnings inside: https://gist.github.com/jrevillard/a14c2109e117efc98b8c84bebc198fec

Could it be an ACL issue ?

[Jerome Revillard]

Could somebody help me solving this issue please ?

Best,

Jerome

[Felipe Cecagno]

Hi Jerome,

I think nobody never tried what you're doing, that's why the issue is new to everyone.

Could you share the instructions to replicate exactly your setup? This way, somebody in the group can easily reproduce the issue and try to find a solution.

I'd focus on the coturn issue to make sure it can communicate through UDP with FreeSWITCH and Kurento.

Best regards,

--

   Felipe Cecagno

--

[Jerome Revillard]

Dear Felipe, thanks for your help,

In order to reproduce my setup approximately, you need to have a docker swarm env (docker swarm init)

You can find my bbb compose file here: https://gist.github.com/jrevillard/d2100494528d2174db279bdaf4cbc863

You need to replace the <%= @... %> variable by what you usually setup in you env file. Also, please adapt the volume to you needs

Also, here is my Traefik compose: https://gist.github.com/jrevillard/31b29b594e56bf3645c450eff12d7421

Same, replace the variables and the volume part.

You first need to start traefik in order to create the needed network: docker stack deploy --compose-file '/path-to-the-traefik-compose/docker-compose.yml' traefik

Then the same for bbb: docker stack deploy --compose-file '/path-to-the-bbb-compose/docker-compose.yml' bigbluebutton

Hopefully, you will have everything ready for testing.

Don't hesitate to ask me if you have any question.

Best,

Jerome

[Jerome Revillard]

Dear all,

Is there anybody able to test please ? I'm still stuck but I really would like to have something working with this nice solution...

Best,

Jerome

[Chad Pilkey]

For a few reasons our container orchestration effort is mostly focused on Kubernetes at the moment. I don't really see much value in dockerizing BBB and then just running one copy of everything on the same node (VM, computer, etc). In my ideal picture we want to have multiple copies of the various components and scale up the cluster that way. As I said before though, we're still in the very early stages and a lot of the work requires application changes to go along with the dockerization of the components. We've been throwing around ideas for how we can approach it, but we haven't settled on a plan yet.