Re: [bigbluebutton-setup] docker bbb on proxmox VM

[Andrew Wells]

HI Leo,

I see several issues with your request and i thought i would reply as i use proxmox as well.

We spend way too much time on this forum on issues that she should be dealt with elsewhere.

I see that you got these instructions from github but are they official BBB install instructions?

If you were to use the method as outlined in this forum, using Ubuntu and not behind nat, i am pretty confident the issues would go away. I have installed it several times, and i would much prefer using Debian but i suck it up and use Ubuntu.

Why not use the official instructions as outlined in the BBB web page and/or the official github page and see if it doesnt resolve it or contact the people from that Github install.

If you are behind NAT, good luck again, as that should be dealt with elsewhere.

Day 75 last day --Hard75

'Change is inevitable! .' - Andy Frisella, Founder of Hard 75

Andrew T. Wells | Tel: Ext. 101 | Direct: 416 907-0100 |Mobile:  416 831-9363 |

e-mail andrew...@kooltel.com |

Kool Telecom Inc. 579 Kingston Road, Suite 130 Toronto, Ontario M4E 1R3 +1 4165150420 -phone +1 8885150420 -toll free +1 4165150419 -fax http://www.kooltel.com Information contained in this transmission is privileged and confidential. It is intended for the use of the individual or entity named above only. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copy of this communication is strictly prohibited. If you have received this communication in error, please immediately notify me.

From: Leo Leonidas <leonid....@freenet.de>
To: BigBlueButton-Setup <bigbluebu...@googlegroups.com>
Sent: 1/16/2023 1:55 AM
Subject: [bigbluebutton-setup] docker bbb on proxmox VM

Hello Guys,
I've spent a ridiculous amount of time on deploying BBB via docker on a Rocky Linux 9 VM inside Proxmox. I tried everything, every conf possibility which came to my mind.
I ordered a second public IP (94.x.x.B) on that Proxmox host (only for BBB).

Can someone please help to get that thing to work, I'm already getting nightmares. I have full controll over any machine and everyting, but no mather what I try, nothing makes BBB working in the end (best result so far with approach 1). I also do not prefer any of these approaches, I just would like to have a working BBB service.


Approach 2:
This approach which actually should be easier/ faster but is not working at all.
Quickly explained, I'm prerouting everything from 94.x.x.B to 10.2.1.2, which is the extra public IP i ordered and the IP of the VM which uses vmbr0. So I'm actually not going over my virualised opnsense/pfsense. On the bbb VM I've chosen greenlight: Y, https proxy: N, Coturn: N and opened the ports on the VMs firewall (same like in approach 1 below) and using a separate nginx docker for ssl.

```
auto enp0s31f6
iface enp0s31f6 inet static
  post-up iptables -t nat -A PREROUTING -d 94.x.x.B -j DNAT --to 10.2.1.2
```

```
auto vmbr0
iface vmbr0 inet static
        address  10.2.1.1/30
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.2.1.1/30' -o enp0s31f6 -j SNAT --to-source 94.x.x.B
        post-down iptables -t nat -D POSTROUTING -s '10.2.1.1/30' -o enp0s31f6 -j SNAT --to-source 94.x.x.B        
```

Freshly installed BigBlueButton but the "docker exec -it bbb-docker-greenlight-1 bundle exec rake conf:check" is showing:
```
Checking environment: Passed
Checking Connection: Failed
Error connecting to BigBlueButton server - Failed to open TCP connection to bbb.nuvariant-labs.com:443 (execution expired)
```
I can access the greenlight UI but getting "Invalid BigBlueButton Endpoint and Secret" after login. I got a prerouting and a postrouting, so really don't know what else is needed for bbb to finally work.
Any ideas?





Approach 1:
This approach has opnsense/pfsense involed:
I'm redirecting everything to opnsense/pfsense via virtual IP - IP Alias (10.10.10.3) + setup a 1:1 NAT (External IP: 10.10.10.3, Internal IP: 10.1.1.57) to route everything to the BBB VM.


Proxmox to opnsense/pfsense
```
auto enp0s31f6
iface enp0s31f6 inet static
  address 94.x.x.A
  netmask 255.255.255.192
  gateway 94.x.x.X
  mtu 1500
  post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
  up ip addr add 94.x.x.B/26 dev $IFACE label $IFACE:0
  down ip addr del 94.x.x.B/26 dev $IFACE label $IFACE:0
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -d 94.x.x.A -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -d 94.x.x.A -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to 10.10.10.3
```


opnsense/pfsense back out
```
auto vmbr10
iface vmbr10 inet static
  address 10.10.10.1/29
  bridge-ports none
  bridge-stp off
  bridge-fd 0
  post-up iptables -t nat -A POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j SNAT --to-source 94.x.x.A
  post-up iptables -t nat -A POSTROUTING -s '10.10.10.3/32' -o enp0s31f6 -j SNAT --to-source 94.x.x.B
  post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
```

As switching ssl certificates for BBB docker is undocumented/ seems not even planed although a lot of people are asking for it, I decided to deploy a separate nginx reverse proxy on the same VM as BBB. It is reachable from the internet and redirecting to https, TLS certs as well as proxying to bbb on port 48087.

```
server {
  listen 80;
  listen [::]:80;
  server_name bbb.my-domain.com;
  ...
  location / {
    return 302 https://$server_name$request_uri;
  }
}

server {
  listen 443 ssl http2 default_server;
  listen [::]:443 ssl http2 default_server;
  server_name bbb.my-domain.com;
  ssl_certificate /etc/ssl/live/my-domain.com.fullchain.pem;
  ssl_certificate_key /etc/ssl/live/my-domain.com.key;
  ssl_protocols TLSv1.3;
  ssl_prefer_server_ciphers off;
  access_log /var/log/nginx/my-domain.com.access.log;
  error_log /var/log/nginx/my-domain.com.error.log;

  location / {
    proxy_http_version 1.1;
    proxy_pass http://127.0.0.1:48087;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Upgrade $http_upgrade;
    #proxy_set_header Connection $connection_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_cache_bypass $http_upgrade;
  }
}

```
This Nginx container is deployed using the conf above, volume mounted the ssl certs and listens on 443 and 80.


On the bbb VM I've chosen greenlight: Y, https proxy: N, Coturn: N,
and opened the ports on the VMs firewall.
```
services: cockpit dhcpv6-client http https ssh
ports: 16384-32768/udp 48087/tcp

```
The web UI is seems to work fine, I can login, create rooms, update the settings, but the microphone and webcam sharing is not. I've executed the check, which looks good except the warning.



"docker exec -it bbb-docker-greenlight-1 bundle exec rake conf:check"
```
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.5.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.

Checking environment: Passed
Checking Connection: Passed
Checking Secret: Passed
Checking SMTP connection: Passed

```


When I click on Microphone ("How would you like to join the audio"), I get the following: Failure on call (reason=ICE error) (error 1004)



When I click on share webcam, it shows my webcam, camera and quality settings, I can see myself, but then on start sharing, I get: Media server failed to process request (error 2200)

I tried on Safari and Firefox.


Executing the curl command without dummy nic results in a timeout:
```
curl --trace-ascii - -k https://94.x.x.B:443/bigbluebutton/api

== Info: Trying 94.x.x.B:443...
== Info: connect to 94.x.x.B port 443 failed: Connection timed out
== Info: Failed to connect to 94.x.x.B port 443: Connection timed out
== Info: Closing connection 0
curl: (28) Failed to connect to 94.x.x.B port 443: Connection timed out



Could not make a WebSocket connection (error 1002)
Connection failure (ICE error 1107)

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-Setup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/dbe41977-cfe6-47a4-a990-835adceb4980n%40googlegroups.com.

[Leo Leonidas]

Yeah sure, was the official BBB docker GitHub (https://github.com/bigbluebutton/docker) at least I think it is official.

Anyway, I made it finally work on a Proxmox VM (Rocky Linux). Everything is working, all checks passing and of course the microphone and webcam. I gave up on the NAT approach, opnsense/pfsense 1:1 NAT + reflection etc. This killed me somewhere, I don't know, it's impossible! The solution was to assign the second public IP directly to my VM and everything was working, I'm so glad it finally works, documented it in here if anyone else is struggling with BBB docker on a VM: https://serverfault.com/questions/1121061/assigned-second-public-ip-to-vm-from-outside-not-reachable/1121266#1121266