Audio crackling & Video problems

[Florian Erfurth]

Hello,

the users are complaining that since last week we have issues with audio (crackling) and video ("doesn't work" ™).

I'm trying to find out the cause.

In Kurento-logs I found a lot of entries like:

2021-01-26T08:23:58,285676 28618 0x00007f844e7fc700 warning kmsutils                  kmsutils.c:506 gap_detection_probe() <kmswebrtcendpoint0_kmsagnosticbin2-0:sink>  GAP of 0 ms at PTS=0:04:19.986141172 (packet loss?); will request a new keyframe

Is this a sign of network problems? Can I be sure it is not caused by the server itself?

If you have an idea where (in logs) I should look and which keyword I should try to look for, that would be great. Especially the sound crackling has the biggest impact for users and I urgently need to fix it.

Best regards

Floh

[ICT Cloud]

Hi,

Yes this appears to be a poor packet flow. Check your packets for loss.

~b10

[Fred Dixon]

Hi Florian,

> the users are complaining that since last week we have issues with audio (crackling) and video ("doesn't work" ™).

Was the audio working well before then?  Did anything change on your BigBlueButton server?  Can you share more details on your setup.  See

  https://docs.bigbluebutton.org/2.2/getting-help.html#help-installing-or-running-bigbluebutton

for information to share to make it easier for others to help.

Regards,... Fred

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-Setup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/c7f96d4f-0a57-4177-accd-8e8e0bf16393n%40googlegroups.com.

--

BigBlueButton Developer

http://bigbluebutton.org/

Like BigBlueButton?  Tweet us at @bigbluebutton

[basisbit]

Are you running BigBlueButton on a dedicated server, are you monitoring CPU usage as well as your network traffic usage?

[Florian Erfurth]

Hello everybody,

thank you very much for your replies. Sorry for missing informations, I completely forgot about that! :-O

Currently all servers are using BigBlueButton Server 2.2.31 (2307). Half of them were newly installed (within the last 2 weeks) and half of them were updated from 2.2.27 (or older, I don't remember). Some of the new and updated servers are affected, so not sure if its caused by the newer version. The servers running older versions (between 2.2.21 and 2.2.27) didn't have the issues, so, yes, the audio was working well before.

I updated by 'apt update && apt dist-upgrade'.

@b10: How do I check for packet loss?

There are also such entries, which I believe are related to the "packet loss" entries, so the packets arrived too late, am I right?

kurento-media-server log:

2021-01-27T08:29:14,850256 28002 0x00007f75f9ffb700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 29355, expected 29361)
2021-01-27T08:29:14,887525 28002 0x00007f75f9ffb700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 29356, expected 29361)
2021-01-27T08:29:14,925583 28002 0x00007f75f9ffb700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 29357, expected 29361)
2021-01-27T08:29:14,964223 28002 0x00007f75f9ffb700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 29358, expected 29361)
2021-01-27T08:29:15,011520 28002 0x00007f75f9ffb700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 29359, expected 29364)
2021-01-27T08:29:15,049754 28002 0x00007f75f9ffb700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 29361, expected 29364)
2021-01-27T08:29:15,055935 28002 0x00007f75f9ffb700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 29362, expected 29364)
2021-01-27T08:29:15,283208 28002 0x00007f75f9ffb700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 29361, expected 29367)
2021-01-27T08:29:15,283321 28002 0x00007f75f9ffb700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 29362, expected 29367)

I double-checked the CPU usage, it was around 30% (checked with htop) and traffic used lesser than 5% of available bandwidth. Unfortunately, I didn't watch while the issue was occurring.

It's not a dedicated server, but a root server with dedicated cpu & ram. They have all been working well for more than a year now. They're all with 4 core cpu and 16GB RAM.

root@conference18:~# bbb-conf --check

BigBlueButton Server 2.2.31 (2307)
                    Kernel version: 4.4.0-200-generic
                      Distribution: Ubuntu 16.04.7 LTS (64-bit)
                            Memory: 16432 MB
                         CPU cores: 4

/usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties (bbb-web)
       bigbluebutton.web.serverURL: https://conference18.xxxxxxxxx.de
                defaultGuestPolicy: ALWAYS_ACCEPT
                 svgImagesRequired: true

/etc/nginx/sites-available/bigbluebutton (nginx)
                       server_name: conference18.xxxxxxxxx.de
                              port: 80, [::]:80
                              port: 443 ssl

/opt/freeswitch/etc/freeswitch/vars.xml (FreeSWITCH)
                       local_ip_v4: xxxxxxxxx
                   external_rtp_ip: xxxxxxxxx
                   external_sip_ip: xxxxxxxxx

/opt/freeswitch/etc/freeswitch/sip_profiles/external.xml (FreeSWITCH)
                        ext-rtp-ip: $${local_ip_v4}
                        ext-sip-ip: $${local_ip_v4}
                        ws-binding: xxxxxxxxx:5066
                       wss-binding: xxxxxxxxx:7443

/usr/local/bigbluebutton/core/scripts/bigbluebutton.yml (record and playback)
                     playback_host: conference18.xxxxxxxxx.de
                 playback_protocol: https
                            ffmpeg: 4.2.4-1ubuntu0.1bbb1~16.04.1

/etc/bigbluebutton/nginx/sip.nginx (sip.nginx)
                        proxy_pass: xxxxxxxxx
                          protocol: http

/usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml (Kurento SFU)
                        kurento.ip: xxxxxxxxx
                       kurento.url: ws://127.0.0.1:8888/kurento
                    kurento.sip_ip: xxxxxxxxx
                    localIpAddress: xxxxxxxxx
               recordScreenSharing: true
                     recordWebcams: true
                  codec_video_main: VP8
               codec_video_content: VP8

/usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml (HTML5 client)
                             build: 1114
                        kurentoUrl: wss://conference18.xxxxxxxxx.de/bbb-webrtc-sfu
                  enableListenOnly: true
                    sipjsHackViaWs: true

/usr/share/bbb-web/WEB-INF/classes/spring/turn-stun-servers.xml (STUN Server)
                              stun:bbbturn.xxxxxxxxx.de


# Potential problems described below

# Warning: No firewall detected.  Recommend using setting up a firewall for your server
#
#     http://docs.bigbluebutton.org/2.2/troubleshooting.html#freeswitch-using-default-stun-server
#

Yes I'm using own bbbturn-server (followed the bbb-docs).

Best regards

Floh

--

You received this message because you are subscribed to the Google Groups "BigBlueButton-Setup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-s...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/14100e86-63b3-4058-a520-eacc40b7245fn%40googlegroups.com.

[basisbit]

Which server provider is hosting those VMs?

[bayden10]

Hi,

One method is on the host where the possible packet issue is you can run tcpdump at the time while then running a session and take note of the time when you here the crackling then analyze the packets with wireshark.

This also sounds like a possible NAT 1:1 issue with your VM/VM-Switch/Router or what ever you are using for NAT 1:1 . . One way to check is run curl https://ipinfo.io/ip on your TURN and BBB hosts to make sure this is working properly for you with forward DNS and rDNS . The public facing IP should be exposed.

You should also run some monitoring as this is best for troubleshooting these types of issues that might be related to resources. The all-in-one is pretty quick to setup and useful  https://bigbluebutton-exporter.greenstatic.dev/installation/all_in_one_monitoring_stack/

~b10

[Florian Erfurth]

Hello Stefan, hello b10

the server provider is netcup. What do you think about that?

There is no NAT, so all servers own an external IP-Address.

Do I need to install all in one monitoring on each servers? I doublechecked the ressource graph in server management of my server provider in the particular part where audio were crackling. No spikes or odd thing in the graphs there.

Best regards

Floh

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/604fab17-aea1-4483-8082-362cfd595bben%40googlegroups.com.