Improve FreeSWITCH configuration files

[Fred Dixon]

Hi everyone,

We've published our FreeSWITCH configuration files for 2.3-dev here
 
 https://github.com/bigbluebutton/bigbluebutton/tree/develop/bbb-voice-conference/config/freeswitch

We're going to modify the build of FreeSWITCH for 2.3 to pull from this directory.  We would welcome improvements to the default configuration, such as

  https://github.com/bigbluebutton/bigbluebutton/issues/11743

and any cleanup/simplification that you think would improve the FreeSWITCH configuration for everyone.

Regards,... Fred

--

BigBlueButton Developer

http://bigbluebutton.org/

Like BigBlueButton?  Tweet us at @bigbluebutton

[Matthias Weiler]

Hi,

we followed the discussions about the tuning of audio parameters last
year an settled with this in our apply-config.sh:

echo " - Tune Audio Parameters"
cat <<HERE > /opt/freeswitch/conf/autoload_configs/opus.conf.xml
<configuration name="opus.conf">
<settings>
<param name="use-vbr" value="0"/>
<param name="use-dtx" value="0"/>
<param name="complexity" value="10"/>
<param name="packet-loss-percent" value="8"/>
<param name="keep-fec-enabled" value="1"/>
<param name="use-jb-lookahead" value="1"/>
<param name="advertise-useinbandfec" value="1"/>
<param name="maxaveragebitrate" value="256000"/>
</settings>
</configuration>
HERE

xmlstarlet ed --inplace -u
"/configuration/profiles/profile[@name='cdquality']/param[@name='interval']/@value"
-v 10 /opt/freeswitch/conf/autoload_configs/conference.conf.xml
xmlstarlet ed --inplace -u
"/configuration/profiles/profile[@name='cdquality']/param[@name='energy-level']/@value"
-v 80 /opt/freeswitch/conf/autoload_configs/conference.conf.xml

We didn't do much testing and comparison but those parameters currently
work fine for us.

Please consider this as input for the discussion, not an actual
suggestion to change the defaults.

Regards,
Matthias

[Martin Thomas Schrott]

Hi together,


as Matthias suggested, these parameters where the best we could achieve
last year after digging into it for several months. I think faircom did
a bit more research based on this - no idea if they changed those
parameters afterwards.


With this audio settings you get a clear sound with less crackles over
all ...

...and I also thought about posting them here - but was not sure, as
these improvements also increase the bandwith and cpu usage.


Would be great if somebody could measure how much more cpu and bandwith
would be needed for this settings.

With that information we could discuss if these  increased requirements
still would be okay and the settings could be the new default.

I know that really many systems are set up with those settings already -
so maybe it would make sense to take them and document how to lower if
the bandwith and cpu is to bad.


What do you think? Could somebody compare the usage of current default
and the optimized settings?


cheers

Martin

[Julien Gribonvald]

Hi folks,

it seems that we can set the maxaveragebitrate like that also:

yq w -i /usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml
conference-media-specs.OPUS.maxaveragebitrate "256000"

I think that somes other conf can be applied here to around.

On my side I tested only this param and it increased the audio details
but it's providing a noise on some micro. There is a recurrent "TAC"
after word pronunciation pauses, after some peoples are ok with that but
not all ! So I disabled it. Else maybe some other params should be added ?

Else thanks for sharing your conf, I will follow this topic.

Thanks

Julien Gribonvald

[Martin Thomas Schrott]

Julien,


in our tests it only worked when set in the opus.conf.xml - retry it
there :-)

and keep in mind you should also increase the interval and the
complexity - otherwise the codec won't be able to handle the higher
rates correctly I guess.


If you do not change to fixed rate (vbr=0) it also won't work properly.


cheers

Martin

[Gilles Pietri]

Hi,

Disagreeing here! Maybe we could set the info about it in the Wiki to
let know about choices there?
https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+And+The+Opus+Audio+Codec
is always a nice read anyway!

Let OPUS be happy, let it use VBR, and do not cap the bitrate that high.
If you mean to have nice quality while not wasting bandwith, set it to
192k, as anything higher won't be noticed, even if it's music.

I have no idea if browsers support dtx, I'd say disabling it is
reasonable, and it probably doesn't save much (energy-level does a lot
more, though it would save a bit of upload bandwith if caught at the
encoder level).

Complexity does not matter that much, on recent CPUs, and … to use the
JS client, that wouldn't make a difference, I think.

There are probably ways to be a bit more clever about the dynamics of
the energy level, I'm not sure what FreeSwitch has to offer in terms of
background noise detection, echo cancellation and what not, but those
are more and more expected by users with poor audio setups…

Regards,

Gilou

[Martin Thomas Schrott]

> Disagreeing here! Maybe we could set the info about it in the Wiki to
> let know about choices there?
> https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+And+The+Opus+Audio+Codec
> is always a nice read anyway!
>
> Let OPUS be happy, let it use VBR, and do not cap the bitrate that high.
> If you mean to have nice quality while not wasting bandwith, set it to
> 192k, as anything higher won't be noticed, even if it's music.

this defenitely is not true - some people (even the opus and freeswitch
config)  also state that even above 40k would not be recognizable - but
you can hear the difference between 192 and 256 for speech only.

You may not hear it on bad quality hardware though.

>
> I have no idea if browsers support dtx, I'd say disabling it is
> reasonable, and it probably doesn't save much (energy-level does a lot
> more, though it would save a bit of upload bandwith if caught at the
> encoder level).
>
> Complexity does not matter that much, on recent CPUs, and … to use the
> JS client, that wouldn't make a difference, I think.

if complexity does not matter in regards of cpu I would suggest do set
it to 10 as this makes a big difference already.

> There are probably ways to be a bit more clever about the dynamics of
> the energy level, I'm not sure what FreeSwitch has to offer in terms of
> background noise detection, echo cancellation and what not, but those
> are more and more expected by users with poor audio setups…

yes, agree here. If the energylevel would be more clever and only react
on the speaker it could improve the bandwith and experience. But! there
are a lot of meetings held with background music and they already have
problems with music being cut off - so this is not a good way to go in
gloobal manner. Make the background noice detection optional (or disable
option) would be good though.


All this said, at least the interval should be halfed, which does
improve a lot and complexity set to 10. - if ok in regards to the
bandwith and cpu usage of corse.

If kbit increase is not noticeable for most, maybe 128k would be a
better way to go - but we did not have good experience with this value.
All crackles only disappered above 228 eventually.

[Gilles Pietri]

Le 01/04/2021 à 13:37, Martin Thomas Schrott a écrit :
>> Let OPUS be happy, let it use VBR, and do not cap the bitrate that high.
>> If you mean to have nice quality while not wasting bandwith, set it to
>> 192k, as anything higher won't be noticed, even if it's music.
>
>
> this defenitely is not true - some people (even the opus and freeswitch
> config)  also state that even above 40k would not be recognizable - but
> you can hear the difference between 192 and 256 for speech only.
>
> You may not hear it on bad quality hardware though.

OK, but for OPUS specifically, the algorithm makes it rather
imperceptible once you've reached 192+ for stereo… And we're talking
about video-conference here. Most won't use high end monitoring speakers
or headphones, allowing to actually distinguish anything.

However, we did double blind (ABX ?) trials using monitoring speakers
and studio headphones on OPUS (not for BBB, but for mumble) to check if
192 vs 256 was actually relevant for stereo music. It is not. And even
if one could argue spectral analysis isn't the whole picture, it's quite
certain they will look exactly the same… At least for stereo @ 48kHz.

I wouldn't argue either with people willing to go even lower for voice
only, but I do have use-cases for proper music streaming over BBB, and
well, most of my use-cases also don't go hungry for ~100 Kbps, so
there's that.

Regards,
Gilou

[Martin Thomas Schrott]

Gilou


you are right, one should not expect, that users will have monitoring
hardware at hand ... but beside the clear audio we noticed a difference
in distortion / crackling during our tests, which was the main reason
for the high value.

Since then there where some Freeswitch updates and the crackling was
reduced also in the lower bandwith - so it now really may not make a big
difference anymore - did not test this in the latest versions.


Still, the main question will be - how high can we go with the values
and quality settings without increasing the bandwith and cpu usage too
much, so users with cheap hardware still will be surved well.


maybe others can also give some input so we could find a good way to go 
for the new defaults.


And beside that - I think Fred also was asking for non audio related
optimisation of the config files. Maybe some parts can be removed - or
optimized.

e.g. vbr = 1 seems to be the default - could this be leaved out at all?
...what about the other values that are set to default?


what about the sip config files / dialplans  ...


so far ...

cheers

Martin

[Julien Gribonvald]

I've just tested these params during a conf, and I confirm that improved
the audio of my collegue who have problems.

So I will put these config in production.

Julien Gribonvald

[sd...@distancelearning.cloud]

Mod_conference is taking everything and converting to L16 PCM and sampling it a 48,000, so whatever the edge it sending it is converted. And then reencoded in opus again at server.

Try all the different combinations of codec settings, and write them to a wav file, before and after mod_conference and do a blind test with users

I believe there is no difference in user experience.

The default settings in freeswitch work fine.

If you mess with the conference sampling rate, expect there to be CPU/Delay consequences form past testing.

Regards,
Stephen

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/f625d5e9-2542-fbd0-8657-08a149d15953%40recia.fr.

[Martin Thomas Schrott]

Stephen

I think nobody was trying to change the  samplerates - which as you
said  - would not make sense as long as FS is converting to wave anyway.

the interval setting in conference.conf.xml though does make a lot sense
to be changed, as this is the interval how often the sound is
transmitted, so 10 is double of 20 - a much higher quality therefore.

It does not change the samplingrate itself in my understanding


most effective would be to get fs working with opus instead - but think
this is not supported for now.


I don't think that the current default is the best way to go, as many
still struggle with audio problems and everybody who changed to these
settings - that where evaluated last year - is more happy than with the
originals.

everytime we had an meeting on a remote bbb and get back on our servers
somebody says: They had a bad audio quality - hadn't they?

Last week we set up a Testserver with 2.3 and in the first meeting
somebody said: The audio is worse than on 2.2 ...

So the audio setting really should be optimized for 2.3 and as far as
resource usage allows increased in it's defaults.

just my 2 ct.

cheers

Martin

[Martin Thomas Schrott]

Hi,


could you tell which ones you changed and what you let on the defaults?
just so we get an idea what helps ...

and does your user have good internet quality or bad bandwith?

cheers

Martin

[Ghazi TRIKI]

Hi,

I want to remind that some exhaustive effort has been put to improve the quality of audio conferences in BigBlueButton. The sum of the efforts are in my reply here https://groups.google.com/g/bigbluebutton-setup/c/3Y7VBllwpX0/m/41X9j8bvCAAJ

For the long version please take a seat and some 🍿 to understand the background of those changes, the choices and the facts.

Some history:

Between April 2019 and February 2020 I was working with a team in Saudi Arabia on a BigBlueButton project, the project was for online Quran teaching. One of their goals was to improve the quality of the audio conference, for the VoIP Network engineer was too much used with Cisco devices and the configurations we tried slightly improved the audio quality, but nothing really consistent that we I can share now.

Then during the BigBlueButton Developer Community Hackathon 2020 organised by Fairkom and Wechange, the team that was gathered worked on improving the audio quality. You can find more about the hackathon in the links below:

https://wechange.de/cms/2020/08/04/bbb-developers-community-hackathon-2020/

https://www.fairkom.eu/summer-hackathon

They did a great work improving the audio quality, however few things were missing, that I will detail later.

One month after the Hackathon, we were asked again at RIADVICE to work on improving the audio quality of the audio conference, the request came from Mainz University. Based on the great job made during the hackathon we evaluated that it was worth taking a closer look.

Technical analysis:

First, when we started, we had a meeting with Simon Millard, the managing director of Noakes from UK. Naokes is the name of a closed source SIP Trunking server; I think it does more than that. He kindly shared his ears and listened to the samples we provided from FreeSwitch to understand what we could cover and which areas that we cannot.

Multiple factors must be took in consideration to understand the final audio quality:

1- Transcoding:

    - Input encoding.

    - Input decoding.

    - L16 - audio linearisation (https://tools.ietf.org/html/rfc2586) - mentioned by Stephen above https://groups.google.com/g/bigbluebutton-dev/c/Fy1_2wBa__M/m/svcyfZG3BgAJ

2 - Network packet loss

3 - Audio muxing (that happens during linearisation)

4 - The muxed output encoding (can be considered as part of the transcoding)

5 - Jitter buffering

Few samples with a pair an amazing ears and it was evident that sure that transcoding was affecting the quality. Same for packet loss, but we can't do much because of the of the opus codec itself.

Implementation:

We also configured a test server with Fairkom settings, we found that it was slightly better but the audio was still in audio. Same for some configuration, some bitrates are obsolete because too high.

We went through FreeSwitch OPUS documentation https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+And+The+Opus+Audio+Codec

The codecs list of FreeSwitch are in the page https://freeswitch.org/confluence/display/FREESWITCH/Audio+Codecs

It is very important to understand the Opus Codec https://tools.ietf.org/html/rfc6716 and how it is doing packet loss resilience https://tools.ietf.org/html/rfc6716#section-2.1.6

Now we dealt with the codecs, the other important factor is the jitter buffer and its configuration in FreeSwitch https://freeswitch.org/confluence/display/FREESWITCH/JitterBuffer, Cisco explain it well https://www.ciscopress.com/articles/article.asp?p=357102

By going through the jitter configuration you understand that more factors are evolved: network type, network condition, codec, distance from the server...

Production facts:

Our configuration was initially deployed in Germany in Mainz University, then across all the German universities that adopted BigBlueButton through ZKI https://www.zki.de/english/

Mainz number are in this link https://www.cms.hu-berlin.de/en/zki-workshop-bbb?set_language=en

Our former Saudi customer highly welcomed the changes, and they had less complaints about audio quality.

We have run more than 500k meetings in our hosting with that improved configuration during and still having it.

Q&A:

Q1: is that configuration creating a delay in the recordings?

A1: We did not notice that.

Q2: Does it use more network traffic?

A2: Yes not always because of the VBR, the network consumption can go down by 30% or higher by 30% for the audio packets.

Q3: Did it remove audio crackling?

A3: No, audio crackling has other different reasons.

-------------------------------

Thank you for reading my reply.

Ghazi Triki

[Martin Thomas Schrott]

Ghazi,

thanks for your input.

the settings faircom extendet where the ones we evaluated last summer - during the hackaton (where we also where involved) this only was taken from our testing results. After the hackaton they worked on it further ... But I guess only to make it stereo as it seems - and switched back to vbr.

the "      <param name="negotiate-bitrate" value="1" />" is no good idea at all - a lot of devices/browsers cannot handle this ... we hd sampling problems when changing this setting.

Also the interval of 20 is worse than an interval of 10 in conference.conf.xml.

If the jitterbuffer works better - this may be interesting - but in my opinion the current buffer settings are quite well and I do not see why to change them.

I understand that vbr  should kept ...

I suggest increasing interval to 10 and complexity to 10.

in regards to the maxaveragebitrate we could increase it - but should test if bad quality lines can handle all that...

I could not see any further important changes in what faircom or you did.

more input and ideas very welcome!

cheers

Martin

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/962e1903-ad78-4813-8b49-d7f0071b4385n%40googlegroups.com.

[Lorenz Schori]

Hi Fred,

On Wed, 31 Mar 2021 22:34:42 -0300
Fred Dixon <ffd...@gmail.com> wrote:

> [...]

> We're going to modify the build of FreeSWITCH for 2.3 to pull from
> this directory. We would welcome improvements to the default
> configuration, such as
>
> https://github.com/bigbluebutton/bigbluebutton/issues/11743
>
> and any cleanup/simplification that you think would improve the
> FreeSWITCH configuration for everyone.

Thanks a lot. I've been working on a minimal BBB configuration for
quite some time over here: https://github.com/znerol/bbb-minimal-fs

I'll be happy to file PRs in order to get us to something like that
step-by-step. Would be cool if we had some github tag in order to track
those efforts. Starting points for everybody who is interested in
helping out with that work:

* https://github.com/bigbluebutton/bigbluebutton/pull/11846
* https://github.com/bigbluebutton/bigbluebutton/pull/11864

Thanks & Cheers,
Lorenz

[Julien Gribonvald]

Hi folks,

I've tested a bit some change, and if you need to accept a lot of users, my result was that we should stay on default conf audio !

Other change are only needed if you want to improve considerably the audio quality, but you will loose on users capacity per servers and bandwidth use.

For that I've applyed the change (the most advanced and corresponding to OPUS docs) provided in attached patch file. The resault was that my platform increased the bandwidth use from around 1GB to 1,5 GB with around 8K users. So now I'm back to default conf expect on complexity to 10 and packet-loss-percent to 8.

On an other side I think that the presentation bitrate also could be reduced, it can help a lot of users !

On my side these change are good on my test:

yq w -i /usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml 'conference-media-specs.H264.tias_content' "300000"
yq w -i /usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml 'conference-media-specs.H264.as_content' "300"
yq w -i /usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml 'conference-media-specs.VP8.tias_content' "250000"
yq w -i /usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml 'conference-media-specs.VP8.as_content' "250"
yq w -i /usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml 'conference-media-specs.OPUS.maxaveragebitrate' "30000"
yq w -i /usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml 'conference-media-specs.OPUS.maxplaybackrate' "48000"

It's an acceptable configuration to accept a maximum of users.

Today with 42 BBB (with default conf) we up to 11,4K users with bbb 2.2.33 (with whiteboard patch). Some servers up to 400 users and one with 490 (it stayed around 20 min and reduced to 460 a bit after during 30 min).

The only thing is that only scalelite can provide stats when servers are up to 380 users, and user doesn't seem to have bad feedback if they don't open too much micro and video.

Thanks,

Julien Gribonvald

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/ad58f3f3-9e9d-c94e-e4fa-23afb7ddfdba%40mtsonline.at.