Any updates on the "crackly" freeswitch audio issue? Has it even made into the "top priorities" list after BBB 2.2 got out of beta?

[Alex]

Hello!

This is related to this open issue

https://github.com/bigbluebutton/bigbluebutton/issues/7007

After 2.2 got officially released I made a fresh install and tested it. Unfortunately it looks like it is still there.

Here is a recording I made on the official demo server:

https://drive.google.com/file/d/1KQ2BhqQ6MMVJd3OM2q7mTnHQ0D2chZKM/view

The same was on my server.

I know the BBB guys tried to fix it in 2019 tweaking some freeswitch settings (kudos to them!).  Any update?

Kind regards,

Alex.

[Roberto Becerra]

Hi Alex, 

We are also facing this issue. 

For now we were making some changes to some Freeswitch files, such as the opus configuration to inccreas the max bitrate and complexity, and actually having less "jitter related control" made things better. 

We are still not quite there with perfect music-ready settings, but much better than a default configuration. 

What is your experience, have you custom configured something?

Best!

Roberto

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/a7a8c6d0-2bb6-40f3-9c23-94b9b4d3b48e%40googlegroups.com.

[Alex]

Hello Roberto!

I haven't tested any other manual changes only the ones the BBB team posted in that thread back then.

Do you mind sharing your custom freeswitch configuration via pastebin or just uploading conf files somewhere.

We don't need perfect music-ready settings, so perhaps your changes would work out really great for us.

Yours,

Alex

On Thursday, March 26, 2020 at 8:33:09 AM UTC+3, Roberto Becerra wrote:

Hi Alex, 

We are also facing this issue. 

For now we were making some changes to some Freeswitch files, such as the opus configuration to inccreas the max bitrate and complexity, and actually having less "jitter related control" made things better. 

We are still not quite there with perfect music-ready settings, but much better than a default configuration. 

What is your experience, have you custom configured something?

Best!

Roberto

On Thu, 26 Mar 2020 at 01:27, Alex <fedotov...@gmail.com> wrote:

Hello!

This is related to this open issue

https://github.com/bigbluebutton/bigbluebutton/issues/7007

After 2.2 got officially released I made a fresh install and tested it. Unfortunately it looks like it is still there.

Here is a recording I made on the official demo server:

https://drive.google.com/file/d/1KQ2BhqQ6MMVJd3OM2q7mTnHQ0D2chZKM/view

The same was on my server.

I know the BBB guys tried to fix it in 2019 tweaking some freeswitch settings (kudos to them!).  Any update?

Kind regards,

Alex.

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigblueb...@googlegroups.com.

[Łukasz Fedorko]

Hi Roberto, can you share any info on Opus tweaks? Cracks are driving us nuts 🙂 Please share anything you can, we'd like to boost bitrate at least.

Regards,
Lukasz

[Roberto Becerra]

Hi there, 

Sure, modified files are in this download link:

https://mab.to/Fw870BJ3T

Though you should use at your own risk, and by no means do we think they are correct, they might be outright nonsense, but they did improve our crackling. What these file modifications do is basically change from cdquality to video-mcu-stereo, increase max bitrate, disable most of  jitter handling, and decrease energy-level for noise suppression. 

Also very important for us was to manually modify flags/config in browsers to disable AGC and AEC, and noise suppression. Best results were experienced in Firefox, where we could get nice stereo signal. 

Now we are in the process of trying to create an extension for these browsers in order to disable those settings, but we are still working on it. If you had any experiences on that I would love to read!

I hope this works for you!

Roberto

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/5c8500d4-c434-43c0-9249-91f24e1c5a66%40googlegroups.com.

[sd...@distancelearning.cloud]

To disable AEC/noise/agc  just need to set constraints in getusermedia, can add this to bbb webrtc_bridge and sip.js code.

Also there is a hack to get chrome stereo going with sdp.

 

Se also can send 2ch binaural audio end to end.

 

Regards,

Stephen.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/CAAeopRh%3DZvKT8QAN4866HMzzgXq0tmVGfrQqk-BSVFALsNfXWA%40mail.gmail.com.

[Alex]

Roberto, thanx so much for this!

We used your settings  with some small changes ( cdquality profile instead of video-mcu-stereo, , we left vbr on, we didn't add "<param name="maxaveragebitrate" value=" and we set <param name="packet-loss-percent" value= to 5 instead of 0)  and it sounds so much better than the default settings!

I bet someone who knows a great deal about freeswitch could tweak it even more

On Thursday, March 26, 2020 at 7:44:07 PM UTC+3, Roberto Becerra wrote:

Hi there, 

Sure, modified files are in this download link:

https://mab.to/Fw870BJ3T

Though you should use at your own risk, and by no means do we think they are correct, they might be outright nonsense, but they did improve our crackling. What these file modifications do is basically change from cdquality to video-mcu-stereo, increase max bitrate, disable most of  jitter handling, and decrease energy-level for noise suppression. 

Also very important for us was to manually modify flags/config in browsers to disable AGC and AEC, and noise suppression. Best results were experienced in Firefox, where we could get nice stereo signal. 

Now we are in the process of trying to create an extension for these browsers in order to disable those settings, but we are still working on it. If you had any experiences on that I would love to read!

I hope this works for you!

Roberto

On Thu, 26 Mar 2020 at 12:16, Łukasz Fedorko <loo...@gmail.com> wrote:

Hi Roberto, can you share any info on Opus tweaks? Cracks are driving us nuts 🙂 Please share anything you can, we'd like to boost bitrate at least.

Regards,
Lukasz

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigblueb...@googlegroups.com.

[Roberto Becerra]

Hi Stephen

Wow, thanks a lot for the tip, you really got me started pulling the thread. Though I would like to further ask for your help if you don't mind. 

I went and found were in those files I could add constraints to getusermedia and found:

./bbb_webrtc_bridge_sip.js:266:  SIP.WebRTC.getUserMedia({audio:true, video:false}, getUserMicMediaSuccess, getUserMicMediaFailure);

 where I replaced 'true' by an object with autoGainControl, etc. 

and 

./sip.js.save:11775:            SIP.WebRTC.getUserMedia(constraints)

Where i replaced (for the sake of testing) constraints for an object with the same AGC etc. 

But still no success, do you have any word on this?

Thanks a lot! 

Roberto 

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/382801d60410%246adf3850%24409da8f0%24%40distancelearning.cloud.

[Flo Gee]

Hi Roberto,

could you share the changes again?

the download link has unfortunately already expired

Regards,

Florian

Am Donnerstag, 26. März 2020 17:44:07 UTC+1 schrieb Roberto Becerra:

Hi there, 

Sure, modified files are in this download link:

https://mab.to/Fw870BJ3T

Though you should use at your own risk, and by no means do we think they are correct, they might be outright nonsense, but they did improve our crackling. What these file modifications do is basically change from cdquality to video-mcu-stereo, increase max bitrate, disable most of  jitter handling, and decrease energy-level for noise suppression. 

Also very important for us was to manually modify flags/config in browsers to disable AGC and AEC, and noise suppression. Best results were experienced in Firefox, where we could get nice stereo signal. 

Now we are in the process of trying to create an extension for these browsers in order to disable those settings, but we are still working on it. If you had any experiences on that I would love to read!

I hope this works for you!

Roberto

On Thu, 26 Mar 2020 at 12:16, Łukasz Fedorko <loo...@gmail.com> wrote:

Hi Roberto, can you share any info on Opus tweaks? Cracks are driving us nuts 🙂 Please share anything you can, we'd like to boost bitrate at least.

Regards,
Lukasz

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigblueb...@googlegroups.com.

[Roberto Becerra]

sure, here it is, same disclaimer as previously!  ;)

https://mab.to/yaKmsq9W8

To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/5cb127a9-07e7-4f6d-974f-61c6e575ccd7%40googlegroups.com.

[Dave Fencik]

Hi Roberto,

I recently joined this community and missed your download link before it expired.  Would you be so kind as to repost it again?  I would greatly appreciate it!

Dave

On Thursday, March 26, 2020 at 11:44:07 AM UTC-5, Roberto Becerra wrote:

Hi there, 

Sure, modified files are in this download link:

https://mab.to/Fw870BJ3T

Though you should use at your own risk, and by no means do we think they are correct, they might be outright nonsense, but they did improve our crackling. What these file modifications do is basically change from cdquality to video-mcu-stereo, increase max bitrate, disable most of  jitter handling, and decrease energy-level for noise suppression. 

Also very important for us was to manually modify flags/config in browsers to disable AGC and AEC, and noise suppression. Best results were experienced in Firefox, where we could get nice stereo signal. 

Now we are in the process of trying to create an extension for these browsers in order to disable those settings, but we are still working on it. If you had any experiences on that I would love to read!

I hope this works for you!

Roberto

On Thu, 26 Mar 2020 at 12:16, Łukasz Fedorko <loo...@gmail.com> wrote:

Hi Roberto, can you share any info on Opus tweaks? Cracks are driving us nuts 🙂 Please share anything you can, we'd like to boost bitrate at least.

Regards,
Lukasz

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigblueb...@googlegroups.com.

[Roberto Becerra]

Hi Dave, 

Someone very nicely put it in a google drive :)

https://drive.google.com/file/d/1d526SnjnTAwS9QfHTyoEHyquptegN364/view

To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/8ebbdaa6-aec8-425e-a8ab-4244bf0480a0%40googlegroups.com.

[Ilya Drz.]

Hi everybody!

We are also struggling with the "crackly" noise problem.

I wonder if anyone has tried setting rtp timer to "none" in Freeswitch Sofia configs?

Here is a couple of links with more info:

https://lists.freeswitch.org/pipermail/freeswitch-users/2017-August/127378.html

https://lists.freeswitch.org/pipermail/freeswitch-users/2013-December/101579.html

It might help but can cause more problems if I understand right.

Any experience with that?

[Rafał Kuszneruk]

Hello Roberto, 

You mentioned you are working on browser extensions to uplift bbb audio on client side. Have you succeeded in it? 

Rafał Kuszneruk

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/CAAeopRgjbKMD0-aqyXrDtY-zLiXrwrg%3DbGQKaxBk_403oD7ztA%40mail.gmail.com.

[ErdaL Bayram]

I am interested to know if anybody has any success with Audio/Music quality on BBB. 

It is still a struggle for us, and couldn't find a fix yet.

Thanks.

[Martin Thomas Schrott]

This should go into setup group please.

and yes, I think all problems are solved in the meantime.

cheers

Martin

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/9eb00886-703b-4dbd-8d75-e69d6832437do%40googlegroups.com.

[BM]

We are also struggling with this issue since a week. Everything was working fine few days before. But now all the sessions are getting crackling noise. Its really annoying for students:(.
Also we checked with multiple parameters like changing energy level but no exact solution is found for the same till date.

In terms of video BBB is best but for audio we really have to improve a lot in comparison with skype n zoom

Pl help on this at topic

[Fred Dixon]

Hi,

> We are also struggling with this issue since a week. Everything was working fine few days before. But now all the sessions are getting crackling noise. 

With one of the users experiencing crackling noise, can you do a test with 

  https://demo.bigbluebutton.org/

and let us know how the audio quality sounds there.

Regards,... Fred

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/50a3531b-10f3-44ff-86ea-174ae22d8d91o%40googlegroups.com.

--

BigBlueButton Developer

http://bigbluebutton.org/

Like BigBlueButton?  Tweet us at @bigbluebutton

[fairkom Gesellschaft]

When playing around with OPUS codec settings I did not really achieve better audio quality. We do not have any crackling noise, though.

Unfortunately, modified files  by Roberto are not available anymore in his download link: https://mab.to/Fw870BJ3T

What I tried in the cdquality section of /opt/freeswitch/conf/autoload_configs/conference.conf.xml

<param name="rate" value="160000"/>

<param name="interval" value="20"/>

<param name="channels" value="2"/>

<param name="energy-level" value="30"/>

and in /usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml

tias_content: "2500000"

as_content: "2500"

maxaveragebitrate: "80000"

maxplaybackrate: "160000"

was:

tias_content: "1500000"

as_content: "1500"

maxaveragebitrate: "30000"

maxplaybackrate: "48000"

--Roland

[Kristoff Scherer]

we are facing he same problems.

Rebooting the server helped but is not the final solution

[Martin Thomas Schrott]

Hi Roland and others,

for us the best audio quality (far better tan zoom and others) can be achieved with the settings below.

keep in mind that virtual servers may cause problems with crackling noises. with bare metal servers there are no issues at all.

cheers

Martin

-- settings --

https://pastebin.com/NCznbRf3

and in /opt/freeswitch/conf/autoload_configs/conference.conf.xml

´´´

    <profile name="cdquality">
      <param name="domain" value="$${domain}"/>
      <param name="rate" value="48000"/>
      <param name="interval" value="10"/>
      <param name="energy-level" value="60"/>

´´´

--

You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/0c3c3b7e-e2ea-43d4-94a7-8af07d8d1859o%40googlegroups.com.

[Fred Dixon]

Thanks Martin for sharing this!

We would be interested if others could try out Martin's proposed changes and share the before/after experience of the audio for your users.

Ideally, you could have a group of people that are experiencing audio issues, and then with this group live

1. Restart BigBlueButton and have them rejoin and confirm they are still experiencing the same issues 

2. Apply Martin's changes, restart BigBlueButton, have them rejoin, and see if the audio improves

Regards,... Fred

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/5742d5cb-bb1a-5d59-9f91-74b4ee669dbb%40mtsonline.at.

[Martin Thomas Schrott]

> Thanks Martin for sharing this!
>

Thanks Fred, you are welcome. we already shared the settings in the
past, but due to the run on bbb this topic was not very relevant to most
of the people at this time ;-)

just to add as a note, for users / areas, where internet lines are slow,
the maxaveragebitrate may be to high, if automatic reduction does not
work (to be tested). So this really is for best audio quality. With very
bad connections the default settings should be quite ok.

But with this settings we also had much less crackling on the virtual
servers, before we switched to bare metal. So it always should be worth
to try this configuration.


and normally the rate should be lowered anyway automatically on bad
connections, just not sure if this works properly, would be great if
someone could test this with slow lines and report if the improved
settings are also good for bad connections, if this would be the case it
may be considdered to have a deeper look if it could be used even as
default.


due to freeswitch / opus docs, speech should not need more than 64k and
settings above should not make any difference, but they defenitely do.


looking forward to any reports or questions ... really was a long way in
spring with richard, riccardo, Alex and others to come to this solution
/ settings. :-)


cheers

Martin

[Jens Elkner]

On Sat, Aug 22, 2020 at 10:29:32PM +0200, Martin Thomas Schrott wrote:
Hi Thomas,
...

> But with this settings we also had much less crackling on the virtual
> servers, before we switched to bare metal. So it always should be worth to
> try this configuration.

Can you define "virtual servers" in your case? qemu? Xen? esxi? LXC? ...

Thanx,
jel.
--
Otto-von-Guericke University http://www.cs.uni-magdeburg.de/
Department of Computer Science Geb. 29 R 027, Universitaetsplatz 2
39106 Magdeburg, Germany Tel: +49 391 67 52768

[Martin Thomas Schrott]

Hallo Jens,


we used vmware / esxi and also tried KVM.

With the settings the crackling could be minimized, but the audio/videeo
sync problems still happened from time to time. With bare metal all
issues disappeared.


cheers

Martin

[DistanceLearning.cloud]

Have you benchmarked the cpu used by mod_conference when setting interval to 10 from the original 20.   I think mod_conference has to sample twice as much and you would see higher cpu usage. Just a guess.

Also does the crackly noise appear in the recordings?  If it doesnt, then you can capture the write raw data of a user and see if if the opus encoding doing this, or is its over the wire, and releated to client decoding?

regards

Stephen

[bayden10]

I would like to chime in here just to point out that this would be more than likely a Network Performance issue along with your TURN server proximity placement based on the participants Geo-location who are complaining

A few suggestions

Bitrate should auto reduce based on what performance level is available at the client side. I believe WebRTC in browsers knows how to do this quite effectively..

You usually don't need to specifically change resolutions, as that's also something that gets calculated and done dynamically based on the available bitrate anyway.

Place your servers (TURN and BBB servers) as close as possible to your users. This will reduce RTT and usually will also improve other metrics such as jitter and packet loss. 

You will need to decide how to allocate your servers to users based on location while doing that, which isn't always simple

At certain bitrates, depending on your scenario and layout of the display, it makes more sense to just reduce the amount of video streams and revert to audio.

You can try to add a network test to run before a session is started for the participants.

Something interesting to add is a bot to help people troubleshoot quality issues during a session, this would be useful in many cases but this would be up to the BBB team & community to develop this based on it's value.

Just some thoughts,

~b10

[Martin Thomas Schrott]

Hi Stephen,

> Have you benchmarked the cpu used by mod_conference when setting
> interval to 10 from the original 20.   I think mod_conference has to
> sample twice as much and you would see higher cpu usage. Just a guess.

not sure if this was for us ... but we did not notice a difference in
cpu load, but we never had high cpu load anyway, so we did not take
special care.

cheers

Martin

[Kenren Taisho]

Dear all,

Good morning. I am running BBB on a bare-metal configuration and server spec hardware.

I tried Martin's config tweak and honestly, we felt improvement in the audio, especially on short meetings.

However, we still experience the following oddities:

1. audio cracklings

2. audio cuts (I don't know how to describe it correctly, but I hope you get the idea)

3. audio level issues (when a  person is speaking, the volume/level of his audio becomes inconsistent)

A little bit of audio/video sync issues was noticed too.

For now, we will stick with Martin's tweak as we concurred that it's still better than the stock config.

We really hope the audio quality will be improved on the coming releases. 

Kind regards

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/fc42f862-f261-2f2a-b8b5-a23f85ad24bb%40mtsonline.at.

[Martin Thomas Schrott]

Hi Kenren,


sounds strange, we do not have any issues with the latest versions ...

are you on the latest bbb version?

what hardware are you using?

cpus?

memory?

do you run any other applications beside bbb?


cheers

Martin

[Kenren Taisho]

Hi Martin,

Yes, I am on the latest 2.2.22 version.

Supermicro server

Single CPU Intel Xeon E5-1620 (8 core)

32G RAM

I don't run any application besides BBB.

In our experience, audio issues usually occur in meetings that are more than an hour.

However, as I said in my previous email - we've decided to use your config adjusting only the energy level a bit lower as our team concurred that the quality is better than the default configuration.

Thank you very much for sharing your config.

Kind regards

[Martin Thomas Schrott]

Hi,

the energylevel is the level of the background (or whole) volume that must be reached to activate the audio stream of the user.

so if you set it lower the stream may be active all the time, even if the user does not want to say something. Just that you keep this in mind.

you could set it to 0 to see if some noise is caused by the on/off switching.  But it also causes, that not speaking users may appear in the "is speaking ..." toast message. That is why we did not lower it further. users where confused as the messages who was speaking did not match anymore. :-)

if you still have cracklings, your clock seems to have a problem. Freeswitch is quite sensitive so try to change your sync tool of your clock or search for other solutions to have your clock being most exact as possible.

Maybe the hardware has an issue here ... :-/

let us know, if you can improve or find the cause.

cheerss

Martin

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/CAGLuZDNkU3RXneQo%3D4wEM5DSuEiaTQE8nkrRYvmiPnSjQEgcBA%40mail.gmail.com.

[sd...@distancelearning.cloud]

Are any of these artifacts in the actual recordings?

 

Regards,

Stephen

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/CAGLuZDPvViFXKmnrd_UBEZSUoTGe3q5mRexX3tg1RCjx7c15tA%40mail.gmail.com.

[Jens Elkner]

On Sun, Aug 23, 2020 at 10:25:37AM +0200, Martin Thomas Schrott wrote:

Hi Martin,

> we used vmware / esxi and also tried KVM.

Ah ok, this makes sense.

> With the settings the crackling could be minimized, but the audio/videeo sync
> problems still happened from time to time. With bare metal all issues
> disappeared.

OK. Just wondered, because many people associate with "virtual server"
native zones like solaris non-global zones, linux containers, docker
bloat, bsd jails, etc. as well. But obviously such zones fall into the
"bare metal" category, too. Furthermore there are a lot of people, which
think on no isolation at all (or in Solaris terms on a global zone),
where every service/SW gets installed directly, when they hear "bare
metal". That's why I usually use the term "[non-]native zones" to avoid
confusion ...

Anyway, thanx a lot,