03/15/2024 TPV Meeting Agenda - WebRTC

[Signal Linden (Bennett Goble)]

Hey OSS folks,

We have an exciting item on the agenda for Friday's TPV meeting: improvements to voice in Second Life. In a nutshell, we'd like to move to a more flexible voice solution based on WebRTC. This will give everyone higher quality audio, better spatialization, faster connections, less firewall issues due to WebRTC NAT hole punching, and in the future will allow us to consider offering features such as the ability to control voice from LSL, define zones for reverb/effects, etc.

Some wins for OSS are: the new solution is based on open source tools and will obviate the need for a separate executable using IPC (SLVoice) --the viewer can directly link against the webrtc library. I'm told this solution may also help the opensim community.

LL and TPVs will need to coordinate and collaborate here. A migration of this scale will need planning and scheduling: we have some ideas here, involving viewer code that can juggle two voice providers. Would it make sense for LL and TPVs to both have beta viewers?

I've attached a document with a summary of the project. The short term plans are:

- Go over details, get feedback, answer questions during Friday's TPV meeting

- We will publicize viewer code and get some compatible regions onto the beta grid for experimentation this week or early next

This work is being tackled by Roxie Linden, who has worked extensively with WebRTC while at High Fidelity. I've been impressed with the performance of the solution so far, but I'm interested in seeing folks outside the company shake out bugs, identify work that's still needed and provide feedback on Aditi.

If you're not able to attend the TPV meeting at 1:00 PM Pacific tomorrow, don't worry, more news will be shared on this list.

-Signal

[Henri Beauchamp]

On Thu, 14 Mar 2024 09:45:05 -0700, Signal Linden (Bennett Goble) wrote:

> Hey OSS folks,
>
> We have an exciting item on the agenda for Friday's TPV meeting:

You mean the meeting held on voice and which voice-impaired people
cannot attend as a result ?... :-D

Seriously, I am *very annoyed* that the TPV meeting is held on voice,
baring some of us, non-English-speaking TPV developers from these
meetings (while I can *write* English fluently, I won't understand a
third of *spoken* English). :-(

Please, consider switching to 100% chat meetings for *all* Linden meetings
(also, if the TPV meeting could be held in chat, then you could also get
rid of the "Open Source Development" meeting, since the latter got a low
attendance and is only mandated by the fact it is "reserved" to us, non
voice-users)...

Alternatively, and since AI is the new trend, did you consider a voice to
chat automated translator in the meeting places ?
Of course the translation quality would need to be excellent (which might
prove challenging with SL-specific jargon).

> .../...

>
> Some wins for OSS are: the new solution is based on open source tools and
> will obviate the need for a separate executable using IPC (SLVoice) --the
> viewer can directly link against the webrtc library. I'm told this solution
> may also help the opensim community.

That's good: since Vivox refuses to provide Linux builds for their client,
I am *very* happy that LL is getting rid of them !
This will also get rid of the dependency on Wine for Linux voice users...

Open Source is also the way to go (and all the way !). :-)

> LL and TPVs will need to coordinate and collaborate here. A migration of
> this scale will need planning and scheduling: we have some ideas here,
> involving viewer code that can juggle two voice providers.

While I do not use voice at all myself (would it be exclusively reserved
to my own usage, I would have gotten fully rid of voice in my viewer
already), I will implement the necessary changes to ensure the transition
and the future compatibility.

> Would it make sense for LL and TPVs to both have beta viewers?

If *absolutely needed*, I will open an "experimental" viewer branch (what
you would call "beta"), but those are only opened for experimental features
(corresponding to LL's project viewers), that are usually not even yet
supported on the main grid, or when the changes are *massive*, or when
there is a stability risk for the viewer.
Otherwise, new features supported on the main grid go straight into my
stable branch; my weekly release rate, plus my occasional emergency release
policy, allow to get things fixed very quickly (days or even hours) in case
of a crash/show stopper bug late discovery...

Regards,

Henri.

[Henri Beauchamp]

On Mon, 18 Mar 2024 11:27:41 +0100, Henri Beauchamp wrote:

> If *absolutely needed*, I will open an "experimental" viewer branch

Well, after I could have a look at the webtrc viewer code, I will very
likely need an experimental branch... The changes are indeed very large
when compared to my code (which is the v1 viewer code, with many fixes
and modifications by me, but that still uses the good old state machine,
instead of the new coroutine-based code).

Also, something is annoying me: llwebrtc got *nothing to do* in the
viewer code sources themselves. It does not make any use of the viewer
classes and is just a wrapper for the libwebtrc library; as such, it
should be built against libwebtrc and distributed as a pre-built
library in place of the webrtc pre-built library, akin to what is
already done for Dullahan.

The fact that llwebrtc also requires a C++20 compiler is extremely
annoying for Linux builds, because only very recent versions of gcc
(gcc 11+) do implement fully C++20, meaning the build system used for
Linux builds would need to be very recent, also meaning a recent glibc,
and thus an incompatibility with systems running a 3+ years old
distribution with an older glibc.
By building llwebrtc *separately* from the viewer, you can instead use
clang as the compiler: even on "old" build systems clang would be able
to compile C++20 code just fine (e.g., my current build system is
Devuan Beowulf, with gcc 7.5 which is not at all C++20-compatible and
clang 13 which is fully C++20-compliant), and the resulting library
will run fine, even on "old" Linux distros...

***********************************************************************
Here is my first (untested !) Linux build of llwebtrtc for Linux:
http://sldev.free.fr/libraries/llwebrtc-linux64-20240320.tar.bz2
and here is the "sources" tarball (with a linux-build.sh script and an
appropriate CMakeLists.txt file), which demonstrates how to build
llwebrtc as a "standalone" pre-built library:
http://sldev.free.fr/libraries/sources/cvlv-llwebrtc-20240320.tar.bz2
Note that I so far did not test standalone llwebrtc builds for Windows
or macOS (I usually rely on LL for those, because I am lazy ! :-D).
***********************************************************************

Another annoyance about webtrc is that it is Google's stuff, with their
silly SHIM (memory allocator wrapper) code, which dirtily hooks into
__libc_malloc(), __libc_free(), etc, sometimes causing issues (memory
leaks at best, crashes at worst) when the viewer is using an alternative
allocator to glibc's malloc (I am using jemalloc which gives very nice
speed benefits). I had this fixed for CEF already, but this requires a
custom/patched build (you will find mine for Linux x86_64 at:
http://sldev.free.fr/libraries/sources/ ).

So, just in case I would need the same kind of trick as with CEF, I
tried and re-built webrtc from sources, and indeed managed to write a
script to do this automatically... Just to notice, after I tried to link
the resulting library to llwebrtc, that there were missing symbols (such
as "AudioDeviceSink")... WOT ?... ARGH !... It was not Googles webrtc,
but a webrtc-sdk fork !... And the latter does not even have build
instructions to rebuild it on a PC (it apparently only builds on
github) !!!

Is there any way to build this thing by ourselves, with our own patches,
without going through github commits & stuff ? O.O

Regards,

Henri.

[Henri Beauchamp]

Greetings,

Regarding webrtc-sdk building, I managed to write a couple scripts and
create a patch to its Python build script to get it to rebuild (without
any need for LL's autobuild) on any Linux x86_64 or arm64 system, from
LL's 3p-webrtc-build repository sources.
For now, my builds do not include any patch for potential issues when
using jemalloc: I want first to test them "as is" and see whether a
crash or memory leak happens, but being able to rebuild (and thus patch)
it by myself ensures that jemalloc usage is "secured" for the future...

Here are the Linux pre-built packages for webrtc-sdk (can be used "as
is" with LL's very latest viewer-voice-webrtc code):
http://sldev.free.fr/libraries/sources/webrtc-114.5735.08-linux64.tar.bz2
http://sldev.free.fr/libraries/sources/webrtc-114.5735.08-linux-arm64.tar.bz2

I also link it to llwebrtc to provide it as a standalone pre-built
libllwebrtc.so library (thus avoiding the dependency on a C++20 compiler
to build the viewer itself):
http://sldev.free.fr/libraries/llwebrtc-linux64-20240321.tar.bz2
http://sldev.free.fr/libraries/llwebrtc-linux-arm64-20240321.tar.bz2

To rebuild webrtc-sdk and/or a standalone llwebrtc yourself under Linux,
you can use:
http://sldev.free.fr/libraries/sources/cvlv-llwebrtc-20240321.tar.bz2
Read the comments in linux-build.sh for details.

Henri.

[Henri Beauchamp]

On Thu, 21 Mar 2024 18:15:53 +0100, Henri Beauchamp wrote:

Sadly, I'm getting into troubles with this mix of static webrtc library
and shared llwebrtc library at viewer link time:

undefined reference to `rtc::SocketAddress::PortAsString[abi:cxx11]() const'
undefined reference to `webrtc::CreateIceCandidate(std::__cxx11::basic_string<char, .../...
undefined reference to `webrtc::CreateSessionDescription(webrtc::SdpType, .../...
undefined reference to `webrtc::CreateAudioDeviceWithDataObserver(webrtc::AudioDeviceModule::AudioLayer, .../...
undefined reference to `rtc::IPAddress::ToString[abi:cxx11]() const'
undefined reference to `webrtc::CreatePeerConnectionFactory(rtc::Thread*, .../...
undefined reference to `rtc::Thread::SetName(std::basic_string_view<char, .../...

Which relates to webrtc/include/webrtc/rtc_base/system/rtc_export.h:

// RTC_EXPORT is used to mark symbols as exported or imported when WebRTC is
// built or used as a shared library.
// When WebRTC is built as a static library the RTC_EXPORT macro expands to
// nothing.

Meaning the said symbols are not exported in the static libwebrtc.a library,
while llwebrtc needs them...

This is going to be a *very painful* backport for Linux viewers ! :-(

Henri.

[Henri Beauchamp]

On Thu, 21 Mar 2024 22:10:46 +0100, Henri Beauchamp wrote:

> Sadly, I'm getting into troubles with this mix of static webrtc library
> and shared llwebrtc library at viewer link time:
>
> undefined reference to `rtc::SocketAddress::PortAsString[abi:cxx11]() const'
> undefined reference to `webrtc::CreateIceCandidate(std::__cxx11::basic_string<char, .../...
> undefined reference to `webrtc::CreateSessionDescription(webrtc::SdpType, .../...
> undefined reference to `webrtc::CreateAudioDeviceWithDataObserver(webrtc::AudioDeviceModule::AudioLayer, .../...
> undefined reference to `rtc::IPAddress::ToString[abi:cxx11]() const'
> undefined reference to `webrtc::CreatePeerConnectionFactory(rtc::Thread*, .../...
> undefined reference to `rtc::Thread::SetName(std::basic_string_view<char, .../...

OK, I think I got it now... It is not a symbol export problem (even
though I found and fixed an issue in LL's run.py script, namely a
missing call to ranlib, which could cause linking issues too, due to
badly ordered symbols in the dependency tree), but the result of
webrtc-sdk being compiled against libc++ (llvm/clang) instead of
libstdc++ (gcc): the ABIs are *incompatible* and thus the symbols
cannot resolve !

It is also impossible to compile llwebrtc under Linux against libc++
and then link it to the viewer using libsdtc++.

A solution would be to compile webrtc-sdk against libsdtc++, but
sadly, this option appears to have been removed, while Google's
webrtc still allows this.

Maybe we could use Google's webrtc, instead of the webrtc-sdk fork ?

Another solution, would be to add a C++ to C wrapper to webrtc-sdk,
akin to what CEF is doing with Chromium, so that the calls done from
llwebrtc to webrtc-sdk are all done via a compatible C ABI.

As it is, webrtc-sdk is *unusable* for Linux viewers.

Regards,

Henri.

[Henri Beauchamp]

Greetings everyone,

Here is some more "fun" with webrtc-sdk for Linux !...

In addition to my previous emails (*), I just found out today, after I
finally backported enough code to my viewer to invoke llwebrtc::init()
on launch, that webrt-sdk is quite happy to "crash" the viewer (by
sending a SIGABRT) when it is not happy about something else (here, it
failed to initialize its audio manager module); instead of politely
bowing out and letting the application continue to run without its help
(which the viewer certainly could do), it crashes it !!! >:-(

Here is the "crash":
--------------
2024-03-29 21:27:25Z INFO: LLVoiceVivox::init: Initializing Vivox voice client.
2024-03-29 21:27:25Z INFO: LLVoiceWebRTC::init: Initializing WebRTC voice.

#
# Fatal error in: ../../../../_source/ubuntu-20.04_x86_64/webrtc/src/media/engine/adm_helpers.cc, line 39
# last system error: 88
# Check failed: 0 == adm->Init() (0 vs. -1)
# Failed to initialize the ADM.Process 1524 stopped
* thread #97, name = 'WebRTCWorkerThr', stop reason = signal SIGABRT
frame #0: 0x0000555545e9d46c libc.so.6`___lldb_unnamed_symbol3514 + 236
.../...
--------------
And the reason for this crash ?... Well, I searched quite a bit (down
to webrtc-sdk sources), and it is apparently because it could not
initialize PulseAudio (and did not even try ALSA instead, as it
should) ! No surprise, since I don't have PulseAudio on my system !

So... I recompiled webrtc-sdk (yet again), this time with the following
added gn args:
rtc_include_pulse_audio=false rtc_include_internal_audio_device=true
Once recompiled, and llwebrtc re-linked to this new build, viewer
binaries linked against it work fine (actual webrtc voice not yet
tested however: more code backport needed for that).
Note that during my searches on the web for similar crashes and
PulseAudio issues, I found out that Google themselves are disabling
PulseAudio in their webrtc build for Chrome... Funny, isn't it ?


Oh, also, Roxie, in llwebrtc.cpp, there is a wrong (inverted,
apparently) debug test "RTC_DCHECK(mPeerConnectionFactory);" at the
very start of LLWebRTCImpl::init(), causing a crash since at this point
mPeerConnectionFactory is not yet initialized and thus "false" (the
test was likely supposed to be RTC_DCHECK(!mPeerConnectionFactory),
but I removed it anyway, as well as the only other RTC_DCHECK in that
file...

So, you will find the new Linux builds for webrtc-sdk and my
standalone llwebrtc on my site:
webrtc-sdk and cvlv-llwebrtc build scripts/patches/sources:
http://sldev.free.fr/libraries/sources/
llwebrtc binaries:
http://sldev.free.fr/libraries/
Currently Linux x86_64 and macOS, Linux aarch64 build still in progress,
and Windows build still on hold until we can get a non-debug build of
webrtc-sdk by LL: any news about it, Roxie ?

Regards,

Henri.


--------------
(*) Here is my last email, to which I replied to myself by mistake,
causing the list not to get it... Citing it for the list archives:

Date: Fri, 22 Mar 2024 15:15:04 +0100
From: Henri Beauchamp <sl...@free.fr>
To: Henri Beauchamp <sl...@free.fr>
Cc: "Signal Linden (Bennett Goble)" <sig...@lindenlab.com>,
ro...@lindenlab.com, sl...@vlan1000.net
Subject: Re: [opensource-dev] 03/15/2024 TPV Meeting Agenda - WebRTC

And... Final last word !

On Fri, 22 Mar 2024 12:02:51 +0100, Henri Beauchamp wrote:

> A solution would be to compile webrtc-sdk against libsdtc++, but
> sadly, this option appears to have been removed, while Google's
> webrtc still allows this.

Well, it appears that the failure to build webrtc-sdk against libsdtc++
(which lead to a ninja error when using the use_custom_libcxx=false
option, ninja complaining it cannot build the third_party/libc++
target), was due to *yet another* bug in run.py, which is forcing
unconditionally the building of libc++ as soon as you select a Linux
target, and regardless of use_custom_libcxx=false usage...

I could therefore patch this other bug and get run.py to build
webrtc-sdk against libsdtc++ !

And as I expected, the missing symbols issue is now gone ! \o/

Here are my latest sources/packages for the libllwebrtc.so pre-built
library that can be used to compile a Linux viewer:

Sources/patch/scripts for rebuilding libllwebrtc.so and/or webrtc-sdk:
http://sldev.free.fr/libraries/sources/cvlv-llwebrtc-20240322.tar.bz2

Build of webrtc-sdk (Devuan Bookworm build system with gcc 8.3,
clang 13, glibc v2.28, libstdc++ v6.0.25):
http://sldev.free.fr/libraries/sources/webrtc-114.5735.08-linux64.tar.bz2

Build of llwebrtc (same build system):
http://sldev.free.fr/libraries/llwebrtc-linux64-20240322.tar.bz2

Linux aarch64 builds are in progress (this will take "some" time,
since done on an Orange PI5+) and will be published later today in
the same locations.

IMPORTANT NOTE: while I could test that linking works fine, I did
not yet backport the webrtc voice code to my viewer and could therefore
NOT test whether the library *does* work or not. In particular, if your
Linux viewer is using jemalloc, you might encounter crashes (see the
reason why here: https://github.com/chromiumembedded/cef/issues/3095 ).
If you do encounter such crashes, let me know (I know how to patch
Google's shitty SHIM code to fix this)...

TODO: provide Windows and macOS pre-built (standalone) llwebrtc library
packages so that we can remove the wrapper code from the viewer sources.
I'd rather let LL do this and concentrate on my backport to my viewer...


Regards,

Henri.

[Henri Beauchamp]

On Sat, 30 Mar 2024 12:51:15 -0700, Roxanne Skelly wrote:

> Hi Henri,
> Feel free to try this build of the webrtc libraries.
>
> It includes:
> * A release windows build.

This webrtc-sdk release build indeed links like a charm and provides me
with a standalone llwebrtc.[lib|dll|exp] library.

I could also link and run a viewer using that library (even if I cannot
yet test WebRTC voice, since my code is not yet ready for that, I could
at least verify that webrtc is initialized and terminated properly).

Thank you ! :-)

I uploaded my builds and an updated cvlv-llwebrtc sources tarball to my
site, in the usual places.

> * A linux build with pulse audio disabled (and internal audio devices
> enabled)

I will keep using my own builds for Linux, the reason being that yours
are done against Ubuntu 22, which means any distro with an older glibc
versions would fail to run the viewer using that library build...
I build mine against Ubuntu 20, which also corresponds to the "sysroot"
used to build CEF, and allows the viewer to run on any 4 years or less
Linux distribution (instead of 2 years or less for your build).

Also, I expect potential issues between webrtc and jemalloc, and in this
case I will need to patch and rebuild webrtc-sdk myself, anyway (I will
of course share my patch with you)...

Thanks again and have a nice week-end ! :-)

Henri.

[Henri Beauchamp]

Greetings,

Here is an update about my experiments with WebRTC.

Sadly, under Linux, WebRTC often deadlocks on disconnection. Here is a stack
trace for the deadlock (it is always the same in these cases):

libc.so.6`___lldb_unnamed_symbol3481 + 198
libc.so.6`pthread_cond_wait + 440
libllwebrtc.so`___lldb_unnamed_symbol3282 + 571
libllwebrtc.so`___lldb_unnamed_symbol4589 + 318
libllwebrtc.so`___lldb_unnamed_symbol4590 + 14
libllwebrtc.so`___lldb_unnamed_symbol4588 + 33
libllwebrtc.so`llwebrtc::LLWebRTCPeerConnectionImpl::terminate() + 203
libllwebrtc.so`llwebrtc::LLWebRTCPeerConnectionImpl::shutdownConnection() + 10
cool_vl_viewer-bin`LLVoiceConnection::breakVoiceConnectionCoro(this=0x0000555567e15930) at llvoicewebrtc.cpp:625:25
cool_vl_viewer-bin`boost::fibers::worker_context<LLCoros::launch(.../...)::$_0>::run_(boost::context::fiber&&) [inlined] boost::function0<void>::operator()(this=0x000055554341fa48) const at function_template.hpp:771:14
cool_vl_viewer-bin`boost::fibers::worker_context<LLCoros::launch(.../...)::$_0>::run_(boost::context::fiber&&) [inlined] LLCoros::toplevel(this=<unavailable>, name="breakVoiceConnectionCoro", callable=LLCoros::callable_t @ 0x000055554341fa48) at llcoros.cpp:168:3

Apparently, pthread_cond_wait() is waiting forever for a condition that is
never met...

This of course freezes the viewer ! :-(

Given the above stack trace, I came to suspect a weird interaction between
pthread mutexes and boost::fiber coroutines (non-coroutine aware mutex ?)...

I thought about possible workarounds, such as breaking down
breakVoiceConnectionCoro() into several steps: the data and audio interfaces
tear down would be posted outside of the coroutine (just before launching it),
the HTTP post would be kept in the coroutine, but on return, it would post the
shutdownConnection() call to the main loop work queue, so that the deadlock-prone
shutdownConnection() function is not called from within a coroutine, and finally
an additional state would be added to the state machine to account for the
splitting of the disconnection process.

However, I told to myself that this won't cover other possible deadlock and
crash issues coming from that webrtc-sdk library (which looks to me as
especially fragile and badly guarded against unexpected connection states)...

So, I finally opted for a more radical approach: since I cannot decently
trust WebRTC to not crash/hang and ruin the viewer stability, and since
it uses its own (non-standard) threading model, let's isolate it entirely
from the viewer binary !

There are two ways to achieve this: create a separate voice client, akin to
what is done with SLVoice(.exe), and launch/communicate with it just what
is done in llviewervivox.cpp... Or, even better: use the viewer plugin
infrastructure to create a webrtc plugin !
I of course opted for the second solution (more elegant, simpler), and I
did implement a webrtc plugin for my viewer; it works like a charm to
isolate the WebRTC calls from the viewer code, and when it crashes or
freezes, all I need is to restart it !
And sadly, the deadlocks indeed still happen within the plugin (meaning the
problem was not related with the coroutines, but likely due to some bug in
the webrtc-sdk library itself). I'm wondering if it won't best to ditch
that buggy webrtc-sdk fork and use Google's webrtc instead (it is working
in CEF, so I would expect it to work too for WebRTC voice)...

Note that the plugin model also got another advantage (beyond viewer
stability): it does not cause any FPS "hiccup" when calling the WebRTC
functions and processing the plugin messaging is extremely lightweight
(just and exchange of an LLSD every now and then, and the read-message-
from-plugin part is even threaded !).

You will find the plugin code in the Cool VL Viewer v1.32.1.0 sources (1)
I just published (this is a new experimental branch of my viewer, for
WebRTC and PBR upcoming updates); you are of course most welcome to reuse
it in your own viewer (and free to re-license it for such a purpose).
Beyond the plugin itself (media_plugins/webrtc/voice_plugin_webrtc.cpp)
which could likely be reused "as is", you would need to backport the few
changes done to LLPluginClassMedia (indra/llplugin/llpluginclassmedia.*)
but I took care to regroup all those changes and comment them, so that
would be an easy task. Then, of course, you would need to backport the
plugin management code I wrote in llvoicewebrtc.*, but here again, that
should prove easy (they are enclosed in #if HB_USE_WEBRTC_PLUGIN/#else/
#endif preprocessor clauses, and thus trivial to understand and follow).

Finally, I am attaching the latest diff between Roxie's llwebrtc and
mine, which is patched to work under Linux (especially to not hang
on llwebrtc::terminate() and to properly list audio devices).



To go further with WebRTC support in my viewer (which still does not
properly work), I am currently faced with the following issues:

- The Linux audio devices are still not properly set by WebRTC (not sure
why: I will likely have to dig into WebRTC sources and instrument them
to find out).
- Under both Linux and Windows, I get a 404 error reply from the server
to the last post of ICE updates (which is the "completed=true" post,
done to the exact same URL as the former ICE candidates updates which
were accepted without error); this causes the voice connection to stop
and be retried (disconnected) after 30s or so...
- The latest Windows build of WebRTC (2) is _again_ a DEBUG build, and
cannot therefore be linked properly with a RELEASE viewer (or plugin).

Regards,

Henri.


(1) http://sldev.free.fr/sources/
(2) https://github.com/secondlife/3p-webrtc-build/releases/tag/m114.5735.08.58
See also:
https://github.com/secondlife/viewer/commit/f5c5c5ec9497211010b5a4dd15bea44d4bbc8482#comments

[Henri Beauchamp]

On Sat, 20 Apr 2024 16:33:43 +0200, Henri Beauchamp wrote:

> To go further with WebRTC support in my viewer (which still does not
> properly work), I am currently faced with the following issues:
>

> .../...

> - Under both Linux and Windows, I get a 404 error reply from the server
> to the last post of ICE updates (which is the "completed=true" post,
> done to the exact same URL as the former ICE candidates updates which
> were accepted without error); this causes the voice connection to stop
> and be retried (disconnected) after 30s or so...

OK, I found the bug (typo or rather copy-paste bug) in my code !
I posted {'candidates':{'completed':1}, .../.../ } as the LLSD body for
mIceCompleted == true, instead of {'candidate':{'completed':1}, .../.../ }
(because I copied "candidates" from the ICE sending code)... I hate those
silly bugs (so hard to find)...

Yet, I would have found it much sooner if, instead of replying with a 404
error ("not found", like if the URL itself was bad), the server had replied
with a 400 (bad request) error... You might want to change this, server-side...

Henri.

[Henri Beauchamp]

On Sat, 20 Apr 2024 16:33:43 +0200, Henri Beauchamp wrote:

> To go further with WebRTC support in my viewer (which still does not
> properly work), I am currently faced with the following issues:
>
> - The Linux audio devices are still not properly set by WebRTC (not sure
> why: I will likely have to dig into WebRTC sources and instrument them
> to find out).

Well, it looks like WebRTC needs more incentives to properly list all ALSA
devices, else it only lists a few, that do not contain the right ones on my
system (AMD Zen4 CPU, which uses USB Audio devices, making things a little bit
more tricky).

The solution is simple (will have to document that for users though):
echo "defaults.namehint.extended on" >>~/.asoundrc
And maybe (not needed for me, but might help on some systems):
echo "defaults.namehint.showall on" >>~/.asoundrc

Henri.

[Roxanne Skelly]

Hi Henri,

Thank you for pushing through the integration and your notes on what's happening.

I'm quite curious about the deadlock you're seeing.  Usually, when I hit those, I look at the stack traces of the deadlocked thread and the other webrtc threads (as well as the main thread) to determine which two threads are waiting on the resource.  That's been quite enlightening in the past when tracking down deadlocks.  You'd need full symbols for webrtc.lib to gain insight to what's going on, though.

Once I determine that cause, and if it's a webrtc bug itself, I usually check the chromium sources for something more recent.  If they've fixed it, great, I can pull in their fix as a patch.  Otherwise, I create my own patch.  The webrtc-sdk stuff contains a mechanism for applying patches.  (you probably know this, but I'm stating it for others.)

AFAIK, the only changes to m114 from stock chromium 114 are those patches.  I would like to update at some point.  WebRTC m114 is only a year old, and got plenty of airtime as part of chromium, so I'm not too worried about the fragility.  It's likely that I'm doing something wrong in connection shutdown.  Then again, I've found bugs in webrtc (and have patched them.)

I'm glad you found a plugin solution for seperating webrtc from the viewer.  I'll take a look at it.  Especially if I can't figure out the stalling/stuttering issue, which is a weird one as I've thrown tracy against the viewer and the Main Thread work done by webrtc is minimal.  It's also strange as it doesn't occur on all machines.  And as it's been seen on Mac and Windows (and apparently linux.)  That's good info, though, as it'll likely help me track down the culprit.

-Roxie

[Henri Beauchamp]

On Mon, 22 Apr 2024 10:43:59 -0700 (PDT), Roxanne Skelly wrote:

> Hi Henri,
> Thank you for pushing through the integration and your notes on what's
> happening.
>
> I'm quite curious about the deadlock you're seeing. Usually, when I hit
> those, I look at the stack traces of the deadlocked thread and the other
> webrtc threads (as well as the main thread) to determine which two threads
> are waiting on the resource. That's been quite enlightening in the past
> when tracking down deadlocks. You'd need full symbols for webrtc.lib to
> gain insight to what's going on, though.

Linux, here, so I'd need to compile a debug build of webrtc-sdk to get the
debug symbols and investigate the deadlock: but I'm afraid I cannot afford
spending time chasing this bug. Due to RL constraints and the little time
left to develop my viewer, I am already "late" on other backports (PBR
terrain, mirrors), and now want to work on those...

With my llwebrtc shutdown + audio devices Ids patch, plus my plugin, plus
the fact I found out today the reasons and the fixes for the issues with
Linux audio devices and the ICE 404 error (see my last two emails in this
list), I now have a "more or less working WebRTC voice", and can concentrate
on the rest.

I will of course keep refining my webrtc backport, but I'd need a couple
things first:

- Something equivalent to the "Voice echo canyon" (I tried your "echo pylons"
but they do nothing at all for me: I tried clicking them, standing on them, to
no avail, including with LL's webrtc viewer).

- A *release* build of the last webrtc-sdk Windows releases (the one you
link to in the last commit is a *debug* build). See:
https://github.com/secondlife/viewer/commit/f5c5c5ec9497211010b5a4dd15bea44d4bbc8482#comments

> WebRTC m114 is only a year old, and got plenty of airtime as part of
> chromium,

Excepted that Chromium disables some parts of WebRTC to replace them with
its own (e.g. audio devices management), or so did I read in various posts
I found on the web while searching for similar bugs to the ones I saw
happening...

> so I'm not too worried about the fragility.

All Google Chromium stuff (i.e. WebRTC included) is fragile, I'm afraid...
Some parts of their code look like they were programmed by total amateurs,
and the hacks they use, especially around memory management, are plain
bogus, e.g. https://github.com/chromiumembedded/cef/issues/3095

I'm glad both (CEF & WebRTC) are now fully isolated (at the process level)
from my viewer thanks to the plugins: at least, when they crash, they do not
bring the viewer down with them !

> It's likely that I'm doing something wrong in connection shutdown.

See my email to you (dated Mon, 22 Apr 2024 11:11:01 +0200) and this
article I found:
https://groups.google.com/g/discuss-webrtc/c/Y5KANqJG1Wo

Since you recently updated from M9x to M114, and the changes described
there have been implemented in M111, I won't be surprised if it would
impact llwebrtc as you initially coded it.

> I'm glad you found a plugin solution for seperating webrtc from the
> viewer. I'll take a look at it. Especially if I can't figure out the
> stalling/stuttering issue, which is a weird one as I've thrown tracy
> against the viewer and the Main Thread work done by webrtc is minimal.
> It's also strange as it doesn't occur on all machines. And as it's been
> seen on Mac and Windows

Well, threads use mutexes to exchange data. When a mutex is locked by a
thread, the other thread must wait... This is, of course, totally random
and depends on the system the viewer runs on, its load, etc...

Also I remember seeing a title for one of your commits akin to "push some
work to coroutines to spread the load" (or something in that vein). But even
if boost's folks call them "fibers", they are not real fibers (they are not
lightweight *threads*), but just plain coroutines all executed *in the main
thread* in a time sharing and cooperative manner. If a coroutine is slow to
yield back to the main coroutine, you will get a FPS "hiccup".
Since you call some WebRTC methods in coroutines, the said coroutines cannot
yield back until the said methods return and you will still get a hiccup via
the llcoro::suspend() call from the main coroutine.

That's why the plugin solution is better (no delay seen from the viewer side
due to WebRTC method calls (or callbacks), since all those are *entirely*
executed in the SLPlugin process).

> (and apparently linux.)

I did not see those, under Linux, but this is just because, without the plugin,
the viewer froze too often and too fast ! :-D

Plus, to judge properly about the "hiccups" impact of WebRTC, we would need
to test the latter in real conditions (not in empty sims), with lots of stuff
rezzing, neighbour sims connecting and disconnecting as we move around, etc...

Best regards,

Henri

[Henri Beauchamp]

Greetings,

Just a short follow-up to say that with Roxie's latest WebRTC build (*), I get
a lot less deadlocks under Linux (in fact, *almost* none): I *suspect* that
the deadlock is related with audio processing not properly being brought up
(or down)...

This is the WebRTC version in use in today's release of the Cool VL Viewer
experimental branches (v1.32.1.1), which seems to work pretty well already
(with the exception of logins in WebRTC sims, after which you still need to
switch voice off and back on to get Voice working).

For developers interested in my WebRTC plugin for their own viewer, they
should also pick up the latest changes that went into v1.32.1.1 (both in
indra/media_plugins/webrtc/voice_plugin_webrtc.cpp and in
indra/newview/llvoicewebrtc.cpp) since bugs have been fixed there.

Regards,

Henri.

(*) https://github.com/secondlife/3p-webrtc-build/releases/tag/m114.5735.08.59-alpha