Bug#1056653: crypto.getRandomValues: "The operation failed for an operation-specific reason"

[Philipp Marek]

Package: firefox
Version: 120.0-2
Severity: normal
X-Debbugs-Cc: phi...@marek.priv.at

I can't reopen 1039566 as it's already archived;
with 120.0-2 I have the same problem again.

$ MOZILLA_DISABLE_PLUGINS=1 firefox -safe-mode

then open any page, open developer tools, and type

>> crypto.getRandomValues(new Uint8Array(3))

to get

Uncaught DOMException: The operation failed for an operation-specific reason
<anonymous> debugger eval code:1
<anonym> debugger eval code:1


That breaks quite a few websites, sadly.


When I noticed the problem I saw that opensc was updated the day before --
that might have something to do with randomness.

https://buildd.debian.org/status/fetch.php?pkg=firefox&arch=amd64&ver=120.0-1&stamp=1700636532&raw=0
doesn't show libopensc or similar being used for compilation, though -
so any breakage would be through a few other libraries.



Debugging attempts with gdb were unsuccessful --
too many threads and too much activity,
too much indirection via async queues,
even when running only one tab.


Is there a way to get a dump similar to what ltrace() should do,
ie. all calls within the binary and to libraries, including arguments and return values?
Perhaps I can find out which part breaks in the first place and start debugging there.


-- Package-specific info:


-- Addons package information

-- System Information:
Debian Release: trixie/sid
APT prefers testing
APT policy: (990, 'testing'), (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 6.5.0-4-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=de_AT.UTF-8, LC_CTYPE=de_AT.UTF-8 (charmap=UTF-8), LANGUAGE=de_AT:de
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages firefox depends on:
ii debianutils 5.14
ii fontconfig 2.14.2-6
ii libasound2 1.2.10-1
ii libatk1.0-0 2.50.0-1
ii libc6 2.37-12
ii libcairo-gobject2 1.18.0-1
ii libcairo2 1.18.0-1
ii libdbus-1-3 1.14.10-3
ii libevent-2.1-7 2.1.12-stable-8
ii libffi8 3.4.4-1
ii libfontconfig1 2.14.2-6
ii libfreetype6 2.13.2+dfsg-1
ii libgcc-s1 13.2.0-5
ii libgdk-pixbuf-2.0-0 2.42.10+dfsg-3
ii libglib2.0-0 2.78.1-4
ii libgtk-3-0 3.24.38-6
ii libnspr4 2:4.35-1.1
ii libnss3 2:3.94-1
ii libpango-1.0-0 1.51.0+ds-3
ii libstdc++6 13.2.0-5
ii libvpx8 1.13.1-2
ii libx11-6 2:1.8.7-1
ii libx11-xcb1 2:1.8.7-1
ii libxcb-shm0 1.15-1
ii libxcb1 1.15-1
ii libxcomposite1 1:0.4.5-1
ii libxdamage1 1:1.1.6-1
ii libxext6 2:1.3.4-1+b1
ii libxfixes3 1:6.0.0-2
ii libxrandr2 2:1.5.2-2+b1
ii procps 2:4.0.4-2
ii zlib1g 1:1.2.13.dfsg-3

Versions of packages firefox recommends:
ii libavcodec58 7:4.4.2-1+b3
ii libavcodec60 7:6.1-3

Versions of packages firefox suggests:
ii fonts-lmodern 2.005-1
ii fonts-stix [otf-stix] 1.1.1-4.1
ii libcanberra0 0.30-11
ii libgssapi-krb5-2 1.20.1-5
ii pulseaudio 16.1+dfsg1-2+b1

-- no debconf information

-- debsums errors found:
debsums: can't check firefox file /usr/share/doc/firefox/changelog.Debian.gz (Wide character in subroutine entry)
debsums: can't check firefox file /usr/share/firefox/browser/chrome/icons/default/default128.png (Wide character in subroutine entry)
debsums: can't check firefox file /usr/share/firefox/browser/chrome/icons/default/default16.png (Wide character in subroutine entry)
debsums: can't check firefox file /usr/share/firefox/browser/chrome/icons/default/default32.png (Wide character in subroutine entry)
debsums: can't check firefox file /usr/share/firefox/browser/chrome/icons/default/default48.png (Wide character in subroutine entry)
debsums: can't check firefox file /usr/share/firefox/browser/chrome/icons/default/default64.png (Wide character in subroutine entry)
debsums: can't check firefox file /usr/share/man/man1/firefox.1.gz (Wide character in subroutine entry)

[Philipp Marek]

Other data points after installing 120.0.1-1:

- Restarting firefox doesn't help

- Rebooting the machine doesn't help
(tried that in case some connected service (dbus?) was the culprit)

[Philipp Marek]

Another data point: The affected FF process has two defunct "Socket
Process"es,
the working FF-esr doesn't.

594571 ? Sl 14:00 \_ /usr/bin/firefox.real
-ProfileManager
594630 ? Z 0:00 | \_ [Socket Process] <defunct>
594753 ? Z 0:00 | \_ [Socket Process] <defunct>


Probably related, every now and then a tab (any URLs) hangs on loading
with a spinning symbol in the tab; copying the URL to another tab
doesn't help, unless the hanging tab gets closed, then the new tab
loads.

So I guess some thread hangs in the background - and so doesn't return
random numbers nor collects the child processes.


I can't make "rr" work to find out why;
"gdb" also has problems, I guess because of the large number of symbols.


Any ideas how to debug this are welcome!

[Philipp Marek]

Ok, I seem to have it fixed locally.


One of the threads in FF ran with much more CPU load, even if no
activity was expected.

"gdb" on it showed lots of

Thread 1 "Isolated Web Co" received signal SIGSYS, Bad system call.
0x00007f5c4e636840 in __libc_open64 (file=file@entry=0x7f5c422af8b0
"/proc/sys/crypto/fips_enabled", oflag=0) at
../sysdeps/unix/sysv/linux/open64.c:41
41 ../sysdeps/unix/sysv/linux/open64.c: Datei oder Verzeichnis
nicht gefunden.
(gdb) bt
#0 0x00007f5c4e636840 in __libc_open64
(file=file@entry=0x7f5c422af8b0 "/proc/sys/crypto/fips_enabled",
oflag=0) at ../sysdeps/unix/sysv/linux/open64.c:41
#1 0x00007f5c4e5bf5b2 in __GI__IO_file_open
(fp=fp@entry=0x7f5c2fd44020, filename=filename@entry=0x7f5c422af8b0
"/proc/sys/crypto/fips_enabled", posix_mode=<optimized out>,
prot=prot@entry=438, read_write=8, is32not64=<optimized out>)
at ./libio/fileops.c:188
#2 0x00007f5c4e5bf76b in _IO_new_file_fopen
(fp=fp@entry=0x7f5c2fd44020, filename=filename@entry=0x7f5c422af8b0
"/proc/sys/crypto/fips_enabled", mode=<optimized out>,
mode@entry=0x7f5c422b05b3 "r", is32not64=is32not64@entry=1)
at ./libio/fileops.c:280
#3 0x00007f5c4e5b2f69 in __fopen_internal (filename=0x7f5c422af8b0
"/proc/sys/crypto/fips_enabled", mode=0x7f5c422b05b3 "r", is32=1) at
./libio/iofopen.c:75
#4 0x00007f5c421faec9 in SECMOD_GetSystemFIPSEnabled () at
/usr/lib/firefox/libnss3.so
#5 0x00007f5c421ec095 in SECMOD_CreateModuleEx () at
/usr/lib/firefox/libnss3.so
#6 0x00007f5c421ed9db in SECMOD_LoadModule () at
/usr/lib/firefox/libnss3.so
#7 0x00007f5c421b38ce in () at /usr/lib/firefox/libnss3.so
#8 0x00007f5c421b40b8 in NSS_NoDB_Init () at
/usr/lib/firefox/libnss3.so
#9 0x00007f5c46729610 in EnsureNSSInitializedChromeOrContent() ()
at ./security/manager/ssl/nsNSSComponent.cpp:203
#10 0x00007f5c4672fce0 in
mozilla::psm::Constructor<nsRandomGenerator, (nsresult
(nsRandomGenerator::*)())0, (mozilla::psm::ProcessRestriction)1>
(aResult=0x7ffc28fc6c18, aIID=...) at
./security/manager/ssl/nsNSSModule.cpp:71

So this seems seccomp-related as well.


An earlier tip I received was "upgrade nss" -- and here are
firefox-local nss libraries that wouldn't be affected by dpkg!

After installing libnss3=2:3.96.1-1 from unstable, doing

/usr/lib/firefox$ mkdir UNUSED
/usr/lib/firefox$ mv libnss3.so libnssutil3.so libssl3.so UNUSED/

and restarting firefox the crypto.getRandomValues() function now works
as expected.


Perhaps firefox shouldn't ship its own nss libraries?!!

[Philipp Marek]

Update:

"strings /usr/lib/firefox/UNUSED/libnss3.so | grep NSS_3. | sort
--version-sort"
gives me 3.79 max,
"dpkg-query -l libnss3" says 2:3.96.1-1

So the libraries included in firefox seem to be much older