Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Bug#982794: firefox-esr: illegal instruction in libxul.so on armhf

34 views
Skip to first unread message

Vincent Arkesteijn

unread,
Feb 14, 2021, 9:20:03 AM2/14/21
to
Package: firefox-esr
Version: 78.7.0esr-1
Severity: normal

Dear Maintainer,

Firefox is killed with SIGILL shortly after startup:
$ firefox-esr -safe-mode
Illegal instruction
$

A gdb session on a dumped core to investigate:
[...]
Core was generated by `firefox-esr'.
Program terminated with signal SIGILL, Illegal instruction.
#0 0xaf6f0ab0 in ?? () from /usr/lib/firefox-esr/libxul.so
(gdb) backtrace
#0 0xaf6f0ab0 in ?? () from /usr/lib/firefox-esr/libxul.so
#1 0xb6f32f40 in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0xbeb44584, env=env@entry=0xb6a0a240) at dl-init.c:72
#2 0xb6f32fe2 in call_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>) at dl-init.c:30
#3 _dl_init (main_map=0xb6a30c00, argc=1, argv=0xbeb44584, env=0xb6a0a240) at dl-init.c:119
#4 0xb6cec52e in __GI__dl_catch_exception (exception=exception@entry=0x0, operate=0xb6f352c1 <call_dl_init>, args=0xbeb41ed8, args@entry=0xbeb41f10)
at dl-error-skeleton.c:182
#5 0xb6f35d04 in dl_open_worker (a=<optimized out>) at dl-open.c:758
#6 0xb6cec4f4 in __GI__dl_catch_exception (exception=exception@entry=0xbeb420d8, operate=0xb6f35869 <dl_open_worker>, args=args@entry=0xbeb420e4)
at dl-error-skeleton.c:208
#7 0xb6f355cc in _dl_open (file=0xbeb4237c "/usr/lib/firefox-esr/libxul.so", mode=-2147483391, caller_dlopen=0xb6f5df85 <_start+2424>, nsid=-2, argc=1,
argv=0xbeb44584, env=0xb6a0a240) at dl-open.c:837
#8 0xb6eeed18 in dlopen_doit (a=0xbeb42344) at dlopen.c:66
#9 0xb6cec4f4 in __GI__dl_catch_exception (exception=exception@entry=0xbeb42300, operate=0xb6eeecc1 <dlopen_doit>, args=args@entry=0xbeb42344)
at dl-error-skeleton.c:208
#10 0xb6cec588 in __GI__dl_catch_error (objname=objname@entry=0xb6a0d2ec, errstring=errstring@entry=0xb6a0d2f0, mallocedp=mallocedp@entry=0xb6a0d2e8,
operate=<optimized out>, args=args@entry=0xbeb42344) at dl-error-skeleton.c:227
#11 0xb6eef3de in _dlerror_run (operate=<optimized out>, args=args@entry=0xbeb42344) at dlerror.c:170
#12 0xb6eeed9e in __dlopen (file=0xbeb4237c "/usr/lib/firefox-esr/libxul.so", mode=<optimized out>) at dlopen.c:87
#13 0xb6f5df84 in _start ()
(gdb) layout asm
0xaf6f0ab0 vmov.i32 q8, #0 ; 0x00000000

This is on a Marvell Dove system, with VFPv3-D16. From /proc/cpuinfo:
Features : swp half thumb fastmult vfp edsp iwmmxt thumbee vfpv3 vfpv3d16 tls

The referenced (NEON?) register Q8 is not available here, nor even the VFPv3-D32 registers D16-D17 that it maps to.

-- Package-specific info:


-- Addons package information

-- System Information:
Debian Release: bullseye/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: armhf (armv7l)

Kernel: Linux 3.5.7-14-ARCH (PREEMPT)
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages firefox-esr depends on:
ii debianutils 4.11.2
ii fontconfig 2.13.1-4.2
ii libatk1.0-0 2.36.0-2
ii libc6 2.31-9
ii libcairo-gobject2 1.16.0-5
ii libcairo2 1.16.0-5
ii libdbus-1-3 1.12.20-1
ii libdbus-glib-1-2 0.110-6
ii libevent-2.1-7 2.1.12-stable-1
ii libffi7 3.3-5
ii libfontconfig1 2.13.1-4.2
ii libfreetype6 2.10.4+dfsg-1
ii libgcc-s1 10.2.1-6
ii libgdk-pixbuf-2.0-0 2.42.2+dfsg-1
ii libglib2.0-0 2.66.6-2
ii libgtk-3-0 3.24.24-1
ii libnspr4 2:4.29-1
ii libnss3 2:3.60-1
ii libpango-1.0-0 1.46.2-3
ii libstdc++6 10.2.1-6
ii libvpx6 1.9.0-1
ii libx11-6 2:1.7.0-2
ii libx11-xcb1 2:1.7.0-2
ii libxcb-shm0 1.14-3
ii libxcb1 1.14-3
ii libxcomposite1 1:0.4.5-1
ii libxdamage1 1:1.1.5-2
ii libxext6 2:1.3.3-1.1
ii libxfixes3 1:5.0.3-2
ii libxrender1 1:0.9.10-1
ii procps 2:3.3.16-5
ii zlib1g 1:1.2.11.dfsg-2

Versions of packages firefox-esr recommends:
ii libavcodec58 7:4.3.1-8

Versions of packages firefox-esr suggests:
pn fonts-lmodern <none>
pn fonts-stix | otf-stix <none>
pn libcanberra0 <none>
ii libgssapi-krb5-2 1.18.3-4
pn libgtk2.0-0 <none>
pn pulseaudio <none>

-- no debconf information

Jochen Sprickerhof

unread,
Jul 1, 2021, 5:00:04 PM7/1/21
to
Hi Hideki and Vincent,

just some information from a bystander. Also Ccing the bug author for
more information. @Vincent: would be great to get your answers as well.

* Hideki Yamane <hen...@iijmio-mail.jp> [2021-06-28 22:35]:
>On Sun, 14 Feb 2021 14:12:17 +0000 Vincent Arkesteijn <vin...@arkesteijn.net> wrote:
>> Firefox is killed with SIGILL shortly after startup:
>> $ firefox-esr -safe-mode
>> Illegal instruction
>
> Can you reproduce it on freshly installed bullseye sytem?

I guess this is still possible given:

>>The referenced (NEON?) register Q8 is not available here, nor even the VFPv3-D32 registers D16-D17 that it maps to.

I only found this reference for NEON on armhf:

"NEON and VFP/VFP2/VFP3 remain an optional part of the architecture."

https://wiki.debian.org/ArmHardFloatPort#VFP

There are strings like -DBUILD_ARM_NEON=1 and -DHAVE_ARM_NEON=1 in the
buildd logs here:

https://buildd.debian.org/status/fetch.php?pkg=firefox-esr&arch=armhf&ver=78.11.0esr-1&stamp=1622606659&raw=0

(I didn't look what they actually do.)

If this is still reproducible, I see two options:
- Disable NEON code.
- Depend on the neon-support dummy package.

>> -- System Information:
>> Debian Release: bullseye/sid
>> APT prefers testing
>> APT policy: (500, 'testing')
>> Architecture: armhf (armv7l)
>
> However,
>
>> Kernel: Linux 3.5.7-14-ARCH (PREEMPT)
>
> It seems that is not the kernel bullseye provides.
>
>
> And it maybe help to provide its hardware information, too.

The bug author wrote:

>>This is on a Marvell Dove system, with VFPv3-D16. From /proc/cpuinfo:
>>Features : swp half thumb fastmult vfp edsp iwmmxt thumbee vfpv3 vfpv3d16 tls

Cheers Jochen
signature.asc

Jochen Sprickerhof

unread,
Jul 1, 2021, 7:40:02 PM7/1/21
to
(somehow my message didn't made it to the BTS, sorry for double post.)

Hi Hideki and Vincent,

just some information from a bystander. Also Ccing the bug author for
more information. @Vincent: would be great to get your answers as
well.

* Hideki Yamane <hen...@iijmio-mail.jp> [2021-06-28 22:35]:
>On Sun, 14 Feb 2021 14:12:17 +0000 Vincent Arkesteijn <vin...@arkesteijn.net> wrote:
>>Firefox is killed with SIGILL shortly after startup:
>>$ firefox-esr -safe-mode
>>Illegal instruction
>
>Can you reproduce it on freshly installed bullseye sytem?

I guess this is still possible given:

>>The referenced (NEON?) register Q8 is not available here, nor even the VFPv3-D32 registers D16-D17 that it maps to.

I only found this reference for NEON on armhf:

"NEON and VFP/VFP2/VFP3 remain an optional part of the architecture."

https://wiki.debian.org/ArmHardFloatPort#VFP

There are strings like -DBUILD_ARM_NEON=1 and -DHAVE_ARM_NEON=1 in the
buildd logs here:

https://buildd.debian.org/status/fetch.php?pkg=firefox-esr&arch=armhf&ver=78.11.0esr-1&stamp=1622606659&raw=0

(I didn't look what they actually do.)

If this is still reproducible, I see two options:
- Disable NEON code.
- Depend on the neon-support dummy package.

>>-- System Information:
>>Debian Release: bullseye/sid
>> APT prefers testing
>> APT policy: (500, 'testing')
>>Architecture: armhf (armv7l)
>
>However,
>
>>Kernel: Linux 3.5.7-14-ARCH (PREEMPT)
>
>It seems that is not the kernel bullseye provides.
>
>
>And it maybe help to provide its hardware information, too.

The bug author wrote:

>>This is on a Marvell Dove system, with VFPv3-D16. From /proc/cpuinfo:
>>Features : swp half thumb fastmult vfp edsp iwmmxt thumbee vfpv3 vfpv3d16 tls

Cheers Jochen
signature.asc

Vincent Arkesteijn

unread,
Jul 4, 2021, 9:30:03 AM7/4/21
to
Control: found -1 78.11.0esr-1

Hi Hideki and Jochen,

Thank you for both of your responses.

On Thu, Jul 01, 2021 at 08:08:44AM +0200, Jochen Sprickerhof wrote:
> * Hideki Yamane <hen...@iijmio-mail.jp> [2021-06-28 22:35]:

> > Can you reproduce it on freshly installed bullseye sytem?

After apt upgrade (firefox now at 78.11.0esr-1), the issue is still there. The offending instruction is the same and the backtrace looks very similar. Given that the cause seems well understood (use of NEON instructions on a non-NEON system), I don't think a fresh install would give us any new information.

> I only found this reference for NEON on armhf:
>
> "NEON and VFP/VFP2/VFP3 remain an optional part of the architecture."
>
> https://wiki.debian.org/ArmHardFloatPort#VFP

In addition:

"VFPv3-D16 is the common denominator of the processors to support here (therefore the recommended build option is -mfpu=vfpv3-d16)"

https://wiki.debian.org/ArmHardFloatPort/VfpComparison#FPU

I couldn't find a more authoritative definition of the supported architecture subset for the armhf port.

> If this is still reproducible, I see two options:
> - Disable NEON code.
> - Depend on the neon-support dummy package.

Agreed.

> > > Kernel: Linux 3.5.7-14-ARCH (PREEMPT)
> > It seems that is not the kernel bullseye provides.

Correct. The default Debian armhf kernel doesn't give me video, and I forgot whether it even boots.

> > And it maybe help to provide its hardware information, too.
> The bug author wrote:
>
> > > This is on a Marvell Dove system, with VFPv3-D16. From /proc/cpuinfo:
> > > Features : swp half thumb fastmult vfp edsp iwmmxt thumbee vfpv3 vfpv3d16 tls

More specifically, this is on a SolidRun CuBox (first generation, so not the CuBox-i or CuBox-M).

I noticed that some time ago, the severity of this bug was raised from normal to serious. While it is serious on my system, I had set it to normal because it likely affects only a relatively small number of systems. And while I would appreciate this bug getting resolved, making it release critical seems unnecessary.

Regards,
Vincent.

Emanuele Rocca

unread,
Mar 3, 2023, 9:20:04 AM3/3/23
to
Hi,

On Sun, Feb 14, 2021 at 02:12:17PM +0000, Vincent Arkesteijn wrote:
> Firefox is killed with SIGILL shortly after startup:
> $ firefox-esr -safe-mode
> Illegal instruction
> $

This is due to the fact that some armhf CPUs do not have support for NEON
instructions.

skia used to detect such support at runtime, but that behavior was removed in
https://github.com/google/skia/commit/809ccf37ec836d0df64afd0b13023fd968d505a4

Firefox seems to erroneously enable NEON in places where it shouldn't. Trying
to figure out exactly where and what's the best way to address this.

Meanwhile, to reproduce and debug this issue on a amd64 machine:

apt install debootstrap qemu-user-static binfmt-support schroot

Trying to run a armhf binary on the x86 host will invoke qemu-arm-static, see:

/usr/sbin/update-binfmts --display qemu-arm
ls -l /usr/libexec/qemu-binfmt/arm-binfmt-P

Create a armhf chroot:

debootstrap --arch=armhf sid /srv/sid-armhf
printf '[armhf]\ntype=directory\ndirectory=/srv/sid-armhf\n' >> /etc/schroot/schroot.conf

Install and run firefox-esr in the chroot:

schroot -c armhf
(armhf)root@ariel:/home/ema# apt install --no-install-recommends firefox-esr

Firefox seems to be working:

(armhf)root@ariel:/home/ema# firefox --help | head -1
Usage: firefox-esr [ options ... ] [URL]

The reason why firefox does not crash here is that the default armhf CPU
emulated by qemu-arm-static has NEON support. We can override that by setting
QEMU_CPU to cortex-r5f (which cannot execute NEON instructions) and reproduce
the issue:

(armhf)root@ariel:/home/ema# QEMU_CPU=cortex-r5f firefox
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction

To get a backtrace, install Firefox's debugging symbols in the chroot:

(armhf)root@ariel:/home/ema# echo 'deb http://deb.debian.org/debian-debug sid-debug main' > /etc/apt/sources.list.d/debug.list
(armhf)root@ariel:/home/ema# apt update && apt install firefox-esr-dbgsym

And do the following on the x86 host:

dpkg --add-architecture armhf
apt install libc6:armhf libc6-dbg:armhf gdb-multiarch

LD_LIBRARY_PATH=/srv/sid-armhf/usr/lib/arm-linux-gnueabihf qemu-arm-static -g 1234 -cpu cortex-r5f /srv/sid-armhf/usr/bin/firefox-esr --private-window

In another terminal, again on the host, this should give you a backtrace:

gdb-multiarch -q /srv/sid-armhf/usr/bin/firefox-esr -ex 'set architecture arm' -ex 'target remote :1234' -ex 'set debug-file-directory /srv/sid-armhf/usr/lib/debug' -ex 'set pagination off' -ex 'continue' -ex 'bt' -ex 'continue' -ex 'exit'

Something like:

Program received signal SIGILL, Illegal instruction.
0x37071dc6 in _GLOBAL__sub_I_SkOpts.cpp () from /srv/sid-armhf/usr/lib/firefox-esr/libxul.so
#0 0x37071dc6 in _GLOBAL__sub_I_SkOpts.cpp () from /srv/sid-armhf/usr/lib/firefox-esr/libxul.so
#1 0x3f7d144c in call_init (env=0x3f208340, argv=0x3ffffc94, argc=1, l=<optimized out>) at dl-init.c:70
#2 call_init (l=<optimized out>, argc=1, argv=0x3ffffc94, env=0x3f208340) at dl-init.c:26
#3 0x3f7d14f2 in _dl_init (main_map=0x3f245f00, argc=1, argv=0x3ffffc94, env=0x3f208340) at dl-init.c:117
#4 0x3f56664a in _dl_catch_exception () from /srv/sid-armhf/usr/lib/arm-linux-gnueabihf/libc.so.6
#5 0x3f7d5b60 in dl_open_worker (a=0x3fffd7b0) at dl-open.c:808
#6 0x3f566614 in _dl_catch_exception () from /srv/sid-armhf/usr/lib/arm-linux-gnueabihf/libc.so.6
#7 0x3f7d5da2 in _dl_open (file=0x3fffda64 "/srv/sid-armhf/usr/lib/firefox-esr/libxul.so", mode=-2147483391,
caller_dlopen=0x4000af81 <_start+4560>, nsid=-2, argc=1, argv=0x3ffffc94, env=0x3f208340) at dl-open.c:884
#8 0x3f4d9da0 in ?? () from /srv/sid-armhf/usr/lib/arm-linux-gnueabihf/libc.so.6
0 new messages