Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64

[Sebastian Krzyszkowiak]

Package: libwebkit2gtk-4.0-37
Version: 2.36.3-1~deb11u1
Severity: important
X-Debbugs-Cc: d...@dosowisko.net, te...@security.debian.org

Dear Maintainer,

since 2.36.0-3~deb11u1, webkit2gtk in bullseye has been almost unusable on
arm64 computers, crashing a lot when navigating between pages.

An easy way to reproduce is to navigate to https://news.ycombinator.com/
and click some link (for example, "new" or "login"); or navigate to
https://mobile.twitter.com/ which makes it crash right away. Crashes
are reproducible in both GNOME Web and in MiniBrowser shipped by
webkit2gtk package.

This has been seen on various machines and in downstream distros like PureOS
as well (https://source.puri.sm/Librem5/debs/epiphany/-/issues/38)

Example backtrace:

[🡕] Process 14397 (WebKitNetworkPr) of user 1000 dumped core.

Stack trace of thread 14397:
#0 0x0000ffff914f92ac __GI_raise (libc.so.6 + 0x372ac)
#1 0x0000ffff914e5ea0 __GI_abort (libc.so.6 + 0x23ea0)
#2 0x0000ffff93f89c50 _Z16WTFCrashWithInfoiPKcS0_i (libjavascriptcoregtk-4.0.so.18 + 0x198c50)
#3 0x0000ffff94f2a138 _ZN3WTF10StackTrace17captureStackTraceEii (libjavascriptcoregtk-4.0.so.18 + 0x1139138)
#4 0x0000ffff94f05a30 WTFReleaseLogStackTrace (libjavascriptcoregtk-4.0.so.18 + 0x1114a30)
#5 0x0000ffff97f02988 _ZN7WebCore13internalErrorERKN3WTF3URLE (libwebkit2gtk-4.0.so.37 + 0x214a988)
#6 0x0000ffff966038e0 _ZN6WebKit29NetworkConnectionToWebProcess12preconnectToESt8optionalIN3WTF16ObjectIdentifierIN7WebCore14ResourceLoaderEEEEONS_29NetworkResourceLoadParametersE (libwebkit2gtk-4.0.so.37 + 0x84b8e0)
#7 0x0000ffff964e0eb8 _ZN6WebKit29NetworkConnectionToWebProcess46didReceiveNetworkConnectionToWebProcessMessageERN3IPC10ConnectionERNS1_7DecoderE (libwebkit2gtk-4.0.so.37 + 0x728eb8)
#8 0x0000ffff9676eb70 _ZN3IPC10Connection15dispatchMessageESt10unique_ptrINS_7DecoderESt14default_deleteIS2_EE (libwebkit2gtk-4.0.so.37 + 0x9b6b70)
#9 0x0000ffff9676ef08 _ZN3IPC10Connection26dispatchOneIncomingMessageEv (libwebkit2gtk-4.0.so.37 + 0x9b6f08)
#10 0x0000ffff94f28ad0 _ZN3WTF7RunLoop11performWorkEv (libjavascriptcoregtk-4.0.so.18 + 0x1137ad0)
#11 0x0000ffff94f819f4 _ZZN3WTF7RunLoopC1EvEN3$_18__invokeEPv (libjavascriptcoregtk-4.0.so.18 + 0x11909f4)
#12 0x0000ffff94f80d88 _ZN3WTF7RunLoop3$_08__invokeEP8_GSourcePFiPvES4_ (libjavascriptcoregtk-4.0.so.18 + 0x118fd88)
#13 0x0000ffff91a53ab4 g_main_dispatch (libglib-2.0.so.0 + 0x53ab4)
#14 0x0000ffff91a53e5c g_main_context_iterate (libglib-2.0.so.0 + 0x53e5c)
#15 0x0000ffff91a541b0 g_main_loop_run (libglib-2.0.so.0 + 0x541b0)
#16 0x0000ffff94f81384 _ZN3WTF7RunLoop3runEv (libjavascriptcoregtk-4.0.so.18 + 0x1190384)
#17 0x0000ffff9674287c _ZN6WebKit20AuxiliaryProcessMainINS_22NetworkProcessMainSoupEEEiiPPc (libwebkit2gtk-4.0.so.37 + 0x98a87c)
#18 0x0000ffff914e6218 __libc_start_main (libc.so.6 + 0x24218)
#19 0x0000000000400874 $x (WebKitNetworkProcess + 0x874)
#20 0x0000000000400874 $x (WebKitNetworkProcess + 0x874)

[🡕] Process 14382 (WebKitWebProces) of user 1000 dumped core.

Stack trace of thread 2:
#0 0x0000ffff854082ac __GI_raise (libc.so.6 + 0x372ac)
#1 0x0000ffff853f4ea0 __GI_abort (libc.so.6 + 0x23ea0)
#2 0x0000ffff87e98c50 _Z16WTFCrashWithInfoiPKcS0_i (libjavascriptcoregtk-4.0.so.18 + 0x198c50)
#3 0x0000ffff88e39138 _ZN3WTF10StackTrace17captureStackTraceEii (libjavascriptcoregtk-4.0.so.18 + 0x1139138)
#4 0x0000ffff88e14a30 WTFReleaseLogStackTrace (libjavascriptcoregtk-4.0.so.18 + 0x1114a30)
#5 0x0000ffff8be11988 _ZN7WebCore13internalErrorERKN3WTF3URLE (libwebkit2gtk-4.0.so.37 + 0x214a988)
#6 0x0000ffff8a9c1824 _ZN6WebKit17WebLoaderStrategy30internallyFailedLoadTimerFiredEv (libwebkit2gtk-4.0.so.37 + 0xcfa824)
#7 0x0000ffff88e90aa0 _ZZN3WTF7RunLoop9TimerBaseC1ERS0_EN3$_38__invokeEPv (libjavascriptcoregtk-4.0.so.18 + 0x1190aa0)
#8 0x0000ffff88e8fd88 _ZN3WTF7RunLoop3$_08__invokeEP8_GSourcePFiPvES4_ (libjavascriptcoregtk-4.0.so.18 + 0x118fd88)
#9 0x0000ffff85962ab4 g_main_dispatch (libglib-2.0.so.0 + 0x53ab4)
#10 0x0000ffff85962e5c g_main_context_iterate (libglib-2.0.so.0 + 0x53e5c)
#11 0x0000ffff859631b0 g_main_loop_run (libglib-2.0.so.0 + 0x541b0)
#12 0x0000ffff88e90384 _ZN3WTF7RunLoop3runEv (libjavascriptcoregtk-4.0.so.18 + 0x1190384)
#13 0x0000ffff8aa7b2b4 _ZN6WebKit20AuxiliaryProcessMainINS_17WebProcessMainGtkEEEiiPPc (libwebkit2gtk-4.0.so.37 + 0xdb42b4)
#14 0x0000ffff853f5218 __libc_start_main (libc.so.6 + 0x24218)
#15 0x0000000000400874 $x (WebKitWebProcess + 0x874)
#16 0x0000000000400874 $x (WebKitWebProcess + 0x874)

-- System Information:
Debian Release: 11.4
APT prefers stable
APT policy: (990, 'stable'), (500, 'stable-updates'), (500, 'stable-security')
Architecture: arm64 (aarch64)
Foreign Architectures: armhf

Kernel: Linux 5.15.52-rockchip64 (SMP w/6 CPU threads; PREEMPT)
Kernel taint flags: TAINT_CRAP
Locale: LANG=pl_PL.UTF-8, LC_CTYPE=pl_PL.UTF-8 (charmap=UTF-8), LANGUAGE=en_US
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages libwebkit2gtk-4.0-37 depends on:
ii bubblewrap 0.4.1-3
ii gstreamer1.0-plugins-base 1.18.4-2
ii gstreamer1.0-plugins-good 1.18.4-2
ii libatk1.0-0 2.36.0-2
ii libc6 2.31-13+deb11u3
ii libcairo-gobject2 1.16.0-5
ii libcairo2 1.16.0-5
ii libegl1 1.3.2-1
ii libenchant-2-2 2.2.15-1
ii libfontconfig1 2.13.1-4.2
ii libfreetype6 2.10.4+dfsg-1+deb11u1
ii libgcc-s1 10.2.1-6
ii libgcrypt20 1.8.7-6
ii libgdk-pixbuf-2.0-0 2.42.2+dfsg-1
ii libgles2 1.3.2-1
ii libglib2.0-0 2.66.8-1
ii libgpg-error0 1.38-2
ii libgstreamer-gl1.0-0 1.18.4-2
ii libgstreamer-plugins-base1.0-0 1.18.4-2
ii libgstreamer1.0-0 1.18.4-2.1
ii libgtk-3-0 3.24.24-4+deb11u2
ii libharfbuzz-icu0 2.7.4-1
ii libharfbuzz0b 2.7.4-1
ii libhyphen0 2.8.8-7
ii libicu67 67.1-7
ii libjavascriptcoregtk-4.0-18 2.36.3-1~deb11u1
ii libjpeg62-turbo 1:2.0.6-4
ii liblcms2-2 2.12~rc1-2
ii libmanette-0.2-0 0.2.5-1
ii libnotify4 0.7.9-3
ii libopenjp2-7 2.4.0-3
ii libpango-1.0-0 1.48.2-1
ii libpangocairo-1.0-0 1.48.2-1
ii libpng16-16 1.6.37-3
ii libseccomp2 2.5.1-1+deb11u1
ii libsecret-1-0 0.20.4-2
ii libsoup2.4-1 2.72.0-2
ii libsqlite3-0 3.34.1-3
ii libstdc++6 10.2.1-6
ii libsystemd0 247.3-7
ii libtasn1-6 4.16.0-2
ii libwayland-client0 1.18.0-2~exp1.1
ii libwayland-egl1 1.18.0-2~exp1.1
ii libwayland-server0 1.18.0-2~exp1.1
ii libwebp6 0.6.1-2.1
ii libwebpdemux2 0.6.1-2.1
ii libwoff1 1.0.2-1+b1
ii libwpe-1.0-1 1.10.0-2
ii libwpebackend-fdo-1.0-1 1.8.0-1
ii libx11-6 2:1.7.2-1
ii libxcomposite1 1:0.4.5-1
ii libxdamage1 1:1.1.5-2
ii libxml2 2.9.10+dfsg-6.7+deb11u2
ii libxrender1 1:0.9.10-1
ii libxslt1.1 1.1.34-4
ii libxt6 1:1.2.0-1
ii xdg-dbus-proxy 0.1.2-2
ii zlib1g 1:1.2.11.dfsg-2+deb11u1

Versions of packages libwebkit2gtk-4.0-37 recommends:
ii gstreamer1.0-gl 1.18.4-2
ii gstreamer1.0-libav 1.18.4-3
ii gstreamer1.0-plugins-bad 1.18.4-3
ii libgl1-mesa-dri 21.2.1-2
ii xdg-desktop-portal-gtk 1.8.0-1

Versions of packages libwebkit2gtk-4.0-37 suggests:
pn gstreamer1.0-alsa <none>

-- no debconf information

[Sebastian Krzyszkowiak]

For the record, I've just realized that the system I reported it from
was lacking bullseye-security updates, but I've checked it now on a
fully up-to-date arm64 system with webkit2gtk 2.36.4-1~deb11u1 and
it's still crashing in the same way.

[Alberto Garcia]

On Sun, Aug 07, 2022 at 10:24:45PM +0200, Sebastian Krzyszkowiak wrote:

> since 2.36.0-3~deb11u1, webkit2gtk in bullseye has been almost
> unusable on arm64 computers, crashing a lot when navigating between
> pages.

Hello,

I'm going to publish 2.36.6-1~deb11u1 this week. It contains a number
of crash fixes and I would appreciate if you could verify whether the
problem still happens with it.

Thanks,

Berto

[Sebastian Krzyszkowiak]

Hi Berto,

I've just tried 2.36.6-1~deb11u1 and it still crashes the same way.

Note that these crashes seem exclusive to backports; as far as I'm
aware they don't happen on bookworm at all.

Cheers,
Sebastian

[Alberto Garcia]

Control: forwarded -1 https://bugs.webkit.org/show_bug.cgi?id=244026

On Wed, Aug 17, 2022 at 01:01:46AM +0200, Sebastian Krzyszkowiak wrote:
> > I'm going to publish 2.36.6-1~deb11u1 this week. It contains a number
> > of crash fixes and I would appreciate if you could verify whether the
> > problem still happens with it.
> >

> I've just tried 2.36.6-1~deb11u1 and it still crashes the same way.
>
> Note that these crashes seem exclusive to backports; as far as I'm
> aware they don't happen on bookworm at all.

Thanks, I just forwarded the bug upstream, I'll try to reproduce it
myself this or next week.

Berto

[Dominique MARTINET]

Hi,

Alberto Garcia wrote on Wed, Aug 17, 2022 at 10:31:22AM +0000:
> Thanks, I just forwarded the bug upstream, I'll try to reproduce it
> myself this or next week.

I've also been observing similar crashes on aarch64 bullseye package
and using bookworm is not an option for me (thanks to proprietary
drivers...)


The traces are slightly different:

/usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitNetworkProcess
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000ffff7cfdfaa0 in __GI_abort () at abort.c:79
#2 0x0000ffff7fa8ac50 in WTFCrashWithInfo(int, char const*, char const*, int) () at WTF/Headers/wtf/Assertions.h:741
#3 0x0000ffff80a2d5a8 in captureStackTrace () at ../Source/WTF/wtf/StackTrace.cpp:79
#4 0x0000ffff80a08ea0 in WTFReleaseLogStackTrace () at ../Source/WTF/wtf/Assertions.cpp:592
#5 0x0000ffff83c06550 in internalError () at ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97
#6 0x0000ffff820e8d1c in preconnectTo () at ../Source/WebKit/NetworkProcess/NetworkConnectionToWebProcess.cpp:735
#7 0x0000ffff81fc62f4 in callMemberFunctionImpl<WebKit::NetworkConnectionToWebProcess, void (WebKit::NetworkConnectionToWebProcess::*)(std::optional<WTF::ObjectIdentifier<WebCore::ResourceLoader> >, WebKit::NetworkResourceLoadParameters&&), std::tuple<std::optional<WTF::ObjectIdentifier<WebCore::ResourceLoader> >, WebKit::NetworkResourceLoadParameters>, 0, 1> () at ../Source/WebKit/Platform/IPC/HandleMessage.h:125
#8 callMemberFunction<WebKit::NetworkConnectionToWebProcess, void (WebKit::NetworkConnectionToWebProcess::*)(std::optional<WTF::ObjectIdentifier<WebCore::ResourceLoader> >, WebKit::NetworkResourceLoadParameters&&), std::tuple<std::optional<WTF::ObjectIdentifier<WebCore::ResourceLoader> >, WebKit::NetworkResourceLoadParameters>, std::integer_sequence<unsigned long, 0, 1> > () at ../Source/WebKit/Platform/IPC/HandleMessage.h:131
#9 handleMessage<Messages::NetworkConnectionToWebProcess::PreconnectTo, WebKit::NetworkConnectionToWebProcess, void (WebKit::NetworkConnectionToWebProcess::*)(std::optional<WTF::ObjectIdentifier<WebCore::ResourceLoader> >, WebKit::NetworkResourceLoadParameters&&)> () at ../Source/WebKit/Platform/IPC/HandleMessage.h:196
#10 didReceiveNetworkConnectionToWebProcessMessage () at DerivedSources/WebKit/NetworkConnectionToWebProcessMessageReceiver.cpp:479
#11 0x0000ffff822543d0 in dispatchMessage () at ../Source/WebKit/Platform/IPC/Connection.cpp:1134
#12 0x0000ffff82254768 in dispatchOneIncomingMessage () at ../Source/WebKit/Platform/IPC/Connection.cpp:1203
#13 0x0000ffff80a2bf40 in operator() () at ../Source/WTF/wtf/Function.h:82
#14 performWork () at ../Source/WTF/wtf/RunLoop.cpp:133
#15 0x0000ffff80a85190 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:80
#16 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:79
#17 0x0000ffff80a84524 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:53
#18 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:45
#19 0x0000ffff7d551ab4 in g_main_context_dispatch () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#20 0x0000ffff7d551e5c in ?? () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#21 0x0000ffff7d5521b0 in g_main_loop_run () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#22 0x0000ffff80a84b20 in run () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:108
#23 0x0000ffff822280d8 in run () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:70
#24 AuxiliaryProcessMain<WebKit::NetworkProcessMainSoup> () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:96
#25 0x0000ffff7cfdfe18 in __libc_start_main (main=0x400878 <__wrap_main>, argc=3, argv=0xfffff1c90058, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#26 0x0000000000400874 in _start ()


/usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitWebProcess
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000ffff99831aa0 in __GI_abort () at abort.c:79
#2 0x0000ffff9c2dcc50 in WTFCrashWithInfo(int, char const*, char const*, int) () at WTF/Headers/wtf/Assertions.h:741
#3 0x0000ffff9d27f5a8 in captureStackTrace () at ../Source/WTF/wtf/StackTrace.cpp:79
#4 0x0000ffff9d25aea0 in WTFReleaseLogStackTrace () at ../Source/WTF/wtf/Assertions.cpp:592
#5 0x0000ffffa0458550 in internalError () at ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97
#6 0x0000ffff9edead30 in internallyFailedLoadTimerFired () at ../Source/WebKit/WebProcess/Network/WebLoaderStrategy.cpp:495
#7 0x0000ffff9d2d723c in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:177
#8 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:169
#9 0x0000ffff9d2d6524 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:53
#10 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:45
#11 0x0000ffff99da3ab4 in g_main_context_dispatch () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#12 0x0000ffff99da3e5c in ?? () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#13 0x0000ffff99da41b0 in g_main_loop_run () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#14 0x0000ffff9d2d6b20 in run () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:108
#15 0x0000ffff9eea47c4 in run () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:70
#16 AuxiliaryProcessMain<WebKit::WebProcessMainGtk> () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:96
#17 0x0000ffff99831e18 in __libc_start_main (main=0x400878 <__wrap_main>, argc=3, argv=0xfffff7b85168, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#18 0x0000000000400874 in _start ()



hopefully the root of the problem is the same and having debug symbols
installed will help pinpoint what's wrong.

(a likely reason it's different is I run with WEBKIT_FORCE_SANDBOX=0
because the sandbox forbids access to GPU driver /dev non-standard
node... but honestly at this point it could be completely different,
I wasn't sure if I should open a new bug report or not :/)

I've downloaded the sources and will start looking into it as free time
permits.


Thanks,
--
Dominique Martinet

[Alberto Garcia]

On Fri, Sep 16, 2022 at 10:18:32AM +0900, Dominique MARTINET wrote:

> Alberto Garcia wrote on Wed, Aug 17, 2022 at 10:31:22AM +0000:
> > Thanks, I just forwarded the bug upstream, I'll try to reproduce it
> > myself this or next week.
>
> I've also been observing similar crashes on aarch64 bullseye package
> and using bookworm is not an option for me (thanks to proprietary
> drivers...)
>
>
> The traces are slightly different:

Sorry for the late reply, and thanks! I think the traces are useful,
I added them to the upstream bug report and I'll try to discuss them
with the development team.

By the way 2.38.0 just came out, it'll probably be backported soon to
bullseye.

Berto

[Alberto Garcia]

On Fri, Sep 16, 2022 at 10:18:32AM +0900, Dominique MARTINET wrote:

> The traces are slightly different:
>
> /usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitNetworkProcess
> (gdb) bt
> #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> #1 0x0000ffff7cfdfaa0 in __GI_abort () at abort.c:79
> #2 0x0000ffff7fa8ac50 in WTFCrashWithInfo(int, char const*, char const*, int) () at WTF/Headers/wtf/Assertions.h:741
> #3 0x0000ffff80a2d5a8 in captureStackTrace () at ../Source/WTF/wtf/StackTrace.cpp:79
> #4 0x0000ffff80a08ea0 in WTFReleaseLogStackTrace () at ../Source/WTF/wtf/Assertions.cpp:592
> #5 0x0000ffff83c06550 in internalError () at ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97
> #6 0x0000ffff820e8d1c in preconnectTo () at ../Source/WebKit/NetworkProcess/NetworkConnectionToWebProcess.cpp:735

Hello Dominique,

do you think you can try this patch?

https://bug-244026-attachments.webkit.org/attachment.cgi?id=462492&action=diff&format=raw&headers=1

If you have problems / are unsure about how to build WebKit I can
provide packages for you. Just let me know.

Berto

[Dominique MARTINET]

Hi Alberto,

Alberto Garcia wrote on Wed, Sep 21, 2022 at 12:54:22PM +0200:
> On Fri, Sep 16, 2022 at 10:18:32AM +0900, Dominique MARTINET wrote:
> > The traces are slightly different:
> >
> > /usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitNetworkProcess
> > (gdb) bt
> > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> > #1 0x0000ffff7cfdfaa0 in __GI_abort () at abort.c:79
> > #2 0x0000ffff7fa8ac50 in WTFCrashWithInfo(int, char const*, char const*, int) () at WTF/Headers/wtf/Assertions.h:741
> > #3 0x0000ffff80a2d5a8 in captureStackTrace () at ../Source/WTF/wtf/StackTrace.cpp:79
> > #4 0x0000ffff80a08ea0 in WTFReleaseLogStackTrace () at ../Source/WTF/wtf/Assertions.cpp:592
> > #5 0x0000ffff83c06550 in internalError () at ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97
> > #6 0x0000ffff820e8d1c in preconnectTo () at ../Source/WebKit/NetworkProcess/NetworkConnectionToWebProcess.cpp:735
>

> do you think you can try this patch?
>
> https://bug-244026-attachments.webkit.org/attachment.cgi?id=462492&action=diff&format=raw&headers=1

Thanks!

> If you have problems / are unsure about how to build WebKit I can
> provide packages for you. Just let me know.

I'm definitely unsure about how to build WebKit, but that can be
fixed... I might need to find somewhere with a bit more memory though,
been trying to build the 2.38 release yesterday and build got OOM killed
even with -j1 on a VM with 6GB of ram :/

If you have time to build a package you might be faster than me, so I'd
appreciate it :)

--
Dominique

[Alberto Garcia]

On Thu, Sep 22, 2022 at 07:45:29AM +0900, Dominique MARTINET wrote:

> If you have time to build a package you might be faster than me, so
> I'd appreciate it :)

Hi, I built two sets of packages, the unpatched vanilla version of
2.38.0 and the one with the patch I mentioned in my previous comment:

https://people.debian.org/~berto/files/webkitgtk-2.38.0-arm64/

Can you test them both and tell me if things get better?

Thanks!

Berto

[Dominique MARTINET]

Hi,

Alberto Garcia wrote on Fri, Sep 23, 2022 at 12:38:58AM +0200:
> I built two sets of packages, the unpatched vanilla version of
> 2.38.0 and the one with the patch I mentioned in my previous comment:
>
> https://people.debian.org/~berto/files/webkitgtk-2.38.0-arm64/
>
> Can you test them both and tell me if things get better?

Thanks!

You were indeed faster than me... But it looks like these packages were
built for debian testing/bookworm (require libc6 >= 2.34), so I cannot
test easily due to proprietary libGL and other drivers woes.

(I actually do have some frankendebian version for testing, I'll give it
a try on Monday, but this will not necessarily have the same problems as
what I had reported here on stable/bullseye)


I should get access to a beefier machine at work and will resume my
building attempts there, and report with both tests (the packages you
provided on bookworm and stable package with patch) hopefully by the end
of Monday.

--
Dominique

[Alberto Garcia]

On Sun, Sep 25, 2022 at 01:40:32PM +0900, Dominique MARTINET wrote:

> > https://people.debian.org/~berto/files/webkitgtk-2.38.0-arm64/

> You were indeed faster than me... But it looks like these packages were
> built for debian testing/bookworm (require libc6 >= 2.34), so I cannot
> test easily due to proprietary libGL and other drivers woes.

Argh, you're right, my fault!!

I just uploaded the packages built for bullseye, same URL, the patched
ones are available already, the unpatched ones soon.

Berto

[Dominique MARTINET]

Alberto Garcia wrote on Mon, Sep 26, 2022 at 12:00:04AM +0000:
> I just uploaded the packages built for bullseye, same URL, the patched
> ones are available already, the unpatched ones soon.

Thanks!

I've tried the patched variant, it seems to fail in the same place?

Here's the new trace with adjusted lines:
----
/usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitNetworkProcess

#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

#1 0x0000ffff817b3aa0 in __GI_abort () at abort.c:79
#2 0x0000ffff8427958c in WTFCrashWithInfo(int, char const*, char const*, int) () at WTF/Headers/wtf/Assertions.h:754
#3 0x0000ffff8525db20 in captureStackTrace () at ../Source/WTF/wtf/StackTrace.cpp:79
#4 0x0000ffff85232d18 in WTFReleaseLogStackTrace () at ../Source/WTF/wtf/Assertions.cpp:602
#5 0x0000ffff884e8934 in internalError () at ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97
#6 0x0000ffff869e6c14 in preconnectTo () at ../Source/WebKit/NetworkProcess/NetworkConnectionToWebProcess.cpp:733
#7 0x0000ffff8689baf8 in callMemberFunctionImpl<WebKit::NetworkConnectionToWebProcess, void (WebKit::NetworkConnectionToWebProcess::*)(std::optional<WTF::ObjectIdentifier<WebCore::ResourceLoader> >, WebKit::NetworkResourceLoadParameters&&), std::tuple<std::optional<WTF::ObjectIdentifier<WebCore::ResourceLoader> >, WebKit::NetworkResourceLoadParameters>, 0, 1> () at ../Source/WebKit/Platform/IPC/HandleMessage.h:131
#8 callMemberFunction<WebKit::NetworkConnectionToWebProcess, void (WebKit::NetworkConnectionToWebProcess::*)(std::optional<WTF::ObjectIdentifier<WebCore::ResourceLoader> >, WebKit::NetworkResourceLoadParameters&&), std::tuple<std::optional<WTF::ObjectIdentifier<WebCore::ResourceLoader> >, WebKit::NetworkResourceLoadParameters>, std::integer_sequence<unsigned long, 0, 1> > () at ../Source/WebKit/Platform/IPC/HandleMessage.h:137
#9 handleMessage<Messages::NetworkConnectionToWebProcess::PreconnectTo, WebKit::NetworkConnectionToWebProcess, void (WebKit::NetworkConnectionToWebProcess::*)(std::optional<WTF::ObjectIdentifier<WebCore::ResourceLoader> >, WebKit::NetworkResourceLoadParameters&&)> () at ../Source/WebKit/Platform/IPC/HandleMessage.h:259
#10 didReceiveNetworkConnectionToWebProcessMessage () at DerivedSources/WebKit/NetworkConnectionToWebProcessMessageReceiver.cpp:357
#11 0x0000ffff86b3ff1c in dispatchMessage () at ../Source/WebKit/Platform/IPC/Connection.cpp:1150
#12 0x0000ffff86b402b4 in dispatchOneIncomingMessage () at ../Source/WebKit/Platform/IPC/Connection.cpp:1219
#13 0x0000ffff8525c4b0 in operator() () at ../Source/WTF/wtf/Function.h:82

#14 performWork () at ../Source/WTF/wtf/RunLoop.cpp:133

#15 0x0000ffff852b5570 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:80

#16 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:79

#17 0x0000ffff852b48f4 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:53

#18 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:45

#19 0x0000ffff81d25ab4 in g_main_context_dispatch () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#20 0x0000ffff81d25e5c in ?? () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#21 0x0000ffff81d261b0 in g_main_loop_run () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#22 0x0000ffff852b4ef8 in run () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:108
#23 0x0000ffff86b11004 in run () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:71
#24 AuxiliaryProcessMain<WebKit::NetworkProcessMainSoup> () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:97
#25 0x0000ffff817b3e18 in __libc_start_main (main=0x400878 <__wrap_main>, argc=3, argv=0xffffcfa17918, init=<optimized out>, fini=<optimized out>,

rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#26 0x0000000000400874 in _start ()



/usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitWebProcess

#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

#1 0x0000ffffa0afdaa0 in __GI_abort () at abort.c:79
#2 0x0000ffffa35c358c in WTFCrashWithInfo(int, char const*, char const*, int) () at WTF/Headers/wtf/Assertions.h:754
#3 0x0000ffffa45a7b20 in captureStackTrace () at ../Source/WTF/wtf/StackTrace.cpp:79
#4 0x0000ffffa457cd18 in WTFReleaseLogStackTrace () at ../Source/WTF/wtf/Assertions.cpp:602
#5 0x0000ffffa7832934 in internalError () at ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97
#6 0x0000ffffa61e6f6c in internallyFailedLoadTimerFired () at ../Source/WebKit/WebProcess/Network/WebLoaderStrategy.cpp:503
#7 0x0000ffffa45ff61c in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:177

#8 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:169

#9 0x0000ffffa45fe8f4 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:53

#10 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:45

#11 0x0000ffffa106fab4 in g_main_context_dispatch () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#12 0x0000ffffa106fe5c in ?? () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#13 0x0000ffffa10701b0 in g_main_loop_run () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#14 0x0000ffffa45feef8 in run () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:108
#15 0x0000ffffa62a1aac in run () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:71
#16 AuxiliaryProcessMain<WebKit::WebProcessMainGtk> () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:97
#17 0x0000ffffa0afde18 in __libc_start_main (main=0x400878 <__wrap_main>, argc=3, argv=0xffffc73c1cc8, init=<optimized out>, fini=<optimized out>,

rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#18 0x0000000000400874 in _start ()

----

Since the patched variant does not resolve the problem I'm not going to
bother with the unpatched -- please ask if you'd like me to try anyway.


Meanwhile I'm continuing my quest to build webkit, pdebuild failed on an
intermediate ldd command because of improper argument escaping, I'll
retry out of the debian build chroot this time so I can iterate
faster...


Thanks!
--
Dominique

[Alberto Garcia]

On Mon, Sep 26, 2022 at 09:21:45AM +0900, Dominique MARTINET wrote:
> Alberto Garcia wrote on Mon, Sep 26, 2022 at 12:00:04AM +0000:
> > I just uploaded the packages built for bullseye, same URL, the patched
> > ones are available already, the unpatched ones soon.
>
> Thanks!
>
> I've tried the patched variant, it seems to fail in the same place?

Ok, thanks. I uploaded the unpatched version, although I suspect it's
not going to make any difference.

Berto

[Dominique MARTINET]

Alberto Garcia wrote on Mon, Sep 26, 2022 at 07:44:24AM +0000:
> > I've tried the patched variant, it seems to fail in the same place?
>
> Ok, thanks. I uploaded the unpatched version, although I suspect it's
> not going to make any difference.

I agree it shouldn't make any difference : I tried anyway, and can
confirm I hit the same trace.

I've also finally finished building yesterday (the last failure I was
referring to wasn't the badly escaped argument -- that was just a
warning -- but ldd segfaulting on qemu-user aarch64 on the webkit so,
presumably because it is too big? I've replaced it with a wrapper around
lddtree and could finish building...), but unfortunately I run into a
different crash before I can test this; I suspect this is due to my
build options, I will try to get closer to the deb package options...
(Or I guess I could actually build the deb package now I have ldd figured
out, but allowing partial builds would be great for actual development...)


I'll report again when/if I make progress, thanks for the follow up and
packages :)

--
Dominique

[Dominique MARTINET]

Dominique MARTINET wrote on Tue, Sep 27, 2022 at 10:10:40AM +0900:
> (Or I guess I could actually build the deb package now I have ldd figured
> out, but allowing partial builds would be great for actual development...)

hmpf!

So I have rebuilt as a package, and it .. just works?

At this point I'm starting to think the build environment matters quite
a bit, would you mind comparing with what I'm doing:

- I do not have any beefy aarch64 machine, so building from a x86_64
host with qemu-user-static and binfmt

- in order to get the sysroot I just run inside a container:
podman run docker.io/arm64v8/debian:bullseye
(using a volume for build data)

- in that container I intalled all required dependencies with apt
build-dep in build tree, I don't think there is anything else, so it
should be similar to pdebuild

- there is a bug with ldd and qemu user: I replace ldd with the
following script
---
#!/bin/sh

lddtree "$@" 2>/dev/null \
| sed -e '/^[^ ]/d' -e 's/^ */\t/' \
-e 's/ld-linux-aarch64.so.1.*/ld-linux-aarch64.so.1/' \
-e 's/$/ (0x0000005501acc000)/'
---
This is used by g-ir-scanner afaiu; build fails for me without it.

- I used the upstream tarball
https://webkitgtk.org/releases/webkitgtk-2.38.0.tar.xz

- I used the debian directory from sid, with a few adjustments:
http://deb.debian.org/debian/pool/main/w/webkit2gtk/webkit2gtk_2.38.0-2.debian.tar.xz
--------
diff -ur orig/debian/patches/series webkitgtk-2.38.0/debian/patches/series
--- orig/debian/patches/series 2022-09-26 18:25:41.000000000 +0900
+++ webkitgtk-2.38.0/debian/patches/series 2022-09-27 10:27:48.811885275 +0900
@@ -5,3 +5,4 @@
dont-detect-sse2.patch
reduce-memory-overheads.patch
fix-ftbfs-hurd.patch
+fix-bullseye.patch
diff -ur orig/debian/rules webkitgtk-2.38.0/debian/rules
--- orig/debian/rules 2022-09-26 18:25:41.000000000 +0900
+++ webkitgtk-2.38.0/debian/rules 2022-09-27 11:09:31.391250044 +0900
@@ -12,8 +12,8 @@
# soup3 build -> 4.1 API packages
# gtk4 build -> 5.0 API packages
ENABLE_SOUP2=YES
-ENABLE_SOUP3=YES
-ENABLE_GTK4=YES
+ENABLE_SOUP3=NO
+ENABLE_GTK4=NO
# Set to YES to use the docs from the tarball instead of generating them.
# Use only if gi-docgen is not available.
USE_PREBUILT_DOCS=NO
diff -ur orig/debian/control.in webkitgtk-2.38.0/debian/control.in
--- orig/debian/control.in 2022-09-26 18:25:41.000000000 +0900
+++ webkitgtk-2.38.0/debian/control.in 2022-09-27 11:11:07.441672820 +0900
@@ -51,7 +51,7 @@
libegl1-mesa-dev,
libgl-dev,
libgles-dev
-Build-Depends-Indep: gi-docgen, libglib2.0-doc, libgtk-3-doc, libsoup2.4-doc
+Build-Depends-Indep: libglib2.0-doc, libgtk-3-doc, libsoup2.4-doc
Standards-Version: 4.6.1.0
Rules-Requires-Root: no
Vcs-Browser: https://salsa.debian.org/webkit-team/webkit
---------

+ fix-bullseye.patch as per link
---------
--- a/Source/WebKit/UIProcess/API/glib/WebKitWebsiteDataManager.cpp
+++ a/Source/WebKit/UIProcess/API/glib/WebKitWebsiteDataManager.cpp
@@ -498,6 +498,9 @@ WebKit::WebsiteDataStore& webkitWebsiteDataManagerGetDataStore(WebKitWebsiteData
configuration->setServiceWorkerRegistrationDirectory(FileSystem::stringFromFileSystemRepresentation(priv->swRegistrationsDirectory.get()));
if (priv->domCacheDirectory)
configuration->setCacheStorageDirectory(FileSystem::stringFromFileSystemRepresentation(priv->domCacheDirectory.get()));
+#if !ENABLE(SERVER_PRECONNECT)
+ configuration->setAllowsServerPreconnect(false);
+#endif
priv->websiteDataStore = WebKit::WebsiteDataStore::create(WTFMove(configuration), PAL::SessionID::generatePersistentSessionID());
priv->websiteDataStore->setIgnoreTLSErrors(priv->tlsErrorsPolicy == WEBKIT_TLS_ERRORS_POLICY_IGNORE);
}
---------

- I built without documentation because no gi-docgen:
DEB_BUILD_OPTIONS=nodoc dpkg-buildpackage


With that package, epiphany works just fine.


I will rebuild the package again without the patch, to check if I can
reproduce the issue without it in my build environment.

That takes quite a while, so will probably only report back tomorrow.


Thanks,
--
Dominique

[Dominique MARTINET]

Hello,

Dominique MARTINET wrote on Tue, Sep 27, 2022 at 01:58:40PM +0900:
> I will rebuild the package again without the patch, to check if I can
> reproduce the issue without it in my build environment.
>
> That takes quite a while, so will probably only report back tomorrow.

So I can confirm the same package as sid's 2.38.0-2 without any extra
patch works just fine if I recompile it -- so it must be a difference in
our build environment.

Happy to send you the list of packages I have installed or anything else
that might be useful to track this down further!


(As a side note, it turns out that even when this does works, gdk opengl
context creation fails on my system because it apparently always try to
initialize the context as opengl first regardless of GLES setting so
this isn't as good as I was hoping... In particular the fallback code
rotates the rendered pixels by 180° for some reason so it's upside down,
but removing that as well is perfect if just a bit slow.
Hopefully I can figure that out next... Did I already say I hate closed
drivers?
But anyway that's unrelated to this bug: epiphany (and MiniBrowser) work
just fine at this point; and I'll open a bug upstream directly for the
fallback code)

--
Dominique

[Alberto Garcia]

On Wed, Sep 28, 2022 at 08:58:28AM +0900, Dominique MARTINET wrote:

> > I will rebuild the package again without the patch, to check if I can
> > reproduce the issue without it in my build environment.
> >
> > That takes quite a while, so will probably only report back tomorrow.
>
> So I can confirm the same package as sid's 2.38.0-2 without any
> extra patch works just fine if I recompile it -- so it must be a
> difference in our build environment.

Hmmm... ok, this may be a bug in the compiler. The official Debian
builds are made using clang (because of bug #1008098), did you use
gcc?

> Happy to send you the list of packages I have installed or anything
> else that might be useful to track this down further!

Yes, that would be helpful.

Berto

[Sebastian Krzyszkowiak]

On Tue, Sep 20, 2022 at 5:44 PM Alberto Garcia <be...@igalia.com> wrote:
> By the way 2.38.0 just came out, it'll probably be backported soon to
> bullseye.

Hi Alberto,

the patch that sets setAllowsServerPreconnect does not fix the
original issue, however, this one does:
https://github.com/WebKit/WebKit/pull/4790

With this applied on top of your recent backport of 2.38.0, WebKit
becomes somewhat usable on arm64 in bullseye. It still crashes on some
pages (such as Twitter), but in a different way (somewhere in JSC)
which probably deserves a separate issue.

Cheers,
Sebastian

[Dominique MARTINET]

Alberto Garcia wrote on Thu, Sep 29, 2022 at 11:19:04AM +0000:
> On Wed, Sep 28, 2022 at 08:58:28AM +0900, Dominique MARTINET wrote:
> > So I can confirm the same package as sid's 2.38.0-2 without any
> > extra patch works just fine if I recompile it -- so it must be a
> > difference in our build environment.
>
> Hmmm... ok, this may be a bug in the compiler. The official Debian
> builds are made using clang (because of bug #1008098), did you use
> gcc?

Hm, that is not specified in the control or rule file?

I've triggered a fresh build to check and can confirm it's using gcc...
Interesting that my build didn't fail on this (well, I guess ldd
segfault was just as bad :-D)

I've had a quick look but wasn't able to figure how to build with clang
to check -- would building the package with clang in CC/CXX suffice,
e.g. `CC=clang CXX=clang++ dpkg-buildpackage` ?
(that appears to do the job, I'll send a new mail on Monday if build
didn't finish in time to test today)

> > Happy to send you the list of packages I have installed or anything
> > else that might be useful to track this down further!
>
> Yes, that would be helpful.

Here's the full list (no more useful text after it):

adduser 3.118
adwaita-icon-theme 3.38.0-1
apt 2.2.4
at-spi2-core 2.38.0-4
autoconf 2.69-14
automake 1:1.16.3-2
autopoint 0.21-4
autotools-dev 20180224.1+nmu1
base-files 11.1+deb11u5
base-passwd 3.5.51
bash 5.1-2+deb11u1
binutils 2.35.2-2
binutils-aarch64-linux-gnu 2.35.2-2
binutils-common:arm64 2.35.2-2
bison 2:3.7.5+dfsg-1
bsdextrautils 2.36.1-8+deb11u1
bsdutils 1:2.36.1-8+deb11u1
bubblewrap 0.4.1-3
build-essential 12.9
bzip2 1.0.8-4
ca-certificates 20210119
ccache 4.2-1
clang 1:11.0-51+nmu5
clang-11 1:11.0.1-2
cmake 3.18.4-2+deb11u1
cmake-data 3.18.4-2+deb11u1
coreutils 8.32-4
cpp 4:10.2.1-1
cpp-10 10.2.1-6
dash 0.5.11+git20200708+dd9ef66-5
dbus 1.12.20-2
dbus-user-session 1.12.20-2
dconf-gsettings-backend:arm64 0.38.0-2
dconf-service 0.38.0-2
debconf 1.5.77
debhelper 13.3.4
debian-archive-keyring 2021.1.1
debianutils 4.11.2
dh-autoreconf 20
dh-strip-nondeterminism 1.12.0-1
dictionaries-common 1.28.4
diffutils 1:3.7-5
dirmngr 2.2.27-2+deb11u2
dmsetup 2:1.02.175-2.1
docbook 4.5-6
docbook-to-man 1:2.0.0-45
docbook-xml 4.5-9
docbook-xsl 1.79.2+dfsg-1
dpkg 1.20.12
dpkg-dev 1.20.12
dwz 0.13+20210201-1
e2fsprogs 1.46.2-2
eatmydata 105-9
emacsen-common 3.0.4
fakeroot 1.25.3-1.1
file 1:5.39-3
findutils 4.8.0-1
flex 2.6.4-8
fontconfig 2.13.1-4.2
fontconfig-config 2.13.1-4.2
fonts-cantarell 0.111-3
fonts-dejavu-core 2.37-2
g++ 4:10.2.1-1
g++-10 10.2.1-6
gcc 4:10.2.1-1
gcc-10 10.2.1-6
gcc-10-base:arm64 10.2.1-6
gcc-9-base:arm64 9.3.0-22
gdb 10.1-1.7
gettext 0.21-4
gettext-base 0.21-4
gir1.2-atk-1.0:arm64 2.36.0-2
gir1.2-atspi-2.0:arm64 2.38.0-4
gir1.2-cloudproviders-0.3.0:arm64 0.3.0-3
gir1.2-colord-1.0:arm64 1.4.5-3
gir1.2-freedesktop:arm64 1.66.1-1+b1
gir1.2-gdkpixbuf-2.0:arm64 2.42.2+dfsg-1+deb11u1
gir1.2-glib-2.0:arm64 1.66.1-1+b1
gir1.2-gst-plugins-base-1.0:arm64 1.18.4-2
gir1.2-gstreamer-1.0:arm64 1.18.4-2.1
gir1.2-gtk-3.0:arm64 3.24.24-4+deb11u2
gir1.2-gudev-1.0:arm64 234-1
gir1.2-harfbuzz-0.0:arm64 2.7.4-1
gir1.2-json-1.0:arm64 1.6.2-1
gir1.2-manette-0.2:arm64 0.2.5-1
gir1.2-notify-0.7:arm64 0.7.9-3
gir1.2-pango-1.0:arm64 1.46.2-3
gir1.2-rest-0.7 0.8.1-1.1
gir1.2-rsvg-2.0:arm64 2.50.3+dfsg-1
gir1.2-secret-1:arm64 0.20.4-2
gir1.2-soup-2.4:arm64 2.72.0-2
git 1:2.30.2-1
git-man 1:2.30.2-1
glib-networking:arm64 2.66.0-2
glib-networking-common 2.66.0-2
glib-networking-services 2.66.0-2
gnome-pkg-tools 0.21.2
gnupg 2.2.27-2+deb11u2
gnupg-l10n 2.2.27-2+deb11u2
gnupg-utils 2.2.27-2+deb11u2
gobject-introspection 1.66.1-1+b1
gperf 3.1-1
gpg 2.2.27-2+deb11u2
gpg-agent 2.2.27-2+deb11u2
gpg-wks-client 2.2.27-2+deb11u2
gpg-wks-server 2.2.27-2+deb11u2
gpgconf 2.2.27-2+deb11u2
gpgsm 2.2.27-2+deb11u2
gpgv 2.2.27-2+deb11u2
grep 3.6-1
groff-base 1.22.4-6
gsettings-desktop-schemas 3.38.0-2
gtk-doc-tools 1.33.2-1
gtk-update-icon-cache 3.24.24-4+deb11u2
gzip 1.10-4+deb11u1
hicolor-icon-theme 0.17-2
hostname 3.23
htop 3.0.5-7
hunspell-en-us 1:2019.10.06-1
icu-devtools 67.1-7
init-system-helpers 1.60
intltool-debian 0.35.0+20060710.5
iso-codes 4.6.0-1
jdupes 1.19.1-1
less 551-2
libacl1:arm64 2.2.53-10
libalgorithm-diff-perl 1.201-1
libalgorithm-diff-xs-perl 0.04-6+b1
libalgorithm-merge-perl 0.08-3
libapparmor1:arm64 2.13.6-10
libapt-pkg6.0:arm64 2.2.4
libarchive-zip-perl 1.68-1
libarchive13:arm64 3.4.3-2+deb11u1
libargon2-1:arm64 0~20171227-0.2
libasan6:arm64 10.2.1-6
libaspell15:arm64 0.60.8-3
libassuan0:arm64 2.5.3-7.1
libatk-bridge2.0-0:arm64 2.38.0-1
libatk-bridge2.0-dev:arm64 2.38.0-1
libatk1.0-0:arm64 2.36.0-2
libatk1.0-data 2.36.0-2
libatk1.0-dev:arm64 2.36.0-2
libatk1.0-doc 2.36.0-2
libatomic1:arm64 10.2.1-6
libatspi2.0-0:arm64 2.38.0-4
libatspi2.0-dev:arm64 2.38.0-4
libattr1:arm64 1:2.4.48-6
libaudit-common 1:3.0-2
libaudit1:arm64 1:3.0-2
libavahi-client3:arm64 0.8-5+deb11u1
libavahi-common-data:arm64 0.8-5+deb11u1
libavahi-common3:arm64 0.8-5+deb11u1
libbabeltrace1:arm64 1.5.8-1+b3
libbinutils:arm64 2.35.2-2
libblkid-dev:arm64 2.36.1-8+deb11u1
libblkid1:arm64 2.36.1-8+deb11u1
libboost-regex1.74.0:arm64 1.74.0-9
libbrotli-dev:arm64 1.0.9-2+b2
libbrotli1:arm64 1.0.9-2+b2
libbsd0:arm64 0.11.3-1
libbz2-1.0:arm64 1.0.8-4
libc-bin 2.31-13+deb11u4
libc-dev-bin 2.31-13+deb11u4
libc-devtools 2.31-13+deb11u4
libc6:arm64 2.31-13+deb11u4
libc6-dbg:arm64 2.31-13+deb11u4
libc6-dev:arm64 2.31-13+deb11u4
libcairo-gobject2:arm64 1.16.0-5
libcairo-script-interpreter2:arm64 1.16.0-5
libcairo2:arm64 1.16.0-5
libcairo2-dev:arm64 1.16.0-5
libcairo2-doc 1.16.0-5
libcap-ng0:arm64 0.7.9-2.2+b1
libcap2:arm64 1:2.44-1
libcap2-bin 1:2.44-1
libcbor0:arm64 0.5.0+dfsg-2
libcc1-0:arm64 10.2.1-6
libclang-common-11-dev 1:11.0.1-2
libclang-cpp11 1:11.0.1-2
libclang1-11 1:11.0.1-2
libcloudproviders-dev:arm64 0.3.0-3
libcloudproviders0:arm64 0.3.0-3
libcolord-dev:arm64 1.4.5-3
libcolord2:arm64 1.4.5-3
libcom-err2:arm64 1.46.2-2
libcrypt-dev:arm64 1:4.4.18-4
libcrypt1:arm64 1:4.4.18-4
libcryptsetup12:arm64 2:2.3.7-1+deb11u1
libctf-nobfd0:arm64 2.35.2-2
libctf0:arm64 2.35.2-2
libcups2:arm64 2.3.3op2-3+deb11u2
libcups2-dev:arm64 2.3.3op2-3+deb11u2
libcupsimage2:arm64 2.3.3op2-3+deb11u2
libcupsimage2-dev:arm64 2.3.3op2-3+deb11u2
libcurl3-gnutls:arm64 7.74.0-1.3+deb11u3
libcurl4:arm64 7.74.0-1.3+deb11u3
libdatrie-dev:arm64 0.2.13-1
libdatrie1:arm64 0.2.13-1
libdazzle-1.0-0:arm64 3.38.0-1
libdb5.3:arm64 5.3.28+dfsg1-0.8
libdbus-1-3:arm64 1.12.20-2
libdbus-1-dev:arm64 1.12.20-2
libdconf1:arm64 0.38.0-2
libdebconfclient0:arm64 0.260
libdebhelper-perl 13.3.4
libdebuginfod1:arm64 0.183-1
libdeflate-dev:arm64 1.7-1
libdeflate0:arm64 1.7-1
libdevmapper1.02.1:arm64 2:1.02.175-2.1
libdpkg-perl 1.20.12
libdrm-amdgpu1:arm64 2.4.104-1
libdrm-common 2.4.104-1
libdrm-nouveau2:arm64 2.4.104-1
libdrm-radeon1:arm64 2.4.104-1
libdrm2:arm64 2.4.104-1
libdw-dev:arm64 0.183-1
libdw1:arm64 0.183-1
libeatmydata1:arm64 105-9
libedit2:arm64 3.1-20191231-2+b1
libegl-dev:arm64 1.3.2-1
libegl-mesa0:arm64 20.3.5-1
libegl1:arm64 1.3.2-1
libegl1-mesa-dev:arm64 20.3.5-1
libelf-dev:arm64 0.183-1
libelf1:arm64 0.183-1
libenchant-2-2:arm64 2.2.15-1
libenchant-2-dev:arm64 2.2.15-1
libepoxy-dev:arm64 1.5.5-1
libepoxy0:arm64 1.5.5-1
liberror-perl 0.17029-1
libevdev-dev:arm64 1.11.0+dfsg-1
libevdev2:arm64 1.11.0+dfsg-1
libexpat1:arm64 2.2.10-2+deb11u4
libexpat1-dev:arm64 2.2.10-2+deb11u4
libext2fs2:arm64 1.46.2-2
libfakeroot:arm64 1.25.3-1.1
libffi-dev:arm64 3.3-6
libffi7:arm64 3.3-6
libfido2-1:arm64 1.6.0-2
libfile-fcntllock-perl 0.22-3+b7
libfile-stripnondeterminism-perl 1.12.0-1
libfontconfig-dev:arm64 2.13.1-4.2
libfontconfig1:arm64 2.13.1-4.2
libfontconfig1-dev:arm64 2.13.1-4.2
libfontenc1:arm64 1:1.1.4-1
libfreetype-dev:arm64 2.10.4+dfsg-1+deb11u1
libfreetype6:arm64 2.10.4+dfsg-1+deb11u1
libfreetype6-dev:arm64 2.10.4+dfsg-1+deb11u1
libfribidi-dev:arm64 1.0.8-2+deb11u1
libfribidi0:arm64 1.0.8-2+deb11u1
libgbm1:arm64 20.3.5-1
libgc1:arm64 1:8.0.4-3
libgcc-10-dev:arm64 10.2.1-6
libgcc-s1:arm64 10.2.1-6
libgcrypt20:arm64 1.8.7-6
libgcrypt20-dev 1.8.7-6
libgd3:arm64 2.3.0-2
libgdbm-compat4:arm64 1.19-2
libgdbm6:arm64 1.19-2
libgdk-pixbuf-2.0-0:arm64 2.42.2+dfsg-1+deb11u1
libgdk-pixbuf-2.0-dev:arm64 2.42.2+dfsg-1+deb11u1
libgdk-pixbuf-xlib-2.0-0:arm64 2.40.2-2
libgdk-pixbuf2.0-0:arm64 2.40.2-2
libgdk-pixbuf2.0-bin 2.42.2+dfsg-1+deb11u1
libgdk-pixbuf2.0-common 2.42.2+dfsg-1+deb11u1
libgirepository-1.0-1:arm64 1.66.1-1+b1
libgirepository1.0-dev:arm64 1.66.1-1+b1
libgl-dev:arm64 1.3.2-1
libgl1:arm64 1.3.2-1
libgl1-mesa-dri:arm64 20.3.5-1
libglapi-mesa:arm64 20.3.5-1
libgles-dev:arm64 1.3.2-1
libgles1:arm64 1.3.2-1
libgles2:arm64 1.3.2-1
libglib2.0-0:arm64 2.66.8-1
libglib2.0-bin 2.66.8-1
libglib2.0-data 2.66.8-1
libglib2.0-dev:arm64 2.66.8-1
libglib2.0-dev-bin 2.66.8-1
libglib2.0-doc 2.66.8-1
libglvnd-dev:arm64 1.3.2-1
libglvnd0:arm64 1.3.2-1
libglx-dev:arm64 1.3.2-1
libglx-mesa0:arm64 20.3.5-1
libglx0:arm64 1.3.2-1
libgmp10:arm64 2:6.2.1+dfsg-1+deb11u1
libgnutls30:arm64 3.7.1-5+deb11u2
libgomp1:arm64 10.2.1-6
libgpg-error-dev 1.38-2
libgpg-error0:arm64 1.38-2
libgpm2:arm64 1.20.7-8
libgraphite2-3:arm64 1.3.14-1
libgraphite2-dev:arm64 1.3.14-1
libgssapi-krb5-2:arm64 1.18.3-6+deb11u2
libgstreamer-gl1.0-0:arm64 1.18.4-2
libgstreamer-plugins-base1.0-0:arm64 1.18.4-2
libgstreamer-plugins-base1.0-dev:arm64 1.18.4-2
libgstreamer1.0-0:arm64 1.18.4-2.1
libgstreamer1.0-dev:arm64 1.18.4-2.1
libgtk-3-0:arm64 3.24.24-4+deb11u2
libgtk-3-common 3.24.24-4+deb11u2
libgtk-3-dev:arm64 3.24.24-4+deb11u2
libgtk-3-doc 3.24.24-4+deb11u2
libgudev-1.0-0:arm64 234-1
libgudev-1.0-dev:arm64 234-1
libharfbuzz-dev:arm64 2.7.4-1
libharfbuzz-gobject0:arm64 2.7.4-1
libharfbuzz-icu0:arm64 2.7.4-1
libharfbuzz0b:arm64 2.7.4-1
libhogweed6:arm64 3.7.3-1
libhunspell-1.7-0:arm64 1.7.0-3
libhyphen-dev:arm64 2.8.8-7
libhyphen0:arm64 2.8.8-7
libice-dev:arm64 2:1.0.10-1
libice6:arm64 2:1.0.10-1
libicu-dev:arm64 67.1-7
libicu67:arm64 67.1-7
libidn2-0:arm64 2.3.0-5
libip4tc2:arm64 1.8.7-1
libisl23:arm64 0.23-1
libitm1:arm64 10.2.1-6
libjbig-dev:arm64 2.1-3.1+b2
libjbig0:arm64 2.1-3.1+b2
libjpeg-dev:arm64 1:2.0.6-4
libjpeg62-turbo:arm64 1:2.0.6-4
libjpeg62-turbo-dev:arm64 1:2.0.6-4
libjson-c5:arm64 0.15-2
libjson-glib-1.0-0:arm64 1.6.2-1
libjson-glib-1.0-common 1.6.2-1
libjson-glib-dev:arm64 1.6.2-1
libjsoncpp24:arm64 1.9.4-4
libk5crypto3:arm64 1.18.3-6+deb11u2
libkeyutils1:arm64 1.6.1-2
libkmod2:arm64 28-1
libkrb5-3:arm64 1.18.3-6+deb11u2
libkrb5support0:arm64 1.18.3-6+deb11u2
libksba8:arm64 1.5.0-3
liblcms2-2:arm64 2.12~rc1-2
liblcms2-dev:arm64 2.12~rc1-2
libldap-2.4-2:arm64 2.4.57+dfsg-3+deb11u1
libldap-common 2.4.57+dfsg-3+deb11u1
libllvm11:arm64 1:11.0.1-2
liblocale-gettext-perl 1.07-4+b1
liblsan0:arm64 10.2.1-6
liblua5.2-0:arm64 5.2.4-1.1+b3
liblz4-1:arm64 1.9.3-2
liblzma-dev:arm64 5.2.5-2.1~deb11u1
liblzma5:arm64 5.2.5-2.1~deb11u1
liblzo2-2:arm64 2.10-2
libmagic-mgc 1:5.39-3
libmagic1:arm64 1:5.39-3
libmanette-0.2-0:arm64 0.2.5-1
libmanette-0.2-dev:arm64 0.2.5-1
libmd0:arm64 1.0.3-3
libmount-dev:arm64 2.36.1-8+deb11u1
libmount1:arm64 2.36.1-8+deb11u1
libmpc3:arm64 1.2.0-1
libmpdec3:arm64 2.5.1-1
libmpfr6:arm64 4.1.0-3
libncurses6:arm64 6.2+20201114-2
libncursesw6:arm64 6.2+20201114-2
libnettle8:arm64 3.7.3-1
libnghttp2-14:arm64 1.43.0-1
libnl-3-200:arm64 3.4.0-1+b1
libnl-genl-3-200:arm64 3.4.0-1+b1
libnotify-dev:arm64 0.7.9-3
libnotify4:arm64 0.7.9-3
libnpth0:arm64 1.6-3
libnsl-dev:arm64 1.3.0-2
libnsl2:arm64 1.3.0-2
libobjc-10-dev:arm64 10.2.1-6
libobjc4:arm64 10.2.1-6
libopengl-dev:arm64 1.3.2-1
libopengl0:arm64 1.3.2-1
libopenjp2-7:arm64 2.4.0-3
libopenjp2-7-dev 2.4.0-3
liborc-0.4-0:arm64 1:0.4.32-1
liborc-0.4-dev:arm64 1:0.4.32-1
liborc-0.4-dev-bin 1:0.4.32-1
libosp5 1.5.2-13+b2
libp11-kit0:arm64 0.23.22-1
libpam-modules:arm64 1.4.0-9+deb11u1
libpam-modules-bin 1.4.0-9+deb11u1
libpam-runtime 1.4.0-9+deb11u1
libpam-systemd:arm64 247.3-7+deb11u1
libpam0g:arm64 1.4.0-9+deb11u1
libpango-1.0-0:arm64 1.46.2-3
libpango1.0-dev:arm64 1.46.2-3
libpango1.0-doc 1.46.2-3
libpangocairo-1.0-0:arm64 1.46.2-3
libpangoft2-1.0-0:arm64 1.46.2-3
libpangoxft-1.0-0:arm64 1.46.2-3
libpcre16-3:arm64 2:8.39-13
libpcre2-16-0:arm64 10.36-2+deb11u1
libpcre2-32-0:arm64 10.36-2+deb11u1
libpcre2-8-0:arm64 10.36-2+deb11u1
libpcre2-dev:arm64 10.36-2+deb11u1
libpcre2-posix2:arm64 10.36-2+deb11u1
libpcre3:arm64 2:8.39-13
libpcre3-dev:arm64 2:8.39-13
libpcre32-3:arm64 2:8.39-13
libpcrecpp0v5:arm64 2:8.39-13
libperl5.32:arm64 5.32.1-4+deb11u2
libpipeline1:arm64 1.5.3-1
libpixman-1-0:arm64 0.40.0-1
libpixman-1-dev:arm64 0.40.0-1
libpng-dev:arm64 1.6.37-3
libpng16-16:arm64 1.6.37-3
libpolkit-agent-1-0:arm64 0.105-31+deb11u1
libpolkit-gobject-1-0:arm64 0.105-31+deb11u1
libprocps8:arm64 2:3.3.17-5
libproxy1v5:arm64 0.4.17-1
libpsl-dev:arm64 0.21.0-1.2
libpsl5:arm64 0.21.0-1.2
libpthread-stubs0-dev:arm64 0.4-1
libpython3-stdlib:arm64 3.9.2-3
libpython3.9:arm64 3.9.2-1
libpython3.9-minimal:arm64 3.9.2-1
libpython3.9-stdlib:arm64 3.9.2-1
libreadline8:arm64 8.1-1
librest-0.7-0:arm64 0.8.1-1.1
librest-dev 0.8.1-1.1
librhash0:arm64 1.4.1-2
librsvg2-2:arm64 2.50.3+dfsg-1
librsvg2-common:arm64 2.50.3+dfsg-1
librsvg2-dev:arm64 2.50.3+dfsg-1
librtmp1:arm64 2.4+20151223.gitfa8646d.1-2+b2
libruby2.7:arm64 2.7.4-1+deb11u1
libsasl2-2:arm64 2.1.27+dfsg-2.1+deb11u1
libsasl2-modules:arm64 2.1.27+dfsg-2.1+deb11u1
libsasl2-modules-db:arm64 2.1.27+dfsg-2.1+deb11u1
libsass1:arm64 3.6.4+20201122-1
libseccomp-dev:arm64 2.5.1-1+deb11u1
libseccomp2:arm64 2.5.1-1+deb11u1
libsecret-1-0:arm64 0.20.4-2
libsecret-1-dev:arm64 0.20.4-2
libsecret-common 0.20.4-2
libselinux1:arm64 3.1-3
libselinux1-dev:arm64 3.1-3
libsemanage-common 3.1-1
libsemanage1:arm64 3.1-1+b2
libsensors-config 1:3.6.0-7
libsensors5:arm64 1:3.6.0-7
libsepol1:arm64 3.1-1
libsepol1-dev:arm64 3.1-1
libsigsegv2:arm64 2.13-1
libsm-dev:arm64 2:1.2.3-1
libsm6:arm64 2:1.2.3-1
libsmartcols1:arm64 2.36.1-8+deb11u1
libsoup-gnome2.4-1:arm64 2.72.0-2
libsoup2.4-1:arm64 2.72.0-2
libsoup2.4-dev:arm64 2.72.0-2
libsoup2.4-doc 2.72.0-2
libsource-highlight-common 3.1.9-3
libsource-highlight4v5 3.1.9-3+b1
libsqlite3-0:arm64 3.34.1-3
libsqlite3-dev:arm64 3.34.1-3
libss2:arm64 1.46.2-2
libssh2-1:arm64 1.9.0-2
libssl1.1:arm64 1.1.1n-0+deb11u3
libstdc++-10-dev:arm64 10.2.1-6
libstdc++6:arm64 10.2.1-6
libsub-override-perl 0.09-2
libsystemd-dev:arm64 247.3-7+deb11u1
libsystemd0:arm64 247.3-7+deb11u1
libtasn1-6:arm64 4.16.0-2
libtasn1-6-dev:arm64 4.16.0-2
libtcl8.6:arm64 8.6.11+dfsg-1
libtext-iconv-perl 1.7-7+b1
libthai-data 0.1.28-3
libthai-dev:arm64 0.1.28-3
libthai0:arm64 0.1.28-3
libtiff-dev:arm64 4.2.0-1+deb11u1
libtiff5:arm64 4.2.0-1+deb11u1
libtiff5-dev 4.2.0-1+deb11u1
libtiffxx5:arm64 4.2.0-1+deb11u1
libtinfo6:arm64 6.2+20201114-2
libtirpc-common 1.3.1-1+deb11u1
libtirpc-dev:arm64 1.3.1-1+deb11u1
libtirpc3:arm64 1.3.1-1+deb11u1
libtool 2.4.6-15
libtsan0:arm64 10.2.1-6
libubsan1:arm64 10.2.1-6
libuchardet0:arm64 0.0.7-1
libudev1:arm64 247.3-7+deb11u1
libunistring2:arm64 0.9.10-4
libunwind-dev:arm64 1.3.2-2
libunwind8:arm64 1.3.2-2
libuuid1:arm64 2.36.1-8+deb11u1
libuv1:arm64 1.40.0-2
libvulkan1:arm64 1.2.162.0-1
libwayland-bin 1.18.0-2~exp1.1
libwayland-client0:arm64 1.18.0-2~exp1.1
libwayland-cursor0:arm64 1.18.0-2~exp1.1
libwayland-dev:arm64 1.18.0-2~exp1.1
libwayland-egl1:arm64 1.18.0-2~exp1.1
libwayland-server0:arm64 1.18.0-2~exp1.1
libwebp-dev:arm64 0.6.1-2.1
libwebp6:arm64 0.6.1-2.1
libwebpdemux2:arm64 0.6.1-2.1
libwebpmux3:arm64 0.6.1-2.1
libwoff-dev:arm64 1.0.2-1+b1
libwoff1:arm64 1.0.2-1+b1
libwpe-1.0-1:arm64 1.10.0-2
libwpe-1.0-dev:arm64 1.10.0-2
libwpebackend-fdo-1.0-1:arm64 1.8.0-1
libwpebackend-fdo-1.0-dev:arm64 1.8.0-1
libx11-6:arm64 2:1.7.2-1
libx11-data 2:1.7.2-1
libx11-dev:arm64 2:1.7.2-1
libx11-xcb-dev:arm64 2:1.7.2-1
libx11-xcb1:arm64 2:1.7.2-1
libxau-dev:arm64 1:1.0.9-1
libxau6:arm64 1:1.0.9-1
libxaw7:arm64 2:1.0.13-1.1
libxcb-dri2-0:arm64 1.14-3
libxcb-dri3-0:arm64 1.14-3
libxcb-glx0:arm64 1.14-3
libxcb-present0:arm64 1.14-3
libxcb-render0:arm64 1.14-3
libxcb-render0-dev:arm64 1.14-3
libxcb-shm0:arm64 1.14-3
libxcb-shm0-dev:arm64 1.14-3
libxcb-sync1:arm64 1.14-3
libxcb-xfixes0:arm64 1.14-3
libxcb1:arm64 1.14-3
libxcb1-dev:arm64 1.14-3
libxcomposite-dev:arm64 1:0.4.5-1
libxcomposite1:arm64 1:0.4.5-1
libxcursor-dev:arm64 1:1.2.0-2
libxcursor1:arm64 1:1.2.0-2
libxdamage-dev:arm64 1:1.1.5-2
libxdamage1:arm64 1:1.1.5-2
libxdmcp-dev:arm64 1:1.1.2-3
libxdmcp6:arm64 1:1.1.2-3
libxext-dev:arm64 2:1.3.3-1.1
libxext6:arm64 2:1.3.3-1.1
libxfixes-dev:arm64 1:5.0.3-2
libxfixes3:arm64 1:5.0.3-2
libxfont2:arm64 1:2.0.4-1
libxft-dev:arm64 2.3.2-2
libxft2:arm64 2.3.2-2
libxi-dev:arm64 2:1.7.10-1
libxi6:arm64 2:1.7.10-1
libxinerama-dev:arm64 2:1.1.4-2
libxinerama1:arm64 2:1.1.4-2
libxkbcommon-dev:arm64 1.0.3-2
libxkbcommon0:arm64 1.0.3-2
libxkbfile-dev:arm64 1:1.1.0-1
libxkbfile1:arm64 1:1.1.0-1
libxml2:arm64 2.9.10+dfsg-6.7+deb11u2
libxml2-dev:arm64 2.9.10+dfsg-6.7+deb11u2
libxml2-utils 2.9.10+dfsg-6.7+deb11u2
libxmu6:arm64 2:1.1.2-2+b3
libxmuu1:arm64 2:1.1.2-2+b3
libxpm4:arm64 1:3.5.12-1
libxrandr-dev:arm64 2:1.5.1-1
libxrandr2:arm64 2:1.5.1-1
libxrender-dev:arm64 1:0.9.10-1
libxrender1:arm64 1:0.9.10-1
libxshmfence1:arm64 1.3-1
libxslt1-dev:arm64 1.1.34-4+deb11u1
libxslt1.1:arm64 1.1.34-4+deb11u1
libxt-dev:arm64 1:1.2.0-1
libxt6:arm64 1:1.2.0-1
libxtst-dev:arm64 2:1.2.3-1
libxtst6:arm64 2:1.2.3-1
libxxf86vm1:arm64 1:1.1.4-1+b2
libxxhash0:arm64 0.8.0-2
libyaml-0-2:arm64 0.2.2-1
libz3-4:arm64 4.8.10-1
libzstd1:arm64 1.4.8+dfsg-2.1
linux-libc-dev:arm64 5.10.140-1
login 1:4.8.1-1
logsave 1.46.2-2
lsb-base 11.1.0
m4 1.4.18-5
make 4.3-4.1
man-db 2.9.4-2
manpages 5.10-1
manpages-dev 5.10-1
mawk 1.3.4.20200120-2
media-types 4.0.0
meson 0.56.2-1
mount 2.36.1-8+deb11u1
ncurses-base 6.2+20201114-2
ncurses-bin 6.2+20201114-2
netbase 6.3
ninja-build 1.10.1-1
opensp 1.5.2-13+b2
openssh-client 1:8.4p1-5+deb11u1
openssl 1.1.1n-0+deb11u3
pango1.0-tools 1.46.2-3
passwd 1:4.8.1-1
patch 2.7.6-7
pax-utils 1.2.9-1
perl 5.32.1-4+deb11u2
perl-base 5.32.1-4+deb11u2
perl-modules-5.32 5.32.1-4+deb11u2
pinentry-curses 1.1.0-4
pkg-config 0.29.2-1
po-debconf 1.0.21+nmu1
procps 2:3.3.17-5
python3 3.9.2-3
python3-distutils 3.9.2-1
python3-gi 3.38.0-2
python3-lib2to3 3.9.2-1
python3-lxml:arm64 4.6.3+dfsg-0.1+deb11u1
python3-mako 1.1.3+ds1-2
python3-markdown 3.3.4-1
python3-markupsafe 1.1.1-1+b3
python3-minimal 3.9.2-3
python3-pkg-resources 52.0.0-4
python3-pygments 2.7.1+dfsg-2.1
python3.9 3.9.2-1
python3.9-minimal 3.9.2-1
rake 13.0.3-1
readline-common 8.1-1
ruby 1:2.7+2
ruby-minitest 5.13.0-1
ruby-net-telnet 0.1.1-2
ruby-power-assert 1.1.7-2
ruby-rubygems 3.2.5-2
ruby-test-unit 3.3.9-1
ruby-xmlrpc 0.3.0-2
ruby2.7 2.7.4-1+deb11u1
rubygems-integration 1.18
sassc 3.6.1+20201027-1
sed 4.7-1
sensible-utils 0.0.14
sgml-base 1.30
sgml-data 2.0.11+nmu1
shared-mime-info 2.0-1
sysprof 3.38.1-1
systemd 247.3-7+deb11u1
systemd-sysv 247.3-7+deb11u1
sysvinit-utils 2.96-7+deb11u1
tar 1.34+dfsg-1
tzdata 2021a-1+deb11u5
ucf 3.0043
util-linux 2.36.1-8+deb11u1
uuid-dev:arm64 2.36.1-8+deb11u1
vim-common 2:8.2.2434-3+deb11u1
vim-nox 2:8.2.2434-3+deb11u1
vim-runtime 2:8.2.2434-3+deb11u1
wayland-protocols 1.20-1
x11-common 1:7.7+22
x11-xkb-utils 7.7+5
x11proto-dev 2020.1-1
x11proto-input-dev 2020.1-1
x11proto-randr-dev 2020.1-1
x11proto-record-dev 2020.1-1
x11proto-xext-dev 2020.1-1
x11proto-xinerama-dev 2020.1-1
xauth 1:1.1-1
xdg-dbus-proxy 0.1.2-2
xkb-data 2.29-2
xml-core 0.18+nmu1
xorg-sgml-doctools 1:1.11-1.1
xserver-common 2:1.20.11-1+deb11u2
xsltproc 1.1.34-4+deb11u1
xtrans-dev 1.4.0-1
xvfb 2:1.20.11-1+deb11u2
xxd 2:8.2.2434-3+deb11u1
xz-utils 5.2.5-2.1~deb11u1
zlib1g:arm64 1:1.2.11.dfsg-2+deb11u2
zlib1g-dev:arm64 1:1.2.11.dfsg-2+deb11u2


Thanks,
--
Dominique

[Dominique MARTINET]

Dominique MARTINET wrote on Fri, Sep 30, 2022 at 01:26:03PM +0900:
> Alberto Garcia wrote on Thu, Sep 29, 2022 at 11:19:04AM +0000:
> > Hmmm... ok, this may be a bug in the compiler. The official Debian
> > builds are made using clang (because of bug #1008098), did you use
> > gcc?
>
> Hm, that is not specified in the control or rule file?
>
> I've triggered a fresh build to check and can confirm it's using gcc...
> Interesting that my build didn't fail on this (well, I guess ldd
> segfault was just as bad :-D)
>
> I've had a quick look but wasn't able to figure how to build with clang
> to check -- would building the package with clang in CC/CXX suffice,
> e.g. `CC=clang CXX=clang++ dpkg-buildpackage` ?
> (that appears to do the job, I'll send a new mail on Monday if build
> didn't finish in time to test today)

build finished, tested -- I can reproduce ! :)

The build has been quite stable with gcc for me, so while I trust
Sebastian the patch he pointed at fixes the immediate issue I'll try to
rebuild with it as well and start looking at the next crashes.

(It's a bit annoying to take 3+ hours to build though so I might have a
stab at running from build dir again first though...)

Cheers,
--
Dominique

[Alberto Garcia]

On Fri, Sep 30, 2022 at 04:57:33PM +0900, Dominique MARTINET wrote:
> The build has been quite stable with gcc for me, so while I trust
> Sebastian the patch he pointed at fixes the immediate issue I'll try
> to rebuild with it as well and start looking at the next crashes.

So let me get this straight... if the packages are built with clang
(that's how bullseye backports are currently being built) then it
crashes, but with gcc it is stable, right?

I made a new build using all the standard Debian tools but with gcc
instead of clang, and this is the result:

https://people.debian.org/~berto/files/webkitgtk-2.38.0-arm64/unpatched-gcc/

Can you check if it works for you? If it does then I'll switch back to
gcc for the next upload.

Berto

[Sebastian Krzyszkowiak]

On Fri, Sep 30, 2022 at 11:05 AM Alberto Garcia <be...@igalia.com> wrote:
> Can you check if it works for you? If it does then I'll switch back to
> gcc for the next upload.

I have tested your unpatched-gcc build on my machines and it appears
perfectly usable - haven't managed to make it crash so far.

I can still see some backtraces showing up in logs due to preconnect
attempts, so applying PR 4790 will likely still be a good idea, but it
doesn't bring down the whole process anymore.

Cheers,
Sebastian

[Alberto Garcia]

Control: tags -1 pending

On Fri, Sep 30, 2022 at 01:01:55PM +0200, Sebastian Krzyszkowiak wrote:

> > Can you check if it works for you? If it does then I'll switch
> > back to gcc for the next upload.
>
> I have tested your unpatched-gcc build on my machines and it appears
> perfectly usable - haven't managed to make it crash so far.

Ok, then the next backport will use gcc instead of clang for arm64
(unfortunately gcc still crashes on amd64).

Thanks!

Berto

[Dominique MARTINET]

Hi Alberto, Sebastian,

Sebastian Krzyszkowiak wrote on Thu, Sep 29, 2022 at 06:15:56PM +0200:
> the patch that sets setAllowsServerPreconnect does not fix the
> original issue, however, this one does:
> https://github.com/WebKit/WebKit/pull/4790
>
> With this applied on top of your recent backport of 2.38.0, WebKit
> becomes somewhat usable on arm64 in bullseye. It still crashes on some
> pages (such as Twitter), but in a different way (somewhere in JSC)
> which probably deserves a separate issue.

Thank you for finding this other PR -- I can confirm that fixes the
original issue which crashed almost immediately on any page, and also
confirm there are leftover crashes with a clang build on this patch.

Here's the backtrace I get:
(gdb) bt
#0 codeBlock () at ../Source/JavaScriptCore/interpreter/RegisterInlines.h:41
#1 codeBlock () at ../Source/JavaScriptCore/interpreter/CallFrameInlines.h:62
#2 operationArithNegateProfiledOptimize () at ../Source/JavaScriptCore/jit/JITOperations.cpp:3361
#3 0x0000ffff74148358 in ?? ()
#4 0x0000ffff2ed44080 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

unfortunately it looks like gdb doesn't manage to unwind it properly (or
indeed corrupted), so that might be difficult to debug.

Since gcc build appears to fix the issue I think that's the way to go
for bullseye, at least until we understand where that makes a
difference...

Sebastian Krzyszkowiak wrote on Fri, Sep 30, 2022 at 01:01:55PM +0200:
> I can still see some backtraces showing up in logs due to preconnect
> attempts, so applying PR 4790 will likely still be a good idea, but it
> doesn't bring down the whole process anymore.

I agree on principle; out of curiousity where did you see these logs?
I do not see anything on stdout/stderr with the gcc build, but I would
assume this to be logged elsewhere or perhaps only if some magic env var
is set?

Thanks,
--
Dominique

[Alberto Garcia]

On Mon, Oct 03, 2022 at 01:07:09PM +0900, Dominique MARTINET wrote:
> Since gcc build appears to fix the issue I think that's the way to
> go for bullseye, at least until we understand where that makes a
> difference...

gcc is the default compiler that we use for WebKit in Debian, the only
reason why it wasn't being used in bullseye in this case it's because
of this bug:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008098

I narrowed this one down as much as possible, we should probably
report it upstream but I didn't have the chance to do it yet.

Berto

[Loïc Yhuel]

The crash in operationArithNegateProfiledOptimize is due to a bug in
clang 10/11.
On AArch64, the frame pointer might be set incorrectly depending on
which registers are saved on the stack.
This is fixed in
https://github.com/llvm/llvm-project/commit/e9d9a61208, which applies
easily on clang 11.

WebKit uses __builtin_frame_address(1) in DECLARE_CALL_FRAME macro,
which due to the bug reads a saved floating point register, instead of
the frame pointer of the caller.

[Alberto Garcia]

On Fri, Oct 28, 2022 at 08:02:19PM +0200, Loïc Yhuel wrote:
> The crash in operationArithNegateProfiledOptimize is due to a bug in
> clang 10/11.
> On AArch64, the frame pointer might be set incorrectly depending on
> which registers are saved on the stack.
> This is fixed in
> https://github.com/llvm/llvm-project/commit/e9d9a61208, which applies
> easily on clang 11.

Backports are going to start using gcc again (except for amd64, where
gcc is broken) so hopefully this helps.

Berto