Important packages that are broken ia64

[John Paul Adrian Glaubitz]

Hello!

The following important packages are broken on ia64:

- grub (git master) does not boot on ia64, crashes when loading stage2 (https://lists.gnu.org/archive/html/grub-devel/2023-07/msg00106.html)
- kernel FTBFS with gcc-13 (https://buildd.debian.org/status/fetch.php?pkg=linux&arch=ia64&ver=6.4.4-2&stamp=1690708282&raw=0)
- glibc testsuite failures, in particular math functions (https://buildd.debian.org/status/fetch.php?pkg=glibc&arch=ia64&ver=2.37-7&stamp=1691155973&raw=0)
- webkit2gtk (https://gitlab.gnome.org/GNOME/gobject-introspection/-/issues/471)

At least the grub problem could certainly be easily bisected, but alas I don't really
have the time and motivation do that. So it would be nice if someone else could at
least bisect grub to find the issue and report it on the grub-devel mailing list.

Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer
`. `' Physicist
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

[John Paul Adrian Glaubitz]

Hi!

On Sun, 2023-08-06 at 15:08 +0000, Pedro Miguel Justo wrote:
> ow much automation can be used for bisecting a problem on grub?
>
> My rx2660 is considerably up to date and still booting.
>
> What do you recommend for best bisecting the offending step?

You cannot automate this as the machine needs to be rebooted all the time.

Basically, you need to do the following:

# apt build-dep grub2
# git clone git://git.savannah.gnu.org/grub.git
# cd grub
# git bisect start
# git bisect good grub-2.06
# git bisect bad HEAD

Then build grub, install and testboot:

# ./bootstrap && ./configure && make && make install
# /usr/local/sbin/grub-install /dev/sda (when sda is your primary disk)

Reboot the machine and chose the "grub" boot entry, "debian" will be your
fallback entry in case you have run into the bug.

If the machine boot successfully using the "grub" boot entry in EFI, go back
into the "grub" source directory and run "git bisect good". Run "make clean"
and repeat the build and install step above.

If a boot fails, reset the machine, boot the "debian" entry and then go back
into the "grub" source directory and run "git bisect bad". Run "make clean"
and repeat the build and install step above.

Repeat this procedure until git shows you the offending commit.

[Jessica Clarke]

On 7 Aug 2023, at 00:00, Pedro Miguel Justo <pm...@texair.net> wrote:

>
>
>
>> On Aug 6, 2023, at 10:13, John Paul Adrian Glaubitz <glau...@physik.fu-berlin.de> wrote:
>>
>> Hi!
>>
>> On Sun, 2023-08-06 at 15:08 +0000, Pedro Miguel Justo wrote:
>>> ow much automation can be used for bisecting a problem on grub?
>>>
>>> My rx2660 is considerably up to date and still booting.
>>>
>>> What do you recommend for best bisecting the offending step?
>>
>> You cannot automate this as the machine needs to be rebooted all the time.
>

> That is what I was expecting.

>
>>
>> Basically, you need to do the following:
>>
>> # apt build-dep grub2
>

> Sourt of embarrassing question but… what is the deb-src right entry for debian ports?

Same as non-ports; there’s no ports copy of the unstable sources.

Jess

[John Paul Adrian Glaubitz]

On Sun, 2023-08-06 at 23:34 +0000, Pedro Miguel Justo wrote:
> > > > # git clone git://git.savannah.gnu.org/grub.git
>

> This leads me to another question: Has anyone been able to install git these days?

# wget http://snapshot.debian.org/archive/debian/20220510T034305Z/pool/main/g/git/git-man_2.36.1-1_all.deb
# dpkg -i git-man_2.36.1-1_all.deb

Then install git normally with apt-get.

[John Paul Adrian Glaubitz]

On Tue, 2023-08-08 at 23:27 +0000, Pedro Miguel Justo wrote:

>
> > On Aug 7, 2023, at 01:03, John Paul Adrian Glaubitz <glau...@physik.fu-berlin.de> wrote:
> >
> > # wget http://snapshot.debian.org/archive/debian/20220510T034305Z/pool/main/g/git/git-man_2.36.1-1_all.deb
> > # dpkg -i git-man_2.36.1-1_all.deb
> >
> > Then install git normally with apt-get.
> >
>

> Thanks Adrian.
>
> From there, all good till the grub-install step, which failed:
>
> pmsjt@itanium:~/grub$ sudo /usr/local/sbin/grub-install /dev/sda
> Installing for ia64-efi platform.
> /usr/local/sbin/grub-install: warning: cannot open directory `/usr/local/share/locale': No such file or directory.

Unless you care about localization (which you hopefully don't during debugging),
you can safely ignore this warning.

> /usr/local/sbin/grub-install: error: efibootmgr: not found.

Well, you need to install efibootmgr. Not sure why it isn't installed on your
machine in the first place since it's needed for grub-install to update the
boot manager settings in firmware and create the "grub" entry.

Just install it with:

# apt install efibootmgr

[Jason Duerstock]

Hi Pedro,

Did you finish the git bisect? What was the commit that broke it?

Thanks,

Jason

On Wed, Aug 9, 2023 at 9:41 PM Pedro Miguel Justo <pm...@texair.net> wrote:
>
> >
> > Well, you need to install efibootmgr. Not sure why it isn't installed on your
> > machine in the first place since it's needed for grub-install to update the
> > boot manager settings in firmware and create the "grub" entry.
> >
> > Just install it with:
> >
> > # apt install efibootmgr
> >
> > Adrian
> >
>

> Thanks Adrian. It is a mystery why it was previously installed. After that I was able to go through 3 or 4 bisections with successful boots. Then, finally, I hit the problem. The trouble is that it seems to have also affected my “debian” entry, so no safe boot:
>
> Loading.: debian
> Starting: debian
> Welcome to GRUB!
>
> 7 0 0x00006B 0x000000000000001E unexpected trap
> 7 0 0x000066 0x000000000000001E trap taken, number in ext PE
> 7 0 0x00003C 0x0000000000005A00 trap taken, offset in ext PE

[John Paul Adrian Glaubitz]

On Thu, 2023-08-10 at 00:43 +0000, Pedro Miguel Justo wrote:
> Well, gladly I had left the latest installation CD you asked us to test
> in the drive. I was able to boot from it and repair GRUB. I am back in
> business.

Great. Hopefully, you will be able to find the problematic commit.

It should be related to the EFI changes by Ard Biesheuvel.

[John Paul Adrian Glaubitz]

Hi Pedro!

On Thu, 2023-08-10 at 00:08 +0000, Pedro Miguel Justo wrote:
>
> Thanks Adrian. It is a mystery why it was previously installed. After that
> I was able to go through 3 or 4 bisections with successful boots. Then,
> finally, I hit the problem. The trouble is that it seems to have also affected
> my “debian” entry, so no safe boot:

Are you sure you didn't overwrite the "debian" entry?

Either way, you should be able to restore the bootloader by booting into rescue
mode using the installation media.

[John Paul Adrian Glaubitz]

Hi Pedro!

On Thu, 2023-08-10 at 12:04 +0000, Pedro Miguel Justo wrote:
> Here it is:
>
> 06edd40db76bb78457ac26156ed5f7b62381bbe8 is the first bad commit
> commit 06edd40db76bb78457ac26156ed5f7b62381bbe8
> Author: Oliver Steffen <oste...@redhat.com>
> Date: Fri May 26 13:35:43 2023 +0200
>
> guid: Unify GUID types
>
> There are 3 implementations of a GUID in GRUB. Replace them with
> a common one, placed in types.h.
>
> It uses the "packed" flavor of the GUID structs, the alignment attribute
> is dropped, since it is not required.
>
> Signed-off-by: Oliver Steffen <oste...@redhat.com>
> Reviewed-by: Daniel Kiper <daniel...@oracle.com>

Did you verify that reverting this commit on git master makes GRUB work again?

If yes, please report this finding to the grub-devel mailing list, CC-ing Oliver Steffen.

[Frank Scheiner]

Hi Pedro, all,

On 10.08.23 14:04, Pedro Miguel Justo wrote:
> Here it is:
>
> 06edd40db76bb78457ac26156ed5f7b62381bbe8 is the first bad commit
> commit 06edd40db76bb78457ac26156ed5f7b62381bbe8
> Author: Oliver Steffen <oste...@redhat.com>
> Date: Fri May 26 13:35:43 2023 +0200
>
> guid: Unify GUID types
>
> There are 3 implementations of a GUID in GRUB. Replace them with
> a common one, placed in types.h.
>
> It uses the "packed" flavor of the GUID structs, the alignment attribute
> is dropped, since it is not required.
>
> Signed-off-by: Oliver Steffen <oste...@redhat.com>
> Reviewed-by: Daniel Kiper <daniel...@oracle.com>
>

I had a quick look into this and the ia64 "related" part ([1]) switches
from an `__attribute__ ((aligned(8)))` struct "grub_efi_guid_t" to an
`__attribute__ ((packed))` struct "grub_guid_t" (see [2], [3] and
below), which might not work as expected on ia64.

`include/grub/types.h` defines:

```
#define GRUB_PACKED __attribute__ ((packed))
```

[1]:
https://git.savannah.gnu.org/cgit/grub.git/diff/grub-core/loader/ia64/efi/linux.c?id=06edd40db76bb78457ac26156ed5f7b62381bbe8

[2]:
https://git.savannah.gnu.org/cgit/grub.git/diff/include/grub/efi/api.h?id=06edd40db76bb78457ac26156ed5f7b62381bbe8

[3]:
https://git.savannah.gnu.org/cgit/grub.git/diff/include/grub/types.h?id=06edd40db76bb78457ac26156ed5f7b62381bbe8

Not sure if it could help to just replace `GRUB_PACKED` with
`__attribute__ ((aligned(8)))` for "grub_guid_t" in [3] or if this
breaks the cases that originally used "grub_efi_packed_guid_t".

Cheers,
Frank

[Frank Scheiner]

Hi Pedro,

On 10.08.23 19:31, Frank Scheiner wrote:
> Hi Pedro, all,
>
> On 10.08.23 14:04, Pedro Miguel Justo wrote:
>> Here it is:
>>
>> 06edd40db76bb78457ac26156ed5f7b62381bbe8 is the first bad commit
>> commit 06edd40db76bb78457ac26156ed5f7b62381bbe8
>> Author: Oliver Steffen <oste...@redhat.com>
>> Date:   Fri May 26 13:35:43 2023 +0200
>>
>>      guid: Unify GUID types
>>
>>      There are 3 implementations of a GUID in GRUB. Replace them with
>>      a common one, placed in types.h.
>>
>>      It uses the "packed" flavor of the GUID structs, the alignment
>> attribute
>>      is dropped, since it is not required.
>>
>>      Signed-off-by: Oliver Steffen <oste...@redhat.com>
>>      Reviewed-by: Daniel Kiper <daniel...@oracle.com>
>>
>
> I had a quick look into this and the ia64 "related" part ([1]) switches
> from an `__attribute__ ((aligned(8)))` struct "grub_efi_guid_t" to an
> `__attribute__ ((packed))` struct "grub_guid_t" (see [2], [3] and
> below), which might not work as expected on ia64.

Damn it, I should have read the commit message instead of the code
changes. It tells the - maybe - important thing already.

Unfortunately there's no info in [4] (where this was introduced) about
why exactly the aligment was intended in the first place.

[4]:
https://git.savannah.gnu.org/cgit/grub.git/commit/?id=837091258d7e9f4af3cc333ec775271f1b767d11

Linux actually uses a minimum alignment of 32 bits, see for example [5].

[5]:
https://github.com/torvalds/linux/blob/master/include/linux/efi.h#L60-L78

@Pedro:
Does it work again with `__attribute__ ((aligned(8)))` instead of
`__attribute__ ((packed))` for "grub_guid"?

Cheers,
Frank

[Frank Scheiner]

Hi Adrian,

On 06.08.23 10:44, John Paul Adrian Glaubitz wrote:
> Hello!
>
> The following important packages are broken on ia64:
>
> - grub (git master) does not boot on ia64, crashes when loading stage2 (https://lists.gnu.org/archive/html/grub-devel/2023-07/msg00106.html)
> - kernel FTBFS with gcc-13 (https://buildd.debian.org/status/fetch.php?pkg=linux&arch=ia64&ver=6.4.4-2&stamp=1690708282&raw=0)

I created a bug report for this, see [1]. In short this happens (1) also
for cross-compilation on amd64 (most likely also for other
cross-compilation host arches), (2) for me always for the same file
(`net/ipv4/fib_semantics.c`) and function ("fib_create_info()"), but
there are others, too, see buildd link, (3) for different kernel
versions and (3) is gone when providing `-fsanitize=undefined` during
compilation.

Details in [1].

[1]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111425

I hope someone picks that up.

Cheers,
Frank

[Frank Scheiner]

Hi again,

Richard Biener suggested `-fno-var-tracking` or `-g0` as workaround and
both indeed workaround the problem (see [2]). I don't yet know how to
limit the addition of `-fno-var-tracking` to KBUILD_CFLAGS to
`net/ipv4/fib_semantics.c` but the following patch:

```
root@dl380-g7:/usr/src# cat workaround-for-gcc-13.patch
diff --git a/net/ipv4/Makefile b/net/ipv4/Makefile
index b18ba8ef93ad..38f8586ebcc6 100644
--- a/net/ipv4/Makefile
+++ b/net/ipv4/Makefile
@@ -3,6 +3,8 @@
# Makefile for the Linux TCP/IP (INET) layer.
#

+KBUILD_CFLAGS += -fno-var-tracking
+
obj-y := route.o inetpeer.o protocol.o \
ip_input.o ip_fragment.o ip_forward.o ip_options.o \
ip_output.o ip_sockglue.o inet_hashtables.o \

```

...makes compilation work for the used kernel configuration ([3]):

```
root@dl380-g7:/usr/src# time ./make-kernel.bash
rx2620-rx2660-rx2800-i2-combined-localmodconfig ia64
linux-on-ramdisk/torvalds-linux/ w-gcc13
[...]
Making kernel...
time make -j24
LOCALVERSION="-ce9ecca0238b140b88f43859b211c9fdfd8e5b70-ia64-w-gcc13"
ARCH=ia64 CROSS_COMPILE=ia64-linux- all
[...]
CC security/integrity/ima/ima_fs.o
CC net/ipv4/fib_semantics.o
CC security/integrity/ima/ima_queue.o
[...]
No errors detected in 26726 functions.

real 3m41.183s
user 69m23.751s
sys 6m57.374s

0
[...]
done
Build artifacts:
/boot/vmlinux-6.6.0-rc2-ce9ecca0238b140b88f43859b211c9fdfd8e5b70-ia64-w-gcc13
/boot/vmlinuz-6.6.0-rc2-ce9ecca0238b140b88f43859b211c9fdfd8e5b70-ia64-w-gcc13
/boot/config-6.6.0-rc2-ce9ecca0238b140b88f43859b211c9fdfd8e5b70-ia64-w-gcc13
/lib/modules/6.6.0-rc2-ce9ecca0238b140b88f43859b211c9fdfd8e5b70-ia64-w-gcc13.tar

real 3m57.203s
user 69m28.325s
sys 6m59.414s
```

Looking into the buildd link above this also needs to be applied for
`/drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c` if it is indeed the
same problem.

[2]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111425#c5

[3]: https://gcc.gnu.org/bugzilla/attachment.cgi?id=55904

The resulting kernel and modules aren't yet tested, though. I plan that
for today and tomorrow.

Cheers,
Frank

[Frank Scheiner]

Hi again,

On 18.09.23 11:43, Frank Scheiner wrote:
> The resulting kernel and modules aren't yet tested, though. I plan that
> for today and tomorrow.

v6.6-rc2 with the acpi build fix (I used the patch from [1]) and the
workaround for gcc-13 (see my prior email) and built with gcc 13.2.0
from [2] works fine (boot to login prompt) on:

* rx2620
* rx2800 i2

More test(ed) systems to follow tomorrow.

Cheers,
Frank

[1]:
https://lore.kernel.org/lkml/CAJZ5v0hnNK4O_HyinvTp01YxXR7V4vzpMhf85yW9M2=52-...@mail.gmail.com/

[2]: https://mirrors.edge.kernel.org/pub/tools/crosstool/

[Frank Scheiner]

On 18.09.23 11:43, Frank Scheiner wrote:

> Richard Biener suggested `-fno-var-tracking` or `-g0` as workaround and
> both indeed workaround the problem (see [2]). I don't yet know how to
> limit the addition of `-fno-var-tracking` to KBUILD_CFLAGS to

> `net/ipv4/fib_semantics.c` [...]

Ok, found what I need on [1]:

```
$(CFLAGS_$@) specifies per-file options for $(CC). The @ part has a
literal value which specifies the file that it is for.
[...]
Example:

# drivers/scsi/Makefile
CFLAGS_aha152x.o = -DAHA152X_STAT -DAUTOCONF
```

[1]:
https://github.com/torvalds/linux/blob/master/Documentation/kbuild/makefiles.rst

I'll give that a try and if it works create a MR on [2] so kernel builds
for ia64 can succeed with gcc-13 until that one is fixed.

[2]: https://salsa.debian.org/kernel-team/linux

Cheers,
Frank

[Frank Scheiner]

Worked for me, MR is here:

https://salsa.debian.org/kernel-team/linux/-/merge_requests/852

Looking forward to new ia64 kernels for Sid.

Cheers,
Frank

[Frank Scheiner]

On 18.09.23 22:34, John Paul Adrian Glaubitz wrote:
> Hello Frank!

>
> On Mon, 2023-09-18 at 22:14 +0200, Frank Scheiner wrote:
>> Worked for me, MR is here:
>>
>> https://salsa.debian.org/kernel-team/linux/-/merge_requests/852
>>
>> Looking forward to new ia64 kernels for Sid.
>

> I don't think this patch is acceptable in its current form as it
> modifies the Makefile globally so that the flag is passed on to
> the host compiler for all architectures.

Yes, thought about that, too. Know a better solution? Is there a way to
make this only affecting compilations for ia64?

Cheers,
Frank

[John Paul Adrian Glaubitz]

Hello Frank!

On Mon, 2023-09-18 at 22:14 +0200, Frank Scheiner wrote:

> Worked for me, MR is here:
>
> https://salsa.debian.org/kernel-team/linux/-/merge_requests/852
>
> Looking forward to new ia64 kernels for Sid.

I don't think this patch is acceptable in its current form as it
modifies the Makefile globally so that the flag is passed on to
the host compiler for all architectures.

[John Paul Adrian Glaubitz]

On Mon, 2023-09-18 at 22:36 +0200, Frank Scheiner wrote:
> > I don't think this patch is acceptable in its current form as it
> > modifies the Makefile globally so that the flag is passed on to
> > the host compiler for all architectures.
>
> Yes, thought about that, too. Know a better solution? Is there a way to
> make this only affecting compilations for ia64?

You could modify CFLAGS globally in debian/rules specifically for ia64:

ifneq (,$(filter $(DEB_HOST_ARCH),ia64))
export DEB_CFLAGS_MAINT_APPEND += -fno-var-tracking
endif

[Frank Scheiner]

On 18.09.23 22:42, John Paul Adrian Glaubitz wrote:
> On Mon, 2023-09-18 at 22:36 +0200, Frank Scheiner wrote:
>>> I don't think this patch is acceptable in its current form as it
>>> modifies the Makefile globally so that the flag is passed on to
>>> the host compiler for all architectures.
>>
>> Yes, thought about that, too. Know a better solution? Is there a way to
>> make this only affecting compilations for ia64?
>
> You could modify CFLAGS globally in debian/rules specifically for ia64:
>
> ifneq (,$(filter $(DEB_HOST_ARCH),ia64))
> export DEB_CFLAGS_MAINT_APPEND += -fno-var-tracking
> endif

That's the other extreme then, everything for ia64 gets compiled with
`-fno-var-tracking`, or? Is that more acceptable?

Maybe the per-file CFLAGS are also effective when included in the
architecture Makefile in `arch/ia64/`?

Or maybe we could query the arch from within the two Makefiles and only
apply `-fno-var-tracking` when we compile for ia64.

I'll have a look into that tomorrow.

Cheers,
Frank

[John Paul Adrian Glaubitz]

On Mon, 2023-09-18 at 22:56 +0200, Frank Scheiner wrote:
> That's the other extreme then, everything for ia64 gets compiled with
> `-fno-var-tracking`, or? Is that more acceptable?

It's certainly more acceptable than messing around with other targets.

> Maybe the per-file CFLAGS are also effective when included in the
> architecture Makefile in `arch/ia64/`?

No idea.

> Or maybe we could query the arch from within the two Makefiles and only
> apply `-fno-var-tracking` when we compile for ia64.
>
> I'll have a look into that tomorrow.

[Frank Scheiner]

The MR ([1]) was updated and now uses `-fno-var-tracking` for the
respective files only when the target architecture is ia64. This works
like so for example:

```
ifeq ($(ARCH),ia64)
CFLAGS_bnx2x_sp.o += -fno-var-tracking
endif
```

Tested via a local kernel build where one case was using a wrong target
architecture (i386) at first to see it failing and then working after
changing that back to the correct target architecture (ia64).

[1]: https://salsa.debian.org/kernel-team/linux/-/merge_requests/852

Cheers,
Frank

[Frank Scheiner]

On 19.09.23 20:33, John Paul Adrian Glaubitz wrote:

> On Tue, 2023-09-19 at 20:26 +0200, Frank Scheiner wrote:
>> The MR ([1]) was updated and now uses `-fno-var-tracking` for the
>> respective files only when the target architecture is ia64. This works
>> like so for example:
>>
>> ```
>> ifeq ($(ARCH),ia64)
>> CFLAGS_bnx2x_sp.o += -fno-var-tracking
>> endif
>> ```
>>
>> Tested via a local kernel build where one case was using a wrong target
>> architecture (i386) at first to see it failing and then working after
>> changing that back to the correct target architecture (ia64).
>>
>> [1]: https://salsa.debian.org/kernel-team/linux/-/merge_requests/852
>

> This looks reasonable to me and I think we can have this merged as a temporary
> workaround so that we can get 6.5 and 6.6 kernels built for ia64 on the buildds.

You're welcome.

Cheers,
Frank

[John Paul Adrian Glaubitz]

On Tue, 2023-09-19 at 20:26 +0200, Frank Scheiner wrote:

> The MR ([1]) was updated and now uses `-fno-var-tracking` for the
> respective files only when the target architecture is ia64. This works
> like so for example:
>
> ```
> ifeq ($(ARCH),ia64)
> CFLAGS_bnx2x_sp.o += -fno-var-tracking
> endif
> ```
>
> Tested via a local kernel build where one case was using a wrong target
> architecture (i386) at first to see it failing and then working after
> changing that back to the correct target architecture (ia64).
>
> [1]: https://salsa.debian.org/kernel-team/linux/-/merge_requests/852

This looks reasonable to me and I think we can have this merged as a temporary
workaround so that we can get 6.5 and 6.6 kernels built for ia64 on the buildds.