Boot regression in Linux 6.1 gone for Montecito in 6.3

Frank Scheiner

unread,

May 20, 2023, 2:20:05 PM5/20/23

to

Dear all,

I today noticed that a new Debian kernel is available from [1]. Thanks
to whoever built that one - most likely Adrian.

[1]:
http://ftp.ports.debian.org/debian-ports/pool-ia64/main/l/linux/linux-image-6.3.0-0-mckinley_6.3.2-1~exp1_ia64.deb

I currently have the rx2620 and the rx2800-i2 at home and gave it a test
on them, and...

...it is working more or less flawlessly on both systems. The boot
regression at where the initramfs is extracted is gone for the rx2620.

@Pedro:
Maybe you give this kernel a try on your rx2660 to see if it is working
for that one, too. It will take me until next week to test it on my
Montvale driven systems and the rx4640 with Madisons.

There is a reproducible issue for the rx2800-i2, though, happening just
after the first occurance of a message from the igb driver for the four
built-in PCI-e NICs (longer log on [2]). I cut the intertwined USB
messages here:

```
[ 37.566439] ------------[ cut here ]------------
[ 37.571640] WARNING: CPU: 7 PID: 155 at kernel/irq/msi.c:196
msi_domain_free_descs+0x240/0x280
[ 37.582425] GSI 19 (level, low) -> CPU 7 (0x0700) vector 75
[ 37.583481] Modules linked in: igb(+) uhci_hcd(+) ehci_pci ehci_hcd
i2c_algo_bit usbcore ptp pps_core i2c_core usb_common
[ 37.603864] CPU: 7 PID: 155 Comm: (udev-worker) Not tainted
6.3.0-0-mckinley #1 Debian 6.3.2-1~exp1
[ 37.622180] Hardware name: hp Integrity rx2800 i2, BIOS 01.93 09/12/2012
[ 37.622180]
[ 37.622180] Call Trace:
[ 37.640068] [<a0000001000156d0>] show_stack.part.0+0x30/0x60
[ 37.640068] sp=e00000084649fa70
bsp=e000000846499c18
[ 37.652236] [<a000000100015950>] show_stack+0x90/0xc0
[ 37.652236] sp=e00000084649fa70
bsp=e000000846499be0
[ 37.696069] [<a0000001014c3750>] dump_stack_lvl+0xf0/0x160
[ 37.696069] sp=e00000084649fc40
bsp=e000000846499b70
[ 37.720069] [<a0000001014c37f0>] dump_stack+0x30/0x60
[ 37.720069] sp=e00000084649fc40
bsp=e000000846499b18
[ 37.720069] [<a00000010006c060>] __warn+0x160/0x200
[ 37.720069] sp=e00000084649fc40
bsp=e000000846499ad0
[ 37.748236] [<a00000010006c400>] warn_slowpath_fmt+0x300/0x360
[ 37.748236] sp=e00000084649fc40
bsp=e000000846499a58
[ 37.776069] [<a000000100199060>] msi_domain_free_descs+0x240/0x280
[ 37.776069] sp=e00000084649fc70
bsp=e000000846499a18
[ 37.800069] [<a000000100199cd0>]
msi_domain_free_msi_descs_range+0x50/0x80
[ 37.800069] sp=e00000084649fc80
bsp=e0000008464999d8
[ 37.826759] [<a000000100c26cf0>] pci_msi_teardown_msi_irqs+0x90/0xe0
[ 37.826759] sp=e00000084649fc90
bsp=e0000008464999b8
[ 37.850941] [<a000000100c26010>] pci_free_msi_irqs+0x30/0xa0
[ 37.850941] sp=e00000084649fc90
bsp=e000000846499990
[ 37.874941] [<a000000100c21ed0>] pci_disable_msix+0xb0/0xe0
[ 37.874941] sp=e00000084649fc90
bsp=e000000846499968
[ 37.898941] [<a000000200baa130>]
igb_reset_interrupt_capability+0x230/0x260 [igb]
[ 37.898941] sp=e00000084649fc90
bsp=e000000846499928
[ 37.920068] [<a000000200bcb9d0>] igb_probe+0x970/0x3380 [igb]
[ 37.920068] sp=e00000084649fc90
bsp=e000000846499858
[ 37.950180] [<a000000100c082d0>] local_pci_probe+0x90/0x140
[ 37.950180] sp=e00000084649fca0
bsp=e000000846499818
[ 37.974181] [<a000000100c09d50>] pci_device_probe+0x190/0x520
[ 37.974181] sp=e00000084649fca0
bsp=e0000008464997c0
[ 37.998180] [<a000000100e61100>] really_probe+0x260/0xb20
[ 37.998180] sp=e00000084649fcc0
bsp=e000000846499770
[ 38.024068] [<a000000100e61c50>] __driver_probe_device+0x290/0x420
[ 38.024068] sp=e00000084649fcc0
bsp=e000000846499730
[ 38.024068] [<a000000100e61e30>] driver_probe_device+0x50/0x160
[ 38.024068] sp=e00000084649fcc0
bsp=e0000008464996e8
[ 38.052236] [<a000000100e626c0>] __driver_attach+0x380/0x4c0
[ 38.052236] sp=e00000084649fcc0
bsp=e0000008464996b0
[ 38.052236] [<a000000100e5bc40>] bus_for_each_dev+0x100/0x1c0
[ 38.052236] sp=e00000084649fcc0
bsp=e000000846499668
[ 38.086181] [<a000000100e5fd40>] driver_attach+0x40/0x60
[ 38.086181] sp=e00000084649fcd0
bsp=e000000846499648
[ 38.110180] [<a000000100e5ea90>] bus_add_driver+0x230/0x520
[ 38.110180] sp=e00000084649fcd0
bsp=e000000846499600
[ 38.134180] [<a000000100e64680>] driver_register+0x100/0x360
[ 38.134180] sp=e00000084649fcd0
bsp=e0000008464995c8
[ 38.142437] [<a000000100c07720>] __pci_register_driver+0xc0/0xe0
[ 38.142437] sp=e00000084649fcd0
bsp=e000000846499598
[ 38.166941] [<a0000002002480c0>] igb_init_module+0xc0/0x120 [igb]
[ 38.166941] sp=e00000084649fcd0
bsp=e000000846499578
[ 38.184236] [<a00000010000a940>] do_one_initcall+0xe0/0x560
[ 38.184236] sp=e00000084649fcd0
bsp=e000000846499548
[ 38.184236] [<a0000001001d2280>] do_init_module+0xa0/0x440
[ 38.184236] sp=e00000084649fd10
bsp=e000000846499510
[ 38.214439] [<a0000001001d7820>] load_module+0x4f20/0x5620
[ 38.214439] sp=e00000084649fd10
bsp=e000000846499378
[ 38.254437] [<a0000001001d84a0>] __do_sys_finit_module+0x180/0x220
[ 38.254437] sp=e00000084649fd90
bsp=e000000846499340
[ 38.264236] [<a0000001001d85d0>] sys_finit_module+0x30/0x60
[ 38.264236] sp=e00000084649fe30
bsp=e0000008464992e8
[ 38.264236] [<a00000010000c9e0>] ia64_ret_from_syscall+0x0/0x20
[ 38.264236] sp=e00000084649fe30
bsp=e0000008464992e0
[ 38.303510] ---[ end trace 0000000000000000 ]---
```

[2]: https://pastebin.com/jrUvjRcN

****

Earlier today I built a 6.4.0-rc2 kernel with kernel config based on
"config-6.1.0-9-mckinley" and localmodconfig matching the needed modules
for my rx2620 **and** rx2800-i2 combined.

Unfortunately the resulting kernel "crashes" as soon as udevd starts and
that for both systems similarly. But yeah, the initramfs issues are gone
for the rx2620. If someone is interested, the error messages are on [3]
and [4] for rx2620 and rx2800-i2 respectively.

[3]: https://pastebin.com/SAUKbG7Z

[4]: https://pastebin.com/v1TTB2x3

Well, rc2 might just be to early to conclude that there is a new error.
And it could also be an issue of my kernel config. I'll try to observe
how later kernels behave.

Cheers,
Frank

John Paul Adrian Glaubitz

unread,

May 20, 2023, 3:20:04 PM5/20/23

to

Hello Frank!

On Sat, 2023-05-20 at 20:19 +0200, Frank Scheiner wrote:
> I today noticed that a new Debian kernel is available from [1]. Thanks
> to whoever built that one - most likely Adrian.

The buildd built the kernel, not me ;-).

> I currently have the rx2620 and the rx2800-i2 at home and gave it a test
> on them, and...
>
> ...it is working more or less flawlessly on both systems. The boot
> regression at where the initramfs is extracted is gone for the rx2620.

And I can confirm that this kernel boots fine on my RX2660 again. The latest
stable kernel from the 6.1 tree (6.1.27) is still affected by this bug though,
so hopefully whatever fixed this bug will get backported to the 6.1 tree.

I got this message after rebooting the system with 6.3.

> Earlier today I built a 6.4.0-rc2 kernel with kernel config based on
> "config-6.1.0-9-mckinley" and localmodconfig matching the needed modules
> for my rx2620 **and** rx2800-i2 combined.
>
> Unfortunately the resulting kernel "crashes" as soon as udevd starts and
> that for both systems similarly. But yeah, the initramfs issues are gone
> for the rx2620. If someone is interested, the error messages are on [3]
> and [4] for rx2620 and rx2800-i2 respectively.
>
> [3]: https://pastebin.com/SAUKbG7Z
>
> [4]: https://pastebin.com/v1TTB2x3
>
> Well, rc2 might just be to early to conclude that there is a new error.
> And it could also be an issue of my kernel config. I'll try to observe
> how later kernels behave.

Maybe you will have time to bisect it tomorrow? :-)

Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer
`. `' Physicist
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

Frank Scheiner

unread,

May 20, 2023, 4:02:15 PM5/20/23

to

Hi Adrian,

On 20.05.23 21:16, John Paul Adrian Glaubitz wrote:
> Hello Frank!
>
> On Sat, 2023-05-20 at 20:19 +0200, Frank Scheiner wrote:
>> I today noticed that a new Debian kernel is available from [1]. Thanks
>> to whoever built that one - most likely Adrian.
>
> The buildd built the kernel, not me ;-).

Yeah, sure. :-) Did you also check out 6.2?

>> I currently have the rx2620 and the rx2800-i2 at home and gave it a test
>> on them, and...
>>
>> ...it is working more or less flawlessly on both systems. The boot
>> regression at where the initramfs is extracted is gone for the rx2620.
>
> And I can confirm that this kernel boots fine on my RX2660 again. The latest
> stable kernel from the 6.1 tree (6.1.27) is still affected by this bug though,
> so hopefully whatever fixed this bug will get backported to the 6.1 tree.
>

>> There is a reproducible issue for the rx2800-i2, though, happening just
>> after the first occurance of a message from the igb driver for the four
>> built-in PCI-e NICs (longer log on [2]). I cut the intertwined USB
>> messages here:

>> [...]

>
> I got this message after rebooting the system with 6.3.

It doesn't actually affect operation noticeably. As the rx2660 has
different NICs (BCM5704) and those are attached via PCI-X but also has
PCIe, I suspect this is actually related to PCIe and not the igb NICs in
the rx2800-i2 then.

>> Earlier today I built a 6.4.0-rc2 kernel with kernel config based on
>> "config-6.1.0-9-mckinley" and localmodconfig matching the needed modules
>> for my rx2620 **and** rx2800-i2 combined.
>>
>> Unfortunately the resulting kernel "crashes" as soon as udevd starts and
>> that for both systems similarly. But yeah, the initramfs issues are gone
>> for the rx2620. If someone is interested, the error messages are on [3]
>> and [4] for rx2620 and rx2800-i2 respectively.
>>
>> [3]: https://pastebin.com/SAUKbG7Z
>>
>> [4]: https://pastebin.com/v1TTB2x3
>>
>> Well, rc2 might just be to early to conclude that there is a new error.
>> And it could also be an issue of my kernel config. I'll try to observe
>> how later kernels behave.
>
> Maybe you will have time to bisect it tomorrow? :-)

Unfortunately not, I'm away tomorrow. Wouldn't it make more sense to
wait for the 6.4 release to be sure it's not due to something unrelated
to ia64 (1 run takes my rx2800-i2 40 mins and it only produces a handful
of modules in that time)?

First I'll try w/o an intramfs and all needed drivers compiled in, to
see if it will boot through then.

Cheers,
Frank

John Paul Adrian Glaubitz

unread,

May 20, 2023, 4:10:03 PM5/20/23

to

Hello!

On Sat, 2023-05-20 at 21:51 +0200, Frank Scheiner wrote:
> On 20.05.23 21:16, John Paul Adrian Glaubitz wrote:
> > Hello Frank!
> >
> > On Sat, 2023-05-20 at 20:19 +0200, Frank Scheiner wrote:
> > > I today noticed that a new Debian kernel is available from [1]. Thanks
> > > to whoever built that one - most likely Adrian.
> >
> > The buildd built the kernel, not me ;-).
>
> Yeah, sure. :-) Did you also check out 6.2?

No, I didn't since Debian's kernel maintainers skipped 6.2:

> https://buildd.debian.org/status/logs.php?pkg=linux&arch=ia64

> > > I currently have the rx2620 and the rx2800-i2 at home and gave it a test
> > > on them, and...
> > >
> > > ...it is working more or less flawlessly on both systems. The boot
> > > regression at where the initramfs is extracted is gone for the rx2620.
> >
> > And I can confirm that this kernel boots fine on my RX2660 again. The latest
> > stable kernel from the 6.1 tree (6.1.27) is still affected by this bug though,
> > so hopefully whatever fixed this bug will get backported to the 6.1 tree.
> >
> > > There is a reproducible issue for the rx2800-i2, though, happening just
> > > after the first occurance of a message from the igb driver for the four
> > > built-in PCI-e NICs (longer log on [2]). I cut the intertwined USB
> > > messages here:
> > > [...]
> >
> > I got this message after rebooting the system with 6.3.
>
> It doesn't actually affect operation noticeably. As the rx2660 has
> different NICs (BCM5704) and those are attached via PCI-X but also has
> PCIe, I suspect this is actually related to PCIe and not the igb NICs in
> the rx2800-i2 then.

OK.

> > > Earlier today I built a 6.4.0-rc2 kernel with kernel config based on
> > > "config-6.1.0-9-mckinley" and localmodconfig matching the needed modules
> > > for my rx2620 **and** rx2800-i2 combined.
> > >
> > > Unfortunately the resulting kernel "crashes" as soon as udevd starts and
> > > that for both systems similarly. But yeah, the initramfs issues are gone
> > > for the rx2620. If someone is interested, the error messages are on [3]
> > > and [4] for rx2620 and rx2800-i2 respectively.
> > >
> > > [3]: https://pastebin.com/SAUKbG7Z
> > >
> > > [4]: https://pastebin.com/v1TTB2x3
> > >
> > > Well, rc2 might just be to early to conclude that there is a new error.
> > > And it could also be an issue of my kernel config. I'll try to observe
> > > how later kernels behave.
> >
> > Maybe you will have time to bisect it tomorrow? :-)
>
> Unfortunately not, I'm away tomorrow. Wouldn't it make more sense to
> wait for the 6.4 release to be sure it's not due to something unrelated
> to ia64 (1 run takes my rx2800-i2 40 mins and it only produces a handful
> of modules in that time)?

You should cross-compile your kernel. I can build a kernel for my rx2660
on an AMD EPYC in 2-3 minutes. And it's definitely better to catch the
regression before the release as this way you will get the fix landed
for 6.4 instead of 6.5.

Frank Scheiner

unread,

May 20, 2023, 4:30:04 PM5/20/23

to

Hi,

On 20.05.23 22:07, John Paul Adrian Glaubitz wrote:
> Hello!
>
> On Sat, 2023-05-20 at 21:51 +0200, Frank Scheiner wrote:
>> On 20.05.23 21:16, John Paul Adrian Glaubitz wrote:
>>> Hello Frank!
>>>
>>> On Sat, 2023-05-20 at 20:19 +0200, Frank Scheiner wrote:
>>>> I today noticed that a new Debian kernel is available from [1]. Thanks
>>>> to whoever built that one - most likely Adrian.
>>>
>>> The buildd built the kernel, not me ;-).
>>
>> Yeah, sure. :-) Did you also check out 6.2?
>
> No, I didn't since Debian's kernel maintainers skipped 6.2:
>
>> https://buildd.debian.org/status/logs.php?pkg=linux&arch=ia64

I see.

>>>> Earlier today I built a 6.4.0-rc2 kernel with kernel config based on
>>>> "config-6.1.0-9-mckinley" and localmodconfig matching the needed modules
>>>> for my rx2620 **and** rx2800-i2 combined.
>>>>
>>>> Unfortunately the resulting kernel "crashes" as soon as udevd starts and
>>>> that for both systems similarly. But yeah, the initramfs issues are gone
>>>> for the rx2620. If someone is interested, the error messages are on [3]
>>>> and [4] for rx2620 and rx2800-i2 respectively.
>>>>
>>>> [3]: https://pastebin.com/SAUKbG7Z
>>>>
>>>> [4]: https://pastebin.com/v1TTB2x3
>>>>
>>>> Well, rc2 might just be to early to conclude that there is a new error.
>>>> And it could also be an issue of my kernel config. I'll try to observe
>>>> how later kernels behave.
>>>
>>> Maybe you will have time to bisect it tomorrow? :-)
>>
>> Unfortunately not, I'm away tomorrow. Wouldn't it make more sense to
>> wait for the 6.4 release to be sure it's not due to something unrelated
>> to ia64 (1 run takes my rx2800-i2 40 mins and it only produces a handful
>> of modules in that time)?
>
> You should cross-compile your kernel. I can build a kernel for my rx2660
> on an AMD EPYC in 2-3 minutes.

Oh I don't have new things... :-/

My "newest" x86 MP system is a Sun X4270 w/2 x Xeon X5570s. I recently
upgraded its memory to 144 GiB so could compile the kernel(s) in a RAM
disk there, not sure how much of a boost that would be. IIRC I used it
for cross-compiling sparc64 kernels some time ago.

> And it's definitely better to catch the
> regression before the release as this way you will get the fix landed
> for 6.4 instead of 6.5.

Cheers,
Frank

Gatis Visnevskis

unread,

May 20, 2023, 5:20:04 PM5/20/23

to

Hi all, i have rx2620 sitting in shelf for some time. Where can i get working iso to try? Last time i booted 3.2.0 kernel.

Gatis

John Paul Adrian Glaubitz

unread,

May 21, 2023, 3:50:05 AM5/21/23

to

Hi Gatis!

On Sat, 2023-05-20 at 23:58 +0300, Gatis Visnevskis wrote:
> Hi all, i have rx2620 sitting in shelf for some time. Where can i
> get working iso to try? Last time i booted 3.2.0 kernel.

Current images can be found here:

> https://cdimage.debian.org/cdimage/ports/snapshots/

Note that the latest image probably won't boot on your RX2620 due to the
kernel issue that Frank mentioned in his original mail, so you will have
to use the image from last December.

unread,

May 25, 2023, 2:32:12 PM5/25/23

to

On 25.05.23 19:22, John Paul Adrian Glaubitz wrote:
> Hi!
>
> On Thu, 2023-05-25 at 19:17 +0200, Frank Scheiner wrote:
>>> This is great! I am definitely going to try building my Itanium kernels from Arm64 now.
>>
>> Say, what Arm64 machine do you have at hand?
>
> My wild guess would be an M1/M2 Mac ;-).

Right, I always forget that they can run GNU/Linux (and also OpenBSD
IIC), too. I was originally thinking about some Ampere Altra powered
server or something like that.

Cheers,
Frank