Boot regression in Linux v6.4-rc3

[Frank Scheiner]

Dear all,

there is a boot regression in effect in Linux v6.4-rc3 that affects at
least:

* rx2620 (w/2 x Montecito and zx1)
* rx2800-i2 (w/1 x Tukwila)

...(see second part of [1] and following posts for more details, [2] and
[3] for the respective logs), example here:

```
ELILO v3.16 for EFI/IA-64
..
Uncompressing Linux... done
Loading file AC100221.initrd.img...done
[ 0.000000] Linux version 6.4.0-rc3 (root@x4270) (ia64-linux-gcc
(GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Thu May 25 15:52:20
CEST 2023
[ 0.000000] efi: EFI v1.1 by HP
[ 0.000000] efi: SALsystab=0x3ee7a000 ACPI 2.0=0x3fe2a000
ESI=0x3ee7b000 SMBIOS=0x3ee7c000 HCDP=0x3fe28000
[ 0.000000] PCDP: v3 at 0x3fe28000
[ 0.000000] earlycon: uart8250 at MMIO 0x00000000f4050000 (options
'9600n8')
[ 0.000000] printk: bootconsole [uart8250] enabled
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x000000003FE2A000 000028 (v02 HP )
[ 0.000000] ACPI: XSDT 0x000000003FE2A02C 0000CC (v01 HP rx2620
00000000 HP 00000000)
[...]
[ 3.793350] Run /init as init process
Loading, please wait...
Starting systemd-udevd version 252.6-1
[ 3.951100] ------------[ cut here ]------------
[ 3.951100] WARNING: CPU: 6 PID: 140 at kernel/module/main.c:1547
__layout_sections+0x370/0x3c0
[ 3.949512] Unable to handle kernel paging request at virtual address
1000000000000000
[ 3.951100] Modules linked in:
[ 3.951100] CPU: 6 PID: 140 Comm: (udev-worker) Not tainted 6.4.0-rc3 #1
[ 3.956161] (udev-worker)[142]: Oops 11003706212352 [1]
[ 3.951774] Hardware name: hp server rx2620 , BIOS
04.29
11/30/2007
[ 3.951774]
[ 3.951774] Call Trace:
[ 3.958339] Unable to handle kernel paging request at virtual address
1000000000000000
[ 3.956161] Modules linked in:
[ 3.951774] [<a0000001000156d0>] show_stack.part.0+0x30/0x60
[ 3.951774] sp=e000000183a67b20
bsp=e000000183a61628
[ 3.956161]
[ 3.956161]
```

[1]: https://lists.debian.org/debian-ia64/2023/05/msg00010.html

[2]: https://pastebin.com/SAUKbG7Z

[3]: https://pastebin.com/v1TTB2x3

With the needed modules compiled into the kernel the rx2620 (only tested
there yet) boots correctly, though for v6.4-rc2 with kernel oopses (with
similar content), for v6.4-rc3 actually w/o kernel oopses.

According to bisecting between:

GOOD: `cec24b8b6bb841a19b5c5555b600a511a8988100` and

BAD: `b6a7828502dc769e1a5329027bc5048222fa210a` (already in effect there)

...the problem was introduced with:

```
root@x4270:/usr/src/linux-on-ramdisk# git bisect bad
ac3b43283923440900b4f36ca5f9f0b1ca43b70e is the first bad commit
commit ac3b43283923440900b4f36ca5f9f0b1ca43b70e
Author: Song Liu <so...@kernel.org>
Date: Mon Feb 6 16:28:02 2023 -0800

module: replace module_layout with module_memory

module_layout manages different types of memory (text, data,
rodata, etc.)
in one allocation, which is problematic for some reasons:

1. It is hard to enable CONFIG_STRICT_MODULE_RWX.
2. It is hard to use huge pages in modules (and not break strict rwx).
3. Many archs uses module_layout for arch-specific data, but it is not
obvious how these data are used (are they RO, RX, or RW?)

Improve the scenario by replacing 2 (or 3) module_layout per module
with
up to 7 module_memory per module:

MOD_TEXT,
MOD_DATA,
MOD_RODATA,
MOD_RO_AFTER_INIT,
MOD_INIT_TEXT,
MOD_INIT_DATA,
MOD_INIT_RODATA,

and allocating them separately. This adds slightly more entries to
mod_tree (from up to 3 entries per module, to up to 7 entries per
module). However, this at most adds a small constant overhead to
__module_address(), which is expected to be fast.

Various archs use module_layout for different data. These data are put
into different module_memory based on their location in module_layout.
IOW, data that used to go with text is allocated with
MOD_MEM_TYPE_TEXT;
data that used to go with data is allocated with MOD_MEM_TYPE_DATA,
etc.

module_memory simplifies quite some of the module code. For example,
ARCH_WANTS_MODULES_DATA_IN_VMALLOC is a lot cleaner, as it just uses a
different allocator for the data. kernel/module/strict_rwx.c is also
much cleaner with module_memory.

Signed-off-by: Song Liu <so...@kernel.org>
Cc: Luis Chamberlain <mcg...@kernel.org>
Cc: Thomas Gleixner <tg...@linutronix.de>
Cc: Peter Zijlstra <pet...@infradead.org>
Cc: Guenter Roeck <li...@roeck-us.net>
Cc: Christophe Leroy <christop...@csgroup.eu>
Reviewed-by: Thomas Gleixner <tg...@linutronix.de>
Reviewed-by: Christophe Leroy <christop...@csgroup.eu>
Reviewed-by: Luis Chamberlain <mcg...@kernel.org>
Signed-off-by: Luis Chamberlain <mcg...@kernel.org>

arch/arc/kernel/unwind.c | 12 +-
arch/arm/kernel/module-plts.c | 9 +-
arch/arm64/kernel/module-plts.c | 13 +-
arch/ia64/kernel/module.c | 24 +--
arch/mips/kernel/vpe.c | 11 +-
arch/parisc/kernel/module.c | 51 ++----
arch/powerpc/kernel/module_32.c | 7 +-
arch/s390/kernel/module.c | 26 +--
arch/x86/kernel/callthunks.c | 4 +-
arch/x86/kernel/module.c | 4 +-
include/linux/module.h | 89 +++++++---
kernel/module/internal.h | 40 ++---
kernel/module/kallsyms.c | 58 ++++---
kernel/module/kdb.c | 17 +-
kernel/module/main.c | 375
++++++++++++++++++++--------------------
kernel/module/procfs.c | 16 +-
kernel/module/strict_rwx.c | 99 ++---------
kernel/module/tree_lookup.c | 39 ++---
18 files changed, 427 insertions(+), 467 deletions(-)

root@x4270:/usr/src/linux-on-ramdisk# git bisect log
git bisect start
# status: waiting for both good and bad commits
# good: [cec24b8b6bb841a19b5c5555b600a511a8988100] Merge tag
'char-misc-6.4-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc
git bisect good cec24b8b6bb841a19b5c5555b600a511a8988100
# status: waiting for bad commit, 1 good commit known
# bad: [b6a7828502dc769e1a5329027bc5048222fa210a] Merge tag
'modules-6.4-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux
git bisect bad b6a7828502dc769e1a5329027bc5048222fa210a
# bad: [3f0dedc39039a75670817a1afffa77b6cee077cb] dmaengine: remove
MODULE_LICENSE in non-modules
git bisect bad 3f0dedc39039a75670817a1afffa77b6cee077cb
# bad: [b10addf37bbcaee66672eb54c15532266c8daea6] module: add
symbol-name to pr_debug Absolute symbol
git bisect bad b10addf37bbcaee66672eb54c15532266c8daea6
# bad: [85e6f61c134f111232d27d3f63667c1bccbbc12d] module: move early
sanity checks into a helper
git bisect bad 85e6f61c134f111232d27d3f63667c1bccbbc12d
# bad: [05777499a81298ef7e4a5e32a6f744f1f937a80c] ARM: dyndbg: allow
including dyndbg.h in decompressor
git bisect bad 05777499a81298ef7e4a5e32a6f744f1f937a80c
# bad: [efaa2496bae66f0a78efa60d9b73ceef5ec63d79] module: fix MIPS
module_layout -> module_memory
git bisect bad efaa2496bae66f0a78efa60d9b73ceef5ec63d79
# bad: [9e07f161717ab8e8ac1206bf82546511e24cbb7b] module: Remove the
unused function within
git bisect bad 9e07f161717ab8e8ac1206bf82546511e24cbb7b
# bad: [ac3b43283923440900b4f36ca5f9f0b1ca43b70e] module: replace
module_layout with module_memory
git bisect bad ac3b43283923440900b4f36ca5f9f0b1ca43b70e
# first bad commit: [ac3b43283923440900b4f36ca5f9f0b1ca43b70e] module:
replace module_layout with module_memory
```

...and merged with commit `b6a7828502dc769e1a5329027bc5048222fa210a`:

```
commit b6a7828502dc769e1a5329027bc5048222fa210a
Merge: d06f5a3f7140 8660484ed1cf
Author: Linus Torvalds <torv...@linux-foundation.org>
Date: Thu Apr 27 16:36:55 2023 -0700

Merge tag 'modules-6.4-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux

Pull module updates from Luis Chamberlain:
"The summary of the changes for this pull requests is:

- Song Liu's new struct module_memory replacement

- Nick Alcock's MODULE_LICENSE() removal for non-modules

- My cleanups and enhancements to reduce the areas where we vmalloc
module memory for duplicates, and the respective debug code which
proves the remaining vmalloc pressure comes from userspace.
[...]
```

Could someone have a look into this, please?

Cheers,
Frank

P.S.
There is also a bug for this specific commit:

```
kmemleaks on ac3b43283923 ("module: replace module_layout with
module_memory")
```

...on [4], reported on 2023-04-03, but I don't know if its content is
related to the problems on ia64.

[4]: https://bugzilla.kernel.org/show_bug.cgi?id=217296

[Song Liu]

Hi Frank,

Thanks for the report.

It seems the error happened during the WARN_ON_ONCE. Could you
please try whether something like the following fixes it?

diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..ae42dfc1a815 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1537,7 +1537,7 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
|| is_init != module_init_layout_section(sname))
continue;

- if (WARN_ON_ONCE(type == MOD_INVALID))
+ if (type == MOD_INVALID)
continue;

s->sh_entsize =
module_get_offset_and_type(mod, type, s, i);


If that doesn't work, maybe we need something like this:

diff --git i/arch/ia64/kernel/module.c w/arch/ia64/kernel/module.c
index 3661135da9d9..4e9a7f0498e2 100644
--- i/arch/ia64/kernel/module.c
+++ w/arch/ia64/kernel/module.c
@@ -815,7 +815,7 @@ apply_relocate_add (Elf64_Shdr *sechdrs, const
char *strtab, unsigned int symind
uint64_t gp;
struct module_memory *mod_mem;

- mod_mem = &mod->mem[MOD_DATA];
+ mod_mem = &mod->mem[MOD_TEXT];
if (mod_mem->size > MAX_LTOFF)
/*
* This takes advantage of fact that
SHF_ARCH_SMALL gets allocated


Song

[Frank Scheiner]

Hi Song,

On 26.05.23 18:49, Song Liu wrote:
> Hi Frank,
>
> Thanks for the report.

Sure, thanks for your help in this.

> It seems the error happened during the WARN_ON_ONCE. Could you
> please try whether something like the following fixes it?
>
> diff --git i/kernel/module/main.c w/kernel/module/main.c
> index 0f9183f1ca9f..ae42dfc1a815 100644
> --- i/kernel/module/main.c
> +++ w/kernel/module/main.c
> @@ -1537,7 +1537,7 @@ static void __layout_sections(struct module
> *mod, struct load_info *info, bool i
> || is_init != module_init_layout_section(sname))
> continue;
>
> - if (WARN_ON_ONCE(type == MOD_INVALID))
> + if (type == MOD_INVALID)
> continue;
>
> s->sh_entsize =
> module_get_offset_and_type(mod, type, s, i);

Ok, tried that as -patch1 on top of v6.4-rc3, but didn't help, see [1].

[1]: https://pastebin.com/UK9v30Ae

> If that doesn't work, maybe we need something like this:
>
> diff --git i/arch/ia64/kernel/module.c w/arch/ia64/kernel/module.c
> index 3661135da9d9..4e9a7f0498e2 100644
> --- i/arch/ia64/kernel/module.c
> +++ w/arch/ia64/kernel/module.c
> @@ -815,7 +815,7 @@ apply_relocate_add (Elf64_Shdr *sechdrs, const
> char *strtab, unsigned int symind
> uint64_t gp;
> struct module_memory *mod_mem;
>
> - mod_mem = &mod->mem[MOD_DATA];
> + mod_mem = &mod->mem[MOD_TEXT];
> if (mod_mem->size > MAX_LTOFF)
> /*
> * This takes advantage of fact that
> SHF_ARCH_SMALL gets allocated

Tried that one as -patch2 on top of v6.4-rc3, but didn't help, see [2].

[2]: https://pastebin.com/gLupBndU

I also tried both patches as -patch1plus2 on top of v6.4-rc3 with a
similar result, see [3].

[3]: https://pastebin.com/7pXBG5vx

Cheers,
Frank

[Song Liu]

Thanks for running the test.

I am not very familiar with the code, but I think we shouldn't hit that
WARN_ON_ONCE. Could you please try with the follow patch to see
which section caused this issue?

Thanks,
Song

diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..caf3d30cd133 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1537,8 +1537,11 @@ static void __layout_sections(struct module

*mod, struct load_info *info, bool i
|| is_init != module_init_layout_section(sname))
continue;

- if (WARN_ON_ONCE(type == MOD_INVALID))

+ if (WARN_ON_ONCE(type == MOD_INVALID)) {
+ pr_warn("%s: section %s (sh_flags
%llx) matched to MOD_INVALID\n", __func__,
+ sname, s->sh_flags);
continue;
+ }

s->sh_entsize =
module_get_offset_and_type(mod, type, s, i);

pr_debug("\t%s\n", sname);

[Linus Torvalds]

On Fri, May 26, 2023 at 2:59 PM Luis Chamberlain <mcg...@kernel.org> wrote:
>
> Not saying that debugging commit ac3b4328392344 ("module: replace
> module_layout with module_memory") is going to be impossible, quite
> the contrary I think it would be good to root cause it, if possible,
> as perhaps it may also be similar to some other future oddball arch
> bug later that may come up.

I don't have any context - the mailing lists in question that
apparently this came in on aren't in lore.

That said, that commit looks odd for the ia64 part.

In particular, this part:

- if (mod->core_layout.size > MAX_LTOFF)
+ struct module_memory *mod_mem;
+
+ mod_mem = &mod->mem[MOD_DATA];

in apply_relocate_add() (file: arch/ia64/kernel/module.c) seems suspect.

The previous place that used to look at "mod->core_layout.base"
converted that to "mod->mem[MOD_TEXT].base". As do other changes in
other architectures.

So that "MOD_DATA" looks *very* wrong. Shouldn't core_layout. be
translated to use "MOD_TEXT" instead?

Nothing else in the ia64 parts strike me as odd, but that one looks wrong to me.

But this is my "monkey see, monkey do" pattern matching reaction, not
from any deeper understanding of the problem (I can't even see the
report) or really even the code.

Linus

[Song Liu]

On Fri, May 26, 2023 at 3:22 PM Linus Torvalds
<torv...@linux-foundation.org> wrote:
>
> On Fri, May 26, 2023 at 2:59 PM Luis Chamberlain <mcg...@kernel.org> wrote:
> >
> > Not saying that debugging commit ac3b4328392344 ("module: replace
> > module_layout with module_memory") is going to be impossible, quite
> > the contrary I think it would be good to root cause it, if possible,
> > as perhaps it may also be similar to some other future oddball arch
> > bug later that may come up.
>
> I don't have any context - the mailing lists in question that
> apparently this came in on aren't in lore.
>
> That said, that commit looks odd for the ia64 part.
>
> In particular, this part:
>
> - if (mod->core_layout.size > MAX_LTOFF)
> + struct module_memory *mod_mem;
> +
> + mod_mem = &mod->mem[MOD_DATA];
>
> in apply_relocate_add() (file: arch/ia64/kernel/module.c) seems suspect.
>
> The previous place that used to look at "mod->core_layout.base"
> converted that to "mod->mem[MOD_TEXT].base". As do other changes in
> other architectures.
>
> So that "MOD_DATA" looks *very* wrong. Shouldn't core_layout. be
> translated to use "MOD_TEXT" instead?

MOD_DATA is likely wrong here. But as Frank tested, changing it to MOD_TEXT
didn't fix the issue. I suspect we missed some special cases when we updated
layout_sections().

Thanks,
Song

[Luis Chamberlain]

On Fri, May 26, 2023 at 12:55:14PM +0200, Frank Scheiner wrote:
> Dear all,
>
> there is a boot regression in effect in Linux v6.4-rc3 that affects at
> least:
>
> * rx2620 (w/2 x Montecito and zx1)
> * rx2800-i2 (w/1 x Tukwila)

Jesus, ia64 is even dropped from qemu as of 2.11. We're now around qemu
7.11 to give some perspective. I'm just wondering how to reproduce
testing easily instead of this ping pong back and forth for this
architecture for some oddball architectures.

Through commit 96ec72a3425d1 ("ia64: Mark architecture as orphaned")
it was noted even the old maintainer no longer had access to working
machines and so it was orphan'd.

Not saying that debugging commit ac3b4328392344 ("module: replace
module_layout with module_memory") is going to be impossible, quite
the contrary I think it would be good to root cause it, if possible,
as perhaps it may also be similar to some other future oddball arch
bug later that may come up.

But certainly just trying to see what options we have to test this
architecture.

And what's the status of removal for ia64 anyway?

Luis

[Frank Scheiner]

Hi,

On 26.05.23 23:01, Song Liu wrote:
> Thanks for running the test.

Thanks for staying with me.

> I am not very familiar with the code, but I think we shouldn't hit that
> WARN_ON_ONCE. Could you please try with the follow patch to see
> which section caused this issue?
>
> Thanks,
> Song
>
> diff --git i/kernel/module/main.c w/kernel/module/main.c
> index 0f9183f1ca9f..caf3d30cd133 100644
> --- i/kernel/module/main.c
> +++ w/kernel/module/main.c
> @@ -1537,8 +1537,11 @@ static void __layout_sections(struct module
> *mod, struct load_info *info, bool i
> || is_init != module_init_layout_section(sname))
> continue;
>
> - if (WARN_ON_ONCE(type == MOD_INVALID))
> + if (WARN_ON_ONCE(type == MOD_INVALID)) {
> + pr_warn("%s: section %s (sh_flags
> %llx) matched to MOD_INVALID\n", __func__,
> + sname, s->sh_flags);
> continue;
> + }
>
> s->sh_entsize =
> module_get_offset_and_type(mod, type, s, i);
> pr_debug("\t%s\n", sname);

I put that as -patch3 on top of 6.4-rc3, the result is on [1].

[1]: https://pastebin.com/KqnWL2pM

I this time put the whole console messages there, just in case some of
the earlier messages could be of any help. To jump to the actual oopses,
search for "Loading, please wait...".

Cheers,
Frank

[Frank Scheiner]

On 27.05.23 00:22, Linus Torvalds wrote:
> [...]

> But this is my "monkey see, monkey do" pattern matching reaction, not
> from any deeper understanding of the problem (I can't even see the
> report) or really even the code.

If it is of any help, my initial report is available for example via:

https://marc.info/?l=linux-ia64&m=168509859125505&w=2

...the whole thread is currently at:

https://marc.info/?t=168509868200003&r=1&w=2

Cheers,
Frank

[Linus Torvalds]

On Sat, May 27, 2023 at 12:01 AM Frank Scheiner <frank.s...@web.de> wrote:
>
> If it is of any help, my initial report is available for example via:
>
> https://marc.info/?l=linux-ia64&m=168509859125505&w=2
>
> ...the whole thread is currently at:
>
> https://marc.info/?t=168509868200003&r=1&w=2

This does make it clear just how great a mailing list archive lore is.
Konstantin, is there any particular reason why
linux...@vger.kernel.org isn't in lore? Is it just a rational hatred
of all things itanium?

Anyway, the WARN_ON() is likely related, but the bug is clearly an
unexpected page fault in __copy_user() when called by load_module().

The ia64 oops output is nasty, presumably because ia64 aggressively
inlines things. It would help a lot if you enabled debug info (maybe
you already do?) and then run the oops through
./scripts/decode_stacktrace.sh which will figure out line numbers,
inlining etc.

Because I don't even see why it would call __copy_user() in the first
place. This is 'finit_module()' that loads the module data from a
file, not user space.

So I guess it must be the strndup_user() in

mod->args = strndup_user(uargs, ~0UL >> 1);

but that doesn't look like it should even care about any module
layout. Plus I would have expected to see strndup_user() in the call
trace, but whatever.

End result: that ia64 trace is very hard to read, and _maybe_ running
it through the decode script might give more information about what it
is that triggers...

Linus

[Frank Scheiner]

Hi,

On 27.05.23 19:08, Linus Torvalds wrote:
> Anyway, the WARN_ON() is likely related, but the bug is clearly an
> unexpected page fault in __copy_user() when called by load_module().
>
> The ia64 oops output is nasty, presumably because ia64 aggressively
> inlines things. It would help a lot if you enabled debug info (maybe
> you already do?)

I believe it is enabled - I have at least CONFIG_DEBUG_KERNEL=y and
CONFIG_DEBUG_INFO=y - my kernel config is on [1] for reference.

[1]: https://pastebin.com/HRQtZ9vb

> and then run the oops through
> ./scripts/decode_stacktrace.sh which will figure out line numbers,
> inlining etc.
>
> Because I don't even see why it would call __copy_user() in the first
> place. This is 'finit_module()' that loads the module data from a
> file, not user space.
>
> So I guess it must be the strndup_user() in
>
> mod->args = strndup_user(uargs, ~0UL >> 1);
>
> but that doesn't look like it should even care about any module
> layout. Plus I would have expected to see strndup_user() in the call
> trace, but whatever.
>
> End result: that ia64 trace is very hard to read, and _maybe_ running
> it through the decode script might give more information about what it
> is that triggers...

Ok, I put the decoded console messages on [2].

[2]: https://pastebin.com/dLYMijfS

Cheers,
Frank

[Linus Torvalds]

On Sat, May 27, 2023 at 11:41 AM Frank Scheiner <frank.s...@web.de> wrote:
>
> Ok, I put the decoded console messages on [2].
>
> [2]: https://pastebin.com/dLYMijfS

Ugh. Apparently ia64 decoding isn't great. But at least it gives
multiple line numbers:

load_module (kernel/module/main.c:2291 kernel/module/main.c:2412
kernel/module/main.c:2868)

except your kernel obviously has those test-patches, so I still don't
know exactly where they are.

But it looks like it is in move_module(). Strange. I don't know how it
gets to "__copy_user" from there...

[ Looks at the ia64 code ]

Oh.

It turns out that it *says* __copy_user(), but the code is actually
shared with the regular memcpy() function, which does

GLOBAL_ENTRY(memcpy)
and r28=0x7,in0
and r29=0x7,in1
mov f6=f0
mov retval=in0
br.cond.sptk .common_code
;;

where that ".common_code" label is - surprise surprise - the common
copy code, and so when the oops reports that the problem happened in
__copy_user(), it actually is in this case just a normal memcpy.

Ok, so it's probably the

memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size);

in move_module() that takes a fault. And looking at the registers,
the destination is in r17/r18, and your dump has

unable to handle kernel paging request at virtual address 1000000000000000
...
r17 : 0fffffffffffffff r18 : 1000000000000000

so it's almost certainly that 'dest' that is bad.

Which I guess shouldn't surprise anybody.

But that's where my knowledge of ia64 and the new module loader layout ends.

Linus

[Frank Scheiner]

Hi,

On 27.05.23 21:34, Linus Torvalds wrote:
> On Sat, May 27, 2023 at 11:41 AM Frank Scheiner <frank.s...@web.de> wrote:
>>
>> Ok, I put the decoded console messages on [2].
>>
>> [2]: https://pastebin.com/dLYMijfS
>
> Ugh. Apparently ia64 decoding isn't great. But at least it gives
> multiple line numbers:
>
> load_module (kernel/module/main.c:2291 kernel/module/main.c:2412
> kernel/module/main.c:2868)
>
> except your kernel obviously has those test-patches, so I still don't
> know exactly where they are.

Erm, I see. I did recreate a vanilla v6.4-rc3 and ran that, decoded
result is on [1] - not sure if it makes it a little better.

[1]: https://pastebin.com/z5XzEnhq

I did also try to build and run a SP kernel to maybe get a better
picture in the traces, but that seems to require FLATMEM, which seems to
not work on that machine or due to the way it is configured (and yeah,
it was also the wrong commit I used for it and it was patched...):

```
[ 0.000000] Linux version
6.4.0-rc3-933174ae28ba72ab8de5b35cb7c98fc211235096-patch3_sp

(root@x4270) (ia64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39)

#1 Sat May 27 21:28:44 CEST 2023
[...]
[ 0.000000] ACPI: SSDT 0x000000003FE35BA8 00013C (v01 HP rx2620
00000006 INTL 20050309)
[ 0.000000] ACPI: Local APIC address (____ptrval____)
[ 0.000000] 1 CPUs available, 1 CPUs total
[...]
[ 0.000000] Kernel panic - not syncing: Cannot use FLATMEM with
246784MB hole
[ 0.000000] Please switch over to SPARSEMEM
[ 0.000000] ---[ end Kernel panic - not syncing: Cannot use FLATMEM
with 246784MB hole
[ 0.000000] Please switch over to SPARSEMEM ]---
```

Thanks for your help and going as far as you could, that's greatly
appreciated. Running that stuff is surely easier than debugging it. :-)

Cheers,
Frank

[Song Liu]

Yeah, it appears we are writing to mod_mem[MOD_INVALID].

>From the log, the following sections are assigned to MOD_INVALID:

[ 4.009109] __layout_sections: section .got (sh_flags 10000002)
matched to MOD_INVALID
[ 4.009109] __layout_sections: section .sdata (sh_flags 10000003)
matched to MOD_INVALID
[ 4.009109] __layout_sections: section .sbss (sh_flags 10000003)
matched to MOD_INVALID

AFAICT, .got should go to rodata, while .sdata and .sbss should go
to (rw)data. However, reading the code before the module_memory
change, I think they were all copied to (rw)data, which is not ideal but
most likely OK.

To match the behavior before the module_memory change, I think
we need something like the following.

Frank, could you please give it a try?

Thanks,
Song

diff --git i/kernel/module/main.c w/kernel/module/main.c

index 0f9183f1ca9f..e4e723e1eb21 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1514,14 +1514,14 @@ static void __layout_sections(struct module

*mod, struct load_info *info, bool i

MOD_RODATA,
MOD_RO_AFTER_INIT,
MOD_DATA,
- MOD_INVALID, /* This is needed to match the masks array */
+ MOD_DATA,
};
static const int init_m_to_mem_type[] = {
MOD_INIT_TEXT,
MOD_INIT_RODATA,
MOD_INVALID,
MOD_INIT_DATA,
- MOD_INVALID, /* This is needed to match the masks array */
+ MOD_INIT_DATA,
};

for (m = 0; m < ARRAY_SIZE(masks); ++m) {

[Frank Scheiner]

Hi Song, Linus,

Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot
regression for me on the rx2620:

```
ELILO v3.16 for EFI/IA-64
..
Uncompressing Linux... done
Loading file AC100221.initrd.img...done
[ 0.000000] Linux version

6.4.0-rc3-44c026a73be8038f03dbdeef028b642880cf1511-patch4 (root@x4270)
(ia64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Sun May
28 09:08:44 CEST 2023

[ 0.000000] efi: EFI v1.1 by HP
[ 0.000000] efi: SALsystab=0x3ee7a000 ACPI 2.0=0x3fe2a000
ESI=0x3ee7b000 SMBIOS=0x3ee7c000 HCDP=0x3fe28000
[ 0.000000] PCDP: v3 at 0x3fe28000
[ 0.000000] earlycon: uart8250 at MMIO 0x00000000f4050000 (options
'9600n8')
[ 0.000000] printk: bootconsole [uart8250] enabled
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x000000003FE2A000 000028 (v02 HP )
[ 0.000000] ACPI: XSDT 0x000000003FE2A02C 0000CC (v01 HP rx2620
00000000 HP 00000000)
[...]

[ 3.810346] Run /init as init process

Loading, please wait...
Starting systemd-udevd version 252.6-1

[ 3.985378] e1000: Intel(R) PRO/1000 Network Driver
[ 3.989378] e1000: Copyright (c) 1999-2006 Intel Corporation.
[ 3.993375] GSI 29 (level, low) -> CPU 7 (0x0700) vector 53
[ 4.030382] ACPI: bus type USB registered
[ 4.030382] usbcore: registered new interface driver usbfs
[ 4.034382] usbcore: registered new interface driver hub
[ 4.034382] usbcore: registered new device driver usb
[ 4.040609] GSI 18 (level, low) -> CPU 0 (0x0000) vector 54
[ 4.040621] ehci-pci 0000:00:01.2: EHCI Host Controller
[...]
[ OK ] Finished systemd-update-ut… - Record Runlevel Change in UTMP.
[ 14.568606] ioc1: LSI53C1030 C0: Capabilities={Initiator,Target}

Debian GNU/Linux 12 rx2620 -

rx2620 login:
```

Great! I'll give it a try on my rx2800-i2, too, but assume it wil work
there, too.

Cheers,
Frank

[Frank Scheiner]

Hi again,

On 28.05.23 09:30, Frank Scheiner wrote:
> [...]

> Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot
> regression for me on the rx2620:
>

> [...]

>
> Great! I'll give it a try on my rx2800-i2, too, but assume it wil work
> there, too.

Indeed, -patch4 also makes it work on the rx2800-i2:

```
ELILO v3.16 for EFI/IA-64
..
Uncompressing Linux... done

Loading file AC10027B.initrd.img...done

[ 0.000000] Linux version
6.4.0-rc3-44c026a73be8038f03dbdeef028b642880cf1511-patch4 (root@x4270)
(ia64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Sun May
28 09:08:44 CEST 2023

[ 0.000000] efi: EFI v2.1 by HP
[ 0.000000] efi: SALsystab=0xdfdd63a18 ESI=0xdfdd63f18 ACPI
2.0=0x3d3c4014 HCDP=0xdffff8798 SMBIOS=0x3d368000
[ 0.000000] PCDP: v3 at 0xdffff8798
[ 0.000000] earlycon: uart8250 at I/O port 0x4000 (options '115200n8')

[ 0.000000] printk: bootconsole [uart8250] enabled
[ 0.000000] ACPI: Early table checksum verification disabled

[ 0.000000] ACPI: RSDP 0x000000003D3C4014 000024 (v02 HP )
[ 0.000000] ACPI: XSDT 0x000000003D3C4580 000124 (v01 HP RX2800-2
00000001 01000013)
[...]
[ 36.649531] Run /init as init process

Loading, please wait...
Starting systemd-udevd version 252.6-1

[ 36.865635] pps_core: LinuxPPS API ver. 1 registered
[ 36.869321] pps_core: Software ver. 5.3.6 - Copyright 2005-2007
Rodolfo Giometti <giom...@linux.it>
[ 36.885025] PTP clock support registered
[ 36.910852] ACPI: bus type USB registered
[ 36.918198] usbcore: registered new interface driver usbfs
[ 36.924762] usbcore: registered new interface driver hub
[ 36.922133] igb: Intel(R) Gigabit Ethernet Network Driver
[ 36.931386] usbcore: registered new device driver usb
[ 36.938149] igb: Copyright (c) 2007-2014 Intel Corporation.
[...]
[ OK ] Finished apt-daily-upgrade… apt upgrade and clean activities.

Debian GNU/Linux 12 rx2800-i2 -

rx2800-i2 login:
```

I'll try to test this on other machines (rx4640 w/Madison, rx2660
w/Montecito/Montvale, rx6600 w/Montvale) as well, starting on Tuesday if
possible.

****

There is an issue - only for the rx2800-i2 - seemingly related to it's
PCIe NIC(s) and MSIs, but this is already there in 6.3.x (see first part
of [1]) and **not related to the problem of this thread (AFAICT)** and -
more important - doesn't affect operation: The machine is working
diskless using one of those interfaces so it can't be that bad. I'll
look into bisecting that issue as well.

[1]: https://lists.debian.org/debian-ia64/2023/05/msg00010.html

Cheers,
Frank

[John Paul Adrian Glaubitz]

Hi!

Sounds good. Would be great to get this into 6.4 before release.

Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer
`. `' Physicist
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

[Song Liu]

On Sun, May 28, 2023 at 1:10 AM Frank Scheiner <frank.s...@web.de> wrote:
>
> Hi again,
>
> On 28.05.23 09:30, Frank Scheiner wrote:
> > [...]
> > Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot
> > regression for me on the rx2620:
> >
> > [...]
> >
> > Great! I'll give it a try on my rx2800-i2, too, but assume it wil work
> > there, too.
>
> Indeed, -patch4 also makes it work on the rx2800-i2:

Thanks for running the test!

I will send the official patch.

Thanks,
Song

[Konstantin Ryabitsev]

On Sat, May 27, 2023 at 10:08:24AM -0700, Linus Torvalds wrote:
> This does make it clear just how great a mailing list archive lore is.
> Konstantin, is there any particular reason why
> linux...@vger.kernel.org isn't in lore? Is it just a rational hatred
> of all things itanium?

We only add things to lore when someone asks, and nobody's asked. :) I guess
I'll consider this an ask and put it on the radar.

Regards,
-K

[Linus Torvalds]

On Tue, May 30, 2023 at 4:21 PM Konstantin Ryabitsev
<konst...@linuxfoundation.org> wrote:
>
> We only add things to lore when someone asks, and nobody's asked. :) I guess
> I'll consider this an ask and put it on the radar.

Thanks. It would probably be good to see if there are any other
vger.kernel.org lists with any appreciable traffic that aren't on
lore.

Linus

[Konstantin Ryabitsev]

Yes, that will auto-fix itself as we continue to migrate things over to
subspace ("new vger").

-K

[Frank Scheiner]

Hi Linus, hi Song,

On 29.05.23 00:46, Song Liu wrote:
> [...]

> Thanks for running the test!
>
> I will send the official patch.
>
> Thanks,
> Song

With the fix merged and to conclude this, I'd like to add that it was a
pleasure to work with you on this problem, although I didn't do much.

Looking forward to the next occasion - for your sake maybe on another
architecture, but can't promise... ;-)

Cheers,
Frank

[Frank Scheiner]

On 31.05.23 21:14, Luis Chamberlain wrote:

> On Wed, May 31, 2023 at 11:16 AM Frank Scheiner <frank.s...@web.de> wrote:
>> Looking forward to the next occasion - for your sake maybe on another
>> architecture, but can't promise... ;-)
>

> I think it would be prudent for Song to also ask you to test his
> future upcoming modules patches on ia64 given how hard it is to
> procure such hardware too.

Sure!

Cheers,
Frank

[Luis Chamberlain]

On Wed, May 31, 2023 at 11:16 AM Frank Scheiner <frank.s...@web.de> wrote:
> Looking forward to the next occasion - for your sake maybe on another
> architecture, but can't promise... ;-)

I think it would be prudent for Song to also ask you to test his
future upcoming modules patches on ia64 given how hard it is to
procure such hardware too.

Luis