Bug#1033301: linux: arm64 kernel size increased from 31 to 39 MB, causing u-boot-rpi to fail

[Aurelien Jarno]

Source: linux
Version: 6.1.15-1
Severity: important
Tags: upstream
X-Debbugs-Cc: vag...@debian.org
Control: affects -1 + u-boot-rpi

Hi,

Following the upgrade of the kernel from 6.1.12-1 to 6.1.15-1 on a
Raspberry Pi 3 Model B Plus, u-boot (from the u-boot-rpi package) failed
to boot with:

| 40175552 bytes read in 1695 ms (23 MiB/s)
| 43794863 bytes read in 1817 ms (23 MiB/s)
| Moving Image from 0x80000 to 0x200000, end=2990000
| ERROR: RD image overlaps OS image (OS=0x200000..0x2990000)

I tracked the issue to a significant increase of the kernel size between
version 6.1.12-1 and 6.15-1:

| 31492 /boot/vmlinuz-6.1.0-5-arm64
| 39236 /boot/vmlinuz-6.1.0-6-arm64

This is more than the 36MB that is allowed by u-boot with the default
load addresses. A workaround is to shift the load addresses at the
u-boot level as in the attached patch.

I have tracked issue on the upstream kernel side to the following commit
on the stable tree:

| commit 3e3e4d234d46e48480a7c7c35399fa811182e8ef
| Author: Masahiro Yamada <masa...@kernel.org>
| Date: Thu Oct 13 08:35:00 2022 +0900
|
| arm64: remove special treatment for the link order of head.o
|
| commit 994b7ac1697b4581b7726d2ac64321e3c840229b upstream.
|
| In the previous discussion (see the Link tag), Ard pointed out that
| arm/arm64/kernel/head.o does not need any special treatment - the only
| piece that must appear right at the start of the binary image is the
| image header which is emitted into .head.text.
|
| The linker script does the right thing to do. The build system does
| not need to manipulate the link order of head.o.
|
| Link: https://lore.kernel.org/lkml/CAMj1kXH77Ja8bSsq2Qj8Ck9iSZKw=1F8Uy-uAWGVDm4-CG=E...@mail.gmail.com/
| Suggested-by: Ard Biesheuvel <ar...@kernel.org>
| Signed-off-by: Masahiro Yamada <masa...@kernel.org>
| Reviewed-by: Nicolas Schier <nic...@fjasle.eu>
| Link: https://lore.kernel.org/r/20221012233500.1...@kernel.org
| Signed-off-by: Will Deacon <wi...@kernel.org>
| Signed-off-by: Tom Saeger <tom.s...@oracle.com>
| Signed-off-by: Greg Kroah-Hartman <gre...@linuxfoundation.org>

The problem is still reproducible on Linus' master.

I am reporting the bug to the linux package as I believed there is no
real reason for such an increase in the kernel size. In case I missed
something and this is actually wanted, the bug can be reassigned to the
u-boot package.

Regards
Aurelien

[Salvatore Bonaccorso]

Hi Aurelien,

Thanks for tracking this down. I would like to loop in Masahiro and
upstream to see if something can/should be done on upstream side.

Masahiro, in short, upstream change 994b7ac1697b ("arm64: remove
special treatment for the link order of head.o") (which got backported
as well to 6.1.14) caused the vmlinuz size to icrease significantly,
causing boot failures on Raspberry Pi 3 Model B Plus with u-boot
parameters previously working. Full quoting the Debian report below

> --- u-boot-2023.01+dfsg.orig/include/configs/rpi.h
> +++ u-boot-2023.01+dfsg/include/configs/rpi.h
> @@ -95,32 +95,32 @@
> * text_offset bytes (specified in the header of the Image) into a 2MB
> * boundary. The 'booti' command relocates the image if necessary. Linux uses
> * a default text_offset of 0x80000. In summary, loading at 0x80000
> - * satisfies all these constraints and reserving memory up to 0x02400000
> - * permits fairly large (roughly 36M) kernels.
> + * satisfies all these constraints and reserving memory up to 0x02a00000
> + * permits fairly large (roughly 42M) kernels.
> *
> * scriptaddr and pxefile_addr_r can be pretty much anywhere that doesn't
> * conflict with something else. Reserving 1M for each of them at
> - * 0x02400000-0x02500000 and 0x02500000-0x02600000 should be plenty.
> + * 0x02a00000-0x02b00000 and 0x02c00000-0x02d00000 should be plenty.
> *
> * On ARM, both the DTB and any possible initrd must be loaded such that they
> * fit inside the lowmem mapping in Linux. In practice, this usually means not
> * more than ~700M away from the start of the kernel image but this number can
> * be larger OR smaller depending on e.g. the 'vmalloc=xxxM' command line
> * parameter given to the kernel. So reserving memory from low to high
> - * satisfies this constraint again. Reserving 1M at 0x02600000-0x02700000 for
> - * the DTB leaves rest of the free RAM to the initrd starting at 0x02700000.
> + * satisfies this constraint again. Reserving 1M at 0x02c00000-0x02d00000 for
> + * the DTB leaves rest of the free RAM to the initrd starting at 0x02d00000.
> * Even with the smallest possible CPU-GPU memory split of the CPU getting
> - * only 64M, the remaining 25M starting at 0x02700000 should allow quite
> + * only 64M, the remaining 19M starting at 0x02d00000 should allow quite
> * large initrds before they start colliding with U-Boot.
> */
> #define ENV_MEM_LAYOUT_SETTINGS \
> "fdt_high=" FDT_HIGH "\0" \
> "initrd_high=" INITRD_HIGH "\0" \
> "kernel_addr_r=0x00080000\0" \
> - "scriptaddr=0x02400000\0" \
> - "pxefile_addr_r=0x02500000\0" \
> - "fdt_addr_r=0x02600000\0" \
> - "ramdisk_addr_r=0x02700000\0"
> + "scriptaddr=0x02a00000\0" \
> + "pxefile_addr_r=0x02b00000\0" \
> + "fdt_addr_r=0x02c00000\0" \
> + "ramdisk_addr_r=0x02d00000\0"
>
> #if CONFIG_IS_ENABLED(CMD_MMC)
> #define BOOT_TARGET_MMC(func) \

Any ideas?

Regards,
Salvatore

[Vagrant Cascadian]

Adding u-boot maintainers for rpi (Matthias Brugger, Peter Robinson)
platforms and u-boot list to CC.

On 2023-03-22, Salvatore Bonaccorso wrote:
> Thanks for tracking this down. I would like to loop in Masahiro and
> upstream to see if something can/should be done on upstream side.
>
> Masahiro, in short, upstream change 994b7ac1697b ("arm64: remove
> special treatment for the link order of head.o") (which got backported
> as well to 6.1.14) caused the vmlinuz size to icrease significantly,
> causing boot failures on Raspberry Pi 3 Model B Plus with u-boot
> parameters previously working. Full quoting the Debian report below

In general it would be nice to not have the kernel grow nearly 25% in
size from a single commit; was that an expected outcome from the above
upstream change? Was the "special treament" originally done to keep the
kernel size down?

As for u-boot, It seems u-boot might need to update the various load
addresses for the kernel, device tree and ramdisk at some point
regardless of weather this particular issue gets fixed in the kernel, as
the kernel will likely continue to grow a bit over time...

Aurelien Jarno included a patch referenced below which bumps the
tolerances in u-boot from 36MB to 42MB.

Bumping from 36MB to 42MB seems address the issue at hand, although
would it make sense to bump even higher... or is that starting to hit
other limitations?


Thanks everyone!


live well,
vagrant

[Aurelien Jarno]

On 2023-03-24 13:58, Vagrant Cascadian wrote:
> Adding u-boot maintainers for rpi (Matthias Brugger, Peter Robinson)
> platforms and u-boot list to CC.
>
> On 2023-03-22, Salvatore Bonaccorso wrote:
> > Thanks for tracking this down. I would like to loop in Masahiro and
> > upstream to see if something can/should be done on upstream side.
> >
> > Masahiro, in short, upstream change 994b7ac1697b ("arm64: remove
> > special treatment for the link order of head.o") (which got backported
> > as well to 6.1.14) caused the vmlinuz size to icrease significantly,
> > causing boot failures on Raspberry Pi 3 Model B Plus with u-boot
> > parameters previously working. Full quoting the Debian report below
>
> In general it would be nice to not have the kernel grow nearly 25% in
> size from a single commit; was that an expected outcome from the above
> upstream change? Was the "special treament" originally done to keep the
> kernel size down?

This is currently being tracked [1], but the issue seems to be linked to
the version of the tools used in Debian, and the fact that we build our
kernels with BTF support, so the issue is likely to be difficult to find.

[1] https://lore.kernel.org/linux-arm-kernel/ZBovCrMX...@aurel32.net/

> As for u-boot, It seems u-boot might need to update the various load
> addresses for the kernel, device tree and ramdisk at some point
> regardless of weather this particular issue gets fixed in the kernel, as
> the kernel will likely continue to grow a bit over time...

Yes that's probably a sane thing to do, at least in Debian.

From the comment above the code, I understand that the main limitation
is when allocating most of the memory to the GPU, leaving only 64MB to
the CPU.
It seems possible to do that on many RPi:
- On 256MB RPi, the maximum gpu_mem value is 192
- On 512MB RPi, the maximum gpu_mem value is 448
- On >= 1GB RPi, the maximum gpu_mem value is 944

Aurelien

--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aure...@aurel32.net http://www.aurel32.net

[Aurelien Jarno]

control: reassign -1 pahole/1.24-4
control: retitle -1 pahole: BTF deduplication issues causing arm64 kernel size increase
control: tag -1 + fixed-upstream
control: tag -1 + patch
control: affects -1 u-boot-rpi src:linux

Hi,

This has been discussed upstream, and it appears that the issue is not
reproducible with pahole 1.25. Looking at the part that needs to be
backported to the Debian package, I have found that the issue is caused
by the following patch backported in version 1.24-4:

02-encode-DW_TAG_unspecified_type-returning-routines-as-void.patch

This patch has an issue, and memory is not initialized after allocation,
so garbage values are used instead. A one-liner fix has been committed
upstream not so long after the initial patch:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=b72f5188856df0abf45e1a707856bb4e4e86153c

I am therefore reassigning the bug to pahole where the bug should be
fixed. Even if Bookworm is now fully frozen, I personally believe this
bug should still be fixed before the release. That said, this has to be
discussed with the release team, and as pahole is a key package you need
a pre-approval *before* the upload to sid.

Regards

Aurelien

--
Aurelien Jarno GPG: 4096R/1DDD8C9B

aure...@aurel32.net http://aurel32.net

[Salvatore Bonaccorso]

Hi Aurelien,

Thanks for tracking this down!

Domenico, I would say, it would eben be good to have this in in the
archive for bookworm before the next (last?) linux upload for bookworm
ideally. This is not yet planned but, the last one should probably be
latest in the week of 15th May.

Regards,
Salvatore

[Domenico Andreoli]

Hi,

Apologies for not having sent a VAC message but I'm unable to handle this in a timely manner.

Please proceed with a NMU.

Thanks,
Dom