Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems

16 views
Skip to first unread message

Salvatore Bonaccorso

unread,
Sep 9, 2023, 4:20:05 AM9/9/23
to
Hi all,

We have problem with the image size of armel builds in bookworm. There
is a pending bookworm-security linux update pending which is currently
blocked due to armel FTBFS due to the image size increase:

https://people.debian.org/~carnil/buildd-logs/linux/linux_6.1.52-1_armel-2023-09-07T08:53:41Z.gz

debian/bin/buildcheck.py debian/build/build_armel_none_marvell armel none marvell
Can't read ABI reference. ABI not checked!
Image size 2753652/2729712, using 100.88%. Too large. Refusing to continue.
make[2]: *** [debian/rules.real:169: debian/stamps/build_armel_none_marvell] Error 1
make[2]: Leaving directory '/<<PKGBUILDDIR>>'
make[1]: *** [debian/rules.gen:1615: build-arch_armel_none_marvell_real_image] Error 2
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
make: *** [debian/rules:39: build-arch] Error 2
dpkg-buildpackage: error: debian/rules binary-arch subprocess returned exit status 2

In fact we are already too narrow to 100% in any case, but there was a
bump between 6.1.41 and 6.1.42 upstream AFAICS:

6.1.52-1 Image size 2751596/2729712, using 100.80%. Too large. Refusing to continue.
6.1.51-1 Image size 2752212/2729712, using 100.82%. Too large. Refusing to continue.
6.1.47-1 Image size 2752676/2729712, using 100.84%. Too large. Refusing to continue.
6.1.45-1 Image size 2751292/2729712, using 100.79%. Too large. Refusing to continue.
6.1.43-1 Image size 2751348/2729712, using 100.79%. Too large. Refusing to continue.
6.1.42-1 Image size 2752924/2729712, using 100.85%. Too large. Refusing to continue.
6.1.41-1 Image size 2701348/2729712, using 98.96%. Image fits. Continuing.
6.1.40-1 Image size 2703956/2729712, using 99.06%. Under 1% space in UNRELEASED. Continuing.
6.1.38-1 Image size 2703076/2729712, using 99.02%. Under 1% space in bookworm. Continuing.

I doupt anybody is sensibly using armel nowdays under bookworm, so my proposed
course of action for unblock the bookworm-security update is:

Either

- ignore the image size and implicitly drop support for devices which would break
due to size constraints, the current upper limit is adjusted for the following:

# Buffalo Linkstation LS-WSXL/WXL/WVL (from stock kernel): 2729776 - 64 = 2729712

or:

- Relese the DSA without armel builds. This is not optimal and for the point release
we need to have to have all builds, but this gives people who still are interested
in this architecture to step up and propose a fix for the problem, otherwise then
disable the image size check, and then effectively dropping some support.

Attached is the result of bloat-o-meter script between 6.1.41 and 6.1.42.

I might put me in a bad spot, but should have been support for armel been
dropped earlier and should we do it for trixie following the same done for
mipsel?

Note that the last time the problem arised already earlier in
experimental and Ben workarounded it there with
https://salsa.debian.org/kernel-team/linux/-/commit/9dfe6d33a4fd220394228b30cbbfdb3b444d36ec
We probably can do that as well here. 60443c88f3a8 ("kallsyms: Improve
the performance of kallsyms_lookup_name()") was in fact backported to
6.1.42. So this is next I would try and disable MPTCP and
FUNCTION_TRACER. But the problem with armel will remain.

armel people, can you have ideally look at it ASAP on the comments
please, I would not like to delay the DSA for linux on
bookworm-security too much.

Thanks for having a look,

Regards,
Salvatore
vmlinux.6.1.41-1..vmlinux.6.1.42-1.txt
vmlinux.6.1.41-1..vmlinux.6.1.42-1.ungrouped.txt

Adrian Bunk

unread,
Sep 9, 2023, 5:10:03 AM9/9/23
to
On Sat, Sep 09, 2023 at 10:15:59AM +0200, Salvatore Bonaccorso wrote:
>...
> - Relese the DSA without armel builds. This is not optimal and for the point release
> we need to have to have all builds, but this gives people who still are interested
> in this architecture to step up and propose a fix for the problem, otherwise then
> disable the image size check, and then effectively dropping some support.
>...
> armel people, can you have ideally look at it ASAP on the comments
> please, I would not like to delay the DSA for linux on
> bookworm-security too much.

Releasing this DSA without armel and sorting out the issue for the point
release sounds like the best option to me.

> Thanks for having a look,
>
> Regards,
> Salvatore

cu
Adrian

Paul Gevers

unread,
Sep 9, 2023, 5:20:04 AM9/9/23
to
Hi Salvatore,

On 09-09-2023 10:15, Salvatore Bonaccorso wrote:
> but should have been support for armel been
> dropped earlier and should we do it for trixie

The kernel for armel went over some hardware limits before (I was
affected with my NAS, where I couldn't upgrade the kernel to bullseye as
documented in the release notes [1]). Is the current situation reaching
the limit for all armel devices, or "just" for some and are the others
probably fine for some years to come?

If we're now reaching the final limit and if it was foreseeable that we
would reach that limit, then yes it would have made sense to drop armel
*before* the bookworm release, but alas. If the kernel team can't
support the kernel on armel, than armel shouldn't be a release
architecture for trixie. If it's only some devices, than we "just" need
to communicate that clearly.

I don't have a clear advice for the current situation in security and
the next point release, let's hope you can stretch the situation a bit
longer. I recall that the kernel package has safety checks in place and
refuses to *try* to install the kernel if it doesn't fit on the
hardware. That means that you don't cripple the hardware of affected
people, but "merely" can't give them security support? I guess it would
be possible (as long as support lasts; no LTS support) for effected
systems to run the security supported bullseye kernel.

Paul

[1]
https://www.debian.org/releases/bullseye/armel/release-notes/ch-information.en.html#no-longer-supported-hardware
OpenPGP_signature.asc

Bastian Blank

unread,
Sep 9, 2023, 6:10:04 AM9/9/23
to
Hi

On Sat, Sep 09, 2023 at 11:13:56AM +0200, Paul Gevers wrote:
> If we're now reaching the final limit and if it was foreseeable that we
> would reach that limit, then yes it would have made sense to drop armel
> *before* the bookworm release, but alas. If the kernel team can't support
> the kernel on armel, than armel shouldn't be a release architecture for
> trixie. If it's only some devices, than we "just" need to communicate that
> clearly.

We have two armel kernel currently:
- "marvell", for some CPU from Marvell, and
- "rpi", for Raspberry Pi 1 and related devices.

The first one is the one with included size limitations, because those
load the kernel from a pre-defined flash partition, whose size can't be
easily changed by the user. This one is now overflowing for the second
to last documented one in the kernel package config.

The second one is for the original Raspberry Pi 1 type. There we don't
have any size limits, as the kernel is loaded from a file system.
However those systems contain a ARMv6 CPU. So our armel port is only
partially usable anyway, as is is built for ARMv4. There exists with
Raspbian a better suited forked distribution with ARMv6 as target.

So yes there is a small number of devices we can still support with the
armel port, but where we are a bad choice.

Everything newer is ARMv7, supported by the armhf port, or ARMv8,
supported by the arm64 port.

Latest popcon for stable is:

linux-image-marvell: 31
linux-image-rpi: 7

Debian itself does not have any armel hardware. Everything is done on
armhf or arm64. Sadly the armhf supporting systems are already in the
progress of drying up. Even some ARMv8 vendors do not longer include
32bit support.

Bastian

--
Each kiss is as the first.
-- Miramanee, Kirk's wife, "The Paradise Syndrome",
stardate 4842.6

Salvatore Bonaccorso

unread,
Sep 9, 2023, 6:50:03 AM9/9/23
to
Hi,
FWIW, following Ben's aproach for unstable, here is my proposed change
for bookworm in the near-term:

https://salsa.debian.org/kernel-team/linux/-/merge_requests/844

I have verified by cross-building that the image size goes down to

Image size 2644124/2729712, using 96.86%. Image fits. Continuing.

which would be sufficient so far.

So we can at least include the above for the point release and
releasing the DSA earlier without the armel builds.

Thank you!

Regards,
Salvatore

Andrew M.A. Cater

unread,
Sep 9, 2023, 7:50:04 AM9/9/23
to
On Sat, Sep 09, 2023 at 11:42:45AM +0200, Bastian Blank wrote:
> Hi
>
> On Sat, Sep 09, 2023 at 11:13:56AM +0200, Paul Gevers wrote:
> > If we're now reaching the final limit and if it was foreseeable that we
> > would reach that limit, then yes it would have made sense to drop armel
> > *before* the bookworm release, but alas. If the kernel team can't support
> > the kernel on armel, than armel shouldn't be a release architecture for
> > trixie. If it's only some devices, than we "just" need to communicate that
> > clearly.
>
> We have two armel kernel currently:
> - "rpi", for Raspberry Pi 1 and related devices.
>
> The second one is for the original Raspberry Pi 1 type. There we don't
> have any size limits, as the kernel is loaded from a file system.
> However those systems contain a ARMv6 CPU. So our armel port is only
> partially usable anyway, as is is built for ARMv4. There exists with
> Raspbian a better suited forked distribution with ARMv6 as target.
>

No - Raspbian contains commercial software now by default. We have to use
Raspberry Pi firmware but Raspberry pi is *not* purely ARM v6 it's ARM v6 with
hardware floating point - and therefore incompatible with Debian and every
other ARM v6 version. Although it is less than optimal, it is still perfectly
supportable by all the packages in armel.

> So yes there is a small number of devices we can still support with the
> armel port, but where we are a bad choice.
>

See above: note that the Raspberry Pi Zero (original version) is still very
much in production and is 32 bit and ARM v6 hardware floating point. It is
still a prime target for armel - and the number of devices produced is not
small.

> Everything newer is ARMv7, supported by the armhf port, or ARMv8,
> supported by the arm64 port.
>
> Latest popcon for stable is:
>
> linux-image-marvell: 31
> linux-image-rpi: 7
>
> Debian itself does not have any armel hardware. Everything is done on
> armhf or arm64. Sadly the armhf supporting systems are already in the
> progress of drying up. Even some ARMv8 vendors do not longer include
> 32bit support.
>

This is, unfortunately, the case. I'm not sure where our sd card images
are built but the Raspberry Pi iamges are built from Gunnar Wolf's scripts
primarily for everything prior to RPi4.

All best, as ever,

Andy

[amac...@debian.org]

Ben Hutchings

unread,
Sep 10, 2023, 11:20:03 AM9/10/23
to
On Sat, 2023-09-09 at 11:13 +0200, Paul Gevers wrote:
> Hi Salvatore,
>
> On 09-09-2023 10:15, Salvatore Bonaccorso wrote:
> > but should have been support for armel been
> > dropped earlier and should we do it for trixie
>
> The kernel for armel went over some hardware limits before (I was
> affected with my NAS, where I couldn't upgrade the kernel to bullseye as
> documented in the release notes [1]). Is the current situation reaching
> the limit for all armel devices, or "just" for some and are the others
> probably fine for some years to come?

The issue exists with many devices supported by the "marvell" flavour.
We also have an "rpi" flavour for armel that supports Raspberry Pi
models with Arm v6 CPUs, and that doesn't have any size limits that
we've needed to worry about yet.

> If we're now reaching the final limit and if it was foreseeable that we
> would reach that limit, then yes it would have made sense to drop armel
> *before* the bookworm release, but alas. If the kernel team can't
> support the kernel on armel, than armel shouldn't be a release
> architecture for trixie. If it's only some devices, than we "just" need
> to communicate that clearly.
[...]

I would be quite happy to drop the "marvell" flavour for trixie. The
size limits are a recurring problem, and the supported SoCs and devices
seem to have been out of production for a long time. In contrast, the
Raspberry Pi models with v6 CPUs are still in production.

(I do have my doubts as to whether it will be possible to continue
supporting a useful user-space for armel, but I certainly can't speak
authoritatively about that.)

Ben.

--
Ben Hutchings - Debian developer, member of kernel, installer and LTS
teams
signature.asc

Gianluca Renzi

unread,
Sep 11, 2023, 3:10:04 AM9/11/23
to
On 9/10/23 04:55, Paul Wise wrote:
> On Sat, 2023-09-09 at 11:42 +0200, Bastian Blank wrote:
>
>> The first one is the one with included size limitations, because those
>> load the kernel from a pre-defined flash partition, whose size can't be
>> easily changed by the user.  This one is now overflowing for the second
>> to last documented one in the kernel package config.
> Seems like this would be solvable by writing a bootloader to the flash
> partition that would be able to load Linux from a normal filesystem?
>
I think any modern (at least more than 10 years of lifespan) bootloader
(u-boot, redboot, barebox, etc.,...) can read from any supported
filesystem (at least ext2, ext3 and ext4 and maybe nand-related
filesystems like yaffs, yaffs2, ubifs), so having the {z|u}Image kernel
stored on those filesystems should be not an issue...

The biggest problem will be the update procedures for those hardware.
All scripts must be changed to reflect the new partition scheme. This is
*not* an easy fix.

Just my $0.02

Regards

Gianluca

--
Eurek s.r.l. |
Electronic Engineering | http://www.eurek.it
via Celletta 8/B, 40026 Imola, Italy | Phone: +39-(0)542-609120
p.iva 00690621206 - c.f. 04020030377 | Fax: +39-(0)542-609212

bodhi

unread,
Sep 11, 2023, 5:30:05 PM9/11/23
to
On Monday, September 11, 2023 at 12:10:04 AM UTC-7, Gianluca Renzi wrote:
> On 9/10/23 04:55, Paul Wise wrote:
> > On Sat, 2023-09-09 at 11:42 +0200, Bastian Blank wrote:
> >
> >> The first one is the one with included size limitations, because those
> >> load the kernel from a pre-defined flash partition, whose size can't be
> >> easily changed by the user. This one is now overflowing for the second
> >> to last documented one in the kernel package config.
> > Seems like this would be solvable by writing a bootloader to the flash
> > partition that would be able to load Linux from a normal filesystem?
> >
> I think any modern (at least more than 10 years of lifespan) bootloader
> (u-boot, redboot, barebox, etc.,...) can read from any supported
> filesystem (at least ext2, ext3 and ext4 and maybe nand-related
> filesystems like yaffs, yaffs2, ubifs), so having the {z|u}Image kernel
> stored on those filesystems should be not an issue...
>
> The biggest problem will be the update procedures for those hardware.
> All scripts must be changed to reflect the new partition scheme. This is
> *not* an easy fix.

I agreed with Paul and also Gianluca. I'm the u-boot maintainer for many Marvell Kirkwood SoC boards. All these boards are actively maintained and up to modern standard, i.e. can boot with file systems on SATA, USB, MMC... We can ignore what's currently in flash partitions and boot with kernels on disk rootfs.

What can I do to help in upgrading the update procedure to boot Debian 12 on Marvell Kirkwood SoCs? can somebody give me some pointer where to start looking for these scripts?

All the best,
Tony

Philippe Clérié

unread,
Sep 11, 2023, 5:50:04 PM9/11/23
to
For what it's worth, if it helps I can pitch in and test.

I have a QNAP HS-210 and a TS-212P that I've retired since the first
time I read that Debian would drop armel arch. I know there have been
workarounds the limitations, and I keep having plans to implement at
least one. But! Good intentions and all! :-)

...
Philippe

------
The trouble with common sense is that it is so uncommon.
(Anonymous)

Domenico Andreoli

unread,
Sep 13, 2023, 1:40:04 AM9/13/23
to
Looks good to me. Thanks!

>
> Thank you!
>
> Regards,
> Salvatore
>

--
rsa4096: 3B10 0CA1 8674 ACBA B4FE FCD2 CE5B CF17 9960 DE13
ed25519: FFB4 0CC3 7F2E 091D F7DA 356E CC79 2832 ED38 CB05
signature.asc

Domenico Andreoli

unread,
Sep 13, 2023, 1:40:04 AM9/13/23
to
On Sun, Sep 10, 2023 at 10:55:49AM +0800, Paul Wise wrote:
> On Sat, 2023-09-09 at 11:42 +0200, Bastian Blank wrote:
>
> > The first one is the one with included size limitations, because those
> > load the kernel from a pre-defined flash partition, whose size can't be
> > easily changed by the user.  This one is now overflowing for the second
> > to last documented one in the kernel package config.
>
> Seems like this would be solvable by writing a bootloader to the flash
> partition that would be able to load Linux from a normal filesystem?

Yes, if the support for the given device is mainlined.

I doubt this is the case for my QNAP though, would not dare to try.
signature.asc

Domenico Andreoli

unread,
Sep 13, 2023, 2:00:03 AM9/13/23
to
On Sun, Sep 10, 2023 at 05:13:52PM +0200, Ben Hutchings wrote:
> On Sat, 2023-09-09 at 11:13 +0200, Paul Gevers wrote:
> > Hi Salvatore,
> >
> > On 09-09-2023 10:15, Salvatore Bonaccorso wrote:
> > > but should have been support for armel been
> > > dropped earlier and should we do it for trixie
> >
> > The kernel for armel went over some hardware limits before (I was
> > affected with my NAS, where I couldn't upgrade the kernel to bullseye as
> > documented in the release notes [1]). Is the current situation reaching
> > the limit for all armel devices, or "just" for some and are the others
> > probably fine for some years to come?
>
> The issue exists with many devices supported by the "marvell" flavour.
> We also have an "rpi" flavour for armel that supports Raspberry Pi
> models with Arm v6 CPUs, and that doesn't have any size limits that
> we've needed to worry about yet.

I happily delayed the issue by changing the layout of the mtd partitions
of my QNAP with a nice script [0]. It's able to handle a good number
of models and I think it's a great solution for the short/medium term.

> > If we're now reaching the final limit and if it was foreseeable that we
> > would reach that limit, then yes it would have made sense to drop armel
> > *before* the bookworm release, but alas. If the kernel team can't
> > support the kernel on armel, than armel shouldn't be a release
> > architecture for trixie. If it's only some devices, than we "just" need
> > to communicate that clearly.
> [...]
>
> I would be quite happy to drop the "marvell" flavour for trixie. The
> size limits are a recurring problem, and the supported SoCs and devices
> seem to have been out of production for a long time. In contrast, the
> Raspberry Pi models with v6 CPUs are still in production.

I would find reasonable if trixie dropped "marvell" flavor. The bell
for the armel QNAP devices has been warning for years already, anybody
caring of their data should move it somewhere else.

> (I do have my doubts as to whether it will be possible to continue
> supporting a useful user-space for armel, but I certainly can't speak
> authoritatively about that.)
>
> Ben.
>
> --
> Ben Hutchings - Debian developer, member of kernel, installer and LTS
> teams

[0] https://github.com/amouiche/qnap_mtd_resize_for_bullseye
signature.asc

Adrian Bunk

unread,
Sep 28, 2023, 9:00:04 PM9/28/23
to
On Sat, Sep 09, 2023 at 10:15:59AM +0200, Salvatore Bonaccorso wrote:
>...
> Note that the last time the problem arised already earlier in
> experimental and Ben workarounded it there with
> https://salsa.debian.org/kernel-team/linux/-/commit/9dfe6d33a4fd220394228b30cbbfdb3b444d36ec
> We probably can do that as well here. 60443c88f3a8 ("kallsyms: Improve
> the performance of kallsyms_lookup_name()") was in fact backported to
> 6.1.42. So this is next I would try and disable MPTCP and
> FUNCTION_TRACER.
>...

Yes, that looks reasonable.

Additionally, one generic cause of bloat is:
debian/changelog: * Enable UNICODE. (closes: #985689)
debian/config/config:CONFIG_UNICODE=y

That's 74 kB uncompressed, and there doesn't seem to be any
justification for not making it modular. It's not urgent since
Bens change handles the immediate problem, but worth changing
in unstable.

cu
Adrian

Fan Naibed

unread,
Oct 2, 2023, 6:30:04 AM10/2/23
to
On 2023-09-09 08:15, Salvatore Bonaccorso wrote:
> Hi all,
...
> I doupt anybody is sensibly using armel nowdays under bookworm, so my
> proposed
> course of action for unblock the bookworm-security update is:
...

FYI, An old, headless, raspberry pi (armel) with GPS and camera, on
Debian testing, was being sensibly used for long-term, slow, time-lapse
photography, where it was good enough for the intended purpose. Another
pi was slated for print server duties, but was mostly just idling
because the particular printer was not supported. Both stopped
responding over the network after upgrades some months ago, so this
decision wasn't involved; the problem was probably caused by systemd
network interface renaming. The hope was to get them going again with a
little debugging, or a re-install if necessary, as builds are still
being made[1]. It's always sad to see support have to be dropped for old
functioning hardware still sensibly serving purposes they can
accomplish. But, understandable with the size growth required to support
newer "better" hardware. Thanks for keeping them alive, and up to date,
as long as they were.


[1] https://raspi.debian.net/daily-images/

Salvatore Bonaccorso

unread,
Oct 3, 2023, 12:40:04 AM10/3/23
to
Hi Adrian,

Sorry for not replying early, busy with preparing the updates.

On Fri, Sep 29, 2023 at 03:41:15AM +0300, Adrian Bunk wrote:
> On Sat, Sep 09, 2023 at 10:15:59AM +0200, Salvatore Bonaccorso wrote:
> >...
> > Note that the last time the problem arised already earlier in
> > experimental and Ben workarounded it there with
> > https://salsa.debian.org/kernel-team/linux/-/commit/9dfe6d33a4fd220394228b30cbbfdb3b444d36ec
> > We probably can do that as well here. 60443c88f3a8 ("kallsyms: Improve
> > the performance of kallsyms_lookup_name()") was in fact backported to
> > 6.1.42. So this is next I would try and disable MPTCP and
> > FUNCTION_TRACER.
> >...
>
> Yes, that looks reasonable.

Great thanks, this landed now for the point release udpates and in
fact we have the armel builds back.

> Additionally, one generic cause of bloat is:
> debian/changelog: * Enable UNICODE. (closes: #985689)
> debian/config/config:CONFIG_UNICODE=y
>
> That's 74 kB uncompressed, and there doesn't seem to be any
> justification for not making it modular. It's not urgent since
> Bens change handles the immediate problem, but worth changing
> in unstable.

Could you fill a bug against src:linux for it, so this might be
further addressed in unstable for armel?

Regards,
Salvatore
0 new messages