consolidate linux-libc-dev headers

[Dimitri John Ledkov]

Currently in both Debian and Ubuntu we ship like close to 40 .deb packages that have linux-libc-dev (for native/multiarch, cross, ports, mipses).

Each one of them is like 1.5MB, however they actually all have the same and repeated content.

the bulk of headers are the same on all arches (and enforced via multi-arch:same), asm-generic is also the same for all of them, and then kernel-arch asm headers are unique - but only for given kernel arches (not all of the debian arch names).

Currently there are only 13 kernel archs for unique asm-arch headers, but in Debian we have multiple reuses of the same kernel archs with different userspace abi properties (arm = armel armhf, mips = 12 mips archs, powerpc = 4, x86 = 3, and so on).

It seems wasteful on disk, build-time, and derivative build time to package all of these separately. Especially since there is a chain of src:linux building native ones + linux-source as a .deb; which is then used by cross-toolchain-base{,-ports,-mips} to unpack and rebuild linux headers again, and publish. When in practice src:linux itself could build an efficient libc-dev packages for all arches as an arch:all package and Multi-Arch:foreign.

I have implemented example packaging of that as a standalone source package https://ppa.launchpadcontent.net/xnox/nonvirt/ubuntu/pool/main/l/linux-uapi/

The linux-libc-dev is native & multiarch uapi headers for all arches. The linux-libc-dev-alpha-cross is all debian arch crosses. It is implemented using hardlinks to maintain all the same paths that are currently being used by all the arch:any linux-libc-dev packages, and the linux-libc-dev-*-cross packages. In total they provide equivalent functionality as 40+ linux-libc-dev* current debs in either Debian or Ubuntu.

Is this something that the debian kernel team could consider supporting / building as either standalone source package, or out of src:linux directly? as this would reduce 40+ .deb duplication in the mirror pools of the same content; and will eliminate build-depends on linux-source (and thus built-using) from src:cross-toolchain-base* and will actually ensure that all kernel headers for native / multiarch / cross arches are updated simultaneously, without need to wait for all arch:any builds to catch up first. It also gives ability to trivially create freestanding toolchains to any of the arches without actually building a full debian port for or having a working kernel for a given port.

The obvious things missing from the packaging is to create all the breaks/replaces/provides, and update cross-toolchain-base packages to match. Also probably using symlinks rather than hard links, with linux-libc-dev-cross likely depending on the native one.

Separately for Ubuntu, due to the number of kernel built, I would likely want to upload source package that produces linux-libc-dev as a separate source package such that linux-libc-dev is actually only updated when needed, rather than on every kernel upload. As there is no need to rev linux-libc-dev as often as kernel uploads are done. But I do hope to see if this approach for linux-libc-dev can be coordinated to ensure compatible cross-toolchain-base changes.

--

okurrr,

Dimitri

[Bastian Blank]

Hi Dimitri

On Thu, Nov 16, 2023 at 06:28:13PM +0000, Dimitri John Ledkov wrote:
> I have implemented example packaging of that as a standalone source package
> https://ppa.launchpadcontent.net/xnox/nonvirt/ubuntu/pool/main/l/linux-uapi/

I actually just implemented something similar, but as part of the Debian
linux packaging. It looks a bit different, sure.

> Is this something that the debian kernel team could consider supporting /
> building as either standalone source package, or out of src:linux directly?

My first thought was actually to make bootstrapping of new architectures
easier.

> The obvious things missing from the packaging is to create all the
> breaks/replaces/provides, and update cross-toolchain-base packages to
> match. Also probably using symlinks rather than hard links, with
> linux-libc-dev-cross likely depending on the native one.

What do you want to do with linux-libc-dev-cross? /usr/$triplet is no
allowed location in Debian anyway.

> Separately for Ubuntu, due to the number of kernel built, I would likely
> want to upload source package that produces linux-libc-dev as a separate
> source package such that linux-libc-dev is actually only updated when
> needed, rather than on every kernel upload. As there is no need to rev
> linux-libc-dev as often as kernel uploads are done.

The question here is: does it provide an advantage to have it as
separate source? Less updates IMHO do not count, as they don't come
with a penalty attached. But I see the downside of fixing the user
space API to a different version then the kernel you actually ship in
the end.

Regards,
Bastian

--
Knowledge, sir, should be free to all!
-- Harry Mudd, "I, Mudd", stardate 4513.3

[Dimitri John Ledkov]

On Thu, 16 Nov 2023 at 19:30, Bastian Blank <wa...@debian.org> wrote:
>
> Hi Dimitri
>
> On Thu, Nov 16, 2023 at 06:28:13PM +0000, Dimitri John Ledkov wrote:
> > I have implemented example packaging of that as a standalone source package
> > https://ppa.launchpadcontent.net/xnox/nonvirt/ubuntu/pool/main/l/linux-uapi/
>
> I actually just implemented something similar, but as part of the Debian
> linux packaging. It looks a bit different, sure.
>

Will check it out, thanks! And yes, indeed there are dozens of ways to
implement this idea.

> > Is this something that the debian kernel team could consider supporting /
> > building as either standalone source package, or out of src:linux directly?
>
> My first thought was actually to make bootstrapping of new architectures
> easier.
>

Indeed that too. But I was told "arm64" would the last one, and then
"riscv64" is really the last one. Not sure if c-sky or riscv128 or
arm64be will be next.

> > The obvious things missing from the packaging is to create all the
> > breaks/replaces/provides, and update cross-toolchain-base packages to
> > match. Also probably using symlinks rather than hard links, with
> > linux-libc-dev-cross likely depending on the native one.
>
> What do you want to do with linux-libc-dev-cross? /usr/$triplet is no
> allowed location in Debian anyway.

I'm not sure about if it is allowed or not, but it is the only
possible way to ship cross toolchains like in all releases since 2015
- https://tracker.debian.org/news/685686/accepted-cross-toolchain-base-2-source-all-into-unstable-unstable/

I think we (all the toolchainy people in debian & ubuntu, which is
like doko xnox helmut and whoever else) agreed to do it this way back
in Debconf in Swiss alps by the lake?! or like cambridge
mini-debconf?!

It's ok if you don't want to integrate `linux-libc-dev-cross` into
src:linux I can upload that into debian as a standalone
src:linux-libc-dev-cross under the toolchain-base maintainers hat that
will build-depend on linux-source and build it for all possible
triplets under the sun. And i think we override the lintian warnings
there. Because it will be easier to have that once, rather than in all
three cross-toolchain-base packages. And it can be updated, without
rebuilding the cross-toolchains themselves. Because it is wasteful to
acutally do cross toolchains bootstraps just to bump a stable point
release of linux headers that like changes nothing.

Or just integrate it into src:linux build too, given all of those
cross headers are established packages since 2015. (and yes using GNU
tripplet as a top level dir)

>
> > Separately for Ubuntu, due to the number of kernel built, I would likely
> > want to upload source package that produces linux-libc-dev as a separate
> > source package such that linux-libc-dev is actually only updated when
> > needed, rather than on every kernel upload. As there is no need to rev
> > linux-libc-dev as often as kernel uploads are done.
>
> The question here is: does it provide an advantage to have it as
> separate source? Less updates IMHO do not count, as they don't come
> with a penalty attached. But I see the downside of fixing the user
> space API to a different version then the kernel you actually ship in
> the end.
>

Fixing to a different (or sooner) API before kernels actually are
ready is one angle that helps Ubuntu. Separately in Ubuntu we have
like 16+ custom/customer kernels (i.e. src:linux-acme) and they all
keep trying to filter ubuntu repos and build what they need and get
confused when they need to build generic kernel and their custom
kernel, and constantly try to build linux-libc-dev & linux-source out
of their custom kernel and that has conflicts and API level
missmatched (in cases when custom kernel is ahead or behind the given
Ubuntu's release "baseline" API). Hence for me it will be easier in
Ubuntu-only to maintain src:linux that only build generic kernel
flavour; and src:linux-uapi builds "stable API" headers and toolchainy
things. And it will simplify filtered archives / installs too, of
having src:linux-$customer + src:linux-uapi to cover their needs.

Speed & frequency of updates matters too, as linux-libc-dev forms part
of the toolchain / buildinfo, and for cases were reproducible builds
matter, having less churn of linux-libc-dev helps a lot with having
stable toolchain version numbers. Also headers change a lot slower,
than underlying kernel. But that's a very niche thing (matters for
minimising reproducible rebuilds, buildd chroot refreshes, containers
that are used as buildds and the like).


--
okurrr,

Dimitri

[Dimitri John Ledkov]

On Thu, 16 Nov 2023 at 19:30, Bastian Blank <wa...@debian.org> wrote:
>

> Hi Dimitri
>
> On Thu, Nov 16, 2023 at 06:28:13PM +0000, Dimitri John Ledkov wrote:
> > I have implemented example packaging of that as a standalone source package
> > https://ppa.launchpadcontent.net/xnox/nonvirt/ubuntu/pool/main/l/linux-uapi/
>
> I actually just implemented something similar, but as part of the Debian
> linux packaging. It looks a bit different, sure.

Actually I cannot find this anywhere yet - is this something you only
have locally so far, or has it been pushed anywhere in a git repo,
pull request or anything?

--
okurrr,

Dimitri

[Bastian Blank]

On Thu, Nov 16, 2023 at 08:07:47PM +0000, Dimitri John Ledkov wrote:
> On Thu, 16 Nov 2023 at 19:30, Bastian Blank <wa...@debian.org> wrote:
> > On Thu, Nov 16, 2023 at 06:28:13PM +0000, Dimitri John Ledkov wrote:
> > > I have implemented example packaging of that as a standalone source package
> > > https://ppa.launchpadcontent.net/xnox/nonvirt/ubuntu/pool/main/l/linux-uapi/
> >
> > I actually just implemented something similar, but as part of the Debian
> > linux packaging. It looks a bit different, sure.
> Will check it out, thanks! And yes, indeed there are dozens of ways to
> implement this idea.

https://salsa.debian.org/kernel-team/linux/-/merge_requests/909

> It's ok if you don't want to integrate `linux-libc-dev-cross` into
> src:linux I can upload that into debian as a standalone
> src:linux-libc-dev-cross under the toolchain-base maintainers hat that
> will build-depend on linux-source and build it for all possible
> triplets under the sun. And i think we override the lintian warnings
> there. Because it will be easier to have that once, rather than in all
> three cross-toolchain-base packages. And it can be updated, without
> rebuilding the cross-toolchains themselves. Because it is wasteful to
> acutally do cross toolchains bootstraps just to bump a stable point
> release of linux headers that like changes nothing.
>
> Or just integrate it into src:linux build too, given all of those
> cross headers are established packages since 2015. (and yes using GNU
> tripplet as a top level dir)

I will add breaks to all those packages, so the cross packages must
adopt a standard schema for the headers. However it does not matter,
because the include list already is correct:

| #include <...> search starts here:
| /usr/lib/gcc-cross/s390x-linux-gnu/13/include
| /usr/lib/gcc-cross/s390x-linux-gnu/13/../../../../s390x-linux-gnu/include
| /usr/include/s390x-linux-gnu
| /usr/include
| End of search list.

Bastian

--
A Vulcan can no sooner be disloyal than he can exist without breathing.
-- Kirk, "The Menagerie", stardate 3012.4

[Bastian Blank]

On Thu, Nov 16, 2023 at 10:27:26PM +0100, Bastian Blank wrote:
> However it does not matter,
> because the include list already is correct:
>
> | #include <...> search starts here:
> | /usr/lib/gcc-cross/s390x-linux-gnu/13/include
> | /usr/lib/gcc-cross/s390x-linux-gnu/13/../../../../s390x-linux-gnu/include
> | /usr/include/s390x-linux-gnu
> | /usr/include
> | End of search list.

I can confirm that this works:

| % cat test.c
| #include <asm/errno.h>
| % dpkg --print-architecture
| arm64
| % s390x-linux-gnu-gcc -E test.c | grep errno.h
| # 1 "/usr/lib/linux/uapi/s390/asm/errno.h" 1 3 4
| # 1 "/usr/include/asm-generic/errno.h" 1 3 4
| # 6 "/usr/include/asm-generic/errno.h" 2 3 4
| # 2 "/usr/lib/linux/uapi/s390/asm/errno.h" 2 3 4

Bastian

--
Beam me up, Scotty! It ate my phaser!

[Helmut Grohne]

Hi Dimitri and Bastian,

On Thu, Nov 16, 2023 at 06:28:13PM +0000, Dimitri John Ledkov wrote:

> Currently in both Debian and Ubuntu we ship like close to 40 .deb packages
> that have linux-libc-dev (for native/multiarch, cross, ports, mipses).
>
> Each one of them is like 1.5MB, however they actually all have the same and
> repeated content.
>
> the bulk of headers are the same on all arches (and enforced via
> multi-arch:same), asm-generic is also the same for all of them, and then
> kernel-arch asm headers are unique - but only for given kernel arches (not
> all of the debian arch names).
>
> Currently there are only 13 kernel archs for unique asm-arch headers, but
> in Debian we have multiple reuses of the same kernel archs with different
> userspace abi properties (arm = armel armhf, mips = 12 mips archs, powerpc
> = 4, x86 = 3, and so on).
>
> It seems wasteful on disk, build-time, and derivative build time to package
> all of these separately. Especially since there is a chain of src:linux
> building native ones + linux-source as a .deb; which is then used by
> cross-toolchain-base{,-ports,-mips} to unpack and rebuild linux headers
> again, and publish. When in practice src:linux itself could build an
> efficient libc-dev packages for all arches as an arch:all package and
> Multi-Arch:foreign.

Thanks for working on this. I would have appreciated copying
debian...@lists.debian.org as I am not following d-ke...@l.d.o.

> The linux-libc-dev is native & multiarch uapi headers for all arches. The
> linux-libc-dev-alpha-cross is all debian arch crosses. It is implemented
> using hardlinks to maintain all the same paths that are currently being
> used by all the arch:any linux-libc-dev packages, and the
> linux-libc-dev-*-cross packages. In total they provide equivalent
> functionality as 40+ linux-libc-dev* current debs in either Debian or
> Ubuntu.

> Is this something that the debian kernel team could consider supporting /
> building as either standalone source package, or out of src:linux directly?
> as this would reduce 40+ .deb duplication in the mirror pools of the same
> content; and will eliminate build-depends on linux-source (and thus
> built-using) from src:cross-toolchain-base* and will actually ensure that
> all kernel headers for native / multiarch / cross arches are updated
> simultaneously, without need to wait for all arch:any builds to catch up
> first. It also gives ability to trivially create freestanding toolchains to
> any of the arches without actually building a full debian port for or
> having a working kernel for a given port.

I agree with this in principle, but I see a number of shortcomings and
hope you can address them.

When creating a new port, linux is one the pieces that comes first.
Hence building a new linux-lib-dev package was a natural with custom
changes was a very natural thing to do. Until now, we never needed to
build architecture-independent packages, so this is a noticeable
deviation. I'll be looking to support this in rebootstrap.

An example where this kinda fails is musl-linux-any. I'm still vaguely
trying to bootstrap this even though it kinda is stuck really hard on
musl vs systemd being unable to cooperate in any meaningful way. I'm not
sure it is reasonable to add these links the regular linux-libc-dev
package as 99% of people will never use them.

The same goes for esoteric architectures such as sh3, s390 (without x),
arc, loong64, and others.

I guess that the list of architectures will always be incomplete for
some, so we probably still need a process for building a modified
linux-libc-dev package only. This probably requires some build profiles.
Is pkg.linux.nokernel pkg.linux.notools pkg.linux.quick a sensible set
of profiles for this? Is there an easily patchable way to add an
architecture?

Let me also summarize how I see this change affecting various
infrastructure.

Bootstrapping will have to branch on whether the default linux-libc-dev
is usable and for many architectures it will be usable thus speeding up
bootstrap there.

For cross building we completely eliminate multi-arch skews of
linux-libc-dev that used to happen every so often. Cool.

For c-t-b, I guess that we can simply cut out linux-libc-dev and remove
all the -cross packages. I hope there is no devil in the detail.

So if the shortcomings end up having viable solutions, this seems like a
rather positive change to me.

Helmut

[Bastian Blank]

Hi Helmut

On Sat, Dec 23, 2023 at 05:15:18PM +0100, Helmut Grohne wrote:
> I guess that the list of architectures will always be incomplete for
> some, so we probably still need a process for building a modified
> linux-libc-dev package only. This probably requires some build profiles.
> Is pkg.linux.nokernel pkg.linux.notools pkg.linux.quick a sensible set
> of profiles for this? Is there an easily patchable way to add an
> architecture?

Let's see. I have some changes pending that make config changes easier.

Also we might be able to add a linux-libc-dev-arch that builds a
standalone version again and is only built with a special build profile,
but it still needs the package to know more information then dpkg does
provide.

Or you inject a new reboostrap-specific package that just adds a
symlink /usr/lib/$DEB_HOST_MULTIARCH/asm pointing to the appropriate
/usr/lib/linux/uapi/$ARCH if linux-libc-dev does not include this.

> For c-t-b, I guess that we can simply cut out linux-libc-dev and remove
> all the -cross packages. I hope there is no devil in the detail.

I would start with adding Provides and later Breaks for them to
linux-libc-dev. The compilers have proper search paths. (And they
violate the policy, so the devils are with those already.)

Bastian

--
Deflector shields just came on, Captain.

[Bastian Blank]

On Sun, Dec 24, 2023 at 04:10:52PM +0100, Bastian Blank wrote:
> On Sat, Dec 23, 2023 at 05:15:18PM +0100, Helmut Grohne wrote:
> > I guess that the list of architectures will always be incomplete for
> > some, so we probably still need a process for building a modified
> > linux-libc-dev package only. This probably requires some build profiles.
> > Is pkg.linux.nokernel pkg.linux.notools pkg.linux.quick a sensible set
> > of profiles for this? Is there an easily patchable way to add an
> > architecture?
>
> Let's see. I have some changes pending that make config changes easier.
>
> Also we might be able to add a linux-libc-dev-arch that builds a
> standalone version again and is only built with a special build profile,
> but it still needs the package to know more information then dpkg does
> provide.
>
> Or you inject a new reboostrap-specific package that just adds a
> symlink /usr/lib/$DEB_HOST_MULTIARCH/asm pointing to the appropriate
> /usr/lib/linux/uapi/$ARCH if linux-libc-dev does not include this.

I think we can also just ship all Linux architectures. We currently do
ship 13 (becoming 14) of the 21 currently supported ones. Then only the
multiarch aliases are missing.

Or, most likely bad idea, we teach the compiler or the libc to look into
/usr/lib/linux/uapi.

Bastian

--
What kind of love is that? Not to be loved; never to have shown love.
-- Commissioner Nancy Hedford, "Metamorphosis",
stardate 3219.8

[Bo YU]

Hi!

On Mon, Dec 25, 2023 at 3:00 AM Bastian Blank <wa...@debian.org> wrote:
>
> On Sun, Dec 24, 2023 at 04:10:52PM +0100, Bastian Blank wrote:
> > On Sat, Dec 23, 2023 at 05:15:18PM +0100, Helmut Grohne wrote:
> > > I guess that the list of architectures will always be incomplete for
> > > some, so we probably still need a process for building a modified
> > > linux-libc-dev package only. This probably requires some build profiles.
> > > Is pkg.linux.nokernel pkg.linux.notools pkg.linux.quick a sensible set
> > > of profiles for this? Is there an easily patchable way to add an
> > > architecture?
> >
> > Let's see. I have some changes pending that make config changes easier.
> >
> > Also we might be able to add a linux-libc-dev-arch that builds a
> > standalone version again and is only built with a special build profile,
> > but it still needs the package to know more information then dpkg does
> > provide.
> >
> > Or you inject a new reboostrap-specific package that just adds a
> > symlink /usr/lib/$DEB_HOST_MULTIARCH/asm pointing to the appropriate
> > /usr/lib/linux/uapi/$ARCH if linux-libc-dev does not include this.
>
> I think we can also just ship all Linux architectures. We currently do
> ship 13 (becoming 14) of the 21 currently supported ones. Then only the
> multiarch aliases are missing.

Unfortunately, there is no kernel to support loong64 in debian-ports.
>From my view, there is no package that can be built from a point, see:
https://buildd.debian.org/status/architecture.php?a=loong64&suite=sid

```
/usr/include/linux/types.h:5:10: fatal error: asm/types.h: No such
file or directory
5 | #include <asm/types.h>
```
just one of these full buildd log:
https://buildd.debian.org/status/fetch.php?pkg=brightnessctl&arch=loong64&ver=0.5.1-3.1&stamp=1703466178&raw=0

So could we fix the issue with the MR?
https://salsa.debian.org/kernel-team/linux/-/merge_requests/879

Or something?

BR,
Bo