Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Need help with Multi-Arch in systemd

99 views
Skip to first unread message

Michael Biebl

unread,
Jul 8, 2021, 5:30:03 PM7/8/21
to
Hi,

a couple of days ago, the following bug report was filed against systemd
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990547

I'm quoting the relevant parts

> I noticed that systemd-machined was failing to start on an arm64
> box. This box had armhf enabled, and turns out systemd had been
> cross-graded over to systemd:armhf[*]. It still had
> systemd-container:arm64 installed, so now I had an arm64
> /lib/systemd/systemd-machine, but an armhf
> /lib/systemd/libsystemd-shared-244.so.
>
> libsystemd-shared-244.so is from the systemd package. Since systemd is
> marked Multi-Arch: foreign, systemd:armhf was able to incorrectly
> satisfy systemd-container:arm64's dependency on systemd.

The systemd package provides a package private library
/lib/systemd/libsystemd-shared-{source:Version}.so, which is used by
binaries that are built from src:systemd. Some of those binaries are
split into separate binary packages, like systemd-container.
Since both systemd is marked as Multi-Arch: foreign, it can happen that
we get this architecture mismatch.

Asking on #debian-systemd, Marco d'Itri suggested, that we move
libsystemd-shared into a separate binary package. This would only help,
if we moved libsystemd-shared into a Multi-Arch location (which means,
we'd have to carry a patch against upstream). It would also mean, that
on every new upstream release, systemd would have to go through NEW.
So I'm not very keen on doing that.

Option 2 would be to drop Multi-Arch: foreign from systemd. This would
mean, that packages like libpam-systemd/libnss-systemd can no longer be
installed for a foreign architecture (even though those modules only use
architecture independent interfaces of systemd).

Option 3 is something I came up with after reading the Multi-Arch
related wiki pages:
https://salsa.debian.org/systemd-team/systemd/-/commit/c379461a14dcd13e0b625bd8faabbcf7fb3d19d6
It would mean marking systemd as Multi-Arch: allowed. And packages that
only use architecture interfaces of systemd would have to use the :any
annotation.


I would appreciate feedback, if option 3 is proper solution or not, or
if there are other, better alternatives. If we go with option 3, should
I inform all rdeps of systemd to update their dependency accordingly,
i.e. do a MBF?
Currently, I only see interpreters like perl, use M-A: allowed, so I'm
not sure if I'm misusing this feature.

Regards,
Michael



OpenPGP_signature

Helmut Grohne

unread,
Jul 9, 2021, 4:50:03 AM7/9/21
to
Hi Michael,

thanks for reaching out!

On Thu, Jul 08, 2021 at 11:03:48PM +0200, Michael Biebl wrote:
> a couple of days ago, the following bug report was filed against systemd
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990547

Imo, this is bug should be serious. In essence, it is a missing
dependency and we track missing dependencies as serious.

At the same time, I find it so unlikely to happen in practice combined
with the difficulty of finding a good solution that I'd propose
bullseye-ignore.

> Asking on #debian-systemd, Marco d'Itri suggested, that we move
> libsystemd-shared into a separate binary package. This would only help, if
> we moved libsystemd-shared into a Multi-Arch location (which means, we'd
> have to carry a patch against upstream). It would also mean, that on every
> new upstream release, systemd would have to go through NEW.
> So I'm not very keen on doing that.

I concur with Marco here and I argue that splitting the library is my
preferred solution. I confirm that you'd need to move the library for
this to work. For the other points I do not follow. I think it would be
ok to move the library using code inside debian/. Is there any reason
why you ruled out that option?

I also disagree with the need to go through NEW more than once. The new
package could quite simply be named libsystemd-private and lack a
.symbols and .shlibs file. Internal users would always use (=
${binary:Version}) anyway. The package description would declare that
any external dependency on libsystemd-private is a bug. Prior art:
libbinutils/binutils-dev.

One could argue that packaging a shared library like that is a policy
violation. If that is the case, then the current bundling is a policy
violation as well. So meh. It really sounds like a fair compromise to
me.

I am actually wondering now whether we should have a separate archive
section for "private" packages. I would define it as follows:

A private package is an implementation detail used by a single source.
Binary packages built from a one source package must not depend on
private packages built from another. Dependencies on private packages
should use tight version requirements (e.g. (= ${binary:Version}).

I suppose that a number of *-common and *-data packages as well as
gcc-N-base could be moved to such a section. User interfaces for package
managers could hide private packages by default.

Regardless of whether we add an archive section, lintian could carry a
list of private packages and enforce the no-dependency rule.

> Option 2 would be to drop Multi-Arch: foreign from systemd. This would mean,
> that packages like libpam-systemd/libnss-systemd can no longer be installed
> for a foreign architecture (even though those modules only use architecture
> independent interfaces of systemd).

That would break a pile of use cases and cause a lot of pain downstream.
Please don't.

> Option 3 is something I came up with after reading the Multi-Arch related
> wiki pages:
> https://salsa.debian.org/systemd-team/systemd/-/commit/c379461a14dcd13e0b625bd8faabbcf7fb3d19d6
> It would mean marking systemd as Multi-Arch: allowed. And packages that only
> use architecture interfaces of systemd would have to use the :any
> annotation.

Yeah, this technically works, but it causes a lot of maintenance effort,
because you get to hunt down every single systemd dependency in the
archive for years. I think our time is worth more than the cost of
introducing a new binary package into the archive.

> I would appreciate feedback, if option 3 is proper solution or not, or if
> there are other, better alternatives. If we go with option 3, should I
> inform all rdeps of systemd to update their dependency accordingly, i.e. do
> a MBF?

The problem I see here is less with annotating all deps once. It is more
the constant effort it incurs. It is not obvious that you should
annotate your systemd dependency :any. People will forget that when
adding systemd dependencies. What I take issue with is the permanent
cost that we keep in the long run.

Hope this helps

Helmut

Michael Biebl

unread,
Jul 9, 2021, 5:20:03 AM7/9/21
to
Am 09.07.21 um 10:28 schrieb Helmut Grohne:
> I concur with Marco here and I argue that splitting the library is my
> preferred solution. I confirm that you'd need to move the library for
> this to work. For the other points I do not follow. I think it would be
> ok to move the library using code inside debian/. Is there any reason
> why you ruled out that option?
>
> I also disagree with the need to go through NEW more than once. The new
> package could quite simply be named libsystemd-private and lack a
> .symbols and .shlibs file.

Splitting out libsystemd-shared (once) will make PID1 very brittle. It
can lead to situations where old systemd + new libsystemd-private is
installed. If the installation is aborted at this point, you have an
unbootable system.
This is why I think it's not a viable approach.


OpenPGP_signature

Michael Biebl

unread,
Jul 9, 2021, 5:50:03 AM7/9/21
to
Am 09.07.21 um 10:28 schrieb Helmut Grohne:
>
> On Thu, Jul 08, 2021 at 11:03:48PM +0200, Michael Biebl wrote:

>> Option 2 would be to drop Multi-Arch: foreign from systemd. This would mean,
>> that packages like libpam-systemd/libnss-systemd can no longer be installed
>> for a foreign architecture (even though those modules only use architecture
>> independent interfaces of systemd).
>
> That would break a pile of use cases and cause a lot of pain downstream.
> Please don't.

Can you iterate on that a bit? I was looking for use cases, but couldn't
come up with them.

>> Option 3 is something I came up with after reading the Multi-Arch related
>> wiki pages:
>> https://salsa.debian.org/systemd-team/systemd/-/commit/c379461a14dcd13e0b625bd8faabbcf7fb3d19d6
>> It would mean marking systemd as Multi-Arch: allowed. And packages that only
>> use architecture interfaces of systemd would have to use the :any
>> annotation.
>
> Yeah, this technically works, but it causes a lot of maintenance effort,
> because you get to hunt down every single systemd dependency in the
> archive for years. I think our time is worth more than the cost of
> introducing a new binary package into the archive.

As said, option 1 is not something I really consider a viable option for
the aforementioned reasons.
>> I would appreciate feedback, if option 3 is proper solution or not, or if
>> there are other, better alternatives. If we go with option 3, should I
>> inform all rdeps of systemd to update their dependency accordingly, i.e. do
>> a MBF?
>
> The problem I see here is less with annotating all deps once. It is more
> the constant effort it incurs. It is not obvious that you should
> annotate your systemd dependency :any. People will forget that when
> adding systemd dependencies. What I take issue with is the permanent
> cost that we keep in the long run.

So, packages need to decide whether they use architecture independent
interfaces of systemd or not and annotate it accordingly. But they only
need to do that once. Isn't this the same as with any python, ruby or
perl dependency?

Maybe having something like the inverse of M-A: allowed would be useful?
Instead of having packages explicitly opt-in via a :any annotation, they
can opt out and request the same architecture even if a package is
marked as M-A: foreign?
So the select few cases which use an architecture dependent interface of
M-A: foreign package would use something like pkg:arch ?










OpenPGP_signature

Timo Röhling

unread,
Jul 9, 2021, 6:10:03 AM7/9/21
to
* Michael Biebl <bi...@debian.org> [2021-07-09 11:01]:
>Splitting out libsystemd-shared (once) will make PID1 very brittle. It
>can lead to situations where old systemd + new libsystemd-private is
>installed. If the installation is aborted at this point, you have an
>unbootable system.
Helmut suggested a tightly versioned dependency, so such a state
would never be consistent. And given that system updates are not
atomic transactions, I'm pretty confident you can interrupt *any*
update of a boot-critical package at a point that leaves the system
unbootable.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀ ╭────────────────────────────────────────────────────╮
⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │
⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄⠀⠀⠀⠀ ╰────────────────────────────────────────────────────╯
signature.asc

Michael Biebl

unread,
Jul 9, 2021, 7:00:04 AM7/9/21
to
Am 09.07.2021 um 11:37 schrieb Timo Röhling:
> * Michael Biebl <bi...@debian.org> [2021-07-09 11:01]:
>> Splitting out libsystemd-shared (once) will make PID1 very brittle. It
>> can lead to situations where old systemd + new libsystemd-private is
>> installed. If the installation is aborted at this point, you have an
>> unbootable system.
> Helmut suggested a tightly versioned dependency, so such a state
> would never be consistent.

That tightly versioned dependency doesn't help unfortunately. There is
still a time window between the new libsystemd-shared and the new
systemd being unpacked.

Michael

Timo Röhling

unread,
Jul 9, 2021, 7:20:02 AM7/9/21
to
* Michael Biebl <bi...@debian.org> [2021-07-09 12:29]:
>That tightly versioned dependency doesn't help unfortunately. There is
>still a time window between the new libsystemd-shared and the new
>systemd being unpacked.
There's also a time window between files in /usr/bin and files in
/usr/lib being replaced.

If you reboot your system with an unfinished/interrupted upgrade,
you cannot assume a consistent or bootable state. Packaging all
stuff in the same package *may* reduce the above time window
somewhat, but it will not eliminate the underlying problem, so it is
a non-solution.

There may be other reasons to object Helmut's proposal, but this is
not it.
signature.asc

Michael Biebl

unread,
Jul 9, 2021, 7:50:03 AM7/9/21
to
Am 09.07.2021 um 13:01 schrieb Timo Röhling:
> * Michael Biebl <bi...@debian.org> [2021-07-09 12:29]:
>> That tightly versioned dependency doesn't help unfortunately. There is
>> still a time window between the new libsystemd-shared and the new
>> systemd being unpacked.
> There's also a time window between files in /usr/bin and files in
> /usr/lib being replaced.
>
> If you reboot your system with an unfinished/interrupted upgrade,
> you cannot assume a consistent or bootable state. Packaging all
> stuff in the same package *may* reduce the above time window
> somewhat, but it will not eliminate the underlying problem, so it is
> a non-solution.
>
> There may be other reasons to object Helmut's proposal, but this is
> not it.

I don't agree.
The time window between dpkg unpacking a single package and ending up in
a inconsistent state is neglibible to apt updating multiple packages
during a dist-upgrade.

Let me be blunt here: I'm not willing to compromise on robustness here.

Guillem Jover

unread,
Jul 9, 2021, 8:10:04 AM7/9/21
to
If the private library has no backwards or forward compatibility (due
to the SONAME used) the time window where the library does not match
the expectations of the stuff linked to it might indeed be too big.
But the reported bug is just a symptom of a bigger issue. This problem
already exists for any of the other binary packages built against this
private shared library, there's a time-window where they will not work
if the installation gets interrupted or fails, compared with public
shared libraries.

This implies to me the correct solution is a name-versioned package,
even though that seems cumbersome given our current archive handling
practices this is IMO the only correct solution. The alternative of
using Multi-Arch:allowed is a workaround for this packaging problem.

Another solution might be to request upstream to stabilize this library?
Yet another solution, which seems suboptimal, might be to statically
link it, but the shared library seems rather big, but perhaps if the
programs linked only use parts of it, that might not be too bad?

Thanks,
Guillem

Helmut Grohne

unread,
Jul 9, 2021, 8:50:02 AM7/9/21
to
Hi Michael,

in your other mail, you gave a very good reason for keeping the systemd
binary and libsystemd-shared.so in the same binary package. That
improves my understanding of why you favour option 3.

On Fri, Jul 09, 2021 at 11:31:15AM +0200, Michael Biebl wrote:
> Am 09.07.21 um 10:28 schrieb Helmut Grohne:
> >
> > On Thu, Jul 08, 2021 at 11:03:48PM +0200, Michael Biebl wrote:
>
> > > Option 2 would be to drop Multi-Arch: foreign from systemd. This would mean,
> > > that packages like libpam-systemd/libnss-systemd can no longer be installed
> > > for a foreign architecture (even though those modules only use architecture
> > > independent interfaces of systemd).
> >
> > That would break a pile of use cases and cause a lot of pain downstream.
> > Please don't.
>
> Can you iterate on that a bit? I was looking for use cases, but couldn't
> come up with them.

I guess you already know what comes next: cross building. 4700 packages
include systemd in their Build-Depends. Many of them would immediately
become cross bd-uninstallable. For others, the installation set would be
satisfied with a foreign systemd, which would quite simply fail to work.

Any kind of mixed-arch system where you want to install a daemon that
depends on systemd for a foreign architecture would also be affected.
Someone attempting to do so might accidentally cross grade systemd in
doing so.

> As said, option 1 is not something I really consider a viable option for the
> aforementioned reasons.

>From my pov, the only reason left against 1 is partial upgrades breaking
the installation. It's a strong one anyhow.

> So, packages need to decide whether they use architecture independent
> interfaces of systemd or not and annotate it accordingly. But they only need
> to do that once. Isn't this the same as with any python, ruby or perl
> dependency?

Yes. For perl and python, we haven't even annotated half of the relevant
Depends with :any yet. Experience tells, the annotation is not gonna
happen.

> Maybe having something like the inverse of M-A: allowed would be useful?
> Instead of having packages explicitly opt-in via a :any annotation, they can
> opt out and request the same architecture even if a package is marked as
> M-A: foreign?
> So the select few cases which use an architecture dependent interface of
> M-A: foreign package would use something like pkg:arch ?

If I were to design Multi-Arch today, I would likely agree. The
transition cost to get there seems prohibitively high though. I've tried
hard to change the spec and I did not succeed (on the Multi-Arch
interpreter problem). Yet, we devised a workaround for that issue. So
you made me thinking, can we somehow implement this with our current
spec? The most important requirements seem to be:

* libsystemd-shared.so and /sbin/systemd need to reside in the same
binary package.
* It shall be possible to depend on libsystemd-shared.so for a
particular architecture.
* A dependency on "systemd" should request a native systemd.

Now let's do something stupid. Rename systemd to systemd-core (taking
all files with it, please refrain from discussing the name unless you
seriously consider doing this). Mark it Multi-Arch: allowed. Add a new,
empty binary package systemd. It is Multi-Arch: foreign and depends on
systemd-core:any. This approach would technically satisfy all three
requirements, but it feels a little crazy to me.

Another possibility (kudos to David Kalnischkies) would be exploiting
something I might call a loophole in the Multi-Arch spec. While we don't
currently use arch-qualified dependencies in the archive, the spec
considers them and dpkg and apt support them (somewhat). So in theory,
we could say that systemd-container Depends: systemd:${DEB_HOST_ARCH}.
I'm not sure whether all resolvers agree on this, but I'd expect this
dependency to pull the right instance (despite being marked Multi-Arch:
foreign). Before jumping onto this, please consider David's warning:
here be dragons.

I'm relaying another question from David: Assuming we'd solve the
libsystemd-shared.so issue, would systemd-container:arch1 actually work
with systemd:arch2 or are there more aspects where the same architecture
is assumed?

As we already are going crazy in the search for solutions to this, let
me propose a scary one. I have no idea whether this works in practice.
Split out libsystemd-shared.so into systemd-private like before, but at
the same time ship it in systemd. That causes a conflict of course, so
systemd-private Conflicts: systemd:${DEB_HOST_ARCH} and systemd
Conflicts: systemd-private:${DEB_HOST_ARCH} as well as Provides:
systemd-private:${DEB_HOST_ARCH}. These Conflicts must be arch-qualified
as otherwise they're interpreted as any instance. We already have
arch-qualified Conflicts in the archive, so this aspect likely works.
I'm less sure about arch-qualified Provides though. In any case, the
(common) native case would be systemd without systemd-private and thus
harmless. Please consider this a crazy thought experiment for now.

And I fear we have another related issue that we swept under the carpet
thus far. man systemd.exec explains SystemCallArchitectures=native.
Thus we should consider it part of the API that systemd provides to
client packages. Of course what "native" means here depends on the
architecture of the systemd binary package. If I were to package a
daemon that uses SystemCallArchitectures=native and depend on systemd, I
could end up in a situation where my daemon architecture differs from
systemd's architecture (due to the present M-A:foreign). Once starting
the daemon, systemd would block all syscalls as they all use the wrong
architecture. So any package that uses SystemCallArchitectures=native
technically should depend on a systemd of the same architecture.
However, a better solution would be to not use
SystemCallArchitectures=native in binary packages at all and spell out
the architecture of the relevant binary package instead. It's a bit
annoying to do so for maintainers as upstreams will quite simply use
native as a value.

Looks like this leaves us between a rock and a hard place. None of the
options seems particularly attractive to me at this point.

Helmut

Timo Röhling

unread,
Jul 9, 2021, 9:40:03 AM7/9/21
to
* Michael Biebl <bi...@debian.org> [2021-07-09 13:24]:
>Let me be blunt here: I'm not willing to compromise on robustness here.
Well, I have had my say and ultimately, it's your package and
your decision, of course.
signature.asc

Michael Biebl

unread,
Jul 9, 2021, 4:50:03 PM7/9/21
to
Hi Guillem,

thanks for your feedback

Am 09.07.21 um 13:46 schrieb Guillem Jover:
> If the private library has no backwards or forward compatibility (due
> to the SONAME used) the time window where the library does not match
> the expectations of the stuff linked to it might indeed be too big.

The libsystemd-shared library is considered an implementation detail and
indeed has no guaranteed backwards or forward compatibility.
The soname is bumped when the first rc1 release is made (e.g v249-rc1 →
libsystemd-shared-249.so) and there might even be incompatible changes
between the rc1 and the final release.

> But the reported bug is just a symptom of a bigger issue. This problem
> already exists for any of the other binary packages built against this
> private shared library, there's a time-window where they will not work
> if the installation gets interrupted or fails, compared with public
> shared libraries.

This observation is correct I'd say. Currently we have the following
split-off binary packages which ship binaries that link against
libsystemd-shared:

systemd-container (4 binaries in $PATH, 7 services, 11 total)
systemd-coredump (1 binary in $PATH, 1 service, 2 total)
systemd-journal-remote (3 services, 3 total)
systemd-timesyncd (1 service, 1 total)

(I'll exclude systemd-tests, as this is a special case)

The main difference here, is that none of those binaries is really
critical for the runtime of the system.
An unbootable system though is one of the worst things that can happen.
Which is why I'm really reluctant to split off libsystemd-shared from
systemd and hopefully also explains why in general I'm not super keen on
chopping up src:systemd unnecessarily.

> This implies to me the correct solution is a name-versioned package,
> even though that seems cumbersome given our current archive handling
> practices this is IMO the only correct solution.

A name-versioned package would certainly be better then an unversioned
libsystemd-shared package. It still would make PID 1 a bit more brittle
though (see e.g. my comment regard rc releases).
It would also need patching of the build system, to override the rpath,
which would have to be multi-arch aware.

$ find . -name meson.build | xargs grep rpath | wc -l
123

This would be a significant patch with a lot of ongoing churn and
maintenance effort. I'm not sure if I'm willing or even able to do that.

> Another solution might be to request upstream to stabilize this library?

The point of libsystemd-shared is to be an internal implementation
detail where ABI can be broken. There are certainly no plans to change
that upstream and I don't think I can convince upstream otherwise on
this matter.

> Yet another solution, which seems suboptimal, might be to statically
> link it, but the shared library seems rather big, but perhaps if the
> programs linked only use parts of it, that might not be too bad?

So, option 4).
I tried that for v249. It's doable, but as this there no upstream
support for that, it requires a patch which is significant and would
require ongoing maintenance effort as well.

Regarding the numbers:

systemd-container (11 binaries)
Installed-Size: [-1278-] {+5644+}

systemd-coredump (2 binaries)
Installed-Size: [-276-] {+1106+}

systemd-journal-remote (3 binaries)
Installed-Size: [-336-] {+1218+}

systemd-timesyncd (1 binary)
Installed-Size: [-209-] {+595+}

I think the increase in size is significant.
More importantly, I'd need help maintaining this patch going forward


I acknowledge, that option 2 and 3 makes Helmut's work on cross-building
harder, although I think that option 3 would be preferrable to 2, as it
as least leaves the door open for making cross-building possible.


So, unless we get a :arch annotation that can be used for M-A:foreign
packages, maybe the best option is

option 5)
do nothing?

I mean, we shipped the M-A: foreign notation for systemd since 2014.
If there is one bug report regarding such an architecture mismatch in 7
years, maybe the most pragmatic approach is to simply ignore it, given
the downsides of the alternatives? I'm not sure what the best course of
action is here.

Regards,
Michael

OpenPGP_signature

Johannes Schauer Marin Rodrigues

unread,
Jul 9, 2021, 5:00:03 PM7/9/21
to
Quoting Helmut Grohne (2021-07-09 14:24:01)
> Another possibility (kudos to David Kalnischkies) would be exploiting
> something I might call a loophole in the Multi-Arch spec. While we don't
> currently use arch-qualified dependencies in the archive, the spec considers
> them and dpkg and apt support them (somewhat). So in theory, we could say
> that systemd-container Depends: systemd:${DEB_HOST_ARCH}. I'm not sure
> whether all resolvers agree on this, but I'd expect this dependency to pull
> the right instance (despite being marked Multi-Arch: foreign). Before jumping
> onto this, please consider David's warning: here be dragons.

No, not all solvers agree on this. Imagine this situation:

Package: pkga
Architecture: amd64
Depends: pkgb:i386
Multi-Arch: no

Package: pkgb
Architecture: i386
Multi-Arch: foreign

dose3, apt and dpkg agree that pkga can be installed because its dependency is
satisfied by pkgb. But if we change pkgb:i386 to pkgb:amd64 to end up with
this:

Package: pkga
Architecture: amd64
Depends: pkgb:amd64
Multi-Arch: no

Package: pkgb
Architecture: i386
Multi-Arch: foreign

Then dose3 can satisfy the dependency of pkg while apt and dpkg cannot. This is
because when building the cudf representation of above two packages, dose3
considers pkg to provide pkgb for all architectures, including amd64, because
it is marked as m-a:foreign.

Thanks!

cheers, josch
signature.asc

Sam Hartman

unread,
Jul 9, 2021, 5:50:03 PM7/9/21
to
>>>>> "Helmut" == Helmut Grohne <hel...@subdivi.de> writes:

Helmut> Looks like this leaves us between a rock and a hard
Helmut> place. None of the options seems particularly attractive to
Helmut> me at this point.

We seem to be in a brainstorming space here, throwing out half baked
ideas because none of us has a great answer. So, I'm going to join the
party. Are there games we could play with prerm scripts in a
systemd-private package to make sure that the old systemd private
library sticks around until the upgrade finishes when upgrading
systemd-private?

--Sam

Adrian Bunk

unread,
Jul 10, 2021, 11:50:02 AM7/10/21
to
On Fri, Jul 09, 2021 at 10:28:57AM +0200, Helmut Grohne wrote:
>...
> I also disagree with the need to go through NEW more than once. The new
> package could quite simply be named libsystemd-private and lack a
> .symbols and .shlibs file. Internal users would always use (=
> ${binary:Version}) anyway. The package description would declare that
> any external dependency on libsystemd-private is a bug. Prior art:
> libbinutils/binutils-dev.
>...

The prior art of libbinutils/binutils-dev is that instead of shared
linking, users are linking to the static version to the private library
(see e.g. #964536).

I don't understand what benefits this switch to static linking gives,
except for making security support harder.

> Helmut

cu
Adrian

Helmut Grohne

unread,
Jul 13, 2021, 4:50:03 PM7/13/21
to
Hi Michael,

I'm not yet fully convinced that we're out of options, but let's for a
moment assume we were.

On Fri, Jul 09, 2021 at 10:26:43PM +0200, Michael Biebl wrote:
> So, unless we get a :arch annotation that can be used for M-A:foreign
> packages, maybe the best option is

Given Johannes' reply, I think that :arch annotations can be a strict
improvement to the status quo. dose will ignore them, so you aren't
worse off with it. apt and dpkg agree to honour them, so they'd
practically solve the symptoms (despite leading dose into non-solutions
to certain satisfiability issues that are mostly relevant to cross
building only).

The one question here is whether those are the semantics that we want
for this corner case of multiarch. A while ago, David changed apt to
make it stop behaving like dose and instead make it behave like dpkg.
That looks like we had consensus back then. Did we document it anywhere?

> option 5)
> do nothing?
>
> I mean, we shipped the M-A: foreign notation for systemd since 2014.
> If there is one bug report regarding such an architecture mismatch in 7
> years, maybe the most pragmatic approach is to simply ignore it, given the
> downsides of the alternatives? I'm not sure what the best course of action
> is here.

If :arch qualifiers are an option, I think they are a strict improvement
over doing nothing. That's something I couldn't convincingly say about
any other option. In any case, I don't think fixing this in time for
bullseye is reasonable, so we do have some time figuring out whether
:arch is a solution.

Helmut

Bálint Réczey

unread,
Jul 13, 2021, 6:10:03 PM7/13/21
to
Hi Michael,
IMO what Guillem recommends, i.e. the name-versioned libsystemd
package with versioned shared library name is still the cleanest and
so far the only reasonably reliable solution. IMO going through NEW in
every few months could be an acceptable cost especially since I was
quite happy with the pace at which NEW is processed recently. I don't
maintain systemd in Ubuntu anymore, but if I were, I'd be happy to
accept this increased maintenance burden for the sake of having a
clean solution for the Multi-Arch problem. If release candidates will
be uploaded only to experimental then the incompatible changes won't
cause problems. Even if RC-s are uploaded to unstable and a rarely
occurring incompatible change takes place then the library package
name can be bumped again.
After we discussed this topic on #debian devel on 2020-05-05 I even
started implementing the solution but did not finish it because it was
not an important problem from Ubuntu's POV.

I think the rpath usage search counted a lot of hits which don't have
to be changed and maybe upstream would be willing to accept the patch.

Cheers,
Balint

Josh Triplett

unread,
Jul 14, 2021, 12:20:03 AM7/14/21
to
Helmut Grohne wrote:
> So you made me thinking, can we somehow implement this with our
> current spec? The most important requirements seem to be:
>
> * libsystemd-shared.so and /sbin/systemd need to reside in the same
> binary package.
> * It shall be possible to depend on libsystemd-shared.so for a
> particular architecture.
> * A dependency on "systemd" should request a native systemd.
>
> Now let's do something stupid. Rename systemd to systemd-core (taking
> all files with it, please refrain from discussing the name unless you
> seriously consider doing this). Mark it Multi-Arch: allowed. Add a new,
> empty binary package systemd. It is Multi-Arch: foreign and depends on
> systemd-core:any. This approach would technically satisfy all three
> requirements, but it feels a little crazy to me.

This seems like a rather reasonable approach, actually. It's a little
unusual, but it has all the advantages of making systemd multi-arch
aware while not creating the trap of making a `systemd` dependency do
the wrong thing, because `systemd` has the right multi-arch dependency
on `systemd-core:any`.

The handful of packages that really do need a same-arch dependency on
systemd (those that are part of systemd itself and need
libsystemd-shared) could depend on `systemd-core` directly, and
everything else can continue depending on systemd with no transition
required. And there's only one trip through NEW, once.

Simon McVittie

unread,
Jul 14, 2021, 7:20:03 AM7/14/21
to
On Thu, 08 Jul 2021 at 23:03:48 +0200, Michael Biebl wrote:
> [a separate libsystemd-shared-249 .deb] would also mean, that on every
> new upstream release, systemd would have to go through NEW

It seems like we're rejecting a good technical solution because
social/organisational factors block it (namely, new binary packages
triggering manual review from the ftp team even if there has not been
any significant change in how the package is organised, resulting in
manual review being artificially frequent for libraries that happen to
break ABI a lot, but infrequent for packages that aren't libraries or
don't break ABI).

This seems like a side-effect of the ftp team's two gatekeeping roles
(legal review and managing the namespaces of source and binary package
names) both having the dak override file as their implementation, rather
than necessarily anything that was designed or intended.

Would it be feasible for dak to have a list of binary package name
regexes mapped to a source package and a section/priority, and auto-accept
packages from the given source package that match the regex, assigning
the given section/priority, without manual action? That would let the
ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/
in libs/optional, for example.

It seems like this would also be good for src:linux, where ABI breaks
are often tied to security fixes that should enter the archive ASAP.

smcv

Michael Biebl

unread,
Jul 14, 2021, 8:10:03 AM7/14/21
to
Am 14.07.21 um 12:59 schrieb Simon McVittie:
> Would it be feasible for dak to have a list of binary package name
> regexes mapped to a source package and a section/priority, and auto-accept
> packages from the given source package that match the regex, assigning
> the given section/priority, without manual action? That would let the
> ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/
> in libs/optional, for example.
>
> It seems like this would also be good for src:linux, where ABI breaks
> are often tied to security fixes that should enter the archive ASAP.

If something fully automated like this would be implemented, I would
have much less concerns with this option.

As it stands today, NEW processing is simply to unpredictable. It can
range from taking a a few hours/days to several months.

Regards,
Michael







OpenPGP_signature

Michael Biebl

unread,
Jul 14, 2021, 9:00:03 AM7/14/21
to
Am 14.07.21 um 13:47 schrieb Michael Biebl:
> Am 14.07.21 um 12:59 schrieb Simon McVittie:
>> Would it be feasible for dak to have a list of binary package name
>> regexes mapped to a source package and a section/priority, and
>> auto-accept
>> packages from the given source package that match the regex, assigning
>> the given section/priority, without manual action? That would let the
>> ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/
>> in libs/optional, for example.
>>
>> It seems like this would also be good for src:linux, where ABI breaks
>> are often tied to security fixes that should enter the archive ASAP.
>
> If something fully automated like this would be implemented, I would
> have much less concerns with this option.


Fwiw, I like your proposal and would like to see it implemented
regardless of the issue at hand as I think it could be generally useful.

Thanks, Simon!


OpenPGP_signature

Timo Röhling

unread,
Jul 14, 2021, 9:20:02 AM7/14/21
to
* Simon McVittie <sm...@debian.org> [2021-07-14 11:59]:
>Would it be feasible for dak to have a list of binary package name
>regexes mapped to a source package and a section/priority, and auto-accept
>packages from the given source package that match the regex, assigning
>the given section/priority, without manual action? That would let the
>ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/
>in libs/optional, for example.
+1

Especially considering that library packages are subject to transitions
anyway, so they already receive much more scrutiny than, say, an
updated Python module.
signature.asc

Bastian Blank

unread,
Jul 14, 2021, 11:00:03 AM7/14/21
to
On Wed, Jul 14, 2021 at 11:59:11AM +0100, Simon McVittie wrote:
> It seems like this would also be good for src:linux, where ABI breaks
> are often tied to security fixes that should enter the archive ASAP.

As security updates are hand approved, accepting by NEW does not help
that much.

Bastian

--
Peace was the way.
-- Kirk, "The City on the Edge of Forever", stardate unknown

Bastian Blank

unread,
Jul 14, 2021, 11:00:03 AM7/14/21
to
On Thu, Jul 08, 2021 at 11:03:48PM +0200, Michael Biebl wrote:
> Asking on #debian-systemd, Marco d'Itri suggested, that we move
> libsystemd-shared into a separate binary package. This would only help, if
> we moved libsystemd-shared into a Multi-Arch location (which means, we'd
> have to carry a patch against upstream). It would also mean, that on every
> new upstream release, systemd would have to go through NEW.
> So I'm not very keen on doing that.

Why do you think? libsystemd-shared is a perfectly valid package name
and it does not change.

However, if you call it libsystemd-shared-249, it is supposed to be
stable.

Bastian

--
Is truth not truth for all?
-- Natira, "For the World is Hollow and I have Touched
the Sky", stardate 5476.4.

Philipp Kern

unread,
Jul 14, 2021, 1:10:02 PM7/14/21
to
And yet it should not dictate technical solutions. We basically see the
same thing with nvidia-graphics-drivers that break your running
applications when the libraries are upgraded and you don't reboot.
Arguably the proper solution is to version them with the full
major/minor version. But I can see how that's a total hassle with NEW
processing for both for the maintainer and to the FTP team.

I do recall that the FTP masters would've been generally open to have
such an auto-approver (but maybe I'm wrong), but that no-one stepped up
yet to code it up?

Kind regards
Philipp Kern

Guillem Jover

unread,
Jul 14, 2021, 6:30:03 PM7/14/21
to
On Wed, 2021-07-14 at 11:59:11 +0100, Simon McVittie wrote:
> On Thu, 08 Jul 2021 at 23:03:48 +0200, Michael Biebl wrote:
> > [a separate libsystemd-shared-249 .deb] would also mean, that on every
> > new upstream release, systemd would have to go through NEW
>
> It seems like we're rejecting a good technical solution because
> social/organisational factors block it (namely, new binary packages
> triggering manual review from the ftp team even if there has not been
> any significant change in how the package is organised, resulting in
> manual review being artificially frequent for libraries that happen to
> break ABI a lot, but infrequent for packages that aren't libraries or
> don't break ABI).

Yes. I was mentioning exactly this the other day on the
#debian-bootstrap IRC channel.

In addition having this automatic support could make life easier for
many other potential packages.

> This seems like a side-effect of the ftp team's two gatekeeping roles
> (legal review and managing the namespaces of source and binary package
> names) both having the dak override file as their implementation, rather
> than necessarily anything that was designed or intended.

Yes, plus section and priority-spaces. But then, I don't see why a binary
package rename should trigger a new legal audit.

> Would it be feasible for dak to have a list of binary package name
> regexes mapped to a source package and a section/priority, and auto-accept
> packages from the given source package that match the regex, assigning
> the given section/priority, without manual action? That would let the
> ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/
> in libs/optional, for example.

What I had in mind was that DAK would gain support for automatic
ACCEPT of binary package renames due to SONAME version bumps,
something like this:

* If the new bin:lib<name><version-b>:
- replaces an existing bin:lib<name><version-a> from the same
source, where <version-a> is lower than <version-b>.
- contains a shared library mapping to that package name.
- is in section */libs or */oldlibs.
* Then → auto-ACCEPT, pre-filling the new section/priority from the
old binary package.
* Otherwise → NEW.

I guess it could potentially be further extended later on to cover
other safe non shared library cases.

But if that's too much to ask, either due to implementation or policy
concerns, I'd take an explicit allowlist letting specific cases
through, such as the systemd one, instead of having to settle for
either suboptimal or wrong solutions for the problem at hand, due
to the currently required workflow being too cumbersome.

Thanks,
Guillem

Michael Biebl

unread,
Jul 14, 2021, 6:40:03 PM7/14/21
to
Am 09.07.21 um 14:24 schrieb Helmut Grohne:
> Now let's do something stupid. Rename systemd to systemd-core (taking
> all files with it, please refrain from discussing the name unless you
> seriously consider doing this). Mark it Multi-Arch: allowed. Add a new,
> empty binary package systemd. It is Multi-Arch: foreign and depends on
> systemd-core:any. This approach would technically satisfy all three
> requirements, but it feels a little crazy to me.

[..]

> And I fear we have another related issue that we swept under the carpet
> thus far. man systemd.exec explains SystemCallArchitectures=native.


You are right. Thinking more about this, splitting out libsystemd-shared
as a Multi-Arch: same library will not help with
SystemCallArchitectures=native, which is used by the services in
systemd-{container,journal-remote,...}.
So splitting out libsystemd-shared, while in theory a nice solution, is
not the right one in this case. We really need to ensure that systemd
and systemd-* are of of the same architecture.

I still think that my idea of having a ":arch" annotation as counterpart
to ":any" would be the most elegant way to achieve this. But since this
is only an idea and not implemented, it's not something I can make use
of. Do you think there is a chance we could convince dpkg and apt
maintainers to add support for that?


Alternatively, your idea of systemd-core/systemd got me thinking.
While I don't like the prospect of moving all (conf)files and state from
one package to another, maybe we can turn this idea around.

We introduce an empty systemd-native package, which is
Architecture:linux-any but *not* Multi-Arch: foreign/same/allowed.

systemd and all systemd-* packages would depend on systemd-native.
This would link the architecture of systemd and systemd-* together and
ensure they are the same.
Would this work for your cross-compile use-case?

Michael

OpenPGP_signature

Helmut Grohne

unread,
Jul 15, 2021, 2:40:03 AM7/15/21
to
Hi Michael,

On Thu, Jul 15, 2021 at 12:22:59AM +0200, Michael Biebl wrote:
> You are right. Thinking more about this, splitting out libsystemd-shared as
> a Multi-Arch: same library will not help with
> SystemCallArchitectures=native, which is used by the services in
> systemd-{container,journal-remote,...}.

That is correct, but it actually goes beyond systemd-* using systemd.
Any other service can face the very same problem.

> So splitting out libsystemd-shared, while in theory a nice solution, is not
> the right one in this case. We really need to ensure that systemd and
> systemd-* are of of the same architecture.

We have two related issues at hand. One is the libsystemd-shared
architecture matching and the other is the SystemCallArchitectures
matching. You appear to imply that both issues need to be addressed by
one common solution. Sure that would be nice, but is it required? Maybe
these issues are not that related after all.

Let us for a moment assume that we could magically make
SystemCallArchitectures work by locking users to the same architecture
as systemd. That would be bad in terms for mixed multiarch systems and
cross grading, because you essentially lock all daemons to the same
architecture as systemd. You fix the problem, by removing flexibility.

I also proposed a solution here, which is avoiding
SystemCallArchitectures=native. Initially, that sounds like a
maintenance nightmare until you notice that it can be solved on a
technical level. We already have dh_installsystemd. How bad would it be
to have dh_installsystemd change .service files and automatically
replace =native with a concrete architecture (in arch:any packages
only)? systemd would no longer detect the architecture of services from
its own one but rather use the one documented by the package. The mixing
of architectures that our earlier solution broke would now partially
work. We'd fix one package and binNMU the pile.

So while these problems are related, I think that forcing the
architectures to equal is a suboptimal solution for
SystemCallArchitectures.

> I still think that my idea of having a ":arch" annotation as counterpart to
> ":any" would be the most elegant way to achieve this. But since this is only
> an idea and not implemented, it's not something I can make use of. Do you
> think there is a chance we could convince dpkg and apt maintainers to add
> support for that?

If you replace :arch with :$DEB_HOST_ARCH, it already works today with
apt and dpkg, but not dose. The question is not whether we can implement
that (it already is), but whether we want to endorse these semantics or
not.

> Alternatively, your idea of systemd-core/systemd got me thinking.
> While I don't like the prospect of moving all (conf)files and state from one
> package to another, maybe we can turn this idea around.
>
> We introduce an empty systemd-native package, which is
> Architecture:linux-any but *not* Multi-Arch: foreign/same/allowed.
>
> systemd and all systemd-* packages would depend on systemd-native.
> This would link the architecture of systemd and systemd-* together and
> ensure they are the same.

I think this technically works. We also have prior art for this. blas
temporarily added such a package as a measure to fix #760936.

> Would this work for your cross-compile use-case?

I don't think this would negatively affect cross compiling. It could
affect people who try to run mixed-architecture systems though.

Given Simon's and Guillem's replies, I presently favour fixing the NEW
processing to package the library separately and fix
SystemCallArchitectures using dh_installsystemd, because it is
technically sound and does not negatively impact multiarch use cases.

Should I file a bug about SystemCallArchitectures such that we can track
it somewhere?

Helmut

Joerg Jaspert

unread,
Jul 19, 2021, 4:40:03 PM7/19/21
to
On 16194 March 1977, Simon McVittie wrote:

> Would it be feasible for dak to have a list of binary package name
> regexes mapped to a source package and a section/priority, and
> auto-accept
> packages from the given source package that match the regex, assigning
> the given section/priority, without manual action? That would let the
> ftp team pre-approve src:systemd to ship /^libsystemd-shared-[0-9]+$/
> in libs/optional, for example.

At some point somewhere I think it was already said, but in general: We
love MRs, we are at https://salsa.debian.org/ftp-team/dak/ and something
doing kind of auto-NEW processing is NOT out of the question.

Exact details have to be hashed out, but this is the wrong thread for
that. While sometimes NEW can be annoying, it regularly catches bugs
even if "only a small change in packaging" happened, so it does have
some reason why its there. But yes, there are some candidates (hello
kernel) that can make use of something with less humans involved.

Your example above doesn't sound too bad as a starter, though haven't
put any further thought into it yet.

--
bye, Joerg

Michael Lustfield

unread,
Jul 20, 2021, 3:30:02 AM7/20/21
to
On Wed, 14 Jul 2021 18:48:24 +0200
Philipp Kern <pk...@debian.org> wrote:

> On 14.07.21 13:47, Michael Biebl wrote:
> > Am 14.07.21 um 12:59 schrieb Simon McVittie:
> I do recall that the FTP masters would've been generally open to have
> such an auto-approver (but maybe I'm wrong), but that no-one stepped up
> yet to code it up?

A few of us came up with some proof of concept designs/models, but we
ultimately dropped the effort when it became clear the team wasn't interested
in such tools. We considered continuing with a tool that would work for
individual users reviewing their own work, but there just wasn't enough
interest for that either.

I'd be happy to help resurrect (the personal-use version of) the project/effort
if there's any chance I won't be working on it by myself...

Joerg Jaspert

unread,
Jul 20, 2021, 6:20:04 PM7/20/21
to
On 16200 March 1977, Michael Lustfield wrote:

>> I do recall that the FTP masters would've been generally open to have
>> such an auto-approver (but maybe I'm wrong), but that no-one stepped
>> up
>> yet to code it up?
> A few of us came up with some proof of concept designs/models, but we
> ultimately dropped the effort when it became clear the team wasn't
> interested
> in such tools. We considered continuing with a tool that would work
> for
> individual users reviewing their own work, but there just wasn't
> enough
> interest for that either.

> I'd be happy to help resurrect (the personal-use version of) the
> project/effort
> if there's any chance I won't be working on it by myself...

The place for that feature to end up in is inside dak, obviously.
That feature should read its config (is it enabled? for the current
package? with which config for it?) from the database (projectb).
And then just hook itself into the part of the uploading that redirects
packages to NEW.

And then one could look if it gets implemented a bit more generic and
not NEW specific, but able to do something like this for any policy
queue. (As in, backports NEW, p-u, whatever). So, configurable per
queue. For the actual feature and the affected packages.

Any MR going in that direction will be great.

--
bye, Joerg

Helmut Grohne

unread,
Jul 22, 2021, 9:00:02 AM7/22/21
to
On Thu, Jul 15, 2021 at 08:08:13AM +0200, Helmut Grohne wrote:
> I also proposed a solution here, which is avoiding
> SystemCallArchitectures=native. Initially, that sounds like a
> maintenance nightmare until you notice that it can be solved on a
> technical level. We already have dh_installsystemd. How bad would it be
> to have dh_installsystemd change .service files and automatically
> replace =native with a concrete architecture (in arch:any packages
> only)? systemd would no longer detect the architecture of services from
> its own one but rather use the one documented by the package. The mixing
> of architectures that our earlier solution broke would now partially
> work. We'd fix one package and binNMU the pile.

I've discussed this with #systemd on libera.chat. That resulted in a new
approach: Letting systemd detect the architecture of the binary being
executed. We've reported that as a feature request at
https://github.com/systemd/systemd/issues/20277. I therefore propose to
temporarily ignore the SystemCallArchitectures issue on the Debian side
and see what upstream thinks about it.

The libsystemd-shared issue can be tackled independently.

Helmut
0 new messages