Bug#1035904: dpkg currently warning about merged-usr systems (revisited) (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Ansgar]

Package: tech-ctte
X-Debbugs-Cc: Russ Allbery <r...@debian.org>, Sean Whitton <spwh...@spwhitton.name>, Helmut Grohne <hel...@subdivi.de>, Luca Boccassi <bl...@debian.org>, debia...@lists.debian.org, debian...@lists.debian.org

On Wed, 2023-05-10 at 14:36 -0700, Russ Allbery wrote:
> Ansgar <ans...@43-1.org> writes:
> > Debian going out of its way to tell derivative users to switch back from
> > merged-/usr to split-/usr is the *opposite* of trying to make things as
> > smooth for them as possible.
>
> Yes, I agree with that part and I think I objected to that at the time.
> Nonetheless, one bad decision doesn't mean that it is Debian policy that
> we don't care about derivatives or their users.  I think we made a mistake
> there which is not in alignment with our ideals or our goals.  We should
> try to reverse that mistake, not double down on it.

Cool, then let's ask tech-ctte.

Dear ctte, please consider overruling the dpkg maintainer to include
the patch from #994388[1].

Thanks,
Ansgar

[1]: https://bugs.debian.org/994388#397

[Ansgar]

On Wed, 2023-05-10 at 23:47 +0200, Ansgar wrote:
> Cool, then let's ask tech-ctte.
>
> Dear ctte, please consider overruling the dpkg maintainer to include
> the patch from #994388[1].
>
> Thanks,
> Ansgar
>
>   [1]: https://bugs.debian.org/994388#397

For derivatives based on Debian stable it might be worth having this
included in the next stable release; this would need a fairly quick
decision on this issue.

Ansgar

[Emilio Pozuelo Monfort]

Hi Sean,

On 11/05/2023 03:59, Sean Whitton wrote:
> Hello,

>
> On Wed 10 May 2023 at 11:47PM +02, Ansgar wrote:
>
>> Dear ctte, please consider overruling the dpkg maintainer to include
>> the patch from #994388[1].
>

> Currently dpkg contains code to emit the merged-/usr warning, that's
> dead code on Debian, but which becomes active when packages from the
> Debian archive are copied unmodified into derivatives.
>
> The heart of the issue is how dpkg is a native package. What we're
> talking about is not the Debian system, but the Debian archive as it
> exists independently of the Debian system.
>
> dpkg has an upstream existence that's independent of Debian, and it's
> perfectly legitimate for that version of dpkg to emit the warning. The
> problem here might be caused by how the Debian archive is implicitly
> being used to distribute upstream dpkg.
>
> This is not in itself a problem -- we distribute a lot of stuff in
> source packages that does not form part of the Debian system. But in
> this case, this distribution that's occurring might conflict with how
> Debian is seeking to provide a product not just to end users, but also
> to those building derivatives.
>
> One simple solution is for dpkg to become a non-native package, carrying
> Debian-specific patches to do things like remove the warning code.

I think you're conflating two independent things.

If you override the dpkg maintainer to remove that warning that occurs on
derivatives, then anyone could NMU dpkg and the maintainer wouldn't introduce it
back, effectively removing the warning from "dpkg upstream".

OTOH if the dpkg maintainer switches to non-native packages, anyone could NMU it
adding the change as a patch, however the maintainer will just NACK the NMU
before or after it happens.

So I don't see a problem with dpkg being native, just like e.g. apt is, and that
won't magically solve the issue at hand.

Cheers,
Emilio

[Simon Richter]

Hi,

On 5/11/23 10:59, Sean Whitton wrote:

>> Dear ctte, please consider overruling the dpkg maintainer to include
>> the patch from #994388[1].

> Currently dpkg contains code to emit the merged-/usr warning, that's
> dead code on Debian, but which becomes active when packages from the
> Debian archive are copied unmodified into derivatives.

The way I see it (but I'm not a dpkg maintainer), the current
implementation is correct, as dpkg does not support aliased directories,
but Debian has decided to use it in such an environment nonetheless. The
tech-ctte decision not to roll back usrmerge accepts responsibility for
this decision, so silencing the warning on Debian is correct, but no one
has accepted that responsibility for derived distributions.

Any derived distribution can easily go on record and request inclusion
in the list of distributions where this warning is suppressed, by typing
the phrase "Yes, I understand that this is a bad idea." into an email
client.

> dpkg has an upstream existence that's independent of Debian, and it's
> perfectly legitimate for that version of dpkg to emit the warning. The
> problem here might be caused by how the Debian archive is implicitly
> being used to distribute upstream dpkg.

I'm skeptical about splitting development and packaging, though,
especially in this context, because we'd need to clarify how far we'd
want upstream dpkg and Debian dpkg to deviate.

Basically, non-native dpkg has two scenarios: either it is maintained by
the same people as now, which causes them extra work for no clear
benefit, or we find maintainers willing to deal with complex bug reports
that upstream is fully entitled to reject because Debian brought this
onto themselves by deciding that one upstream project's "unsupported
configuration" needs to be avoided by moving to another upstream
project's "unsupported configuration."

Those same people who would volunteer to maintain such a package could
spend their energy finding a solution that can be merged into "upstream"
dpkg, that would be more productive.

Simon

[Luca Boccassi]

On Thu, 11 May 2023 21:16:34 +0900 Simon Richter <s...@debian.org>
wrote:

The crux of the issue is that we are hearing how negatively affecting
derivatives in any way, even purely theoretically, is a big no-no, in
this very same thread and topic. Reaching out and asking for
directions/help/whatever is not enough in that context. So it follows
that it cannot be enough in this context either, and it must be fixed
instead.

Or alternatively, we can establish that a documentation/post-facto
approach is enough for derivatives, and then that's valid for all
changes and transitions.

Either of these are valid approaches.

What I cannot find acceptable is that some changes get a free pass, and
some get roadblocks after roadblocks thrown at them.

--
Kind regards,
Luca Boccassi

[Ansgar]

On Wed, 2023-05-10 at 19:01 -0700, Sean Whitton wrote:
> On Wed 10 May 2023 at 11:47PM +02, Ansgar wrote:

> > Cool, then let's ask tech-ctte.
> >
> > Dear ctte, please consider overruling the dpkg maintainer to
> > include
> > the patch from #994388[1].
> >
> > Thanks,
> > Ansgar
> >
> >   [1]: https://bugs.debian.org/994388#397
>

> This would require a new, maintainer-overruling vote.
> Our existing decisions do not apply, so far as I can tell.

Yes, I agree.

> I have written a separate message to the bug and to debian-dpkg with
> a proposal to avoid having to have such a vote.

That seems to be about an implementation detail on how to apply the
patch. I don't think that is the core of the issue?

The core issue as I see it is as follows:

- Debian has decided to support only merged-/usr, including possibly
moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
as the interpreter in binaries.

- This change breaks on non-merged-/usr systems, including derivatives
that do not revert *all* relevant changes. (Do you know one that
does this or plans to do so?)

- dpkg recommends derivative users to move to non-merged-/usr.

I think this contradiction is not good and the core conflict. For me a
distribution should have some coherence. It is not just a distribution
of unrelated parts (like linux, libc, dpkg, dash, ...), but also
integrates them to work together.

And this also means not one package telling users to do X which breaks
other packages. Or (if other packages would do similar things as dpkg)
one package asking users to do X and the other asking users to do the
opposite of X. Just imagine dpkg asking users to move to split-/usr and
then another package starting to warn users to move back to merged-
/usr. Would that be a good state? I think not which is why this bug
exists.

Do you think this summary of the issue is right?

Is there some consensus about how this issue should be solved or do we
need a longer discussion to explore the solution space?

Ansgar

[Steve McIntyre]

On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
>
>The core issue as I see it is as follows:
>
>- Debian has decided to support only merged-/usr, including possibly
> moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> as the interpreter in binaries.

WTF? *Nobody* has been talking about breaking ABI like this, that I've
seen. The interpreter must *not* be changed willy-nilly.

--
Steve McIntyre, Cambridge, UK. st...@einval.com
"I've only once written 'SQL is my bitch' in a comment. But that code
is in use on a military site..." -- Simon Booth

[Luca Boccassi]

On Fri, 12 May 2023 at 09:40, Steve McIntyre <st...@einval.com> wrote:
>
> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >
> >The core issue as I see it is as follows:
> >
> >- Debian has decided to support only merged-/usr, including possibly
> > moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> > as the interpreter in binaries.
>
> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> seen. The interpreter must *not* be changed willy-nilly.

Nothing's happening 'willy-nilly'. We are discussing a bunch of
seemingly crazy options, as in, "what would _actually_ explode if we
do this or do that?", on this very d-devel thread. I posted a longer
version here some days ago:

https://lists.debian.org/debian-gcc/2023/05/msg00030.html

Kind regards,
Luca Boccassi

[Steve McIntyre]

Oh holy fuck.

You're talking about changing ABI by doing this. That *is* utterly
crazy. No.

--
Steve McIntyre, Cambridge, UK. st...@einval.com

"...In the UNIX world, people tend to interpret `non-technical user'
as meaning someone who's only ever written one device driver." -- Daniel Pead

[Luca Boccassi]

On Fri, 12 May 2023 at 11:40, Steve McIntyre <st...@einval.com> wrote:
>
> On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
> >On Fri, 12 May 2023 at 09:40, Steve McIntyre <st...@einval.com> wrote:
> >>
> >> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >> >
> >> >The core issue as I see it is as follows:
> >> >
> >> >- Debian has decided to support only merged-/usr, including possibly
> >> > moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> >> > as the interpreter in binaries.
> >>
> >> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> >> seen. The interpreter must *not* be changed willy-nilly.
> >
> >Nothing's happening 'willy-nilly'. We are discussing a bunch of
> >seemingly crazy options, as in, "what would _actually_ explode if we
> >do this or do that?", on this very d-devel thread. I posted a longer
> >version here some days ago:
> >
> >https://lists.debian.org/debian-gcc/2023/05/msg00030.html
>
> Oh holy fuck.
>
> You're talking about changing ABI by doing this. That *is* utterly
> crazy. No.

It's a thought experiment on a mailing list. If we can't even have
those anymore, something went very wrong somewhere.

You seem to be aware of things that wouldn't work anymore (I think?).
If you have a couple of minutes to spare, may I please ask you to
reply to that thread with such examples? I am genuinely interested in
understanding and talking about it. Thank you.

Kind regards,
Luca Boccassi

[Steve McIntyre]

On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
>On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
>>On Fri, 12 May 2023 at 09:40, Steve McIntyre <st...@einval.com> wrote:
>>>
>>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
>>> >
>>> >The core issue as I see it is as follows:
>>> >
>>> >- Debian has decided to support only merged-/usr, including possibly
>>> > moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
>>> > as the interpreter in binaries.
>>>
>>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
>>> seen. The interpreter must *not* be changed willy-nilly.
>>
>>Nothing's happening 'willy-nilly'. We are discussing a bunch of
>>seemingly crazy options, as in, "what would _actually_ explode if we
>>do this or do that?", on this very d-devel thread. I posted a longer
>>version here some days ago:
>>
>>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
>
>Oh holy fuck.
>
>You're talking about changing ABI by doing this. That *is* utterly
>crazy. No.

People have asked me to expand on this further...

I've been involved in defining ABI before, specifically for
armhf. It's not a quick and easy process. It needs buy-in from all
sides to make things work, and people *really* value the
interoperability that it enables.

The interpreter path is one of the most important parts of the ABI
spec, the bit that makes binaries compatible between all the various
stakeholders: compiler/tools people, distros, software vendors,
etc. Lots of the rest of the details downstream of this can be
changed, and people do this all the time - compare multilib to
multi-arch for example. That all works fine *so long as* the runtime
linker can be located and started OK.

Changing the interpreter path would mean moving to a Debian-specific
ABI, breaking that compatibility. Hand-waving that away with (and I
quote):

"The vast majority of distros today ship the loader in /usr/lib as
/lib is just a symlink, so it would be interoperable."

is appalling arrogance. No. You do *not* get to break ABI with that
argument. The point of the ABI spec is that *everybody* follows
it. You don't change it just because you think it'll make your life a
little easier when bootstrapping a system.

--
Steve McIntyre, Cambridge, UK. st...@einval.com

Welcome my son, welcome to the machine.

[Luca Boccassi]

The loader is still available via the old path, so external/third
party/local/other software works unchanged. This should negatively
only affect our 1st party packages, when running on a non-merged
distro.
And are _all_ our packages really 100% compatible with other distros
at all? Are they even supposed to be?

For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
machine, when I try to run it, it fails:

root@focal:/tmp# wget
http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
--2023-05-12 12:46:17--
http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27572 (27K) [application/vnd.debian.binary-package]
Saving to: 'efibootmgr_17-2_amd64.deb'

efibootmgr_17-2_amd64.deb
100%[===============================================>] 26.93K
--.-KB/s in 0.04s

2023-05-12 12:46:17 (740 KB/s) - 'efibootmgr_17-2_amd64.deb' saved [27572/27572]

root@focal:/tmp# dpkg -x efibootmgr_17-2_amd64.deb ebm
root@focal:/tmp# ./ebm/bin/efibootmgr
./ebm/bin/efibootmgr: /lib/x86_64-linux-gnu/libc.so.6: version
`GLIBC_2.34' not found (required by ./ebm/bin/efibootmgr)

Should I file a severity: serious bug against efibootmgr because it is
not interoperable?

The answer is obviously not, because it would be absurd to expect a
binary compiled against libraries from one distro to "just work" on an
entirely different distro. Glibc itself is not forward compatible, and
is allowed to add new symbols, that are not present in older versions,
and packages are allowed to depend on them. Aren't those also ABI
breakages? What about all the libraries that bump soname? What about
binaries that rely on newer kernel interfaces, or IPC interfaces?

So, what I am asking is, what actual, real difference does it make if,
by default (and with an override available for example), packages
built on Debian for Debian record the ld path to point to its (actual)
location on Debian, via say a compiler spec file that is injected in a
deb build?
There very likely is some real difference and impact, and I am
genuinely and honestly asking what it could be. If nothing else, it's
an interesting topic, even if likely nothing comes out of it.

> Changing the interpreter path would mean moving to a Debian-specific
> ABI, breaking that compatibility. Hand-waving that away with (and I
> quote):
>
> "The vast majority of distros today ship the loader in /usr/lib as
> /lib is just a symlink, so it would be interoperable."
>
> is appalling arrogance. No. You do *not* get to break ABI with that
> argument. The point of the ABI spec is that *everybody* follows
> it. You don't change it just because you think it'll make your life a
> little easier when bootstrapping a system.

AFAIK there are at least 3 distros where the default interpreter path
is changed to follow distro-specific customizations: Gentoo, Nix,
Guix. So evidently, some people *do* get to "break ABI", and not
everybody follows it. So why can't we at least _talk_ about it, pros
and cons, advantages and problems, without the tones of the discussion
needlessly escalating?

Kind regards,
Luca Boccassi

[Simon Richter]

Hi,

On 5/12/23 02:51, Luca Boccassi wrote:

> Or alternatively, we can establish that a documentation/post-facto
> approach is enough for derivatives, and then that's valid for all
> changes and transitions.

For the context of this bug, the notice *is* the after-the-fact
documentation of an interface change, i.e. the bare minimum.

It would have been better to get input from derived distributions about
this interface change before it happened, but given that the interface
change was not caused by a change in dpkg code, but by the introduction
of the usrmerge package, there was not much of a remedy available then.

Simon

[Luca Boccassi]

Not really? This notice is new, and suggests something that will make
the downstream irreparably incompatible with Debian, which I thought
was a big no-no.
If negatively affecting downstreams is a problem, this needs to be
reverted. If it's enough to say somewhere else "actually, that is bad
advice, if you follow it you are on your own", then that's fine too by
me, but then it's fine for other changes as well.

Kind regards,
Luca Boccassi

[Steve McIntyre]

So why the hell do you want to break this in the first place? Does a
symlink in the "wrong" place offend you for some reason? For that you
want to change a core assumption in *every single binary* in Debian?
Believe me, I've been here in the past when we made changes in armhf
to accommodate earlier mistakes. That was just for one
architecture. What possible benefit do you see in this change?

>And are _all_ our packages really 100% compatible with other distros
>at all? Are they even supposed to be?
>
>For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
>machine, when I try to run it, it fails:
>
>root@focal:/tmp# wget
>http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
>--2023-05-12 12:46:17--
>http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
>Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
>Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... connected.
>HTTP request sent, awaiting response... 200 OK
>Length: 27572 (27K) [application/vnd.debian.binary-package]
>Saving to: 'efibootmgr_17-2_amd64.deb'
>
>efibootmgr_17-2_amd64.deb
>100%[===============================================>] 26.93K
>--.-KB/s in 0.04s
>
>2023-05-12 12:46:17 (740 KB/s) - 'efibootmgr_17-2_amd64.deb' saved [27572/27572]
>
>root@focal:/tmp# dpkg -x efibootmgr_17-2_amd64.deb ebm
>root@focal:/tmp# ./ebm/bin/efibootmgr
>./ebm/bin/efibootmgr: /lib/x86_64-linux-gnu/libc.so.6: version
>`GLIBC_2.34' not found (required by ./ebm/bin/efibootmgr)
>
>Should I file a severity: serious bug against efibootmgr because it is
>not interoperable?

You're wilfully missing the point, and you know it.

>The answer is obviously not, because it would be absurd to expect a
>binary compiled against libraries from one distro to "just work" on an
>entirely different distro. Glibc itself is not forward compatible, and
>is allowed to add new symbols, that are not present in older versions,
>and packages are allowed to depend on them. Aren't those also ABI
>breakages? What about all the libraries that bump soname? What about
>binaries that rely on newer kernel interfaces, or IPC interfaces?
>
>So, what I am asking is, what actual, real difference does it make if,
>by default (and with an override available for example), packages
>built on Debian for Debian record the ld path to point to its (actual)
>location on Debian, via say a compiler spec file that is injected in a
>deb build?
>There very likely is some real difference and impact, and I am
>genuinely and honestly asking what it could be. If nothing else, it's
>an interesting topic, even if likely nothing comes out of it.

I have better things to do than argue about this. I refuse to engage
with this right now. You're talking about breaking things for *no*
discernible benefit that I've seen any discussion about.

>> Changing the interpreter path would mean moving to a Debian-specific
>> ABI, breaking that compatibility. Hand-waving that away with (and I
>> quote):
>>
>> "The vast majority of distros today ship the loader in /usr/lib as
>> /lib is just a symlink, so it would be interoperable."
>>
>> is appalling arrogance. No. You do *not* get to break ABI with that
>> argument. The point of the ABI spec is that *everybody* follows
>> it. You don't change it just because you think it'll make your life a
>> little easier when bootstrapping a system.
>
>AFAIK there are at least 3 distros where the default interpreter path
>is changed to follow distro-specific customizations: Gentoo, Nix,
>Guix. So evidently, some people *do* get to "break ABI", and not
>everybody follows it. So why can't we at least _talk_ about it, pros
>and cons, advantages and problems, without the tones of the discussion
>needlessly escalating?

Again: *why* do you want to do this? For all the value here, should we
also discuss switching to PE-COFF from ELF for our binaries? That's
more commonly used...

--
Steve McIntyre, Cambridge, UK. st...@einval.com

Is there anybody out there?

[Luca Boccassi]

As it was mentioned on the list, because it makes bootstrapping
self-contained, that's a real and concrete benefit that some
developers like Helmut care greatly about, and that's why we are
talking about it. To me, it sounds very attractive to have a
self-contained and canonicalized distro-wide configuration. If the
canonical location where certain files are stored in /usr/bin or
/usr/lib, it seems sensible to me to configure Debian software to look
for it where we actually put it, while maintaining compatibility for
external/local software so that it keeps working. And it is also
unclear so far what would outright break - the externally defined ABI
in terms of where the loader can be accessed at, would still be
respected. Hence why questions are being asked. Nobody's being forced
to do anything, this is just a discussion.

> >And are _all_ our packages really 100% compatible with other distros
> >at all? Are they even supposed to be?
> >
> >For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
> >machine, when I try to run it, it fails:
> >
> >root@focal:/tmp# wget
> >http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
> >--2023-05-12 12:46:17--
> >http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
> >Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
> >Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... connected.
> >HTTP request sent, awaiting response... 200 OK
> >Length: 27572 (27K) [application/vnd.debian.binary-package]
> >Saving to: 'efibootmgr_17-2_amd64.deb'
> >
> >efibootmgr_17-2_amd64.deb
> >100%[===============================================>] 26.93K
> >--.-KB/s in 0.04s
> >
> >2023-05-12 12:46:17 (740 KB/s) - 'efibootmgr_17-2_amd64.deb' saved [27572/27572]
> >
> >root@focal:/tmp# dpkg -x efibootmgr_17-2_amd64.deb ebm
> >root@focal:/tmp# ./ebm/bin/efibootmgr
> >./ebm/bin/efibootmgr: /lib/x86_64-linux-gnu/libc.so.6: version
> >`GLIBC_2.34' not found (required by ./ebm/bin/efibootmgr)
> >
> >Should I file a severity: serious bug against efibootmgr because it is
> >not interoperable?
>
> You're wilfully missing the point, and you know it.

I'm trying to determine where the boundary lies. What are the
expectations for interoperability? Are all executables expected to be
self-contained? Can they rely on external config that is guaranteed to
be there on Debian but not elsewhere? Can they rely on external
libraries that are guaranteed to be there on Debian but not elsewhere?
Can they rely on external symlinks that are guaranteed to be there on
Debian but not elsewhere? How is this all defined, and most
importantly, what are the actual use cases being covered?

> >The answer is obviously not, because it would be absurd to expect a
> >binary compiled against libraries from one distro to "just work" on an
> >entirely different distro. Glibc itself is not forward compatible, and
> >is allowed to add new symbols, that are not present in older versions,
> >and packages are allowed to depend on them. Aren't those also ABI
> >breakages? What about all the libraries that bump soname? What about
> >binaries that rely on newer kernel interfaces, or IPC interfaces?
> >
> >So, what I am asking is, what actual, real difference does it make if,
> >by default (and with an override available for example), packages
> >built on Debian for Debian record the ld path to point to its (actual)
> >location on Debian, via say a compiler spec file that is injected in a
> >deb build?
> >There very likely is some real difference and impact, and I am
> >genuinely and honestly asking what it could be. If nothing else, it's
> >an interesting topic, even if likely nothing comes out of it.
>
> I have better things to do than argue about this. I refuse to engage
> with this right now. You're talking about breaking things for *no*
> discernible benefit that I've seen any discussion about.

It is entirely up to you of course, however just saying "things would
break" without mentioning what or how does not help to further our
understanding of the matter at hand. So far reactions are either one
of "what??! weird, but it could actually work!" and "what??! no,
because no!". In the absence of somebody explaining "there's use case
X that we support as per policy/decision/custom/workflow Y that would
break because of Z" I'm finding it very difficult to come to the
conclusion that this would actually be problematic, in practice. I am
here to be enlightened.

> >> Changing the interpreter path would mean moving to a Debian-specific
> >> ABI, breaking that compatibility. Hand-waving that away with (and I
> >> quote):
> >>
> >> "The vast majority of distros today ship the loader in /usr/lib as
> >> /lib is just a symlink, so it would be interoperable."
> >>
> >> is appalling arrogance. No. You do *not* get to break ABI with that
> >> argument. The point of the ABI spec is that *everybody* follows
> >> it. You don't change it just because you think it'll make your life a
> >> little easier when bootstrapping a system.
> >
> >AFAIK there are at least 3 distros where the default interpreter path
> >is changed to follow distro-specific customizations: Gentoo, Nix,
> >Guix. So evidently, some people *do* get to "break ABI", and not
> >everybody follows it. So why can't we at least _talk_ about it, pros
> >and cons, advantages and problems, without the tones of the discussion
> >needlessly escalating?
>
> Again: *why* do you want to do this? For all the value here, should we
> also discuss switching to PE-COFF from ELF for our binaries? That's
> more commonly used...

Actually I'd prefer Mach-O - its native graceful support for optional
dependencies (dlopen-like) is really nice! But I digress, and "why" is
explained above and in earlier mails.

Kind regards,
Luca Boccassi

[Luca Boccassi]

On Sun, 14 May 2023 at 22:37, Josh Triplett <jo...@joshtriplett.org> wrote:
>
> On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:

> > The loader is still available via the old path, so external/third
> > party/local/other software works unchanged. This should negatively
> > only affect our 1st party packages, when running on a non-merged
> > distro.
> > And are _all_ our packages really 100% compatible with other distros
> > at all? Are they even supposed to be?
>

> People build things on Debian that are not Debian packages. People
> compile binaries on Debian, and expect them to work on any system that
> has sufficiently new libraries.
>
> This is *not* about Debian packages failing to work on other
> distributions; this is about *software compiled on Debian* faliing to
> work in other environments.

Why would "software compiled on Debian" fail to work in other
environments? Well, there are many reasons actually, people invented
containers/flatpaks/snaps exactly for that reason. But nothing do with
anything discussed here though, as far as I can tell?

> If you build a dynamically linked binary that only depends on glibc, you
> can expect it to be reasonably portable, to any system that uses glibc
> and has a sufficiently new version.
>
> Debian stable is, in fact, one of the common environments people use to
> compile binaries for distribution.

"sufficiently new version" is doing a lot of work there. We have
shlibs dependencies for a reason. In fact, the most common environment
used to distribute binaries is the EOL Ubuntu 16.04, slowly switching
to the soon-to-be-EOL 18.04. glibc is not forward-compatible, new
symbols are added all the time and are used all the time, and you
don't jump on a brand new distribution to do that kind of work, that
would be self-defeating.

> > So, what I am asking is, what actual, real difference does it make if,
> > by default (and with an override available for example), packages
> > built on Debian for Debian record the ld path to point to its (actual)
> > location on Debian, via say a compiler spec file that is injected in a
> > deb build?
>

> Making binaries built *on* Debian different than binaries built *for*
> Debian would introduce a needless additional source of complexity,
> compared to just compiling code the same way in both cases.

That's not how it works today already. There are several significant
differences between just running "gcc sources.c" and building a
package via debhelper on a buildd, they are not the same thing at all,
and they haven't been since forever, there are dozens of
compiler/linker options that the Debian package build environment
sets. Or will you now also ask the distribution to rollback multiarch,
hardening, SOURCE_DATE_EPOCH, -ffile-prefix-map and all the other
reproducibility options, and so on? These and many more are all
"needless additional sources of complexity, compared to just compiling
code the same way" too. Because guess what, there are people who
couldn't possibly care less about
multiarch/security/reproducibility/etc, and there will also be a
subset of users who considers a subset of those compiler options
"needless". So are you going to push to have all of that reverted? And
also are you going to propose a Policy change that forbids adding any
new compiler/linker option to the package build process?

> To frame this in different terms: consider that one of the major goals
> of systemd has been to harmonize across distributions and eliminate
> needless variations that don't serve much actual purpose (e.g.
> variations in config file paths for the same config file). Consider how
> much effort systemd went to work with distributions, understand and deal
> with the *important* variations, and try to convince them to abandon the
> *unimportant* variations. Now imagine if someone came along and said
> "let's patch systemd to put unit files in /purple/; it'll work with
> everything in our distribution".

Pretty sure the Nix folks are already doing pretty much that. And if
it works for their case, all the power to them.

> Or, imagine if someone said "let's inject an argument to gzip, only for
> building the .gz files sihpped in our packages of course, to modify the
> gzip header and remove a few of the extraneous additional fields; it'll
> be fine, because we've patched our gzip to parse it"

Not really related, archives are _intended_ to be opened anywhere for
any reason. Do you have any actual related use case that would no
longer work? Because that would be the easiest and most convincing
counter-factual that could be provided.

> The x86-64 ABI is set. Feel free to make the case to the next
> architecture designer that their new ABI should have the dynamic linker
> in `/usr/lib`. That would *not* have the same downsides, as long as
> everyone agrees on a path.

In practice it is not, though. There are other distributions that
change PT_INTERP for their own purposes, they've already been listed
in this thread. And I am still not hearing any concrete, factual use
case that would be impaired by such a change. I'm beginning to
seriously think there aren't any? Is that really the case?

Kind regards,
Luca Boccassi

[Peter Pentchev]

On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
> On Sun, 14 May 2023 at 22:37, Josh Triplett <jo...@joshtriplett.org> wrote:
> >
> > On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> > > The loader is still available via the old path, so external/third
> > > party/local/other software works unchanged. This should negatively
> > > only affect our 1st party packages, when running on a non-merged
> > > distro.
> > > And are _all_ our packages really 100% compatible with other distros
> > > at all? Are they even supposed to be?
> >
> > People build things on Debian that are not Debian packages. People
> > compile binaries on Debian, and expect them to work on any system that
> > has sufficiently new libraries.
> >
> > This is *not* about Debian packages failing to work on other
> > distributions; this is about *software compiled on Debian* faliing to
> > work in other environments.
>
> Why would "software compiled on Debian" fail to work in other
> environments? Well, there are many reasons actually, people invented
> containers/flatpaks/snaps exactly for that reason. But nothing do with
> anything discussed here though, as far as I can tell?

If an ELF executable, compiled on Debian, records its interpreter as
/usr/lib/ld-linux.so.2, what happens when one tries to run it on
a non-usr-merged system? Even one with a recent enough glibc version?

G'luck,
Peter

--
Peter Pentchev ro...@ringlet.net ro...@debian.org p...@storpool.com
PGP key: http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115 C354 651E EFB0 2527 DF13

[Russ Allbery]

Luca Boccassi <bl...@debian.org> writes:

> Why would "software compiled on Debian" fail to work in other
> environments? Well, there are many reasons actually, people invented
> containers/flatpaks/snaps exactly for that reason. But nothing do with
> anything discussed here though, as far as I can tell?

My understanding is that this specific thread is about a mention that we
may want to change PT_INTERP to /usr/lib64/ld-linux-x86-64.so.2 or some
similar path.

If PT_INTERP points to a file that doesn't exist, the program is obviously
not going to run. The Linux x86_64 ABI says it must point to
/lib64/ld-linux-x86-64.so.2. If we build binaries that use some other
value, then we are not building ABI-compliant binaries and they may not
run on other systems. This is the whole point of an ABI.

An obvious specific example of such a system would be one that didn't
merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other
path, but that's just one obvious example. There may be others; the whole
point of an ABI is that you do not change things like this, not even if
you can't personally imagine why your change wouldn't be harmful. There's
a whole process for changing an ABI that involves everyone else agreeing
as well, and unless one goes through that process, the ABI is what it is.
Debian not building ABI-compliant binaries would be highly surprising.

Incidentally, that remains true even if we only do that in distribution
packages. I certainly have copied binaries from a Debian package to other
Linux systems before for various reasons and expected them to run. Sure,
this might not work for other reasons outside of our control, but that's
no reason to be gratuitously incompatible by breaking the ABI,
particularly for what seem to be annoyances of our own creation with known
workarounds.

--
Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>

[Luca Boccassi]

On Mon, 15 May 2023 at 01:07, Peter Pentchev <ro...@ringlet.net> wrote:
>
> On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
> > On Sun, 14 May 2023 at 22:37, Josh Triplett <jo...@joshtriplett.org> wrote:
> > >
> > > On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> > > > The loader is still available via the old path, so external/third
> > > > party/local/other software works unchanged. This should negatively
> > > > only affect our 1st party packages, when running on a non-merged
> > > > distro.
> > > > And are _all_ our packages really 100% compatible with other distros
> > > > at all? Are they even supposed to be?
> > >
> > > People build things on Debian that are not Debian packages. People
> > > compile binaries on Debian, and expect them to work on any system that
> > > has sufficiently new libraries.
> > >
> > > This is *not* about Debian packages failing to work on other
> > > distributions; this is about *software compiled on Debian* faliing to
> > > work in other environments.
> >
> > Why would "software compiled on Debian" fail to work in other
> > environments? Well, there are many reasons actually, people invented
> > containers/flatpaks/snaps exactly for that reason. But nothing do with
> > anything discussed here though, as far as I can tell?
>
> If an ELF executable, compiled on Debian, records its interpreter as
> /usr/lib/ld-linux.so.2, what happens when one tries to run it on
> a non-usr-merged system? Even one with a recent enough glibc version?

This is not about locally built ELF executables, no difference in those.

Kind regards,
Luca Boccassi

[Luca Boccassi]

On Mon, 15 May 2023 at 01:14, Russ Allbery <r...@debian.org> wrote:
>
> Luca Boccassi <bl...@debian.org> writes:
>
> > Why would "software compiled on Debian" fail to work in other
> > environments? Well, there are many reasons actually, people invented
> > containers/flatpaks/snaps exactly for that reason. But nothing do with
> > anything discussed here though, as far as I can tell?
>
> My understanding is that this specific thread is about a mention that we
> may want to change PT_INTERP to /usr/lib64/ld-linux-x86-64.so.2 or some
> similar path.
>
> If PT_INTERP points to a file that doesn't exist, the program is obviously
> not going to run. The Linux x86_64 ABI says it must point to
> /lib64/ld-linux-x86-64.so.2. If we build binaries that use some other
> value, then we are not building ABI-compliant binaries and they may not
> run on other systems. This is the whole point of an ABI.

This is not about locally compiled software or such, only packages
(and maybe even just a subset of them).

> An obvious specific example of such a system would be one that didn't
> merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other
> path, but that's just one obvious example. There may be others; the whole
> point of an ABI is that you do not change things like this, not even if
> you can't personally imagine why your change wouldn't be harmful. There's
> a whole process for changing an ABI that involves everyone else agreeing
> as well, and unless one goes through that process, the ABI is what it is.
> Debian not building ABI-compliant binaries would be highly surprising.

That's self-evidently not true, as there are other distributions where
that already happens, it's been already mentioned. Besides, we are not
talking about sacred religious texts - the point is making things
work. If they do, is it _really_ non-compliant/incompatible?

> Incidentally, that remains true even if we only do that in distribution
> packages. I certainly have copied binaries from a Debian package to other
> Linux systems before for various reasons and expected them to run. Sure,
> this might not work for other reasons outside of our control, but that's
> no reason to be gratuitously incompatible by breaking the ABI,
> particularly for what seem to be annoyances of our own creation with known
> workarounds.

Thanks, that's the first actual real example mentioned so far. And
it's an interesting one: taking a $random Debian package and using it
on a completely different, non-Debian system. Is that a supported use
case? If so, does that mean that I can go ahead and raise a Severity:
serious bug on any package that doesn't work in such a way?

Kind regards,
Luca Boccassi

[Steve McIntyre]

On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:

The ABI has been agreed and set down in documentation that *just
about* everybody has been following since its inception. This includes
the most basic set of definitions of what an x86-64 program must look
like, including the interpreter path. If this path is changed, then
*at the most basic level* we'd be making programs that are not valid
by the ABI we've agreed to. This is an *external interface contract*,
not something we should ever consider changing without significant
cross- and inter-project discussion.

Pointing at gentoo or nixos as examples of projects that have decided
to break compatibility doesn't cut it, I'm afraid. They're well known
for changing fundamental things around Linux and (basically) not
caring about interoperability. That attitude is *not* Debian's.

--
Steve McIntyre, Cambridge, UK. st...@einval.com

"The problem with defending the purity of the English language is that
English is about as pure as a cribhouse whore. We don't just borrow words; on
occasion, English has pursued other languages down alleyways to beat them
unconscious and rifle their pockets for new vocabulary." -- James D. Nicoll

[Steve McIntyre]

I'm *trying* to assume good faith here, but I'm running out of energy
to do so.

Russ has described copying *binaries* out of packages and running them
elsewhere. I've done that too, from time to time. This is one of the
things made possible by the ABI contract being followed.

You are the one proposing to break that contract, thereby
*guaranteeing* this will fail on systems where otherwise it could
work. I think the onus is on *you* to justify why this is a valid and
useful thing to do. Your apparent lack of care for agreed standards
here is horrifying.

--
Steve McIntyre, Cambridge, UK. st...@einval.com

"We're the technical experts. We were hired so that management could
ignore our recommendations and tell us how to do our jobs." -- Mike Andrews

[Russ Allbery]

Luca Boccassi <bl...@debian.org> writes:

> That's self-evidently not true, as there are other distributions where
> that already happens, it's been already mentioned.

You've mentioned this a couple of times but I don't think I've seen the
message where the details were explained. Maybe this was only in your
message posted to debian-gcc, which wasn't part of this thread? (It's
also possible that I just missed it somewhere.)

That message only mentions GUIX, which I don't know very much about, but
my recollection (maybe wrong?) is that it's a NIX variant that is doing
special tricks to support immutable package trees and
roll-forward/roll-back upgrades. I can see why that might be motivation
to build incompatible binaries in order to preserve some other invariant
they're trying for as the point of their distribution (in particular, I
suspect they're pinning binaries to a specific version of the dynamic
loader as part of the whole immutable tree strategy). That's a perfectly
fine decision in a distribution that's trying to do something very
different and is a bit of a science experiment, but I don't think that
describes Debian.

(Also, no slight on the GUIX folks, but GUIX is not exactly an, uh, major
player in Linux distributions, and I'm not sure how much they care about
compatibility with anyone else.)

> Besides, we are not talking about sacred religious texts - the point is
> making things work. If they do, is it _really_
> non-compliant/incompatible?

I understand your point in making this argument, but please understand
that this sort of willingness to change things if one didn't think they
would cause problems didn't work very well, and was part of what led to
the development of standardized ABIs in the first place. Those of us who
have been around longer than Linux have ABIs have a bit of a strong
reaction here (I think this is also what you're seeing from Steve),
because we remember the bad old days. I still have compatibility code
around to handle the fact that gcc on IRIX miscompiled calls to inet_ntoa
because gcc didn't correctly implement the IRIX ABI.

People are very bad at judging whether their new idea would be *really*
incompatible. This is why these days everyone tries to follow the ABI
pretty closely.

And in any case, changing PT_INTERP is trivially and obviously
incompatible; the binary will simply not run on a system that doesn't have
that path. So it's not like we have to carefully judge nuance here. Your
argument, so far as I can tell, is basically "but no one will ever want to
run those binaries on a non-/usr-merged system anyway," which is basically
conceding the incompatibility point since the ABI doesn't require merged
/usr.

There's also some other history here: Debian is not super-happy with the
PT_INTERP because ideally we'd prefer it use a path compatible with our
multiarch approach. I believe we raised that and no one had any interest
in trying to change anything, so we lived with the limitations that
creates. (And I think that was the right decision.)

> Thanks, that's the first actual real example mentioned so far. And it's
> an interesting one: taking a $random Debian package and using it on a
> completely different, non-Debian system. Is that a supported use case?
> If so, does that mean that I can go ahead and raise a Severity: serious
> bug on any package that doesn't work in such a way?

I feel like you're distorting my argument here to try to make some sort of
slippery slope argument, and it's coming across as possibly more
aggressive than you had intended.

The world does not divide neatly into supported and unsupported use cases.
There are a lot of things I do to computers that I expect to work in some
situations but not in others. That includes, say, having a Debian chroot
on some other OS and running binaries from that chroot without going into
the chroot. Often that will absolutely not work. Sometimes it will work,
and it's convenient that it will work for some obscure recovery situations
or other weird one-off use cases. I've also copied files from working
systems to broken systems running a different distribution before, and
there's a list of caveats as long as my arm, but sometimes it's a quick
fix for something.

But mostly my reaction is because breaking the ABI is a Really Big Deal.
Constructing the Linux ABI and getting the details actually published was
a hard-fought, arduous endeavor. I doubt anyone enjoyed it; it's the sort
of annoying compatibility work that provides tons of small, subtle
benefits and takes a great deal of truly thankless work, and people often
don't realize all the tiny ways that it has made the world a better place,
or the range of weird compatibility problems that can arise from messing
with it. Diverging from it is not something to do lightly, precisely
*because* it's often extremely difficult to understand what the effects
could be or what might break.

While I appreciate how it would make bootstrapping Debian somewhat more
convenient in this case, I am unconvinced that this is a good enough
reason to undermine one of the foundations of what makes Linux a
collective and fairly mutually compatible ecosystem.

I realize it's not necessarily obvious that changing PT_INTERP for some
binaries is a big deal, in part because it's not even obvious that it's
part of the ABI. That's why people who are familiar with the ABI process
are jumping in to say "please don't touch that, this is a big deal to us."

[Luca Boccassi]

On Mon, 15 May 2023 at 02:26, Russ Allbery <r...@debian.org> wrote:
>
> Luca Boccassi <bl...@debian.org> writes:
>
> > That's self-evidently not true, as there are other distributions where
> > that already happens, it's been already mentioned.
>
> You've mentioned this a couple of times but I don't think I've seen the
> message where the details were explained. Maybe this was only in your
> message posted to debian-gcc, which wasn't part of this thread? (It's
> also possible that I just missed it somewhere.)
>
> That message only mentions GUIX, which I don't know very much about, but
> my recollection (maybe wrong?) is that it's a NIX variant that is doing
> special tricks to support immutable package trees and
> roll-forward/roll-back upgrades. I can see why that might be motivation
> to build incompatible binaries in order to preserve some other invariant
> they're trying for as the point of their distribution (in particular, I
> suspect they're pinning binaries to a specific version of the dynamic
> loader as part of the whole immutable tree strategy). That's a perfectly
> fine decision in a distribution that's trying to do something very
> different and is a bit of a science experiment, but I don't think that
> describes Debian.
>
> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh, major
> player in Linux distributions, and I'm not sure how much they care about
> compatibility with anyone else.)

This is a counter-example to confute the assertion that *everybody*
does the same thing, which has been made multiple times. I'm not sure
whether it's an experiment or not, I mean if you ask their
users/developers I think they'd tell you it's very much production
ready, but it is largely irrelevant: it exists, and that's the only
reason why it was mentioned, as it shows that it is _possible_ to do
that and be a working distribution. Doesn't imply it's automatically
desirable, but that was not the intention.

> > Besides, we are not talking about sacred religious texts - the point is
> > making things work. If they do, is it _really_
> > non-compliant/incompatible?
>
> I understand your point in making this argument, but please understand
> that this sort of willingness to change things if one didn't think they
> would cause problems didn't work very well, and was part of what led to
> the development of standardized ABIs in the first place. Those of us who
> have been around longer than Linux have ABIs have a bit of a strong
> reaction here (I think this is also what you're seeing from Steve),
> because we remember the bad old days. I still have compatibility code
> around to handle the fact that gcc on IRIX miscompiled calls to inet_ntoa
> because gcc didn't correctly implement the IRIX ABI.
>
> People are very bad at judging whether their new idea would be *really*
> incompatible. This is why these days everyone tries to follow the ABI
> pretty closely.
>
> And in any case, changing PT_INTERP is trivially and obviously
> incompatible; the binary will simply not run on a system that doesn't have
> that path. So it's not like we have to carefully judge nuance here. Your
> argument, so far as I can tell, is basically "but no one will ever want to
> run those binaries on a non-/usr-merged system anyway," which is basically
> conceding the incompatibility point since the ABI doesn't require merged
> /usr.

Not quite: my argument is that binaries from these packages are not
intended and not required to be ran on non-Debian systems, so there's
no incompatibility introduced in the first place - everything still
works where it is supposed to, exactly as it was before.

> There's also some other history here: Debian is not super-happy with the
> PT_INTERP because ideally we'd prefer it use a path compatible with our
> multiarch approach. I believe we raised that and no one had any interest
> in trying to change anything, so we lived with the limitations that
> creates. (And I think that was the right decision.)
>
> > Thanks, that's the first actual real example mentioned so far. And it's
> > an interesting one: taking a $random Debian package and using it on a
> > completely different, non-Debian system. Is that a supported use case?
> > If so, does that mean that I can go ahead and raise a Severity: serious
> > bug on any package that doesn't work in such a way?
>
> I feel like you're distorting my argument here to try to make some sort of
> slippery slope argument, and it's coming across as possibly more
> aggressive than you had intended.

No aggression intended whatsoever, sorry if it appeared that way. I am
trying to understand what the rules are.

> The world does not divide neatly into supported and unsupported use cases.
> There are a lot of things I do to computers that I expect to work in some
> situations but not in others. That includes, say, having a Debian chroot
> on some other OS and running binaries from that chroot without going into
> the chroot. Often that will absolutely not work. Sometimes it will work,
> and it's convenient that it will work for some obscure recovery situations
> or other weird one-off use cases. I've also copied files from working
> systems to broken systems running a different distribution before, and
> there's a list of caveats as long as my arm, but sometimes it's a quick
> fix for something.

Vast, vast majority of binaries from existing packages will already
not work out of the box in that use case though. Why are the existing
incompatibilities allowed, and this isn't?

> But mostly my reaction is because breaking the ABI is a Really Big Deal.
> Constructing the Linux ABI and getting the details actually published was
> a hard-fought, arduous endeavor. I doubt anyone enjoyed it; it's the sort
> of annoying compatibility work that provides tons of small, subtle
> benefits and takes a great deal of truly thankless work, and people often
> don't realize all the tiny ways that it has made the world a better place,
> or the range of weird compatibility problems that can arise from messing
> with it. Diverging from it is not something to do lightly, precisely
> *because* it's often extremely difficult to understand what the effects
> could be or what might break.

I am afraid I am a bit more "pragmatic" than that. I am very
interested in what works and what doesn't. Whether it conforms to the
letter of the Sacred Ancient Texts... interesting for sure, but
secondary.

> While I appreciate how it would make bootstrapping Debian somewhat more
> convenient in this case, I am unconvinced that this is a good enough
> reason to undermine one of the foundations of what makes Linux a
> collective and fairly mutually compatible ecosystem.
>
> I realize it's not necessarily obvious that changing PT_INTERP for some
> binaries is a big deal, in part because it's not even obvious that it's
> part of the ABI. That's why people who are familiar with the ABI process
> are jumping in to say "please don't touch that, this is a big deal to us."

Let me ask a simple policy question: why do I have to abide to your
uncodified, unwritten use case of running a Debian program from a
Debian package from outside a Debian chroot on a foreign, unmodified
non-Debian system, but you don't have to abide by my uncodified,
unwritten use case of running a Debian program on a Debian system with
only a Debian /usr? And also, why it is only this change that has to
abide to such use case, and the myriad of cases that cannot possibly
work in such a setup are given a free pass (I mean I'm pretty sure the
only executables that are guaranteed to work as you mention are golang
and rustlang, where everything is statically compiled, everything else
is already rc-buggy by that standard)? Isn't this the very reason we
have a Policy for, so that we don't get to cherry-pick arbitrary use
cases to block things we don't like? Are you really sure we want to be
in a place where anybody can bring up an out-of-policy use case as a
valid reason to block something they don't like?

Again, I am fine if we want to say that Debian-specific changes and
Debianisms are bad and we cannot do anything that jeopardises
interoperability, cross-distribution harmony and mutual compatibility.
But let's do that then, and start by writing it down in Policy so that
it applies fairly and equally to all cases.

Kind regards,
Luca Boccassi

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Steve McIntyre (2023-05-15 02:54:02)

> On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
> >On Sun, 14 May 2023 at 22:37, Josh Triplett <jo...@joshtriplett.org> wrote:
> >
> >> The x86-64 ABI is set. Feel free to make the case to the next
> >> architecture designer that their new ABI should have the dynamic linker
> >> in `/usr/lib`. That would *not* have the same downsides, as long as
> >> everyone agrees on a path.
> >
> >In practice it is not, though. There are other distributions that
> >change PT_INTERP for their own purposes, they've already been listed
> >in this thread. And I am still not hearing any concrete, factual use
> >case that would be impaired by such a change. I'm beginning to
> >seriously think there aren't any? Is that really the case?
>
> The ABI has been agreed and set down in documentation that *just
> about* everybody has been following since its inception. This includes
> the most basic set of definitions of what an x86-64 program must look
> like, including the interpreter path. If this path is changed, then
> *at the most basic level* we'd be making programs that are not valid
> by the ABI we've agreed to. This is an *external interface contract*,
> not something we should ever consider changing without significant
> cross- and inter-project discussion.
>
> Pointing at gentoo or nixos as examples of projects that have decided
> to break compatibility doesn't cut it, I'm afraid. They're well known
> for changing fundamental things around Linux and (basically) not caring about
> interoperability. That attitude is *not* Debian's.

me and Luca have different ideas about how bootstrapping a Debian chroot should
look like and I don't want to make an argument *for* changing PT_INTERP here as
I think that keeping compatibility with others by following ABI is a good thing
and because I think (and hope -- but Helmut is doing that analysis right now)
that the debootstrap problem can be solved in a way I envision without changing
PT_INTERP. But what I do not understand about the argument against Luca's
proposal is:

Obviously, with Luca's proposal, binaries from packages built with a different
dynamic linker path in them would not work on distributions without merged-/usr
symlinks. But if the property of stuff from Debian being able to run on
non-Debian non-merged-/usr systems is an important one, then why was it okay to
have merged-/usr as the default? Because with merged-/usr we already changed
the interface contract for a lot of things because now binaries and libraries
can also be found at other locations than on non-merged-/usr systems. A script
with a /usr/bin/bash shebang built on and for Debian will not work on a system
without the symlinks.

So did we not years ago decide, that the result of the "cross- and
inter-project discussion" is, that everybody is going merged-/usr and that's
why we need it too and that's why it is okay to build a system where binaries
and scripts built for it just may not run on those other systems that do not do
it? With merged-/usr we already *did* "change fundamental things around" for
reasons that are really not clear to me (but which i do not want to discuss
here) and as a result did not "care about interoperability" (with those who do
not also adopt it). In my own Debian work I so far only got extra work because
of merged-/usr and I do not see the benefits (yet) and I was hoping that

"changing fundamental things around Linux and (basically) not caring about

interoperability" was *not* Debian's attitude but alas here we are.

So have we not already burned the bridges to the non-merged-/usr world? Why was
it okay back then to say "we can make this change because all other important
players are doing merged-/usr so we can/have to as well". And now in the
PT_INTERP discussion somehow we care again about those systems? I thought we
already had the "cross- and inter-project discussion" about merged-/usr and
because the result was "yes, go for it" we did it too. But if that is the case,
why do we now care for a subset of the interoperability problems caused by
merged-/usr for systems that don't have it?

As I said, I don't care much about the PT_INTERP value but I don't understand
yet, why this argument about interoperability with non-merged-/usr systems is
working now but it didn't wasn't enough to stop another very fundamental change
in how we build a Linux distro.

Thanks!

cheers, josch

[Bdale Garbee]

Merged-/usr seems to me to have brought great pain with no discernable benefit to Debian so far, and I at least have completely lost the thread on what the point of doing it was supposed to be.  So, using it as a justification for further harm to user and system expectations isn't compelling.

Bdale

[Luca Boccassi]

On Mon, 15 May 2023 at 13:51, Bdale Garbee <bd...@gag.com> wrote:
>
> Merged-/usr seems to me to have brought great pain with no discernable benefit to Debian so far, and I at least have completely lost the thread on what the point of doing it was supposed to be. So, using it as a justification for further harm to user and system expectations isn't compelling.

Are you able to provide an example of such "harm"?

Kind regards,
Luca Boccassi

[Steve McIntyre]

Hey Johannes,

On Mon, May 15, 2023 at 06:48:04AM +0200, Johannes Schauer Marin Rodrigues wrote:
>Quoting Steve McIntyre (2023-05-15 02:54:02)
>>

>> Pointing at gentoo or nixos as examples of projects that have decided
>> to break compatibility doesn't cut it, I'm afraid. They're well known
>> for changing fundamental things around Linux and (basically) not caring about
>> interoperability. That attitude is *not* Debian's.
>
>me and Luca have different ideas about how bootstrapping a Debian chroot should
>look like and I don't want to make an argument *for* changing PT_INTERP here as
>I think that keeping compatibility with others by following ABI is a good thing
>and because I think (and hope -- but Helmut is doing that analysis right now)
>that the debootstrap problem can be solved in a way I envision without changing
>PT_INTERP. But what I do not understand about the argument against Luca's
>proposal is:
>
>Obviously, with Luca's proposal, binaries from packages built with a different
>dynamic linker path in them would not work on distributions without merged-/usr
>symlinks. But if the property of stuff from Debian being able to run on
>non-Debian non-merged-/usr systems is an important one, then why was it okay to
>have merged-/usr as the default? Because with merged-/usr we already changed
>the interface contract for a lot of things because now binaries and libraries
>can also be found at other locations than on non-merged-/usr systems. A script
>with a /usr/bin/bash shebang built on and for Debian will not work on a system
>without the symlinks.

Despite the massive upheavals of merged-/usr in *other* ways, it's
actually a *minor* change as far as compatibility is concerned
here. The runtime linker (aka interpreter) is responsible for
resolving symbols and finding needed libraries. So long as *that* bit
works OK, then ~everything else should follow OK. This is how it's
possible to have things work across distros: binaries don't actually
care where the libraries live, or whether they came from rpm or deb
packaging, etc.

The issue at hand here is that the interpreter path is the most basic
thing that matters for this compatibility. Break this and *nothing*
can work.

>So did we not years ago decide, that the result of the "cross- and
>inter-project discussion" is, that everybody is going merged-/usr and that's
>why we need it too and that's why it is okay to build a system where binaries
>and scripts built for it just may not run on those other systems that do not do
>it? With merged-/usr we already *did* "change fundamental things around" for
>reasons that are really not clear to me (but which i do not want to discuss
>here) and as a result did not "care about interoperability" (with those who do
>not also adopt it). In my own Debian work I so far only got extra work because
>of merged-/usr and I do not see the benefits (yet) and I was hoping that
>"changing fundamental things around Linux and (basically) not caring about
>interoperability" was *not* Debian's attitude but alas here we are.
>
>So have we not already burned the bridges to the non-merged-/usr world? Why was
>it okay back then to say "we can make this change because all other important
>players are doing merged-/usr so we can/have to as well". And now in the
>PT_INTERP discussion somehow we care again about those systems? I thought we
>already had the "cross- and inter-project discussion" about merged-/usr and
>because the result was "yes, go for it" we did it too. But if that is the case,
>why do we now care for a subset of the interoperability problems caused by
>merged-/usr for systems that don't have it?

This change is absolutely *not* needed to make merged-/usr work; if
anybody is claiming that it is, then they are not being 100% honest
with us. All the other distros doing merged-/usr have done it without
making this change, and it's also been working OK for us so far
without this change.

Breaking an agreed interface contract like this is axiomatically
*wrong* and will hurt us and the rest of the Linux ecosystem.

--
Steve McIntyre, Cambridge, UK. st...@einval.com

"You can't barbecue lettuce!" -- Ellie Crane

[Luca Boccassi]

That is absolutely true, it is not mandatory. It is one possible
solution (of many) to a particular use case being sounded out, that's
all. I don't think it was mentioned by anybody as needed, if it was,
happy to clarify.

Kind regards,
Luca Boccassi

[Russ Allbery]

Luca Boccassi <bl...@debian.org> writes:
> On Mon, 15 May 2023 at 02:26, Russ Allbery <r...@debian.org> wrote:

>> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh,
>> major player in Linux distributions, and I'm not sure how much they
>> care about compatibility with anyone else.)

> This is a counter-example to confute the assertion that *everybody* does
> the same thing, which has been made multiple times. I'm not sure whether
> it's an experiment or not, I mean if you ask their users/developers I
> think they'd tell you it's very much production ready, but it is largely
> irrelevant: it exists, and that's the only reason why it was mentioned,
> as it shows that it is _possible_ to do that and be a working
> distribution. Doesn't imply it's automatically desirable, but that was
> not the intention.

Ah, okay, I'm happy to agree with that point: you can violate the ABI and
continue to be a working distribution. (There are a lot of parts of the
ABI that if you violated them you would not have a working distribution,
but this is not one of them so far as I can tell.)

> Not quite: my argument is that binaries from these packages are not
> intended and not required to be ran on non-Debian systems, so there's no
> incompatibility introduced in the first place - everything still works
> where it is supposed to, exactly as it was before.

I think we're saying the same thing but quibbling over phrasing. I'd put
that as saying that it's fine for the binaries of certain core Debian
packages to be incompatible with the ABI because they're not intended to
be used outside of Debian. (In other words, I'm talking about
incompatibility as a concrete, testable property of a binary, and I think
you're talking about incompatibility as a more abstract concept of a
distribution.)

> No aggression intended whatsoever, sorry if it appeared that way. I am
> trying to understand what the rules are.

Well, the rule that I'd ideally set is don't break the ABI, even if it's
not obvious why breaking the ABI is a bad idea or you can't see any bad
consequences that could come from it, unless the reason for breaking the
ABI is absolutely central to the mission and purpose of Debian.

That said, it's not like we've never shipped a binary in Debian with a
different PT_INTERP. (I vaguely remember that some programming language
uses PT_INTERP tricks for some sort of private binary scheme? Objective
CAML or something? I ran across it once years ago and can't remember the
details. Also, IIRC klibc does some sort of PT_INTERP trick in some
situations that I don't remember the details of, although I don't think it
does that with general binaries.) So I do see your point that you would
prefer the rule to be more pragmatic than that.

My counterargument is that this proposal seems to mostly be about avoiding
having to create a symlink at a critical point in the bootstrap process,
and while it's tricky to get the timing right (and therefore kind of
annoying), the resulting usable system has to have that symlink anyway (I
think there's no disagreement about that). Not following the ABI for core
binaries seems like a scary change with unknown consquences to a bunch of
core packages to solve what looks like a relatively minor (if admittedly
annoying!) problem.

Note that the target of PT_INTERP on Debian is *already* a symlink, to
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 which was the multiarch path
that we actually want to use. We're already ensuring compatibility with a
symlink and I think we should just keep doing that and not venture into
these waters.

>> The world does not divide neatly into supported and unsupported use
>> cases. There are a lot of things I do to computers that I expect to
>> work in some situations but not in others. That includes, say, having
>> a Debian chroot on some other OS and running binaries from that chroot
>> without going into the chroot. Often that will absolutely not work.
>> Sometimes it will work, and it's convenient that it will work for some
>> obscure recovery situations or other weird one-off use cases. I've
>> also copied files from working systems to broken systems running a
>> different distribution before, and there's a list of caveats as long as
>> my arm, but sometimes it's a quick fix for something.

> Vast, vast majority of binaries from existing packages will already
> not work out of the box in that use case though.

I'm not sure why you think this is true and it makes me wonder if maybe my
intuition about cross-distribution compatibility is wrong. I would expect
to be able to copy, say, most (all?) binaries from coreutils from a Debian
system to some other distribution and run it (and that's exactly the sort
of binary that is useful in this kind of cross-distribution rescue case).
Is this not true today? What breaks?

Note that we're not talking about complicated packages with lots of
runtime like, say, Emacs. As I understand it your proposal wouldn't
change PT_INTERP for that binary anyway. We're presumably talking about
the kind of binaries that you need to bootstrap a minimal system, so
packages like coreutils or bash. And I would indeed expect those binaries
to be generally portable, as long as the same critical shared libraries
are available on other systems (in this case, PCRE2 and ncurses).

> Why are the existing incompatibilities allowed, and this isn't?

Because it breaks the ABI. I know you don't find that answer satisfying,
but that really is my answer.

> I am afraid I am a bit more "pragmatic" than that. I am very interested
> in what works and what doesn't. Whether it conforms to the letter of the
> Sacred Ancient Texts... interesting for sure, but secondary.

Right, this is our point of fundamental disagreement.

> Let me ask a simple policy question: why do I have to abide to your
> uncodified, unwritten use case of running a Debian program from a Debian
> package from outside a Debian chroot on a foreign, unmodified non-Debian
> system, but you don't have to abide by my uncodified, unwritten use case
> of running a Debian program on a Debian system with only a Debian /usr?

Well, the concrete answer here is that in order to make this change,
you're going to have to convince a bunch of Debian core package
maintainers to go along with this change and build their binaries in ways
that break the ABI. I believe at least some of them are going to have the
same reaction that Steve and I had, and will require a lot of convincing.

> And also, why it is only this change that has to abide to such use case,
> and the myriad of cases that cannot possibly work in such a setup are
> given a free pass (I mean I'm pretty sure the only executables that are
> guaranteed to work as you mention are golang and rustlang, where
> everything is statically compiled, everything else is already rc-buggy
> by that standard)?

I really would be surprised if coreutils didn't work, but maybe I'm wrong?
It does appear to generally have a PCRE2 requirement, but I would expect a
compatible PCRE2 in other distributions of a similar vintage.

> Isn't this the very reason we have a Policy for, so that we don't get to
> cherry-pick arbitrary use cases to block things we don't like?

Policy indeed probably doesn't say that you have to follow the Linux ABI
for normal binaries that aren't doing weird language-ecosystem-specific
things, but that's partly because it's so foundational and so automatic in
normal uses of compilers and linkers that I don't think we ever had any
reason to put it into Policy. It certainly didn't occur to me that anyone
would want to change the ABI for a subset of Debian packages.

Also, I thought you didn't care about Sacred Ancient Texts like Policy. :)

> Are you really sure we want to be in a place where anybody can bring up
> an out-of-policy use case as a valid reason to block something they
> don't like?

Yes, I absolutely want to be in a place where anyone can raise any reason
that matters to them in public discussion as a reason not to do something
they think would be harmful! This is the whole point of working
collaboratively together on a shared endeavor. Everyone gets a say! That
doesn't mean they get a *veto*, but they get a *say*, and that say is
weighted by how much perceived expertise they have and how directly the
plan affects their work in Debian.

I personally am certainly not expecting to have a veto or even that strong
of a say here. This doesn't affect any of my packages so far as I can
tell, and I'm far from an expert on the ABI. I've just been around for
long enough to pick up something by osmosis. I mostly jumped in because
it felt like you and Steve were just yelling at each other and I thought I
might be able to explain some of where he was coming from in a way that
may make more sense.

> Again, I am fine if we want to say that Debian-specific changes and
> Debianisms are bad and we cannot do anything that jeopardises
> interoperability, cross-distribution harmony and mutual compatibility.
> But let's do that then, and start by writing it down in Policy so that
> it applies fairly and equally to all cases.

I would love to get into a situation where Policy is a comprehensive guide
to everything people would like to have rules about in Debian, but I am
also pretty sure that if I quit my day job and made Policy my full-time
job, that would still not be done in ten years.

Distributions are complicated with a lot of moving parts and a lot of
those aren't written down. This is why we have people with substantial
personal expertise around who can think through novel situations and try
to figure out what the possible options and consequences are. It's a good
thing!

My starting point is that "follow the ABI" is like a safety interlock.
Whenever you park a car on a hill, you set the parking brake. You don't
try to figure out whether this hill is steep enough to make the car roll,
or do complex geometry calculations to try to figure out where the car
might roll too; you just set the parking brake always. That makes it a
habit, which has its own valuable properties. It means that you set the
parking brake in a bunch of places where it's unnecessary to do so in some
objective sense, but who cares, it's a habit, it's automatic.

You're saying "we don't need to set the parking brake here." You might be
right! I'm thinking through the negative consequences of that, but I'm
not saying that the specific examples that have come to mind so far are
all that compelling. My primary argument is that the logic of always
setting the parking brake and not thinking about it is what's compelling,
and you're asking us to give up a point of consistency that acts like a
safety interlock, and I'm saying that makes me very uncomfortable because
it requires we go through this complex process of figuring out what's
might go wrong and we also create an inconsistency at a core level of the
distribution that may have unforseen consequences. It's a lot easier
mentally and conceptually to just always set the parking brake, and
complexity is the enemy of any large software project.

[Bdale Garbee]

I could. 

Can you provide an example of actual value delivered to Debian from merged-/usr?

Bdale

[Matthew Vernon]

On 15/05/2023 16:54, Bdale Garbee wrote:
> I could.
>
> Can you provide an example of actual value delivered to Debian from
> merged-/usr?

With respect, I don't think this line of argument is going to get us
very far - this bug isn't about whether we should undo usr-merge, so I
don't think a debate on the merits or otherwise of usr-merge is germane.

[I think this applies to the contrary side of the argument also]

Regards,

Matthew

[Bdale Garbee]

You are of course correct. 

I remain unconvinced that anything related to the work on merged-/usr to date should be considered as a positive justification for actions discussed in this thread, but we can just let the rest drop.

Bdale

[Sam Hartman]

>>>>> "Matthew" == Matthew Vernon <mat...@debian.org> writes:

Matthew> On 15/05/2023 16:54, Bdale Garbee wrote:
>> I could.
>>
>> Can you provide an example of actual value delivered to Debian
>> from merged-/usr?

Matthew> With respect, I don't think this line of argument is going
Matthew> to get us very far - this bug isn't about whether we should
Matthew> undo usr-merge, so I don't think a debate on the merits or
Matthew> otherwise of usr-merge is germane.

I actually think the whole ABI issue is basically irrelevant for this
bug.
I think that in this bug at least we are hoping to discuss the
appropriateness of the dpkg warning for sources downstreams grab from
the Debian archivje.
That issue seems quite sufficient for one bug:-)

[Simon McVittie]

On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
> People build things on Debian that are not Debian packages. People
> compile binaries on Debian, and expect them to work on any system that
> has sufficiently new libraries.

*raises hand*

Hello, I represent an example of those people. With my $work hat on,
I'm involved in maintaining a family of Debian derivatives (the Steam
Runtime) that is used to run Steam games on arbitrary distributions,
including but not limited to Debian. Some of these binaries are built
on a Debian derivative, but run on an arbitrary other distribution,
using a RPATH[1] to find their non-glibc dependencies.

At the moment those binaries are built in ancient environments (based
on Ubuntu 12.04 and Debian 8), but as newer versions of glibc become
ubiquitous, we'll want to start relying on newer versions of Debian
(which will still be very old versions *at the time*, but I'm sure that
by 2030 or 2040 we'll want to be using versions of Debian that, in 2023,
are not released yet). In this use-case, yes we do need to be using the
interoperable ELF interpreter path.

We were able to use (Ubuntu and) Debian for this *because* Debian is
a relatively "ordinary" distribution that tends to follow cross-distro
standards. The major counterexample is multiarch library paths, but
multiarch library paths have some compelling technical advantages, and
because there's no ambiguity about which architecture uses which directory,
they're actually better for interop in some ways than /usr/lib (which
is ambiguous, because the Red Hat family of distros expects to find 32-bit
libraries there, but the Arch family expects 64-bit libraries, and it's
not possible to construct a tree where both get what they want).

I've spent a lot of the last 5 years working on putting Steam games in
containers so that they'll work more reliably on all distros, including
Debian, with less reliance on weird library search paths (although we
still have to use binaries built on an ancient Debian derivative with a
non-trivial RPATH to bootstrap the container environment). Because of
constraints around recent GPUs needing current graphics drivers even
if running on an otherwise ancient library stack, Steam's container
runtime has to construct a hybrid environment where glibc is either the
one from the host or the one from the container runtime environment,
whichever is newer; and while doing that, we have experienced some
broken situations that were caused by distributions diverging from the
interoperable ELF interpreter. One concrete example is that Arch Linux
uses the interoperable ELF interpreter for *almost* all executables,
but uses a different ELF interpreter for executables from the glibc
source package, for whatever reason; we were able to work around this,
but for a while it caused Arch to fail to launch these containers where
Debian/Fedora/etc. could.

This is not something that any of the distributions involved is
going to formally support, so the overall system consists of various
unsupported-but-works-in-practice things happening; but there is
absolutely no chance of Valve/Steam shipping separate binaries for
each distro (there are too many distros for that, even if you don't
count source-based distros like Gentoo that have an almost unlimited
number of concrete instantiations as binaries). However loudly we
might insist that our small subset-of-a-subset is the one they should
be targeting, what we're going to get is binaries that work on "mostly
interoperable" distros. As a result, if we want games on Linux (and
anything else requiring binary interop) to continue to be a thing,
pragmatically we should aim to be one of those "mostly interoperable"
distros, and encourage our friends and/or rivals in other distros to
do the same. Otherwise, the most likely outcome is for developers to go
back to an attitude of "I'm not going to support Linux, too many random
differences" and releasing for Windows only, and we all lose out.

As a result of all this, I would strongly prefer our compiler to
default to hard-coding the same interoperable ELF interpreter that
glibc upstream and the various major distros have agreed on (for example
/lib64/ld-linux-x86-64.so.2 on amd64), and I would also prefer it if we
can use that interoperable path in all the binaries we ship, including
src:glibc and the rest of the transitively Essential set.

One way that I like to think about this sort of thing is that we have a
finite number of "weird Debianism" tokens available, and we should aim
to spend them on the things that give us the best cost/benefit ratio.
We've never considered changing the ELF interpreter to be one of those,
to the extent of having a /lib64 solely for its benefit (on amd64)
even though by policy we don't generally use lib64.

(Incidentally, Steam's container runtime is always a merged-/usr
environment, to provide an environment that maximizes the probability
of any given script or binary working correctly; but it also goes to
considerable lengths to work on both merged- and unmerged-/usr host
systems. I think that's the right way round: it's conservative with what
it requires, but liberal with what it can run.)

smcv

[1] at the moment, genuinely a RPATH and not the modern RUNPATH for
annoying technical reasons

[Sam Hartman]

>>>>> "Sam" == Sam Hartman <hart...@debian.org> writes:

Sam> Hi. Off list, I wanted to try to explain what I think merged

My apology for sending a mail intended to be private to the bug. It was
not my intent to clutter an already cluttered discussion. I was really
just trying to help provide what understanding I had to a friend.

--Sam

[Sam Hartman]

Hi.
Off list, I wanted to try to explain what I think merged /usr has
brought us that is positive.
I want to stress that I'm not a huge fan of merged /usr, and I know
you've encouraged me not to argue from a devil's advocate position in
the past.
All the things I cite here are things I actually think are a positive
value, but I fully agree they do not justify the change to me.

* Normalization of paths. Whether things ended up in /usr/bin or /bin
tended to vary between unixes and especially between linux distros.
That has produced a lot of complexity over the years in build systems.
But it's also produced a lot of non-portable software--stuff written
either for Redhat or Ubuntu that manages to get the path wrong for the
other, and doesn't have the complexity of something like autoconf to
detect the change.
It's kind of nice to ignore all that.

* There's work, from people related to the systemd crowd, to effectively
get to a place where the initial state of a system is very close to
empty with a few symlinks and a read-only /usr. And then possibly
things get filled in a bit on first boot if necessary. So, for
example the systemd style split of /usr/lib/systemd/system vs
/etc/systemd/system where the /etc path generally gets to be empty at
least initially is part of this.
In some ways it reminds me of how AIX dealt with diskless
workstations. (I realize that's based on how Sun did it, but I never
dove into SunOS or Solaris as deep as AIX).
The goal is more for containers or for deploying updates to EOT
devices than for diskless.
The ideais kind of cool.
The updater/installer/container creater knows very little and can
start with an initial state.
It's easy to figure out what has been customized because anything in
/etc is a customization.
doing a factory reset is fairly easy.
It works well with signed OS images and ostree for deploying updates/the
sort of thing Endless does.
And yet there are probably other ways to get all the same benefits.

--Sam

[Simon McVittie]

On Mon, 15 May 2023 at 06:48:04 +0200, Johannes Schauer Marin Rodrigues wrote:
> Obviously, with Luca's proposal, binaries from packages built with a different
> dynamic linker path in them would not work on distributions without merged-/usr
> symlinks. But if the property of stuff from Debian being able to run on
> non-Debian non-merged-/usr systems is an important one, then why was it okay to
> have merged-/usr as the default? Because with merged-/usr we already changed
> the interface contract for a lot of things because now binaries and libraries
> can also be found at other locations than on non-merged-/usr systems. A script
> with a /usr/bin/bash shebang built on and for Debian will not work on a system
> without the symlinks.

pre-bookworm gcc writes /lib64/ld-linux-x86-64.so.2 into the ELF header
of amd64 binaries and I think post-bookworm gcc should continue to do so
even though that has never been the physical path to the ELF interpreter
on Debian (it's really /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2, or
historically a version-numbered path). People who are only testing on
Debian systems, even in pre-merged-/usr releases like Debian 9, could
already have been relying on /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
existing; and now that we're saying merged-/usr is mandatory,
people who are only testing on Debian >= 12 could also start
relying on /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 or
/usr/lib64/ld-linux-x86-64.so.2; but we arrange for both/all paths
to exist, and we advise developers that they should only rely on the
interoperable /lib64/ld-linux-x86-64.so.2, treating all other paths that
lead to the same binary as an implementation detail.

I don't think it's any different to say that both /usr/bin/sh and /bin/sh
exist and will work, but ask that everyone should continue to use /bin/sh
and treat /usr/bin/sh as an implementation detail.

The way I think about merged /usr is a variant of Postel's principle:
be conservative in what you require, but be liberal about what will work.
A merged-/usr system can run software that assumes merged-/usr, and can
also run software that doesn't (because of the compatibility symlinks);
a non-merged-/usr system can run software that doesn't assume merged-/usr,
but software that makes that assumption will sometimes fail to work on it.

Consistent with that, despite being in favour of merged-/usr myself,
I didn't switch my development laptop or my autopkgtest VM images
to the merged-/usr setup until it became effectively mandatory
during the bookworm cycle - because if a package was doing something
not-maximally-portable, I wanted my development laptop or my test VM to
be one of the places it would fail to work, so that I would notice and
report that bug.

Conversely, I *did* switch non-development computers (servers and family
laptops) to merged-/usr, because on those systems the important thing is
for software to work, even if in theory it "doesn't deserve" to work.

On post-bookworm, merged-/usr Debian systems, if we keep using #!/bin/sh
and #!/usr/bin/perl scripts, and avoiding #!/usr/bin/sh and #!/bin/perl
scripts, then I think that's a positive thing for cross-distro interop -
and using the interoperable path to the ELF interpreter (dynamic linker)
is completely consistent with that.

As far as I can see, post-bookworm Lintian will continue to warn about
the non-interoperable spellings like #!/usr/bin/sh and #!/bin/perl,
unless changed to special-case them.

Note that the paths that are canonical from the point of view
of cross-distro interoperability (like #!/bin/sh) are not always
the same as the paths that are canonical from the point of view of
realpath(). *At the moment*, they are usually canonical from dpkg's
point of view, but that won't be the case forever, and I think that's
fine. /lib64/ld-linux-x86-64.so.2 isn't considered to be canonical by
either realpath() or dpkg either (dpkg knows it's a symlink, even on
non-merged-/usr systems where /lib64 is a real directory), but it's
canonical for the x86_64-linux-gnu ABI and that's what we say is most
important.

> With merged-/usr we already *did* "change fundamental things around" for
> reasons that are really not clear to me (but which i do not want to discuss
> here) and as a result did not "care about interoperability" (with those who do
> not also adopt it).

Didn't we? With merged /usr, the "theoretically correct" #!/bin/sh
scripts that have always worked will continue to work, and additionally,
"incorrect" #!/usr/bin/sh scripts will *also* work. That sounds like
caring about interoperability to me!

The one piece of interop we lose with merged-/usr becoming mandatory is
that if a developer has only tested on Debian (and other merged-/usr distros
like Fedora), if they have used a less-interoperable spelling of the path
like #!/usr/bin/sh or #!/bin/perl, their testing will not highlight that
source of potential non-interop with others.

However, I don't think it's credible to claim that "if you only test on
Debian, your software might not work on non-Debian systems" is something
new; if you aim to support some other distro as a first-class citizen,
there's no substitute for actually testing on it (or having users,
bug reporters and contributors test on it for you).

smcv

[Jonathan Carter]

On 2023/05/15 20:26, Sam Hartman wrote:
> I want to stress that I'm not a huge fan of merged /usr, and I know
> you've encouraged me not to argue from a devil's advocate position in
> the past.

And this is where I stop reading any further.

To merge or not to merge is no longer an interesting or more
importantly, a relevant discussion. The project has made the choice to
implement it, after we've been one of the last major distributions to
hold out on implementing it. Sure, it's been bumpy, and there's
potential downsides, but none of the technical problems are as harmful
as still trying to make this a debate.

So, please, file bugs or fix bugs and support the people who need to
complete the remaining bits of merged-/usr, or otherwise please move on.

-Jonathan

[Luca Boccassi]

It's not even a proposal, it's a discussion to see "what would
actually break if X happened". That's why I'm asking for concrete use
cases, rather than theoretical points. Also, I am interested in what
the rules are. For example, glibc compatibility is just as important
if not more, why does that follow a different standard? If anything,
adding a missing symlink is trivial and a single command. Adding a
missing symbol to a shared object... not quite. There is an endless
list of downstream-only debianisms that break cross-compatibility in
just the same way (if not worse), and yet nobody cares about those.
Why?

Is that really the case? Let's test that hypothesis:

root@focal:/tmp# grep VERSION /etc/os-release
VERSION="20.04 LTS (Focal Fossa)"
VERSION_ID="20.04"
VERSION_CODENAME=focal
root@focal:/tmp# wget
http://ftp.uk.debian.org/debian/pool/main/c/coreutils/coreutils_9.1-1_amd64.deb
--2023-05-16 02:07:48--
http://ftp.uk.debian.org/debian/pool/main/c/coreutils/coreutils_9.1-1_amd64.deb
Resolving ftp.uk.debian.org (ftp.uk.debian.org)...
2001:1b40:5600:ff80:f8ee::1, 78.129.164.123
Connecting to ftp.uk.debian.org
(ftp.uk.debian.org)|2001:1b40:5600:ff80:f8ee::1|:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 2896560 (2.8M) [application/octet-stream]
Saving to: 'coreutils_9.1-1_amd64.deb'

coreutils_9.1-1_amd64.deb
100%[===============================================>] 2.76M
9.66MB/s in 0.3s

2023-05-16 02:07:48 (9.66 MB/s) - 'coreutils_9.1-1_amd64.deb' saved
[2896560/2896560]

root@focal:/tmp# dpkg -x coreutils_9.1-1_amd64.deb cu
root@focal:/tmp# ./cu/usr/bin/stat --help
./cu/usr/bin/stat: /lib/x86_64-linux-gnu/libselinux.so.1: no version
information available (required by ./cu/usr/bin/stat)
./cu/usr/bin/stat: /lib/x86_64-linux-gnu/libc.so.6: version
`GLIBC_2.33' not found (required by ./cu/usr/bin/stat)
./cu/usr/bin/stat: /lib/x86_64-linux-gnu/libc.so.6: version
`GLIBC_2.34' not found (required by ./cu/usr/bin/stat)

Whoops. Does this make coreutils rc-buggy now? ;-)

> > Why are the existing incompatibilities allowed, and this isn't?
>
> Because it breaks the ABI. I know you don't find that answer satisfying,
> but that really is my answer.

Its purpose is compatibility. If there's no compatibility in the first
place, what's its purpose? Or to put it in another way, wouldn't it be
more useful to talk about compatibility requirements instead?

And it probably should _not_ say anything about it. However, if
cross-distribution compatibility is a core requirement, if not for the
whole archive for a subset of it, shouldn't that be defined and
written down in the policy?

> Also, I thought you didn't care about Sacred Ancient Texts like Policy. :)

Policy continuously changes, so it's not really Sacred, is it now :-)

> > Are you really sure we want to be in a place where anybody can bring up
> > an out-of-policy use case as a valid reason to block something they
> > don't like?
>
> Yes, I absolutely want to be in a place where anyone can raise any reason
> that matters to them in public discussion as a reason not to do something
> they think would be harmful! This is the whole point of working
> collaboratively together on a shared endeavor. Everyone gets a say! That
> doesn't mean they get a *veto*, but they get a *say*, and that say is
> weighted by how much perceived expertise they have and how directly the
> plan affects their work in Debian.

It did look like a veto to me. More importantly, isn't relying on
passersby to spot alleged harmful changes dangerous, especially for
undocumented, uncodified and untested use cases, like unspecified and
vague cross-compatibility requirements?

> I personally am certainly not expecting to have a veto or even that strong
> of a say here. This doesn't affect any of my packages so far as I can
> tell, and I'm far from an expert on the ABI. I've just been around for
> long enough to pick up something by osmosis. I mostly jumped in because
> it felt like you and Steve were just yelling at each other and I thought I
> might be able to explain some of where he was coming from in a way that
> may make more sense.

I don't believe I've done any yelling here.

What if "setting the parking brake" is not enough, as the wheels are
already off and rolling downhill, as shown above, because while
everybody was looking at the parking brakes lever somebody ran off
with the bolts that kept the wheels attached? Why is it worth worrying
about compatibility with something that is already not compatible, and
it's not expected to be compatible for almost all other aspects?

Kind regards,
Luca Boccassi

[Luca Boccassi]

This sounds like a very interesting use case, and the first real one
mentioned, which is great to see - but I do not fully follow yet, from
what you are saying it seems to me that what you need is for your
binaries to use the usual pt_interp, that bit is clear. But why does
it matter if /usr/bin/ls on the host uses a different one? It's your
executables that you ship as part of that runtime that are the entry
points that need the usual loader path for your chroot-on-steroids,
no? The loader would still be reachable as it always was in this
theoretical exercise. I am probably missing something in how this
works in details.

Kind regards,
Luca Boccassi

[Simon McVittie]

On Tue, 16 May 2023 at 02:50:48 +0100, Luca Boccassi wrote:
> This sounds like a very interesting use case, and the first real one
> mentioned, which is great to see - but I do not fully follow yet, from
> what you are saying it seems to me that what you need is for your
> binaries to use the usual pt_interp, that bit is clear. But why does
> it matter if /usr/bin/ls on the host uses a different one?

We don't need to run the ls from the host, but we do need to run
glibc-related executables like ldconfig and localedef from either the
host or the container runtime, whichever is newer. Because glibc is
a single source package, executables and libraries within the glibc
bubble sometimes make use of private symbols in libraries that are also
within the glibc bubble (and IMO they have a right to do so), even though
executables from outside glibc would be discouraged or disallowed from
doing so. This means that when we have chosen a particular version of
glibc (which, again, must be whichever one is newer), we try to use its
matching version for *everything* originating in the glibc source package.

In principle we could get exactly the same situation if we've imported a
library from the host system (as a dependency of the graphics stack) that
calls an executable as a subprocess and expects it to be >= the version
it was compiled for - hopefully not (/usr)/bin/ls, but potentially others.

The wider point than my specific use-case, though, is that when there's a
standard, you can't predict what other software authors have looked at the
statement "you can rely on this" and ... relied on it. See also Russ's
(as ever, excellent) mails to the same thread.

I appreciate that you are trying to explore the edges of the
problem/constraint space and say "what if we did this, could that work?",
and it's good that you are doing that; but part of that process is
working with the other people on this list when they say "no, we can't
do that because...", and respecting their input.

Thanks,
smcv

[Luca Boccassi]

On Tue, 16 May 2023 at 09:27, Simon McVittie <sm...@debian.org> wrote:
>
> On Tue, 16 May 2023 at 02:50:48 +0100, Luca Boccassi wrote:
> > This sounds like a very interesting use case, and the first real one
> > mentioned, which is great to see - but I do not fully follow yet, from
> > what you are saying it seems to me that what you need is for your
> > binaries to use the usual pt_interp, that bit is clear. But why does
> > it matter if /usr/bin/ls on the host uses a different one?
>
> We don't need to run the ls from the host, but we do need to run
> glibc-related executables like ldconfig and localedef from either the
> host or the container runtime, whichever is newer. Because glibc is
> a single source package, executables and libraries within the glibc
> bubble sometimes make use of private symbols in libraries that are also
> within the glibc bubble (and IMO they have a right to do so), even though
> executables from outside glibc would be discouraged or disallowed from
> doing so. This means that when we have chosen a particular version of
> glibc (which, again, must be whichever one is newer), we try to use its
> matching version for *everything* originating in the glibc source package.
>
> In principle we could get exactly the same situation if we've imported a
> library from the host system (as a dependency of the graphics stack) that
> calls an executable as a subprocess and expects it to be >= the version
> it was compiled for - hopefully not (/usr)/bin/ls, but potentially others.

Thanks for the clarification, so if I understood correctly, your use
case is that sometimes (eg: when they are newer) you pull binaries
(eg: ldconfig) from the host, and run them from the container? So, in
case let's say ldconfig on the host points to /usr/lib/ld, but because
your container is not usr-merged, it wouldn't find the interpreter and
fail?

> The wider point than my specific use-case, though, is that when there's a
> standard, you can't predict what other software authors have looked at the
> statement "you can rely on this" and ... relied on it. See also Russ's
> (as ever, excellent) mails to the same thread.
>
> I appreciate that you are trying to explore the edges of the
> problem/constraint space and say "what if we did this, could that work?",
> and it's good that you are doing that; but part of that process is
> working with the other people on this list when they say "no, we can't
> do that because...", and respecting their input.

I respect and appreciate the input, but I want to understand it too,
hence the "because..." part is what I was looking for - so thanks for
providing it, it is really useful.

Kind regards,
Luca Boccassi

[Roger Lynn]

On 15/05/2023 19:00, Simon McVittie wrote:
> On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
>> People build things on Debian that are not Debian packages. People
>> compile binaries on Debian, and expect them to work on any system that
>> has sufficiently new libraries.
>
> *raises hand*
>
> Hello, I represent an example of those people. With my $work hat on,
> I'm involved in maintaining a family of Debian derivatives (the Steam
> Runtime) that is used to run Steam games on arbitrary distributions,
> including but not limited to Debian. Some of these binaries are built
> on a Debian derivative, but run on an arbitrary other distribution,
> using a RPATH[1] to find their non-glibc dependencies.

As another much smaller example, which I hope is still relevant, I have
taken binaries from Debian stable or oldstable armel and run them for
diagnostic purposes on embedded boards which only have Busybox installed and
for which I don't have a convenient build environment.

Regards,
Roger

PS Apologies if I've followed up incorrectly - I read debian-devel through
an NNTP gateway.

[Ansgar]

Hi,

On Thu, 2023-05-11 at 00:32 +0200, Ansgar wrote:
> On Wed, 2023-05-10 at 23:47 +0200, Ansgar wrote:
> > Cool, then let's ask tech-ctte.
> >
> > Dear ctte, please consider overruling the dpkg maintainer to
> > include
> > the patch from #994388[1].
> >
> > Thanks,
> > Ansgar
> >
> >   [1]: https://bugs.debian.org/994388#397
>
> For derivatives based on Debian stable it might be worth having this
> included in the next stable release; this would need a fairly quick
> decision on this issue.

The full freeze is approaching and there has been no progress on this
issue. Does the ctte think a decision before the release is still
possible?

As asked earlier I'm also interested in whether the ctte thinks there
is enough consensus about how this issue should be solved or do we
need a longer discussion to explore the solution space?

I admit not having read all mails in the thread as it went fairly off
topic IMHO.

Ansgar

[Ansgar]

Hi,

On Thu, 2023-05-18 at 10:48 -0700, Sean Whitton wrote:

> On Thu 18 May 2023 at 07:21PM +02, Ansgar wrote:
>
> > The full freeze is approaching and there has been no progress on
> > this
> > issue. Does the ctte think a decision before the release is still
> > possible?
>

> Not speaking for the whole ctte, but I don't think that is possible.

Why not?

Do you think the implications of removing the warning are unclear?

Do you think we need to explore alternative solutions?

Ansgar

[Gunnar Wolf]

Ansgar dijo [Thu, May 18, 2023 at 07:55:03PM +0200]:

> Why not?
>
> Do you think the implications of removing the warning are unclear?
>
> Do you think we need to explore alternative solutions?

(I am no longer part of the Committee, answering just as another
developer)

dpkg has many bits that make it special. It has been discussed whethe
dpkg should be a native package or it should become non-native; if it
were non-native, having a patch that contradicts the upstream author's
wishes would be easier (and I'm not saying that I'd be up for patching
this warning out as it is).

If we were to force a patch silencing out this warning right now and
for all of the Bookworm cycle, and the dpkg authors disagree with it,
they could remove (even omit to include it) in any updates. Upstream
has repeatedly expressed their opposition to the way usrmerge has been
brought forward, and the warning silenced specifically for Debian is
already the best compromise situation we have been able to reach --
even though we are aware the situation is far from ideal.

[Ansgar]

On Thu, 2023-05-18 at 12:14 -0600, Gunnar Wolf wrote:
> Ansgar dijo [Thu, May 18, 2023 at 07:55:03PM +0200]:
> > Why not?
> >
> > Do you think the implications of removing the warning are unclear?
> >
> > Do you think we need to explore alternative solutions?
>
> (I am no longer part of the Committee, answering just as another
> developer)
>
> dpkg has many bits that make it special. It has been discussed whethe
> dpkg should be a native package or it should become non-native; if it
> were non-native, having a patch that contradicts the upstream
> author's wishes would be easier (and I'm not saying that I'd be up
> for patching this warning out as it is).

Do you think this implementation detail is relevant for what we do in
Debian? I don't care how a patch is applied and don't think that detail
has to be part of the decision.

I also don't see any further active discussion on this aspect (unless I
missed something).

> If we were to force a patch silencing out this warning right now and
> for all of the Bookworm cycle, and the dpkg authors disagree with it,
> they could remove (even omit to include it) in any updates.

So? That is the case with any ruling the ctte makes, including the non-
binding one the ctte just did under Constitution 6.1.5.

> Upstream
> has repeatedly expressed their opposition to the way usrmerge has
> been brought forward, and the warning silenced specifically for
> Debian is already the best compromise situation we have been able to
> reach -- even though we are aware the situation is far from ideal.

If the best solution we have been able to reach is telling users of
derivative distributions to configure their system in a way that is
expected to cause breakage, then it would be worth documenting that
this is the case and we cannot do more for derivative users.

If the ctte believes this to be fine, then the ctte can decide to not
overrule the maintainer.

I don't think this is a good reason to delay the decision indefinitely
unless there is some reason to believe something will change within a
reasonable period of time (which I don't see happening).

Ansgar

[Luca Boccassi]

We heard so much in the past couple of weeks about how important it is
for the project not to cause issues for derivatives and
cross-compatibility use cases, even speculatively. This is not even
speculative, it is certain to cause damage (as we experienced first
hard last year), I don't see how we can ignore it after all of these
discussions.

Kind regards,
Luca Boccassi

[Bastian Blank]

Hi Gunar

On Thu, May 18, 2023 at 12:14:42PM -0600, Gunnar Wolf wrote:
> dpkg has many bits that make it special. It has been discussed whethe
> dpkg should be a native package or it should become non-native; if it
> were non-native, having a patch that contradicts the upstream author's
> wishes would be easier (and I'm not saying that I'd be up for patching
> this warning out as it is).

But why does the state of the package (native vs non-native) can have
any effect on a CTTE decision? Or do you want to say I can block CTTE
from reaching any kind of decision just by uploading a package as
native? Sorry, but this does not compute.

> If we were to force a patch silencing out this warning right now and
> for all of the Bookworm cycle, and the dpkg authors disagree with it,
> they could remove (even omit to include it) in any updates.

Sure, but this is a direct violation of a CTTE decision. How often do
you think someone could do that?

Bastian

--
"Life and death are seldom logical."
"But attaining a desired goal always is."
-- McCoy and Spock, "The Galileo Seven", stardate 2821.7

[Matthias Klumpp]

Am Do., 18. Mai 2023 um 20:39 Uhr schrieb Luca Boccassi <bl...@debian.org>:
> [...]

> We heard so much in the past couple of weeks about how important it is
> for the project not to cause issues for derivatives and
> cross-compatibility use cases, even speculatively. This is not even
> speculative, it is certain to cause damage (as we experienced first
> hard last year), I don't see how we can ignore it after all of these
> discussions.

Speaking as maintainer of two Debian derivatives (PureOS and an
internal one), keeping this warning means we will need to patch dpkg
which of course is possible, but also a bit annoying. It is also odd
that Debian's configuration suddenly becomes "invalid" just by
changing the name of the OS.
(FWIW, PureOS has been usrmerged before Debian did that officially,
and so was Ubuntu - so far we haven't experienced any issues and our
users are happy - syncing dpkg without patching it will for sure cause
a lot of confusion though).

Cheers,
Matthias

--
I welcome VSRE emails. See http://vsre.info/

[Gunnar Wolf]

Bastian Blank dijo [Thu, May 18, 2023 at 09:05:44PM +0200]:

> But why does the state of the package (native vs non-native) can have
> any effect on a CTTE decision? Or do you want to say I can block CTTE
> from reaching any kind of decision just by uploading a package as
> native? Sorry, but this does not compute.

> (...)

> Sure, but this is a direct violation of a CTTE decision. How often do
> you think someone could do that?

During my time as a Technical Committe member, /usr-merge was the
point we most came back to. And yes, the way the TC decisions were
dodged or omitted by the dpkg maintainers was... quite depressing.

However, my reply should only be read regarding what I believe should
be done in the following ~month before the release.

Of course, I don't see the situation as ideal, nor as something that
should persist. I hope a _fixed_ dpkg is uploaded and becomes part of
Bookworm's first point release.

But, even if it were on the table (which is not AFAICT), I would (in a
strictly personal capacity) oppose the TC requiring such a patch at
this point.

[Arnaud Rebillout]

On 19/05/2023 01:33, Luca Boccassi wrote:
> We heard so much in the past couple of weeks about how important it is
> for the project not to cause issues for derivatives and
> cross-compatibility use cases, even speculatively. This is not even
> speculative, it is certain to cause damage (as we experienced first
> hard last year), I don't see how we can ignore it after all of these
> discussions.

Speaking as Kali maintainer, we patched it out already a while ago:
https://gitlab.com/kalilinux/packages/dpkg/-/commit/bff5fa3c

Best,

Arnaud

[Ansgar]

Hi,

On Fri, 2023-05-19 at 07:09 -0700, Sean Whitton wrote:
> On Thu 18 May 2023 at 07:55PM +02, Ansgar wrote:
>
> > Why not?
>
> We will not move that fast.

So there is no real reason?

As there doesn't seem to be anything you think need to be talked about
that is missing to get to a decision (given you ignored all other
questions multiple times).

Do you expect this to take longer than 2-3 months? I would suggest to
just use a GR in that case: it seems quicker and less painful than a
multi-month long process for a simple(!) question.

Ansgar

[Helmut Grohne]

Hi Ansgar,

I'm speaking with a CTTE hat here, but not representing CTTE consensus.

On Wed, May 10, 2023 at 11:47:42PM +0200, Ansgar wrote:
> Dear ctte, please consider overruling the dpkg maintainer to include
> the patch from #994388[1].

I think we need to reject this request on multiple levels.

On a social level, I see that you are frustrated with how dpkg is being
maintained and how communication does not work out in practice. While
part of that can be attributed to the dpkg maintainer, this goes both
ways and the way you refer this matter to the CTTE without even
attempting to resolve it by other means just serves to deepen that
dispute. I see that the dpkg maintainer has recently contributed
constructively to the discussion about how dpkg can be part of a
solution for the problems resulting from the /usr merge. I have a hard
time saying the same about your interaction here. Therefore, I think
your request should be rejected as not being serious.

On a process level, I think I miss attempts to resolve this with the
dpkg maintainer in a constructive way. The present discussion clearly
shows that dpkg's support for how Debian deals with merged /usr is
lacking. We are dealing with multiple file-loss scenarios (something we
otherwise consider grave) and issuing a warning about such behaviour
seems fine to me. On the other hand, much of the project seems to agree
that the way this warning is worded is far from optimal and evidently
leads to confused users. And while it may seem that any attempt at
resolving may lead nowhere, the same can be said about our dpkg
maintainer's concerns about our /usr-merge strategy and him pointing out
real problems affecting Debian installations in practice. Given the
recent constructive communication, I think it is far from clear that
this option is exhausted. In particular, acknowledging the problems
presented and proposing strategies for dealing with them could go a long
way towards a cooperative solution. At present, I see the options to
keep and to delete the warning on the table, but there clearly is
unexplored middle ground. As such, I think your request should be
rejected as not having exhausted the solution space.

On a technical level, Debian's support for merged /usr is currently
founded on the moratorium preventing breakage. Please observe that this
moratorium applies to Debian and Debian only. We have implemented
processes to validate this. If a derivative wants to use merged /usr
(which probably every derivative should), it certainly needs to prevent
breakage in some way - presumably using a similar process. I think that
the cost of patching dpkg is minor compared to the cost of a process
that prevents breakage arising from our merged /usr strategy. Given
this, a warning-by-default (worded in a better way) for derivatives
actually can be a good thing, because it makes derivatives aware that
they cannot just ignore merged /usr, but have to act. As such, I think
your request should be rejected since the measure is technically
reasonable in principle.

Does anyone mind just closing the bug?

At the same time, I recommend changing the warning. The amount of
feedback we get regarding the warning should make it clear that the
current wording is still seen as offensive and causes confusion. Keeping
the warning in its current form also serves little but deepening the
dispute.

For instance, the wording "going behind dpkg's back" may be considered
technically correct, but it also can be objectively described as passive
aggressive. Just deleting this aspect and instead saying "This system
uses merged-usr-via-aliased-dirs which violates core assumptions of
dpkg." would probably keep the intended message in a less
confrontational way. Elaborating that file loss is being mitigated by a
process (moratorium) on Debian would also help. Looking into the wiki, a
recommendation of dpkg-fsys-usrunmess should caution that using it now
breaks other tool's assumptions such as systemd and is generally not
being tested with QA anymore.

Even if one were thinking that the warning were appropriate as is,
adapting it again due to community feedback would demonstrate empathy
and be a step towards a cooperative solution.

I think our priority should be finding a way to terminate the
moratorium.

Helmut

[Luca Boccassi]

On Sun, 21 May 2023 at 15:51, Ansgar <ans...@43-1.org> wrote:

>
> On Sun, 2023-05-21 at 16:25 +0200, Helmut Grohne wrote:
> > On a process level, I think I miss attempts to resolve this with the
> > dpkg maintainer in a constructive way.
>

> The patch was already suggested to the dpkg maintainer and rejected.

>
> > Does anyone mind just closing the bug?
>

> Yes, I do.
>
> Please pass a resolution that you don't want to override the dpkg
> maintainer and that telling derivative users to configure their system
> in a way that will cause breakage is okay to do for packages in Debian.

I do as well. Recently a very strong consensus emerged that even
accidentally causing damage and/or incompatibility to
downstreams/external use cases is strongly frowned upon, to say the
least. Talking the talk is easy, now we have to walk the walk. Both
the warning and the 'unmess' tool cause significant damage and break
cross-compatibility, so they both need to be removed.

A "mind the moratorium" message would be of course very sensible to have.

Kind regards,
Luca Boccassi

[Ansgar]

On Sun, 2023-05-21 at 16:25 +0200, Helmut Grohne wrote:

> On a process level, I think I miss attempts to resolve this with the
> dpkg maintainer in a constructive way.

The patch was already suggested to the dpkg maintainer and rejected.

> Does anyone mind just closing the bug?

Yes, I do.

Please pass a resolution that you don't want to override the dpkg
maintainer and that telling derivative users to configure their system
in a way that will cause breakage is okay to do for packages in Debian.

Ansgar

[Matthew Vernon]

Hi,

This thread has rather veered off the initial bug report.

On 11/05/2023 13:16, Simon Richter wrote:
> Hi,

>
> On 5/11/23 10:59, Sean Whitton wrote:
>
>>> Dear ctte, please consider overruling the dpkg maintainer to include
>>> the patch from #994388[1].
>

>> Currently dpkg contains code to emit the merged-/usr warning, that's
>> dead code on Debian, but which becomes active when packages from the
>> Debian archive are copied unmodified into derivatives.
>
> The way I see it (but I'm not a dpkg maintainer), the current
> implementation is correct, as dpkg does not support aliased directories,
> but Debian has decided to use it in such an environment nonetheless. The
> tech-ctte decision not to roll back usrmerge accepts responsibility for
> this decision, so silencing the warning on Debian is correct, but no one
> has accepted that responsibility for derived distributions.
>
> Any derived distribution can easily go on record and request inclusion
> in the list of distributions where this warning is suppressed, by typing
> the phrase "Yes, I understand that this is a bad idea." into an email
> client.

I have considerable sympathy for this point of view. Further, given
ongoing (and quite fruitful) discussion on how to resolve the
outstanding issues around /usr-merge and dpkg, I don't think the
question of dpkg's warning (and its unfortunate wording) is one that is
useful for the technical committee (and the dpkg maintainers) to be
spending time on right now.

I think I would feel differently if there were derivatives who had asked
the dpkg maintainers to likewise exclude their distro from the warning
had been rebuffed (though I suspect such folk will just be patching it
out in their own builds).

Likewise I would expect that once we have finished sorting out the
outstanding /usr-merge & dpkg issues that the warning would be removed.

But those scenarios aren't where we're at now, so I think the project
should continue to focus on moving ourselves to the point where dpkg
does support /usr-merge as implemented in Debian.

Regards,

Matthew

[Ansgar]

On Tue, 2023-06-06 at 11:04 +0100, Sean Whitton wrote:
> I don't think the TC has or should have any authority over dpkg
> upstream, but with dpkg being a native package, any implementation of
> our decision for the Debian archive is also implemented for dpkg
> upstream.  And it might be that the dpkg developers would be against
> the TC override solely or mostly because of this fact.  So possibly
> changing that would resolve things peacefully.

Given the Debian project owns dpkg.org, the upstream mailing list is
@lists.debian.org, official releases are published on deb.debian.org
and so on, I think the Debian project *is* the upstream for dpkg.

Ansgar

[Ansgar]

On Tue, 2023-06-13 at 20:01 +0100, Matthew Garrett wrote:
> After discussing this at our monthly meeting, we concluded that the
> technical committee isn't going to take action on this at the moment.
> There's a legitimate technical consideration behind the warning, and
> it's necessary for derivatives to ensure that they handle the
> associated scenarios properly.

Okay, I take that as "Debian doesn't care about derivatives".
Suggesting users to revert to split-/usr *will* break stuff for users
once more code Debian assumes merged-/usr.

> While those underlying technical issues exist, we
> think it's premature for the committee to intervene on this specific
> issue.

Okay, I guess the very long drama about this last year was not
sufficiently long and we did not discuss it in sufficient detail.

I'm a bit disappointed how the ctte appears to do as much as they can
to avoid deciding on this :-(

Ansgar

[Matthew Garrett]

On Tue, Jun 13, 2023 at 09:14:53PM +0200, Ansgar wrote:
> On Tue, 2023-06-13 at 20:01 +0100, Matthew Garrett wrote:
> > After discussing this at our monthly meeting, we concluded that the
> > technical committee isn't going to take action on this at the moment.
> > There's a legitimate technical consideration behind the warning, and
> > it's necessary for derivatives to ensure that they handle the
> > associated scenarios properly.
>
> Okay, I take that as "Debian doesn't care about derivatives".
> Suggesting users to revert to split-/usr *will* break stuff for users
> once more code Debian assumes merged-/usr.

With my non-CTTE hat on, I think this is a demonstration that Debian
*does* care about derivatives. It's important to ensure that derivatives
are aware of the subtle interactions between dpkg and usrmerge,
otherwise they may suffer from hard to diagnose issues on upgrades. The
message from dpkg says nothing about reverting to split-/usr, and
instead points at a wiki page. If our documentation on how to avoid
these issues is incomplete (ie, if we're only describing how to avoid it
by using split-/usr rather than describing the mitigations we've imposed
within Debian to avoid triggering the issues), perhaps a better approach
would be to improve that documentation? We don't benefit our users by
pretending there isn't a problem here.

[Ansgar]

Matthew Garrett writes:
> With my non-CTTE hat on, I think this is a demonstration that Debian
> *does* care about derivatives. It's important to ensure that derivatives
> are aware of the subtle interactions between dpkg and usrmerge,
> otherwise they may suffer from hard to diagnose issues on upgrades. The
> message from dpkg says nothing about reverting to split-/usr, and
> instead points at a wiki page. If our documentation on how to avoid
> these issues is incomplete (ie, if we're only describing how to avoid it
> by using split-/usr rather than describing the mitigations we've imposed
> within Debian to avoid triggering the issues), perhaps a better approach
> would be to improve that documentation? We don't benefit our users by
> pretending there isn't a problem here.

Okay, I followed your recommendation and added a small warning to the
FAQ: https://wiki.debian.org/Teams/Dpkg/FAQ?action=diff&rev1=78&rev2=79

Ansgar