Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

Package: debian-policy
X-Debbugs-CC: bi...@debian.org pkg-systemd...@lists.alioth.debian.org

It has come to my attention that there is one package in Debian using
dpkg-divert to mask a systemd configuration file (an udev rule).
Speaking as one of the maintainers, both upstream and downstream, I
find this greatly undesirable for several reasons that I will outline
later. Hence I would like to propose explicitly mentioning that dpkg-
divert must not be used for systemd configuration files (units, rules,
etc), and instead the supported workflow (drop-ins, masking, etc) must
be used, both by packages and administrators. This is already standard
practice, and again there is only one instance that needs correcting as
far as I understand, and I have already provided a bug and a MR for
that [1][2]. So the impact of this policy change should be minimal, and
it's mostly to ensure more such instances are accidentally added in the
future.

I have a draft policy update, that adds a paragraph to the dpkg-divert
section of the policy. It is attached here, and also available on Salsa
on my fork [3].

The full reasoning below is what I provided on the MR for the one
existing instance, and I'll copy it mostly unchanged as I hope it's
exhaustive enough. It uses the one existing instance I found as
concrete example. This is not intended to single out the maintainer or
assign blame, but merely to illustrate the point with a concrete and
real use case. Quoting from the MR:

One of the main goals behind the systemd (and its udev component)
project is to unify how the low-level userspace components of a Linux
distro work, so that the exact same mechanisms, patterns, behaviours
and interfaces apply to a multitude of use cases, implementations and
tools. A core part of this is the configuration system. The
configuration system supports a complex schema of main contents,
overrides, drop-ins, masking and aliasing. This system is used and
understood by all systemd components, across all Linux distributions,
with the same interface, look and feel, so that users can feel at home
and know how to work the system regardless of the vendor, and so that
programs can rely on a stable and common interface that doesn't have to
be endlessly customized depending on which vendor or distribution it is
used by. The concept of 'masking' a configuration is well understood,
ubiquitous and fully supported by all the tooling, including in input
_and_ output, and logging, and so on. By using the supported mechanism
for masking we ensure that there are no surprises for users, coders and
vendors. When an unsupported masking mechanism is used, as in this
case, the fact that the original item is masked is completely hidden to
the systemd components, and thus to the interface provided to the user.
This causes at best confusion and misunderstanding, and at worst bugs
that will inevitably fall on the systemd maintainers, causing increased
workload for an already over stretched team.

A simple and obvious example of what I am referring to is already
included in the git commit for this change. Consider what happens when
udev parses the vanilla configuration (ie: without amazon-ec2-utils
installed):

$ udevadm test /dev/cdrom 2>&1 | grep 60-cdrom_id
Reading rules file: /lib/udev/rules.d/60-cdrom_id.rules

The user/system/log is clearly notified that the file is parsed, there
are no errors, and thus is used.

Now consider what happens when the current version of amazon-ec2-utils
is installed:

$ udevadm test /dev/cdrom 2>&1 | grep 60-cdrom_id
Reading rules file: /lib/udev/rules.d/60-cdrom_id.rules

It looks exactly the same. But something extremely different is
actually happening, in fact the opposite! The file is empty, so the
vanilla rule is effectively masked, but nothing and nobody is notified
of this very important fact when udev is running. One would have to
query dpkg-divert to figure this out, but this is something that even
someone like me, who can reasonably consider myself a proficient Debian
user, would not think of looking at.

Now finally consider what happens if amazon-ec2-utils with my proposed
change is installed:

$ udevadm test /dev/cdrom 2>&1 | grep 60-cdrom_id
Skipping overridden file '/lib/udev/rules.d/60-cdrom_id.rules'.
Skipping empty file: /etc/udev/rules.d/60-cdrom_id.rules

There is an extremely clear, obvious and expected indication that the
original rule is ignored because it is overridden, _and_ that the
override is empty, as it's a masking operation. Now udev is a bit older
than the other components, so "masking" is not named explicitly as it
would be for example by systemctl, but it is the same operation and the
same result and the same interface. Something I should probably improve
a bit upstream, I will take a note of this.

Here's another example with systemctl and units. I have masked systemd-
homed.service via `systemctl mask` and this is what the interface shows
me when I check the status:

$ systemctl status -l --no-pager systemd-homed.service
○ systemd-homed.service
Loaded: masked (Reason: Unit systemd-homed.service is masked.)
Active: inactive (dead) since Sun 2023-05-07 18:59:51 BST; 7s ago
<snip>

Message is clear and unambiguous, I masked the unit so it will not run.
Here's what happens if I unmask it and instead dpkg-divert the vanilla
unit with a /bin/true no-op:

# systemctl status systemd-homed.service
○ systemd-homed.service
Loaded: loaded (/usr/lib/systemd/system/systemd-homed.service; enabled; preset: enabled)
Active: inactive (dead) since Sun 2023-05-07 18:59:51 BST; 4min 49s ago
<snip>

From the output, it's all good. The vanilla unit is enabled and should
run and provide its services on the next boot or when I next start it,
except it's not - it's been dpkg-diverted to a no-op, but there's no
indication there. This is poor UI, poor experience, poor integration.
And it's just the most basic, obvious example of what can go wrong: I
cannot guarantee that there won't be more, higher severity issues, but
I can certainly state that I will provide no support when they
inevitably surface, as this is not a supported mechanism.

The supported mechanism for masking, for both local admins and
packages, is masking via symlinks to /dev/null in /etc. And again I
must stress that packages configure things in /etc too, not only
administrators! This is fully expected and supported. For example, deb-
systemd-helper from init-system-helpers
(https://sources.debian.org/src/init-system-helpers/1.65.2/script/deb-systemd-helper/
) is used in Debian/Ubuntu and derivatives to enable/disable/mask/alias
units, and these changes are performed from auto generated (by
debhelper
https://sources.debian.org/src/debhelper/13.11.4/dh_installsystemd/)
maintainer script snippets not too dissimilar to what I have in this
MR. And this does not only apply to files that the packages own! It
also applies cross-packages. For example, by installing the 'dbus-
broker' package, an alias to dbus.service that effectively masks and
takes over dbus.service from the dbus-daemon package is installed in
/etc by the maintainer script:

$ ls -l /etc/systemd/system/dbus.service
lrwxrwxrwx 1 root root 43 May 6 17:45 /etc/systemd/system/dbus.service
-> /usr/lib/systemd/system/dbus-broker.service

From `/var/lib/dpkg/info/dbus-broker.postinst`:

# Automatically added by dh_installsystemd/13.11.4
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then
# The following line should be removed in trixie or trixie+1
deb-systemd-helper unmask 'dbus-broker.service' >/dev/null || true

# was-enabled defaults to true, so new installations run enable.
if deb-systemd-helper --quiet was-enabled 'dbus-broker.service'; then
# Enables the unit on first installation, creates new
# symlinks on upgrades if the unit file has changed.
deb-systemd-helper enable 'dbus-broker.service' >/dev/null || true
else
# Update the statefile to add new symlinks (if any), which need to be
# cleaned up on purge. Also remove old symlinks.
deb-systemd-helper update-state 'dbus-broker.service' >/dev/null || true
fi
fi
# End automatically added section

This is supported, intended and expected, as it's the correct mechanism
to do aliasing and overriding. No local user action is performed,
everything is done by the package being installed. In effect, the user
action IS installing the package.

Ensuring Debian packages use a supported, intended and expected
mechanism to manage and configure systemd files is fundamental for the
ability of the systemd downstream maintainers team to be able to fully
and effectively support Debian and its users. Deviating from the
expected path causes additional workload, unexpected incidents and
general discomfort for an already over-worked and small team, while
providing no tangible benefit, as the net result is the same.

[0] https://salsa.debian.org/cloud-team/amazon-ec2-utils/-/merge_requests/2
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035667
[3] https://salsa.debian.org/bluca/policy/-/tree/systemd_overrides

--
Kind regards,
Luca Boccassi

[Sam Hartman]

>>>>> "Luca" == Luca Boccassi <bl...@debian.org> writes:

Luca> It has come to my attention that there is one package in
Luca> Debian using dpkg-divert to mask a systemd configuration file
Luca> (an udev rule). Speaking as one of the maintainers, both
Luca> upstream and downstream, I find this greatly undesirable for
Luca> several reasons that I will outline later. Hence I would like
Luca> to propose explicitly mentioning that dpkg- divert must not be
Luca> used for systemd configuration files (units, rules, etc), and
Luca> instead the supported workflow (drop-ins, masking, etc) must
Luca> be used, both by packages and administrators.

First, I thought there was already a prohibition about one package
mucking with another package's configuration.
In many cases it sounds like it's already against policy for one package
to change the default systemd or udev configuration--at least for
packages in the archive.
(I am skepticle that amazon-ec2-utils complies with policy in general).


In cases where the change being made is permitted by policy, I think you
have made a compelling case to vastly prefer the native systemd and udev
mechanisms to dpkg-divert.
I don't think that your case is strong enough to forbid dpkg-divert.

As far as I can tell reading your reasoning, dpkg-divert *works fine*.
It just gives surprising results to someone coming from the systemd
universe.

But consider a package that diverts several resources, several of them
systemd and several of them not systemd.
The maintainer might legitimately want to use the same mechanism for all
the overriding/masking so that systemd resources and non-systemd
resources were handled the same.

There's a real trade off there, and we generally leave those to
maintainers.

So, I'd support policy advice (ENCOURAGED) rather than policy
requirements (MUST) in this case.

I do think that if a maintainer violates this advice without a good
reason, important is a more appropriate bug severity than wishlist, but
unfortunately we don't have a good way to specify that in policy
language.

I would not support policy recommendation language (RECOMMENDED/SHOULD)
because that has a connotation that failing to follow the recommendation
is always a bug, and I don't think that's true here.

--Sam

[Russ Allbery]

I think your X-Debbugs-Cc was syntactically invalid and thus didn't work.
I manually added in the other addresses in this reply.

Luca Boccassi <bl...@debian.org> writes:

> It has come to my attention that there is one package in Debian using
> dpkg-divert to mask a systemd configuration file (an udev rule).
> Speaking as one of the maintainers, both upstream and downstream, I find
> this greatly undesirable for several reasons that I will outline
> later. Hence I would like to propose explicitly mentioning that

> dpkg-divert must not be used for systemd configuration files (units,

> rules, etc), and instead the supported workflow (drop-ins, masking, etc)
> must be used, both by packages and administrators. This is already
> standard practice, and again there is only one instance that needs
> correcting as far as I understand, and I have already provided a bug and
> a MR for that [1][2]. So the impact of this policy change should be
> minimal, and it's mostly to ensure more such instances are accidentally
> added in the future.

> I have a draft policy update, that adds a paragraph to the dpkg-divert
> section of the policy. It is attached here, and also available on Salsa
> on my fork [3].

The part of Policy that you edited with this patch is basically
unmaintained and should ideally be removed in favor of actual Policy. (I
had started looking at that a long time ago and then never finished.) All
of those appendices from the old packaging manual predate better
documentation maintained elsewhere (such as in the dpkg package) and are
ambiguous with regards to whether they set requirements for Debian
packages, document things for local administrators, or something else.
The Policy manual warns that they may not be normative, and people often
don't think to read them (for good reason).

In the case of diversions, while I certainly agree with your proposed
rule, I suspect Policy should say something stronger and more general,
namely that no package in Debian should divert a file from another package
unless this is arranged cooperatively between the packages to solve some
specific (unusual) problem. To me, this feels similar to the case of one
package modifying the configuration files of another package, where we
explicitly prohibit one package modifying the configuration of another
package except through an interface provided by the package whose
configuration is being modified.

In other words, dpkg-divert is primarily for local administrators,
non-Policy-compliant local packages that are doing unusual things, and the
occasional rare problem that requires special coordination between
packages, not something that Debian packages should be doing to other
packages without explicit coordination.

The rule about systemd and udev files doesn't entirely fall out of that
statement, so we can still include a specific statement about them, noting
that drop-ins and masking make dpkg-divert unnecessary (and those
facilities produce better tool behavior) and therefore it should not be
used.

So, ideally, the way I'd prefer to move forward is for us to add a new
section to the main Policy manual on diversions (probably 10.11), document
that this is primarily a tool for local administrators and local packages
to override the behavior of Debian, and that its use between Debian
packages should be rare, should involve coordination between the packages,
and should only be used to solve problems that cannot be handled through
other facilities such as alternatives or package-specific tools like
systemd's support for drop-ins and masking. And then explicitly call out
systemd and udev configuration as cases where dpkg-divert should not be
used, alongside conffiles and critical system files.

--
Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>

[Luca Boccassi]

The problem is that when there are udev/systemd bugs they get directed
to the systemd team, not to the package shipping a divert, because
diverts are essentially invisible to normal users. It is already very
difficult and very time consuming to support these packages,
especially udev because we are essentially dealing with the
intersection of an infinite combination of hardware and software, and
anything that makes our lives harder is detrimental to the project at
large. If there was a significant, or even any benefit at all, then it
could be discussed, but I fail to see any. I do not find the
theoretical point about multiple diversions very compelling - we use
different mechanisms for different things all the time, but more
importantly such a case simply has never surfaced in the 10 past years
or so since we ship systemd by default, and longer since we ship udev.

Moreover, as upstream developer, I can guarantee you that masking via
diversion like this is NOT supported, and there could be more bugs
lurking, and we categorically do not intend to support or help with
such cases. I have no intention to spend any time investigating
whether such bugs exist and document them. Therefore, given there is
only one case in the whole distro so impact is minimal, the best
option to me seems to flat out forbid it, so that we never get into
that bad situation in the first place.

Kind regards,
Luca Boccassi

[Luca Boccassi]

On Mon, 8 May 2023 at 16:48, Russ Allbery <r...@debian.org> wrote:
>
> I think your X-Debbugs-Cc was syntactically invalid and thus didn't work.
> I manually added in the other addresses in this reply.

Thanks - email is hard!

I'd like this to go a step further - rules, units and so on can (and
must!) be shipped by other packages, not just from src:systemd.
But as I mentioned in the other reply, bugs come to the systemd bug
tracker most often, and make our lives more difficult, even if it's a
third package that ships the configuration.
So, I'd very much want to make the rule stronger for this use case,
and forbid it even if the respective maintainers agree between
themselves that package A's unit should be diverted by package B's,
because ultimately both A and B are feeding configuration into
systemd/udev/etc, and this is just not a supported mechanism to apply
such changes.

> In other words, dpkg-divert is primarily for local administrators,
> non-Policy-compliant local packages that are doing unusual things, and the
> occasional rare problem that requires special coordination between
> packages, not something that Debian packages should be doing to other
> packages without explicit coordination.
>
> The rule about systemd and udev files doesn't entirely fall out of that
> statement, so we can still include a specific statement about them, noting
> that drop-ins and masking make dpkg-divert unnecessary (and those
> facilities produce better tool behavior) and therefore it should not be
> used.
>
> So, ideally, the way I'd prefer to move forward is for us to add a new
> section to the main Policy manual on diversions (probably 10.11), document
> that this is primarily a tool for local administrators and local packages
> to override the behavior of Debian, and that its use between Debian
> packages should be rare, should involve coordination between the packages,
> and should only be used to solve problems that cannot be handled through
> other facilities such as alternatives or package-specific tools like
> systemd's support for drop-ins and masking. And then explicitly call out
> systemd and udev configuration as cases where dpkg-divert should not be
> used, alongside conffiles and critical system files.

Ok, I can look at adding 10.11 - I very naively searched for existing
paragraphs mentioning diverts and found the one I extended, I did not
realize it was not proper part of Policy, thanks for the pointer.

Kind regards,
Luca Boccassi

[Russ Allbery]

Sam Hartman <hart...@debian.org> writes:

> In cases where the change being made is permitted by policy, I think you
> have made a compelling case to vastly prefer the native systemd and udev
> mechanisms to dpkg-divert. I don't think that your case is strong
> enough to forbid dpkg-divert.

> As far as I can tell reading your reasoning, dpkg-divert *works fine*.
> It just gives surprising results to someone coming from the systemd
> universe.

I think (and please correct me, Luca, if I'm wrong) that Luca is trying to
declare, on behalf of the systemd maintainers, that this method of
disabling a systemd configuration file is unsupported and may not work.
To me that does warrant a Policy "should not" even if in specific
situations it works currently, because it implies that this approach is
fragile and may well stop working or cause bugs in the future with no
further notification (since that's essentially the definition of
unsupported).

Also, even apart from that, I personally would support a Policy "should
not" for using diversions in any case where another mechanism is available
that solves the same problem. Diversions are a low-level tool with a lot
of sharp edges and should be used very carefully and avoided when there is
some other approach available.

> But consider a package that diverts several resources, several of them
> systemd and several of them not systemd. The maintainer might
> legitimately want to use the same mechanism for all the
> overriding/masking so that systemd resources and non-systemd resources
> were handled the same.

I'm not really convinced by this the way that I would be if we were
talking about alternatives. With alternatives, the slave links mean that
managing a group of similar changes together is important, but dpkg-divert
has no equivalent and every diversion already has to be maintained
separately. Given that, I think the burden of asking people to use
masking instead of diversion for systemd configuration files is a fairly
minor request, so I weigh the problems on the systemd side higher than
what feels like a modest convenience.

[Luca Boccassi]

On Mon, 8 May 2023 at 18:31, Russ Allbery <r...@debian.org> wrote:
>
> Sam Hartman <hart...@debian.org> writes:
>
> > In cases where the change being made is permitted by policy, I think you
> > have made a compelling case to vastly prefer the native systemd and udev
> > mechanisms to dpkg-divert. I don't think that your case is strong
> > enough to forbid dpkg-divert.
>
> > As far as I can tell reading your reasoning, dpkg-divert *works fine*.
> > It just gives surprising results to someone coming from the systemd
> > universe.
>
> I think (and please correct me, Luca, if I'm wrong) that Luca is trying to
> declare, on behalf of the systemd maintainers, that this method of
> disabling a systemd configuration file is unsupported and may not work.
> To me that does warrant a Policy "should not" even if in specific
> situations it works currently, because it implies that this approach is
> fragile and may well stop working or cause bugs in the future with no
> further notification (since that's essentially the definition of
> unsupported).
>
> Also, even apart from that, I personally would support a Policy "should
> not" for using diversions in any case where another mechanism is available
> that solves the same problem. Diversions are a low-level tool with a lot
> of sharp edges and should be used very carefully and avoided when there is
> some other approach available.

Yes, that is precisely what I meant.

I have applied your suggestions and added a 10.10 chapter that has a
generic 'should' rule, and a more strict 'must' rule for systemd
files. I am pushing to the same branch, if you prefer me to attach
directly to the mail too let me know and I can do that, otherwise the
branch is on Salsa:

https://salsa.debian.org/bluca/policy/-/tree/systemd_overrides

I kept the wording for the dpkg-divert appendix too, because people
are bound to find it when googling, so as long as it's there I think
it should mention this too. Once it gets removed/reworked it can go.

On request from Marco, the kmod maintainer, I've also added the same
constraint for modprobe.d/ files, for exactly the same reason, as kmod
supports overrides, drop-ins and so on. I've kept it as a separate
commit on top of the other changes, given I am not involved with kmod
directly.

Kind regards,
Luca Boccassi

[Luca Boccassi]

On Mon, 8 May 2023 at 20:00, Sean Whitton <spwh...@spwhitton.name> wrote:
>
> Hello,

>
> On Mon 08 May 2023 at 08:48AM -07, Russ Allbery wrote:
>
> > In other words, dpkg-divert is primarily for local administrators,
> > non-Policy-compliant local packages that are doing unusual things, and the
> > occasional rare problem that requires special coordination between
> > packages, not something that Debian packages should be doing to other
> > packages without explicit coordination.
> >
> > The rule about systemd and udev files doesn't entirely fall out of that
> > statement,
>

> Hmm, could you explain why?

>
> > so we can still include a specific statement about them, noting that
> > drop-ins and masking make dpkg-divert unnecessary (and those
> > facilities produce better tool behavior) and therefore it should not
> > be used.
> >
> > So, ideally, the way I'd prefer to move forward is for us to add a new
> > section to the main Policy manual on diversions (probably 10.11), document
> > that this is primarily a tool for local administrators and local packages
> > to override the behavior of Debian, and that its use between Debian
> > packages should be rare, should involve coordination between the packages,
> > and should only be used to solve problems that cannot be handled through
> > other facilities such as alternatives or package-specific tools like
> > systemd's support for drop-ins and masking. And then explicitly call out
> > systemd and udev configuration as cases where dpkg-divert should not be
> > used, alongside conffiles and critical system files.
>

> I don't mean to dismiss the significant impact on the systemd
> maintainers that's being claimed, but specifically calling out udev and
> systemd configuration files seems strangely specific, for Policy, to me.
>
> The general prohibition seems justified, and then I would be inclined to
> use the systemd and udev configuration files case as an example, which
> explains and justifies the existence of the rule. So instead of
>
> [don't use dpkg-divert on other packages's files] In particular,
> packages should not divert systemd and udev configuration files.
> Doing so leads to confusing command output.
>
> it seems more natural to me to do something like
>
> [don't use dpkg-divert on other packages's files] For example,
> diverting another package's systemd units or udev configuration
> files can result in confusing command output.
>
> If this was a mistake that maintainers seemed to habitually make, the
> former would make sense, but so far I think we have just one or a few
> concrete examples, and so the correct thing to do seems to me to be to
> add normative language for only the general case.

Hi,

The specific difference, for which I think an explicit call out is
needed, is because these config files are shipped by some packages but
are not used _by_ them, they are consumed by systemd (or udev, or
kmod, etc). Specifically, if package A ships a.service, and package B
overrides it, even if the maintainers of A and B agree, that's still
not good enough for me, as they are really affecting systemd, which is
the consumer and the provider of the interface they are using, and
ultimately the first port of call for bug reports. This is especially
true for udev.

So in my latest revision of the patch, the general rule is as
requested by Russ and as you mention it, but there is an explicit,
stricter rule to cover this case, which is important to me. Policy
calls out core component software in many places, such as dpkg, and
systemd is already mentioned in other parts of the policy, so it did
not seem too far-fetched to me.

I am of course open to re-wording, adjustments, etc as deemed necessary.

Changeset at: https://salsa.debian.org/bluca/policy/-/tree/systemd_overrides

Kind regards,
Luca Boccassi

[Russ Allbery]

Sean Whitton <spwh...@spwhitton.name> writes:
> On Mon 08 May 2023 at 08:48AM -07, Russ Allbery wrote:

>> In other words, dpkg-divert is primarily for local administrators,
>> non-Policy-compliant local packages that are doing unusual things, and
>> the occasional rare problem that requires special coordination between
>> packages, not something that Debian packages should be doing to other
>> packages without explicit coordination.

>> The rule about systemd and udev files doesn't entirely fall out of that
>> statement,

> Hmm, could you explain why?

It didn't fall out of the above statement because the systemd unit file
may not be shipped with the systemd package, but by some other random
package, so you could have an explicit coordination with the package that
provides the unit file but still be doing something that the systemd
maintainers don't want to support.

I think it does fall out of the somewhat squishier statement that you
shouldn't use diversions when there's some other available mechanism to
accomplish the same goal.

> I don't mean to dismiss the significant impact on the systemd
> maintainers that's being claimed, but specifically calling out udev and
> systemd configuration files seems strangely specific, for Policy, to me.

I think they're a special case of the general rule that if there's some
mechanism other than diversions to do what you want, you should use it
instead, but it's such a common special case that we should call it out
explicitly, particularly since a lot of people right now don't seem to
know about masking or drop-ins. So in other words, I think I basically
agree with this, but I think it's worth spending some words on systemd and
udev, more as a communication strategy than anything else.

[Guillem Jover]

On Mon, 2023-05-08 at 08:48:49 -0700, Russ Allbery wrote:
> […] I suspect Policy should say something stronger and more general,

> namely that no package in Debian should divert a file from another package
> unless this is arranged cooperatively between the packages to solve some
> specific (unusual) problem.

,--- §3.9 ---
You should not use "dpkg-divert" on a file belonging to another
package without consulting the maintainer of that package first. When
adding or removing diversions, package maintainer scripts must provide
the "--package" flag to "dpkg-divert" and must not use "--local".
`---

> To me, this feels similar to the case of one
> package modifying the configuration files of another package, where we
> explicitly prohibit one package modifying the configuration of another
> package except through an interface provided by the package whose
> configuration is being modified.

,--- §10.7.4 ---
If the configuration file cannot be shared as described above, the
packages must be marked as conflicting with each other. Two packages
that specify the same file as a "conffile" must conflict. This is an
instance of the general rule about not sharing files. Neither
alternatives nor diversions are likely to be appropriate in this case;
in particular, "dpkg" does not handle diverted "conffile"s well.
`---

Regards,
Guillem

[Russ Allbery]

Guillem Jover <gui...@debian.org> writes:
> On Mon, 2023-05-08 at 08:48:49 -0700, Russ Allbery wrote:

>> […] I suspect Policy should say something stronger and more general,
>> namely that no package in Debian should divert a file from another
>> package unless this is arranged cooperatively between the packages to
>> solve some specific (unusual) problem.

> ,--- §3.9 ---
> You should not use "dpkg-divert" on a file belonging to another
> package without consulting the maintainer of that package first. When
> adding or removing diversions, package maintainer scripts must provide
> the "--package" flag to "dpkg-divert" and must not use "--local".
> `---

Oh, thank you! I had completely forgotten that we said something about
this under maintainer scripts.

That doesn't entirely cover this case (because systemd and udev may not be
"that package" in this sense), but it covers much of the general case.

[Luca Boccassi]

Would you like me to reword/move the new snippet?

[Luca Boccassi]

On Tue, 09 May 2023 00:31:20 +0100 Luca Boccassi <bl...@debian.org>
wrote:

Gentle ping. Any suggestion on how to reword/change the proposal?

[Luca Boccassi]

On Wed, 10 May 2023 23:40:44 +0100 Luca Boccassi <bl...@debian.org>

With the only occurrence now removed, this policy change should now be
impactless: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035667

[Luca Boccassi]

On Sun, 4 Jun 2023 at 11:54, Sean Whitton <spwh...@spwhitton.name> wrote:
>
> Hello Luca,

>
> On Mon 08 May 2023 at 08:07PM +01, Luca Boccassi wrote:
>
> > The specific difference, for which I think an explicit call out is
> > needed, is because these config files are shipped by some packages but
> > are not used _by_ them, they are consumed by systemd (or udev, or
> > kmod, etc). Specifically, if package A ships a.service, and package B
> > overrides it, even if the maintainers of A and B agree, that's still
> > not good enough for me, as they are really affecting systemd, which is
> > the consumer and the provider of the interface they are using, and
> > ultimately the first port of call for bug reports. This is especially
> > true for udev.
> >
> > So in my latest revision of the patch, the general rule is as
> > requested by Russ and as you mention it, but there is an explicit,
> > stricter rule to cover this case, which is important to me. Policy
> > calls out core component software in many places, such as dpkg, and
> > systemd is already mentioned in other parts of the policy, so it did
> > not seem too far-fetched to me.
>

> I'm afraid I'm not convinced. I'd second a patch where systemd is used
> as an example of the rule, as I suggested.

The existing policy is too weak for this case, ie: it's a "should". It
needs to be a "must" for these specific cases. Also the existing
policy only covers diverting from other packages, not from 'self' -
that needs to be forbidden too. There was one such example,
iptables-persistent, and it has been fixed in Bookworm, so to be clear
this is a zero-net-effect policy change, ie, no packages will suddenly
become rc-buggy, as the two existing instances have already been
fixed.

If you prefer, I can reword the general rule to be stricter, ie:
"packages must not use diversions where native mechanisms are
available" or so. Would this be better?

> Thank you for the additional commit regarding kmod. It is good to have
> been made aware of issue, but let's discuss it in a separate bug after
> making this change -- the considerations might be quite different.

>
> On Tue 09 May 2023 at 12:31AM +01, Luca Boccassi wrote:
>
> > On Mon, 08 May 2023 14:14:30 -0700 Russ Allbery <r...@debian.org> wrote:
> >
> >> Oh, thank you! I had completely forgotten that we said something
> >> about this under maintainer scripts.
> >>
> >> That doesn't entirely cover this case (because systemd and udev may
> >> not be "that package" in this sense), but it covers much of the
> >> general case.
> >
> > Would you like me to reword/move the new snippet?
>

> Yes, thank you. I will review the new version.

Any specific suggestions? IE, where it should be, etc.

Kind regards,
Luca Boccassi

[Luca Boccassi]

In the interest of speeding things up a bit, I've done some rewording
as suggested - moved to the exiting chapter, and use the systemd files
only as an example:

https://salsa.debian.org/bluca/policy/-/commit/5058bd2f8c742c3d8695e2c98ee3a597d431ffd7

Off-topic - any reasons MRs are disabled on the policy repo? It would
be much nicer and quicker to use the Gitlab review process I think,
like we do for other packages.

Kind regards,
Luca Boccassi

[Bill Allombert]

On Sun, Jun 04, 2023 at 12:25:54PM +0100, Luca Boccassi wrote:
> If you prefer, I can reword the general rule to be stricter, ie:
> "packages must not use diversions where native mechanisms are
> available" or so. Would this be better?

"native mechanisms" seems to vague.

Cheers,
--
Bill. <ball...@debian.org>

Imagine a large red swirl here.

[Luca Boccassi]

On Sun, 4 Jun 2023 19:39:49 +0200 Bill Allombert <ball...@debian.org>
wrote:

> On Sun, Jun 04, 2023 at 12:25:54PM +0100, Luca Boccassi wrote:
> > If you prefer, I can reword the general rule to be stricter, ie:
> > "packages must not use diversions where native mechanisms are
> > available" or so. Would this be better?
>
> "native mechanisms" seems to vague.

Well, I originally had written precisely what I meant, but was told on
this thread to "add normative language for only the general case"
instead, so I've done that. That's always going to sound vague. I
cannot do both at the same time.

As you can see from the latest revision, right now the rule is general
but it's immediately followed by a very concrete and documented
example. Open to precise suggestions on how to improve it.

[Luca Boccassi]

On Tue, 6 Jun 2023 at 11:58, Sean Whitton <spwh...@spwhitton.name> wrote:
>
> Hello,
>

> I think what's a bit peculiar here is using "must" for a case where
> there might be package-specific exceptions. In other cases, Policy uses
> "should" for these cases. Typically "must" rules are simple and
> completely determinate.
>
> Maintainers do take our "should"s seriously -- let's use that here.
> It can always be strengthened later.

My understanding is that "should" is effectively optional, ie: it is
not enough to make a package rc-buggy, and while they are generally
followed, there is no actual hard requirement to do so. That is not
enough for me and for this use case. So again my goal here is to make
an hypothetical future dpkg-divert or alternative of a unit file
(wherever it is shipped from) instantly and unambiguously rc-buggy,
with no leeway (again there are no longer any such cases in the
archive as of bookworm, so nothing is affected). If you can suggest an
alternative appropriate wording that can achieve the same effect, that
would be fine to me - I don't mind how it's written, as long as it
meets this requirement.

> > Diversions and alternatives should be used primarily as a tool for
> > local administrators and local packages to override the behaviour of
> > Debian. Its use between Debian packages should be rare, should involve
> > coordination between the packages and their maintainers, and must only

> > be used to solve problems that cannot be handled through other

> > facilities or native mechanisms. In other words, packages in Debian
> > must not divert a file from another package unless this is arranged
> > cooperatively between the packages to solve some specific and unusual
> > problem.
>
> How about:
>
> Diversions and alternatives are primarily tools for local
> administrators and local packages to override Debian's default
> behaviour. Maintainers should prefer to use other, package-specific
> facilities for overriding configuration, such as systemd's unit file
> overrides, wherever possible.
>
> If diversions and alternatives are to be used, maintainers should
> co-ordinate with the maintainers of all the packages involved. For
> example, configuration files used by systemd components should not
> be diverted with dpkg-divert or the alternatives system without
> agreement between not only the maintainers of the packages that ship
> the files, but also the maintainers of the relevant systemd
> components.

As above, as long as this wording makes any offending package
rc-buggy, it's fine by me, but my understanding is that using 'should'
doesn't guarantee that. Please correct me if I'm wrong here.

> I would also like to suggest "their own mechanisms for overriding parts
> of configuration" over "native overriding mechanisms" in the appendices.

Sure that sounds good.

Kind regards,
Luca Boccassi

[Dominik George]

> The whole project is moving toward git and Salsa

Sorry for the noise, but as you are clearly misattributing this to me (I am part of the project, so "the whole project" includes me):

I am not, and do not want to, move bugs and patches to Git and Salsa. I consider it a huge advantage of Debian that I can contribute limitless with something as barrierfree as an e-mail.

If you voice your opinion, please do not impose it on me. Thanks!

-nik

[Luca Boccassi]

Ok, how about: "the whole project, minus nature...@debian.org who
appears to be unfamiliar with the concept of hyperboles, is moving
toward git and Salsa". Better?

Snarks aside, allowing merge requests to be open on Salsa in
_addition_ to attachments to the BTS, as the vast majority of other
packages already do, doesn't take away anything from you, but would
add quite a lot for the rest of us, who find ourselves very limited
and very much barrier-ized by clunky, old and painful email-based
processes.

Kind regards,
Luca Boccassi

[Bill Allombert]

I do not think there is consensus that this should be a RC bug outside of
a case by case basis. It is already awkward to mention systemd explicitly
in this paragraph.

The diversion system is made precisely to work around other packages behavior,
this is a feature not a bug. That it should only be used as last resort, I
think everyone agree. But when it is, it should not be a RC bug.

[Luca Boccassi]

On Tue, 6 Jun 2023 14:03:41 +0200 Bill Allombert <ball...@debian.org>
wrote:

This is a technical matter, I'm not sure what 'consensus' means in this
context? Things _will not work_ as expected by shoe-horning dpkg's
overrides onto systemd mechanisms, they _will_ break in weird and
unexpected ways, and we as maintainers _will not support it_ - whether
somebody else agrees or disagrees with this won't change any of it.

It means that, when such bug happens, and it turns out there's a
divert/override in the way, the bug will be an immediate close+wontfix
and the reporter will be told they are on their own. And that's a best
case scenario where it is all apparent, while actually it's more likely
this will be hidden away and require a large waste of time to figure
out. Isn't it better to ensure this cannot happen in the first place?
Once more: there are no such cases as of bookworm in the archive, so
the impact today is nil. Wouldn't it be prudent to ensure it stays that
way in the strictest possible terms?

[Bill Allombert]

On Tue, Jun 06, 2023 at 01:38:51PM +0100, Luca Boccassi wrote:
> > The diversion system is made precisely to work around other packages
> behavior,
> > this is a feature not a bug. That it should only be used as last
> resort, I
> > think everyone agree. But when it is, it should not be a RC bug.
>
> This is a technical matter, I'm not sure what 'consensus' means in this
> context? Things _will not work_ as expected by shoe-horning dpkg's
> overrides onto systemd mechanisms, they _will_ break in weird and
> unexpected ways, and we as maintainers _will not support it_ - whether
> somebody else agrees or disagrees with this won't change any of it.

Consensus is the way the Debian Policy update process works.
But you do not need changes in Policy to report bugs about package that breaks
others, there is the "grave" severity already.

[Luca Boccassi]

That does not help, given currently policy allows it, without changes
they could just say "policy allows me, so go fix it yourself". What
then?

[Russ Allbery]

Luca Boccassi <bl...@debian.org> writes:

> --- a/policy/ap-pkg-alternatives.rst
> +++ b/policy/ap-pkg-alternatives.rst
> @@ -24,3 +24,7 @@ See the :manpage:`update-alternatives(8)` man page for details.
> If ``update-alternatives`` does not seem appropriate you may wish to
> consider using diversions instead.
>
> +Do not attempt to use alternatives for files belonging or used by components
> +that support native overriding mechanisms, such as ``systemd`` unit files. Read
> +:doc:`ch-binary` for more information.

"Do not attempt" in US English reads a little weirdly (I know you're
copying an existing UK English phrasing). I instead suggest:

Do not use alternatives for ``systemd`` configuration files. See
:doc:`ch-binary` for more information.

(See below for why this is systemd-specific.) This doesn't use our normal
normative language conventions, which I think is correct since this is a
non-normative appendix.

> diff --git a/policy/ap-pkg-diversions.rst b/policy/ap-pkg-diversions.rst
> index fe360d1..09367d7 100644
> --- a/policy/ap-pkg-diversions.rst
> +++ b/policy/ap-pkg-diversions.rst
> @@ -81,3 +81,7 @@ when the file does not exist.
> Do not attempt to divert a conffile, as ``dpkg`` does not handle it
> well.
>
> +Do not attempt to divert files belonging or used by components that support
> +native overriding mechanisms, such as ``systemd`` unit files. Read
> +:doc:`ch-binary` for more information.

Similarly here:

Do not use diversions for files that have their own native override
mechanisms, such as ``systemd`` unit files. See :doc:`ch-binary` for
more information.

> diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
> index e517f26..e36d028 100644
> --- a/policy/ch-binary.rst
> +++ b/policy/ch-binary.rst
> @@ -371,6 +371,37 @@ against earlier versions of something that previously did not use
> ``update-alternatives``; this is an exception to the usual rule that
> versioned conflicts should be avoided.)
>
> +Diversions and alternatives should be used primarily as a tool for local
> +administrators and local packages to override the behaviour of Debian. Its use
> +between Debian packages should be rare, should involve coordination between the
> +packages and their maintainers, and must only be used to solve problems that
> +cannot be handled through other facilities or native mechanisms.

I think this not correct for alternatives. They are intended for use
between Debian packages, so we need to distinguish between alternatives
and diversions here. This might be clearer if we created a subsection for
alternatives and diversions and added a bit of additional context from the
appendices (and ideally removed the appendices). That's not something to
do on this bug, just noting mostly for Sean and future work.

I think there's also a bit more justification than we need. The
justification is useful for discussing the Policy bug, but once we've
decided on an approach, I think we can just point at the correct
alternative mechanism.

How about something like this:

Diversions are primarily intended as a tool for local administrators
or local packages to override the behavior of Debian. While there are
some circumstances where one Debian package may need to divert a file
of another Debian package, those circumstances are rare and diversions
should only be used as a last resort when no other suitable mechanism
exists. Diversion of a file in one Debian package by another Debian
package should be coordinated between the maintainers of those
packages.

Diversions must not be used when a suitable override mechanism that
accomplishes the same goal is already available.

One specific case of that rule is that configuration files used by
``systemd`` components, such as `units,
<https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Description>`_
`udev rules,
<https://www.freedesktop.org/software/systemd/man/udev.html#Rules%20Files>`_
`tmpfiles.d,
<https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html#Configuration%20Directories%20and%20Precedence>`_
`modules-load.d,
<https://www.freedesktop.org/software/systemd/man/modules-load.d.html#Configuration%20Format>`_,
`sysusers
<https://www.freedesktop.org/software/systemd/man/sysusers.d.html#Configuration%20Directories%20and%20Precedence>`_
and other such files, including those specific to systemd daemons
(e.g.: `/etc/systemd/system.conf).
<https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html>`_
must not be diverted by any Debian package. Instead, use masking and
drop-ins.

"masking" and "drop-ins" here should ideally be links to the relevant
documentation.

We then have to figure out what to say about alternatives. I think the
above rule is too strong for alternatives; it's often convenient to use
slave alternatives even in places where there may be a native override
mechanism, although it sounds like the systemd maintainers think this is
always inappropriate for systemd. Here, I wonder if we do just want a
systemd-specific rule. The systemd-specific rule shouldn't reference
masking or drop-ins, though, I think, since I can't see how you would do
what alternatives does using masking and drop-ins (one package would
always override the other, I think, rather than allowing
update-alternatives to choose between them).

I think the correct observation here is that alternatives really don't
make any sense for daemons. You don't want to pick between implemenations
of a system service using alternatives for a few reasons, but one of the
most obvious is that running update-alternatives to change the preferred
implementation doesn't stop and restart the service (nor should it know to
do that), so you end up putting your system in a confusing state.

Therefore, I wonder if we want to just say something simpler here about
alternatives:

Alternatives must never be used for ``systemd`` configuration files.
The alternatives system does not know how to apply changes to services
when updating alternatives, so the resulting behavior would be
confusing and unpredictable.

I keep wanting to suggest an alternative, but I'm not sure there's an
obvious alternative to suggest (the options are going to be very
situation-specific), so I'm inclined to just leave it at that.

[Russ Allbery]

Luca Boccassi <bl...@debian.org> writes:

> Snarks aside, allowing merge requests to be open on Salsa in _addition_
> to attachments to the BTS, as the vast majority of other packages
> already do, doesn't take away anything from you, but would add quite a
> lot for the rest of us, who find ourselves very limited and very much
> barrier-ized by clunky, old and painful email-based processes.

Merge requests are useful when the goal is to be able to merge the merge
request as-is at the end of some code review process. My guess is that
this happens about 20% of the time with Policy. The other 80% of the time
I tweak wording or phrasing or formatting when applying the change, and
also add an upgrading-checklist entry and other bookkeeping that I don't
want to have to explain to everyone proposing new wording.

Policy is not a program and we're not taking source code changes. The goal
of the discussion is to converge on the semantics of what we are going to
say, which often requires multi-paragraph discussions where the normal
tools of email such as quoting are more useful. I find trying to have an
email-style discussion in Salsa to be annoying and tedious. It also
fragments the record, whereas having it in the bug means I can, at any
time, load the entire bug into Gnus and re-read the discussion of how we
arrived at the decision in the same tool that I use for reading all other
Debian discussions.

[Dominik George]

> Ok, how about: "the whole project, minus nature...@debian.org who
> appears to be unfamiliar with the concept of hyperboles, is moving
> toward git and Salsa". Better?

No.

Your "hyperbole" very much read as "Come on, minority who cares about
the mail workflow, you're weird anachronists, get onto the Salsa train already!"

So that's what I am criticizing.

-nik

[Luca Boccassi]

Sure, and you just coincidentally happened to leave out the part that
says the exact opposite:

> Snarks aside, allowing merge requests to be open on Salsa in
> _addition_ to attachments to the BTS, as the vast majority of other
> packages already do, doesn't take away anything from you, but would
> add quite a lot for the rest of us, who find ourselves very limited
> and very much barrier-ized by clunky, old and painful email-based
> processes.

Which means you are actually doing what you accuse me of - you are not
content with being able to keep using mail workflows for yourself, you
want to actively stop everybody from being able to use Salsa too.

[Luca Boccassi]

It depends on your pain threshold. For me it's much easier and simpler
and less painful to do all of that on Salsa, as opposed to wade through
a bunch of random emails and scrolling up and down through the BTS. I'm
sure there were some suggestions earlier in the thread, that I now have
forgotten about, and they will be lost because there's no way I'm going
back wading through dozens of random replies to figure out what is
applied and what is not. On Salsa, this would be an 'unresolved' review
comment that I can click 'solved' once handled.

I'm not suggesting that you stop using emails to send your changes -
I'm simply asking to reconsider making policy work like the vast
majority of other parts of Debian, and _also_ accepts merge requests on
Salsa, in _addition_ to emails. Then the submitter can choose. I
suspect you are going to get a lot more contributions this way, that is
certainly the experience from other teams that have enabled Salsa from
what I can hear (eg: kernel team).

[Bill Allombert]

On Tue, Jun 06, 2023 at 09:36:31PM +0100, Luca Boccassi wrote:
> On Tue, 6 Jun 2023 20:51:46 +0200 Dominik George
> <nature...@debian.org> wrote:
> > > Ok, how about: "the whole project, minus
> nature...@debian.org who
> > > appears to be unfamiliar with the concept of hyperboles, is moving
> > > toward git and Salsa". Better?
> >
> > No.
> >
> > Your "hyperbole" very much read as "Come on, minority who cares about
> > the mail workflow, you're weird anachronists, get onto the Salsa
> train already!"
> >
> > So that's what I am criticizing.
>
> Sure, and you just coincidentally happened to leave out the part that
> says the exact opposite:

This is beside the point. Your problematic statement was
"The whole project is moving toward git and Salsa ".
This is not conducive of productive interaction.

[Bill Allombert]

That simply not how Policy works.
Policy is for promoting interoperability and documenting current practices.
"Policy is not a stick to beat people with" as Manoj used to say.

If you are suggesting a policy change so that you can report RC bugs on other
packages, you are on the wrong track.

[Luca Boccassi]

On Wed, 7 Jun 2023 00:01:42 +0200 Bill Allombert <ball...@debian.org>
wrote:

It was not problematic at all, unless you are deliberately trying to
see problems where there are none. It was an hyperbole: a huge number
of packages and teams are using effectively Salsa workflows for
collaboration today. Are you denying this?

[Luca Boccassi]

On Wed, 7 Jun 2023 00:02:06 +0200 Bill Allombert <ball...@debian.org>

No, I am suggesting a policy change so that we do not end up in a messy
and unmaintainable situation, which I thought was one of the goals. The
current practice is that packages are not using diversions and
alternatives to take over systemd files. As I have already specified a
number of times, the number of packages that need changes following
this change is zero - because I have _already_ done the required work
to make it so. After having done this work for Bookwrom, I am
translating this current practice into policy, to ensure we don't
regress.

[Russ Allbery]

Luca Boccassi <bl...@debian.org> writes:

> It depends on your pain threshold.

Salsa is worse for me for working on Policy in pretty much every respect
than email with patches. That's not a statement about my pain threshold.
It's a fundamental disagreement with you about which tools work better for
this specific type of maintenance.

> For me it's much easier and simpler and less painful to do all of that
> on Salsa, as opposed to wade through a bunch of random emails and
> scrolling up and down through the BTS.

I understand that you don't like email. The Policy process is optimized
for the Policy editors, rather than for the preferences of people who are
working on a specific change that they care about. I appreciate that this
can be annoying, but it's fairly typical for open source work. See, for
example, the Linux kernel, which likewise doesn't take merge requests, no
matter how inconvenient that may be for contributors.

> I'm sure there were some suggestions earlier in the thread, that I now
> have forgotten about, and they will be lost because there's no way I'm
> going back wading through dozens of random replies to figure out what is
> applied and what is not.

That's fine; this is part of the job of the Policy editors.

> I'm not suggesting that you stop using emails to send your changes - I'm
> simply asking to reconsider making policy work like the vast majority of
> other parts of Debian, and _also_ accepts merge requests on Salsa, in
> _addition_ to emails.

I have considered it several times, and I considered it again as part of
this recent conversation before writing my reply.

> Then the submitter can choose. I suspect you are going to get a lot more
> contributions this way,

Maximizing contribution of merge requests doesn't help the bottleneck for
Policy changes. Policy changes are often complex and require thought and
research. Most of the patches we get are incorrect or incomplete to start
with. The bottleneck is time from people with a lot of in-depth Debian
knowledge to think deeply about a problem.

If the people who routinely provide detailed and incredibly useful
analyses of Policy problems say that they would rather use Salsa to do so,
that's important feedback and I would like to hear that feedback. But the
goal here is not to maximize development speed. It's to think hard about
a problem and get the answer correct.

[Luca Boccassi]

On Tue, 06 Jun 2023 16:46:04 -0700 Russ Allbery <r...@debian.org> wrote:
> Luca Boccassi <bl...@debian.org> writes:

> > I'm not suggesting that you stop using emails to send your changes
- I'm
> > simply asking to reconsider making policy work like the vast
majority of
> > other parts of Debian, and _also_ accepts merge requests on Salsa,
in
> > _addition_ to emails.
>
> I have considered it several times, and I considered it again as part
of
> this recent conversation before writing my reply.

Sure, it's your package so it's up to you.

[Russ Allbery]

Bill Allombert <ball...@debian.org> writes:
> On Sun, Jun 04, 2023 at 12:25:54PM +0100, Luca Boccassi wrote:

>> If you prefer, I can reword the general rule to be stricter, ie:
>> "packages must not use diversions where native mechanisms are
>> available" or so. Would this be better?

> "native mechanisms" seems to vague.

I suggested "must not be used when a suitable override mechanism that
accomplishes the same goal is already available." Does that sound okay?
It's a bit weaker and opens the door to arguments about whether the native
override mechanism accomplishes the same goal, but I think that's a
feature here rather than a bug.

[Russ Allbery]

Sean Whitton <spwh...@spwhitton.name> writes:

> I think what's a bit peculiar here is using "must" for a case where
> there might be package-specific exceptions. In other cases, Policy uses
> "should" for these cases. Typically "must" rules are simple and
> completely determinate.

I prefer that too, but in this case, it feels like must is appropriate for
at least systemd configuration files. And also, just intuitively, I feel
like must is correct when people are using diversions rather than a native
mechanism. Diversions add weird edge cases and we really shouldn't be
using them lightly.

The wording I proposed and that Luca has now adopted therefore uses must
with a caveat. Does that sound okay to you?

I think must is too strong for alternatives, where I think there are more
likely to be package-specific exceptions. But I'd like to take a harder
line on diversions, and I do think that matches how we use diversions
between two Debian packages (work around some weird situation where we
have no better option).

[Russ Allbery]

Luca Boccassi <bl...@debian.org> writes:

> There is a native alternative: aliases. We already do that today in the
> archive for the well-known 'dbus' unit. The dbus package ships the
> reference implementation, and dbus-broker which is an alternative
> implementation ships its own unit, which also lists 'dbus.service' as an
> alias. I have listed this in addition to your suggested paragraph.

Thanks! This looks good to me. I've reviewed all the bug traffic and
feel like this addresses the concerns, but not exactly in the way that
Sean, Bill, and Sam preferred. I'd therefore like to see if this works
for them or if they still feel like it's too strong before I second it.
But if I were the only one deciding, I'd second this wording. (Probably
not that surprising since you took all of my wording suggestions.)

[Bill Allombert]

It is fine with me. It is in line with the notion that diversion are a last
resort, but not precluded entirely.

[Bill Allombert]

On Tue, Jun 06, 2023 at 07:56:14PM -0700, Russ Allbery wrote:
> Sean Whitton <spwh...@spwhitton.name> writes:
>
> > I think what's a bit peculiar here is using "must" for a case where
> > there might be package-specific exceptions. In other cases, Policy uses
> > "should" for these cases. Typically "must" rules are simple and
> > completely determinate.
>
> I prefer that too, but in this case, it feels like must is appropriate for
> at least systemd configuration files. And also, just intuitively, I feel
> like must is correct when people are using diversions rather than a native
> mechanism. Diversions add weird edge cases and we really shouldn't be
> using them lightly.
>
> The wording I proposed and that Luca has now adopted therefore uses must
> with a caveat. Does that sound okay to you?

I do not think appropriate for the policy to list systemd or any
other packages specifically in this section.

If a package set up a diversion that breaks another, then it is buggy,
whatever policy say. If the diversion does not cause any breakage, there is
no purpose for policy to declare it a RC bug.

In general, policy proscription are only useful when the description of a
better mechanism is provided. But there is no place for that in this section.

It would more suitable to have a separate section or document defining the
interface between systemd and other packages, that would explain how to avoid
diversion by providing better solutions rather than direct proscriptions, with
more details than just "use native systemd configuration files".

[Luca Boccassi]

On Wed, 7 Jun 2023 12:23:15 +0200 Bill Allombert <ball...@debian.org>
wrote:

> On Tue, Jun 06, 2023 at 07:56:14PM -0700, Russ Allbery wrote:
> > Sean Whitton <spwh...@spwhitton.name> writes:
> >
> > > I think what's a bit peculiar here is using "must" for a case
where
> > > there might be package-specific exceptions.  In other cases,
Policy uses
> > > "should" for these cases.  Typically "must" rules are simple and
> > > completely determinate.
> >
> > I prefer that too, but in this case, it feels like must is
appropriate for
> > at least systemd configuration files.  And also, just intuitively,
I feel
> > like must is correct when people are using diversions rather than a
native
> > mechanism.  Diversions add weird edge cases and we really shouldn't
be
> > using them lightly.
> >
> > The wording I proposed and that Luca has now adopted therefore uses
must
> > with a caveat.  Does that sound okay to you?
>
> I do not think appropriate for the policy to list systemd or any
> other packages specifically in this section.

There are already mentions of systemd and unit files in policy before
this change, in other sections. For what reason are those fine and this
is not?

> If a package set up a diversion that breaks another, then it is
buggy,
> whatever policy say. If the diversion does not cause any breakage,
there is
> no purpose for policy to declare it a RC bug.

Earlier you wrote:

> Policy is for promoting interoperability and documenting current
practices.

This is what this change is doing - promoting interoperability by
forbidding known broken and unsupported practices, and instead steering
toward known working and supported practices. It also documents what
the current practice w.r.t. overriding and aliasing is as of Bookworm.

> In general, policy proscription are only useful when the description
of a
> better mechanism is provided.  But there is no place for that in this
section.

It is explained how to use a better mechanism? With links to all the
relevant documentation et cetera. Are you saying it's not exhaustive
enough and you want more details added? I am wary of excessively
redefining and duplicating existing documentation, especially because
it will naturally evolve (in backward-compatible ways) and any such
copy would get out of date and be confusing.

[Russ Allbery]

Bill Allombert <ball...@debian.org> writes:
> On Tue, Jun 06, 2023 at 07:56:14PM -0700, Russ Allbery wrote:

>> I prefer that too, but in this case, it feels like must is appropriate
>> for at least systemd configuration files. And also, just intuitively,
>> I feel like must is correct when people are using diversions rather
>> than a native mechanism. Diversions add weird edge cases and we really
>> shouldn't be using them lightly.

>> The wording I proposed and that Luca has now adopted therefore uses
>> must with a caveat. Does that sound okay to you?

> I do not think appropriate for the policy to list systemd or any other
> packages specifically in this section.

My rationale for listing systemd specifically is:

1. This is a common case where one package wants to change the behavior of
another and I expect it to become increasingly common as people make
more use of systemd security features. For example, I would expect
cases where the default systemd configuration for a service disallows
access to large swaths of the file system, and then installing a plugin
for that package grants additional access to specific paths used by
that plugin.

2. We understand what people are trying to do in this case and can offer
very specific guidance, whereas we can't in general for arbitrary
packages. This makes Policy more useful for packagers; instead of
seeing a general rule that they can't use diversions but being left on
their own to figure out how to solve their problem, Policy can
explicitly say "here are the mechanisms to use for this case, they can
do everything you would want to do with diversions."

I would like to add more documentation like this around systemd-related
things to Policy because systemd is complicated and has a lot of options,
so people who aren't deeply familiar with it will easily miss best
practices in its reams of very good but overwhelming documentation, and
Debian should be opinionated about steering people towards best practices
for our primary init system, just as we were very opinionated about how to
write init scripts for sysvinit.

> If a package set up a diversion that breaks another, then it is buggy,
> whatever policy say. If the diversion does not cause any breakage, there
> is no purpose for policy to declare it a RC bug.

I don't really agree with this, and I thought we had reached some
agreement in the other branch of this thread that this isn't quite
complete and it's okay to ban diversions if there's a better mechanism
available.

I think that if diversions and some other mechanism work equally well
technically, we should forbid using diversions and require using the other
mechanism. This is in part based on the long threads that came out of the
/usr-merge discussions, which made clear that diversions are a very
complicated feature with a lot of edge cases, and it's possible to get
into trouble with them because they're manipulating the packaging system
at a very low level. If there's some alternative that's less invasive, I
do feel like using diversions is a fairly serious bug because it adds to
the instability of the system.

In isolation, you could talk me into it being an important bug rather than
an RC bug, but that feels like a lot of nuance here and must feels cleaner
and more straightforward.

That being said, I'd rather back down to should than remove the
systemd-specific details, since I think the latter are quite valuable.

> In general, policy proscription are only useful when the description of
> a better mechanism is provided. But there is no place for that in this
> section.

I'm not sure I understand this statement, since describing a better
mechanism is exactly the point of that language about systemd. We link
directly to those better mechanisms (masking, drop-ins, and, for
alternatives, aliases).

I definitely agree that Policy should have a whole section devoted to
systemd and its configuration files, but I don't want to try to boil the
ocean in one bug. But yes, we should be working towards that, IMO.

[Bill Allombert]

On Wed, Jun 07, 2023 at 08:19:08AM -0700, Russ Allbery wrote:
> > In general, policy proscription are only useful when the description of
> > a better mechanism is provided. But there is no place for that in this
> > section.
>
> I'm not sure I understand this statement, since describing a better
> mechanism is exactly the point of that language about systemd. We link
> directly to those better mechanisms (masking, drop-ins, and, for
> alternatives, aliases).

What I meant was that this section in not the right place for systemd or other
software configuration detail, because nobody will look for systemd
configuration detail in the dpkg-divert section.

> I definitely agree that Policy should have a whole section devoted to
> systemd and its configuration files, but I don't want to try to boil the
> ocean in one bug. But yes, we should be working towards that, IMO.

Yes that what I wanted to say.

[Holger Levsen]

On Wed, Jun 07, 2023 at 08:19:08AM -0700, Russ Allbery wrote:
> I would like to add more documentation like this around systemd-related
> things to Policy because systemd is complicated and has a lot of options,
> so people who aren't deeply familiar with it will easily miss best
> practices in its reams of very good but overwhelming documentation, and
> Debian should be opinionated about steering people towards best practices
> for our primary init system, just as we were very opinionated about how to
> write init scripts for sysvinit.

[...]

> I definitely agree that Policy should have a whole section devoted to
> systemd and its configuration files, but I don't want to try to boil the
> ocean in one bug. But yes, we should be working towards that, IMO.

yes! (to everything in said email.) thank you.


--
cheers,
Holger

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

Only change is constant.

[Bill Allombert]

On Tue, Jun 06, 2023 at 11:34:17PM +0100, Luca Boccassi wrote:
> On Wed, 7 Jun 2023 00:01:42 +0200 Bill Allombert <ball...@debian.org>

> > This is beside the point. Your problematic statement was
> > "The whole project is moving toward git and Salsa ".
> > This is not conducive of productive interaction.
>
> It was not problematic at all, unless you are deliberately trying to
> see problems where there are none. It was an hyperbole: a huge number
> of packages and teams are using effectively Salsa workflows for
> collaboration today. Are you denying this?

This is not important. What matters is that this statement could be read as
excluding from the project or belittling developers that are not moving toward
Salsa. This is inappropriate. Even assuming it was not your intent, such
statements creates friction in the project by making people uncomfortable
without any benefit, and so should be avoided.

[Russ Allbery]

Sean Whitton <spwh...@spwhitton.name> writes:

> I still find myself feeling queasy about adding this must-with-caveat.
> It feels like with the caveat you're trying to get something between
> "must" and "should", but then rather than build down from "must", why
> not build up from "should"? I would like to propose:

> Maintainers should strongly prefer using other overriding
> mechanisms, instead of diversions, whenever those other mechanisms
> are sufficient to accomplish the same goal. In other words,
> diversions in packages should be considered a last resort.

This sounds good to me. The argument for building up from should instead
of down from must seems compelling.

[Luca Boccassi]

That essentially means it's fine to use diversions and ship releases
using them, so that's exactly what will happen as per Murphy's law.

> I think that this accomplishes everything, in terms of guidance for
> our
> maintainers, that the must-with-caveat wording does. But it avoids
> saying that using diversions over a native mechanism in and of itself
> renders a package RC-buggy, which doesn't seem right to me.
>
> My own experience with diversions is limited, and so I accept that I
> may
> well be naively underestimating the badness of the edge cases.

Why would it not be RC-buggy? Not RC-buggy means we are happy if it
ships in a release. What does that buy us? Why wouldn't we want to
direct maintainers toward the better alternative, that is current
practice as of today, and instead let them reintroduce a mechanism that
we agree is inferior and was just removed from the distribution?

[Russ Allbery]

Luca Boccassi <bl...@debian.org> writes:

> That essentially means it's fine to use diversions and ship releases
> using them, so that's exactly what will happen as per Murphy's law.

I think we're reaching a consensus that "must" is appropriate for the
systemd configuration files, so this discussion is about how to phrase the
general guidance that isn't specific to systemd.

I'm therefore not really understanding the argument that you're trying to
make here. Are you trying to say that we should just delete everything in
Policy that isn't a "must" because it serves no useful purpose? If not,
then why are you taking this incredibly aggressive position that general
guidance is pointless unless it says "must"?

[Luca Boccassi]

Waait, maybe I misunderstood - I thought the change was requested for
all additions, including systemd files? Is that not the case? (It
would be obvious what it refers to on a merge request :-P )
If I can keep a 'must' for systemd config files, then I'm ok with changing.

Kind regards,
Luca Boccassi

[Russ Allbery]

Luca Boccassi <bl...@debian.org> writes:

> I.e.: if the attached version works, then that's good enough for me.

Seconded. Thank you for your work on multiple revisions of this patch!

[Luca Boccassi]

On Wed, 14 Jun 2023 at 16:27, Russ Allbery <r...@debian.org> wrote:
>
> Luca Boccassi <bl...@debian.org> writes:
>
> > I.e.: if the attached version works, then that's good enough for me.
>
> Seconded. Thank you for your work on multiple revisions of this patch!

Thanks for your patience and feedback!

Kind regards,
Luca Boccassi

[Holger Levsen]

On Tue, Jun 13, 2023 at 11:04:07PM +0100, Luca Boccassi wrote:
> ---
> policy/ap-pkg-alternatives.rst | 3 +++
> policy/ap-pkg-diversions.rst | 3 +++
> policy/ch-binary.rst | 35 ++++++++++++++++++++++++++++++++++
> 3 files changed, 41 insertions(+)
>
> diff --git a/policy/ap-pkg-alternatives.rst b/policy/ap-pkg-alternatives.rst
> index ffa2163..6f7780f 100644
> --- a/policy/ap-pkg-alternatives.rst
> +++ b/policy/ap-pkg-alternatives.rst
> @@ -24,3 +24,6 @@ See the :manpage:`update-alternatives(8)` man page for details.
> If ``update-alternatives`` does not seem appropriate you may wish to
> consider using diversions instead.
>
> +Do not use alternatives for ``systemd`` configuration files. See
> +:doc:`ch-binary` for more information.
> +
> diff --git a/policy/ap-pkg-diversions.rst b/policy/ap-pkg-diversions.rst
> index fe360d1..d299d04 100644
> --- a/policy/ap-pkg-diversions.rst
> +++ b/policy/ap-pkg-diversions.rst
> @@ -81,3 +81,6 @@ when the file does not exist.
> Do not attempt to divert a conffile, as ``dpkg`` does not handle it
> well.
>
> +Do not use diversions for files that have their own native override mechanisms,
> +such as ``systemd`` unit files. See :doc:`ch-binary` for more information.
> +
> diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
> index e517f26..19635e7 100644
> --- a/policy/ch-binary.rst
> +++ b/policy/ch-binary.rst
> @@ -371,6 +371,41 @@ against earlier versions of something that previously did not use
> ``update-alternatives``; this is an exception to the usual rule that
> versioned conflicts should be avoided.)
>
> +Diversions are primarily intended as a tool for local administrators or local
> +packages to override the behavior of Debian. While there are some circumstances
> +where one Debian package may need to divert a file of another Debian package,
> +those circumstances are rare and diversions should only be used as a last resort
> +when no other suitable mechanism exists. Diversion of a file in one Debian
> +package by another Debian package should be coordinated between the maintainers
> +of those packages. Maintainers should strongly prefer using other overriding
> +mechanisms, instead of diversions, whenever those other mechanisms are
> +sufficient to accomplish the same goal. In other words, diversions in packages
> +should be considered a last resort.
> +
> +One specific case of that rule is that configuration files used by
> +``systemd`` components, such as `units,
> +<https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Description>`_
> +`udev rules,
> +<https://www.freedesktop.org/software/systemd/man/udev.html#Rules%20Files>`_
> +`tmpfiles.d,
> +<https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html#Configuration%20Directories%20and%20Precedence>`_
> +`modules-load.d,
> +<https://www.freedesktop.org/software/systemd/man/modules-load.d.html#Configuration%20Format>`_,
> +`sysusers
> +<https://www.freedesktop.org/software/systemd/man/sysusers.d.html#Configuration%20Directories%20and%20Precedence>`_
> +and other such files, including those specific to systemd daemons
> +(e.g.: `/etc/systemd/system.conf).
> +<https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html>`_
> +must not be diverted by any Debian package. Instead, use `masking and drop-ins
> +<https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Description>`_.
> +
> +Alternatives must never be used for ``systemd`` configuration files. The
> +alternatives system does not know how to apply changes to services when updating
> +alternatives, so the resulting behavior would be confusing and unpredictable.
> +Instead, `aliases
> +<https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Description>`_
> +can be used to provide alternative implementations of the same named unit.
> +

seconded.

and wondering, if there should be a recommendation similar to consulting debia...@l.d.o
when introducing epochs, or...

..the other way round: should it be explicitly spelled out that, unlike for epochs,
in general there's no need to consult -devel for diversions for packages as they are
generally ment for local admins and only in very very very rare cases...

(maybe be even more verbose about "other overriding mechanisms"?!)

but in any case seconded, things can always be improved, and this is good.

--
cheers,
Holger

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

Punk ist nicht tot.
Punk trägt Maske, ist solidarisch und schützt sich und andere.
(@Kreuzpirat)