Bug#1031325: e2fsprogs 1.47.0 introduces a breaking change into Bookworm, breaking grub and making installations of Ubuntu and Debian releases via debootstrap impossible

[Adrian Bunk]

On Tue, Feb 14, 2023 at 08:46:53PM -0500, Theodore Ts'o wrote:
>...
> I will draw the analogy of building a program which links against
> glibc for Bookworm resulting in a binary that will not run on Buster.
> We expect that, and we tell people to use build chroots. This is not
> something which is actionable, and we don't hold back glibc updates
> just because you can no longer build on Debian 10.0 something that
> won't work on Debian 9.0, or 8.0.
>...
> We can change the default for mke2fs.conf file for Debian. I don't
> think it's warranted, and I'm not aware of any other distribution
> doing this. The fact that file system featuers that fix certain
> problems (such as the 2038 bug) will cause older versions the
> distribution to fail to accept that file system is always going to be
> the case. So how long do we hold back some new feature? A year? Two
> years? Five years? Ten years? Again, we don't do this with shared
> library linkages; we just tell people to use a build chroot.
>...
> So if we are to hold e2fsprogs and xfsprogs to the standard that a
> file system created by default must work on all older Debian and
> Ubuntu distributions to some arbitrary point in history,
>...

A rule of thumb is that any combination/mix of packages permitted by the
package manager from two adjacent Debian releases should work whenever
reasonably possible, since this reduces problems for our users during
an upgrade, when using backports, or when temporarily going back to the
version of a package from the previous stable due to a regression.

For normal library dependencies
Depends: libc6 (>= 2.34)
will do the right thing automatically.

e2fsprogs adding Breaks against older versions of bootloaders and other
software that lacks support for the new default might be a good idea.

The situtation is different when a relationship cannot be reliably
expressed by package dependencies.

For the kernel the answer to your "how long" is that packages should
also work with the kernel from the previous and the kernel from the
next Debian release.

Some time ago there was a discussion regarding Qt in Debian 10 using the
getrandom syscall that was not available in the kernel in Debian 8.
This was not considered supported (or supportable) since Debian 8 and
Debian 10 are not adjacent releases, but Qt in Debian 10 using a feature
not supported by the kernel in Debian 9 might have resulted in a lot of
problems when still running the old kernel during or after an upgrade
from Debian 9 to Debian 10.

If the feature stays enabled by default in bookworm:
Everyone using grub is an x86 thing, for embedded ARM it is more common
to use a once ported ancient u-boot on your hardware forever.[1]
A bug against the release-notes pseudo-package with text describing
the incompatibility and possible workarounds would be appreciated.

> Best regards,
>
> - Ted

cu
Adrian

[1] u-boot in bullseye does already "support" metadata_csum_seed
by refusing to write to filesystems that have it enabled:
https://salsa.debian.org/debian/u-boot/-/commit/2e7365518acdb8fb6e9be332c8a6c57b457188d9

[Theodore Ts'o]

On Wed, Feb 15, 2023 at 04:06:55PM -0700, Sam Hartman wrote:
>
> You argue about shared libraries for non-packaged binaries.
> I think we mostly don't care about that, and again, I think that's at
> least a generally recognized thing that came out of our focus on
> packages and package dependencies.

Note that package dependencies doesn't allow a binary created on
Debian N to work on Debian N-1. It just *prevents* the package from
being installed on Debian N-1. If you care about allowing the package
to be instaslled on Debian N-1, that's what build chroots are for.
That's how we create packages for backports, after all.

I'm making a similar argument for mkfs and bootable images for older
distributions. Just as you use a build chroot when you build a
package for Debian N-1, you should use a chroot when building a
bootable image for Debian N-1. And that means using the chroot for
*everything*; not just installing the grub bootloader, but also
running mkfs.

(Note that using the sid grub bootloader while using the sid's
mkfs.ext4 would work in this particular case, but if you want a pure
"Bullseye" image which is identical to what running the Bullseye d-i
would create, then you should run Bullseye's mkfs, not sid's mkfs.)

- Ted

[Russ Allbery]

"Theodore Ts'o" <ty...@mit.edu> writes:
> On Wed, Feb 15, 2023 at 04:06:55PM -0700, Sam Hartman wrote:

>> You argue about shared libraries for non-packaged binaries. I think we
>> mostly don't care about that, and again, I think that's at least a
>> generally recognized thing that came out of our focus on packages and
>> package dependencies.

> Note that package dependencies doesn't allow a binary created on Debian
> N to work on Debian N-1. It just *prevents* the package from being
> installed on Debian N-1. If you care about allowing the package to be
> instaslled on Debian N-1, that's what build chroots are for. That's how
> we create packages for backports, after all.

> I'm making a similar argument for mkfs and bootable images for older
> distributions. Just as you use a build chroot when you build a package
> for Debian N-1, you should use a chroot when building a bootable image
> for Debian N-1. And that means using the chroot for *everything*; not
> just installing the grub bootloader, but also running mkfs.

Regardless of which analogy feels more correct here, the reality on the
ground is that using a current mke2fs to create file systems for older
Debian versions, unlike running new binaries on old systems, is something
that used to work. People have been doing that and relied on it working.
We've made a change that broke it. That change has benefits and
justifications as all changes do, but from the perspective of the people
with that use case, it's a regression. So we should either document it or
revert it, and I think the debate is over which of those two to do.
Ideally we do this with an eye to what reduces this problem in the future.

It had never occurred to me before that the version of the system on which
I run mke2fs would matter as long as I didn't pick a newer file system
type (ext5 or something). Now I know! Until today, I had no idea ext4
even *had* "incompat" features or that mke2fs could make file systems that
grub couldn't understand. I'm sure this is well-documented in the ext4
world (although ext4(5) could both be advertised a bit more prominently
and be more explicit about this problem, IMO). I'd just never had any
reason to seek out that documentation, since creating file systems with
the defaults has always just worked for me.

In that sense, it's a testament to the maintenance quality of ext4, but
the result of mke2fs just working all these years is that most users do
not read the details of the ext4 man page since they've never needed to.
That would make me lean towards making documentation a large part of the
solution here. I'm sure I'm not alone in having no idea that the version
or configuration of mk2efs mattered at this level unless I'd changed the
defaults. It seems like we'd save people a lot of future trouble if we
could find ways to communicate to system constructors that the file system
needs to be built from a chroot.

In other words, if this is a compatibility rule that is important when
using these tools, it needs to be WAY more prominently documented than it
is right now. That that feels more important to me than adding a
one-release backward-compatibility guarantee. A lot of environments
regularly use oldstable or even oldoldstable for specific applications, so
we'd still break people's use cases if they don't know about the need to
build file systems from a chroot. Given that this will presumably happen
again in the future since it's apparently a normal part of ext4
development, it's not a one-time problem and it's something that we
somehow have to communicate to users to avoid ongoing fragility.

It feels to me (not a release team member!) like the strongest argument
for making a change other than documentation is the proximity to the
release at which the change was made (which is indeed quite late in the
release process for this critical of a component of the operating system).
But I'm also not sure that matters to most of our users? I think most
users are only going to care when stable is released; people doing
production work on unstable or testing already know that they're signing
up for occasional breakage. So the proximity-to-release argument to me
feels most relevant if this change is specifically a problem for the
release process and would hold up things Debian itself needs to do, due to
(for example) it being difficult to change tooling so that file systems
are generated from a chroot. I have no idea if that's the case.

--
Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>

[Michael Prokop]

* Sebastian Ramacher [Thu Feb 16, 2023 at 09:09:08AM +0100]:
> On 2023-02-15 13:17:38 -0700, Sam Hartman wrote:

> > But for example you're not leaving a lot of time for asking programs
> > like vmdb2 or fai-diskimage to adjust how they call fsck.
> > If you made this change a few months ago, it would be reasonable to file
> > bugs against those packages and ask them to adjust how they call
> > mkfs.ext4.

> To better understand the impact of this change, I was wondering which
> tools / image builders in the archive would be affected by this change.
> I've cloned the bug to vmdb2, but what about others?

I didn't verify it yet, but AFAICT grml-debootstrap is affected as
well, since it supports installing older Debian releases from within
more recent Debian/Grml environments and uses mkfs.ext4 as default.

BTW, we had a similar situation already in the past when mkfs.ext4
of e2fsprogs >=1.43~WIP.2015.05.18-1 enabled the "metadata_csum"
feature, while this was unsupported on jessie and older releases.
(grml-debootstrap provides a workaround for this.)

regards
-mika-

[Sam Hartman]

>>>>> "Sebastian" == Sebastian Ramacher <sram...@debian.org> writes:

Sebastian> To better understand the impact of this change, I was
Sebastian> wondering which tools / image builders in the archive
Sebastian> would be affected by this change. I've cloned the bug to
Sebastian> vmdb2, but what about others?

It will affect fai-diskimage in the fai package..
I believe that's used by the cloud team (or was) to create cloud images;
I don't know if their use case is affected, because I don't know what OS
they use to create what images.

[Adrian Bunk]

On Wed, Feb 15, 2023 at 12:05:41AM +0100, Daniel Leidert wrote:
>...
> Reasons:
>...
> - - the change makes it impossible to create filesystems with this version of
> e2fsprogs and then run a grub-install from a target system that does not cope
> with that feature; basically breaking the debootstrap method of installing
> Debian or Ubuntu onto a server (violating #4 of the Debian social contract)
>...
> Instead, turning on this feature should be postponed for the next release cycle
> where a proper transition can be done.
>...

Daniel, you are contradicting yourself when claiming that a change that
would allegedly violate the Debian social contract could be done in the
next release cycle.

> Daniel Leidert

cu
Adrian

[Adrian Bunk]

On Thu, Feb 16, 2023 at 05:48:22PM +0100, Daniel Leidert wrote:

> Actually, I'm not.
>...

If not being able to install bullseye from bookworm is a violation of
the Debian social contract, then the same rationale applies to not
being able to install bullseye from trixie being a violation of
the Debian social contract.

In [1] you are arguing with problems installing Ubuntu 20.04 this
way from bookworm with the e2fsprogs change, the same will apply
to installing Ubuntu 22.04 from trixie.

QED

> I have also written in [1] how I think the transition
> should be handled (IMO)
>...

I am currently spending time trying to summarize the situation and open
questions, and I am a bit underwhelmed by the inaccuracies and lack of
technical detail in your emails.

The instructions you cite in [1] are for installing bullseye from
non-Debian systems. What bookworm ships does not matter much there,
these instructions will be wrong as soon as some *other* distribution
like Fedora changes the default.

I am wondering how exactly your often repeated "there is no grub
upstream release with support for it" would be relevant in practice.
Whether it's 2.06-8 or 2.07-1 in bookworm shouldn't make a difference.

Sebastian has now created #1031364 for your original vmdb2 problem,
everyone discussing in #1030939 seems to have missed that tools in
bookworm creating images for < bookworm must handle such changes.
That's not different from debootstrap having code to handle
apt-transport-https being required in some older releases.

> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030939#108
>
> Regards, Daniel

cu
Adrian

[Sam Hartman]

>>>>> "Adrian" == Adrian Bunk <bu...@debian.org> writes:

Adrian> On Thu, Feb 16, 2023 at 05:48:22PM +0100, Daniel Leidert wrote:
>> Am Donnerstag, dem 16.02.2023 um 18:37 +0200 schrieb Adrian Bunk:
>> > On Wed, Feb 15, 2023 at 12:05:41AM +0100, Daniel Leidert wrote:
>> > > ... > > Reasons: > > ... > > - - the change makes it
>> impossible to create filesystems with this version of > >  
>> e2fsprogs and then run a grub-install from a target system that
>> does not cope > >   with that feature; basically breaking the
>> debootstrap method of installing > >   Debian or Ubuntu onto a
>> server (violating #4 of the Debian social contract) > > ... > >
>> Instead, turning on this feature should be postponed for the next
>> release cycle > > where a proper transition can be done. > > ...
>> >
>> > Daniel, you are contradicting yourself when claiming that a
>> change that > would allegedly violate the Debian social contract
>> could be done in the > next release cycle.
>>
>> Actually, I'm not. ...

Adrian> If not being able to install bullseye from bookworm is a
Adrian> violation of the Debian social contract, then the same
Adrian> rationale applies to not being able to install bullseye from
Adrian> trixie being a violation of the Debian social contract.

I don't think that arguing about whether this is a social contract
violation makes a lot of sense.
It seems fairly clear there is not a consensus about that.

But if we're going to do that, I suggest that Adrian is putting words
into Daniel's mouth a bit.
Daniel has said he believes this situation violates the Social Contract
#4, but has not explained why.
Adrian has taken one interpretation.
I'll propose another: "This violates the social contract because we are
not prioritizing our users. Prioritizing users requires that we give
them notice of incompatible changes."
I personally think that prioritizing users requires no such thing, and
that this change is not a violation of the SC.
But you don't need to take the straw man position Adrian is assuming
Daniel implies to believe this is a SC violation.

But seriously, let's give up the whole is this an SC violation part of
the discussion and move on with constructive aspects:

* The RT has asked to understand the impact of the change.

* Several people have proposed better documentation because it's clear
that user (and image builder) expectations are not aligned with
filesystem maintainer expectations.

* Anyone could prepare patches to image building software to use mkfs
options that will work with bullseye. You could also try to prepare
patches to run mkfs out of a chroot or container of the guest OS for
the image. I appreciate Russ strongly favors this solution, but as
someone who has dug into image building tools a fair bit over the
years, I think there are significant complexity and performance
downsides to that approach. I also think that changing the mkfs
options is more likely to get an unblock than changing the structure
of how the tool works.



* People could try to judge consensus on debian-devel or debian-policy
about whether we want a stability guarantee ?+/- 1 release on issues
like this. My suspicion is that you will not find a consensus, and
that if the RT decides they don't want this change in bookworm, Ted
will change the defaults, and if the RT is unwilling to block, we're
left with documentation.

I think all the above is more productive than arguing about whether this
is or is not an SC violation.

[Daniel Leidert]

Am Donnerstag, dem 16.02.2023 um 20:10 +0200 schrieb Adrian Bunk:

[..]

> I am currently spending time trying to summarize the situation and open
> questions, and I am a bit underwhelmed by the inaccuracies and lack of
> technical detail in your emails.

Well, I didn't have weeks to prepare. I had <24 hours and gave you
already enough information so you did not have to start from scratch.

I will summarize my points at the bottom.

> The instructions you cite in [1] are for installing bullseye from
> non-Debian systems.

That is simply not true. Those are general instructions, they are not
limited to non-Debian systems. Most server providers have exctly *one*
rescue system from where I can do a clean installation with deboostrap
(and that even usually is a Debian). I cannot choose to use one that
hasn't an e2fsprogs that has this breaking change enabled. Say for
example, grml, used by multiple providers I know as rescue system and
based on Debian, picks up Bookworm with e2fsprogs with that change. Now
users trying to install anything other than a Debian Bookworm using the
deboostrap method will run into the situation that "grub-install" will
fail, and it won't even indicate that they will have to tune the just
created ext4 filesystem or even change /etc/mke2fs.conf. I spent a few
hours until I tracked it down. And the situation right now is, that I
can simply install any system with the deboostrap method. I'm not aware
that there are any breakages or incompatibilities.

> What bookworm ships does not matter much there,
> these instructions will be wrong as soon as some *other* distribution
> like Fedora changes the default.

Fedora isn't used much as a rescue system, don't you think? Have you
ever encountered that? I do custom server setups with deboostrap for
almost two decades now. I haven't seen any distribution so far that
changed the created filesystem to be incomatible with grub-install from
the systems that might be installed. Most of the rescue systems were
Debian based, JFTR.

> I am wondering how exactly your often repeated "there is no grub
> upstream release with support for it" would be relevant in practice.
> Whether it's 2.06-8 or 2.07-1 in bookworm shouldn't make a difference.

You completely miss the point here. It would lead to exactly the same
situation if 2.07 would be the *first* to support it and could be
shipped with Bookworm as long as e2fsprogs makes this breaking change
now. But it makes a huge difference if 2.07 with a fix is released in
around the same time as Bookworm and can spread until Trixie is
prepared and the breaking change is postponed to Trixie. Ubuntu 24
would have picked up that fix by then. 22 and maybe even 20 would
probably have picked it up either. Even bullseye could get a patch to
deal with that. The breakage would have less impact than it has now,
while nothing is prepared.

And it is completely illusional to say that people should first create
a Bullseye chroot to then do a deboostrap setup of a target system from
that chroot, as Theodore suggested. Well, I'm more than underwhlemed by
suggestions like this.

> Sebastian has now created #1031364 for your original vmdb2 problem,
> everyone discussing in #1030939 seems to have missed that tools in
> bookworm creating images for < bookworm must handle such changes.
> That's not different from debootstrap having code to handle
> apt-transport-https being required in some older releases.

I agree. So don't you think introducing this now is a really bad
timing?

I checked a search engine to find out what this feature even does.
Turns out, there were less than 500 hits. It is a feature available
since kernel 4.4 and not widely used nor default. So what is the gain
here? I also tried to understand why our users would need to be able to
change the UUID of the filesystem. In 20 years with Debian, I haven't
encountered a situation where this has been necessary (I didn't even
know that one could). My gut feeling is, that this feature is only
useful to a handful of people. I haven't heard any explanation so far
why this needs to be turned on by default just now. The whole
discussion so far has been Theodore argueing why he doesn't care about
his actions and why he doesn't have to.

If this feature should be turned on, then I still think that doing this
for Trixie is the better choice. The tools affected can be fixed to
work around the issue. The other distributions can pick up the grub-
install fix.

And JFTR: The attitude I preceived since I got into the discussion with
the simple sentence that fixing grub in Bookworm might not be enough,
can be summarized as "I/we don't care". So, sorry, I care, even if my
less excellent mails might be underwhelming for you.

Daniel

[Sam Hartman]

>>>>> "Adrian" == Adrian Bunk <bu...@debian.org> writes:

Adrian> Below is my attempt to give an overview of the situation,
Adrian> feel free to amend/correct if anything is missing or wrong.


I believe your summary is correct and includes the issues I am aware of.
I believe I am following things enough that I have confidence in that
conclusion.

Thanks much.

[Sebastian Ramacher]

Also cloned/reassigned the bug to fai, thanks.

Cheers
--
Sebastian Ramacher

[Russ Allbery]

Adrian Bunk <bu...@debian.org> writes:

> The image creators could just set the features they enable to what they
> copied from /etc/mke2fs.conf from the target distribution, a label with
> a timestamp wouldn'tbring much benefit here.

That's a very good point and I'm embarrassed it wasn't immediately obvious
to me. Sorry about the noise.

[Theodore Ts'o]

On Fri, Feb 17, 2023 at 08:51:33AM -0800, Russ Allbery wrote:
> Adrian Bunk <bu...@debian.org> writes:
>
> > The image creators could just set the features they enable to what they
> > copied from /etc/mke2fs.conf from the target distribution, a label with
> > a timestamp wouldn'tbring much benefit here.
>
> That's a very good point and I'm embarrassed it wasn't immediately obvious
> to me. Sorry about the noise.

One other thing I woud add here is (a) this whole discussion of
mke2fs.conf only helps for ext4, and the general problem is something
that extends to all file system. The immediate question may be ext4
specific, but as I mentioned earlier, XFS is enabling the "bigtime"
feature for the first time in Bookworm. So enabling what may be
convenient, but ultimately an anti-pattern is something that hopefully
in the long-term Debian should be striving towards. Yes, it's
annoying and and extra work. So is using build chroots if we are
building packages for a older Distro versions. But it's the right
thing to do.

Secondly, (b) there may be a misapprehension that it is possible to
get an identical file system just by controlling the contents of
mke2fs and/or specifying the file system features on the command line.
While this is mostly true, it is not the whole story. For example,
the size and location of the journal is determined hueristically, and
in the past, this has changed as we have discovered that (for example)
making the default journal size larger would result in better
performance. The location of the journal has also changed from the
beginning of storage device (low-numbered LBA's) to the middle of the
device.

So just as a binary compiled using the default gcc on Buster might be
different from a binary build using Sid, even if you you force the use
of older glibc shared libraries at link time or use -static to
statically link with the glibc installed in your random desktop
environment, the results from using mke2fs on Debian N may be
different from using mke2fs on Debian M, even if you control for thea
file system features. (And other things which _are_ controllable on
mke2fs.conf, but which go beyond mke file system features. For
example, in Bookworm starting with e2fsprogs 1.46.4, the default inode
size is forced to always be 256 bytes, even for small file systems.
This was not true in Buster and older Debian distributions.)

So if you want a bootable image that is identical to what would be
created by using the Buster or Bullseye debian-interaller, you
*really* want to use the mkfs that was supplied by Buster or Bullseye,
and that means running the mkfs from a chroot.

Best regards,

- Ted

[Adrian Bunk]

On Fri, Feb 17, 2023 at 02:08:28PM -0500, Theodore Ts'o wrote:
>...

> So enabling what may be
> convenient, but ultimately an anti-pattern is something that hopefully
> in the long-term Debian should be striving towards. Yes, it's
> annoying and and extra work. So is using build chroots if we are
> building packages for a older Distro versions. But it's the right
> thing to do.

Don't assume "host = target" would always be possible.

Your proposed solution only works for the trivial
"building Debian on Debian" case.

The same general problem applies in various "building non-Debian
embedded Linux filesystem on Debian" situations where the target
chroot does not contain mkfs.ext4.

Or if it is present, it would be a chroot for the target architecture
where you might have to run mkfs.ext4 under qemu.

All image building tools that support bookworm (including all image
building tools we ship in bookworm) have to ensure that what they
produce works on the intended target. There is no general solution
*how* to do that that could be applied in all cases.

> Secondly, (b) there may be a misapprehension that it is possible to
> get an identical file system just by controlling the contents of
> mke2fs and/or specifying the file system features on the command line.

>...

It is clear that different versions of tools like mkfs.ext4 or gzip
might produce different output.

If identical results matter, you can just define $distribution as
build environment and then expect that for example the host tools
from Debian bookworm will always create the same output when building
a Yocto distribution.

[Theodore Ts'o]

On Fri, Feb 17, 2023 at 12:43:01PM -0700, Sam Hartman wrote:
>
> I am not entirely convinced that using current rather than guest
> tools for image building is an anti-pattern. You've been working on
> filesystems for a long time; I've been working on various image
> building projects since my first watchmaker project at MIT.

For what it's worth, I've been building test appliances[1] for the
purposes of file system testing since 2013 (initially just for Qemu,
but now for Cloud[2] environments as well as Android[3]). Admittedly,
I haven't been doing this as long as you, but I'm not unfamiliar with
building appliance images using debootstrap, and I've *always* built
my KVM/Qemu images using a build chroot[4], including the mkfs and
debootstrap steps.

[1] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/gen-image
[2] https://thunk.org/gce-xfstests
[3] https://thunk.org/android-xfstests
[4] https://github.com/tytso/xfstests-bld/blob/master/setup-buildchroot

So I'm happy to talk to you off-line about best practices for building
images, but this is something that I do *regularly*, since there are
weekly updates from upstream xfstests. Thus, I have personal
experience that using a build chroot for image creation really is
*not* *that* *hard*. For that matter, when I'm doing test appliance
development, I'm running regression tests[5] several times a day which
build images in a chroot.

[5] https://github.com/tytso/xfstests-bld/blob/master/selftests/appliance

Everything is automated. No fuss, no muss, no dirty dishes. :-)
Futher, it provides better build reproducibility, no matter who is
building the image, and it also makes full GPL compliance (if you are
publishing the test appliances) much, *much* easier --- I can provide
an exhaustive list of all components (including mkfs.ext4!) and control
scripts involved with the creation of the image.

Cheers,

- Ted

[Theodore Ts'o]

On Fri, Feb 17, 2023 at 10:34:29PM +0200, Adrian Bunk wrote:
>
> The same general problem applies in various "building non-Debian
> embedded Linux filesystem on Debian" situations where the target
> chroot does not contain mkfs.ext4.

In practice, if the root file system is using ext4, the target chroot
has to have e2fsprogs installed so that fsck.ext4 exists. So I'm not
sure it's all that unreasonable?

> Or if it is present, it would be a chroot for the target architecture
> where you might have to run mkfs.ext4 under qemu.

Sure, but that's not that difficult; I have a script[1] that will
setup chroots for foreign architectures which use binfmt_misc to run
(for example) arm32 or arm64 binaries using qemu on an x86 machine,
without needing to boot an arm32/arm64 kernel. For a more detailed
explanation see slide #8 of this slide deck[2]. The setup-buildchroot
script is turn-key; my experience is that several new college grads
and interns hired at $WORK have had no problems setting it up.

[1] https://github.com/tytso/xfstests-bld/blob/master/setup-buildchroot#L364
[2] https://thunk.org/android-xfstests

Also, in practice, if you are building an image for a foreign system,
you'll need to have a qemu setup to run the second stage of the
debootstrap in any case. I've just found it simpler to run the mkfs
and debootstrap in a chroot using qemu-user-static compared to messing
around with debootstrap --second-stage, which requires running a
chroot and qemu in any case.

> All image building tools that support bookworm (including all image
> building tools we ship in bookworm) have to ensure that what they
> produce works on the intended target. There is no general solution
> *how* to do that that could be applied in all cases.

Well, the general solution we can give them is instructions to adjust
/etc/mke2fs.conf on those systems needing to run those image building
systems. It's a one-line change. This can be documented in the
release notes, or in an e2fsprogs NEWS entry.

If the RT really insists, we can edit /etc/mkefs.conf for Bookworm to
not enable metadata_csum_seed by default. This will make it more
difficult for root file systems cloud VM's to be able adjust the file
system UUID on the fly, while the file system is mounted, so that's
the tradeoff. Quite frankly, the distro that I really care about for
$WORK is Arch, since Google's Cloud Optimized OS is based off of Arch
userspace packages. So if the Debian Release Team would like to
disable metadata_csum_seed by default in e2fsprogs, I will make that
change and upload a new version of e2fsprogs 1.47.0.

I don't think that's the right way to go, since I don't consider image
building to be a super-common use case, and those who do that can edit
/etc/mke2fs.conf on their own. However, I accept that this is the
Release Team's call.

If we do make this change to mke2fs.conf for Bookworm, my intention is
to upload a version of e2fsprogs which reverts this change as soon as
Bookworm ships as stable, so that Debian 13 will enable
metadata-csum-seed by default. At that point, people using Sid or
Debian Testing will have problems if they want to build images for
Bullseye, and I'll note that Daniel, who originally registered this
objection, was building images using Debian Unstable. So this will
will only give him a reprieve for a few months before he will need to
push changes to vmdb2 or make that one-line change to
/etc/mke2fs.conf.

If the Debian release team could let me know which path they would
prefer, I would appreciate it. At this point, the grub2 package
version (2.06-8) which supports metadata-csum-seed has migrated to
testing. So the only thing the e2fsprogs migration is now blocked on
is the RT's decision on this bug.

Best regards,

- Ted

[Adrian Bunk]

On Fri, Feb 17, 2023 at 09:28:59PM -0500, Theodore Ts'o wrote:
> On Fri, Feb 17, 2023 at 10:34:29PM +0200, Adrian Bunk wrote:
> >
> > The same general problem applies in various "building non-Debian
> > embedded Linux filesystem on Debian" situations where the target
> > chroot does not contain mkfs.ext4.
>
> In practice, if the root file system is using ext4, the target chroot
> has to have e2fsprogs installed so that fsck.ext4 exists. So I'm not
> sure it's all that unreasonable?

>...

In Yocto e2fsprogs-e2fsck and e2fsprogs-mke2fs are separate packages.


The general problem is also not limited to mkfs.

The target might have busybox tar, but you might want to use GNU tar for
generating tarballs that will be unpacked on the target.

I have once seen some old version of busybox tar choke on some specific
tarball generated by GNU tar, but that's not a reason for a general rule
that all tarballs that might be unpackaged on the target must be created
by busybox tar.


I do see your point that reducing differences between host tools and
target can be desirable, but there are limits how far this is feasible
or whether it is the best solution in a given situation.

When differences exist, it is the responsibility of the image creation
tools to deal with the differences.

[Paul Gevers]

Dear Ted,

On 16-02-2023 23:24, Theodore Ts'o wrote:
> But, if the Debian Release team would like to override my position, my
> suggestion would be to just change the default for /etc/mke2fs.conf
> for *everyone* running Debian bookworm, and with the understanding
> that this will be reverted in Debian testing after the next stable
> release, and that moving forward, we make it the image building tools
> problem if they want to support this highly dubious practice of using
> Debian N+X's mkfs to build images for Debian N.

The Release Team discussed this topic in their IRC meeting of this
evening. I would like to affirm your position as a maintainer of
e2fsprogs to make this kind of changes in your package and that's up to
image builders to cope with that. However, because of the *timing* of
the change we ask you to revert the change of the default settings for
bookworm and please enable them once trixie opens.

For future default changes, consider planning them before the transition
freeze.

Thanks

Paul