Bug#1030846: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Cyril Brulebois]

Package: e2fsprogs
Version: 1.47.0-1
Severity: grave
Justification: renders package unusable
X-Debbugs-Cc: debia...@lists.debian.org, gr...@packages.debian.org

(Please keep debia...@lists.debian.org and gr...@packages.debian.org
in the loop.)

Hi,

Spotted via debian-installer tests: grub-install fails with “unknown
filesystem” when trying to run a simple `grub-install /dev/sda` with
an all-default installation.

While there might be something wrong about e2fsprogs-udeb specifically,
possibly only affecting the installer context, I'm not sure what that
means for existing systems, hence the severity.

The “e2fsprogs gets a new upstream release and new flags” hypothesis was
confirmed by building d-i with e2fsprogs-udeb_1.46.6-1_amd64.udeb
rebranded as 1.48, which made the problem disappear.

Cheers,
--
Cyril Brulebois (ki...@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[Sven Joachim]

On 2023-02-08 11:39 +0100, Cyril Brulebois wrote:

> Package: e2fsprogs
> Version: 1.47.0-1
> Severity: grave
> Justification: renders package unusable
> X-Debbugs-Cc: debia...@lists.debian.org, gr...@packages.debian.org
>
> (Please keep debia...@lists.debian.org and gr...@packages.debian.org
> in the loop.)
>
> Hi,
>
> Spotted via debian-installer tests: grub-install fails with “unknown
> filesystem” when trying to run a simple `grub-install /dev/sda` with
> an all-default installation.
>
> While there might be something wrong about e2fsprogs-udeb specifically,
> possibly only affecting the installer context, I'm not sure what that
> means for existing systems, hence the severity.
>
> The “e2fsprogs gets a new upstream release and new flags” hypothesis was
> confirmed by building d-i with e2fsprogs-udeb_1.46.6-1_amd64.udeb
> rebranded as 1.48, which made the problem disappear.

It is the metadata_csum_seed feature that is now enabled by default
which grub-install does not grok. This has been reported "recently"[1].

Cheers,
Sven


1. https://bugs.debian.org/866603

[Theodore Ts'o]

On Wed, Feb 08, 2023 at 11:39:48AM +0100, Cyril Brulebois wrote:
>
> Spotted via debian-installer tests: grub-install fails with “unknown
> filesystem” when trying to run a simple `grub-install /dev/sda` with
> an all-default installation.

The fix for this issue landed in the grub2 repository on June 11,
2021:

commit 7fd5feff97c4b1f446f8fcf6d37aca0c64e7c763
Author: Javier Martinez Canillas <jav...@redhat.com>
Date: Fri Jun 11 21:36:16 2021 +0200

fs/ext2: Ignore checksum seed incompat feature

This incompat feature is used to denote that the filesystem stored its
metadata checksum seed in the superblock. This is used to allow tune2fs
changing the UUID on a mounted metdata_csum filesystem without having
to rewrite all the disk metadata. However, the GRUB doesn't use the
metadata checksum at all. So, it can just ignore this feature if it
is enabled. This is consistent with the GRUB filesystem code in general
which just does a best effort to access the filesystem's data.

The checksum seed incompat feature has to be removed from the ignore
list if the support for metadata checksum verification is added to the
GRUB ext2 driver later.

Suggested-by: Eric Sandeen <esan...@redhat.com>
Suggested-by: Lukas Czerner <lcze...@redhat.com>
Signed-off-by: Javier Martinez Canillas <jav...@redhat.com>
Reviewed-by: Lukas Czerner <lcze...@redhat.com>
Reviewed-by: Daniel Kiper <daniel...@oracle.com>

Unfortunately, this just missed the last grub release, since grub 2.06
was tagged on June 8, 2021. <Sigh>

There are two ways we can fix this. One is we can disable the
checksum seed feature in the Debian release of mke2fs.conf. Or we can
land this above-mentioned commit into the grub2 package. Since the
hard freeze is almost upon us, I'm happy to handle this either way.

If we *are* going to backport some grub2 patches, may also make a plug
for this one, which is also in the upstream grub2 git repo:

commit 2e9fa73a040462b81bfbfe56c0bc7ad2d30b446b
Author: Theodore Ts'o <ty...@mit.edu>
Date: Tue Aug 30 22:41:59 2022 -0400

fs/ext2: Ignore the large_dir incompat feature

Recently, ext4 added the large_dir feature, which adds support for
a 3 level htree directory support.

The GRUB supports existing file systems with htree directories by
ignoring their existence, and since the index nodes for the hash tree
look like deleted directory entries (by design), the GRUB can simply do
a brute force O(n) linear search of directories. The same is true for
3 level deep htrees indicated by large_dir feature flag.

Hence, it is safe for the GRUB to ignore the large_dir incompat feature.

Fixes: https://savannah.gnu.org/bugs/?61606

Signed-off-by: Theodore Ts'o <ty...@mit.edu>
Reviewed-by: Daniel Kiper <daniel...@oracle.com>

Otherwise, what can happen is that users may choose to enable the
large_dir feature, and if they do it on the root file system, they can
end up making their system unbootable. I've had some Arch and Gentoo
users get bitten by this....

> The “e2fsprogs gets a new upstream release and new flags” hypothesis was
> confirmed by building d-i with e2fsprogs-udeb_1.46.6-1_amd64.udeb
> rebranded as 1.48, which made the problem disappear.

Alternatively, I can apply this patch to e2fsprogs:

diff --git a/misc/mke2fs.conf.in b/misc/mke2fs.conf.in
index b7fc95df7..ff47bdb39 100644
--- a/misc/mke2fs.conf.in
+++ b/misc/mke2fs.conf.in
@@ -11,7 +11,7 @@
features = has_journal
}
ext4 = {
- features = has_journal,extent,huge_file,flex_bg,metadata_csum,metadata_csum_seed,64bit,dir_nlink,extra_isize,orphan_file
+ features = has_journal,extent,huge_file,flex_bg,metadata_csum,64bit,dir_nlink,extra_isize,orphan_file
}
small = {
blocksize = 1024

Which will kind of suck, since the reason for enabling
metadata_csum_seed is to be able to reliably change the label and file
system uuid on a mounted file system, which matters for certain cloud
workloads. Specifically, for Google's Cloud Optimized OS, which
attempts to update the file system UUID and resize the root file
system to fit the size of the cloud-emulated block device on two
separate, racing systemd unit files. This which works 99.9% of the
time, but when you launch a huge number of cloud images, that last
0.1% of the time is really noticeable.

But that's for COS; if we have to disable the metadata_csum_seed
feature on Debian file systems, I can live with that.

Cheers,

- Ted

[Steve McIntyre]

Control: reassign -1 grub2

On Wed, Feb 08, 2023 at 02:02:44PM -0500, Theodore Ts'o wrote:
>
>The fix for this issue landed in the grub2 repository on June 11,
>2021:
>
>commit 7fd5feff97c4b1f446f8fcf6d37aca0c64e7c763
>Author: Javier Martinez Canillas <jav...@redhat.com>
>Date: Fri Jun 11 21:36:16 2021 +0200

...

>If we *are* going to backport some grub2 patches, may also make a plug
>for this one, which is also in the upstream grub2 git repo:
>
>commit 2e9fa73a040462b81bfbfe56c0bc7ad2d30b446b
>Author: Theodore Ts'o <ty...@mit.edu>
>Date: Tue Aug 30 22:41:59 2022 -0400

I've just queued these up in our repo for the next grub upload, due in
a few days.

--
Steve McIntyre, Cambridge, UK. st...@einval.com
"... the premise [is] that privacy is about hiding a wrong. It's not.
Privacy is an inherent human right, and a requirement for maintaining
the human condition with dignity and respect."
-- Bruce Schneier

[Theodore Ts'o]

On Wed, Feb 08, 2023 at 09:12:05PM +0000, Steve McIntyre wrote:
>
> I've just queued these up in our repo for the next grub upload, due in
> a few days.

Many thanks, Steve!

- Ted

[Sven Joachim]

Control: clone -1 -2
Control: reopen -2
Control: reassign -2 e2fsprogs 1.47.0-1
Control: severity -2 serious

Thanks from me as well :-). To prevent e2fsprogs from migrating to
testing before grub2 and breaking d-i, I am reassigning a copy of this
bug back to e2fsprogs. It may be closed once grub2 2.06-8 enters
Bookworm. Perhaps it would also be a good idea to add a versioned
"Breaks: grub2-common (<< 2.06-8~)" to e2fsprogs.

Cheers,
Sven

[Cyril Brulebois]

Theodore Ts'o <ty...@mit.edu> (2023-02-09):
> On Thu, Feb 09, 2023 at 06:55:08PM +0100, Sven Joachim wrote:
> > That is not going to help, because IIUC grub-install is run from the
> > target system that you are installing, and there is no
> > grub2-common-udeb.
>
> Right, but if the conflict in e2fsprogs-udeb prevents the installer
> from pulling in an overly new version of e2fsprogs-udeb, that woul be
> sufficient, no?

As Sven rightfully pointed out, there are two different environments:
- installer, with anna and udebs (but that would be the same with apt
and debs);
- target.

There are no connections between both environments, so you can't have
package relationships in a cross-environment manner.

The installer uses whatever was available at build-time, or for
components that aren't built-in, whatever is available on the
installation image, or on the mirror. There's no “pulling in an overly
new version” that can be avoided. There's *one* version in the suite,
that's the one that's getting used, there's no alternative.

TL;DR: Some Conflicts, even if that were possible (which it is not)
wouldn't achieve anything (except probably not offering any ext*
option at the partioning step — not a huge win).

> The reason why I really don't want to add the conflicts with e2fsprogs
> 1.47 is because for people who are using sid, there is aboslutely
> nothing wrong with installing e2fsprogs 1.47. It's only if they use
> the installer that sucks in e2fsprogs that there's a problem --- it's
> not an inherent conflict with grub per se. If it were, then we
> shouldn't allow installation of e2fsprogs 1.46 because grub2 upstream
> is painfully slow in putting out releases --- and that's not
> reasonable.

*sigh* @ the heavy finger pointing in various parts of your mail.

> Now, what I *could* do is change the mke2fs.conf in e2fsprogs-udeb to
> remove the default use of the csum-seed feature ---- but is it worth
> it? Hopefully the new version of grub2 will come out soon, at which
> point I would then need to revert the hacked mke2fs.conf --- and your
> dup'ing this bug should prevent the installer from picking up
> e2fsprogs 1.47 until the new version of grub gets released.

Right now what I'm most concerned with is getting grub2 fixed in
unstable, candidate for migration into testing. Until that happens,
e2fsprogs must be kept out of testing, so that the installer cannot get
a too-new e2fsprogs with a too-old grub2.

Whether we introduce a relationship between both packages making britney
wait on grub2's being ready to allow e2fsprogs to migrate alongside it,
or we keep an RC bug on e2fsprogs to keep it out of testing until grub2
gets fixed and migrated into testing… doesn't matter much to me.


I'll word it differently because I'd like to avoid more mails on this
thread: The installer pulls components for the target system from a
single distribution, there's no set of existing packages that can be
kept around (as that would be the case for existing systems using
testing or unstable), so we need packages in the distribution being
installed to be consistent.

Right now: unstable is broken; testing isn't.


[ snip stuff about grub design ]

> Holding back file system development because grub2 uptsream is super
> slow doesn't seem like a reasonable way forward, so I really don't
> want to set this precedent.

The Bookworm freeze has started, we need to be able to install stuff, so
we need a solution for the grub2/e2fsprogs situation *now*.

I really don't care about “setting a precedent”.

[ snip brainstorming ]

[Theodore Ts'o]

On Fri, Feb 10, 2023 at 08:31:04AM +0100, Cyril Brulebois wrote:
> > Holding back file system development because grub2 uptsream is super
> > slow doesn't seem like a reasonable way forward, so I really don't
> > want to set this precedent.
>
> The Bookworm freeze has started, we need to be able to install stuff, so
> we need a solution for the grub2/e2fsprogs situation *now*.
>
> I really don't care about “setting a precedent”.

But that problem has already been solved by cloning the bug back to
e2fpsrogs (#1030939) which will prevent e2fsprogs from transitioning
to testing, no? So what's the problem.

The fact that grub2 upstream is super-slow in doing releases is
already a pain in *ass that has caused much pain over the years.
Adding a conflict just adds more pain, without adding any additional
benefit. So what's the point?

- Ted

[Steve McIntyre]

On Tue, Feb 14, 2023 at 12:50:18PM +0100, Daniel Leidert wrote:
>Am Dienstag, dem 14.02.2023 um 10:45 +0000 schrieb Steve McIntyre:
>> On Tue, Feb 14, 2023 at 11:34:13AM +0100, Daniel Leidert wrote:
>> > Am Montag, dem 13.02.2023 um 21:35 -0500 schrieb Theodore Ts'o:
>>
>> ...
>>
>> > > But a generalized requirement that we be able to use debootstrap and
>> > > vmdb2 to be able to bootstrap an arbitrarily old distribution doesn't
>> > > seem to be reasonable.
>> >
>> > You are completyl wrong. This breaks a standard way of installing any
>> > system supported by deboostrap with a grub without a fix to deal with
>> > this version of e2fsprogs. This isn't just about vmdb2.
>> >
>> > What you are saying is ignorant.
>> >
>> > If this isn't cared about, then this version of e2fsprogs shouldn't
>> > make it into Bookworm. We are in the middle of a freeze and this breaks
>> > toolchains and a standard way (see [1]) of installing Debian.
>>
>> Sorry Daniel, but I have to (mostly!) agree with Ted here. If you're
>> creating an image of an older release using newer tools then you'll
>> need to be aware that sometimes the newer tools will create things
>> that don't work there. If there's a bug here, I would strongly argue
>> that it's in vmdb2. deboostrap (for example) includes some
>> release-specific knowledge to cope with issues like this.
>
>debootstrap does nothing related to grub. So it is a bad example.

That's why I said *like* this, not *exactly* like this. debootstrap
has had knowledge of things like fs layouts etc. that older releases
need (e.g. merged-/usr).

>Again I refer to [1]: If the host system contains the problematic
>e2fsprogs and the target system doesn't contain a grub with the fix
>[2], then this breaks installations. This breaks older systems *and*
>current systems. For example, I neither see the necessary grub patch
>in both Ubuntu 20.04 and 22.04 either. So they also cannot be
>installed using the deboostrap method and the toolchain in Sid (and
>Bookworm if e2fsprogs makes it there).

Breakages happen like this, and this has happened before in similar
circumstances. If you're installing an older system using brand-new
tools, you need to be aware of the potential for things to not
work. In this particular case, all you need to do is tweak the flags
on the ext4 filesystem when you create it. This isn't that hard...

>[1] https://www.debian.org/releases/stable/amd64/apds03
>[2] Even "our" grub only contains a patch and not an upstream version
>with support. So how can you even expect the target system to contain a
>fix and be able to handle the created filesystem?!
>
>Whe whole handling is completely wrong here. First, grub should have
>been fixed upstream. And the change in e2fsprogs should have been done
>only after "fixed" grub versions had settled. If we do it the other way
>around, we have to patch grub in affected distributions as well. And
>for Debian that means at least to patch Bullseye and any other release
>we want to be able to install from Bookworm. I even a lot of companies
>using Buster still.

And I know of folks still working on Stretch and Jessie. How far back
do you want to tie things??

>> If we don't allow for this kind of change, that wouldn't allow us to
>> *ever* make breaking changes in some packages, and that's just not
>> sustainable.
>
>I'm critizicing the way of handling that breaking change and the
>ignorance shown reagarding the impact, not that fact that there is a
>breaking change. And it breaks a lot! This doesn't affect just a few
>minor use cases. It affects the basic way of installing a clean Ubuntu
>or Debian (or derivative) on a remote server using the debootstrap
>method.

People using these tools need to be aware of the potential issue. What
would happen if you ran debootstrap with a filesystem that the target
distro doesn't know how to mount at all, for example?

>And again: We are in the middle of a freeze here. And e2fsprogs pushes
>a breaking change that is not even handled by any existing grub
>upstream release, and is also not properly handled within Debian?!

Grub upstream is already known to be problematic in terms of release
cycles. We now know about this particular issue (thanks Ted!) and
we've fixed it in unstable (and soon testing).

--
Steve McIntyre, Cambridge, UK. st...@einval.com

Who needs computer imagery when you've got Brian Blessed?

[Daniel Leidert]

Am Dienstag, dem 14.02.2023 um 14:58 +0000 schrieb Steve McIntyre:
> On Tue, Feb 14, 2023 at 12:50:18PM +0100, Daniel Leidert wrote:
>
>

[..]
> Breakages happen like this,

This breakage is *unnecessary* and it breaks at the momnent *all*
debootstrap installations except for doing a bookworm/sid installation
themselves. That is just stupid.

Get down from your high horse and ackknowledge the problems your
behavior causes.

> and this has happened before in similar
> circumstances.

No it has not. You are doing a breakage on purpose during a freeze
period while the transition period is over. Do it with a proper
transition during the next release cycle.


> >
[..]

> > Whe whole handling is completely wrong here. First, grub should have
> > been fixed upstream. And the change in e2fsprogs should have been done
> > only after "fixed" grub versions had settled. If we do it the other way
> > around, we have to patch grub in affected distributions as well. And
> > for Debian that means at least to patch Bullseye and any other release
> > we want to be able to install from Bookworm. I even a lot of companies
> > using Buster still.
>
> And I know of folks still working on Stretch and Jessie. How far back
> do you want to tie things??

How about staying on topic and explaining, why this transition is
necessary and has to be done the wrong way around, instead of picking
the fact that older systems still exist? You break almost *all*
installation right now. You also broke an Ubuntu 20.04 or 22.04 LTS
installation. Are they new enough?

[..]

> >
> > I'm critizicing the way of handling that breaking change and the
> > ignorance shown reagarding the impact, not that fact that there is a
> > breaking change. And it breaks a lot! This doesn't affect just a few
> > minor use cases. It affects the basic way of installing a clean Ubuntu
> > or Debian (or derivative) on a remote server using the debootstrap
> > method.
>
> People using these tools need to be aware of the potential issue. What
> would happen if you ran debootstrap with a filesystem that the target
> distro doesn't know how to mount at all, for example?

That is different from introducing a breaking change for which no grub
upstream release has a fix yet. So basically the only system able to
handle a filesystem created with e2fsprogs 1.47.0 is Sid right now. Can
you please ignore your ego and see the problems you are causing?

You push a breaking change for no reason at all. What is the gain here
compared to the issues you are causing?

> > And again: We are in the middle of a freeze here. And e2fsprogs
> > pushes
> > a breaking change that is not even handled by any existing grub
> > upstream release, and is also not properly handled within Debian?!
>
> Grub upstream is already known to be problematic in terms of release
> cycles.

That is not enough and it is not a solution for the problem. There is
*no* grub version out there supporting this, except for the patched
version is Sid. Is this the sign for a working transition?

> We now know about this particular issue (thanks Ted!) and
> we've fixed it in unstable (and soon testing).

Which helps exactly nobody, as it still breaks all other Debian or
Ubuntu installation.

I cannot belive that you intentionally break one of the standard
methods to install Debian or Ubuntu for no reason at all and without a
proper transition. Version 1.47.0 of e2fsprogs contains nothing
necessary for Bookworm. I'll bring this to the attention of the release
team. I'm sick of your ignorance. Do a proper transition and don't
start it during a freeze.


Daniel

[Theodore Ts'o]

There is another issue with vmdb2 if you are using XFS. Starting with
xfsprogs 5.15 (which is already in testing), bigtime is enabled by
default, so that newly created XFS file systems won't be subject to
timestamp overflow in 2038. Grub didn't land support for this feature
until 8b1e5d1936ff ("fs/xfs: Add bigtime incompat feature support") in
May 2021, despite the fact that XFS has had this feature for years and
years and years.

So if you aren't using the latest security fixes, and you are using
XFS as the boot partition --- it won't work on buster and bullseye.
"Fortunately", there were were massive number security vulnerabilities
in grub2 which forced a backport of grub2 2.06 to bullseye and buster,
so if you have the security updates enabled, you'll probably be OK ---
but it was only because of massive number of security problems forced
that backport.


In any case, a version of grub that will support the csum_seed feature
will be landing in Bookworm in just a few days. So at that point,
you'll be able to create VM images for Bookworm and Sid that will work
with the e2fsprogs in sid. The current plan of record is that it will
only be at that point that e2fsprogs will be allowed to migrate into
Bookworm.

For slowly moving upstreams like grub2, distributions *have* to take
updates before grub2 finally gets around to doing a release --- to get
security fixes if nothing else! The support for csum_seed has been in
Fedora and other distributions for a while, since the patches had
landed in grub2 in June 2021. I probably should have made sure the
feature had landed in Debian's grub2 packaging earlier; that's my bad,
and my apologies for that.

Note that Debian's grub2 has well over 100 patches, nearly all of
which are backports from grub2's git repo. So the argument that
"there doesn't exist a formally released grub2 release" isn't
particularly compelling, since all the distros are backporting
patches. The only question is how *many* commits release has an
individual distribution taken.


By the way, in the case of the csum_seed feature, it's pretty
straightforward to just run "tune2fs -O ^metadata_csum_seed
/tmp/boot.img". If the UUID has been changed since the file system
was created, you'll have to do this while the file system is unmounted
and it will take a few seconds, but that's almost certainly not the
case with vmdb2.

- Ted

[Daniel Leidert]

Am Dienstag, dem 14.02.2023 um 12:34 -0500 schrieb Theodore Ts'o:
>
[..]

> In any case, a version of grub that will support the csum_seed feature
> will be landing in Bookworm in just a few days.  So at that point,
> you'll be able to create VM images for Bookworm and Sid that will work
> with the e2fsprogs in sid.  The current plan of record is that it will
> only be at that point that e2fsprogs will be allowed to migrate into
> Bookworm.

As soon as this version hits testing, you have successfully disabled
the last working environment to use vmdb2 to create images of Ubuntu
and Debian. As soon as this version hits Testing, one then can no
longer build images for either any Ubuntu release nor Debian Bullseye
or older. I can then only build images for Bookworm and Sid. Please
think about that.

You *seriously* break the debootstrap method of installing Debian or
Ubuntu. vmdb2 ist just another tool that is broken by now, a tool that
I use very often. I explained the impacts of what you are doing often
enough now. By now the impact should be clear. And you are still not
dealing with the aftermath of your intended action?! You haven't done a
proper transition. You haven't given any affected toolchain the
necessary time to adopt. You haven't documented how to disable that
breaking change when creating filesystems for distributions where grub
does not support this (there is not even a NEWS entry). You haven't
even announced that breaking change. And yet you are going to break one
of our two standard methods of installing Debian or Ubuntu. How is any
of what you have been saying so far addressing any of these issues?

> [..]

> By the way, in the case of the csum_seed feature, it's pretty
> straightforward to just run "tune2fs -O ^metadata_csum_seed
> /tmp/boot.img".  If the UUID has been changed since the file system
> was created, you'll have to do this while the file system is
> unmounted
> and it will take a few seconds, but that's almost certainly not the
> case with vmdb2.

Well, how do you intend to distribute that information, so people who
have to use the deboostrap method to install a Debian or Ubuntu release
with a grub not supporting csum_seed (basically all existing releases,
except for the upcoming Bookworm) know that?

I do not understand why you are pushing 1.47.0 over 1.46.6, which you
had uploaded just five days before the former. You haven't even
presented a reason yet.

Regards, Daniel

[Sebastian Ramacher]

Control: clone -1 -2
Control: reassign -2 fai 6.0
Control: clone -1 -3
Control: reassign -3 grml-debootstrap 0.102

On 2023-02-15 21:04:36 +0100, Sebastian Ramacher wrote:
> Control: clone -1 -2

> Control: reassign -2 vmdb2 0.26-2
>
> On 2023-02-14 01:01:38 +0100, Daniel Leidert wrote:
> > Hi Steve,
> >
> > I believe that your fix to grub2 in Sid is not enough to handle
> > #1030939/#1030846.
> >
> > This problem breaks e.g. vmdb2. I can no longer create a Bullseye
> > system image with vmdb2 on Sid, because the grub-install step in the
> > Bullseye chroot now fails, because the created filesystem (created with
> > e2fsprogs on Sid) cannot be recognized by grub. I have to downgrade
> > e2fsprogs to the version in Testing to get the build going again. It
> > also means that a Bookworm system can never be used to format and
> > debootstrap a Bullseye or Buster system that requires a grub
> > installation.
> >
> > I guess that the fix applied to grub2 in Sid would have to be applied
> > to grub2 in Bullseye as well (and basically to any grub2 package in any
> > Debian or Ubuntu or Raspbian release supported by debootstrap).
> >
> > This situation is really messy. It breaks basically all my image builds
> > with vmdb2.
>
> Regardless of the outcome of #1031325, this issue will need to be fixed
> in vmdb2 eventually. vmdb2, similar to other bootstraping tools, has to
> account for the feature and disable it if necessary for older
> distributions.
>
> Cloning and reassign to vmdb2.

Based on more feedback from #10313225, I am also cloning and reassigning
this issue to fai and grml-debootstrap. Dear maintainers, please check
whether this issue is relevant for your packages.

Cheers
--
Sebastian Ramacher

[Paul Gevers]

Control: clone -1 -2
Control: reassign -2 freedom-make 0.32
Control: clone -1 -3
Control: reassign -3 debos 1.1.1-2

The same appears to apply for debos and freedom-maker. Dear maintainers,

please check whether this issue is relevant for your packages.

Paul

[Christopher Obbard]

Control: severity -1 important
Control: retitle -1 e2fsprogs generates filesystems which cannot be
mounted on systems with older e2fsprogs

It turns out for debos the situation is a bit different. Since debos
uses packages on the host to prepare the partitions, new features in
e2fsprogs from unstable which are unsupported in the target suite
causes the built system to not boot.

One such failure at boot-time of a Debian testing system is when
systemd-fsck tries to check a filesystem, it fails to mount the device:

/dev/mmcblk0p5 has unsupported feature(s): FEATURE_C12

So, in short we cannot run images built for suites with older e2fsprogs
on a host system which has e2fsprogs >=1.47.0-1.

Fixing grub2 will not fix the issue for debos, so I have retitled the
bug.

In debos with newer e2fsprogs, we can set the following flag on a
partition to disable the feature which allows the system running older
e2fsprogs to boot, but older e2fsprogs doesn't allow us to set this
flag:

features: [ "^orphan_file" ]

So to my mind, the bug is in e2fsprogs as it doesn't maintain backwards
compatibility.

I have reduced the severity of the bug since a workaround for debos
does exist. I wonder whether the bug should be reassigned to e2fsprogs
?

[Christopher Obbard]

[Theodore Ts'o]

On Tue, Feb 21, 2023 at 12:17:20PM +0000, Christopher Obbard wrote:
> Control: severity -1 important
> Control: retitle -1 e2fsprogs generates filesystems which cannot be
> mounted on systems with older e2fsprogs
>
> It turns out for debos the situation is a bit different. Since debos
> uses packages on the host to prepare the partitions, new features in
> e2fsprogs from unstable which are unsupported in the target suite
> causes the built system to not boot.
>
> One such failure at boot-time of a Debian testing system is when
> systemd-fsck tries to check a filesystem, it fails to mount the device:
>
> /dev/mmcblk0p5 has unsupported feature(s): FEATURE_C12
>
> So, in short we cannot run images built for suites with older e2fsprogs
> on a host system which has e2fsprogs >=1.47.0-1.
>
> Fixing grub2 will not fix the issue for debos, so I have retitled the
> bug.
>
> In debos with newer e2fsprogs, we can set the following flag on a
> partition to disable the feature which allows the system running older
> e2fsprogs to boot, but older e2fsprogs doesn't allow us to set this
> flag:
>
> features: [ "^orphan_file" ]
>
> So to my mind, the bug is in e2fsprogs as it doesn't maintain backwards
> compatibility.

E2fsprogs is always going to be adding new features. Even if I dumb
down /etc/mk2fs.conf to not enable this feature in Bookworm, it *will*
enable this new feature in the next release of Debian. This is true
for all file systems; we add new features to improve user experience.

So packages that are creating file systems to be consumed by older
target OS's must either (a) be aware of these changes for all file
systems (for example XFS is enabling the "bigtime" feature by default
in Bookworm to fix y2038 bug by expanding the size of the timestamp in
their inodes), OR, (b) run the mke2fs from the target OS in build
chroot.

This might get tricky for some target OS's, such as for example, GNU
Hurd, but the Hurd *absolutely* needs to either use mkfs.ext4 from
Hurd, or if you are using the mke2fs from Linux, you'll need
"mkfs.ext2 -O hurd".

The assumption that mkfs's from file systems never change and provide
full backwards compatibility for all target OS's is very much an
anti-pattern.

Best regards,

- Ted

[Gunnar Wolf]

tags 1031364 + upstream,forwarded,patch
thanks

I have reported this bug to the upstream author as issue #69:

https://gitlab.com/larswirzenius/vmdb2/-/issues/69

And proposed a simplistic patch as merge request #106:

https://gitlab.com/larswirzenius/vmdb2/-/merge_requests/106

Just for completeness sake, I'm including the patch in this report as
well:

diff --git a/vmdb/plugins/mkfs_plugin.py b/vmdb/plugins/mkfs_plugin.py
index 7bb32b6..f0bef3b 100644
--- a/vmdb/plugins/mkfs_plugin.py
+++ b/vmdb/plugins/mkfs_plugin.py
@@ -51,6 +51,17 @@ class MkfsStepRunner(vmdb.StepRunnerInterface):
cmd.append("-L")
cmd.append(label)

+ # Ext4 filesystem features large_dir and metadata_csum_seed
+ # are known to make versions of GRUB older than 2.06-8 unable
+ # to boot. Keep this around at least until it is no longer
+ # likely enough(?) users will try to build older target
+ # systems.
+ if fstype == "ext4":
+ cmd.append("-O")
+ cmd.append("^large_dir")
+ cmd.append("-O")
+ cmd.append("^metadata_csum_seed")
+
options = values["options"] or None
if options:
for opt in options.split(" "):

[Andreas Henriksson]

Control: forwarded -1 https://github.com/go-debos/debos/issues/78

Seems like the bookworm / metadata_csum_seed problem has happened in the
past for a different e2fsprogs feature and is discussed in the above
upstream github issue, so I'm marking this as forwarded to have the
two issues linked.

Regards,
Andreas Henriksson