Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Seeing progross during fsck on boot
137 views
Skip to first unread message
Mike
Sep 1, 2022, 6:00:05 PM9/1/22
to
Folks,
A long time, maybe 11 years ago, I built a NAS box based around comodity
hardware and the Debian of the day. It's currently been through several
apt-get dist-upgrades and currently running Debian 11 with loads of old
config grandfathered into it.
It's a server, so runs 24x7 but every few months or so falls on its
arse. Periodically it wants to fsck the disks, either because they've
gone 20 mounts or so without a fsck or more often because they've gone
however many days it is without a fsck.
This I can live with. I can even live with the fact that with several
TB file systems and quite a few files, the processs takes around fuor
hours. I'd just be nice to have some progress reported. While it
managed to spit details of any file system errors that have been found
and corrected, other than that it sits there utterly silent. For hours.
I can see that it's doing something as 1) it's expected behaviour and 2)
I can see the disk light solid on. Nevertheless, it would be nice to
see the progress bar indicting how it was getting along.
Does aoyone have any idea how to enable this? I have no idea where to
look. I've had several attempts at finding the answer with a well-known
Internet search engine but while I've found many similar questions and
even sometimes the same one as I'm asking but so far an answer has
proved illusive.
Does anyone have any suggestions for a solution or indeed where one may
look for one?
Regards,
Mike.
A long time, maybe 11 years ago, I built a NAS box based around comodity
hardware and the Debian of the day. It's currently been through several
apt-get dist-upgrades and currently running Debian 11 with loads of old
config grandfathered into it.
It's a server, so runs 24x7 but every few months or so falls on its
arse. Periodically it wants to fsck the disks, either because they've
gone 20 mounts or so without a fsck or more often because they've gone
however many days it is without a fsck.
This I can live with. I can even live with the fact that with several
TB file systems and quite a few files, the processs takes around fuor
hours. I'd just be nice to have some progress reported. While it
managed to spit details of any file system errors that have been found
and corrected, other than that it sits there utterly silent. For hours.
I can see that it's doing something as 1) it's expected behaviour and 2)
I can see the disk light solid on. Nevertheless, it would be nice to
see the progress bar indicting how it was getting along.
Does aoyone have any idea how to enable this? I have no idea where to
look. I've had several attempts at finding the answer with a well-known
Internet search engine but while I've found many similar questions and
even sometimes the same one as I'm asking but so far an answer has
proved illusive.
Does anyone have any suggestions for a solution or indeed where one may
look for one?
Regards,
Mike.
David Wright
Sep 1, 2022, 8:50:05 PM9/1/22
to
On Thu 01 Sep 2022 at 22:51:44 (+0100), Mike wrote:
> It's a server, so runs 24x7 but every few months or so falls on its
> arse. Periodically it wants to fsck the disks, either because they've
> gone 20 mounts or so without a fsck or more often because they've gone
> however many days it is without a fsck.
>
> This I can live with. I can even live with the fact that with several
> TB file systems and quite a few files, the processs takes around fuor
> hours. I'd just be nice to have some progress reported. While it
> managed to spit details of any file system errors that have been found
> and corrected, other than that it sits there utterly silent. For hours.
> I can see that it's doing something as 1) it's expected behaviour and 2)
> I can see the disk light solid on. Nevertheless, it would be nice to
> see the progress bar indicting how it was getting along.
You don't say which filesystem format you're using. The native
> It's a server, so runs 24x7 but every few months or so falls on its
> arse. Periodically it wants to fsck the disks, either because they've
> gone 20 mounts or so without a fsck or more often because they've gone
> however many days it is without a fsck.
>
> This I can live with. I can even live with the fact that with several
> TB file systems and quite a few files, the processs takes around fuor
> hours. I'd just be nice to have some progress reported. While it
> managed to spit details of any file system errors that have been found
> and corrected, other than that it sits there utterly silent. For hours.
> I can see that it's doing something as 1) it's expected behaviour and 2)
> I can see the disk light solid on. Nevertheless, it would be nice to
> see the progress bar indicting how it was getting along.
formats, ext2–4, can give you a progress bar with the -C option,
but only if filesystems are checked one at a time. With:
# fsck -f -n -C /dev/sdz4
I get progress bars on the first two passes, but the other three
flash by too quickly to make a judgment (29GB partition on a caddy).
I haven't delved into -C's optional argument because I'm using
an ordinary xterm.
Cheers,
David.
David
Sep 1, 2022, 10:00:05 PM9/1/22
to
On Fri, 2 Sept 2022 at 07:59, Mike <deb...@norgie.net> wrote:
[... fsck ... it would be nice to see the progress bar ...]
> Does anyone have any suggestions for a solution or indeed where one may
> look for one?
Hi,
To add to other replies, there is information about the '-C' option of the fsck
command in two different manpages. Run the below commands to display them:
man 8 fsck
man 8 e2fsck
(when you run fsck on a ext2/3/4 filesystem, it calls e2fsck to do the work).
[... fsck ... it would be nice to see the progress bar ...]
> Does anyone have any suggestions for a solution or indeed where one may
> look for one?
To add to other replies, there is information about the '-C' option of the fsck
command in two different manpages. Run the below commands to display them:
man 8 fsck
man 8 e2fsck
(when you run fsck on a ext2/3/4 filesystem, it calls e2fsck to do the work).
Porter Smith
Sep 1, 2022, 11:40:05 PM9/1/22
to
Along the lines of what the others have said, as a policy it is always an extremely excellent idea to make frequent and thorough backupa an store them in two or three separate places as cloud based storage is abundant and dirt cheap.
Mike
Sep 3, 2022, 5:00:06 AM9/3/22
to
On Fri, Sep 02, 2022 at 12:31:35AM +0200, DdB wrote:
> Just thinking about your request ...
>
> Imagine this: You run "fsck -N ..." and get a rough estimate about the
> time necessary to get the I/O and the job done, then it would be easy to
> set up some timed countdown in parallel with the real fsck job, just for
> you to have an idea about the time left.
>
> Alternatively, copy the whole device to another place, peform the fsck
> there and decide, if copying back the result would be faster of just
> running fsck on the original device.
>
> What i am intending to make you think about: Sometimes it is way more
> difficult and time consuming to get a rough estimate of some result,
> than to actually just get it for real. (You may talk to some math PHD
> about that.) ... not worth the effort, because even you would prefer to
> get the result as fast as possible and not wait twice the time just to
> know ahead of time, when the job is likely to finish.
>
> Does that make sense to you too?
>
>
Hi,
Thanks for the replies.
Rereading my original request, I think perhaps I wasn't entirely clear
on a couple of points:
1) When there server starts, the default behaviour is to fsck the disks
if they have been longer than 120 days (or something like that) without
a fsck. It's a server and runs 24x7, so you can bet your bottom dollar
that if if it does reboot, it will need to fsck. You can also bet that
it was an uncontrolled shutdown because the UPS caught fire or
something, so it's probably wise to run fsck. I haev no issue with this
happening
2) I'm not too worried about how long it's actually going to take. The
main issue is that I've had a couple of instances where the box has
rebooted, I've sat around waiting for it to reboot, wondered why it
hasn't, plugged a monitor in and seen jack on the screen, then just as I
panic about what major issue could be wrong with it and key CTRL+ALT+DEL
to see if I missed anything... I see the disk lights are solid on and
assume it must be running fsck. Sometimes it does find some erros and
spits that out to the screen. Otherwise it just sits there like a dog
that's been shown a card trick.
Maybe I'm being nostalgic but I seem to recall in days gone by that fsck
printed a progress bar out of hashes to show how it was getting along.
Someone asked about the file systems in use. Some are ext3 and some are
ext4.
TL;DR: When my server boots up and decdies to spent four hours fscking
the disks, I'd just like to see some indiction that it's still alive and
doing something :-)
Regards,
Mike.
> Just thinking about your request ...
>
> Imagine this: You run "fsck -N ..." and get a rough estimate about the
> time necessary to get the I/O and the job done, then it would be easy to
> set up some timed countdown in parallel with the real fsck job, just for
> you to have an idea about the time left.
>
> Alternatively, copy the whole device to another place, peform the fsck
> there and decide, if copying back the result would be faster of just
> running fsck on the original device.
>
> What i am intending to make you think about: Sometimes it is way more
> difficult and time consuming to get a rough estimate of some result,
> than to actually just get it for real. (You may talk to some math PHD
> about that.) ... not worth the effort, because even you would prefer to
> get the result as fast as possible and not wait twice the time just to
> know ahead of time, when the job is likely to finish.
>
> Does that make sense to you too?
>
>
Hi,
Thanks for the replies.
Rereading my original request, I think perhaps I wasn't entirely clear
on a couple of points:
1) When there server starts, the default behaviour is to fsck the disks
if they have been longer than 120 days (or something like that) without
a fsck. It's a server and runs 24x7, so you can bet your bottom dollar
that if if it does reboot, it will need to fsck. You can also bet that
it was an uncontrolled shutdown because the UPS caught fire or
something, so it's probably wise to run fsck. I haev no issue with this
happening
2) I'm not too worried about how long it's actually going to take. The
main issue is that I've had a couple of instances where the box has
rebooted, I've sat around waiting for it to reboot, wondered why it
hasn't, plugged a monitor in and seen jack on the screen, then just as I
panic about what major issue could be wrong with it and key CTRL+ALT+DEL
to see if I missed anything... I see the disk lights are solid on and
assume it must be running fsck. Sometimes it does find some erros and
spits that out to the screen. Otherwise it just sits there like a dog
that's been shown a card trick.
Maybe I'm being nostalgic but I seem to recall in days gone by that fsck
printed a progress bar out of hashes to show how it was getting along.
Someone asked about the file systems in use. Some are ext3 and some are
ext4.
TL;DR: When my server boots up and decdies to spent four hours fscking
the disks, I'd just like to see some indiction that it's still alive and
doing something :-)
Regards,
Mike.
Greg Wooledge
Sep 3, 2022, 8:10:06 AM9/3/22
to
On Sat, Sep 03, 2022 at 09:56:34AM +0100, Mike wrote:
> Rereading my original request, I think perhaps I wasn't entirely clear
> on a couple of points:
I thought it was clear. Some of the responses completely baffled me.
> Rereading my original request, I think perhaps I wasn't entirely clear
> on a couple of points:
One person even mentioned xterm -- like, *what*? How does xterm come
into play when you're booting a server and there's no GUI yet?
> Maybe I'm being nostalgic but I seem to recall in days gone by that fsck
> printed a progress bar out of hashes to show how it was getting along.
which ran (originally/mostly) in series, instead of in parallel. The
fsck progress bar(s) seem to have been lost in the move to systemd,
perhaps because writing a progress bar isn't considered viable in a
highly parallel boot system where other things might also be trying to
write to the console. Or perhaps for other reasons.
It appears that systemd-fsck(8) is the documentation for the new regime.
Note this paragraph:
systemd-fsck does not know any details about specific filesystems, and
simply executes file system checkers specific to each filesystem type
(/sbin/fsck.type). These checkers will decide if the filesystem should
actually be checked based on the time since last check, number of
mounts, unclean unmount, etc.
Also note that /lib/systemd/systemd-fsck is a compiled program, not a
shell script, so we can't just go in and edit the thing to make it pass -C.
Now, here's an interesting thing: fsck.ext(4) refers to e2fsck.conf(5).
Did you know that e2fsck (aka fsck.ext4) has a configuration file? I
didn't either.
e2fsck.conf(5) says that the pathname is /etc/e2fsck.conf (which doesn't
exist by default on Debian 11). It also says that there are some options
available, which can be set in the configuration file:
report_features
If this boolean relation is true, e2fsck will print the file
system features as part of its verbose reporting (i.e., if the
-v option is specified)
report_time
If this boolean relation is true, e2fsck will run as if the op‐
tions -tt are always specified. This will cause e2fsck to print
timing statistics on a pass by pass basis for full file system
checks.
report_verbose
If this boolean relation is true, e2fsck will run as if the op‐
tion -v is always specified. This will cause e2fsck to print
some additional information at the end of each full file system
check.
Unfortunately, I couldn't find any wording about the progress bar. I don't
see how to turn that on from the config file.
So... sadly, I don't see a simple way to get the progress bar back. You
could play around with this e2fsck.conf file and see what the options do,
or you could edit the freakin' systemd SOURCE CODE to change how
systemd-fsck calls fsck. I don't see a sensible way to insert a wrapper
script into the execution flow, so I really wouldn't try going down that
road.
If none of these options work, you might need to talk to the systemd people
about it, maybe putting in a feature request, or seeing if one of them knows
a trick that'll work.
Frank McCormick
Sep 3, 2022, 8:50:06 AM9/3/22
to
SIGNALS The following signals have the following effect when sent
to e2fsck.
SIGUSR1 This signal causes e2fsck to start displaying a
completion bar or emitting progress information. (See
discussion of the -C option.)
SIGUSR2 This signal causes e2fsck to stop displaying a
completion bar or emitting progress information.
--
Frank McCormick
David
Sep 3, 2022, 9:00:08 AM9/3/22
to
On Sat, 3 Sept 2022 at 18:57, Mike <deb...@norgie.net> wrote:
> Thanks for the replies.
>
> Rereading my original request, I think perhaps I wasn't entirely clear
> on a couple of points:
[...]
> Thanks for the replies.
>
> Rereading my original request, I think perhaps I wasn't entirely clear
> on a couple of points:
Hi again.
Those points seemed clear to me.
> Maybe I'm being nostalgic but I seem to recall in days gone by that fsck
> printed a progress bar out of hashes to show how it was getting along.
> TL;DR: When my server boots up and decdies to spent four hours fscking
> the disks, I'd just like to see some indiction that it's still alive and
> doing something :-)
indication if the replies that you already received have satisfied your
request for help, or not.
So I took another look at your request. I'm now able to provide more
information than last time. Which I am writing up because learned a few
things, so I thought I would share that with the list, so that we can help
each other figure things out.
There are four parts to this message:
1) Before systemd
2) Now with systemd
3) But what about initrd
4) Final part
The final part is the only part that's actually useful, probably :)
Part 1) Before systemd
----------------------
In the manpage 'man 5 fstab' read about the "sixth field".
In the manpage 'man 8 fsck' read about the '-A' option.
In the manpage 'man 8 fsck' read about the '-C' option.
In the manpage 'man 8 e2fsck' read about the '-C' option.
All of the above explains how fsck will work when /sbin/fsck or
/usr/sbin/fsck is in use. Before systemd, that external binary was run by
the init system during the boot process, and there would have been some
script somewhere that a sysadmin could change its arguments. So the above
information might have been of use in that situation.
The manpage 'man 8 fsck' documents several environment variables that I had
not noticed before, so just out of curiousity I went looking for how they
might be used and I discovered the following.
Part 2) Now with systemd
------------------------
/etc/fstab still works as described above. However there is documentation
out there that suggests that the work of fsck has now been taken over by
systemd-fsck, which is documented by 'man 8 systemd-fsck.service' and its
friends.
The /usr/lib/systemd/system/systemd-fsck-root.service file contains the
line:
ExecStart=/lib/systemd/systemd-fsck
I looked at the source code for that file here:
https://github.com/systemd/systemd/blob/main/src/fsck/fsck.c
which is interesting because it says:
"This program expects one or no arguments."
revealing that there is no way for the system administrator to
provide arguments or environment variables to the boot-time fsck process.
The source code shows that systemd-fsck generates these arguments
internally before it passes them to /sbin/fsck.
I imagine that could be overcome by copying the above service file to
/etc/lib/systemd/system/systemd-fsck-root.service and editing the above
ExecStart line to use /sbin/fsck instead.
I did try running /lib/systemd/systemd-fsck manually, but I don't have any
large drives here that are unencrypted, so every filesystem that I have
available to test just completed too quickly to show any progress bar.
(More about that in part 4 ... )
Part 3) But what about initrd
-----------------------------
When I looked at the Archlinux wiki at
https://wiki.archlinux.org/title/Fsck
it describes a different mechanism where fsck is done by the initrd.
The Debian machinery is different, but this idea made me think
that maybe this has nothing to do with systemd.
So, I did boot a test machine (Debian 11) with "break=bottom" and had
a dumb poke around the initrd, and I found this (transcribed by hand,
blanks omitted):
-----begin-----
(initramfs) cat /run/initramfs/fsck.log
Log of fsck -C -a -T -t ext4 /dev/sda2
Sat Sep 3 10:53:44 2022
SATPRO_X: clean, 34049/794624 files, 445226/3173376 blocks
Sat Sep 3 10:53:44 2022
-----end-----
And I also noticed that the console messages say (transcribed by hand):
-----begin-----
Begin: Will now check root file system ... fsck from util linux 2.36.1
[/sbin/fsck.ext4 (1) -- /dev/sda2] fsck.ext4 -a -C0 /dev/sda2
SATPRO_X: clean, 34049/794624 files, 445226/3173376 blocks
done.
-----end-----
And after more poking around, I'm guessing that fsck is called from
/usr/share/initramfs-tools/scripts function _checkfs_once() with '-C' (in
the fsck.log above) and that fsck.ext4 reports (in the console message
above) that it takes that to mean '-C0'.
So, it is currently my naive understanding that all of this is happening in
the initrd, before systemd starts. Which would mean that Part 2 of this
message is completely irrelevant. Someone please do correct me if that is
wrong.
Part 4) Final part
------------------
> Someone asked about the file systems in use. Some are ext3 and some are
> ext4.
algorithm that is orders of magnitude faster than ext3.
So if you convert your ancient ext3 filesystems to ext4, I expect you will
find that the fsck process that used to take hours will then take only
a minute or two, or even less, and so you won't care about the
progress bar any more. And
I suspect that is why no-one seems to care about this progress bar issue
these days.
Charles Curley
Sep 3, 2022, 10:20:05 AM9/3/22
to
On Sat, 3 Sep 2022 22:57:19 +1000
David <bounci...@gmail.com> wrote:
Nice write-up, especially the last part.
One nit-pick
> I imagine that could be overcome by copying the above service file to
> /etc/lib/systemd/system/systemd-fsck-root.service and editing the
> above ExecStart line to use /sbin/fsck instead.
I believe on Debian that should be
/etc/systemd/system/systemd-fsck-root.service
There is a systemd command for editing systemd files which will if
necessary do that copy transparently for you. I forget right now what
that is.
--
Does anybody read signatures any more?
https://charlescurley.com
https://charlescurley.com/blog/
David <bounci...@gmail.com> wrote:
Nice write-up, especially the last part.
One nit-pick
> I imagine that could be overcome by copying the above service file to
> /etc/lib/systemd/system/systemd-fsck-root.service and editing the
> above ExecStart line to use /sbin/fsck instead.
/etc/systemd/system/systemd-fsck-root.service
There is a systemd command for editing systemd files which will if
necessary do that copy transparently for you. I forget right now what
that is.
--
Does anybody read signatures any more?
https://charlescurley.com
https://charlescurley.com/blog/
David Wright
Sep 3, 2022, 11:10:05 AM9/3/22
to
On Sat 03 Sep 2022 at 08:03:38 (-0400), Greg Wooledge wrote:
> On Sat, Sep 03, 2022 at 09:56:34AM +0100, Mike wrote:
> > Rereading my original request, I think perhaps I wasn't entirely clear
> > on a couple of points:
>
> I thought it was clear. Some of the responses completely baffled me.
> One person even mentioned xterm -- like, *what*? How does xterm come
> into play when you're booting a server and there's no GUI yet?
TL;DR sorry to shock your sensibilities by the mention of xterm.
> On Sat, Sep 03, 2022 at 09:56:34AM +0100, Mike wrote:
> > Rereading my original request, I think perhaps I wasn't entirely clear
> > on a couple of points:
>
> I thought it was clear. Some of the responses completely baffled me.
> One person even mentioned xterm -- like, *what*? How does xterm come
> into play when you're booting a server and there's no GUI yet?
--
It doesn't. If you read what I wrote, you will see that I mentioned
xterm as a disclaimer that /I/ don't know anything about the arguments
to -C because /I/ was using xterm.
In case you missed it, my post was to point out that fsck can still,
with appropriate filesystems, spit out a progress bar of equals signs.
There was no indication then of which filesystems were in use until
the OP's second post, nor of their disposition, nor which init system
is being used. With sysvinit, I think it could be trivial to check the
disks sequentially, /if/ there were several disks /and/ the increase
in time taken were less important than the confidence gained in seeing
signs of life.
I don't have the time or confidence to delve into bowels of systemd
and initrds like David.
> > Maybe I'm being nostalgic but I seem to recall in days gone by that fsck
> > printed a progress bar out of hashes to show how it was getting along.
> Near as I can tell, that was a feature of the old sysv-rc scripts,
> which ran (originally/mostly) in series, instead of in parallel. The
> fsck progress bar(s) seem to have been lost in the move to systemd,
> perhaps because writing a progress bar isn't considered viable in a
> highly parallel boot system where other things might also be trying to
> write to the console. Or perhaps for other reasons.
others. As it's pointed out somewhere or other, you don't know whether
fsck is intact and competent to check other filesystems until it's
been checked itself.
When I booted my system this morning, I naturally selected the FSCK
option in Grub, and sure enough, I saw a progress bar as the root
filesystem was checked. Nothing for the others, though. (I use
systemd. I'm sure it's trivial to edit sysvinit scripts to add -C
and make sequential if either is not the default.)
As for why -C bothered me, I think I'm dredging up some wrinkle
I encountered when I wanted a cron job to print a little table to
the screen 30 secnds after booting. I remember having to open a
tty specifically, because stdout didn't seem to appear on the screen.
(I guess it would just try to email it instead.)
I also ?know that some people don't see all the booting messages on
the screen because they have some sort of ?splash screen (is that
what plymouth is all about?). So I don't know if -C <some argument>
is meant to cope with possibilities like that, or when fscking
non-root disks is still occurring while a graphical login screen
is displayed.
Cheers,
David.
David Wright
Sep 3, 2022, 11:10:05 AM9/3/22
to
On Sat 03 Sep 2022 at 08:18:18 (-0600), Charles Curley wrote:
> On Sat, 3 Sep 2022 22:57:19 +1000
> David <bounci...@gmail.com> wrote:
>
> Nice write-up, especially the last part.
>
> One nit-pick
>
>
> > I imagine that could be overcome by copying the above service file to
> > /etc/lib/systemd/system/systemd-fsck-root.service and editing the
> > above ExecStart line to use /sbin/fsck instead.
>
> I believe on Debian that should be
> /etc/systemd/system/systemd-fsck-root.service
>
> There is a systemd command for editing systemd files which will if
> necessary do that copy transparently for you. I forget right now what
> that is.
systemctl edit
> On Sat, 3 Sep 2022 22:57:19 +1000
> David <bounci...@gmail.com> wrote:
>
> Nice write-up, especially the last part.
>
> One nit-pick
>
>
> > I imagine that could be overcome by copying the above service file to
> > /etc/lib/systemd/system/systemd-fsck-root.service and editing the
> > above ExecStart line to use /sbin/fsck instead.
>
> I believe on Debian that should be
> /etc/systemd/system/systemd-fsck-root.service
>
> There is a systemd command for editing systemd files which will if
> necessary do that copy transparently for you. I forget right now what
> that is.
Controlled by SYSTEMD_EDITOR, subject to the vagaries of su/sudo.
Cheers,
David.
David
Sep 3, 2022, 11:10:05 AM9/3/22
to
On Sun, 4 Sept 2022 at 00:18, Charles Curley
be a simple change (lib --> etc) but I neglected to delete the 'lib'.
Also I noticed another error in my transcription of the console message,
I missed the hyphen in the package name, it should be:
Begin: Will now check root file system ... fsck from util-linux 2.36.1
<charle...@charlescurley.com> wrote:
> On Sat, 3 Sep 2022 22:57:19 +1000 David <bounci...@gmail.com> wrote:
> > I imagine that could be overcome by copying the above service file to
> > /etc/lib/systemd/system/systemd-fsck-root.service and editing the
> > above ExecStart line to use /sbin/fsck instead.
>
> I believe on Debian that should be
> /etc/systemd/system/systemd-fsck-root.service
Hi Charles, yes thanks for picking up that edit mistake, it was supposed to
> On Sat, 3 Sep 2022 22:57:19 +1000 David <bounci...@gmail.com> wrote:
> > I imagine that could be overcome by copying the above service file to
> > /etc/lib/systemd/system/systemd-fsck-root.service and editing the
> > above ExecStart line to use /sbin/fsck instead.
>
> I believe on Debian that should be
> /etc/systemd/system/systemd-fsck-root.service
be a simple change (lib --> etc) but I neglected to delete the 'lib'.
Also I noticed another error in my transcription of the console message,
I missed the hyphen in the package name, it should be:
Begin: Will now check root file system ... fsck from util-linux 2.36.1
Greg Wooledge
Sep 3, 2022, 11:40:06 AM9/3/22
to
On Sat, Sep 03, 2022 at 10:06:56AM -0500, David Wright wrote:
> When I booted my system this morning, I naturally selected the FSCK
> option in Grub, and sure enough, I saw a progress bar as the root
> filesystem was checked. Nothing for the others, though. (I use
> systemd. I'm sure it's trivial to edit sysvinit scripts to add -C
> and make sequential if either is not the default.)
That's great information. It sounds like the fsck of the root file system
> When I booted my system this morning, I naturally selected the FSCK
> option in Grub, and sure enough, I saw a progress bar as the root
> filesystem was checked. Nothing for the others, though. (I use
> systemd. I'm sure it's trivial to edit sysvinit scripts to add -C
> and make sequential if either is not the default.)
is handled inside the initramfs, and that the scripts/whatever which do
it are already passing the -C option.
So all that remains is to find a way to pass that option for the *other*
file systems, which are presumably done outside of the initramfs.
Unfortunately, that's where I hit a wall. It doesn't look like systemd
provides a friendly way to add that option. (I'll be happy to be proven
wrong.)
john doe
Sep 3, 2022, 1:00:05 PM9/3/22
to
On 9/3/2022 4:18 PM, Charles Curley wrote:
> On Sat, 3 Sep 2022 22:57:19 +1000
> David <bounci...@gmail.com> wrote:
>
> Nice write-up, especially the last part.
>
> One nit-pick
>
>
>> I imagine that could be overcome by copying the above service file to
>> /etc/lib/systemd/system/systemd-fsck-root.service and editing the
>> above ExecStart line to use /sbin/fsck instead.
>
> I believe on Debian that should be
> /etc/systemd/system/systemd-fsck-root.service
>
> There is a systemd command for editing systemd files which will if
> necessary do that copy transparently for you. I forget right now what
> that is.
>
I guess the CMD [1] in question is:
> On Sat, 3 Sep 2022 22:57:19 +1000
> David <bounci...@gmail.com> wrote:
>
> Nice write-up, especially the last part.
>
> One nit-pick
>
>
>> I imagine that could be overcome by copying the above service file to
>> /etc/lib/systemd/system/systemd-fsck-root.service and editing the
>> above ExecStart line to use /sbin/fsck instead.
>
> I believe on Debian that should be
> /etc/systemd/system/systemd-fsck-root.service
>
> There is a systemd command for editing systemd files which will if
> necessary do that copy transparently for you. I forget right now what
> that is.
>
$ systemctl edit [ <--full> ] <SERVICE-NAME>
[1] https://www.freedesktop.org/software/systemd/man/systemctl.html#
--
John Doe
Sven Joachim
Sep 3, 2022, 1:10:05 PM9/3/22
to
/etc/systemd/system.conf, or boot with the "systemd.show_status=auto"
option. See the systemd manpage:
,----
| KERNEL COMMAND LINE
| [...]
| systemd.show_status
| Takes a boolean argument or the constants error and auto. Can
| be also specified without an argument, with the same effect as
| a positive boolean. If enabled, the systemd manager (PID 1)
| shows terse service status updates on the console during
| bootup. With error, only messages about failures are shown,
| but boot is otherwise quiet. auto behaves like false until
| there is a significant delay in boot. Defaults to enabled,
| unless quiet is passed as kernel command line option, in which
| case it defaults to error. If specified overrides the system
| manager configuration file option ShowStatus=, see systemd-
| system.conf(5).
`----
Cheers,
Sven
David Wright
Sep 3, 2022, 3:10:05 PM9/3/22
to
On Sat 03 Sep 2022 at 11:31:27 (-0400), Greg Wooledge wrote:
> On Sat, Sep 03, 2022 at 10:06:56AM -0500, David Wright wrote:
> > When I booted my system this morning, I naturally selected the FSCK
> > option in Grub, and sure enough, I saw a progress bar as the root
> > filesystem was checked. Nothing for the others, though. (I use
> > systemd. I'm sure it's trivial to edit sysvinit scripts to add -C
> > and make sequential if either is not the default.)
>
> That's great information. It sounds like the fsck of the root file system
> is handled inside the initramfs, and that the scripts/whatever which do
> it are already passing the -C option.
In view of Sven's post, I should point out my boot line was:
> On Sat, Sep 03, 2022 at 10:06:56AM -0500, David Wright wrote:
> > When I booted my system this morning, I naturally selected the FSCK
> > option in Grub, and sure enough, I saw a progress bar as the root
> > filesystem was checked. Nothing for the others, though. (I use
> > systemd. I'm sure it's trivial to edit sysvinit scripts to add -C
> > and make sequential if either is not the default.)
>
> That's great information. It sounds like the fsck of the root file system
> is handled inside the initramfs, and that the scripts/whatever which do
> it are already passing the -C option.
linux /boot/vmlinuz-5.10.0-17-amd64 root=LABEL=toto05 ro systemd.show_status=true quiet forcefsck
I also wrote:
> > The root filesystem should never be checked at the same time as
> > others.
LABEL=toto05 / ext4 errors=remount-ro 0 1
UUID=C027-B627 /boot/efi vfat umask=0077 0 1
/dev/mapper/swap none swap nofail
LABEL=toto06 /home ext4 errors=remount-ro,nofail,noauto,user,exec,suid 0 2
LABEL=toto04 /axisbus ext4 errors=remount-ro,nofail 0 2
which has two filesystems marked 1 in field six.
AFAIK that is the default.
> So all that remains is to find a way to pass that option for the *other*
> file systems, which are presumably done outside of the initramfs.
> Unfortunately, that's where I hit a wall. It doesn't look like systemd
> provides a friendly way to add that option. (I'll be happy to be proven
> wrong.)
rebuilt the initfamfs just in case (though fstab is empty there),
then rebooted with forcefsck again.
I was worried about the fact that I get a 2–3 second blank during
boot up, but no fear, when the login prompt appeared, toto04 was
still being checked.
The appearance is quite different:
[ OK ] Started Disk Manager. (42.9% complete)
Checking in progress on 1 disk (43.2% complete)
Debian GNU/Linux 11 (bullseye) axis 192.168.1.14 … …
Checking in progress on 0 disks (100.0% complete)
_
Those are the final texts: the numbers were obviously increasing
while being displayed, but no visual progress bar.
I didn't see anything like that earlier today, nor after I had
restored the fstab to normality (forcefsck each time). But
I should point out that toto06 has a LUKS-encrypted filesystem
on it, which can only be decrypted after I login as user unlock
and type the passphrase.
Cheers,
David.
Mike
Sep 4, 2022, 8:20:06 AM9/4/22
to
Thanks for the very detailed description. This was just what I was
after. I'd kind of figured a few things, that it likely needed some
kind of switch to fsck to produce output and likely systemd was either
not passing that flag or swallowing the output. I've never delved into
how disks get fscked on boot, either with systemd or sysv, so I wasn't
really sure where to start looking.
Your explanation was very helpful and I think also the last point was a
good one. I've converted the remaining ext3 filesystems to ext4 and
will see how that goes.
It was an interesting point too that someone suggested about
ShowStatus=auto. That sounds helpful, athough when I look in
system.conf I notice two things:
The comment at the top:
# Entries in this file show the compile time defaults.
And the entry itself:
#ShowStatus=yes
This suggests that it should be printing status stuff. I checked my
config on my desktop and saw the same. That I recall only prints one
line during startup for a service that fails to start and that I've
never bothered to fix.
I'll do some more digging on my desktop to understand the ShowStatus
thing but for now I'll be happy to see how I get along with ext4.
Thanks again to everyone who offered their input into this. It's been
very helpful for me.
Regards,
Mike.
David Wright
Sep 4, 2022, 10:30:06 AM9/4/22
to
On Sun 04 Sep 2022 at 13:15:24 (+0100), Mike wrote:
> On Sun, Sep 04, 2022 at 01:01:03AM +1000, David wrote:
> > On Sun, 4 Sept 2022 at 00:18, Charles Curley
> > <charle...@charlescurley.com> wrote:
> > > On Sat, 3 Sep 2022 22:57:19 +1000 David <bounci...@gmail.com> wrote:
> >
> > > > I imagine that could be overcome by copying the above service file to
> > > > /etc/lib/systemd/system/systemd-fsck-root.service and editing the
> > > > above ExecStart line to use /sbin/fsck instead.
> > >
> > > I believe on Debian that should be
> > > /etc/systemd/system/systemd-fsck-root.service
> >
> > Hi Charles, yes thanks for picking up that edit mistake, it was supposed to
> > be a simple change (lib --> etc) but I neglected to delete the 'lib'.
> >
> > Also I noticed another error in my transcription of the console message,
> > I missed the hyphen in the package name, it should be:
> > Begin: Will now check root file system ... fsck from util-linux 2.36.1
> >
> On Sun, Sep 04, 2022 at 01:01:03AM +1000, David wrote:
> > On Sun, 4 Sept 2022 at 00:18, Charles Curley
> > <charle...@charlescurley.com> wrote:
> > > On Sat, 3 Sep 2022 22:57:19 +1000 David <bounci...@gmail.com> wrote:
> >
> > > > I imagine that could be overcome by copying the above service file to
> > > > /etc/lib/systemd/system/systemd-fsck-root.service and editing the
> > > > above ExecStart line to use /sbin/fsck instead.
> > >
> > > I believe on Debian that should be
> > > /etc/systemd/system/systemd-fsck-root.service
> >
> > Hi Charles, yes thanks for picking up that edit mistake, it was supposed to
> > be a simple change (lib --> etc) but I neglected to delete the 'lib'.
> >
> > Also I noticed another error in my transcription of the console message,
> > I missed the hyphen in the package name, it should be:
> > Begin: Will now check root file system ... fsck from util-linux 2.36.1
> >
> Thanks for the very detailed description. This was just what I was
> after. I'd kind of figured a few things, that it likely needed some
> kind of switch to fsck to produce output and likely systemd was either
> not passing that flag or swallowing the output. I've never delved into
> how disks get fscked on boot, either with systemd or sysv, so I wasn't
> really sure where to start looking.
As I said, we don't know the disposition of your disks. The root
> after. I'd kind of figured a few things, that it likely needed some
> kind of switch to fsck to produce output and likely systemd was either
> not passing that flag or swallowing the output. I've never delved into
> how disks get fscked on boot, either with systemd or sysv, so I wasn't
> really sure where to start looking.
partition isn't a problem: I've made no changes like the above, but
I can see it's being checked. But if you also need to check more than
one other partition at boot time, then the penalty for obtaining a
progress indication may be serialisation, which sounds undesirable
in your case (parallel takes four hours).
> Your explanation was very helpful and I think also the last point was a
> good one. I've converted the remaining ext3 filesystems to ext4 and
> will see how that goes.
> It was an interesting point too that someone suggested about
> ShowStatus=auto. That sounds helpful, athough when I look in
> system.conf I notice two things:
>
> The comment at the top:
>
> # Entries in this file show the compile time defaults.
>
> And the entry itself:
>
> #ShowStatus=yes
>
> This suggests that it should be printing status stuff. I checked my
> config on my desktop and saw the same. That I recall only prints one
> line during startup for a service that fails to start and that I've
> never bothered to fix.
is passed as kernel command line option, in which case it defaults
to error." AIUI the Debian default /is/ quiet, and you can see it
in my kernel command line that I posted. (IIRC, not including quiet
can be somewhat overwhelming in the level of detail it spews out.)
So you need, as I do, to use the kernel command line to set it,
rather than system.conf.
> I'll do some more digging on my desktop to understand the ShowStatus
> thing but for now I'll be happy to see how I get along with ext4.
>
> Thanks again to everyone who offered their input into this. It's been
> very helpful for me.
David.
David
Sep 4, 2022, 11:30:05 AM9/4/22
to
On Sun, 4 Sept 2022 at 22:15, Mike <deb...@norgie.net> wrote:
> Thanks for the very detailed description. This was just what I was
> after. I'd kind of figured a few things, that it likely needed some
> kind of switch to fsck to produce output and likely systemd was either
> not passing that flag or swallowing the output. I've never delved into
> how disks get fscked on boot, either with systemd or sysv, so I wasn't
> really sure where to start looking.
>
> Your explanation was very helpful and I think also the last point was a
> good one. I've converted the remaining ext3 filesystems to ext4 and
> will see how that goes.
Yes, I feel confident that will give you a huge benefit. You can test it,
> Thanks for the very detailed description. This was just what I was
> after. I'd kind of figured a few things, that it likely needed some
> kind of switch to fsck to produce output and likely systemd was either
> not passing that flag or swallowing the output. I've never delved into
> how disks get fscked on boot, either with systemd or sysv, so I wasn't
> really sure where to start looking.
>
> Your explanation was very helpful and I think also the last point was a
> good one. I've converted the remaining ext3 filesystems to ext4 and
> will see how that goes.
see below.
I spent a bit more time looking at this and I am confident now about the
conclusions I have reached.
My previous message was "thinking out load" and somewhat inconclusive, the
contents of this message are more useful.
I have been curious in the past about how all this works, so I had an
interest in looking into it, and am happy to share the results.
1) At boot, the filesystems which have 1 in the 'passno' (sixth) field of
/etc/fstab are checked by /sbin/fsck in the initrd.
I believe systemd is irrelevant, because it is not running at that time.
2) After systemd starts, as explained by 'man 8 systemd-fsck', systemd only
checks the root filesystem if that was not already done by the initrd.
Having systemd check the root file system is something that we want to
avoid, because it would need to unmount it.
When running, systemd also checks the filesystems which have values of
2 or higher in the 'passno' field of etc/fstab, because these are not
checked by the initrd processes.
3) The command 'fsck' works on many filesystems. It is provided by the
'util-linux' package.
In the case of ext2/3/4 filesystems, 'fsck' invokes 'e2fsck' also known
as 'fsck.ext4' to do the work. Those are provided by the 'e2fsprogs'
package.
4) systemd-fsck also uses 'e2fsck' to do the work on ext2/3/4 systems.
5) Not important, but the '-C' argument of fsck and e2fsck are slightly
different.
Giving '-C' to fsck causes it to invoke e2fsck with '-C0'.
Evidence:
# fsck -C -N /dev/sda1
fsck from util-linux 2.36.1
[/usr/sbin/fsck.ext4 (1) -- /mnt/Z] fsck.ext4 -C0 /dev/sda1
#
'-N' invokes a "dry-run", which just prints the command that fsck
passes to its helper program, instead of executing it
Note that e2fsck and fsck.ext4 are the same program.
6) e2fsck is capable of checking multiple filesystems at the same time, for
faster completion when there are multiple drives with the same number in
the 'passno' field of /etc/fstab.
But, as noted by David Wright, e2fsck is only capable of displaying one
progress bar at a time.
7) Looking at the source of fsck
https://sources.debian.org/src/e2fsprogs/1.46.5-2/e2fsck/unix.c
around line 445 is interesting.
It confirms that when fsck gives output like this, often seen at boot:
<device>: clean, 123/456 files, 789/1011 blocks
that the keyword "clean" is fsck telling us "I looked briefly at the
<device> and I decided that it did not need to be scanned".
The code shows tht when this occurs, no progress bar is displayed.
A progress bar is only displayed when fsck actually scans the device.
8) e2fsck can be forced to scan a device by giving it the '-f' option. This
will be done by the initrd code if 'forcefsck' is specified on the
kernel boot command. That is quite an easy test for anyone to do.
9) This kernel parameter 'forcefsck' is detected by
/usr/share/initramfs-tools/init
and slightly modified before being passed to the _checkfs_once()
function in
/usr/share/initramfs-tools/functions
10) I have confirmed that adding 'forcefsck' does cause a progress bar to
be displayed. I tested here on Debian 10 i386 because I needed a slow
machine to see it, because ext4 checking is so fast.
11) I also confirmed that the fsck progress bar is displayed even if
'quiet' is also specified on the kernel command line.
Greg Wooledge
Sep 4, 2022, 3:20:06 PM9/4/22
to
On Mon, Sep 05, 2022 at 01:19:29AM +1000, David wrote:
> 6) e2fsck is capable of checking multiple filesystems at the same time, for
> faster completion when there are multiple drives with the same number in
> the 'passno' field of /etc/fstab.
>
> But, as noted by David Wright, e2fsck is only capable of displaying one
> progress bar at a time.
On some old Red Hat systems at work, I have observed e2fsck performing
> 6) e2fsck is capable of checking multiple filesystems at the same time, for
> faster completion when there are multiple drives with the same number in
> the 'passno' field of /etc/fstab.
>
> But, as noted by David Wright, e2fsck is only capable of displaying one
> progress bar at a time.
parallel scans of file systems on multiple disks. On the systems in
question, it showed one progress bar *at a time* in a cyclic rotation.
That is, it would show the progress bar for file system number one, for
a few seconds, and then the progress bar for file system number two, for
a few seconds, and so on.
I don't know whether the e2fsck on Debian does this. Or even whether
modern Red Hat descendant systems do it. It's just what I saw in
the past.
0 new messages