fs corruption

1 view
Skip to first unread message

bad sector

unread,
Jun 20, 2022, 7:43:26 PMJun 20
to

Is an OS-level mishandling of an ext4 file-system possible to an extent that causes a system lockup requiring a journal replay on the next boot?

One of my data HD's is a 3.5" 4tb black WD with a single GPT partition and just less than 10,000 hours, i.e. brand new :-)

I noticed a lockup on 2 or 3 occasions necessarily requiring a hard reset which forced a journal replay on the next boot. On the last such boot there was a fury of disk 'roll' similar to what I heard on the journal replays with the exception that this time its only partition became invisible/unmountable (set to be mounted in fstab). I tried a few fsck swithches to no avail.

I dd backups regularly so even if it's unrecoverable I'm not into any significant data loss but the filesystem recovery is a challenge. I have no other 4tb drive, only a few 2tbs. On a testdisk test-run it occured to me that if the disk were more than half full then I may have to recover to 2 drives. I never used testdisk before, is this possible? Is there a better utility for such a task?

David W. Hodgins

unread,
Jun 20, 2022, 8:08:06 PMJun 20
to
On Mon, 20 Jun 2022 19:43:20 -0400, bad sector <forgetski@postit_invalid_.gov> wrote:
> Is an OS-level mishandling of an ext4 file-system possible to an extent that causes a system lockup requiring a journal replay on the next boot?

<snip details>

In theory it's possible. I've never seen it happen where it wasn't the result of a
hardware (drive head crash, bad connectors or wires), power (inluding lightning
strikes nearby), or user induced (kill power while disk is being written) problems.

I've seen cases where the power supply was enough for normal usage, but adding
one more plug in usb drive would cause spontaneous power offs during peak usage.

Strip out all hardware not absolutely needed. Disconnect printer, dvd drive, etc.

Boot to run level 1 (rescue.target). Check dmesg and any available logs for
indications of whats going wrong.

Regards, Dave Hodgins

bad sector

unread,
Jun 21, 2022, 12:26:12 AMJun 21
to
The user homes were linked to this data disk so
the first sign was that I could not log-in, the
partition wasn't mounted.

Once I knew that this was no joke I looked into
the backup folder on another disk with
ls -t | head -n1
to get the youngest file/folder date (11 Jun 2022)

so I knew when I had made the last backup

fsck suggests some (default?) alternate superblock
locations but for this disk *testdisk* coughed up 7
real ones hiding like dogshit in tall grass

superblock 71663616, blocksize=4096 [dataX-4tb]
superblock 78675968, blocksize=4096 [dataX-4tb]
superblock 102400000, blocksize=4096 [dataX-4tb]
superblock 214990848, blocksize=4096 [dataX-4tb]
superblock 512000000, blocksize=4096 [dataX-4tb]
superblock 550731776, blocksize=4096 [dataX-4tb]
superblock 644972544, blocksize=4096 [dataX-4tb]

I don't wanna take anything away from cgsecurity
the makers of testdisk but maybe this superblock
finding feature could be included in fsck as it
would I think reduce the number of buttons being
chewed off seat cushions by dimwits like yours truly.

Feeding the middle one of these to fsck got it
running for the goal-line (took a looooong time).

# fsck.ext4 -y -b 214990848 -B 4096 /dev/sdb1

Something I discovered though was that there were
some empty folders in the backup (made with some
scripts or just copypasted. I'll have to come up
with a better idea, some event triggered script
that makes dated folder copies and deletes all but
the last 3 of each folder copy found. But I have
a million other things to do :-(

J.O. Aho

unread,
Jun 21, 2022, 1:45:01 AMJun 21
to
On 21/06/2022 01.43, bad sector wrote:

> I dd backups regularly so even if it's unrecoverable I'm not into any significant data loss but the
> filesystem recovery is a challenge. I have no other 4tb drive, only a few 2tbs. On a testdisk test-run
> it occured to me that if the disk were more than half full then I may have to recover to 2 drives.

You could use a lvm setup where you have multiple HDD's (SSD's or mix),
you could then store the dd file on the lvm without problem.

Using btrfs or zfs as the file system, then you can add multiple disks
without the need of lvm.

--

//Aho

bad sector

unread,
Jun 21, 2022, 6:35:07 AMJun 21
to
On 6/21/22 01:44, J.O. Aho wrote:
> On 21/06/2022 01.43, bad sector wrote:
>
>> I dd backups regularly so even if it's unrecoverable I'm not into
>> any significant data loss but the filesystem recovery is a
>> challenge. I have no other 4tb drive, only a few 2tbs. On a
>> testdisk test-run it occured to me that if the disk were more than
>> half full then I may have to recover to 2 drives.
>
> You could use a lvm setup where you have multiple HDD's (SSD's or
> mix), you could then store the dd file on the lvm without problem.

hmmm, I didn't know you could spread LVM on several media. I could get really confused by something like that :-)

> Using btrfs or zfs as the file system, then you can add multiple
> disks without the need of lvm.

but I did read once that btrsf was dd-unfriendly

I mis-spoke in my OP, I was under the impression that testdisk would recover the goods onto the same drive, like good-copy duplicates, but the warning it gave me probably meant something else. Three days ago I didn't know that testdisk existed so maybe it does that too. The success (in my reply to David) consisted of ALL the content being rebuilt in "lost & found" in two or three numbered folders. It's only my system partitions that I back up using dd, the rest is little scripts and copy-paste.

So,

Question-1,

Is btrsf more robust in terms not of backups but in terms of the CAUSES of this incident? I never heard that ext4 could be problematic if used correctly.


Question-2

My 4tb barfed, I'm about 99% sure it was a software fubar, as I said the lockups hapened on the same system and the subsequent drum-roll like fs-replay disk-rolls on another one next booted. I'm not going to point fingers because I'm not sure which. Is 99% good enough to go or do I give the drive to someone I wanna feel sorry for only in public? My other drives, all 2tb's, don't have to use GPT and as I always say prejudice may not be elegant but it sure works as a part of a defensive knee-jerk volley.

J.O. Aho

unread,
Jun 21, 2022, 9:09:29 AMJun 21
to
On 21/06/2022 12.34, bad sector wrote:
> On 6/21/22 01:44, J.O. Aho wrote:

>> You could use a lvm setup where you have multiple HDD's (SSD's or
>> mix), you could then store the dd file on the lvm without problem.
>
> hmmm, I didn't know you could spread LVM on several media. I could get really confused by something like that :-)
>
>> Using btrfs or zfs as the file system, then you can add multiple
>> disks without the need of lvm.
>
> but I did read once that btrsf was dd-unfriendly

You shouldn't dd a btrfs partition, there is btrfs send/receive for
making data dumps, but now I was talking about just storing an image
created with dd on the btrfs/zfs.

> I mis-spoke in my OP, I was under the impression that testdisk would recover the goods onto the same drive

It seems it has a forensic element to it's design, you don't want to
destroy the original.
At least that is my guess, as I never used it.

> The success (in my reply to David) consisted of ALL the content being rebuilt in "lost & found" in two or three numbered folders.

yeah, that is the files/chunks that couldn't be determined where to
place them.

> It's only my system partitions that I back up using dd, the rest is little scripts and copy-paste.

Not that this will change anything for you, but as I have setup is two
SSD with btrfs, one for / and one for /home, as there are less things
happening on / that is work keeping, I take a snapshot once a month and
btrfs send/receive it to an HDD, for the /home I do the same once a week.

btrfs can diff two snapshots and send the difference between those and
that way I get content wise a exact copy on the HDD's, and if something
would happen with the SSD I can copy back the data to the current disks
or to a new one.



> Is btrsf more robust in terms not of backups but in terms of the CAUSES of this incident?
> I never heard that ext4 could be problematic if used correctly.

I would say that all filesystems has their strength and weaknesses, I do
think btrfs is good enough for production environment, but there are
things that works less well too It's different from ext, so there are
new things to learn, specially when you "raid" together multiple
devices. There are some "raid" modes you should avoid, but traditional
"raid" 0/1/10 works quite fine.

The most resilient filesystem I have used was raiserfs, but for some
people that one has been more of issues and the good thing for you is
that you don't have to consider it as an option as it's on it's way out
of the kernel.

The one I was a bit looking forward to (before I spend money on larger
SSD) is bcachefs, it's still in early stages but would make it easy to
add a small SSD as data cache for your HDDs, would make the file system
feel a bit snappier for those things you do more frequently. I don't
recommend to use it until it's at least merged into the kernel.



> My 4tb barfed, I'm about 99% sure it was a software fubar, as I said the lockups hapened
> on the same system and the subsequent drum-roll like fs-replay disk-rolls on another one
> next booted.

The risk for data corruption is high when you have high write when you
decide to reset the computer, as it's difficult to say what data is
written down at the moment, so it can be the meta data used for
replaying or it can be the actual data. Even the most robust filesystem
could get corrupted badly if the reset comes in the wrong moment.

Had you had a raid 1 system, it could have been that one disk had been
corrupted and the other not, or in worst case both... that's why high
end raid cards comes with a battery, so that the last writes will be
finished during a reset.


> My other drives, all 2tb's, don't have to use GPT and as I always say prejudice may
> not be elegant but it sure works as a part of a defensive knee-jerk volley.

It's up to you is MBR good enough or not for your needs, but most
distributions are using GPT if possible nowadays.

--

//Aho

bad sector

unread,
Jun 21, 2022, 10:08:12 AMJun 21
to
On 6/21/22 09:09, J.O. Aho wrote:
> On 21/06/2022 12.34, bad sector wrote:
>> On 6/21/22 01:44, J.O. Aho wrote:
>
>>> You could use a lvm setup where you have multiple HDD's (SSD's
>>> or mix), you could then store the dd file on the lvm without
>>> problem.
>>
>> hmmm, I didn't know you could spread LVM on several media. I could
>> get really confused by something like that :-)
>>
>>> Using btrfs or zfs as the file system, then you can add multiple
>>> disks without the need of lvm.
>>
>> but I did read once that btrsf was dd-unfriendly
>
> You shouldn't dd a btrfs partition, there is btrfs send/receive for
> making data dumps, but now I was talking about just storing an image
> created with dd on the btrfs/zfs.

thanks, I get it, storing large dd files is ok but dd-ing a btrfs partition is a no-no

>> I mis-spoke in my OP, I was under the impression that testdisk
>> would recover the goods onto the same drive
>
> It seems it has a forensic element to it's design, you don't want to
> destroy the original. At least that is my guess, as I never used it.

seems VERY capable with many arms and legs! Worth a good read ..when I get around to it!

>> The success (in my reply to David) consisted of ALL the content
>> being rebuilt in "lost & found" in two or three numbered folders.
>
> yeah, that is the files/chunks that couldn't be determined where to
> place them.

something like that, directory corruption thingie


>> My 4tb barfed, I'm about 99% sure it was a software fubar, as I
>> said the lockups hapened on the same system and the subsequent
>> drum-roll like fs-replay disk-rolls on another one next booted.
>
> The risk for data corruption is high when you have high write when
> you decide to reset the computer, as it's difficult to say what data
> is written down at the moment, so it can be the meta data used for
> replaying or it can be the actual data. Even the most robust
> filesystem could get corrupted badly if the reset comes in the wrong
> moment.

Those resets came on total lockups, I had no choice, and the lockups didn't come on a shutdoen but just in the middle of some music work or such. And THEN on the next reboot to the same or a different system I'd get the disk spooling as usual when doing a replay. The last one of these, or thre lockup/reset leading to it, is what didn't work as-per supposed-to.


> Had you had a raid 1 system, it could have been that one disk had
> been corrupted and the other not, or in worst case both... that's why
> high end raid cards comes with a battery, so that the last writes
> will be finished during a reset.

I had thought of parallel raid a long time ago but I figured that it won't protect me from myself; if I accidentally delete, it'll get mirrored too. I'm happy with dd'd system partitions, been doing them for like 25 years, and for the rest I'll have to study up some scripting like ....whenever I save a file it gets saved as a new version in the triple backup folders with the date of the last save without touching the previous two bu's.


>> My other drives, all 2tb's, don't have to use GPT and as I always
>> say prejudice may not be elegant but it sure works as a part of a
>> defensive knee-jerk volley.
>
> It's up to you is MBR good enough or not for your needs, but most
> distributions are using GPT if possible nowadays.

I partition and sometimes format beforehand, the distros do with existing partitions ONLY. At my age the amount of data worth keeping is in freefall :-))))


J.O. Aho

unread,
Jun 21, 2022, 10:58:20 AMJun 21
to
On 21/06/2022 16.08, bad sector wrote:

> I had thought of parallel raid a long time ago but I figured that it won't protect me from myself;
> if I accidentally delete, it'll get mirrored too.

Yes, raid is not to protect you from doing bad things, it's just to
protect you from a sudden hardware failure causing you data loss.

> I'm happy with dd'd system partitions, been doing them for like 25 years,
> and for the rest I'll have to study up some scripting like ....whenever I save
> a file it gets saved as a new version in the triple backup folders with the date
> of the last save without touching the previous two bu's.

If I need versioning, I use git, then you can revert to an older
version, as I don't often need this, so I do not use NILFS which can
version your files automatically.

Sure dd has some advantages, but it duplicates unchanged data over and
over again, that's what I like with btrfs, it will not duplicate the
data when I take a snapshot, it will just store changes.



>> It's up to you is MBR good enough or not for your needs, but most
>> distributions are using GPT if possible nowadays.
>
> I partition and sometimes format beforehand, the distros do with
> existing partitions ONLY. At my age the amount of data worth keeping is in freefall :-))))

I tend to use distros that require you to do a lot of manual work, it's
really seldom I have the opportunity to let an installer decide for me.

In a way I mess the old days when I had my own blend of RedHat 7.2,
which I used quite many years after EOL with updated system parts as
kernel and glibc and I had even gnome2 libs so I could use apps written
for that horrible creation. Life felt a lot simpler back then.


--

//Aho



Reply all
Reply to author
Forward
0 new messages