after the update to ALTF, fsck throws an HD error and refuses to ForceFix [SOLVED: last resort fix if superblock has different block count than physical device, and FS cannot be repaired because of that]

[Markus Quandt]

Hi all,

I just updated to ALT-F and I am really overwhelmed with how capable this looks, and how smooth the upgrade went in principle...

However, the auto-check detected a file system error (must have been there before, but the D-Link firmware and myself apparently succesfully ignored that for years) that now prevents the main data volume from mounting.

• Unable to automatically fix md1, mounting Read Only: fsck 1.41.14 (22-Dec-2010)
  /dev/md1: The filesystem size (according to the superblock) is 487855872 blocks
  The physical size of the device is 487855856 blocks
  Either the superblock or the partition table is likely to be corrupt!
  
  
  /dev/md1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
  	(i.e., without -a or -p options)

As advised in the interface, I then initiated a ForceFix, giving this (but don't know if I should have waited for the word 'fixing' in red to disappear before switching to the status screen?)

Anyway, for now I get this:

• Checking md1 finished with status code 8: fsck 1.41.14 (22-Dec-2010)
  e2fsck 1.41.14 (22-Dec-2010)
  The filesystem size (according to the superblock) is 487855872 blocks
  The physical size of the device is 487855856 blocks
  Either the superblock or the partition table is likely to be corrupt!
  Abort? yes

This is where my wisdom ends. I have little idea about file systems, partitions, etc, especially not under Linux.

I did wonder whether a SHRINK and subsequent ENLARGE operation on the FS would not constitute an automatic correction here, but don't dare starting this at this stage.

Background info:

The device is a DNS-320 A1, the update happened today with ALT-F 1.0, the disks are two WD 2TB (NOT of the exact identical type!) in an RAID1 setup. Filesystem is EXT3 (carried over from D-Link), I skipped the 'abracadabra' screen altogether in ALT-F when configuring for the first time.

Disks and FS worked totally fine under the original D-Link firmware, except that I wanted to get rid of SMB1, and that Windows 10 on my client PCs does not play well enough with NFS.

Thanks for any advice you might be able to give (and again, thanks for what appears to be a great piece of software)

Markus

PS: I could also post detailed FS info if that helps, as I found the corresponding screen...

[João Cardoso]

On Monday, 8 June 2020 22:51:22 UTC+1, Markus Quandt wrote:

Hi all,

I just updated to ALT-F and I am really overwhelmed with how capable this looks, and how smooth the upgrade went in principle...

However, the auto-check detected a file system error (must have been there before, but the D-Link firmware and myself apparently succesfully ignored that for years) that now prevents the main data volume from mounting.

Mounting read/write mode, you can still see its contents in read only mode, right?

 

• Unable to automatically fix md1, mounting Read Only: fsck 1.41.14 (22-Dec-2010)
  /dev/md1: The filesystem size (according to the superblock) is 487855872 blocks
  The physical size of the device is 487855856 blocks
  Either the superblock or the partition table is likely to be corrupt!
  
  
  /dev/md1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
  	(i.e., without -a or -p options)

As advised in the interface, I then initiated a ForceFix, giving this (but don't know if I should have waited for the word 'fixing' in red to disappear before switching to the status screen?)

Anyway, for now I get this:

• Checking md1 finished with status code 8: fsck 1.41.14 (22-Dec-2010)
  e2fsck 1.41.14 (22-Dec-2010)
  The filesystem size (according to the superblock) is 487855872 blocks
  The physical size of the device is 487855856 blocks
  Either the superblock or the partition table is likely to be corrupt!
  Abort? yes

The last "Abort? yes" is odd, did you run fsck manualy?

When running "forceFix", the command used is 'fsck -fy /dev/md1', which don't ask for questions, assuming force (f) and a answer of yes (y) to all questions (-fy)

That is the only way I know to tentatively fix the 16 blocks mismatch between the fs size which is bigger than the RAID device enclosing it.

Other than growing the RAID device, if possible, or the partitions that make up (the components) the RAID, if possible.

But I don't have the characteristics of your system (it is in System->Utilities, Logs, System Configuration)

So your chances are to made a backup, if possible, and rerun the force fix or unmount the md1 device and run the fsck -fy from the command line.

This is where my wisdom ends. I have little idea about file systems, partitions, etc, especially not under Linux.

I did wonder whether a SHRINK and subsequent ENLARGE operation on the FS would not constitute an automatic correction here, but don't dare starting this at this stage.

Yes, but that requires a sucessful fsck first...

[Markus Quandt]

Hi, thanks for the quick response!

The "ForceFix" was initated as an action from ALT-Fs File Systems page, and it auto-answers 'yes' to the abort question...

Unfortunately, the device was and is not being mounted, not even in ro mode.

But I have accessed the box via SSH in the meantime and am running a manual e2fsck -fy on the device right now. For the second time actually, the first one did not fix the size mismatch. Following advice found per various Linux forums, I have also tried 'resize2fs -f', but that again aborts, with an error about 'short read' (logically, in fact), and sends me back to fsck.

I guess I'll have to re-partition. That's of course a pain, but I'll be able to reconstruct most of the data from other media, and nothing on the disk was vital anyway.

Let's see what is left after the new fsck. It's now trying to 'fix' lots of new errors...

M.

[João Cardoso]

On Tuesday, 9 June 2020 16:57:25 UTC+1, Markus Quandt wrote:

Hi, thanks for the quick response!

The "ForceFix" was initated as an action from ALT-Fs File Systems page, and it auto-answers 'yes' to the abort question...

Unfortunately, the device was and is not being mounted, not even in ro mode.

hmmm, that means that the force fix hasn't able to fix the mismatch. But you was able to see the fs contents when it was in read only mode?

 

But I have accessed the box via SSH in the meantime and am running a manual e2fsck -fy on the device right now. For the second time actually, the first one did not fix the size mismatch. Following advice found per various Linux forums, I have also tried 'resize2fs -f', but that again aborts, with an error about 'short read' (logically, in fact), and sends me back to fsck.

I guess I'll have to re-partition. That's of course a pain,

Not really, recovering data from backups is much worse and lengtly. To repartition and reformat just use Disk->Wizard and select "raid1" and "ext4".

 

but I'll be able to reconstruct most of the data from other media, and nothing on the disk was vital anyway.

Let's see what is left after the new fsck. It's now trying to 'fix' lots of new errors...

Let us know if fsck -fy is able to fix the short read due to the fs and device size mismatch.

[Markus Quandt]

Hi, no, I couldn't see the device content. The file system was recognized by ALT-F and the linux tools, but that was basically all.

ATM, I'm keeping my finger on the 'y' key to keep fsck going...

Best, M.

[Markus Quandt]

One more question - under the DISK/Wizard page, choosing the RAID1 and ext4 options (presently, I have ext3!), it tells me it would re-format, and I would loose all data. Not encouraging, if you mention that this might avoid restoring files?!

[João Cardoso]

On Tuesday, 9 June 2020 19:14:17 UTC+1, Markus Quandt wrote:

One more question - under the DISK/Wizard page, choosing the RAID1 and ext4 options (presently, I have ext3!), it tells me it would re-format, and I would loose all data. Not encouraging, if you mention that this might avoid restoring files?!

No, you said "I guess I'll have to re-partition... [and] reconstruct most of the data from other media"

Thus my suggestion to use the Disk Wizard instead of doing every steps of re-partition, creating RAID and creating the fs. The Disk Wizard only avoids doing all those steps, it will not recover or preserve disk data. Repartition and reformating *will* of course erase all disks data. 

[Markus Quandt]

Ah, sorry, I misunderstood. Thanks anyways!

[Markus Quandt]

Good news:

I finally tried this, found in yet another Linux forum:

mke2fs -S /dev/md1 && fsck /dev/md1

This DID rewrite superblocks and FS information, then again forced a check. That check cleared the excess blocks in the affected inode, and after that started yet another check. After a third or maybe even fourth check, each of which fixed inode, block counts, and whatever, I was indeed able to mount the volume again!!!

One thing this did was, however, to convert the file system to EXT2. (Don't know whether this is an effect of mke2fs itself or the subsequent 'repairs'. One of those was to delete the journal inode.)

I guess that a few files on the drive are actually damaged or lost, but that is comparably easy to fix. I am now backing up (again) all user data from the drive, and then will try a conversion to EXT4.

Or would EXT4 not be the advisable route?

Thanks again,

Markus

[João Cardoso]

On Wednesday, 10 June 2020 16:10:10 UTC+1, Markus Quandt wrote:

Good news:

I finally tried this, found in yet another Linux forum:

mke2fs -S /dev/md1 && fsck /dev/md1

Great! But that is one advise that I would not give (but could point to). According to the mke2fs manual page:

      -S     Write superblock and group descriptors only.  This is an extreme measure to
             be  taken  only  in  the  very unlikely case that all of the superblock and
             backup superblocks are corrupted,  and  a  last-ditch  recovery  method  is
             desired  by  experienced  users.   It  causes  mke2fs  to  reinitialize the
             superblock and group descriptors, while not touching the  inode  table  and
             the  block and inode bitmaps.  The e2fsck program should be run immediately
             after this option is used, and there is no guarantee that any data will  be
             salvageable.   Due  to  the wide variety of possible options to mke2fs that
             affect the on-disk layout, it is critical to specify exactly the same  for-
             mat  options, such as blocksize, fs-type, feature flags, and other tunables
             when using this option, or the filesystem will be  further  corrupted.   In
             some  cases,  such  as filesystems that have been resized, or have had fea-
             tures enabled after format time, it is impossible to overwrite all  of  the
             superblocks  correctly, and at least some filesystem corruption will occur.
             It is best to run this on a full copy of the filesystem  so  other  options
             can be tried if this doesn't work.

 

This DID rewrite superblocks and FS information, then again forced a check. That check cleared the excess blocks in the affected inode, and after that started yet another check. After a third or maybe even fourth check, each of which fixed inode, block counts, and whatever, I was indeed able to mount the volume again!!!

One thing this did was, however, to convert the file system to EXT2. (Don't know whether this is an effect of mke2fs itself or the subsequent 'repairs'. One of those was to delete the journal inode.)

I guess that a few files on the drive are actually damaged or lost, but that is comparably easy to fix. I am now backing up (again) all user data from the drive, and then will try a conversion to EXT4.

Or would EXT4 not be the advisable route?

It is.

Depending on your final results you might want to edit the post subject, or add a "how to solve a short read error on fsck" or "how to fix a mismatch between a filesystem and its enclosing device" on the post first line itself.

[Markus Quandt]

Well, I certainly was in the 'last-ditch' situation, after having tried everything I could find short of re-partitioning. :-)