xfs_repair: Sorry, could not find valid secondary superblock

[Warren Post]

A friend's mdv2009.1 box locked up during a big printing job and he had to
force a shutdown with Alt-SysRq-R S E I U B. Upon reboot it came back in
single user mode and would not mount the hard disk. I booted it from a
live CD and performed the following on hda1, what I presume to be the root
partition:

# mount /dev/hda1
mount: wrong fs type, bad option, bad superblock on /dev/hda1,
missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

# dmesg | tail
XFS: bad version
XFS: SB validate failed
XFS: bad version
XFS: SB validate failed
XFS: bad version
XFS: SB validate failed
XFS: bad version
XFS: SB validate failed
XFS: bad version
XFS: SB validate failed

# badblocks -sn /dev/hda1
Checking for bad blocks (non-destructive read-write test)
Testing with random pattern: done

xfs_check /dev/hda1
bad sb version # 0xb4b4 in ag 0
bad sb version # 0xb4a4 in ag 1
(really big snip)
sb_fdblocks 653, counted 1677
WARNING: this may be a newer XFS filesystem.

# xfs_repair /dev/hda1
bad primary superblock - bad version number !!!
attempting to find secondary superblock...
(big snip)
Sorry, could not find valid secondary superblock
Exiting now.

Searching the web I find numerous requests for help with the same issue
but no one reports success. So I'm out of ideas. What else might I try to
bring this box back to life?

--
Warren Post · New Media Copán
http://my.opera.com/wpost/

[unruh]

On 2011-11-24, Warren Post <inv...@invalid.invalid> wrote:
> A friend's mdv2009.1 box locked up during a big printing job and he had to
> force a shutdown with Alt-SysRq-R S E I U B. Upon reboot it came back in
> single user mode and would not mount the hard disk. I booted it from a
> live CD and performed the following on hda1, what I presume to be the root
> partition:
>
> # mount /dev/hda1
> mount: wrong fs type, bad option, bad superblock on /dev/hda1,
> missing codepage or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so

They are usually sda1, not hda1

Buy a new disk, reinstall a newer version, and then recover from the
backup he made a few days ago.

[freemont]

On Wed, 23 Nov 2011 20:18:56 -0600, Warren Post writ:

As was suggested to me years ago under similar circumstances, maybe you
can compare the output of df and fdisk -l with the entries in /etc/fstab.
Don't assume that /dev/hda1 contains /. What is fstab trying to mount,
and what do the partitions actually contain?

This information may be of some help... I don't pretend to be an expert,
but I've run into similar problems before. :-)

--
⁂ "Because all you of Earth are idiots!"
¯`·.¸¸.·´¯`·-> ※freemont※ <-·´¯`·.¸¸.·´¯

[Jim Beard]

"hda1, I presume" Presumption like assumption can be a fatal
mistake.

If you have lshw on the system, use it to see what disks are out
there. If on the system, blkid can be used to check individual
partitions. Look at /etc/fstab to see what perhaps _should_ be
out there. Take a look at /etc/boot/grub/menu.lst (if using
grub) to see what grub thinks the root partition is. For grub's
purposes, (hd0,0) is the first partition of the first hard drive
(/dev/[hs]da1), irrespective of whether an hda or sda

If the root partition is sda1 or anything other than hda1, go to
work on that. Ignore all below until you find out what the
problem is on the real root partition.

If root partition is hda1, either the disk mbr is corrupt
(pointing to the wrong place) or there is no valid superblock on
hda1. Replacing the disk and restoring from backup should cure
either problem. How old is the disk? It may be at end of
service life quite independent of the current problem, and fixing
the current problem may accomplish little. I note that most
filesystems today have multiple superblocks, which provides
redundancy to cope with corruption, and if none of the redundant
superblocks is good the disk is probably too far gone to be worth
bringing back to life.

Repair of a corrupt mbr might be possible, but that is something
I have never done. If the problem is on hda1, a reformat of the
partition might do the trick. I would move the start point of
the partition (delete the partition, recreate at a different
start address) and hope that would put a good track at the start
for the superblock to go on.

Cheers!

jim b.

--
UNIX is not user unfriendly; it merely
expects users to be computer-friendly.

[Warren]

On Nov 24, 6:26 am, freemont <freemontspamme...@freemontsoffice.com>
wrote:

> As was suggested to me years ago under similar circumstances, maybe you
> can compare the output of df and fdisk -l with the entries in /etc/fstab.
> Don't assume that /dev/hda1 contains /. What is fstab trying to mount,
> and what do the partitions actually contain?

First off, my apologies for posting via Google Groups. I'm on the box
in question running a live distro with no Usenet client on it.

df will be of limited use, given that all but one of the partitions of
the hard disk are unmountable:

bt ~ # mount -a
mount: devpts already mounted or /dev/pts busy

mount: wrong fs type, bad option, bad superblock on /dev/hda1,
missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

mount: wrong fs type, bad option, bad superblock on /dev/hda7,

missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

mount: wrong fs type, bad option, bad superblock on /dev/hda8,

missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

mount: wrong fs type, bad option, bad superblock on /dev/hda9,

missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

Here's what can be mounted (see last line):

bt ~ # mount
aufs on / type aufs (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda5 on /mnt/hda5 type ext3 (rw,noatime)

With that caveat, here's what df and fdisk say:

bt ~ # df -h
Filesystem Size Used Avail Use% Mounted on
aufs 261M 9.0M 252M 4% /
/dev/hda5 8.0G 147M 7.4G 2% /mnt/hda5
/dev/sda1 3.7G 2.8G 732M 80% /mnt/sda1

bt ~ # fdisk -l

Disk /dev/hda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 1051 8442126 83 Linux
/dev/hda2 1052 19456 147838162+ 5 Extended
/dev/hda5 1052 2101 8434093+ 83 Linux
/dev/hda6 2102 2241 1124518+ 82 Linux swap
/dev/hda7 2242 2636 3172806 83 Linux
/dev/hda8 2637 3298 5317483+ 83 Linux
/dev/hda9 3299 19456 129789103+ 83 Linux

Disk /dev/sda: 4007 MB, 4007657472 bytes
18 heads, 18 sectors/track, 24158 cylinders
Units = cylinders of 324 * 512 = 165888 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 25 24159 3909696 83 Linux

The previous day's backup of /etc/fstab says:

# Entry for /dev/sda1 :
UUID=f9f5a379-224d-48d6-a523-d3acd3f97a39 / xfs relatime 1 1
# Entry for /dev/sda9 :
UUID=844ee38a-2705-4a2d-8820-cdcc1f6f50e9 /home xfs relatime 1 2
/dev/cdrom /media/cdrom auto
umask=0,users,iocharset=utf8,noauto,ro,exec 0 0
/dev/fd0 /media/floppy auto
umask=0,users,iocharset=utf8,noauto,exec,flush 0 0
# Entry for /dev/sda5 :
UUID=c910b8c5-6e34-423d-a093-f8e1a1c0c825 /next_install ext3 relatime
1 2
# Entry for /dev/sda7 :
UUID=47985ce3-bec0-4d29-a64c-6a3b6bdffcaf /opt xfs relatime 1 2
none /proc proc defaults 0 0
none /tmp tmpfs defaults 0 0
# Entry for /dev/sda8 :
UUID=40518fd6-f3b2-45d6-8396-262a1ec45f86 /var/www/html xfs relatime 1
2
# Entry for /dev/sda6 :
UUID=1aeb634e-e9cd-45ce-b418-18156d286d1f swap swap defaults 0 0

The box's owner doesn't know what partition is for what, but between
fdisk and what he remembers of his partition layout, my best guess is:

sda1/hda1: / (my guess)
sda2/hda2: extended partition (per fdisk)
sda5/hda5: an empty partition to become the new root partition when
upgrading (my guess)
sda6/hda6: swap (per fdisk)
sda7/hda7: /opt (my guess)
sda8/hda8: /var/www/html (my guess)
sda9/hda9: /home (my guess)

...so my focus is on sda1/hda1, being the closest thing I can confirm
to being the root partition.

[Warren]

On Nov 24, 2:02 pm, Jim Beard <jdbe...@patriot.net> wrote:

> "hda1, I presume"   Presumption like assumption can be a fatal
> mistake.

Agreed, but the box's owner doesn't know what partition is for what,
so I've had to make my best guess based on fdisk and what he remembers
of his partition layout. My reply to freemont has the details of why
hda1/sda1 is my buest guess. In the absence of better data I'm
focusing on that partition.

> If you have lshw on the system, use it to see what disks are out
> there.  If on the system, blkid can be used to check individual
> partitions.  Look at /etc/fstab to see what perhaps _should_ be
> out there.  Take a look at /etc/boot/grub/menu.lst (if using
> grub) to see what grub thinks the root partition is.

The live distro I'm using does not have lshw, but a physical
inspection of the box's innards, a peek at the BIOS, qtparted, and the
below tools all show consistent datac: a single HD with the partitions
indicated below.

blkid says:

bt ~ # blkid
/dev/sda1: LABEL="backup" UUID="85315c54-5097-476b-be39-00a8e91859a6"
TYPE="ext2"
/dev/hda1: UUID="f9f5a379-224d-48d6-a523-d3acd3f97a39" TYPE="xfs"
/dev/hda5: UUID="c910b8c5-6e34-423d-a093-f8e1a1c0c825" SEC_TYPE="ext2"
TYPE="ext3"
/dev/hda6: UUID="1aeb634e-e9cd-45ce-b418-18156d286d1f" TYPE="swap"
/dev/hda7: UUID="47985ce3-bec0-4d29-a64c-6a3b6bdffcaf" TYPE="xfs"
/dev/hda8: UUID="40518fd6-f3b2-45d6-8396-262a1ec45f86" TYPE="xfs"
/dev/hda9: UUID="844ee38a-2705-4a2d-8820-cdcc1f6f50e9" TYPE="xfs"

/etc/boot/grub/menu.lst is not in the backups. I don't know what the
bootloader is, but would be surprised if it's anything other than the
default bootloader for mdv2009.1.

> If root partition is hda1, either the disk mbr is corrupt
> (pointing to the wrong place) or there is no valid superblock on
> hda1.  Replacing the disk and restoring from backup should cure
> either problem.  How old is the disk?  It may be at end of
> service life quite independent of the current problem, and fixing
> the current problem may accomplish little.

Good points. The disk is old enough that the owner doesn't remember
how old it is, which I suppose is an answer in itself.

The owner is extremely insistent that I first attempt to revive the
system without reinstalling, and failing that, that I format the
existing disk rather than buying a new one. So despite your good
advice and my own gut, I'll proceed thus, unless I can show him some
diagnostics that suggest a physical hard disk problem. badblocks
didn't output anything useful for this purpose. Is there some other
tool that can show me hard disk health?

[Jim Beard]

The file to check is /boot/grub/menu.lst, not
/etc/boot/grub/menu.ls. (hd0,0) will be the first hard drive,
first partition, regardless if hda or sda. Given the question of
why hda5 (below) is ext2/ext3, this may be critical.

On 11/24/2011 01:05 PM, Warren wrote:

> inspection of the box's innards, a peek at the BIOS, qtparted, and the
> below tools all show consistent datac: a single HD with the partitions
> indicated below.
>
> blkid says:
>
> bt ~ # blkid
> /dev/sda1: LABEL="backup" UUID="85315c54-5097-476b-be39-00a8e91859a6"
> TYPE="ext2"

Is the above a DVD-RW in a DVD drive? If so, from this and other
posts, it looks like /dev/sda is a 4GB DVD that has been
partitioned for backup by partition. If this is not what is
supposed to be there, the problem may be with the hard drive
controller rather than the drive.

> /dev/hda1: UUID="f9f5a379-224d-48d6-a523-d3acd3f97a39" TYPE="xfs"
> /dev/hda5: UUID="c910b8c5-6e34-423d-a093-f8e1a1c0c825" SEC_TYPE="ext2"
> TYPE="ext3"
> /dev/hda6: UUID="1aeb634e-e9cd-45ce-b418-18156d286d1f" TYPE="swap"
> /dev/hda7: UUID="47985ce3-bec0-4d29-a64c-6a3b6bdffcaf" TYPE="xfs"
> /dev/hda8: UUID="40518fd6-f3b2-45d6-8396-262a1ec45f86" TYPE="xfs"
> /dev/hda9: UUID="844ee38a-2705-4a2d-8820-cdcc1f6f50e9" TYPE="xfs"

Swap is normally in hda5 or sda5. The ext2/ext3 file system in
hda5 makes me wonder if that was set up specfically for the
partition to boot from, rather than from xfs. This does look
like a normal internal hard drive, though, with xfs the standard
file system type.

>
> /etc/boot/grub/menu.lst is not in the backups. I don't know what the
> bootloader is, but would be surprised if it's anything other than the
> default bootloader for mdv2009.1.

As mentioned, /boot/grub/menu.lst
The default for Mandriva and before that Mandrake was grub
(rather than LILO) so I would look again for the file. We need
to know exactly where booting starts.

> Good points. The disk is old enough that the owner doesn't remember
> how old it is, which I suppose is an answer in itself.
>
> The owner is extremely insistent that I first attempt to revive the
> system without reinstalling, and failing that, that I format the
> existing disk rather than buying a new one. So despite your good
> advice and my own gut, I'll proceed thus, unless I can show him some
> diagnostics that suggest a physical hard disk problem. badblocks
> didn't output anything useful for this purpose. Is there some other
> tool that can show me hard disk health?

See if the smartmontools package has been installed. If
available, try smartctl and see what it tells you. You might try
fsck or fsck.xfs and see if it tells you anything different from
xfs_check.

[unruh]

No, the default was lilo for a long time (I started whith it
whenMandrake started). That is why I am still using lilo.
Not enough time to learn something new.

[Jim Beard]

On 11/24/2011 05:04 PM, unruh wrote:

> No, the default was lilo for a long time (I started whith it
> whenMandrake started). That is why I am still using lilo.
> Not enough time to learn something new.

I started with Mandrake 8, and started with grub, which I assume
was the default even then. Before that, yes LILO was the default
at some point.

[David W. Hodgins]

On Wed, 23 Nov 2011 21:18:56 -0500, Warren Post <inv...@invalid.invalid> wrote:

> A friend's mdv2009.1 box locked up during a big printing job and he had to
> force a shutdown with Alt-SysRq-R S E I U B. Upon reboot it came back in
> single user mode and would not mount the hard disk. I booted it from a
> live CD and performed the following on hda1, what I presume to be the root
> partition:
>
> # mount /dev/hda1
> mount: wrong fs type, bad option, bad superblock on /dev/hda1,
> missing codepage or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> # dmesg | tail
> XFS: bad version
> XFS: SB validate failed

I think the problem is that the live cd you're booting from doesn't
support the version of xfs used in the 2009.1 system, or it doesn't
support the 256 bit inodes introduced in that version.

Try a newer live cd,

Regards, Dave Hodgins

--
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

[Warren Post]

On Thu, 24 Nov 2011 16:27:13 -0600, Jim Beard <jdb...@patriot.net> wrote:

> I started with Mandrake 8, and started with grub, which I assume was the
> default even then.

I started with mdk8.1, and lilo was default, although grub was available.

[Aragorn]

On Thursday 24 November 2011 23:49, Warren Post conveyed the following
to alt.os.linux.mandriva...

> On Thu, 24 Nov 2011 16:27:13 -0600, Jim Beard <jdb...@patriot.net>
> wrote:
>
>> I started with Mandrake 8, and started with grub, which I assume was
>> the default even then.
>
> I started with mdk8.1, and lilo was default, although grub was
> available.

To the best of my knowledge, LILO was still the default in Mandrake
10.0, 10.1 and Mandriva 2005 Limited Edition, albeit that I've never
used the latter two. GRUB has always been offered as an alternative
choice ever since 8.1 or so, or possibly sooner.

Both LILO and GRUB (legacy) are cool. GRUB2, in my opinion, is an
abomination. It's one of those "let's make it so complicated that the
user won't be trying to tweak it anymore, and one size must fit all"
bloatware things.

I've chosen to avoid GRUB2 like the plague for as long as possible. ;-)

--
= Aragorn =
(registered GNU/Linux user #223157)

[Jim Beard]

On 11/24/2011 05:38 PM, David W. Hodgins wrote:
> On Wed, 23 Nov 2011 21:18:56 -0500, Warren Post
> <inv...@invalid.invalid> wrote:
>
>> A friend's mdv2009.1 box locked up during a big printing job
>> and he had to
>> force a shutdown with Alt-SysRq-R S E I U B. Upon reboot it
>> came back in
>> single user mode and would not mount the hard disk. I booted it
>> from a
>> live CD and performed the following on hda1, what I presume to
>> be the root
>> partition:
>>
>> # mount /dev/hda1
>> mount: wrong fs type, bad option, bad superblock on /dev/hda1,
>> missing codepage or other error
>> In some cases useful info is found in syslog - try
>> dmesg | tail or so
>>
>> # dmesg | tail
>> XFS: bad version
>> XFS: SB validate failed
>
> I think the problem is that the live cd you're booting from doesn't
> support the version of xfs used in the 2009.1 system, or it doesn't
> support the 256 bit inodes introduced in that version.
>
> Try a newer live cd,

Bingo!

I had forgotten about the change in inodes, I think because my
mind did not wish to retain the memories of the awful problems
that resulted.

[David W. Hodgins]

On Thu, 24 Nov 2011 20:10:08 -0500, Jim Beard <jdb...@patriot.net> wrote:

> On 11/24/2011 05:38 PM, David W. Hodgins wrote:
>> I think the problem is that the live cd you're booting from doesn't
>> support the version of xfs used in the 2009.1 system, or it doesn't
>> support the 256 bit inodes introduced in that version.
>> Try a newer live cd,

> Bingo!

Does that mean the problem has been sorted out?

> I had forgotten about the change in inodes, I think because my
> mind did not wish to retain the memories of the awful problems
> that resulted.

That's precisely why I do remember that it started with 2009.1. :-)

[Warren Post]

On Thu, 24 Nov 2011 16:38:21 -0600, David W. Hodgins
<dwho...@nomail.afraid.org> wrote:

> On Wed, 23 Nov 2011 21:18:56 -0500, Warren Post
> <inv...@invalid.invalid> wrote:
>
>> XFS: bad version
>> XFS: SB validate failed
>
> I think the problem is that the live cd you're booting from doesn't
> support the version of xfs used in the 2009.1 system, or it doesn't
> support the 256 bit inodes introduced in that version.

I think you've got it! I was using BackTrack 3, which I now see was
released in 2008. I'm downloading BackTrack 5 R1 (released 3 months ago)
and will report back.

That might explain an unusual occurrence today: the box booted off the
hard disk and stayed up long enough for me to see that the root partition
was about full and to delete the offending logs. The box seems to be
running fine now. Naturally, I installed hddtemp and smartmontools (both
of which suggest that the disk is physically fine), and will use the live
distro to run xfs_check on all the partitions, but things seem all right.

My guess is that there was something funky about the big print jobs that
filled the logs, and /var/log on this box is on the root partition. Then
my bad luck was to use an elderly live distro to do diagnostics, and the
xfs version issue misled me into thinking the filesystem was hosed. An up
to date version of xfs_check should confirm my guess. I'll report back,
probably after the weekend.

[Warren Post]

On Thu, 24 Nov 2011 14:41:46 -0600, Jim Beard <jdb...@patriot.net> wrote:

> The file to check is /boot/grub/menu.lst, not /etc/boot/grub/menu.ls.

The box now boots -- see my other post to that effect -- so this becomes
moot.

>> bt ~ # blkid
>> /dev/sda1: LABEL="backup" UUID="85315c54-5097-476b-be39-00a8e91859a6"
>> TYPE="ext2"
>
> Is the above a DVD-RW in a DVD drive? If so, from this and other posts,
> it looks like /dev/sda is a 4GB DVD that has been partitioned for backup
> by partition.

A 4GB USB memory stick used for backup. I had inserted it to take a look
at the box's /etc/fstab. Good call.

> Swap is normally in hda5 or sda5. The ext2/ext3 file system in hda5
> makes me wonder if that was set up specfically for the partition to boot
> from, rather than from xfs.

Given that the box's owner was very hazy on his partitioning scheme, I
suspect he meant to have made that partition XFS like all the others and
did EXT by mistake.

> See if the smartmontools package has been installed. If available, try
> smartctl and see what it tells you.

Good advice. As mentioned in another post, they find nothing amiss, and
make me suspect that the filesystem "problem" was a false alarm caused by
obsolete versions of xfs_check and xfs_repair.

> You might try fsck or fsck.xfs and see if it tells you anything
> different from xfs_check.

Heh, after reading man fsck.xfs, I'm sure it will give a clean bill of
health, too. ;-)

Seriously, I'll try an up to date version of xfs_check and report back,
probably after the weekend.

[Warren Post]

On Thu, 24 Nov 2011 16:38:21 -0600, David W. Hodgins
<dwho...@nomail.afraid.org> wrote:

> I think the problem is that the live cd you're booting from doesn't
> support the version of xfs used in the 2009.1 system, or it doesn't
> support the 256 bit inodes introduced in that version.
> Try a newer live cd,

You were right. With a newer live distro, I fixed the problem in just a
few minutes. Thanks!