mdadm and fsck

[Joey L]

I have a broken raid1 drive set - i was able to fix 1 but the other raid still has issues.

i have gone through the tutorials online to bring it back online but still unable to bring it back.

Do i have to run fsck on the individual member drives ???

my /proc/mdstat looks like:

Personalities : [raid1]

md1 : active raid1 sdc1[0] sdd1[1](S)

      976758841 blocks super 1.2 [2/1] [U_]

md0 : active raid1 sda1[2] sdb1[1]

      972654456 blocks super 1.2 [2/1] [_U]

      [==========>..........]  recovery = 50.7% (493690944/972654456) finish=416.5min speed=19162K/sec

my fstab is :

# <file system> <mount point>   <type>  <options>       <dump>  <pass>

proc            /proc           proc    defaults        0       0

# / was on /dev/md0 during installation

UUID=611258bc-4ace-4779-b44d-73ba1949f7db /               ext4    errors=remount-ro 0       1

# swap was on /dev/md1 during installation

UUID=f9dc126d-9593-475a-ab0e-b4126146d885 none            swap    sw              0       0

/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0

/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

/dev/md1        /mnt/raid  ext4 rw        0       1

my questions is :

1. how can i bring it back online.

2. what is the (S) mean ?

3. how do i run fsck or other utility on ext4 filesystem ?

4. do i run it on the raid1 device /dev/md0 or on the individual devices /dev/sda and /devsdb

thanks

mjh

[Andrew McGlashan]

Hi,

Joey L wrote:
> I have a broken raid1 drive set - i was able to fix 1 but the other raid
> still has issues.

You fixed the file system with fsck. One drive [of md0] is considered
fine, the other is becoming fine during the rebuild.

> i have gone through the tutorials online to bring it back online but
> still unable to bring it back.
>
> Do i have to run fsck on the individual member drives ???

No.

> my /proc/mdstat looks like:
> Personalities : [raid1]
> md1 : active raid1 sdc1[0] sdd1[1](S)
> 976758841 blocks super 1.2 [2/1] [U_]

Not sure, but it _may_ be that md1 will rebuild itself using the
"*S*pare" when md0 is fully rebuilt.

> md0 : active raid1 sda1[2] sdb1[1]
> 972654456 blocks super 1.2 [2/1] [_U]
> [==========>..........] recovery = 50.7% (493690944/972654456)
> finish=416.5min speed=19162K/sec

..

> my questions is :
> 1. how can i bring it back online.

If one RAID1 member is fine, then the file system should be accessible.
It will just operate in degraded mode in the meantime.

> 2. what is the (S) mean ?

This is a hot spare, not sure why it isn't rebuilding now.... whilst
md0 is rebuilding.

> 3. how do i run fsck or other utility on ext4 filesystem ?

You only run fsck to fix a file system, not to fix a RAID set. If the
file system won't mount, then you could try fsck on the md0 or md1
device as required _before_ you mount it

> 4. do i run it on the raid1 device /dev/md0 or on the individual devices
> /dev/sda and /devsdb

You would run fsck on the md0 device in your case, but that is your root
file system -- is it mounted now? If it is, then no fsck....

Your md1 seems to have been your swap partition, swap doesn't have any
file system on it to fsck. But it is now /mnt/raid ????

It might be useful for to supply output from blkid if you need more
help. And also output from the following:
mdadm --detail /dev/md0
mdadm --detail /dev/md1

That [mdadm --detail .... ] will give more details than /proc/mdstat


--
Kind Regards
AndrewM

Andrew McGlashan


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/4E74C98A...@affinityvision.com.au

[Andrew McGlashan]

Hi,

You might also find the attached script useful, I place it in
/etc/profile.d/ and it runs when I login.

When I am doing a rebuild, I do the following to monitor the progress:

watch -n .5 $CMDADM2_CMD

That will repeat the command set each half a second.

Cheers
AndrewM

[Andrew McGlashan]

Andrew McGlashan wrote:
> You might also find the attached script useful, I place it in
> /etc/profile.d/ and it runs when I login.

You might want this other file too, it defines the system binary
locations (and more) as used in the other script... ;-)

A.

[tv.d...@googlemail.com]

17/09/2011 17:18, Joey L wrote:
> I have a broken raid1 drive set - i was able to fix 1 but the other raid
> still has issues.
> i have gone through the tutorials online to bring it back online but
> still unable to bring it back.
>
> Do i have to run fsck on the individual member drives ???

Short answer: NO ! You run fsck on raid volumes on which the (umounted)
file-system resides, that is "fsck /dev/md1".

>
> my /proc/mdstat looks like:
> Personalities : [raid1]
> md1 : active raid1 sdc1[0] sdd1[1](S)
> 976758841 blocks super 1.2 [2/1] [U_]
>
> md0 : active raid1 sda1[2] sdb1[1]
> 972654456 blocks super 1.2 [2/1] [_U]
> [==========>..........] recovery = 50.7% (493690944/972654456)
> finish=416.5min speed=19162K/sec
>
> my fstab is :
> # <file system> <mount point> <type> <options> <dump> <pass>
> proc /proc proc defaults 0 0
> # / was on /dev/md0 during installation
> UUID=611258bc-4ace-4779-b44d-73ba1949f7db / ext4
> errors=remount-ro 0 1
> # swap was on /dev/md1 during installation
> UUID=f9dc126d-9593-475a-ab0e-b4126146d885 none swap sw
> 0 0
> /dev/scd0 /media/cdrom0 udf,iso9660 user,noauto 0 0
> /dev/fd0 /media/floppy0 auto rw,user,noauto 0 0
> /dev/md1 /mnt/raid ext4 rw 0 1
>
>
> my questions is :
> 1. how can i bring it back online.

Well, it depends, what did you try so far, did you "--run" it ? Is md1
not accessible as it is ? It should be even in degraded mode.
What does "mdadm --detail /dev/md1" and "mdadm --examine /dev/sdd1" say
? What's in your mdadm.conf ?

md0 is in recovery mode, leave it alone for a few hours (416/60) and see
how it goes.

> 2. what is the (S) mean ?

sdd1 is a spare. If you want to manually add it as a member of the
array, you can fail and remove it, zero superblocks, then add it back to
the raid. You should see the raid rebuilding right after.
As to why it is not automatically included in the array as a replacement
for the failed member when the array is run, I don't know.

> 3. how do i run fsck or other utility on ext4 filesystem ?
> 4. do i run it on the raid1 device /dev/md0 or on the individual devices
> /dev/sda and /devsdb

Run fsck directly on the raid volume, file-system must not be mounted.
Unless you are running an old version of e2fsprogs ext4 support should
not be a problem.

> thanks
> mjh

>
>
>


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/4E75057D...@googlemail.com

[Joey L]

Thanks for the scripts and the reply !! now i think i understand raid on linux a lot better - but do have additional questions :)

My /dev/md0 the root filesystem came back - Thank God !! and you :)

Info you requested :

root@rider:~# mdadm --detail /dev/md0

/dev/md0:

        Version : 1.2

  Creation Time : Mon Jun 27 08:51:23 2011

     Raid Level : raid1

     Array Size : 972654456 (927.60 GiB 996.00 GB)

  Used Dev Size : 972654456 (927.60 GiB 996.00 GB)

   Raid Devices : 2

  Total Devices : 2

    Persistence : Superblock is persistent

    Update Time : Sat Sep 17 22:53:59 2011

          State : clean

 Active Devices : 2

Working Devices : 2

 Failed Devices : 0

  Spare Devices : 0

           Name : debian:0

           UUID : f696d568:1f4226e3:42f0a70a:68e284a7

         Events : 295206

    Number   Major   Minor   RaidDevice State

       2       8        1        0      active sync   /dev/sda1

       1       8       17        1      active sync   /dev/sdb1

-- this looks fine !

But my issue is /dev/md1 -- i rebooted the machine - now it does not come up at all and do not know which drive i think is okay - I see both in fdisk -l  and they are okay.

I get this:

root@rider:~# mdadm --detail /dev/md1

mdadm: cannot open /dev/md1: No such file or directory

root@rider:~# fdisk -l /dev/sdc

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes

255 heads, 63 sectors/track, 121601 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000d27a0

   Device Boot      Start         End      Blocks   Id  System

/dev/sdc1               1      121601   976760001   fd  Linux raid autodetect

root@rider:~# fdisk -l /dev/sdc1

Disk /dev/sdc1: 1000.2 GB, 1000202241024 bytes

255 heads, 63 sectors/track, 121600 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

     Device Boot      Start         End      Blocks   Id  System

root@rider:~# fdisk -l /dev/sdd

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes

255 heads, 63 sectors/track, 121601 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000d4cc8

   Device Boot      Start         End      Blocks   Id  System

/dev/sdd1               1      121601   976760001   fd  Linux raid autodetect

root@rider:~# fdisk -l /dev/sdd1

Disk /dev/sdd1: 1000.2 GB, 1000202241024 bytes

255 heads, 63 sectors/track, 121600 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

Disk /dev/sdd1 doesn't contain a valid partition table

How do I determine which drive is the key drive or the one to add first to /dev/md1

thanks

mjh

[Joey L]

Thanks for the info and quick reply.

My /dev/md0 has rebuilt itself and is okay.

My /dev/md1 is gone after a reboot - and is no longer showing -

I get nothing when i run:

root@rider:~# mdadm --detail /dev/md1

mdadm: cannot open /dev/md1: No such file or directory

Questions for you:

1. How do I figure out which drive that made up /dev/md1  DRIVES: /dev/sdc or /dev/sdc1 and /dev/sdd or /dev/sdd1  is okay to be the master where I would rebuild the second ??

2. How do I "zero superblocks" a drive ?? I know how to fail, remove and add to a raid device.

3. Should I use /dev/sdd1 as the master ?? the system marked it as a spare before.

thanks

mjh

[Andrew McGlashan]

Hi Joey,

Joey L wrote:
> Thanks for the scripts and the reply !! now i think i understand raid on
> linux a lot better - but do have additional questions :)

Great.

Yes.

> But my issue is /dev/md1 -- i rebooted the machine - now it does not
> come up at all and do not know which drive i think is okay - I see both
> in fdisk -l and they are okay.
>
> I get this:
>
> root@rider:~# mdadm --detail /dev/md1
> mdadm: cannot open /dev/md1: No such file or directory

Then the RAID isn't detected and assembled, not even degraded.

The fdisk tool should be run for each device, not the partitions, so in
this case just /dev/sdc and /dev/sdd -- not /dev/sdc1 or /dev/sdd1

> root@rider:~# fdisk -l /dev/sdc
>
> Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
> 255 heads, 63 sectors/track, 121601 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x000d27a0
>
> Device Boot Start End Blocks Id System
> /dev/sdc1 1 121601 976760001 fd Linux raid
> autodetect

> root@rider:~# fdisk -l /dev/sdd
>
> Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
> 255 heads, 63 sectors/track, 121601 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x000d4cc8
>
> Device Boot Start End Blocks Id System
> /dev/sdd1 1 121601 976760001 fd Linux raid
> autodetect

> How do I determine which drive is the key drive or the one to add first
> to /dev/md1

I would try the following to see if the file system can be mounted
successfully. If one errors and the other works fine and you can see
your files okay, then that should help -- perhaps then, make a backup of
the "good" drive, just in case.

mkdir /mnt/sdc1 /mnt/sdd1
mount -o ro -type ext4 /dev/sdc1 /mnt/sdc1
mount -o ro -type ext4 /dev/sdd1 /mnt/sdd1

Hope that gets you enough to work out what is going on.

--
Kind Regards
AndrewM

Andrew McGlashan

Broadband Solutions now including VoIP

--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/4E75765...@affinityvision.com.au

[tv.d...@googlemail.com]

18/09/2011 05:04, Joey L wrote:
> Thanks for the info and quick reply.
>
> My /dev/md0 has rebuilt itself and is okay.
> My /dev/md1 is gone after a reboot - and is no longer showing -
> I get nothing when i run:
>
> root@rider:~# mdadm --detail /dev/md1
> mdadm: cannot open /dev/md1: No such file or directory

You could try to examine each members of the array, see if meta-data are
found or if everything vanished. You can use
"mdadm --examine /dev/sdd1"

/dev/md1 not neing found doesn't mean everything is lost, just means it
hasn't been detected or assembled at boot time, so udev didn't create
the device /dev/md? for it. But maybe the array can be brought back (at
least in degraded mode) manually. You are using 1.2 metadata on your
arrays, is you mdadm.conf info up to date ?

>
>
> Questions for you:
> 1. How do I figure out which drive that made up /dev/md1 DRIVES:
> /dev/sdc or /dev/sdc1 and /dev/sdd or /dev/sdd1 is okay to be the
> master where I would rebuild the second ??

There is no "master" in a raid1 array, otherwise data wouldn't be truly
redundant. A drive is becoming a temporary "gold master" when it is
alone in the array (the other one is failed). So regarding your "md1"
array sdc1 is the last known working member, this is what you should use
to try to start and rebuild the array.

When you "--examine" a member of an array, you'll be shown the array
UUID it belongs to, and the device UUID (if meta-data are not gone).
Personally after creating a raid I always take good note of those
informations (outside of the array) in case I need to do "manual"
intervention. If you need to boot from a live-cd to recover a damaged
system there is no warranty that the array names or array members names
will be consistent with what it was on the working system. But the
UUID's will.

And don't mix up /dev/sd? (the physical disk device) with /dev/sd?? (the
partition which is included in the array). There could be only one
partition on the disk, but not necessarily, or the raid could be a
partitioned on. You have no business using the physical device level to
troubleshoot a raid array, only if you want to image a damaged device
for later forensic work you'll need to work at disk level.
Don't take my word for it, try to:
mdadm --examine /dev/sdc
You will see no mdadm raid meta-data on it.

>
> 2. How do I "zero superblocks" a drive ?? I know how to fail, remove and
> add to a raid device.

mdadm --zero-superblock /dev/sdc1

You have to fail and remove the drive first. You must do that when you
break an array and want to later reuse the members in a new array. You
should carefully read man mdadm. Unfortunately the raid wiki seems to be
down at the moment due to kernel.org problems, hope it's back online soon.
https://www.raid.wiki.kernel.org/index.php/Linux_Raid

>
> 3. Should I use /dev/sdd1 as the master ?? the system marked it as a
> spare before.

/dev/sdc1 is the last known working member of the array, so I would
start with it. But you can try to unplug one or the other device to
prevent any further damage to it, and try to start the array in degraded
mode from either sdd1 or sdc1, you'll see what works, and what data you
can find on it.
Be aware that if you restart the array and it is rebuilt from one
member, the data on the other will be effectively wiped out. The
"--assume-clean" flag could prevent that, but then you'll have no
warranty that the data are consistent on the array. Work on a degraded
array with only one member for now is my advice.
You can try to force the assembling of a degraded or "dirty" array at
boot time with the boot option "md-mod.start_dirty_degraded=1" (see
"modinfo md_mod"), but don't do that with the two members plugged in, it
could further damaged it and compromise your chances to recover from one
of the array member. It is by far a better practice for recovery to boot
without assembling the array, then manually assemble and run it as needed.

> thanks
> mjh
>
>
>
>

Hope it helps.

--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/4E75C417...@googlemail.com

[Bob Proulx]

tv.d...@googlemail.com wrote:
> > My /dev/md1 is gone after a reboot - and is no longer showing -
> > I get nothing when i run:

Was /dev/md1 created by you somewhere along the way after having
installed the system previously? Did you remember to update the
/etc/mdadm/mdadm.conf file with the new UUID numbers for it?

mdadm --detail --scan

Then update /etc/mdadm/mdadm.conf appropriately.

Is this part of an LVM group that contains the root partition? I
think not or you would not have been able to boot. But if it were
then you would also need to rebuild the initrd with the updated
mdadm.conf contents.

dpkg-reconfigure linux-image-$(uname -r)

In Lenny partitions marked autoraid were automatically assembled. In
Squeeze this has been changed and automatic assembly no longer happens.

Bob

[Joey L]

Hi Andrew - 

Thanks for the reply - 

I tried to mount the individual drives that make up my /dev/md1 and this is what i get :

root@rider:~# mount -o ro -t ext4 /dev/sdc1 /mnt/sdc1

mount: wrong fs type, bad option, bad superblock on /dev/sdc1,

       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try

       dmesg | tail  or so

root@rider:~#

root@rider:~#

root@rider:~# mount -o ro  /dev/sdc1 /mnt/sdc1

mount: unknown filesystem type 'linux_raid_member'

root@rider:~# mount -o ro -t ext4 /dev/sdc1 /mnt/sdd1

mount: wrong fs type, bad option, bad superblock on /dev/sdc1,

       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try

       dmesg | tail  or so

root@rider:~# mount -o ro  /dev/sdc1 /mnt/sdd1

mount: unknown filesystem type 'linux_raid_member'

**I can not seem to mount /dev/sdc1 or /dev/sdd1 ---  I get errors of unknown file type.

root@rider:~# mdadm --examine /dev/sdd1

/dev/sdd1:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : 6b483902:96e628c4:6a0187a1:1b75d1e2

           Name : rider:1  (local to host rider)

  Creation Time : Sun Jul 10 01:14:55 2011

     Raid Level : raid1

   Raid Devices : 2

 Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB)

     Array Size : 1953517682 (931.51 GiB 1000.20 GB)

  Used Dev Size : 1953517682 (931.51 GiB 1000.20 GB)

    Data Offset : 2048 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : 64e63c40:209f1801:644a6a5c:f6aed69c

    Update Time : Sat Sep 17 19:37:37 2011

       Checksum : 19807534 - correct

         Events : 19928

   Device Role : spare

   Array State : A. ('A' == active, '.' == missing)

   Array State : A. ('A' == active, '.' == missing)

root@rider:~# mdadm --examine /dev/sdc1

/dev/sdc1:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : 6b483902:96e628c4:6a0187a1:1b75d1e2

           Name : rider:1  (local to host rider)

  Creation Time : Sun Jul 10 01:14:55 2011

     Raid Level : raid1

   Raid Devices : 2

 Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB)

     Array Size : 1953517682 (931.51 GiB 1000.20 GB)

  Used Dev Size : 1953517682 (931.51 GiB 1000.20 GB)

    Data Offset : 2048 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : 78b680cf:9131f4dd:00a622b1:6a5b6b67

    Update Time : Sat Sep 17 09:27:31 2011

       Checksum : a4ee50c4 - correct

         Events : 19922

   Device Role : Active device 0

   Array State : AA ('A' == active, '.' == missing)

It looks like the /dev/sdc1 is the active one...is that the first one i should add the /dev/md1 ??

Should I zero out the other drive and add it back in ?? 

thanks

[tv.d...@googlemail.com]

19/09/2011 15:59, Joey L wrote:
> Hi Andrew -
> Thanks for the reply -
> I tried to mount the individual drives that make up my /dev/md1 and this
> is what i get :
>
> root@rider:~# mount -o ro -t ext4 /dev/sdc1 /mnt/sdc1
> mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> root@rider:~#
> root@rider:~#
> root@rider:~# mount -o ro /dev/sdc1 /mnt/sdc1
> mount: unknown filesystem type 'linux_raid_member'
> root@rider:~# mount -o ro -t ext4 /dev/sdc1 /mnt/sdd1
> mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> root@rider:~# mount -o ro /dev/sdc1 /mnt/sdd1
> mount: unknown filesystem type 'linux_raid_member'

Why would you want to mount it this way ? Since the meta-data are there
try to start the array with mdadm:

mdadm -A --run --uuid=6b483902:96e628c4:6a0187a1:1b75d1e2 /dev/sdc1

You can use "/dev/md1" instead of "--uuid=", "--run" means the array as
to be started even in degraded mode (one member only).
When done try to mount /dev/md1, and see what's on it.

>
> **I can not seem to mount /dev/sdc1 or /dev/sdd1 --- I get errors of
> unknown file type.

Did you change the filesystem type to "fd" a.k.a. "linux raid
auto-detect" at creation time ? This is not necessary and not
recommended with 1.2 meta-data, it use to be mandatory when booting from
a 0.90 type meta-data array.

[trim]

>
> It looks like the /dev/sdc1 is the active one...is that the first one i
> should add the /dev/md1 ??
> Should I zero out the other drive and add it back in ??
> thanks

Yes, if /dev/md1 starts with /dev/sdc1, mount it and check that your
data are there and intact. If so, "--zero-superblock" on /dev/sdd1 and
add it back to the array, then watch the resync.


Have fun.

--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/4E7754D4...@googlemail.com