Adding a new USB disk to an existing RAID5 array (2 disks)

[Craig Lusher]

Hi all,

I want a 4TB RAID 5 array.

I have two internal 2TB disks and one external USB 2TB disk connected as RAID5, but with only 1.8TB of usable space and a spare drive. Please could somebody suggest how to expand to provide 4TB with parity? Thank you in advance!

Initially I built and configured my 323 with 2x 2TB disks in RAID5 (1.8TB usable). I copied over my data from a 2TB USB drive to the array and plugged that USB drive into the 323.

I then erased the USB disk using the partitioner tool and clicked 'copy to' from Disk A (internal) to Disk C (USB) to get the same sized partition. Then I went to the RAID tool and selected 'Disk C' and clicked 'add' to add it to the array. I successfully added a USB drive to the Array, but it is marked as a 'spare'. Please can somebody explain how to change the config so that I have ~4TB of space and parity?

Huge thanks in advance!

Craig

[João Cardoso]

On Thursday, April 2, 2015 at 3:58:58 PM UTC+1, Craig Lusher wrote:

Hi all,

I want a 4TB RAID 5 array.

I have two internal 2TB disks and one external USB 2TB disk connected as RAID5, but with only 1.8TB of usable space and a spare drive. Please could somebody suggest how to expand to provide 4TB with parity? Thank you in advance!

Initially I built and configured my 323 with 2x 2TB disks in RAID5

How did you did that? Using Alt-F  Disk wizard?

(1.8TB usable)

Please attach the System Configuration log

[Craig Lusher]

Hi,

Thank you for your fast reply

Yes thats right using the wizard. Logs attached.

Also, I realised that I had to click 'enlarge' RAID and then 'enlarge' filesystem. When I click enlarge RAID nothing happened, so I clicked enlarge filesystem and it said 'enlarging' but again nothing happened.

I then decided to fail a disk so that it would rebuild onto the spare USB. It is doing that now as we speak!

Many thanks!

[João Cardoso]

On Thursday, April 2, 2015 at 6:50:03 PM UTC+1, Craig Lusher wrote:

Hi,

Thank you for your fast reply

Yes thats right using the wizard. Logs attached.

Also, I realised that I had to click 'enlarge' RAID

Not really, that is only needed when all the partitions that holds the RAID components have been "enlarged" since the RAID creation.

 

and then 'enlarge' filesystem.

that is only needed if the RAID size increases (after being enlarged)

 

When I click enlarge RAID nothing happened, so I clicked enlarge filesystem and it said 'enlarging' but again nothing happened.

Nothing has to happens...

 

I then decided to fail a disk so that it would rebuild onto the spare USB. It is doing that now as we speak!

Can't find nothing wrong in the log you posted. However it was created at boot time when the RAID was not yet assembled.

To generate a new one "start" the User service (it will not be keep running, its a one-shot event). Or post the contents of the RAID Details (RAID Operations, Detail)

[Craig Lusher]

Below is the RAID log and I have attached the new system log.

/dev/md0: Version : 0.90 Creation Time : Wed Mar 25 21:16:49 2015 Raid Level : raid5 Array Size : 1952987648 (1862.51 GiB 1999.86 GB) Used Dev Size : 1952987648 (1862.51 GiB 1999.86 GB) Raid Devices : 2 Total Devices : 3 Preferred Minor : 0 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Thu Apr 2 19:38:03 2015 State : active Active Devices : 2 Working Devices : 3 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 512K UUID : 142b9225:d8aed49c:cced5de7:ca715931 (local to host NAS) Events : 0.691785 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 1 8 2 1 active sync /dev/sda2 2 8 34 - spare /dev/sdc2 /proc/mdstat: Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md0 : active raid5 sdc2[2](S) sdb2[0] sda2[1] 1952987648 blocks level 5, 512k chunk, algorithm 2 [2/2] [UU] bitmap: 0/15 pages [0KB], 65536KB chunk unused devices: 

On Thursday, 2 April 2015 15:58:58 UTC+1, Craig Lusher wrote:

[João Cardoso]

Any news?

I can't see nothing wrong in the RAID device or its components, but is odd that the device is only 1.8TiB (notice that's TiB, 1024 * 1024 * 1024 * 1024 bytes, not TB)

In any case you seems to have issues with the disk, you have a lot of disks errors (both ata1 and ata2)

Apr  2 19:08:41 NAS user.err kernel: ata1.00: exception Emask 0x10 SAct 0x2 SErr 0x1000000 action 0x6 frozen

Apr  2 19:08:41 NAS user.err kernel: ata1.00: edma_err_cause=000400a0 pp_flags=00000003, EDMA self-disable, SError=01000000

Apr  2 19:08:41 NAS user.err kernel: ata1.00: cmd 60/00:08:40:18:10/04:00:00:00:00/40 tag 1 ncq 524288 in

Apr  2 19:08:41 NAS user.err kernel:          res 41/84:08:40:18:10/84:00:00:00:00/40 Emask 0x10 (ATA bus error)

Apr  2 19:08:41 NAS user.info kernel: ata1: hard resetting link

Apr  2 19:08:42 NAS user.info kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Apr  2 19:08:42 NAS user.info kernel: ata1.00: configured for UDMA/133

Apr  2 19:08:42 NAS user.info kernel: ata1: EH complete

And there are also errors in the filesystem:

EXT4-fs error (device md0): ext4_lookup:1437: inode #48759424: comm smbd: deleted inode referenced: 48791718

EXT4-fs error (device md0): ext4_lookup:1437: inode #48759424: comm smbd: deleted inode referenced: 48791718

although they maybe be caused by the disk errors (that have propagated to your files...)

You should perform a SMART test, Disk->Utilities->Health. Start with short tests, then see its current status.

Does the external USB enclosure has its own power supply? Do you power it up before powering up the NAS?

When you copied the partition table from one device to the other, are you sure that the destination device capacity was equal or greater (in number of sectors, not TB!) to the origin disk? If it was smaller, even if by a small amount, the system will try to access areas in the disk that don't exist!

[Craig Lusher]

Hi João,

Yep the USB drive is externally powered and I boot up before DNS 323.

Would the errors be due to the fact the RAID was de-graded at the time?

I'm running a AMART test (long) on both drives.

will let you know the results.

I'm certain the drives were the same size (sectors) but will double check.

Many thanks,

Craig

[João Cardoso]

On Saturday, April 4, 2015 at 9:20:54 AM UTC+1, Craig Lusher wrote:

Hi João,

Yep the USB drive is externally powered and I boot up before DNS 323.

Would the errors be due to the fact the RAID was de-graded at the time?

No, the "ata1.00: exception Emask" and following are a hardware issue, either at the disk or the box sata hardware (or both)

 

I'm running a AMART test (long) on both drives.

Other report says that errors identical to yours appear when the drive spins up after being in standby for a while.

The SMART tests, if started from the webUI, disables drive spindown, or the test will fails. If the tests completes OK and the system continues to work fine afterwards, see if the errors reappear after re-enabling spindow in Disk->Utilities, Submit.

will let you know the results.

I'm certain the drives were the same size (sectors) but will double check.

They are identical, I have now checked that in the log.

But there is something that is puzzling me: from the log you are running RC4.1, but in RC4.1 the Disk Wizard and the RAID Setup creates RAID with metadata 1.0, while yours is 0.9!

Was you in 4.0 previously?

Applied any fix?

then upgraded to 4.1?

and have you run the "before upgrade to RC4.1" script?

I can't also reproduce your 1.8TiB RAID5 capacity issue when building it from two 1.8TiB disk partitions.

I created a RAID5 from two 28GB partitions using Disk->RAID and the result degraded RAID capacity has 52GB. The ext4 filesystem has 52BG after creating it using Disk->Filesystem.

Sure, I haven't used 2TB disk partitions neither the Disk Wizard, but the code and procedure is identical. So there is something wrong in your side.

At first I though that the issue could be the fact that you are using metadata 0.9 (why are you using it?), but the mdadm manual  says that the limitation on the 0.9 metadata is on the components capacities, not the resulting RAID capacity. Is the manual page wrong? or is it a wording misunderstood? It says:

             [metadata] 0, 0.90

                     Use the original 0.90 format superblock.  This format limits arrays to 28 compo-

                     nent  devices  and  limits  component  devices of levels 1 and greater to 2 ter-

                     abytes.

[Craig Lusher]

Hiya,

Thanks again for getting back to me.

Yes I did upgrade the firmware... and no I didn't run the script before hand...

To correct this, I followed the steps exactly and ran the script before re-flashing. However I still see the RAID as 0.9...

really not sure what to do now. I was hoping that would fix everthing.

I've attached my config once again.

Thanks,

Craig

[João Cardoso]

On Sunday, April 5, 2015 at 1:13:03 AM UTC+1, Craig Lusher wrote:

Hiya,

Thanks again for getting back to me.

Yes I did upgrade the firmware... and no I didn't run the script before hand...

To correct this, I followed the steps exactly and ran the script before re-flashing. However I still see the RAID as 0.9...

There are two issues:

-the disk errors that might put your data at risk

-why did the RAID use metadata 0.9

Can you please post the output of the following commands, after ssh/telnet the box and login as the 'root' user, same password as the webUI?

ls /Alt-F/usr/www/cgi-bin/ # will show if stray RC4 files are still in use
cat /etc/bay # will show which disk is which
mdadm --detail/dev/md0 # will show the current RAID setup/usage
mdadm --examine /dev/sd*2 # will show the current RAID components setup/usage

really not sure what to do now. I was hoping that would fix everthing.

You didn't report the SMART test results nor your attempt to fail one of the disks to force a rebuild.

Do you have access to all your data on the box? Do you still have a backup?

Faced with hardware errors I would be sure to have a backup!

If you don't now have a backup (because it was on the USB disk which is now part of the RAID) I would revert that and keep the backup disk disconnected from the box until all issues are solved.

I've attached my config once again.

Yes, but it looks to be the one generated at boot time, not the one generated after boot using Services->User,user, StartNow

[Craig Lusher]

Ah... ok... Will backup to USB right now! Thank you for the warning!

I cant seem to find the output of the SMART test... where would I find this?

I've attached latest log.

Thank you

The output from commands are below, however some of the commands didn't work... As below:

"[root@NAS]# ls /Alt-F/usr/www/cgi-bin/

ls: /Alt-F/usr/www/cgi-bin/: No such file or directory

[root@NAS]# cat /etc/bay

right_dev=sda

sda=right

right_cap="2.0TB"

right_fam="Seagate SV35"

right_mod="ST2000VX000-9YW164"

left_dev=sdb

sdb=left

left_cap="2.0TB"

left_fam="Seagate SV35"

left_mod="ST2000VX000-9YW164"

usbc_dev=sdc

sdc=usbc

usbc_cap="2.0TB"

usbc_fam="BUFFALO "

usbc_mod="External HDD "

[root@NAS]# mdadm --detail/dev/md0

mdadm: unrecognized option `--detail/dev/md0'

Usage: mdadm --help

  for help

[root@NAS]# mdadm --detail/dev/md0

mdadm: unrecognized option `--detail/dev/md0'

Usage: mdadm --help

  for help

[root@NAS]# mdadm --examine /dev/sd*2

/dev/sda2:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 142b9225:d8aed49c:cced5de7:ca715931

  Creation Time : Wed Mar 25 21:16:49 2015

     Raid Level : raid5

  Used Dev Size : 1952987648 (1862.51 GiB 1999.86 GB)

     Array Size : 1952987648 (1862.51 GiB 1999.86 GB)

   Raid Devices : 2

  Total Devices : 3

Preferred Minor : 0

    Update Time : Mon Apr  6 00:00:19 2015

          State : clean

Internal Bitmap : present

 Active Devices : 2

Working Devices : 3

 Failed Devices : 0

  Spare Devices : 1

       Checksum : 4c1ea990 - correct

         Events : 691786

         Layout : left-symmetric

     Chunk Size : 512K

      Number   Major   Minor   RaidDevice State

this     1       8        2        1      active sync   /dev/sda2

   0     0       8       18        0      active sync   /dev/sdb2

   1     1       8        2        1      active sync   /dev/sda2

   2     2       8       34        2      spare   /dev/sdc2

/dev/sdb2:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 142b9225:d8aed49c:cced5de7:ca715931

  Creation Time : Wed Mar 25 21:16:49 2015

     Raid Level : raid5

  Used Dev Size : 1952987648 (1862.51 GiB 1999.86 GB)

     Array Size : 1952987648 (1862.51 GiB 1999.86 GB)

   Raid Devices : 2

  Total Devices : 3

Preferred Minor : 0

    Update Time : Mon Apr  6 00:00:19 2015

          State : clean

Internal Bitmap : present

 Active Devices : 2

Working Devices : 3

 Failed Devices : 0

  Spare Devices : 1

       Checksum : 4c1ea99e - correct

         Events : 691786

         Layout : left-symmetric

     Chunk Size : 512K

      Number   Major   Minor   RaidDevice State

this     0       8       18        0      active sync   /dev/sdb2

   0     0       8       18        0      active sync   /dev/sdb2

   1     1       8        2        1      active sync   /dev/sda2

   2     2       8       34        2      spare   /dev/sdc2

/dev/sdc2:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 142b9225:d8aed49c:cced5de7:ca715931

  Creation Time : Wed Mar 25 21:16:49 2015

     Raid Level : raid5

  Used Dev Size : 1952987648 (1862.51 GiB 1999.86 GB)

     Array Size : 1952987648 (1862.51 GiB 1999.86 GB)

   Raid Devices : 2

  Total Devices : 3

Preferred Minor : 0

    Update Time : Sun Apr  5 01:05:39 2015

          State : clean

Internal Bitmap : present

 Active Devices : 2

Working Devices : 3

 Failed Devices : 0

  Spare Devices : 1

       Checksum : 4c1d677c - correct

         Events : 691786

         Layout : left-symmetric

     Chunk Size : 512K

      Number   Major   Minor   RaidDevice State

this     2       8       34        2      spare   /dev/sdc2

   0     0       8       18        0      active sync   /dev/sdb2

   1     1       8        2        1      active sync   /dev/sda2

   2     2       8       34        2      spare   /dev/sdc2"

On Thursday, 2 April 2015 15:58:58 UTC+1, Craig Lusher wrote:

[Craig Lusher]

Hi any opinions on what I should do?

I have copied all my data off, so clear to destroy raid and start again with fresh install, but how do I do this?

Many thanks,

Craig

[João Cardoso]

On Friday, April 10, 2015 at 12:16:58 AM UTC+1, Craig Lusher wrote:

Hi any opinions on what I should do?

I have copied all my data off, so clear to destroy raid and start again with fresh install, but how do I do this?

Unplug the backup USB disk, reboot just to start afresh, and use the Disk Wizard to create the RAID5. 

I expect it to create a RAID5 with metadata 1.0, which shall solve the capacity issue. Don't restore the data from the backup until you are sure that it's working OK, i.e., the degraded RAID5 capacity should be reported as being  3.6TB (actually 3.6TiB) and no disk hardware errors should appear in the system log.

I think that you have missexecuted the RC4 to RC4.1 script, and you used the Disk Wizard from 4.0 -- just a guess.

 

But before doing the above post the SMART tests -- Disk->Utilities->Health, Status. The bottom part of the report:

SMART Error Log Version: 1

No Errors Logged

SMART Self-test log structure revision number 1

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error

# 1  Short offline       Completed without error       00%      3049         -

...

# 5  Extended offline    Completed without error       00%      2959         -

If you see something else, like

 # 5  Extended offline    Aborted by host               10%     13209         -

It means that the test didn't complete or completed on error, possibly because the drive spin down.

But that's best to attache both disks SMART reports.

Many thanks,

Craig