Upgrade to a larger hard drive (copy/cloning)

2,840 views
Skip to first unread message

Steven Bone

unread,
Mar 5, 2013, 8:55:13 PM3/5/13
to al...@googlegroups.com
What is the best way to upgrade to new drives?  I have two 1 TB drives that are not set up in a RAID configuration.  I'm wondering if there is a way that I could simply remove the right drive, replace it with a new 2 TB drive, copy the left drive to the new right drive, and do the same with the drive that was initially in the right position.  If this is possible, how do you do it?  

Would you merely partition the new drive and use the "Directory Browse" page to copy them (or would you use rsync, or others?)

Thank you for the firmware - Keep up the good work!  

Joao Cardoso

unread,
Mar 6, 2013, 9:27:04 AM3/6/13
to al...@googlegroups.com
On Wednesday, March 6, 2013 1:55:13 AM UTC, Steven Bone wrote:
What is the best way to upgrade to new drives?  I have two 1 TB drives that are not set up in a RAID configuration.  I'm wondering if there is a way that I could simply remove the right drive, replace it with a new 2 TB drive, copy the left drive to the new right drive, and do the same with the drive that was initially in the right position.  If this is possible, how do you do it?  

The way you described seems OK to me.

To simplify the new disks setup (initial partitioning and filesystem creation), I would boot the box with the new disks and use the Disk Wizard first(*); then I would remove (using Disk->Utilities->eject or powering off the box) one of the new disks, insert one of the old ones and copy it, then do the same for the other disk.

(*) a report says that the Disk Wizard sees "no disks" when they are not partitioned; if this happens to you, use first the Disk Partitioner to create a dummy partition on each disk before using the Disk Wizard.
 
Would you merely partition the new drive and use the "Directory Browse" page to copy them (or would you use rsync, or others?)

The Folder/Directory Browse copy  operation runs at about 6MB/sec, while a plain 'cp -a' from the command line runs at about 9MB/sec (this is "fixed" for RC3). This is the fastest disk to disk copy speed that you will get.
As 'cp -a' might have problems with certain folder layouts (very large wide and/or deep folders structures), you might want to do the copy one folder at a time. The Folder browse does not has this limitation (neither RC3)

Thank you for the firmware - Keep up the good work!  

Thanks.

shredhead

unread,
Mar 27, 2014, 6:31:53 PM3/27/14
to
I was also considering a hard drive upgrade.

The approach I thought of is a little different though...

1) break the mirror
2) pull one drive, replace with new
3) copy /sda4 to new drive since that's where Alt-F is located.
4) recreate the mirror, tell it to rebuild. 
5) put in second drive, sync the mirror
6) enlarge the raid (mirror)

I'm wondering if this is a good approach and if it is better or worse than the straight copy.. which one is faster to get finished....

I was also considering doing away with the /sda4 and /sdb4, just putting Alt-F on the mirrored partition. One partition for swap, the rest of the drive for data. To do that I would obviously need to move Alt-F. I did find a thread where this move procedure is explained. Actually I'm the one who asked about that around 2 years ago. :)  https://groups.google.com/forum/#!searchin/alt-f/move$20Alt-F/alt-f/Bme24yqxNds/pryX1gJL2qkJ  I don't remember why I needed to move it then but I do know it worked. 

is there a down side to just having one partition the size of the entire hard drive (other than swap space)? I guess I just worry about /Alt-F not having a mirrored copy. 

Message has been deleted

João Cardoso

unread,
Mar 28, 2014, 3:54:53 PM3/28/14
to al...@googlegroups.com


On Thursday, March 27, 2014 7:59:55 PM UTC, shredhead wrote:
I was also considering a hard drive upgrade.

The approach I thought of is a little different though...

1) break the mirror
2) pull one drive, replace with new
3) copy /sda4 to new drive since that's where Alt-F is located.
4) recreate the mirror, tell it to rebuild. 
5) put in second drive, sync the mirror
6) enlarge the raid (mirror)

I'm wondering if this is a good approach

Yes, it should work (you missed step 2b), partition new disk, and 6b), enlarge filesystem).
More about Alt-F latter.

But you don't need to break the mirror and copy data. Replacing disks is like repairing a degraded array, you deliberated make each disk to "fail", one at a time. And the data will always be available without downtime.

The following is only for metadata 0.9 and if the new disk is smaller than 2.2TB, as bigger disks require RAID metadata 1.x and that can't be changed without breaking the array:

1-fail, then remove one disk component from the array, then unplug the disk. the arrays is still be  working in degraded mode
2-insert the new disk and partition it, one partition of type RAID (this is an Alt-F RAID webUI requirement)
3-add the new disk RAID partition to the RAID as a new component and wait for rebuild to finish and the array leaving the degraded state
4-repeat step 1,2,3 for the other disk
5-enlarge both disks RAID partitions (watchout, ***DON'T*** change its start sectors!)
6-remove write-intent bitmap from the array, enlarge the RAID, re-create bitmap
7-enlarge the filesystem

All the above can be done using the webUI.

Step 1 can be done using Disk->Maintenance, Eject, which will stop swap on the disk, unmount all "non-RAID" filesystems, fail and then remove all disk RAID partitions from RAID arrays (I haven't tested this for a long time, hopefully it will be still working, check it's OK before continuing with step 2)

Step 2 partitioning can eventually make step 5 unneeded, not sure. If the RAID partition in the new disk is created with the desired final size, only part of it will be used.
From the mdadm manual:

   SIZE CHANGES
       Normally when an array is built the "size" it taken from the smallest of
       the drives.  If all the small drives in an arrays are, one  at  a  time,
       removed and replaced with larger drives, then you could have an array of
       large drives with only a small amount used.  In this situation, changing
       the  "size"  with  "GROW" mode will allow the extra space to start being
       used.  If the size is increased in this way,  a  "resync"  process  will
       start to make sure the new parts of the array are synchronised.

       Note  that when an array changes size, any filesystem that may be stored
       in the array will not automatically grow to use the space.  The filesys-
       tem will need to be explicitly told to use the extra space.

       Also  the size of an array cannot be changed while it has an active bit-
       map.  If an array has a bitmap, it must be removed before the  size  can
       be changed. Once the change it complete a new bitmap can be created.

 
and if it is better or worse than the straight copy.. which one is faster to get finished....

I was also considering doing away with the /sda4 and /sdb4, just putting Alt-F on the mirrored partition.

That's OK. For everytime usage there will be no problems with that, only flexibility might be missed, because you have to "unmount" Alt-F when disk maintenance is needed.
E.g., if you ssh the box you might find that you can't unmount filesystems, or you might find that you can't use your favorite text editor, or...
The normal setup is to have Alt-F as one folder in the user filesystem, but if you have empty sda4/sdb4 inherited from the D-Link fw, then using them for Alt-F increases flexibility.

The Alt-F data is not critical, it's not like your child anniversary photos, you can always re-install Alt-F packages latter, you only have aditional work setting it up again.
 
One partition for swap, the rest of the drive for data. To do that I would obviously need to move Alt-F. I did find a thread where this move procedure is explained. Actually I'm the one who asked about that around 2 years ago. :)  https://groups.google.com/forum/#!searchin/alt-f/move$20Alt-F/alt-f/Bme24yqxNds/pryX1gJL2qkJ  I don't remember why I needed to move it then but I do know it worked. 

is there a down side to just having one partition the size of the entire hard drive (other than swap space)? I guess I just worry about /Alt-F not having a mirrored copy. 

You can use a RAID just for it, if you like, but there is no irreplaceable data.
 


On Wednesday, March 6, 2013 9:27:04 AM UTC-5, João Cardoso wrote:

shredhead

unread,
Mar 31, 2014, 8:32:50 AM3/31/14
to al...@googlegroups.com
so far...

1) fail the disk, then remove.
 - Alt-F web page shows errors, missing parts of the code for the menu system itself - a reboot fixes this
2) insert new disk, partition it, add to RAID
- Alt-F partition wizard seems to be the best option for me as new disk is 3TB and I don't know the numbers to use in the partitioning page. I tried copying the partitions from the old disk even though they were the wrong size, but since I inherited disks from DNS firmware, the partition type where the data resides is Linux, not RAID. So I'm using the wizard in hopes that I have the right partition type on the new disk. 

I also noticed something I wish were not the case. if I put in both new disks and hook up one of the old ones USB, it can't see the md0 (RAID). I'd hoped to do that in order to just copy the data over but no dice. My usb portable drives don't work, only the usb converter thingie that has its own power for the drive. I think in another thread I saw that the usb port can't provide power. bummer. 

I will add more to this as things move forward, to benefit others who are doing hard drive upgrades when they have a mirror and want a mirror in the new setup. 

João Cardoso

unread,
Mar 31, 2014, 4:47:21 PM3/31/14
to al...@googlegroups.com


On Monday, March 31, 2014 1:32:50 PM UTC+1, shredhead wrote:
so far...

1) fail the disk, then remove.
 - Alt-F web page shows errors,

It would be great if you did a screenshot. Was you running RC3 at the time?
 
missing parts of the code for the menu system itself - a reboot fixes this
2) insert new disk, partition it, add to RAID
- Alt-F partition wizard seems to be the best option for me as new disk is 3TB

The procedure in the previous post is intended for smaller than 2TB disks. Bigger disks (really partitions) require a RAID metadata version 1.x. As the previous post procedure preserves the RAID, it will have metadata 0.9. In the best scenario, after enlarging the array it will be limited to 2TB.
 
and I don't know the numbers to use in the partitioning page. I tried copying the partitions from the old disk even though they were the wrong size, but since I inherited disks from DNS firmware, the partition type where the data resides is Linux, not RAID. So I'm using the wizard in hopes that I have the right partition type on the new disk.

Yes. But your will have a new degraded RAID, which you must "destroy" before adding the component to the existing RAID. All that can be done using the webUI.

I also noticed something I wish were not the case. if I put in both new disks and hook up one of the old ones USB, it can't see the md0 (RAID). I'd hoped to do that in order to just copy the data over but no dice. My usb portable drives don't work, only the usb converter thingie that has its own power for the drive. I think in another thread I saw that the usb port can't provide power.

Yes, there is no other reason.

shredhead

unread,
Apr 1, 2014, 6:58:08 AM4/1/14
to
Thanks for the tips. 

The procedure I followed was pretty much, fail one drive, remove it, add one of the new ones, let it repair, then do the same for the second drive. Fortunately I have a sata to usb converter kit, so I have the old drive hooked up via usb (which let me copy Alt-F folder to the new drive). 

As expected, I have data on the two new drives (md1) and although the partitions are the size of the drives (I actually bought 2TB not 3TB as I'd originally posted) the file system is still the size of the old drives and "enlarge" did not work I assume because of the metadata version. 

Unfortunately I feel foolish now having spent so much time on this... 

The old drives used partition type Linux not RAID. This must be something original DNS firmware did even though I had a mirror and I have to assume that when I first installed Alt-F I told the disk wizard not to mess with my disks. So I can mount that partition, wipe out the 2TB drives, recreate the mirror with new metadata version, and copy the data over to the new mirror. If it blows up I still have the second original disk to copy from. 

João Cardoso

unread,
Apr 1, 2014, 9:41:43 AM4/1/14
to al...@googlegroups.com


On Tuesday, April 1, 2014 12:04:11 AM UTC+1, shredhead wrote:
Thanks for the tips. 

The procedure I followed was pretty much, fail one drive, remove it, add one of the new ones, let it repair, then do the same for the second drive. Fortunately I have a sata to usb converter kit, so I have the old drive hooked up via usb (which let me copy Alt-F folder to the new drive). 

As expected, I have data on the two new drives (md1) and although the partitions are the size of the drives (I actually bought 2TB not 3TB as I'd originally posted) the file system is still the size of the old drives and "enlarge" did not work I assume because of the metadata version. 

That was working OK last time I checked. It depends on how the partitions are now setup, please post the screenshots of the Status and Disk Partitioner for both (new) disks.

So, after replacing both RAID disks, you have to

1-enlarge the RAID
2-enlarge the md filesystem

Enlarging the RAID makes it use the full size of its smaller RAID component partition, while enlarging the filesystem make it use all space of the device (RAID in your case)
 

Unfortunately I feel foolish now having spent so much time on this... 

I don't think so 
 

The old drives used partition type Linux not RAID. This must be something original DNS firmware did even though I had a mirror and I have to assume that when I first installed Alt-F I told the disk wizard not to mess with my disks. So I can mount that partition, wipe out the 2TB drives, recreate the mirror with new metadata version, and copy the data over to the new mirror. If it blows up I still have the second original disk to copy from. 

Wait a little more, let's analyse the screenshots, have a diagnostic and get a cure (looks like your physician talk, doesn't it?, just replace screenshots with Rx)

Ryan Rosario

unread,
Aug 3, 2015, 12:23:40 AM8/3/15
to Alt-F
how would I approach this if I'm going from 2x1TB drives to 2x3TB drives?

D Ace

unread,
Jan 24, 2019, 4:46:19 PM1/24/19
to al...@googlegroups.com
Hey All,
First, many Kudos to João! Without his work, we'd be sitting with antiquated paper-weights on our desks!

Now my spin on this topic is as such, running Alt-F v1.0 and I had 2 x 3TB drives in RAID 1 configuration with a DNS 323. I had one drive go toast on me so tried got another 3TB drive to replace.

Did all the procedures from article (https://sourceforge.net/p/alt-f/wiki/How%20to%20fix%20a%20degraded%20RAID1%20array/) to rebuild the raid but lo and behold, the surviving drive has a read error and that kills the rebuild.

I can still read from the drive and am in the process of copying all the data off of the remaining drive. Now the question. I'm looking to up my space and have an 2 x 8TB drives that I want to use but one of them is currently storing the data that I'm getting off of my remaining 3 TB drive. My process was going to be as follows:

1) Compete the backup of the 3 TB drive
2) Power down
3) Remove both 3 TB drives and add the new 8 TB drive
4) Power up and ??? Prep the 8 TB drive for raiding??? <- This is where I'm not too sure on whether the single drive by itself can be "prep'ed" for raiding by itself or what the process should be.
5) Copy all the data from the current 8 TB drive to the 8 TB drive now in the DNS 323
6) Power down
7) Put the 2nd 8 TB drive in
8) Power up and then start the rebuild RAID process as detailed in the article I linked to earlier.

Any steps or thoughts on this would be appreciated.
Thanks again...

João Cardoso

unread,
Jan 25, 2019, 2:03:39 PM1/25/19
to Alt-F


On Thursday, 24 January 2019 21:46:19 UTC, D Ace wrote:
Hey All,
First, many Kudos to João! Without his work, we'd be sitting with antiquated paper-weights on our desks!

Now my spin on this topic is as such, running Alt-F v1.0 and I had 2 x 3TB drives in RAID 1 configuration with a DNS 323. I had one drive go toast on me so tried got another 3TB drive to replace.

Did all the procedures from article (https://sourceforge.net/p/alt-f/wiki/How%20to%20fix%20a%20degraded%20RAID1%20array/) to rebuild the raid but lo and behold, the surviving drive has a read error and that kills the rebuild.

I can still read from the drive and am in the process of copying all the data off of the remaining drive.

That can fail for some files or folders, because if the (SMART) read errors occurs in disk sectors used by actual files, that specific file copy fails and whole copy process might also fail and be stops at that point. You must verify at least the number of files and folders and space occupied on both filesystems.
 
Now the question. I'm looking to up my space and have an 2 x 8TB drives that I want to use but one of them is currently storing the data that I'm getting off of my remaining 3 TB drive. My process was going to be as follows:

1) Compete the backup of the 3 TB drive
2) Power down
3) Remove both 3 TB drives and add the new 8 TB drive
4) Power up and ??? Prep the 8 TB drive for raiding??? <- This is where I'm not too sure on whether the single drive by itself can be "prep'ed" for raiding by itself or what the process should be.

You can create a degraded RAID with that disk
 
5) Copy all the data from the current 8 TB drive to the 8 TB drive now in the DNS 323
6) Power down
7) Put the 2nd 8 TB drive in
8) Power up and
 
then start the rebuild RAID process as detailed in the article I linked to earlier.

Any steps or thoughts on this would be appreciated.

You are doing it "by the book", but that involves a lot of copy operations. There could be a faster way, but depends on your linux expertise.
I would put the 3TB plus the two 8TB drives already setup as RAID in a linux PC and perform the copies there, which is some 5x faster at least.

Are you doing the copy in the nas? drive to drive operations are not that fast...
As you have already started the copy and I don't know how you are doing it, it's difficult to give clues.

If you want to use only the NAS, remember that your current 3TB disk is degraded, so you can remove the failed disk from the RAID (it certainly has been already kicked off), and then from the box, which means that now you can insert one of the 8TB drives in the box and create a second degraded RAID. You can then copy from one RAID to the other; after that, stop the 3TB RAID, remove the disk, replace it with the 2nd 8TB, add it to the 8TB RAID and resync (copy) starts. Two copies only.

Otherwise, remember that a "normal" filesystem on the 8TB drive with data on it can be upgraded to a degraded RAID (wiki, but see next).
The destination 8TB drives should be already partitioned under Alt-F, using a large (2GB?) 1st partition swap (need by fsck), the rest with ext4.
The wiki on turning a standard fs to RAID does some lengthy operations, such as a fsck and shrinking to the minimum possible. It is possible to shrink a fs only a small bit, enough to create the RAID metadata (version 1.0, *not* the default 1.1 or 1.2, which will destroy the data). The wiki also says to copy using the "Folders Browse" operations, which is NOT adequate for such big amounts of data.

So, it all depends on how you are actually doing the copy operation to the 8TB disk and how it was partitioned and filesystem formatted.

D Ace

unread,
Jan 28, 2019, 1:39:28 PM1/28/19
to al...@googlegroups.com
Hi João,
Thanks for the response.
I'm mostly on the windows platform and work over the network so usually get 10-15mb transfer speeds and can live with that for copying (old school)...
The big question is from step 4...using the article I referenced, Do I copy the Disk -> Partitioner section, do I copy the existing partition from the existing 3TB drive to the 8TB drive and if so, 
a) is the 8TB drive now only seen as a 3TB drive (- yes, I see that now)? 
b) if it is viewed as only 3TB fo rthe degraded array, how/do I have to expand that partition (i.e. how would I go about that?)

I'm plowing through right now by just trying the above (partitioned the 8TB using the 3TB raid partition, taking out the 3TB drive - only 8TB in the Dlink 323 - and hunting for how to expand partition)
Any help is appreciated
Thanks
Dave

UPDATE: Dummy here...should have went straight to wizard! It's currently running so hopefully that will put my 8TB drive into degraded raid status and I can go from there.
Will update

D Ace

unread,
Feb 7, 2019, 8:24:16 AM2/7/19
to Alt-F
Finally Done. Was able to get 2 x 8TB drives in RAID up and running.
Thanks for all you've done and continue to do João!
D

On Tuesday, March 5, 2013 at 8:55:13 PM UTC-5, Steven Bone wrote:
Reply all
Reply to author
Forward
0 new messages