Slow copycontent when down to last 105 GB of New EXT4 HDD from OLD EXT3 HDD

[Peter Canada]

HI All,

First I would like to thank the author for his long and hard work on this project, I'll see if there is somewhere to contribute financially a small amount somewhere on this site. Anyhow, I installed a new hdd into my DNS 323 and plan on retiring my old one after a couple years of use. So I decided to format the new drive with EXT4 for bigger files and use copycontent in setup/folders sections to copy from the old WD1002 model to the WD1003 EXT4 model of HDD and the copying was going about 20 GB an hour about until it got to the last 107 GB.and slowed down to about 0.1 GB an hour. It should have been finished by now if it hadn't slowed. I was hoping to find out the cause and if I can fix this so it finishes or if you guys have any solutions? Is there a way to safely stop the copy progress and just use rsync maybe? I have attached a pic of the current status and when I check setup/folders it says folder operation currently in progress.

P.S. DO you guys have an IRC channel for Alt-F anywhere?

[Peter Canada]

I should also mention Windows 7 is no longer showing the NAS Shares in its network list.

[João Cardoso]

On Tuesday, July 22, 2014 8:25:12 AM UTC+1, Peter Canada wrote:

I should also mention Windows 7 is no longer showing the NAS Shares in its network list.

The system is very busy, that might explain it.
The webUI is still working, right?

On Tuesday, July 22, 2014 1:22:46 AM UTC-6, Peter Canada wrote:

HI All,

First I would like to thank the author for his long and hard work on this project, I'll see if there is somewhere to contribute financially a small amount somewhere on this site. Anyhow, I installed a new hdd into my DNS 323 and plan on retiring my old one after a couple years of use. So I decided to format the new drive with EXT4 for bigger files and use copycontent in setup/folders sections to copy from the old WD1002 model to the WD1003 EXT4 model of HDD and the copying was going about 20 GB an hour

My measurements at the time gave about 9MB/sec disk to disk copy, which translates to about 30MB/s (with RC4, with RC3 it was 6MB/s).
I just don't remember if it was MB or MiB...
 

about until it got to the last 107 GB.and slowed down to about 0.1 GB an hour.

What disk led is blinking more: the one from the source or the one from the destination disk?
It might be copying small files. It is not the same to copy one 1GB file and 1000 1MB files.
It might have reached a spot in the disk which is requiring several data read retries -- see System->Utilities->View Logs, kernel Log, look at the end to see if any disk read errors/retries are happening.
Also see the Running Processes log, look for the 'cp' command in the Command column and see how is its %VSZ %CPU columns -- please attach that log.
ah, and keep the Autorefresh checkbox in the Status page unchecked!
 

It should have been finished by now if it hadn't slowed. I was hoping to find out the cause and if I can fix this so it finishes or if you guys have any solutions? Is there a way to safely stop the copy progress

No. If you stop the copy process (it can be done!) you will have a hard time figuring out where it stops copying, and you will have to restart all again.
 

and just use rsync maybe

It's worse. rsync asks for a lot of memory that the 323 don't have, swapping will start and the disks throughput will drop yet more.
If I had waited so long I would wait a bit more.

? I have attached a pic of the current status and when I check setup/folders it says folder operation currently in progress.

P.S. DO you guys have an IRC channel for Alt-F anywhere?

IRC? No, I'm too old for these new exciting technologies -- do you know Morse code? :)
But if someone want's to set it up I might try to use it.

[Peter Canada]

I am not sure I can tell the difference but it looks like the left side is stopping more., but here are a couple files!

[Peter Canada]

On Tuesday, July 22, 2014 7:14:32 AM UTC-6, João Cardoso wrote:

On Tuesday, July 22, 2014 8:25:12 AM UTC+1, Peter Canada wrote:

I should also mention Windows 7 is no longer showing the NAS Shares in its network list.

The system is very busy, that might explain it.
The webUI is still working, right?

 

Yes more so now then before.

On Tuesday, July 22, 2014 1:22:46 AM UTC-6, Peter Canada wrote:

HI All,

First I would like to thank the author for his long and hard work on this project, I'll see if there is somewhere to contribute financially a small amount somewhere on this site. Anyhow, I installed a new hdd into my DNS 323 and plan on retiring my old one after a couple years of use. So I decided to format the new drive with EXT4 for bigger files and use copycontent in setup/folders sections to copy from the old WD1002 model to the WD1003 EXT4 model of HDD and the copying was going about 20 GB an hour

My measurements at the time gave about 9MB/sec disk to disk copy, which translates to about 30MB/s (with RC4, with RC3 it was 6MB/s).
I just don't remember if it was MB or MiB... 

I think it was going about 4 MB/s before the slowdown, would compressed video files affect the speed?
 

about until it got to the last 107 GB.and slowed down to about 0.1 GB an hour.

What disk led is blinking more: the one from the source or the one from the destination disk?
It might be copying small files. It is not the same to copy one 1GB file and 1000 1MB files.
It might have reached a spot in the disk which is requiring several data read retries -- see System->Utilities->View Logs, kernel Log, look at the end to see if any disk read errors/retries are happening.
Also see the Running Processes log, look for the 'cp' command in the Command column and see how is its %VSZ %CPU columns -- please attach that log.
ah, and keep the Autorefresh checkbox in the Status page unchecked!

I attached both files in another post !

 

It should have been finished by now if it hadn't slowed. I was hoping to find out the cause and if I can fix this so it finishes or if you guys have any solutions? Is there a way to safely stop the copy progress

No. If you stop the copy process (it can be done!) you will have a hard time figuring out where it stops copying, and you will have to restart all again.

Not fond of this idea either.
 

 

and just use rsync maybe

It's worse. rsync asks for a lot of memory that the 323 don't have, swapping will start and the disks throughput will drop yet more.
If I had waited so long I would wait a bit more.

Yeah I will wait but at this rate it will take 50 more days to complete, definitely don't want to wait that long.
 

? I have attached a pic of the current status and when I check setup/folders it says folder operation currently in progress.

P.S. DO you guys have an IRC channel for Alt-F anywhere?

IRC? No, I'm too old for these new exciting technologies -- do you know Morse code? :)
But if someone want's to set it up I might try to use it.

LOL the only always on text chat, good old back to basic. Where I go to get answers as a last resort. I might consider opening a channel but doesn't seem like the userbase for DNS 3xx is large. Any plans for DNS 343 etc support is it alot more work?

[Peter Canada]

As a side note in the last few minutes I think the speed just jumped up, I will know shortly.

[João Cardoso]

On Tuesday, July 22, 2014 3:00:15 PM UTC+1, Peter Canada wrote:

On Tuesday, July 22, 2014 7:14:32 AM UTC-6, João Cardoso wrote:

On Tuesday, July 22, 2014 8:25:12 AM UTC+1, Peter Canada wrote:

I should also mention Windows 7 is no longer showing the NAS Shares in its network list.

The system is very busy, that might explain it.
The webUI is still working, right?

 

Yes more so now then before.

Don't need to do that, just watch the disk leds. And take a break every now and then and go watch your laundry-washer machine dial rotating:-(
Joking, I understand what you mean.
 

On Tuesday, July 22, 2014 1:22:46 AM UTC-6, Peter Canada wrote:

HI All,

First I would like to thank the author for his long and hard work on this project, I'll see if there is somewhere to contribute financially a small amount somewhere on this site. Anyhow, I installed a new hdd into my DNS 323 and plan on retiring my old one after a couple years of use. So I decided to format the new drive with EXT4 for bigger files and use copycontent in setup/folders sections to copy from the old WD1002 model to the WD1003 EXT4 model of HDD and the copying was going about 20 GB an hour

My measurements at the time gave about 9MB/sec disk to disk copy, which translates to about 30MB/s (with RC4, with RC3 it was 6MB/s).
I just don't remember if it was MB or MiB... 

I think it was going about 4 MB/s before the slowdown, would compressed video files affect the speed?

No, a bit is a bit, it doesn't matter what it means.

 

about until it got to the last 107 GB.and slowed down to about 0.1 GB an hour.

What disk led is blinking more: the one from the source or the one from the destination disk?
It might be copying small files. It is not the same to copy one 1GB file and 1000 1MB files.
It might have reached a spot in the disk which is requiring several data read retries -- see System->Utilities->View Logs, kernel Log, look at the end to see if any disk read errors/retries are happening.
Also see the Running Processes log, look for the 'cp' command in the Command column and see how is its %VSZ %CPU columns -- please attach that log.
ah, and keep the Autorefresh checkbox in the Status page unchecked!

I attached both files in another post !

hmmm, everything looks OK, no disk read errors/retries.

 3928  3927 root     D N   2272   4%  36% du -sm /mnt/Personal/Public/.
  562   559 root     D    73648 121%   7% cp -a . /mnt/Personal/Public

The cp is not using much cpu (it's an I/O process), although it is using too much memory, 121%, which means that disk swapping is happening, lowering the disk throughput.

But... to display the completion bar (which you are not using anymore) the amount of data copied is *measured* (the 'du' process) every now and then, and *that* is causing the low throughput. That's what one pays for having a visual feedback of what is happening.
I have to fix that! Currenty it is calculated every 10 seconds, which is OK for GB copy/moves, but not for TB operations.

 

It should have been finished by now if it hadn't slowed. I was hoping to find out the cause and if I can fix this so it finishes or if you guys have any solutions? Is there a way to safely stop the copy progress

No. If you stop the copy process (it can be done!) you will have a hard time figuring out where it stops copying, and you will have to restart all again.

Not fond of this idea either.
 

 

and just use rsync maybe

It's worse. rsync asks for a lot of memory that the 323 don't have, swapping will start and the disks throughput will drop yet more.
If I had waited so long I would wait a bit more.

Yeah I will wait but at this rate it will take 50 more days to complete, definitely don't want to wait that long.

In that case, if you have access to a a linux computer it's faster to physically attach the disks to it and perform a manual copy.
If you don't want all that work, use the command 'cp -a' from the command line. To see how far it is you have to login in another console and use the 'df -h' command.

To stop the current copy, just kill the 'cp -a' process, use 'kill 562'. The already copied data will be unaltered.
To just stop the progress bar from delaying the whole copy, you have to periodically (every 10 seconds) kill the 'du' process, doing something such as:

while true; do while ! pidof du; do echo -n '.'; sleep 1; done; echo "kill the bastard"; killall du; echo "waiting for it"; sleep 5; done; done

To finish the above "natural born du killer" just hit CTRL-C

 

? I have attached a pic of the current status and when I check setup/folders it says folder operation currently in progress.

P.S. DO you guys have an IRC channel for Alt-F anywhere?

IRC? No, I'm too old for these new exciting technologies -- do you know Morse code? :)
But if someone want's to set it up I might try to use it.

LOL the only always on text chat, good old back to basic. Where I go to get answers as a last resort. I might consider opening a channel but doesn't seem like the userbase for DNS 3xx is large.

Yes, it's not.
 

Any plans for DNS 343 etc support is it alot more work?

Without having one it is not even possible.
 

[Peter Canada]

Joao,

Just so you know that the system resume copying at a much improved rate at about 8 am this morning and should finish to night it is on its last 14 GB or so.

[João Cardoso]

On Wednesday, July 23, 2014 4:01:01 AM UTC+1, Peter Canada wrote:

Joao,

Just so you know that the system resume copying at a much improved rate at about 8 am this morning and should finish to night it is on its last 14 GB or so.

Thanks for reporting back the outcome.
Most time people don't bother to communicate the outcome of an issue, and I don't know whether the proposed solution it worked or not.

In your case the copy probably reached folders with many small files, such as pictures or music, and the copy throughput drops a lot in that case. The figures I gave before, 9MB/s are for a single big file.

I also suggested a cure to increase the throughput, the "natural born du killer" script, that only works when the progressbar is being displayed, what was not your case. And as most users abandon the page after a few hours, at most, it is unlikely that it will be of any use.


 

[Peter Canada]

Well it slowed again this time at around 64 GB, it is currently at 63.3GB and has about 5 GB to go and seems to be doing about 0.1 GB every few hours.............sighs. I attached the logs again in case you were interested. Right now I am just trying to get everything off the old drive. I plan on moving some of what is on the new drive to another new drive of the same model once I get the old drive finished. Maybe, I will rig up a linux box using an old computer I have to increase transfer speeds, some mentioned ftp, is that or would that be really faster?

[João Cardoso]

On Thursday, July 24, 2014 3:31:26 AM UTC+1, Peter Canada wrote:

Well it slowed ag

 

ain this time at around 64 GB, it is currently at 63.3GB and has about 5 GB to go and seems to be doing about 0.1 GB every few hours.............sighs. I attached the logs again in case you were interested.

Nothing new.

I still find the "Load" bargraph at 4 on the Status page too high. Do you have the AutoRefresh checkbox checked? DON'T -- it's nice but consumes much juice.

Right now I am just trying to get everything off the old drive. I plan on moving some of what is on the new drive to another new drive of the same model once I get the old drive finished. Maybe, I will rig up a linux box using an old computer I have to increase transfer speeds,

That's the fastest way.

some mentioned ftp, is that or would that be really faster?

ftp is, (the real ftp, not sftp nor ftps, that some clients use by default and that Alt-F also has), accordingly to my experience, the fastest network protocol, but even so you might have 20MB/s, so copy across the network if you have no other chance. Some people put the box disk on an USB enclosure or adapter and attach it to the desktop computer.

[Peter Canada]

No I have note used auto refresh at all during this procedure, fyi and as for the load I have seen it as high as 11.xx before it slowed down the first time. I do have a USB 3.0/ESATA Enclosure that I bought as well, I may just use that and try finding a real ftp program.

[Peter Canada]

After 7 Days and 14 hours the copy process finished 916GB finally finished. Just for reference, now checking drives to confirm proper copy.

[Tom Schmidt]

I did a similar rsync to an external USB-SATA connector on a 2TB disk
on my DNS-323-B1 of about 1.5TB of data, and it took about a week too.
I was attributing that slowness on the USB. Once that was done, I
then moved the 2TB disk internal and had the RAID mirroring rebuild
itself to a second bare drive, which took about 12 hours.

These are inexpensive boxes, so their throughput is not very high.

Tom

> --
> You received this message because you are subscribed to the Google Groups
> "Alt-F" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to alt-f+un...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/alt-f/6ffe6213-7822-44c5-bf1a-d023dd46271e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Peter Canada]

Great to know!

[João Cardoso]

On Friday, July 25, 2014 1:58:09 AM UTC+1, Tom Schmidt wrote:

I did a similar rsync to an external USB-SATA connector on a 2TB disk
on my DNS-323-B1 of about 1.5TB of data, and it took about a week too.

 

I was attributing that slowness on the USB.

Disk to disk copy is a slow operation in the box, be it between the two internal drives or between an internal drive and an USB attached one.
That's why RAID5, which uses an external USB drive, has not a worse performance that RAID1!