Thank you, and...

[notoneofmyseeds]

Thank you again and again, João Cardoso.

I have a very big and important project here; managing decades worth of
very valuable family data.

I need to transfer from one HD to another, as I upgraded my HDs.

I agonized over using any file manager to do this for fear of loosing
data because we speak here of a lot, and lot of data, with mixed
characteristics; name schemes, etc.

I recalled seeing a function to copy and paste and tried that. I've
never used it. I only used that portion of Alt-F to give permissions and
create folders, etc.

Right this minute, it's copying from an external HD to an internal HD on
the DNS, loads and loads of data.

So far, no problem. I'm elated. And feel the need to say, from deep in
my heart;

Thank you João Cardoso!

It is for this reason I've been asking lately about rsync, etc. No
responses there, but so far this is working. I will still need to find a
secure and reliable way to backup to an external USB drive plugged to
the NAS via a HUB. But that's another day.

For this minute; I'm very, very happy. When folks like me get to you on
this group and you're upset, please remember how you've made me happy. I
am confident bypassing a file manager and doing it all through Alt-F
will be good. And I'm grateful for your work, very deeply!

Thank you, João Cardoso!

[João Cardoso]

On Thursday, 20 August 2015 08:46:34 UTC+1, notoneofmyseeds wrote:

Thank you again and again, João Cardoso.

I have a very big and important project here; managing decades worth of
very valuable family data.

I need to transfer from one HD to another, as I upgraded my HDs.

I agonized over using any file manager to do this for fear of loosing
data because we speak here of a lot, and lot of data, with mixed
characteristics; name schemes, etc.

I recalled seeing a function to copy and paste and tried that. I've
never used it. I only used that portion of Alt-F to give permissions and
create folders, etc.

Right this minute, it's copying from an external HD to an internal HD on
the DNS, loads and loads of data.

Please notice that I only tested that functionality (Setup->Folders, Copy/Paste) with a few GB of data.

Keep and eye on the swap usage in the Status page, if it grows (keeps growing) too much, its a bad sign. Eventually the operation will complete, but with a very low throughput.

[notoneofmy]

On 15-08-20 6:38 PM, João Cardoso wrote:
> Please notice that I only tested that functionality (Setup->Folders,
> Copy/Paste) with a few GB of data.
> Keep and eye on the swap usage in the Status page, if it grows (keeps
> growing) too much, its a bad sign. Eventually the operation will complete,
> but with a very low throughput.

I will do that. And thanks.

And if it does not work; would you please suggest, as I still do think
my best options are with Alt-F (for example during the copy, Firefox
crashed on the computer, but since the copying was done within Alt-F,
this was not a problem). So, what other options within Alt-F would you
suggest, if this does not work as well. As I have about 2TB of data to
process this way; copy over to the new HDs.

Thanks a lot.

[notoneofmyseeds]

On 20.08.2015 18:38, João Cardoso wrote:
> Please notice that I only tested that functionality (Setup->Folders,
> Copy/Paste) with a few GB of data.
> Keep and eye on the swap usage in the Status page, if it grows (keeps
> growing) too much, its a bad sign. Eventually the operation will
> complete, but with a very low throughput.

And you're right, it's not working.

It's been on for the past 8 hours copying 858GB data.

And just now, I checked, lights are still flashing and I tried to log
into the WebUI, as they machine had crashed. It won't let me login. I've
attached the results. If I click OK, it brings me back to the login
screen, waiting for password. And when I enter the password, the whole
process starts again.

Oh, now I'm sweating all over. Wow!

[notoneofmyseeds]

I managed to get in and this is what we see. Attached.
Is there another alternative? Please?

On 20.08.2015 18:38, João Cardoso wrote:

[João Cardoso]

On Thursday, 20 August 2015 20:42:50 UTC+1, notoneofmyseeds wrote:

I managed to get in and this is what we see. Attached.
Is there another alternative? Please?

The best way to copy/clone a disk or big filesystem is to use rsync, even if you have to use it several times, as it "restarts" the copy from the point where it was before. There are good tutorials available.

rsync can use a lot of memory when there are many files to deal with (100bytes per file, if I remember correctly), so swapping might occur and performance degrades. The box has only 64MB of memory,  so about 320000 files are +/- the limit for one rsync operation. Copy one sub-folder from your external disk filesystem at a time -- Turn a big insoluble problem into many manageable small problems -- that's engineering!

I have once researched this rsync issue on small memory machines and used the following script, but I have not tested it on the TB range and don't know if it's still working (odd file/dir names, e.g). And notice the script top comment: 15MB/sec, make your own math.

#!/bin/sh

# cp -a did 48MB/s

# rsync did 15MB/s

#set -x

# create NFILES lines files list to rsync

if test $# != 2; then

        echo rsync-split.sh source_dir dest_dir

        exit 1

fi

SDIR=$1/

DDIR=${2%%/}

echo $SDIR $DDIR

#OPTS="--stats -ahx --info=progress2" # --no-dirs

OPTS="-ahx --info=progress2" # --no-dirs

NFILES=100000 # 10MB memory + 10MB in memory file flist

#NFILES=10000 # 1MB memory

cd $SDIR

i=0; rm -f /tmp/foo

find . | while read -r t; do

 echo "$t" >> /tmp/foo

 i=$((i+1))

 if test $i = $NFILES; then

        wait

        i=0; mv /tmp/foo /tmp/bar

        rsync --files-from=/tmp/bar $OPTS $SDIR $DDIR &

 fi

done

if test -s /tmp/foo; then

        wait

        rsync --files-from=/tmp/foo $OPTS $SDIR $DDIR

fi

rm -f /tmp/foo /tmp/bar

[notoneofmy]

On 15-08-21 6:03 PM, João Cardoso wrote:
> The best way to copy/clone a disk or big filesystem is to use rsync, even
> if you have to use it several times, as it "restarts" the copy from the
> point where it was before. There are good tutorials available.
>
> rsync can use a lot of memory when there are many files to deal with
> (100bytes per file, if I remember correctly), so swapping might occur and
> performance degrades. The box has only 64MB of memory, so about 320000

> files are ± the limit for one rsync operation. Copy one sub-folder from

> your external disk filesystem at a time -- Turn a big insoluble problem
> into many manageable small problems -- that's engineering!
>
> I have once researched this rsync issue on small memory machines and used
> the following script, but I have not tested it on the TB range and don't
> know if it's still working (odd file/dir names, e.g). And notice the script
> top comment: 15MB/sec, make your own math.

Hi,

Thanks for this.

I was hoping for a simpler solution.

It looks like I will abandon this project, as I can't seem to find a
very good and simply solution.

It's very unfortunate that a package such as rsync does not have a GUI
that can connect to network and USB drives.

Thanks anyway.

Cheers