Move data from one DNS-323 to another using ftp or rsync (newbie with Linux)

[Mr. C]

Joao,

This is probably more of an education issue for me than a problem with Alt-f, but I have already spent a fair amount of time reading the posts here and googling without success.  If any other users can point me in the right direction, I will happily work through it so we can let Joao focus on this great project.

I have two DNS-323's (NASD12, NASD13), both have Alt-f installed (RC2), one is Rev B1, the second is Rev C1.  I successfully installed two 3TB drives in the Rev C1 (NASD13) (by command line sgdisk commands)  and I am now trying to move or copy data from one to the other. 

I would like to get rsync working eventually, but would be happy to use ftp now to get the data moved.

I am trying to run this by the command line, as I don't want to involve the windows machine (except for issuing the commands) and let the copy run over night.

So, I figured I would try ftp after reading that it would be fastest method.

"ftp" gives me "-sh: ftp" not found"

So I then figured I would try rsync:
"rsync -v -e root@NASD12:/mnt/Volume_1/NASD1/group.txt /Public/RW/TCF" gave me a response byt no file copy of
building file list ... done
drwxr-xr-x       4096 2013/02/04 23:36:34 TCF

sent 25 bytes  received 12 bytes  74.00 bytes/sec
total size is 0  speedup is 0.00


Any further attempts to use rsync (with command variations) give essentially the same error
 rsync -v  root@NASD12:/mnt/Volume_1/NASD1/group.txt /Public/RW/TCF
ssh: Exited: Error resolving 'NASD12' port '22'. Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]

So now I am lost as to what to try next.

As I say, I am a relative newbie to linux, but an old Dos command line guy, so I am comfortable with that.

Any help is greatly appreciated.

[Joao Cardoso]

On Tuesday, February 5, 2013 12:23:06 PM UTC, Mr. C wrote:

Joao,

This is probably more of an education issue for me than a problem with Alt-f, but I have already spent a fair amount of time reading the posts here and googling without success.  If any other users can point me in the right direction, I will happily work through it so we can let Joao focus on this great project.

I have two DNS-323's (NASD12, NASD13), both have Alt-f installed (RC2), one is Rev B1, the second is Rev C1.  I successfully installed two 3TB drives in the Rev C1 (NASD13) (by command line sgdisk commands)  and I am now trying to move or copy data from one to the other. 

I would like to get rsync working eventually, but would be happy to use ftp now to get the data moved.

I am trying to run this by the command line, as I don't want to involve the windows machine (except for issuing the commands) and let the copy run over night.

So, I figured I would try ftp after reading that it would be fastest method.

"ftp" gives me "-sh: ftp" not found"

 There is no ftp client in Alt-F, you can use 'wget' for non-interactive operations (wget even has a mirror option, that keeps dirs in sync for backup purposes)

So I then figured I would try rsync:
"rsync -v -e root@NASD12:/mnt/Volume_1/NASD1/group.txt /Public/RW/TCF" gave me a response byt no file copy of
building file list ... done
drwxr-xr-x       4096 2013/02/04 23:36:34 TCF

Remove the '-e'. And start with a reasonably sized file, to better understand the output

As the manual says:

       -e, --rsh=COMMAND

              This  option  allows  you  to choose an alternative remote shell program to use for

              communication between the local and remote copies of  rsync.  Typically,  rsync  is

              configured to use ssh by default, but you may prefer to use rsh on a local network.

              If this option is used with [user@]host::module/path, then the remote shell COMMAND

              will be used to run an rsync daemon on the remote host, and all data will be trans-

              mitted  through  that  remote shell connection, rather than through a direct socket

              connection to a running rsync daemon on the remote host.  See  the  section  "USING

              RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" above.

              Command-line  arguments are permitted in COMMAND provided that COMMAND is presented

              to rsync as a single argument.  You must use spaces (not tabs or other  whitespace)

              to  separate  the  command and args from each other, and you can use single- and/or

              double-quotes to preserve spaces in an argument (but not backslashes).   Note  that

              doubling  a  single-quote  inside  a single-quoted string gives you a single-quote;

              likewise for double-quotes (though you need to pay attention to which  quotes  your

              shell is parsing and which quotes rsync is parsing).  Some examples:

                  -e 'ssh -p 2234'

                  -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'

              (Note  that  ssh  users  can alternately customize site-specific connect options in

              their .ssh/config file.)

              You can also choose the remote shell program using the RSYNC_RSH environment  vari-

              able, which accepts the same range of values as -e.

              See also the --blocking-io option which is affected by this option.

sent 25 bytes  received 12 bytes  74.00 bytes/sec
total size is 0  speedup is 0.00


Any further attempts to use rsync (with command variations) give essentially the same error

For errors to make sense you have to include the commands that originate them.

 

 rsync -v  root@NASD12:/mnt/Volume_1/NASD1/group.txt /Public/RW/TCF
ssh: Exited: Error resolving 'NASD12' port '22'. Name or service not known

NASD12 is not defined in your /etc/hosts (on linux) or resolved by your nameserver.

If you do 'ping NASD12' you should have the same error.

[Mr. C]

Joao,


Thanks, did a bunch more reading on rsync, and played in the setting up of the services in alt-f.

Did get it running.  As I figured, it was me needing the education.

One last favour to ask.... as I am moving a couple hundred Gig of data - do you know how to have rsync continue running when I close down the Putty/Telnet session
(I tried nohup, but it did not work, and my reading leads me to believe trying to set it as a background task will fail as well)

Thanks for helping educate this linux newbie.

THANKS for Alt-f (and your patience with us weekend warriors).

Mr. C

[Christian Lobaugh]

best way to run long running commands that will persist beyond your current session (I use this all the time)

nohup command &

examples:

sudo nohup rsync -avz /mnt/sda2/files /mnt/sdb2/files &

nohup unrar e file.to.extract &

It's the combo of nohup and the & together that makes this work smoothly.  I'll let you read if you are interested in the details (many smarter than I am have explained this).

Hope this helps

Christian

--
You received this message because you are subscribed to the Google Groups "Alt-F" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alt-f+un...@googlegroups.com.
To post to this group, send email to al...@googlegroups.com.
Visit this group at http://groups.google.com/group/alt-f?hl=en.

To view this discussion on the web visit https://groups.google.com/d/msg/alt-f/-/wIiDDIVOLUoJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Joao Cardoso]

:-)

I did a fast test (TV showed nothing interesting) 

source: my desktop linux computer (silver)

destination: the box, on a RAID0/ext4 filesystem

commands issued in the box

Tranfering pictures:

# wget -nv -m --no-parent -nH -P destdir --user=myuser --password=mypass ftp://

silver/Pictures

...

Downloaded: 1543 files, 899M in 1m 15s (12.0 MB/s)

# time rsync -av jcard@silver:Pictures destdir

...

sent 27779 bytes  received 943078635 bytes  1544809.85 bytes/sec

real    10m 9.64s

Video clips:

ftp: Downloaded: 16 files, 178M in 13s (13.6 MB/s)

rsync: sent 281 bytes  received 186951388 bytes  1564449.11 bytes/sec

       real    1m 59.28s

Video:

ftp: Downloaded: 1 file, 887M in 1m 6s (13.5 MB/s)

rsync: sent 34 bytes  received 930098793 bytes  1429821.41 bytes/sec

       real    10m 49.33s

Thus rsync over ssh is roughly 8x slower than ftp.

This means that instead of a whole day transfering data using ftp,

a whole week would be needed when using rsync.

If someone able to use rsync over a plain sh and publish the results (and commands)?

Or using ftps and/or sftp? So my results are unbiased?

Joao 

PS-Please notice that the wget command is unnecessary complex, I just copied it from the Alt-F backup script.

[Mr. C]

Christian,

Thanks, I would love to give it a try, but I have not figured out how to use rsync without a userid/password. 
I did some playing with setting the /etc/rsyncd.conf file, but the changes did not survive a reboot.  Could easily be something I am doing wrong (still a newbie...)

I will keep reading and see what I can come up with....

[Mr. C]

Wow, what a speed difference.

I guess I need to know how to get ftp client installed in Alt-f.

With my limited Linux experience I was trying app-get and apt-get but they are "not found".

Sorry for my ignorance here, but how do I get an ftp client installed?
   - Do I need to install ffp first?
   - Do I need to know what distribution of Linux that Alt-f is based on and just download the appropriate .tar (or tgz, or...) file?

Thanks... yet again...

[Joao Cardoso]

On Thursday, February 7, 2013 3:07:14 AM UTC, Mr. C wrote:

Christian,

Thanks, I would love to give it a try, but I have not figured out how to use rsync without a userid/password. 

Read the Services->System->backup, Configure online help (?) icon, specially the last part:

In the rsync/ssh mode, password-less login using ssh cryptographic keys has to be manually setup.

If a .ssh/identity file is not found in the Run As home folder, ssh keys will be created for him and a file with instructions left in his home folder. After following the instructions the backup should proceed normally.

If host is silver, Folder is workspace and Run As is jcard, then the backup will be located in /Backup/silver_jcard_workspace/.

So, you can setup a backup of type rsync just to have the keys generated for you, together with the README instructions, even if you don't intend to use it. :-)

[Joao Cardoso]

On Thursday, February 7, 2013 3:15:44 AM UTC, Mr. C wrote:

Wow, what a speed difference.

I guess I need to know how to get ftp client installed in Alt-f.

You didn't read carefully enough:

source: my desktop linux computer (silver)

destination: the box, on a RAID0/ext4 filesystem

commands issued in the box

Tranfering pictures:

# wget -nv -m --no-parent -nH -P destdir --user=myuser --password=mypass ftp://

silver/Pictures

 

You don't need an ftp client in the nas, use 'wget', that comes in the base firmware.

With my limited Linux experience I was trying app-get and apt-get but they are "not found".

Sorry for my ignorance here, but how do I get an ftp client installed?
   - Do I need to install ffp first? 

   - Do I need to know what distribution of Linux that Alt-f is based

none, Alt-F is my own "distribution"  :-)

[Joao Cardoso]

[The following was classified as SPAM by Google Groups, and vanished after I allowed it.

This is becoming an habit, so in the future I will save myself all this typing and will only write :GG sucks!]

Mr. C said:

Trying wget ....

 wget -t3 -r 192.168.2.12:/mnt/Volume_1/NASD1/IBML/*

--2013-02-05 16:01:14--  ftp://192.168.2.12//mnt/Volume_1/NASD1/IBML/*

           => `192.168.2.12/mnt/Volume_1/NASD1/IBML/.listing'

Connecting to 192.168.2.12:21... connected.

Logging in as anonymous ...

You could do wget -t3 -r root@192.168.2.12:/mnt/Volume_1/NASD1/IBML ( remove the '/*', the -r(ecursive) option does that)

Also, I hope that you don't have html files there, as '-r' means that any external references will be followed. 

 

The server refuses login.  

So, what/where do I set some access controls on to allow access?

Services->Network->inetd Configure->ftp 

Remember, I read just enough to be dangerous (on the command line).

Rant away, cause I have been doing that against ftp, wget, rsync, ....

(all very quietly, as I KNOW my education is lacking).

But be sure you know that I am not ranting against Alt-f, You have

restored the life in these boxes and I REALLY appreciate being able to

work at this to get 6TB of storage!

[Mr. C]

On Thursday, February 7, 2013 12:04:52 AM UTC-5, Joao Cardoso wrote:

On Thursday, February 7, 2013 3:15:44 AM UTC, Mr. C wrote:

Wow, what a speed difference.

I guess I need to know how to get ftp client installed in Alt-f.

You didn't read carefully enough:

source: my desktop linux computer (silver)

destination: the box, on a RAID0/ext4 filesystem

commands issued in the box

Tranfering pictures:

# wget -nv -m --no-parent -nH -P destdir --user=myuser --password=mypass ftp://

silver/Pictures

 

Joao,

OK, I have been reading and trying...  On NASD12 I moved my data to the /Public/RW/VM_Backups directory, and have the /Public/RW directory set up for Anonymous ftp in the

vsftpd server section.  My wget command  (I removed the -nv as I want it to be verbose so I can "TRY" to see what is going on)

wget -m --no-parent -nH -P /Public/RW/projects ftp://NASD12/VM_Backups

From my NASD12 box I issue the above command and get lots of info but here is what I assume is important... It Resolves, connects, logs in, creates it's list, downloads the list file, then moves over to the ftp information does a SIZE VM_Backups, PASV, RETR VM_Backups  but gets No Such File.

Essentially the same if I put a slash on the end of the VM_Backups, just a "No Such Directory" message

So I next try putting the full path in wget -m --no-parent -nH -P /Public/RW/projects ftp://NASD12/mnt/Volume_1/Public/RW/VM_Backups  (With and Without the slash at the end) and get the same "No Such Directory" and just changes what directory it is referring to.

I know that this is more "education" for me, but I am not sure what else I can try to help me learn.

Any other kind linux folks care to "hold my hand" through this?

THANKS

[Joao Cardoso]

On Thursday, February 7, 2013 9:04:55 PM UTC, Mr. C wrote:

On Thursday, February 7, 2013 12:04:52 AM UTC-5, Joao Cardoso wrote:

On Thursday, February 7, 2013 3:15:44 AM UTC, Mr. C wrote:

Wow, what a speed difference.

I guess I need to know how to get ftp client installed in Alt-f.

You didn't read carefully enough:

source: my desktop linux computer (silver)

destination: the box, on a RAID0/ext4 filesystem

commands issued in the box

Tranfering pictures:

# wget -nv -m --no-parent -nH -P destdir --user=myuser --password=mypass ftp://

silver/Pictures

 

Joao,

OK, I have been reading and trying...  On NASD12 I moved my data to the /Public/RW/VM_Backups directory, and have the /Public/RW directory set up for Anonymous ftp in the

vsftpd server section.  My wget command  (I removed the -nv as I want it to be verbose so I can "TRY" to see what is going on)

wget -m --no-parent -nH -P /Public/RW/projects ftp://NASD12/VM_Backups

Everything looks OK. 

In my example I miss to say that I used --user=jcard and that Pictures is a folder right under jcard home folder.

When you login as the anonymous ftp user, you should be in the "Anonymous folder:" specified in the web page.

There can exists permission problems, it depends on how you move the folders (it would be simpler to just redefine the "Anonymous folder:" ...

If you login as user jdoe, "John Doe", you will be in "/home/John Doe" directory, user 'root' under "/root", etc. You might not be able to use "cd .." to go to the upper folder, it depends on how "Restrict folders" is checked or not (there are some issues with this).

I can only recommend you to use a client line ftp client from any desktop computer, login as the ftp or anonymous or guest user, depends on the client, but ultimately it will use 'ftp', and after login try using the 'dir', 'cd' and 'get' commands, to see if they work as expected. Use 'help get", etc to get a short half a line help on the commands.

Then use wget.

But this is not Alt-F related, I'm sure you will find good tutorials out there. You are in the right track now, follow it.

[Mr. C]

Thank You Joao.

Your idea of using another system to test the ftp led me to the issue...

VSFTPD server will not allow an anonymous connection to a "root" directory that is Writeable.

I had to change the "Anonymous Directory" to one that was NOT Writeable, and then the ftp was able to connect.

Then I was able to issue my wget command from the command line and it is running!!!

THANKS
for Alt-F
for your Patience
for your alternate testing ideas.

[Joao Cardoso]

Your idea of using another system to test the ftp led me to the issue...

VSFTPD server will not allow an anonymous connection to a "root" directory that is Writeable.

I had to change the "Anonymous Directory" to one that was NOT Writeable, and then the ftp was able to connect.

Yes, this is know issue, already discussed here.

Solved? please mark the topic as complete.

Please notice that my suggestion of using ftp instead of rsync is only applicable to the throughput issue, ftp does not replaces rsync, they where conceived for different purposes.

If you want to keep folders in sync after this initial download, then you must probably use rsync (although wget has a mirror option that only downloads new or modified files, as rsync does ((but rsync only downloads the part of the file that has changed, not the whole file!)))

[Mr. C]

On Friday, February 8, 2013 10:44:21 AM UTC-5, Joao Cardoso wrote:

Your idea of using another system to test the ftp led me to the issue...

VSFTPD server will not allow an anonymous connection to a "root" directory that is Writeable.

I had to change the "Anonymous Directory" to one that was NOT Writeable, and then the ftp was able to connect.

Yes, this is know issue, already discussed here.

WHERE?  What topic?  Not trying to be a pain, but as a Linux newbie, If I don't know the terms, I don't know what to search on, so I can't find that information.

Solved? please mark the topic as complete.

Please notice that my suggestion of using ftp instead of rsync is only applicable to the throughput issue, ftp does not replaces rsync, they where conceived for different purposes.

If you want to keep folders in sync after this initial download, then you must probably use rsync (although wget has a mirror option that only downloads new or modified files, as rsync does ((but rsync only downloads the part of the file that has changed, not the whole file!)))

Understood about rsync vs ftp/wget.

Will mark as complete.

Thanks