SCP to copy from one NAS to another

[JimBobUK]

Hi,

Wondered if you could help

I am using the following to copy files from one nas to another:

scp -r root @ 192.168.2.115:/mnt/sdb2/Movies/ /mnt/sdb2/

Everything works fine but its slow to copy. Looking around i see I can use -c switch but when I try -c arcfour or -c blowfish it ignores -c...... in the command

Am I doing this wrong?

TIA

[JimBobUKII]

Sorry for double post. I didn't think the first post went through. 

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Alt-F" group.
To post to this group, send email to al...@googlegroups.com.
To unsubscribe from this group, send email to alt-f+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/alt-f?hl=en.
To view this discussion on the web visit https://groups.google.com/d/msg/alt-f/-/AhJvpshxYeYJ.
 
 

[JimBobUKII]

Anyone help me with this? 

Sent from my iPhone

[JimBobUKII]

Hmmmm still no luck. I have tried -c arcfour and -c blowfish but it doesn't understand that part if the command. It is either the version if Linux doesn't know what that switch is or it's me being very dim :)

Sent from my iPhone

[Joao Cardoso]

On Monday, January 21, 2013 7:26:00 PM UTC, JimBobUK wrote:

Hi,

Wondered if you could help

I am using the following to copy files from one nas to another:

scp -r root @ 192.168.2.115:/mnt/sdb2/Movies/ /mnt/sdb2/

Everything works fine but its slow to copy.

What is "slow"? You have to quantify.

A quick and dirty test shows:

Writing to the DNS232 (RAID)  using scp I get about 2MBs, using ftp I get 13MBs, 6.5 time faster

Writing to a DNS325 (standard) using scp I get about  7.5MBs, using ftp I get 26MBs, 3.5 times faster

When using ftp the DNS325 is about 2 times faster then the DNS323, when using scp it is about 3.5 times faster.

scp uses ssh, which uses encryption, which is very slow when no FPU  (floating point hardware) or crypto hardware is available.

Neither the 323 nor 325 has a FPU, both has a hardware crypto accelerator.

The DNS325 is using the vendors firmware, which probably has the crypto hardware working, while the Alt-F kernel has no crypto hardware enabled by default.

Enabling the crypto hardware on Alt-F had mixed results. I tried to use it when working on the cryptsetup package, and a 2 fold performance was obtained encrypting devices when using the crypto accelarator, (ssh also had improvements, don't remember the numbers) but some crypto algorithms returned errors, so I giveup using it -- its still an option). Search on the forum for crypto or cryptsetup or similar.

NAS    ftp scp

DNS323 13  2

DNS325 26  7.5

Do you need an encrypted channel? Can you try using ftps or sftp and publish the results, just for future reference?

Remember that not all ftps/sftp clients work fine with vsftpd, and that vsftpd uses the sftp-server from the openssh package (even if you have not installed the openssh package)

 

Looking around i see I can use -c switch but when I try -c arcfour or -c blowfish it ignores -c...... in the command

Remember that Alt-F is using dropbear as the ssh server. You can install openssh instead, but then you might have to fine tune some things.

 

Am I doing this wrong?

Got the wrong NAS? :-)

 

TIA

[JimBobUK]

Me again :)

 

All I want to do is copy files from one NAS to the other, both are intenal without intenet access so security for copying not an issue.

I would like to know the best way to copy the files from one to the other. I am moving files from one NAS that has 2TB HDD and want to move them to another NAS with 3TB HDD in

If you could explain the easiest and fastest way to copy the files that would be great

 

TIA

[nesta...@gmail.com]

On Wednesday, January 23, 2013 3:18:37 AM UTC-7, JimBobUK wrote:

Me again :)

 

All I want to do is copy files from one NAS to the other, both are intenal without intenet access so security for copying not an issue.

I would like to know the best way to copy the files from one to the other. I am moving files from one NAS that has 2TB HDD and want to move them to another NAS with 3TB HDD in

If you could explain the easiest and fastest way to copy the files that would be great

 

TIA

 

He did explain how you could do it. And if you don't have the NAS he mentioned, then perhaps you can't. Try using ftp. I had similar problems, copying hundreds of gigagbytes of data and it was painfully slow. I did not see the measurements as I copied using windows explorer. I will try ftp and see how fast that goes. It looks like the DNS325 is the darling.

....I just tried copy a video_ts folder using Filezilla; on both top and bottom files being copied, I'm getting 4.7 and 4.6 respectively. Of course this changes from time to time. Later I will try a single file, like an avi file, and see. But from the eyes, this seems way faster than what I was doing in explorer.

[Joao Cardoso]

On Wednesday, January 23, 2013 10:18:37 AM UTC, JimBobUK wrote:

Me again :)

 

All I want to do is copy files from one NAS to the other, both are intenal without intenet access so security for copying not an issue.

I would like to know the best way to copy the files from one to the other. I am moving files from one NAS that has 2TB HDD and want to move them to another NAS with 3TB HDD in

If you could explain the easiest and fastest way to copy the files that would be great

You have to give *all* details, we don't know your setup and don't have telepathic capabilities :-)

I don't know what firmware the NAS are running, what its capabilities are... does both have Alt-F? do you want to copy from a Alt-F powered NAS to another Alt-F powered NAS? Or only one of them has Alt-F? Which one? 

If security is of no concern, neither at the network level nor at the computer where you execute the commands (in both the nas root password can be "watched"), and if the data itself is not sensitive, than ftp is the fastest protocol. 

Anyway, *assuming* that the source NAS has ftp running and that you can ftp it as the 'root' user (is that really necessary?) with 'rootpasswd', try the following in the destination NAS

 wget -nv -m -nH --no-parent -P destinanation_folder --cut-dirs=1 ftp://root:rootpasswd@source_nas_ip/../origin_folder

e.g.

wget -nv -m -nH --no-parent -P /tmp --cut-dirs=1 ftp://root:xp...@192.168.2.115/../mnt/sdb2/Movies

will recursively copy all /mnt/sdb2/Movies contents from the 'nas' source to the /tmp folder, at the end you will find under /tmp/mnt/sdb4/Movies everything

This assumes that the source nas has Alt-F running, and that after you ftp it as root you will find yourself in the '/root' folder (that is the reason for the '/../' and the --cut-dirs=1 option)

But this is not really Alt-F related, you are now in the right track, follow it.

[Christian Lobaugh]

I'm personally a big fan of rsync for situations like this, mainly because it is good at keeping permissions, links (even hard-links), etc, and as a bonus, it will allow you to pick up where you left off if the copy is interrupted.

I'm not sure where rsync stands in the speed department.  By default it uses encryption, but it can be setup to go without.  I know there is an option to have it compress things to lower bandwidth, but this does not necessarily mean faster transfer as you have to take into consideration CPU usage, etc.  My guess is that it will not be as fast as ftp/wget on the DNS machines, but it should run significantly faster than scp if you use it without encryption, and offers some things that ftp does not.

If you just want to move the files and don't care about the other stuff, then ftp/wget will be your fastest option. 

--

You received this message because you are subscribed to the Google Groups "Alt-F" group.
To post to this group, send email to al...@googlegroups.com.
To unsubscribe from this group, send email to alt-f+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/alt-f?hl=en.

To view this discussion on the web visit https://groups.google.com/d/msg/alt-f/-/bmXv9dluhC8J.
 
 

[Joao Cardoso]

On Wednesday, January 23, 2013 4:09:50 PM UTC, Christian Lobaugh wrote:

I'm personally a big fan of rsync for situations like this, mainly because it is good at keeping permissions, links (even hard-links), etc, and as a bonus, it will allow you to pick up where you left off if the copy is interrupted.

I'm not sure where rsync stands in the speed department.

I agree, rsync is good to keep things in sync, but it is not fast.

Because of that slowness, Alt-F backup script does a plain copy for the first backup, and then for the next backups it uses rsync.

When the source is not local, it mounts the source locally for NFS/CIFS or updates the mirror copy for FTP/HTTP, then proceeds as in above.

Much faster then doing a rsync over machines (for the case of our small box)

 

  By default it uses encryption, but it can be setup to go without.

How can that be accomplished? Please give a cmdline example for Alt-F. I was not able to use the --rsh option...

As a matter of fact I'm not particularly fond of rsync... Its only advantage, in my opinion, is the way how it keeps files in sync, by transmitting only the (compressed) differences between them; this shows that it was conceived for low bandwidth situations and no consideration for low power computers (the delta-xfer algorithm). And the default way of establishing that files are in sync, using its length, is... well...

And it is helpless for a real life resync, say between your two or three laptops/computers, when you might have doing changes in all of them and don't remember which.

For this situation, 'unison' is a winner, with multiparty syncing (but unison is now too old and not maintained anymore I think)

[Christian Lobaugh]

On Wed, Jan 23, 2013 at 12:17 PM, Joao Cardoso <whoami...@gmail.com> wrote:

On Wednesday, January 23, 2013 4:09:50 PM UTC, Christian Lobaugh wrote:

I'm personally a big fan of rsync for situations like this, mainly because it is good at keeping permissions, links (even hard-links), etc, and as a bonus, it will allow you to pick up where you left off if the copy is interrupted.

I'm not sure where rsync stands in the speed department.

I agree, rsync is good to keep things in sync, but it is not fast.

Because of that slowness, Alt-F backup script does a plain copy for the first backup, and then for the next backups it uses rsync.

When the source is not local, it mounts the source locally for NFS/CIFS or updates the mirror copy for FTP/HTTP, then proceeds as in above.

Much faster then doing a rsync over machines (for the case of our small box)

 

  By default it uses encryption, but it can be setup to go without.

How can that be accomplished? Please give a cmdline example for Alt-F. I was not able to use the --rsh option...

I've never done it, but my understanding is that you use the rsync daemon for this.  I saw it mentioned while I was reading about rsync, but I skipped over the details, so I'm not much help to you on this. 
 

As a matter of fact I'm not particularly fond of rsync... Its only advantage, in my opinion, is the way how it keeps files in sync, by transmitting only the (compressed) differences between them; this shows that it was conceived for low bandwidth situations and no consideration for low power computers (the delta-xfer algorithm). And the default way of establishing that files are in sync, using its length, is... well...

And it is helpless for a real life resync, say between your two or three laptops/computers, when you might have doing changes in all of them and don't remember which.

For this situation, 'unison' is a winner, with multiparty syncing (but unison is now too old and not maintained anymore I think)

I don't use rsync for any synching beyond basic backups.  I've never really dug into it's strengths and weaknesses in that area, although I do know it's got a lot of fans.

The main advantage for me with rsync is it's copy/move functionality, and how it can preserve extended attributes and hard links (which I use a lot).  This is a critical need for me when I'm copying/moving files to a different partition (local or remote) and I'm willing to sacrifice speed for this.  I don't use it a lot for this, but when I need it, it does the job well, and comes built in to just about every linux distro.
 

  I know there is an option to have it compress things to lower bandwidth, but this does not necessarily mean faster transfer as you have to take into consideration CPU usage, etc.  My guess is that it will not be as fast as ftp/wget on the DNS machines, but it should run significantly faster than scp if you use it without encryption, and offers some things that ftp does not.

If you just want to move the files and don't care about the other stuff, then ftp/wget will be your fastest option. 

--
You received this message because you are subscribed to the Google Groups "Alt-F" group.
To post to this group, send email to al...@googlegroups.com.
To unsubscribe from this group, send email to alt-f+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/alt-f?hl=en.

To view this discussion on the web visit https://groups.google.com/d/msg/alt-f/-/0tczbQXeQCMJ.