scp UNBELIEVABLY slow
Ignoramus16148
laptop to my basement server. Wired network. HTTP test yields about 6
megabytes per second transfer speed.
SCP is only doing 188 kB per second, or 36 times slower.
I am copying from Ubuntu Hardy to Fedora 7.
WTF, how can I debug this. Thanks.
i
AZ Nomad
do a google search for "scp performance"?
Use NFS instead of SCP?
Ignoramus16148
I want to use SCP. It is a bad security to se NFS in this manner.
i
NoStop
Bad security on YOUR local LAN?
Cheers.
--
The world can't afford the rich.
Q: What OS is built for lusers?
A: Which one requires running lusermgr.msc to create them?
Francis (Frank) adds a new "gadget" to his Vista box ...
Download it here: http://tinyurl.com/2hnof6
Ignoramus16148
Even if someone was downloading torrents through my wifi, it would
only account for a few megabits. So while your idea is interesting, I
do not think that it could account for it.
However, plot thickened.
My laptop was connected to the basement server through three gigabit
switches A, B, C. That's where I had this terrible performance.
I moved it to the basement where it is connected through switches C
and D (D is a 100 megabit switch).
Here, the performance improved dramatically to 3.9 megaBYTES per
second.
Go figure.
Why SCP was 36 times slower than HTTP going through A, B, C but not C
and D, is a mystery.
i
Frank
Get a real OS.
Switch to Vista.
urbuttoo is a fukkin toy POS os.
Frank
johnny bobby bee
> I am on a good network here. I need to copy a bunch of AVI from a
> laptop to my basement server. Wired network. HTTP test yields about 6
> megabytes per second transfer speed.
>
> SCP is only doing 188 kB per second, or 36 times slower.
HTTP test? Maybe you meant 6Mbps (megabits not megabytes).
Have you looked into SSHFS? It mounts a remote directory locally using
SSH and fuse.
http://elwoodicious.com/2006/07/05/sshfs-ubuntu-and-you/
I get about 12-15 MB/sec (~100Mbps).
--
As we enjoy great advantages from inventions of others, we should be
glad of an opportunity to serve others by any invention of ours;
and this we should do freely and generously.
--Benjamin Franklin
Jon Solberg
But Ubuntu users are able to spell beyond pre school level.
Ignoramus6291
> Ignoramus16148 wrote:
>> I am on a good network here. I need to copy a bunch of AVI from a
>> laptop to my basement server. Wired network. HTTP test yields about 6
>> megabytes per second transfer speed.
>>
>> SCP is only doing 188 kB per second, or 36 times slower.
>
> HTTP test? Maybe you meant 6Mbps (megabits not megabytes).
No, HTTP test was about 6 megabytes per second, and SCP test was 180
kilobytes per second. No typo. However, see another post titled "plot
thickens".
i
Mark Crowder
"Ignoramus16148" <ignoram...@NOSPAM.16148.invalid> wrote in message
news:EqWdnYILIYOE-HXa...@giganews.com...
if you don't get it sorted out, I've had pretty good luck with:
rsync -e ssh ...
there's some documentation referenced from:
http://rsync.samba.org/documentation.html
regards,
mark
Walter Mautner
....
> urbuttoo is a fukkin toy POS os.
Whose butts (oo) are a Point Of Sales toy?
--
vista policy violation: Microsoft optical mouse found penguin patterns
on mousepad. Partition scan in progress to remove offending
incompatible products. Reactivate MS software.
Linux 2.6.24. [LinuxCounter#295241,ICQ#4918962]
Moog
> Frank wrote:
>
> ....
>> urbuttoo is a fukkin toy POS os.
>
> Whose butts (oo) are a Point Of Sales toy?
Walter. I can't believe you're getting embroiled with this obvious
troll.
He's a divot. Leave him and his braincell to "lol" and invent *funny*
names for stuff.
He's a complete and utter waste of good skin.
--
Moog
"If this is gonna be that kinda party I'm gonna stick my dick in the
mashed potatoes"
AZ Nomad
BWAHAHAHAHAHAHAHAHA! With vista, you will be told that you shouldn't be
copying the files and that the authorities have been alerted.
AZ Nomad
>Ignoramus16148 wrote:
>> I am on a good network here. I need to copy a bunch of AVI from a
>> laptop to my basement server. Wired network. HTTP test yields about 6
>> megabytes per second transfer speed.
>>
>> SCP is only doing 188 kB per second, or 36 times slower.
>HTTP test? Maybe you meant 6Mbps (megabits not megabytes).
>Have you looked into SSHFS? It mounts a remote directory locally using
>SSH and fuse.
>http://elwoodicious.com/2006/07/05/sshfs-ubuntu-and-you/
>I get about 12-15 MB/sec (~100Mbps).
Why bother with the overhead of SSH on a home network? I get 50-70MB/s (bytes,
not bits) via NFS to my file server and that is without large packets as I
have a mixed 100/1000 network.
Cork Soaker
Dick.
Frank
Unruh
I think he wants to know why scp is so slow, not how he can get around it.
Is ssh slow-- EG use rsync over ssh. Is it also slow?
AZ Nomad
Yes. The encryption overhead makes anything over ssh much slower than
other protocols.
Balwinder S Dheeman
I'm sorry, Frank could not get your message due to filters, but would
like to bring to you notice a never answered questions here again --
Have you ever looked what C:\WINDOWS contains? if yes, can you tell us
how many people know what's useful and what you can discard and, or get
rid off? Why that shit Windows and even Vista stores the temporary files
in system directory? Why the MicroShit is controlling your machines and
money?
Seriously, Windows and, or Vista are not a silver bullet either, neither
they are clean machines or real operating systems and that's why other
operating systems still exist and, or are being maintained.
>> urbuttoo is a fukkin toy POS os.
>> Frank
--
Dr Balwinder S "bsd" Dheeman Registered Linux User: #229709
Anu'z Linux@HOME (Unix Shoppe) Machines: #168573, 170593, 259192
Chandigarh, UT, 160062, India Gentoo, Fedora, Debian/FreeBSD/XP
Home: http://cto.homelinux.net/~bsd/ Visit: http://counter.li.org/
Ignoramus6291
What I found is that if I use a chain of switches A, B, C I get
180 kbytes/sec throughput with ssh. If I use switches C, D, I get 3.9
MB/second (over 20x faster).
However, for A,B,C as well as for C, D the HTTP speed is roughly the
same IIRC.
i
Ignoramus6291
and speed of NFS over A, B and C is 4.5 megabytes per second.
i
johnny bobby bee
> Yes. The encryption overhead makes anything over ssh much slower than
> other protocols.
Much slower, that's a bit of a stretch.
As I said, I use SSHFS and i get 100Mbps on a plain 10/100 Mb network.
Andrew Gideon
> and speed of NFS over A, B and C is 4.5 megabytes per second.
I wonder if the packets used by SSH are larger than those used by NFS. I
vaguely recall some NFS setting which by default keeps packets small.
But I could be easily misremembering.
Is there anything else involved that might alter packet sizes? For
example, are you using some type of packet tagging (ie. 802.1q)?
- Andrew
Ignoramus13009
Not intentionally. I set MTU to 1492 manually and it did not help.
i
Andrew Gideon
> Not intentionally. I set MTU to 1492 manually and it did not help.
But did you try playing with the rsize and wsize options for NFS?
Also, did you try a smaller MTU? Perhaps that would help performance
through switch A for some reason.
Is your NFS using TCP or UDP? I cannot imagine why it would matter, but
if it is using UDP perhaps that is the significant difference.
- Andrew
Ignoramus13009
>
>> Not intentionally. I set MTU to 1492 manually and it did not help.
>
> But did you try playing with the rsize and wsize options for NFS?
No. But keep in mind, NFS is fine mostly (though not near the gigabit
speeds). The issue is with scp.
> Also, did you try a smaller MTU? Perhaps that would help performance
> through switch A for some reason.
>
> Is your NFS using TCP or UDP? I cannot imagine why it would matter, but
> if it is using UDP perhaps that is the significant difference.
i
Unruh
>AZ Nomad wrote:
>> Yes. The encryption overhead makes anything over ssh much slower than
>> other protocols.
>Much slower, that's a bit of a stretch.
>As I said, I use SSHFS and i get 100Mbps on a plain 10/100 Mb network.
Well depends on your machine. If he is running an old IBM 8086 machine, I
am sure what he says is true-- that ssh adds appreciably to the time.
(Mind you he sure would not get even 10Mb/s without encryption)
Ignoramus13009
Speed is not the issue. Something about ssh/scp and a unknown network
problem that shows up over ssh, is the issue. For example, here's the
output of scp speed (31 megabytes per second) and HTTP speed (112
MB/Sec) going between routers B and C (but not A). The other server is
not my laptop.
Here SCP is 3 times slower than HTTP, which does not surprise me due
to encryption.
But between laptop, A, B, C, and my server, the speed diff. was 36
times. The laptop was able to go a lot faster without routers A and B.
So I am beginning to place blame on router A. Something is wrong with
it when I do scp.
i
Tauno Voipio
>
>
> What I found is that if I use a chain of switches A, B, C I get
> 180 kbytes/sec throughput with ssh. If I use switches C, D, I get 3.9
> MB/second (over 20x faster).
>
> However, for A,B,C as well as for C, D the HTTP speed is roughly the
> same IIRC.
>
> i
Check that there are no duplex mismatches. A port erroneously
forced full-duplex can slow down an Ethernet link drastically
due to unnecessary collisions generated.
The protocols have different needs for return traffic, which
will be the victim in duplex mismatch.
--
Tauno Voipio
tauno voipio (at) iki fi
Andrew Gideon
>> But did you try playing with the rsize and wsize options for NFS?
>
> No. But keep in mind, NFS is fine mostly (though not near the gigabit
> speeds). The issue is with scp.
But you're looking for the difference. If NFS defaults to smaller packet
sizes, that might be the difference.
- Andrew
Andrew Gideon
> So I am beginning to place blame on router A. Something is wrong with it
> when I do scp.
Is it a router or a switch? If it's a router, another difference is the
network connected on the "far" side of A.
How "smart" is A? Might it have rate limiting on SSH about which you
don't know (or have forgotten)?
- Andrew
Ignoramus13009
>
>> So I am beginning to place blame on router A. Something is wrong with it
>> when I do scp.
>
> Is it a router or a switch? If it's a router, another difference is the
> network connected on the "far" side of A.
It is a DIR-655 Wifi router. However, I used a Ethernet connection
between laptop and DIR-655 and not wifi.
> How "smart" is A? Might it have rate limiting on SSH about which you
> don't know (or have forgotten)?
I would be surprised, but this is already surprising.
i
Ignoramus13009
Tauno, your explanation so far is the only one that is consistent with
all available information.
I will try scp again, then will switch to half duplex, and see if that
makes any difference.
i
Dan Espen
If you suspect collisions, the easiest way to see them
is with the ifconfig command.
"Normal" for collisions is zero.
Ignoramus13009
I will check tonight Dan. Did not know that it was so easy. Collisions
may be between the laptop and A, or between A and B (I suspect the
latter). The second case is not so easy.
i
AZ Nomad
>On 2008-03-26, Tauno Voipio <tauno....@INVALIDiki.fi> wrote:
>> Ignoramus6291 wrote:
>>>
>>>
>>> What I found is that if I use a chain of switches A, B, C I get
>>> 180 kbytes/sec throughput with ssh. If I use switches C, D, I get 3.9
>>> MB/second (over 20x faster).
>>>
>>> However, for A,B,C as well as for C, D the HTTP speed is roughly the
>>> same IIRC.
>>>
>>> i
>>
>>
>> Check that there are no duplex mismatches. A port erroneously
>> forced full-duplex can slow down an Ethernet link drastically
>> due to unnecessary collisions generated.
>>
>> The protocols have different needs for return traffic, which
>> will be the victim in duplex mismatch.
>>
>Tauno, your explanation so far is the only one that is consistent with
>all available information.
His explanation is only valid if a network hub is used and I haven't seen
one sold in ten years. Network switches buffer the traffic and collisions
are impossible.
Darren Dunham
>> I will try scp again, then will switch to half duplex, and see if that
>> makes any difference.
>
> If you suspect collisions, the easiest way to see them
> is with the ifconfig command.
> "Normal" for collisions is zero.
Collisions cannot occur on a full duplex interface. By definition, it
will be zero. However if the other end is set to half-duplex, you
should see errors on the interface (almost certainly receive errors).
--
Darren Dunham ddu...@taos.com
Senior Technical Consultant TAOS http://www.taos.com/
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >
Dances With Crows
Darren Dunham staggered into the Black Sun and said:
> In comp.security.ssh Dan Espen wrote:
>>> I will try scp again, then will switch to half duplex, and see if
>>> that makes any difference.
>> If you suspect collisions, the easiest way to see them is with the
>> ifconfig command. "Normal" for collisions is zero.
> Collisions cannot occur on a full duplex interface.
Um... broadcast packets can collide. This is relatively rare, but it
does happen. 3F89CAF3...@house-from-hell.demon.co.uk said 62
million packets, 1 collision.
> However if the other end is set to half-duplex, you should see errors
> on the interface (almost certainly receive errors).
Yes. Other posts from the OP make me think the problem is related to
having multiple 802.11 routers involved in the chain, rather than
something wrong with the protocol being used or duplex settings. The OP
didn't ever reply to my post about using scp -v and checking the output
for anything odd, either.
--
I will rule you all with my iron fist. YOU! Obey the fist!
--Invader Zim
My blog and resume: http://crow202.dyndns.org:8080/wordpress/
Matt G|There is no Darkness in Eternity/But only Light too dim for us to see
Darren Dunham
> Darren Dunham staggered into the Black Sun and said:
>> Collisions cannot occur on a full duplex interface.
>
> Um... broadcast packets can collide. This is relatively rare, but it
> does happen. 3F89CAF3...@house-from-hell.demon.co.uk said 62
> million packets, 1 collision.
What do you mean by a "broadcast packet"? At the media level (where
collisions occur), there's no such distinction that I can think of. All
packets are seen by all recipients on the link.
But regardless of the actual topology, an interface that is configured
for full-duplex should ignore or disable collision detection. Traffic
cannot cause a collision to be reported to the driver. If it is, it's
either not really in full-duplex mode or it's a bug.
> Other posts from the OP make me think the problem is related to having
> multiple 802.11 routers involved in the chain, rather than something
> wrong with the protocol being used or duplex settings. The OP didn't
> ever reply to my post about using scp -v and checking the output for
> anything odd, either.
I definitely agree that the behavior seems to be more involved with the
path chain than a simple duplex mismatch.
Chris Mattern
>
> Yes. The encryption overhead makes anything over ssh much slower than
> other protocols.
Depends on the encryption you use. Try Blowfish.
--
Christopher Mattern
NOTICE
Thank you for noticing this new notice
Your noticing it has been noted
And will be reported to the authorities
John Thompson
> Ignoramus16148 wrote:
>
>> I am on a good network here. I need to copy a bunch of AVI from a
>> laptop to my basement server. Wired network. HTTP test yields about 6
>> megabytes per second transfer speed.
>>
>> SCP is only doing 188 kB per second, or 36 times slower.
>>
>>
>> WTF, how can I debug this. Thanks.
> Get a real OS.
> Switch to Vista.
AFAIK, Vista doesn't include scp or any other secure file transfer
software at all.
--
John (jo...@os2.dhs.org)
Hadron
Nonsense. You can install rsync on Windows too.
As well as ssh and scp and .....
--
< doogie> asuffield: how do you think dpkg was originally written? :|
< asuffield> by letting iwj get dangerously near a computer
-- in #debian-devel
NoStop
> John Thompson <jo...@vector.os2.dhs.org> writes:
>
>> ["Followup-To:" header set to comp.os.linux.misc.]
>> On 2008-03-25, Frank <f...@tri.olk> wrote:
>>> Ignoramus16148 wrote:
>>>
>>>> I am on a good network here. I need to copy a bunch of AVI from a
>>>> laptop to my basement server. Wired network. HTTP test yields about 6
>>>> megabytes per second transfer speed.
>>>>
>>>> SCP is only doing 188 kB per second, or 36 times slower.
>>>>
>>>> I am copying from Ubuntu Hardy to Fedora 7.
>>>>
>>>> WTF, how can I debug this. Thanks.
>>
>>> Get a real OS.
>>> Switch to Vista.
>>
>> AFAIK, Vista doesn't include scp or any other secure file transfer
>> software at all.
>
> Nonsense. You can install rsync on Windows too.
>
> As well as ssh and scp and .....
>
I can imagine YOU doing that. Anyone with half a brain would just stick with
Linux.
Cheers.
--
The world can't afford the rich.
Q: What OS is built for lusers?
A: Which one requires running lusermgr.msc to create them?
Francis (Frank) adds a new "gadget" to his Vista box ...
Download it here: http://tinyurl.com/2hnof6
Ignoramus10476
> Hadron wrote:
>
>> John Thompson <jo...@vector.os2.dhs.org> writes:
>>
>>> ["Followup-To:" header set to comp.os.linux.misc.]
>>> On 2008-03-25, Frank <f...@tri.olk> wrote:
>>>> Ignoramus16148 wrote:
>>>>
>>>>> I am on a good network here. I need to copy a bunch of AVI from a
>>>>> laptop to my basement server. Wired network. HTTP test yields about 6
>>>>> megabytes per second transfer speed.
>>>>>
>>>>> SCP is only doing 188 kB per second, or 36 times slower.
>>>>>
>>>>> I am copying from Ubuntu Hardy to Fedora 7.
>>>>>
>>>>> WTF, how can I debug this. Thanks.
>>>
>>>> Get a real OS.
>>>> Switch to Vista.
>>>
>>> AFAIK, Vista doesn't include scp or any other secure file transfer
>>> software at all.
>>
>> Nonsense. You can install rsync on Windows too.
>>
>> As well as ssh and scp and .....
>>
> I can imagine YOU doing that. Anyone with half a brain would just stick with
> Linux.
On the Windows XP computer that I use at work, I run X windows, the
whole works of xterms and binutils, perl, cron, and firefox.
That stuff actually works, but not quite as well as under Linux. For
example, you cannot access mounted network drives from cron.
i
Roman Ratnaweera
> I can imagine YOU doing that. Anyone with half a brain would just stick with
> Linux.
>
> Cheers.
>
While I agree about sticking to Linux, perhaps we'll leave out the heat
when conveying that to poor Hadron, now shall we?
Not very constructive for the poor fellow. No need to taunt him when
he's already cursed by the vi$ta disease...
John Thompson
On 2008-03-29, Hadron <hadro...@googlemail.com> wrote:
> John Thompson <jo...@vector.os2.dhs.org> writes:
>
>> AFAIK, Vista doesn't include scp or any other secure file transfer
>> software at all.
> Nonsense. You can install rsync on Windows too.
>
> As well as ssh and scp and .....
Of course, you can install those. But I said I didn't think secure file
transfer softare was *INCLUDED* with Windows. OTOH, every linux
distribution I've tried (and I've been using linux since 1999) includes
ssh and rsync, and a boatload of other useful software you'd have to
search out and install separately for Windows.
--
John (jo...@os2.dhs.org)
Hadron
> ["Followup-To:" header set to comp.os.linux.misc.]
>
> On 2008-03-29, Hadron <hadro...@googlemail.com> wrote:
>> John Thompson <jo...@vector.os2.dhs.org> writes:
>>
>>> AFAIK, Vista doesn't include scp or any other secure file transfer
>>> software at all.
>
>> Nonsense. You can install rsync on Windows too.
>>
>> As well as ssh and scp and .....
>
> Of course, you can install those. But I said I didn't think secure file
> transfer softare was *INCLUDED* with Windows. OTOH, every linux
Define included? I install from a netinst generally and none of those
things are on that either.
> distribution I've tried (and I've been using linux since 1999) includes
> ssh and rsync, and a boatload of other useful software you'd have to
> search out and install separately for Windows.
The point is that these things are there and easily available for
Windows too. Trying to use them as some sort of bonus for linux is
absurd IMO. That is what X and X over ssh is for!
--
modconf (0.2.37) stable unstable; urgency=medium
[...]
* Eduard Bloch:
- fixed Makefile broken Marcin Owsiany a while ago. The default manpage
has been overwritten with the polish translation. I still wonder why
nobody noticed this before. Closes: #117474
[...]
-- Eduard Bloch <bl...@debian.org> Sun, 28 Oct 2001 12:53:27 +0100
John Thompson
> John Thompson <jo...@vector.os2.dhs.org> writes:
>
>> Of course, you can install those. But I said I didn't think secure file
>> transfer softare was *INCLUDED* with Windows. OTOH, every linux
> Define included? I install from a netinst generally and none of those
> things are on that either.
Doesn't the netinst installation process offer to install ssh, rsync,
whatever? I don't remember, the last network installation I did
was Debian MIPS on an SGI Indy and that was several years ago. The
DVD/CD install process does in any case.
But I am confident that the Windows install process doesn't offer any
such option.
> The point is that these things are there and easily available for
> Windows too.
Define "easily available." :-)
--
John (jo...@os2.dhs.org)
Wolfgang
> On 2008-03-25, NoStop <nos...@nospam.com> wrote:
>> Bad security on YOUR local LAN?
>>
>
> Even if someone was downloading torrents through my wifi, it would
> only account for a few megabits. So while your idea is interesting, I
> do not think that it could account for it.
>
> However, plot thickened.
>
> My laptop was connected to the basement server through three gigabit
> switches A, B, C. That's where I had this terrible performance.
>
> I moved it to the basement where it is connected through switches C
> and D (D is a 100 megabit switch).
>
> Here, the performance improved dramatically to 3.9 megaBYTES per
> second.
>
> Go figure.
>
> Why SCP was 36 times slower than HTTP going through A, B, C but not C
> and D, is a mystery.
i have seen effects like this due to the mtu und mtu-path discovery.
Does your net allow icmp?
Do you compare HHTP throughput by large files like your files you want
transfer, or just by loading some webpages?
Have you tried to use another cipher, like blowfish instead 3DES?
Wolfgang