8-STABLE outgoing scp stalling frequently.
Jonathan Chen
I've noticed that on a recent 8-STABLE/amd64, scp(1) appears to be
stalling very frequently. This is the output from a "scp -v -v"
of a 300Mb file from a local to a remote within an internal network:
[.. authentication negotiation ...]
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 131072
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
Sending file modes: C0700 367085370 file1
file1 0% 192KB 140.0KB/s 42:39 ETAdebug2: channel 0: rcvd adjust 65593
file1 0% 256KB 78.2KB/s - stalled -debug2: channel 0: rcvd adjust 81920
file1 0% 336KB 41.6KB/s - stalled -debug2: channel 0: rcvd adjust 81920
file1 0% 416KB 26.9KB/s - stalled -debug2: channel 0: rcvd adjust 81920
file1 0% 496KB 17.1KB/s - stalled -debug2: channel 0: rcvd adjust 81920
file1 0% 576KB 12.4KB/s - stalled -debug2: channel 0: rcvd adjust 81920
file1 0% 656KB 11.3KB/s - stalled -debug2: channel 0: rcvd adjust 81920
file1 0% 736KB 9.7KB/s - stalled -debug2: channel 0: rcvd adjust 81920
file1 0% 816KB 9.9KB/s - stalled -debug2: channel 0: rcvd adjust 81920
file1 0% 896KB 9.0KB/s - stalled -debug2: channel 0: rcvd adjust 81920
file1 0% 976KB 9.5KB/s - stalled -debug2: channel 0: rcvd adjust 81920
/etc/ssh/ssh_config is untouched. Oddly enough a scp of a remote file
to the local machine is blindingly fast.
Does anyone know what's happening here? Any tips on how to track down
what the problem is? The network config appears to be fine - fetch(1) will
have downloads speeds of up to 300KB/s.
Cheers.
--
Jonathan Chen <jo...@chen.org.nz>
----------------------------------------------------------------------
"With sufficient thrust, pigs fly just fine. However, this is not necessarily
a good idea. It is hard to be sure where they are going to land, and it
could be dangerous sitting under them as they fly overhead." -- RFC 1925
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
Mike Tancsa
>Hi,
>
>I've noticed that on a recent 8-STABLE/amd64, scp(1) appears to be
>stalling very frequently. This is the output from a "scp -v -v"
>of a 300Mb file from a local to a remote within an internal network:
Hi,
Is it on the same ethernet segment, or does it go through a
router(s) and is the path asymmetric? I noticed similar symptoms on a
jailed box where the round trip was asymmetric and an intermediary
router was generating a lot of ICMP redirects that were ignored by
the sending host.
---Mike
--------------------------------------------------------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mi...@sentex.net
Providing Internet since 1994 www.sentex.net
Cambridge, Ontario Canada www.sentex.net/mike
Jonathan Chen
> At 02:36 PM 2/2/2010, Jonathan Chen wrote:
> >Hi,
> >
> >I've noticed that on a recent 8-STABLE/amd64, scp(1) appears to be
> >stalling very frequently. This is the output from a "scp -v -v"
> >of a 300Mb file from a local to a remote within an internal network:
>
> Hi,
> Is it on the same ethernet segment, or does it go through a
> router(s) and is the path asymmetric?
The 2 hosts are going in on cables to the same switch. I've tried
other ports and hosts , but experience the same problem.
--
Jonathan Chen <jo...@chen.org.nz>
----------------------------------------------------------------------
"Irrationality is the square root of all evil"
- Douglas Hofstadter
Adam Vande More
> The 2 hosts are going in on cables to the same switch. I've tried
> other ports and hosts , but experience the same problem.
>
I run a similar network setup at home, and am unable to replicate your
experience regardless which host is being copied to/from. Perhaps you have
a more specific issue eg nic driver ?
--
Adam Vande More
Jeremy Chadwick
> On Tue, Feb 02, 2010 at 02:44:09PM -0500, Mike Tancsa wrote:
> > At 02:36 PM 2/2/2010, Jonathan Chen wrote:
> > >Hi,
> > >
> > >I've noticed that on a recent 8-STABLE/amd64, scp(1) appears to be
> > >stalling very frequently. This is the output from a "scp -v -v"
> > >of a 300Mb file from a local to a remote within an internal network:
> >
> > Hi,
> > Is it on the same ethernet segment, or does it go through a
> > router(s) and is the path asymmetric?
>
> The 2 hosts are going in on cables to the same switch. I've tried
> other ports and hosts , but experience the same problem.
Do you see this behaviour both directions, or just unidirectional? E.g.
does the problem happen in both of the below examples, or just one?
box1$ scp user@box2:/file .
box1$ scp file user@box2:/some/path
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
Mike Tancsa
>On Tue, Feb 2, 2010 at 2:51 PM, Jonathan Chen
><<mailto:jo...@chen.org.nz>jo...@chen.org.nz> wrote:
>The 2 hosts are going in on cables to the same switch. I've tried
>other ports and hosts , but experience the same problem.
>
>
>I run a similar network setup at home, and am unable to replicate
>your experience regardless which host is being copied
>to/from. Perhaps you have a more specific issue eg nic driver ?
This is to a box on the same ethernet and subnet
0(ich10)% dd if=/dev/urandom of=testfile bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 1.879028 secs (55804169 bytes/sec)
0(ich10)% scp testfile 192.168.1.207:/dev/null
testfile
100% 100MB 12.5MB/s 00:08
0(ich10)% scp -C testfile 192.168.1.207:/dev/null
testfile
100% 100MB 11.1MB/s 00:09
0(ich10)%
From RELENG_8 to RELENG_7
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
ether 00:1c:c0:95:0d:0d
inet 192.168.1.219 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
uname -a
FreeBSD ich10.sentex.ca 8.0-STABLE FreeBSD 8.0-STABLE #13: Tue Jan 19
12:24:57 EST
2010 mdta...@ich10.sentex.ca:/usr/obj/usr/src/sys/server i386
If you are using an em nic, does
sysctl -w dev.em.0.stats=1
give any interesting numbers in /var/log/messages ?
em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 0
em0: Receive No Buffers = 0
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 0
em0: watchdog timeouts = 0
em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 1003041
em0: Good Packets Xmtd = 690690
em0: TSO Contexts Xmtd = 15709
em0: TSO Contexts Failed = 0
Results from RELENG_8 to RELENG_9 are about the same, but on a different nic
0(ich10)% scp testfile 10.255.255.117:/dev/null
testfile
100% 100MB 11.1MB/s 00:09
0(ich10)%
igb1: Excessive collisions = 0
igb1: Sequence errors = 0
igb1: Defer count = 0
igb1: Missed Packets = 0
igb1: Receive No Buffers = 0
igb1: Receive Length Errors = 0
igb1: Receive errors = 0
igb1: Crc errors = 0
igb1: Alignment errors = 0
igb1: Collision/Carrier extension errors = 0
igb1: RX overruns = 0
igb1: watchdog timeouts = 0
igb1: XON Rcvd = 0
igb1: XON Xmtd = 0
igb1: XOFF Rcvd = 0
igb1: XOFF Xmtd = 0
igb1: Good Packets Rcvd = 36484881
igb1: Good Packets Xmtd = 50922117
igb1: TSO Contexts Xmtd = 5452450
igb1: TSO Contexts Failed = 0
>--
>Adam Vande More
--------------------------------------------------------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mi...@sentex.net
Providing Internet since 1994 www.sentex.net
Cambridge, Ontario Canada www.sentex.net/mike
_______________________________________________
Jonathan Chen
> On Wed, Feb 03, 2010 at 08:36:16AM +1300, Jonathan Chen wrote:
> > Hi,
> >
> > I've noticed that on a recent 8-STABLE/amd64, scp(1) appears to be
> > stalling very frequently. This is the output from a "scp -v -v"
> > of a 300Mb file from a local to a remote within an internal network:
[...]
> > Does anyone know what's happening here? Any tips on how to track down
> > what the problem is? The network config appears to be fine - fetch(1) will
> > have downloads speeds of up to 300KB/s.
>
> But how about upload speeds? It seems that's where scp is suffering as
> well.
This is the obvious test that I should have done; and you're hit the
nail on the head. bge(4) on 8-STABLE (csup'd 4-Feb-2010) has a very
bad upload speed.
I've just tried using ftp to transfer some files:
Upload speed: starts at 63 KB/s, falls rapidly before stalling.
Download speeds: starts at 9 MB/s, increasing slightly before completing.
Device from dmesg:
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x00a002> mem 0xf1bf0000-0xf1bfffff irq 17 at device 0.0 on pci9
--
Jonathan Chen <jo...@chen.org.nz>
----------------------------------------------------------------------
"I don't want to achive immortality through my works..
I want to achieve it through not dying" - Woody Allen
Pyun YongHyeon
> On Tue, Feb 02, 2010 at 11:20:29PM +0200, Nikos Ntarmos wrote:
> > On Wed, Feb 03, 2010 at 08:36:16AM +1300, Jonathan Chen wrote:
> > > Hi,
> > >
> > > I've noticed that on a recent 8-STABLE/amd64, scp(1) appears to be
> > > stalling very frequently. This is the output from a "scp -v -v"
> > > of a 300Mb file from a local to a remote within an internal network:
>
> [...]
> > > Does anyone know what's happening here? Any tips on how to track down
> > > what the problem is? The network config appears to be fine - fetch(1) will
> > > have downloads speeds of up to 300KB/s.
> >
> > But how about upload speeds? It seems that's where scp is suffering as
> > well.
>
> This is the obvious test that I should have done; and you're hit the
> nail on the head. bge(4) on 8-STABLE (csup'd 4-Feb-2010) has a very
> bad upload speed.
>
> I've just tried using ftp to transfer some files:
>
> Upload speed: starts at 63 KB/s, falls rapidly before stalling.
> Download speeds: starts at 9 MB/s, increasing slightly before completing.
>
I'm not sure but recently added code to support TSO may cause the
issue. Would you show me verbose boot output(only bge(4) related
one)? To rule out possible TSO issue, disable TSO and try it
again(#ifconfig bge0 -tso). Does it make any difference?
> Device from dmesg:
> bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x00a002> mem 0xf1bf0000-0xf1bfffff irq 17 at device 0.0 on pci9
>
Jonathan Chen
> On Thu, Feb 04, 2010 at 11:52:55AM +1300, Jonathan Chen wrote:
> > On Tue, Feb 02, 2010 at 11:20:29PM +0200, Nikos Ntarmos wrote:
> > > On Wed, Feb 03, 2010 at 08:36:16AM +1300, Jonathan Chen wrote:
> > > > Hi,
> > > >
> > > > I've noticed that on a recent 8-STABLE/amd64, scp(1) appears to be
> > > > stalling very frequently. This is the output from a "scp -v -v"
> > > > of a 300Mb file from a local to a remote within an internal network:
> >
> > [...]
> > > > Does anyone know what's happening here? Any tips on how to track down
> > > > what the problem is? The network config appears to be fine - fetch(1) will
> > > > have downloads speeds of up to 300KB/s.
> > >
> > > But how about upload speeds? It seems that's where scp is suffering as
> > > well.
> >
> > This is the obvious test that I should have done; and you're hit the
> > nail on the head. bge(4) on 8-STABLE (csup'd 4-Feb-2010) has a very
> > bad upload speed.
> >
> > I've just tried using ftp to transfer some files:
> >
> > Upload speed: starts at 63 KB/s, falls rapidly before stalling.
> > Download speeds: starts at 9 MB/s, increasing slightly before completing.
> I'm not sure but recently added code to support TSO may cause the
> issue. Would you show me verbose boot output(only bge(4) related
> one)?
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x00a002> mem 0xf1bf0000-0xf1bfffff irq 17 at device 0.0 on pci9
bge0: Reserved 0x10000 bytes for rid 0x10 type 3 at 0xf1bf0000
bge0: adjust device control 0x2000 -> 0x5000
bge0: attempting to allocate 1 MSI vectors (1 supported)
bge0: using IRQ 258 for MSI
bge0: CHIP ID 0x0000a002; ASIC REV 0x0a; CHIP REV 0xa0; PCI-E
bge0: Disabling fastboot
bge0: Disabling fastboot
miibus0: <MII bus> on bge0
bge0: bpf attached
bge0: Ethernet address: 00:1d:09:d2:d1:9e
bge0: [MPSAFE]
bge0: [FILTER]
bge0: Disabling fastboot
bge0: Disabling fastboot
bge0: link UP
>To rule out possible TSO issue, disable TSO and try it
> again(#ifconfig bge0 -tso). Does it make any difference?
Yup, it sure does! With a TSO disabled, my upload and download speeds
are pretty much symmetrical at a decent 10MB/s.
Thanks!
--
Jonathan Chen <jo...@chen.org.nz>
----------------------------------------------------------------------
Do not take life too seriously.
You will never get out of it alive.
Pyun YongHyeon
Hmm, that means TSO was broken on your controller. Because BCM5755
or newer controllers have no known TSO issues I don't know why the
controller fails on TSO. Very recent controllers use new TSO format
but I don't think your controller is one of them and FreeBSD has no
support for these controllers anyway.
Would you show me the output of "pciconf -lcv" of your bge(4)
controller?
Jonathan Chen
> On Thu, Feb 04, 2010 at 03:00:15PM +1300, Jonathan Chen wrote:
> > On Wed, Feb 03, 2010 at 05:25:03PM -0800, Pyun YongHyeon wrote:
bge0@pci0:9:0:0: class=0x020000 card=0x01fe1028 chip=0x167314e4 rev=0x02 hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetXtreme BCM5755M Gigabit Ethernet PCIe'
class = network
subclass = ethernet
cap 01[48] = powerspec 3 supports D0 D3 current D0
cap 03[50] = VPD
cap 09[58] = vendor (length 120)
cap 05[e8] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[d0] = PCI-Express 1 endpoint max data 128(128) link x1(x1)
This is on a Dell Latitude D830 Laptop.
Cheers.
--
Jonathan Chen <jo...@chen.org.nz>
----------------------------------------------------------------------
"Beer. Now there's a temporary solution."
- Homer Simpson
Olivier Cochard-Labbé
> Hi,
>
> I've noticed that on a recent 8-STABLE/amd64, scp(1) appears to be
> stalling very frequently.
I've got the same problem since I've upgraded from 7.2 to 8-stable (32bit).
My NIC is a vge(4), with txsum and rxsum disabled.
dmesg:
vge0: <VIA Networking Gigabit Ethernet> port 0xfc00-0xfcff mem
0xfdfff000-0xfdfff0ff irq 18 at device 14.0 on pci0
I can't SCP big file too because my transferts stall and abord.
Regards,
Olivier
Pyun YongHyeon
> On Tue, Feb 2, 2010 at 8:36 PM, Jonathan Chen <jo...@chen.org.nz> wrote:
> > Hi,
> >
> > I've noticed that on a recent 8-STABLE/amd64, scp(1) appears to be
> > stalling very frequently.
>
> I've got the same problem since I've upgraded from 7.2 to 8-stable (32bit).
>
> My NIC is a vge(4), with txsum and rxsum disabled.
> dmesg:
> vge0: <VIA Networking Gigabit Ethernet> port 0xfc00-0xfcff mem
> 0xfdfff000-0xfdfff0ff irq 18 at device 14.0 on pci0
>
> I can't SCP big file too because my transferts stall and abord.
>
I guess I fixed all known vge(4) issues, how recent stable/8 you
use?
Olivier Cochard-Labbé
>
> I guess I fixed all known vge(4) issues, how recent stable/8 you
> use?
>
Hi,
my mistake, it's a release and not a stable version that I'm using:
uname -a
FreeBSD dev.bsdrp.net 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #0: Tue
Jan 5 16:02:27 UTC 2010
ro...@i386-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386
Regards,
Olivier
Pyun YongHyeon
> On Tue, Feb 9, 2010 at 10:51 PM, Pyun YongHyeon <pyu...@gmail.com> wrote:
> >
> > I guess I fixed all known vge(4) issues, how recent stable/8 you
> > use?
> >
>
> Hi,
>
> my mistake, it's a release and not a stable version that I'm using:
>
All issues except poor Tx performance was fixed. I'm still not
sure whether the poor Tx performance on PCIe VT6130 comes from the
limitation of controller which seems to limit the number of
outstanding DMA cycles.
> uname -a
>
> FreeBSD dev.bsdrp.net 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #0: Tue
> Jan 5 16:02:27 UTC 2010
> ro...@i386-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386
>
Ok, if you encounter vge(4) issues in stable/8 let us know.
Pyun YongHyeon
I committed a fix which disables TSO on BCM5755M. Still have no
idea why it fails though.
Thanks for reporting!