fxp performance with POLLING
Bartosz Stec
With POLLING enabled I experience about 10%-25% performance drop when
copying files over network. Tested with both SAMBA and NFS. Is it normal?
FreeBSD 7.1-PRERELEASE #0: Sat Sep 6 01:52:12 CEST 2008
fxp0: <Intel 82801DB (ICH4) Pro/100 Ethernet> port 0xc800-0xc83f mem
0xe1021000-0xe1021fff irq 20 at device 8.0 on pci1
# ifconfig fxp0
fxp0: flags=9843<UP,BROADCAST,RUNNING,SIMPLEX,LINK0,MULTICAST>
metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:20:ed:42:87:13
inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
BTW overall SAMBA performance still sucks on 7.1-pre as much as on
RELENG_5 ...:( - 7.5 MB/s peak.
--
Bartosz Stec
Pieter de Goeje
> Hello again :)
>
> With POLLING enabled I experience about 10%-25% performance drop when
> copying files over network. Tested with both SAMBA and NFS. Is it normal?
Yes. You don't want to use polling unless you set kern.hz to 10000 or
something in that range. If you have a NIC with interrupt moderation, polling
should almost never be necessary. Note that polling can still be useful for
routers, because it allows you to have a much more responsive system even
when handling heavy network traffic.
--
Pieter de Goeje
Chuck Swiger
> On Friday 03 October 2008, Bartosz Stec wrote:
>> Hello again :)
>>
>> With POLLING enabled I experience about 10%-25% performance drop when
>> copying files over network. Tested with both SAMBA and NFS. Is it normal?
>
> Yes. You don't want to use polling unless you set kern.hz to 10000 or
> something in that range.
HZ = 1000 or 2000 is fine for most purposes, at least up through T3 level
bandwidth. For a home LAN or small business office of a half-dozen machines
using DSL/Cable (~ 1-5 MBs up), even a P2-300 or VIA C3 600 at HZ=250 works OK
as a firewall/router. The main thing that using polling does is that it adds
a reasonably fixed amount of latency (ie, the poll interval) but gives solid
processing performance even under heavy load, just as you say:
> If you have a NIC with interrupt moderation, polling
> should almost never be necessary. Note that polling can still be useful for
> routers, because it allows you to have a much more responsive system even
> when handling heavy network traffic.
Note that he's got the link0 flag going, so that should mean he's using
firmware with the fxp NIC which does interrupt moderation.
Regards,
--
-Chuck
Alfred Perlstein
7.5MB is 75% effeciency of a 100mbit card. Not amazing, but
not "sucks".
Where do you see faster performance?
Between windows machines on the same hardware or linux server?
--
- Alfred Perlstein
Bartosz Stec
tried polling only because I found some suggestions on mailing lists,
that it could improve performance with SAMBA on FreeBSD. As you see at
the top of this thread - not in my case :) I also tried sysctl tunings,
and smb.conf settings, also suggested on maling lists, with no or very
little improvements noticed. Most of suggestions unfortunately end with
"change OS to Linux if you want to use SAMBA". I think I will try to
change NIC to 1Gbit - hope that helps :) Or maybe there's some "FreeBSD
and SAMBA tuning guide" which I didn't found?
--
Bartosz Stec
Jeremy Chadwick
Can you please test network I/O using something like netperf or one of
the other network-benchmark tools and not things like NFS or Samba
which rely on disk I/O and other aspects?
--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
Clifton Royston
...
> I remember when on FreeBSD 4.x I was able to copy files from samba and
> to samba up to 12MB/s on 100Mbit lan.
This part seems unlikely, particularly as bit rates are measured in
decimal millions not computer millions.
12*8*1024*1024 = 100,663,296 so that would mean not merely zero but
negative packet and network overhead.
-- Clifton
--
Clifton Royston -- clif...@iandicomputing.com / clif...@lava.net
President - I and I Computing * http://www.iandicomputing.com/
Custom programming, network design, systems and network consulting services
Andrei Kolu
have barely 8MB/s on 100Mbit and 27MB/s on Gigabit lan. Netperf shows
900Mbit/s in any direction, small variety with different switches (ca
10% difference).
Bartosz Stec
It was first time i was using nerperf so I'm not sure I did it
correctly. I installed netperf port on SAMBA serwer (IP 192.168.0.2),
and also download windows binary to windows xp machine (IP
192.168.0.10). All tests ran for one minute.
First test - netperf on FreeBSD and netserver on Windows:
# netperf -l 60 -t TCP_STREAM -H 192.168.0.10
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.0.10 (192.168.0.10) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
8192 32768 32768 60.00 93.97
# netperf -l 60 -t TCP_SENDFILE -H 192.168.0.10
TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.0.10 (192.168.0.10) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
8192 32768 32768 60.00 93.45
# netperf -l 60 -t TCP_RR -H 192.168.0.10
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.0.10 (192.168.0.10) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
32768 65536 1 1 60.00 2433.99
8192 8192
# ifconfig fxp0
fxp0: flags=9843<UP,BROADCAST,RUNNING,SIMPLEX,LINK0,MULTICAST>
metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:20:ed:42:87:13
inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
Second test - netperf on Windows and netserver on FreeBSD:
Unfortunately won't run:
C:\software>netperf-a4 -l 60 -H 192.168.0.2
TCP STREAM TEST to 192.168.0.2
recv_response: partial response received: 0 bytes
Hovewer, thanks to Alfred Perlstein who send mefollowing link:
http://www.mavetju.org/mail/view_message.php?list=freebsd-net&id=755111&thread=no&tag=yes,
I set SO_SNBUF and SO_RCVBUF in smb.conf to 2920. Without any additional
tuning in sysctl I now got about 8MB/s which is *much* better result
than before. It still could be better than that if I am reading netpertf
results correctly :)
Thanks Alfred!
--
Bartosz Stec
Pete French
you are assuming eight bits per byte - but this is a serial line so
you should use ten bits per byte instead.
-pete.
Adam McDougall
>>> BTW overall SAMBA performance still sucks on 7.1-pre as much as on
>>> RELENG_5 ...:( - 7.5 MB/s peak.
>>>
>>
>> 7.5MB is 75% effeciency of a 100mbit card. Not amazing, but
>> not "sucks".
>>
>> Where do you see faster performance?
>>
>> Between windows machines on the same hardware or linux server?
>>
>>
> It sucks because it is a peak performance. About 5-6 MB/s average. I
> tried polling only because I found some suggestions on mailing lists,
> that it could improve performance with SAMBA on FreeBSD. As you see at
> the top of this thread - not in my case :) I also tried sysctl tunings,
> and smb.conf settings, also suggested on maling lists, with no or very
> little improvements noticed. Most of suggestions unfortunately end with
> "change OS to Linux if you want to use SAMBA". I think I will try to
> change NIC to 1Gbit - hope that helps :) Or maybe there's some "FreeBSD
> and SAMBA tuning guide" which I didn't found?
>
Please try experimenting with "socket options" in smb.conf, I've found
that some tuning is desirable on any OS with Samba, but these are the
values that worked best for me with Windows XP clients in mind. Win2003
clients seemed much faster without tuning (same base code as XP 64bit)
and I suspect it has a different SMB implementation. I'd suggest
starting with "socket options = TCP_NODELAY IPTOS_THROUGHPUT
SO_RCVBUF=8192 SO_SNDBUF=8192" and if you aren't satisfied, experiment
with the numbers and which options are enabled. Be sure that the
client has been disconnected from Samba completely to make sure you are
testing the values in the config file. I'm pretty sure with these
tunings I was able to get closer to 10MB/sec on 100Mbit, which satisfies
me for the average user.
# Most people will find that this option gives better performance.
# See smb.conf(5) and /usr/share/doc/samba-doc/htmldocs/speed.html
# for details
# You may want to add the following on a Linux system:
# SO_RCVBUF=8192 SO_SNDBUF=8192
# socket options = TCP_NODELAY
# For some reason, 8192 is pretty fast on a XP lab 100Mb client. Other
sizes tested and dissapointing in that situation. Windows Server 2k3 on
gig is much faster, and likes larger values. There might be some merit
in testing 49152 in some situations. (20080617)
# TCP_NODELAY makes a huge improvement. IPTOS_THROUGHPUT is negligible
locally.
# mcdouga9 20070110
socket options = TCP_NODELAY IPTOS_THROUGHPUT SO_RCVBUF=8192
SO_SNDBUF=8192
# socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
Andrei Kolu
> On Mon, Oct 06, 2008 at 10:29:35AM +0300, Andrei Kolu wrote:
> ...
>
>> I remember when on FreeBSD 4.x I was able to copy files from samba and
>> to samba up to 12MB/s on 100Mbit lan.
>>
>
> This part seems unlikely, particularly as bit rates are measured in
> decimal millions not computer millions.
>
> 12*8*1024*1024 = 100,663,296 so that would mean not merely zero but
> negative packet and network overhead.
>
> -- Clifton
>
--------------------------------------------------------------------
(TEG-MGBSX, TEG-MGBS10, TEW-MGBS40, TEW-MGBS80)
Compliant with IEEE 802.3z Gigabit Ethernet and Fiber Channel Standards
Industry standard SFP package
Duplex LC connector
1.0625Gbps Fiber Channel Compliant
1.25Gbps Gigabit Ethernet Compliant
--------------------------------------------------------------------
So, I guess that 100Mbit and 1000Mbit is not set in stone and you can
actually achieve higher speeds than "standard".
BTW:
1 megabit = 106 = 1,000,000 bits which is equal to 125,000 bytes.
100 megabit = 12,500,000 bytes = 12,5MB
Or I am wrong?
Jeremy Chadwick
Interesting. I use gigE on my home network, and most of my Samba shares
consist of very large files (ISO images and HD videos). Setting the
buffer sizes to 8KB results in horrible performance; a 420MB ISO image
taking ~50 seconds to copy. Copying is from the Samba server to the
Windows machine. ZFS is used on FreeBSD, across 4 disks in a raidz1
pool; all disks are SATA300 with 16MByte cache.
Increasing the Samba buffers to 64KB (65536) drops the transfer to 30
seconds, and increasing it to 128KB (131072) drops to 15-20
seconds.
NICs involved are an Attansic L1E (on Windows XP, with TCP buffer sizes
tuned in the registry, ditto with enabling TCP window resizing), and an
em(4) (on FreeBSD). There are two switches between the boxes: an HP
ProCurve, and a generic D-Link switch. Everything is gigE.
Additionally, don't forget about IPTOS_LOWDELAY, and the "write cache
size" and "read size" options in smb.conf.
Joe Koberg
That was a rule of thumb in the heyday of async serial lines, which used
a start and stop bit per byte.
However, ethernet at 100Mbit is 4B5B coded at a 125mhz rate. So the raw
synchronous data rate really is 12.5Mbytes/s. Minus the sync preamble
of 8 bytes per packet and the mandatory inter-frame-gap of 12 bytes
that's a physical layer rate of (12.5M * (1500/(1500+20))) or 12.34Mbyte/s.
Even in the later days of modems this rule applied less and less,
because the modulation schemes became synchronous.
Joe Koberg
joe_at_osoft_dot_us
sth...@nethelp.no
> synchronous data rate really is 12.5Mbytes/s. Minus the sync preamble
> of 8 bytes per packet and the mandatory inter-frame-gap of 12 bytes
> that's a physical layer rate of (12.5M * (1500/(1500+20))) or 12.34Mbyte/s.
You need add Ethernet header (14 bytes) + CRC (4 bytes). This means you
have a maximum data rate, assuming 1500 byte MTU, of 12.5M * 1500/1538
= 12.19 Mbyte/s. And for those used to powers of two, M here means one
million, not 1048576.
Steinar Haug, Nethelp consulting, sth...@nethelp.no
Pete French
> a start and stop bit per byte.
>
> However, ethernet at 100Mbit is 4B5B coded at a 125mhz rate. So the raw
Errr, 4B5B *is* 10 bits per byte surely?
> Even in the later days of modems this rule applied less and less,
> because the modulation schemes became synchronous.
Gig ether is mainly 8B10, as is Firewire, SATA, FibreChannel and a
load of others I can't remember off the top of my head. I wouldn't
stay it's a hard and fast rule, but it still gives a better estimate
than dividing by eight which is what people naiively do.
Mind you, it assumes that you know the real bit rate, which in the
case of 100baseT is, as you say, actualy 125mbits/sec.
-pete.
Joe Koberg
>> However, ethernet at 100Mbit is 4B5B coded at a 125mhz rate. So the raw
>>
>
> Errr, 4B5B *is* 10 bits per byte surely?
> Gig ether is mainly 8B10, as is Firewire, SATA, FibreChannel and a
>
> case of 100baseT is, as you say, actualy 125mbits/sec.
>
You are right. It definitely is 10 bits per byte clocked at a higher
rate. I guess the "100mbit/s" rate is so strongly associated with the
technology that I glossed right over that.
Joe