Re: em forwarding performance (was Proposed 6.2 em RELEASE patch
Mike Tancsa
I am moving this thread over to performance after this post as it
makes more sense to continue there.
At 11:54 PM 11/19/2006, Mike Tancsa wrote:
>At 04:30 PM 11/13/2006, Scott Long wrote:
>>Mike Tancsa wrote:
>>>At 12:15 AM 11/13/2006, Scott Long wrote:
>>>
>>>>Is this with EM_INTR_FAST enabled also?
>>>
>>>Without it, the 2 streams are definitely lossy on the management interface
>>>
>>> ---Mike
>>
>>Ok, and would you be able to test the polling options as well?
>
>
>Note about platforms. The HEAD w Patch is a patch
>gle...@freebsd.org asked me to test. FastFWD is with
>net.inet.ip.fastforwarding on. Also with FastFWD set to one, I
>always used the kernel options ADAPTIVE_GIANT commented out and
>added NO_ADAPTIVE_MUTEXES. INET6 was removed from all kernels as
>well. With these kernel changes, and fast forwarding on, I was able
>to keep the box r2 responsive from the console as while blasting
>packets across its 2 interfaces. Otherwise, the box seemingly
>livelocked. For the linux kernel config, it was pretty well the
>default, except I removed INET6, IPSEC and disabled iptables. The
>LINUX kernel was 2.6.18.2 on FC5.
>
>The first test is with UDP netperf.
>/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t
>UDP_STREAM -- -m 10 -s 32768 -S 32768
>/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t
>UDP_STREAM -- -m 64 -s 32768 -S 32768
>/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t
>UDP_STREAM -- -m 128 -s 32768 -S 32768
>/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t
>UDP_STREAM -- -m 200 -s 32768 -S 32768
Again, this is the same setup as described at http://www.tancsa.com/blast.jpg
My goals of this testing was to understand the following :
1) the new em driver to make sure it works well for me and give it a
good shake out under load for RELENG_6
2) understand the implications of various configs on forwarding
performance of SMP vs UP vs Polling vs Fast Interrupts and to avoid
livelock when there is a lot of pps
In this round of testing, I tried of RELENG_6 i386 in UP config as
well. Although raw packets / second (pps) forwarding was faster, the
box was pretty unresponsive and userland apps (i.e. routing) and made
the config pretty unusable with fast_forwarding enabled. Once ipfw
was loaded, the box would totally lock up and routing daemons was
start to spin out of control as hold timers expired. Bottom line,
there is slightly less raw pps performance out of an SMP config, but
the box seems to be far more resilient against a high pps attack.
RELENG_6 SMP with #define EM_FAST_INTR 1 in if_em.c
net.inet.ip.fastforwarding=1
with ADAPTIVE_GIANT removed from the kernel and
NO_ADAPTIVE_MUTEXES added
gives decent pps forwarding rates, without the box becoming live
locked. Note, in the real world, given an average packet size of
~600 bytes, a box in this config should be able to route and firewall
gigabit speeds without much issue and can sustain moderate DDoS pps
attacks over the net.
For the routing test, I used 2 peers each sending ~ 195K routes.
I re-tested the single UDP stream with 194k routes loaded in the
kernel routing table and 2 bgp peers. Then, while blasting across the
box, I cleared the session which had all the routes installed in the
kernel so that the daemon would have to reinstall all the routes to
point to the other peer. While this was happening (10 seconds on SMP
boxes, MUCH longer ~ 1min on UP, sometimes total failure) I was
measuring throughput. On UP it didnt drop too much, a bit more on
SMP, but convergence was quite fast, about 10 seconds. Similarly,
installing ipfw rules on the UP version made the box totally live
lock in non polling mode, but seemed to perform well enough in
polling mode. pf lagged quite far behind
The biggest difference seems to be in the efficiency of the firewall
rules in LINUX vs FreeBSD. Granted, the rules I inserted, are poorly
written, but adding rules did seem to have a linearly negative impact
on performance, where as rules via iptables did not significantly
impact forwarding rates. However, in the LINUX test, I seemed to
trigger some race in bgpd when doing the clear that took a little
more proding with FreeBSD, but is there as well :( So it might be
back to version .98
The table is also up at http://www.tancsa.com/blast.html which might
be easier to read
Straight Routing test One Stream 194K Routes bgp
clear and single ipfw 5 ipfw ruipfw 10 pf 1 pf 5
Linux 581,310 590,584
579,833 582,963 575,718 579,442
FreeBSD HEAD Nov 20
+FastFWD 540,695 529,425 439,980
398,283 370,458
FreeBSD HEAD Nov 11 441,560
RELENG6 i386 407,403
RELENG6 i386
FastFWD 557,580 562,547 484,250
425,791 386,644 353,856 333,293
FreeBSD HEAD w Patch 422,294
FreeBSD HEAD w Patch
FastFWD 567,290 564,340 482,093 436,205
381,459 359,248 361,458
AMD64 RELENG6 w
FastFWD 574,590 549,233 486,737
400,050 320,129 296,760 273,824
AMD64 RELENG6 polling 285,917
AMD64 RELENG6 polling FastFWD 512,042
RELENG6 i386 polling FastFWD 558,600 550,041
RELENG6 i386 polling FastFWD HZ=2000 565,520 563,068 373,904
RELENG_6 UP i386 400,188
RELENG_6 UP i386
FastFWD 584,900 582,534 560,033 560,323
RELENG_6 UP i386 FastFWD
Polling 585,934 476,629
422,633 393,301
Straight Routing test 2 streams opposite direction
Linux 473,810 452,205
408,932
FreeBSD HEAD Nov 11 204,043
FreeBSD HEAD nov 20 +
fastFWD 312,140 250,277
223,289 208,551
RELENG6 i386 165,461
RELENG6 i386
FastFWD 368,960 353,437
216,597 206,077 194,47669,50067,385
FreeBSD HEAD w Patch 127,832
FreeBSD HEAD w Patch
FastFWD 346,220 404,785
249,617 234,047157,603
AMD64 RELENG6 w Polling 155,659
AMD64 RELENG6 w Polling FastFWD 231,541
RELENG6 i386 polling FastFWD 319,350 312,621
RELENG6 i386 polling FastFWD HZ=2000 300,280 299,018
RELENG_6 UP i386 FastFWD
Polling 342,551
229,652 205,322
Mike Tancsa
>The table is also up at http://www.tancsa.com/blast.html which might
>be easier to read
Decided to test with RELENG_4 as a comparison. Quite a difference.
With polling and fast forwarding on, I can use 2 routers to blast
through at almost 1Mpps. Even with ipfw loaded, it performs as
RELENG_6 and above does without ipfw. Updated stats in the table at
the above URL.
One other aspect I have not looked at yet are some of the compile
time tunables for em. Looking at the stats from the dual port em nic
below, does anyone have any suggestions on what to adjust ? This is
a PCIe dual port Pro 1000 PT.
em0: Excessive collisions = 0
em0: Symbol errors = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 271924
em0: Receive No Buffers = 1662344
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Carrier extension errors = 0
em0: RX overruns = 0
em0: watchdog timeouts = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 149
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 272047
em0: Good Packets Rcvd = 175954557
em0: Good Packets Xmtd = 28676200
em1: Excessive collisions = 0
em1: Symbol errors = 0
em1: Sequence errors = 0
em1: Defer count = 0
em1: Missed Packets = 58484
em1: Receive No Buffers = 56645
em1: Receive Length Errors = 0
em1: Receive errors = 0
em1: Crc errors = 0
em1: Alignment errors = 0
em1: Carrier extension errors = 0
em1: RX overruns = 0
em1: watchdog timeouts = 0
em1: XON Rcvd = 0
em1: XON Xmtd = 589
em1: XOFF Rcvd = 0
em1: XOFF Xmtd = 59073
em1: Good Packets Rcvd = 28676216
em1: Good Packets Xmtd = 175954480
---Mike
Jeremie Le Hen
Thank you for spending that much time for benchmarking, this is really
interesting.
Though this is a little bit off topic, I'm quite puzzled by the fact
that having filtering rules on Linux or not doesn't change the result
much. NetFitler keeps track of *all* connections even if there are no
ruleset loaded -- you don't have to ask for it, so I guess you are
simply wiping filtering rules, but you don't disable connection
tracking. AFAIK, you can only disable it by either unloading the
`conntrack'' module or recompiling the kernel without it, if built in.
It would be interesting to know the real performance of Linux as a mere
router if we want a true comparision with FreeBSD performances.
Regards,
--
Jeremie Le Hen
< jeremie at le-hen dot org >< ttz at chchile dot org >
Mike Tancsa
>Hi Mike,
>
>Thank you for spending that much time for benchmarking, this is really
>interesting.
Hi,
More to come, and if you can think of other tests let me
know. Next is VLAN performance.
>Though this is a little bit off topic, I'm quite puzzled by the fact
>that having filtering rules on Linux or not doesn't change the result
>much. NetFitler keeps track of *all* connections even if there are no
>ruleset loaded -- you don't have to ask for it
Not sure, but I would unload iptables from the kernel when
testing. I will check again today as I want to go back and test the
LINUX kernel in UP mode to see what difference it makes.
>It would be interesting to know the real performance of Linux as a mere
>router if we want a true comparision with FreeBSD performances.
As just a router, they seem fairly close without any firewall
rules. RELENG_4 seems to be the only clear leader in terms of raw
pps, at least in these tests. I am still puzzled by the fact that
ipfw does relatively poorly compared to RELENG_4.
I also have some PCIe bge nics I will try and test. There are some
patches that Bruce Evans posted and I would like to see how they perform.
---Mike
Mike Tancsa
>It would be interesting to know the real performance of Linux as a mere
>router if we want a true comparision with FreeBSD performances.
Re-tested, this time with a LINUX UP kernel and there is not that
much difference in overall speeds. I added a few IPTABLES rules which
loaded a few of the modules.
[root@r2 ~]# iptables -A OUTPUT -p tcp -s 10.90.2.1 --dport 135:139 -j DROP
[root@r2 ~]# iptables -A OUTPUT -p tcp -s 11.90.3.1 --dport 445 -j DROP
[root@r2 ~]# iptables -A OUTPUT -p tcp -s 0.0.0.0/0 --dport 448 -j REJECT
[root@r2 ~]#
[root@r2 ~]# lsmod
Module Size Used by
ipt_REJECT 5248 1
xt_tcpudp 3456 3
iptable_filter 3328 1
ip_tables 12872 1 iptable_filter
x_tables 14212 3 ipt_REJECT,xt_tcpudp,ip_tables
autofs4 21508 2
sunrpc 153660 1
acpi_cpufreq 8204 0
dm_mirror 21968 0
dm_multipath 18568 0
dm_mod 57112 2 dm_mirror,dm_multipath
video 17028 0
sbs 16192 0
i2c_ec 5376 1 sbs
button 7184 0
battery 10500 0
asus_acpi 16792 0
ac 5636 0
pcspkr 3456 0
i2c_nforce2 7552 0
i2c_core 21504 2 i2c_ec,i2c_nforce2
serio_raw 7300 0
tg3 99972 0
e1000 114752 0
ide_cd 38432 0
cdrom 34848 1 ide_cd
forcedeth 39940 0
sata_nv 11652 0
libata 98068 1 sata_nv
sd_mod 20864 0
scsi_mod 133672 2 libata,sd_mod
ext3 129416 1
jbd 58152 1 ext3
ehci_hcd 30984 0
ohci_hcd 19844 0
uhci_hcd 23052 0
[root@r2 ~]#
Stopping it does seem to unload the klds
[root@r2 ~]# /etc/rc.d/init.d/iptables stop
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]
[root@r2 ~]#
[root@r2 ~]# lsmod
Module Size Used by
ipt_REJECT 5248 0
xt_tcpudp 3456 0
x_tables 14212 2 ipt_REJECT,xt_tcpudp
autofs4 21508 2
sunrpc 153660 1
acpi_cpufreq 8204 0
dm_mirror 21968 0
dm_multipath 18568 0
dm_mod 57112 2 dm_mirror,dm_multipath
video 17028 0
sbs 16192 0
i2c_ec 5376 1 sbs
button 7184 0
battery 10500 0
asus_acpi 16792 0
ac 5636 0
pcspkr 3456 0
i2c_nforce2 7552 0
i2c_core 21504 2 i2c_ec,i2c_nforce2
serio_raw 7300 0
tg3 99972 0
e1000 114752 0
ide_cd 38432 0
cdrom 34848 1 ide_cd
forcedeth 39940 0
sata_nv 11652 0
libata 98068 1 sata_nv
sd_mod 20864 0
scsi_mod 133672 2 libata,sd_mod
ext3 129416 1
jbd 58152 1 ext3
ehci_hcd 30984 0
ohci_hcd 19844 0
uhci_hcd 23052 0
[root@r2 ~]#
RELENG_4 still seems to be king, speed wise for raw forwarding and
firewalling performance. I tried Dragon Fly, but the em NICs lock up
with a few packets blasted at it both in SMP and UP mode. Also tried
booting one of their development kernels, but no difference.
I might give OpenBSD a quick try as a reference.
I also tried a few variations in the kernel from CURRENT. Removing
things like KTRACE and PREMPTION seem to have very little impact, but
some. Also tried changing the hardware timer to TSC, but that didnt
have any impact.
Vlad Galu
> At 08:09 AM 11/22/2006, Jeremie Le Hen wrote:
>
> >It would be interesting to know the real performance of Linux as a mere
> >router if we want a true comparision with FreeBSD performances.
>
> Re-tested, this time with a LINUX UP kernel and there is not that
> much difference in overall speeds. I added a few IPTABLES rules which
> loaded a few of the modules.
>
Can you please completely remove the iptables support from your
Linux configuration, as well as removing support for any packet filter
in FreeBSD? Also, please enable fast_forwarding.
--
If it's there, and you can see it, it's real.
If it's not there, and you can see it, it's virtual.
If it's there, and you can't see it, it's transparent.
If it's not there, and you can't see it, you erased it.
Mike Tancsa
> Can you please completely remove the iptables support from your
>Linux configuration, as well as removing support for any packet filter
>in FreeBSD? Also, please enable fast_forwarding.
I did that a while ago. See
http://www.tancsa.com/blast.html
---Mike
Vlad Galu
>
> > Can you please completely remove the iptables support from your
> >Linux configuration, as well as removing support for any packet filter
> >in FreeBSD? Also, please enable fast_forwarding.
>
> I did that a while ago. See
I'm sorry, I had too much coffee :(
>
> http://www.tancsa.com/blast.html
>
>
> ---Mike
Mike Tancsa
>>test setup description at
>>http://www.tancsa.com/blast.html
More testing :) This time with a pair of PCIe 1x bge nics, as well
as using the patch at
http://lists.freebsd.org/pipermail/freebsd-net/2006-November/012389.html
As I switched to bge, I could test against Dragonfly as well since
its em driver is broken with the dual port PT1000
Couple of interesting notes. Unlike the em nics, I dont see the box
being overwhelmed with packets to the point of netstat -ni showing
input errors. I dont know if this is just a measurement issue, or
the driver is smarter about throttling back the sender rate with the
switch ? I also noticed that the initial blast, as seen by the
routing box, sees pps rates in around 500k, but then slowly goes down
to about 460k and settles in around that rate but then eventually go
back up towards 500kpps.
26479 0 1588740 2 0 332 0
511592 0 30695562 3 0 386 0
508043 0 30482580 2 0 332 0
508026 0 30481560 2 0 332 0
input (Total) output
packets errs bytes packets errs bytes colls
506675 0 30400628 2 0 332 0
511805 0 30708402 3 0 498 0
477792 0 28667520 2 0 332 0
454527 0 27271620 2 0 332 0
454815 0 27288900 2 0 332 0
453920 0 27235200 2 0 444 0
449027 0 26941620 2 0 332 0
449383 0 26962980 2 0 332 0
461838 0 27710280 2 0 332 0
455068 0 27304080 2 0 332 0
455014 0 27300840 2 0 332 0
411416 0 24684960 4 0 664 0
3 0 180 2 0 332 0
2 0 120 2 0 332 0
4 0 293 2 0 332 0
2 0 120 2 0 332 0
443378 0 26602680 2 0 332 0
455782 0 27346920 2 0 332 0
453432 0 27205920 2 0 332 0
453816 0 27228960 2 0 332 0
450561 0 27033660 2 0 332 0
input (Total) output
packets errs bytes packets errs bytes colls
450317 0 27019020 2 0 332 0
454149 0 27248940 2 0 444 0
With the above patch and from current as of today (including
http://lists.freebsd.org/pipermail/cvs-all/2006-November/197096.html)
the box is *very* responsive even when two blasts of packets are
going in opposite directions!
Here initially is one blast coming from 192.168.88.176 to
192.168.44.1, then I start up another blast from 192.168.44.244 to
192.168.88.206. The box is totally responsive from the management interface
[r2-current]# vmstat -i
interrupt total rate
irq4: sio0 929 0
irq14: ata0 1489 0
irq15: ata1 67 0
irq16: bge1 3158402 1980
irq17: bge0 4918 3
irq19: bge2 2358601 1478
cpu0: timer 3187013 1998
irq256: em0 4 0
irq257: em1 4 0
cpu1: timer 3186657 1997
Total 11898084 7459
[r2-current]#
[r2-current]# ifstat -b
bge0 bge1 bge2
Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out
0.47 1.67 222466.8 0.00 0.00 174265.6
0.94 1.17 233668.8 0.00 0.00 183040.6
0.47 1.17 212013.7 0.00 0.00 166077.4
0.94 1.17 220571.9 0.00 0.00 172781.3
0.47 1.17 240600.0 0.00 0.00 188470.0
0.94 1.17 239261.9 0.00 0.00 187421.8
0.47 1.17 210286.3 0.00 0.00 164724.2
32.20 46.60 236021.0 0.00 0.00 184883.1
9.18 8.46 240039.5 0.00 0.00 188030.9
7.31 7.54 237144.6 0.00 0.00 185763.3
7.59 7.62 231464.9 0.00 0.00 181314.2
0.47 2.47 210767.9 0.00 0.00 165101.5
1.40 2.47 216543.6 0.00 0.00 169625.8
0.94 2.47 213539.4 0.00 0.00 167272.6
1.40 2.47 214488.8 0.00 0.00 168016.2
0.94 2.47 197169.2 21056.33 24296.24 154449.2
1.40 2.47 117003.6 101840.6 117507.9 91652.81
2.20 2.89 119036.2 100521.9 115987.3 93245.06
1.87 2.47 117262.4 101812.5 117475.5 91855.57
0.94 2.47 118018.5 101145.1 116706.3 92447.80
1.40 2.47 116557.3 102412.8 118168.2 91303.23
0.94 2.47 117745.3 101736.3 117388.0 92233.84
1.40 2.47 117146.3 102174.3 117893.4 91764.60
0.94 2.47 116071.5 102631.6 118421.1 90922.66
0.94 2.47 116262.2 102233.0 117961.1 91072.04
0.94 2.47 116043.0 102018.6 117713.2 90899.94
1.87 2.47 115890.5 101801.2 117463.4 90780.86
0.94 2.47 117120.8 101347.2 116939.1 91744.60
1.40 2.47 118051.4 100909.8 116434.4 92473.56
0.94 2.47 118767.3 102499.5 118268.6 93034.42
1.40 2.47 116915.7 101836.6 117503.8 91583.94
1.40 3.34 116517.4 102566.9 118346.0 91271.97
1.73 2.89 116316.7 102750.2 118558.4 91114.77
0.94 2.47 117356.1 101465.1 117074.6 91928.93
1.40 2.47 117906.8 101133.9 116693.0 92360.30
1.82 2.47 116012.6 102530.9 118305.3 90876.51
1.40 2.47 115859.8 103404.1 119311.9 90756.86
0.94 2.47 117364.7 102034.7 117732.3 91935.72
1.40 2.47 117721.1 100830.2 116342.5 92214.86
1.40 2.47 116521.8 102289.6 118026.9 91275.45
0.94 2.47 117061.8 101941.4 117624.2 91698.38
bge0 bge1 bge2
Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out
1.87 2.47 91718.83 119571.1 137967.1 71846.42
0.94 3.47 0.00 182065.9 210075.6 0.00
1.40 2.47 0.00 150718.8 173906.3 0.00
0.47 2.47 0.00 0.00 0.00 0.00
1.40 2.47 0.00 0.00 0.00 0.00
0.94 2.47 0.00 0.00 0.00 0.00
1.40 2.47 0.00 0.00 0.00 0.00
[r2-current]# netstat -ni 1
input (Total) output
packets errs bytes packets errs bytes colls
449655 0 26979300 449622 0 21134055 0
462036 0 27722160 462018 0 21715820 0
457761 0 27465660 457814 0 21514989 0
454539 0 27272340 454446 0 21363555 0
469598 0 28175880 469277 0 22175141 0
501175 0 30070500 501315 0 24809563 0
500932 0 30055920 500938 0 24777688 0
502038 0 30122322 501827 0 24855812 0
501276 0 30076560 501521 0 24804540 0
501378 0 30082680 501418 0 24821941 0
501099 0 30065940 501067 0 24806099 0
502164 0 30129840 501890 0 24862704 0
501911 0 30114660 502006 0 24856474 0
499672 0 29980320 499854 0 24742438 0
499575 0 29974502 499586 0 24738495 0
498042 0 29882520 498027 0 24659622 0
500104 0 30006240 500065 0 24751670 0
500569 0 30034140 500653 0 24768087 0
506240 0 30374400 506113 0 25066903 0
500648 0 30038880 500563 0 24777197 0
input (Total) output
packets errs bytes packets errs bytes colls
501650 0 30099000 501554 0 24845586 0
501332 0 30079962 501550 0 24825352 0
500664 0 30039840 500713 0 24778844 0
501305 0 30078300 501235 0 24812561 0
500147 0 30008873 499945 0 24769644 0
501150 0 30069000 501284 0 24827402 0
503208 0 30192420 503357 0 24909495 0
499578 0 29974740 499528 0 24717105 0
500846 0 30050760 500855 0 24799047 0
501295 0 30077700 501261 0 24825246 0
496749 0 29804940 496919 0 24691116 0
449463 0 26967780 449513 0 23372416 0
427765 0 25665900 427857 0 22244044 0
2 0 120 2 0 316 0
Results along with ipfw rule tests at http://www.tancsa.com/blast.html
I also compared Dragonfly using the bge nics and their behaviour is
quite different as compared to FreeBSD. Out of the box its "slower"
but it keeps its forwarding speed with or without ipfw
loaded. Development kernel results are up there as well and it
behaves much like the other BSDs but is notably faster on the
firewall rule processing.
However, FreeBSD HEAD when in UP mode will live lock with 2 blasts of
packets going in opposite directions. The management interface as
well as the serial console is not responsive at all.
[r2-dragonfly]# netstat -ni
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
bge0 1500 <Link#1> 00:10:18:14:15:12 600 0 232 0 0
bge0 1500 fe80:1::210 fe80:1::210:18ff: 0 - 0 - -
bge0 1500 192.168.43 192.168.43.224 280 - 223 - -
bge1 1500 <Link#2> 00:10:18:14:27:d5 76475440 1271 8606533 0 0
bge1 1500 192.168.88 192.168.88.223 4 - 4 - -
bge1 1500 fe80:2::210 fe80:2::210:18ff: 0 - 0 - -
bge2 1500 <Link#3> 00:10:18:14:38:d2 8631578 2465 76061495 0 0
bge2 1500 192.168.44 192.168.44.223 1 - 1 - -
bge2 1500 fe80:3::210 fe80:3::210:18ff: 0 - 0 - -
Massimo Lusetti
> I might give OpenBSD a quick try as a reference.
That would be very interesting.
BTW you really did a good and very compete job, thanks!
Regards
--
Massimo.run();
Divacky Roman
> At 03:28 AM 11/24/2006, Massimo Lusetti wrote:
> >On Thu, 2006-11-23 at 11:52 -0500, Mike Tancsa wrote:
> >
> >> I might give OpenBSD a quick try as a reference.
> >
> >That would be very interesting.
>
>
> I also posted some results with PMC compiled into the kernel
>
> ipfw compiled into the kernel, with 1 rule
>
> http://www.tancsa.com/pmc/
I see generic_bzero/bcopy used quite often. why dont you define
cpu I586_CPU
in your kernel config?
also... bde@ just commited some bzero-related optimizations to routing
code.. might be work trying..
just an idea
roman
Mike Tancsa
>On Fri, Nov 24, 2006 at 03:27:40PM -0500, Mike Tancsa wrote:
> > At 03:28 AM 11/24/2006, Massimo Lusetti wrote:
> > >On Thu, 2006-11-23 at 11:52 -0500, Mike Tancsa wrote:
> > >
> > >> I might give OpenBSD a quick try as a reference.
> > >
> > >That would be very interesting.
> >
> > OpenBSD 4.0 i386 panics on boot.
> >
> > I also posted some results with PMC compiled into the kernel
> >
> > ipfw compiled into the kernel, with 1 rule
> >
> > http://www.tancsa.com/pmc/
>
>I see generic_bzero/bcopy used quite often. why dont you define
>cpu I586_CPU
>in your kernel config?
Hi,
I had
cpu I486_CPU
cpu I586_CPU
cpu I686_CPU
Are you saying
cpu I586_CPU
cpu I686_CPU
or just have
cpu I586_CPU
>also... bde@ just commited some bzero-related optimizations to routing
>code.. might be work trying..
I think that was already in there, but I will cvsup upto today and
see if there is any difference.
---Mike
Mike Tancsa
>I see generic_bzero/bcopy used quite often. why dont you define
>cpu I586_CPU
>in your kernel config?
Hi,
I cvsup'd to todays kernel and re-ran some of the tests, controlling
for CPU defs in the kernel. Posted at http://www.tancsa.com/blast.html
Statistically, I think the results are too close to say they are different.
---Mike
Steven Hartland
> I cvsup'd to todays kernel and re-ran some of the tests, controlling
> for CPU defs in the kernel. Posted at
> http://www.tancsa.com/blast.html
>
> Statistically, I think the results are too close to say they are
> different.
Whats wrong with that web page the display is totally broken :(
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postm...@multiplay.co.uk.
Mike Tancsa
>Whats wrong with that web page the display is totally broken :(
Try it now.
---Mike
Divacky Roman
> At 04:03 PM 11/24/2006, Divacky Roman wrote:
> >On Fri, Nov 24, 2006 at 03:27:40PM -0500, Mike Tancsa wrote:
> >> At 03:28 AM 11/24/2006, Massimo Lusetti wrote:
> >> >On Thu, 2006-11-23 at 11:52 -0500, Mike Tancsa wrote:
> >> >
> >> >> I might give OpenBSD a quick try as a reference.
> >> >
> >> >That would be very interesting.
> >>
> >> OpenBSD 4.0 i386 panics on boot.
> >>
> >> I also posted some results with PMC compiled into the kernel
> >>
> >> ipfw compiled into the kernel, with 1 rule
> >>
> >> http://www.tancsa.com/pmc/
> >
> >I see generic_bzero/bcopy used quite often. why dont you define
> >cpu I586_CPU
> >in your kernel config?
>
> Hi,
>
> I had
>
> cpu I486_CPU
> cpu I586_CPU
> cpu I686_CPU
performance because optimized bzero/bcopy is included (its not
included if you only have I686_CPU).
I dont understand why the generic version is used.
Ivan Voras
> hm.. now I am confused. the rule is that having I586_CPU improves
> performance because optimized bzero/bcopy is included (its not
> included if you only have I686_CPU).
>
> I dont understand why the generic version is used.
I believe the consensus was that I486 line disables it, i.e. to leave
just I586 and I686 in.
Matthew D. Fuller
Divacky Roman, and lo! it spake thus:
>
> hm.. now I am confused. the rule is that having I586_CPU improves
> performance because optimized bzero/bcopy is included (its not
> included if you only have I686_CPU).
Haven't we been by this before? It's not included even if you have
I586_CPU either.
See src/sys/i386/isa/npx.c, line 432:
#ifdef I586_CPU_XXX <--------
^^^
This has been disabled since r1.95, in 2001 (in 5-CURRENT days).
There may be SOMETHING about including I586_CPU that speeds things up,
but it ain't that.
--
Matthew Fuller (MF4839) | full...@over-yonder.net
Systems/Network Administrator | http://www.over-yonder.net/~fullermd/
On the Internet, nobody can hear you scream.
Nick Pavlica
Mike,
Have you done any testing on Solaris 10, or OpenSolaris? I
understand that it has a very robust IP stack. It would be
interesting to see how the three stack up against each other (FBSD,
LINUS, SunOS).
--Nick
Mike Tancsa
I installed OpenSolaris elsewhere this week. Yeah, might be
interesting to try out. I will give it a test on Monday if the NICs
are supported. ipfilter seems to be the firewall software.
---Mike
Divacky Roman
> On Sat, Nov 25, 2006 at 09:36:05AM +0100 I heard the voice of
> Divacky Roman, and lo! it spake thus:
> >
> > hm.. now I am confused. the rule is that having I586_CPU improves
> > performance because optimized bzero/bcopy is included (its not
> > included if you only have I686_CPU).
>
> Haven't we been by this before? It's not included even if you have
> I586_CPU either.
>
> See src/sys/i386/isa/npx.c, line 432:
>
> #ifdef I586_CPU_XXX <--------
> ^^^
>
> This has been disabled since r1.95, in 2001 (in 5-CURRENT days).
> There may be SOMETHING about including I586_CPU that speeds things up,
> but it ain't that.
the log says:
People are still having problems with i586_* on UP machines and SMP
machines, so just hack it to disable them for now until it can be fixed.
noone has been able to step up and fix that? for more then 5 years?
huh.. I guess thats quite low hanging fruit worth shoting down..
Mike Tancsa
>
>Mike,
> Have you done any testing on Solaris 10, or OpenSolaris? I
>understand that it has a very robust IP stack. It would be
Did a quick default install. Results are not so interesting since one
stream livelocks the box. Basic stats at http://www.tancsa.com/blast.html
If there are some OpenSolaris wizards out there who want me to tune,
I am happy to retest...
---Mike
Massimo Lusetti
Mike Tancsa <mi...@sentex.net> wrote:
> OK, I added OpenBSD to the mix as well. Results are pretty crappy
> with the base default install. With one stream, the box essentially
> live locks. This was just with the stock kernels from the CD. The
> PCIe bge nics dont work, so I cant test those. I had a look at their
> errata page and there seems to be some updates to those 2 nics so if
> there is interest I can try compiling in those fixes and re-testing
FWIW I would definitively like to see it. But thanks for going so far..
--
Massimo
Mike Tancsa
I will give it another try tomorrow as I dont have much time today.
However, I did one test with a new CPU. I changed the cpu from an AMD
3800 to an AMD4600. So going from a CPU with a clock speed of 2Ghz to
2.4Gz. The difference is sort of what one would expect. In the ipfw
tests, (with a sample of 40 seconds) opposite path forwarding with 10
ipfw rules went from 208 Kpps to 235 Kpps or about a 10% increase as
opposed to the ~ 15% increase in clock speed. Perhaps bus is more of
a limiting factor that CPU but then again some the difference could
be explained with normal variance.
---Mike
Mike Tancsa
>
> > I might give OpenBSD a quick try as a reference.
>
>That would be very interesting.
OK, I added OpenBSD to the mix as well. Results are pretty crappy
with the base default install. With one stream, the box essentially
live locks. This was just with the stock kernels from the CD. The
PCIe bge nics dont work, so I cant test those. I had a look at their
errata page and there seems to be some updates to those 2 nics so if
there is interest I can try compiling in those fixes and re-testing
---Mike
Mike Tancsa
>FWIW I would definitively like to see it. But thanks for going so far..
Tried it with the patch branch. With the em nics, the box locks up
with 2 streams. It works now with bge, but rates are pretty slow
(220Kpps), and very slow with pf enabled. This was done with
SMP. While forwarding, the box is not responsive at all.
---Mike
Mike Tancsa
Did some more tests, this time using a single NIC interface in
trunking mode. Strangely enough, the speed is a little faster on
HEAD. Perhaps less interrupt processing ? Results in the usual place
http://www.tancsa.com//blast.html
---Mike
Robert Watson
You may want to datestamp the version of HEAD you're using in each test
(assuming it changes between tests). Among other things, I set
net.isr.direct=1 as the default a day or two ago, which will affect a broad
range of performance characteristics. There have also been VLAN tagging
related performance improvements made in HEAD that aren't present in RELENG_6,
avoiding extra memory allocations and frees in storing the tags.
Robert N M Watson
Computer Laboratory
University of Cambridge
Mike Tancsa
>You may want to datestamp the version of HEAD you're using in each
>test (assuming it changes between tests).
Ahh, Good point. I have been using the sources always from Nov 24th
to make comparisons as similar as possible for the tests with 7.0
>which will affect a broad range of performance
>characteristics. There have also been VLAN tagging related
>performance improvements made in HEAD that aren't present in
>RELENG_6, avoiding extra memory allocations and frees in storing the tags.
I ran the tests against RELENG_6 then as well and have posted those
results. There does indeed seem to be a slight improvement in HEAD
over RELENG_6.
---Mike
Nick Pavlica
Mike,
I'm not an OpenSolaris/Solaris expert, but was curious which build
you were testing with. I have noticed various results with my testing
depending on which build I was using. I have done most of my
testing/learning on Solaris 10 06/06, and have played with Solaris
Express B50 with good results. I did see some quirks in
SolarisExpress CE or B52 at the time of this writing. Of course I
patched all of these boxes before I did my testing which was mostly
centered around disk I/O performance on UFS and ZFS, and some
experimentation with Zones/Containers. I'm surprised that the console
locked up during your tests. My limited experience with Solaris 10+
thus far has been positive in terms of performance and stability.
When I have stressed my test systems, they remained responsive and
seemed to have better performance than FC6 and Ubuntu6.10 when
copying large files across my network.
Thanks for digging in with this testing, I hope you keep at it.
--Nick
Mike Tancsa
>>Did a quick default install. Results are not so interesting since one
>>stream livelocks the box. Basic stats at http://www.tancsa.com/blast.html
>>
>>If there are some OpenSolaris wizards out there who want me to tune,
>>I am happy to retest...
>
>Mike,
> I'm not an OpenSolaris/Solaris expert, but was curious which build
>you were testing with.
Hi,
I grabbed the latest DVD bits that were available at the time.
# uname -a
SunOS interlope 5.11 snv_52 i86pc i386 i86pc
>SolarisExpress CE or B52 at the time of this writing. Of course I
>patched all of these boxes before I did my testing which was mostly
>centered around disk I/O performance on UFS and ZFS, and some
>experimentation with Zones/Containers.
Didnt do any patches. The only thing I did was kill off X and disable
and enable ipfilter. Its quite possible there was other cruft
running that I didnt know about, but like I said, this was my first
exposure to OpenSolaris so I have no idea if there are things I
should have set.
> I'm surprised that the console
>locked up during your tests.
>My limited experience with Solaris 10+
>thus far has been positive in terms of performance and stability.
It does recover afterwards, but pretty well all other processes stop
as the CPU I guess is pegged dealing with all the interrupts.
Thinking further about my tests, it doesnt really do that great of a
job of simulating normal real world conditions. In the real world,
the packet sizes will vary and the speeds will be all over the
place. I am wondering if some of these modern nics have that in mind
with their design. But then again, this is sort of the scenario
when a firewal gets blasted by a high PPS attack :(
>When I have stressed my test systems, they remained responsive and
>seemed to have better performance than FC6 and Ubuntu6.10 when
>copying large files across my network.
But thats pretty different then my test setup. All the OSes I tested
can do that no problem :)
>Thanks for digging in with this testing, I hope you keep at it.
Yeah I inadvertently slighted the NetBSD folks by leaving them
out. So I guess I better give them a try as well.
The part that really surprises me is the drop in performance as
firewall rules are added to RELENG_6 and above. Both LINUX and
RELENG_4 seem to scale well with the number of rules added but
RELENG_6 takes a big drop.
---Mike
Ivan Voras
> Yeah I inadvertently slighted the NetBSD folks by leaving them out. So
> I guess I better give them a try as well.
>
> The part that really surprises me is the drop in performance as firewall
> rules are added to RELENG_6 and above. Both LINUX and RELENG_4 seem to
> scale well with the number of rules added but RELENG_6 takes a big drop.
Wasn't there some important setting in ipfw you can tweak if you need
lots of ipfw rules? Size of some hash table?
Quick Googling found this: http://info.iet.unipi.it/~luigi/ip_dummynet/
and net.inet.ip.fw.dyn_buckets: 256. AFAIK the hash size needed to be
tweaked manually in the code, and net.inet.ip.fw.dyn_buckets: 256 is
listed as read-only so this might be it. Maybe mailing Luigi will help
finding out...
Mike Tancsa
I was told offlist "there is additional per-packet locking overhead
not seen in RELENG_4 where all processing is covered by the same spl."...
---Mike