Any way to eliminate bonding interface dropped packets?
Yong Huang
the network traffic is heavy, we see dropped packets. Is there any way
to improve this?
/etc/modprobe.conf (bond0 omitted here because it has much less
traffic and works fine):
alias eth2 e1000
alias eth3 e1000
install bond1 /sbin/modprobe bonding -o bond1 miimon=100 mode=1
primary=eth2
options e1000 FlowControl=1,1
$ uname -a
Linux <myhostname> 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007
x86_64 x86_64 x86_64 GNU/Linux
Note the 52 packets dropped since bootup. It always happens to
receiving.
$ ifconfig bond1
bond1 Link encap:Ethernet HWaddr 00:1B:78:58:F4:9C
inet addr:192.168.2.11 Bcast:192.168.2.255 Mask:
255.255.255.0
inet6 addr: fe80::21b:78ff:fe58:f49c/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:150304549 errors:0 dropped:52 overruns:0 frame:0
TX packets:148948403 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:111541084012 (103.8 GiB) TX bytes:109717011822
(102.1 GiB)
eth2 is the primary or active slave (eth3 is passive).
$ ifconfig eth2 | grep dropped
RX packets:150487351 errors:0 dropped:52 overruns:0 frame:0
TX packets:149129271 errors:0 dropped:0 overruns:0 carrier:0
Is there any magic we can add here?
$ grep -v ^# /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.ip_local_port_range=1024 65000
fs.file-max=1515611
kernel.sem=250 32000 100 128
kernel.shmmax=8589934592
kernel.shmall=4194304
kernel.shmmni=4096
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 1048576
net.core.wmem_max = 1048576
vm.nr_hugepages = 5120
Receiving flow control is on:
# ethtool -a eth2
Pause parameters for eth2:
Autonegotiate: on
RX: on
TX: off
Thanks.
Yong Huang
David Schwartz
> We're using two e1000 interfaces bonded to one pseudo interface. When
> the network traffic is heavy, we see dropped packets. Is there any way
> to improve this?
I'm sorry you are seeing a .000000345% packet loss rate. Ethernet is
best effort, and this seems pretty darned good to me. If you can't
deal with a packet loss rate this low, something is *very* wrong with
how you are using it.
DS
David Schwartz
Sorry, should be .0000346%, but the point stands. That's close enough
to perfect as had better make no difference.
DS
buck
<yon...@yahoo.com> wrote:
>$ ifconfig bond1
>bond1 Link encap:Ethernet HWaddr 00:1B:78:58:F4:9C
> inet addr:192.168.2.11 Bcast:192.168.2.255 Mask:
>255.255.255.0
> inet6 addr: fe80::21b:78ff:fe58:f49c/64 Scope:Link
> UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
> RX packets:150304549 errors:0 dropped:52 overruns:0 frame:0
> TX packets:148948403 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:111541084012 (103.8 GiB) TX bytes:109717011822
>(102.1 GiB)
>Thanks.
>
>Yong Huang
I know nothing of bonding, but can you set txqueuelen (anywhere, not
just on bond1)? The zero size says there is no buffer. This probably
won't help because RX is the problem, but I've had some (small)
success with other devices that have no buffer by setting txqueuelen
to 1000.
--
buck
Yong Huang
> On Thu, 13 Mar 2008 15:36:32 -0700 (PDT), Yong Huang
>
Thanks, buck. I did 'ifconfig bond1 txqueuelen 1000' (eth2 and eth3
are already 1000). The rate of dropped packets increase remains the
same.
To David, thanks for your comments. I think I have to agree with you
we're already near perfection so we'll probably live with it. We're
stress testing this 8-node Oracle RAC (cluster).
Yong Huang