Slow Network - Dropped Packets

[Steve Waltner]

I'm trying to get a 43P Model 140 up on the network and am getting
terrible performance with it. All high-speed network operations are slow
(X, FTP, NFS). I'm communicating between two ports on a Cisco 2948G
switch. It appears as though the AIX systems is dropping incoming
packets. Symptoms are as follows:

- Stutters on X screen updates across network. I'm running smit on the
AIX system and displaying it across the network. When moving windows
around so that large portions of the window are updated at one time, the
updates will pause for a second or two before being fully displayed.
- NFS transfers are SLOW
- FTP transfers run at about 1.8K/sec instead of the 1.2M/sec that you
should get across 10Base-T

When it stutters like this, I see the dropped packets on entstat
increase. It appears as though they are increasing due to CRC errors.
The system is talking to a Solaris 2.6 system on the network for NFS/FTP
and a HP Envizex II X terminal for X. Here is the output from a few
commands. Does anyone have an idea for a solution?

# entstat -t ent0
-------------------------------------------------------------
ETHERNET STATISTICS (ent0) :
Device Type: IBM PCI Ethernet Adapter (22100020)
Hardware Address: 00:06:29:ba:87:19
Elapsed Time: 12 days 0 hours 18 minutes 13 seconds

Transmit Statistics: Receive Statistics:
-------------------- -------------------
Packets: 661508 Packets: 4482114
Bytes: 299310835 Bytes: 736853624
Interrupts: 0 Interrupts: 4502600
Transmit Errors: 0 Receive Errors: 895
Packets Dropped: 0 Packets Dropped: 20825
Bad Packets: 0
Max Packets on S/W Transmit Queue: 7
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 1

Broadcast Packets: 288 Broadcast Packets: 3742375

Multicast Packets: 4 Multicast Packets: 0
No Carrier Sense: 0 CRC Errors: 20825
DMA Underrun: 0 DMA Overrun: 0
Lost CTS Errors: 0 Alignment Errors: 10440
Max Collision Errors: 0 No Resource Errors: 0
Late Collision Errors: 0 Receive Collision Errors:
0
Deferred: 0 Packet Too Short Errors:
949
SQE Test: 0 Packet Too Long Errors: 0
Timeout Errors: 0 Packets Discarded by
Adapter: 0
Single Collision Count: 0 Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 1

General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 1
Driver Flags: Up Broadcast Running
Simplex AlternateAddress 64BitSupport
# no -a
extendednetstats = 0
thewall = 32720
sockthresh = 85
sb_max = 1048576
somaxconn = 1024
clean_partial_conns = 0
net_malloc_police = 0
rto_low = 1
rto_high = 64
rto_limit = 7
rto_length = 13
inet_stack_size = 16
arptab_bsiz = 7
arptab_nb = 25
tcp_ndebug = 100
ifsize = 8
arpqsize = 1
ndpqsize = 50
route_expire = 1
send_file_duration = 300
fasttimo = 200
routerevalidate = 0
nbc_limit = 0
nbc_max_cache = 131072
nbc_min_cache = 1
nbc_pseg = 0
nbc_pseg_limit = 32720
strmsgsz = 0
strctlsz = 1024
nstrpush = 8
strthresh = 85
psetimers = 20
psebufcalls = 20
strturncnt = 15
pseintrstack = 12288
lowthresh = 90
medthresh = 95
psecache = 1
subnetsarelocal = 1
maxttl = 255
ipfragttl = 60
ipsendredirects = 1
ipforwarding = 0
udp_ttl = 30
tcp_ttl = 60
arpt_killc = 20
tcp_sendspace = 16384
tcp_recvspace = 16384
udp_sendspace = 9216
udp_recvspace = 41920
rfc1122addrchk = 0
nonlocsrcroute = 0
tcp_keepintvl = 150
tcp_keepidle = 14400
bcastping = 0
udpcksum = 1
tcp_mssdflt = 512
icmpaddressmask = 0
tcp_keepinit = 150
ie5_old_multicast_mapping = 0
rfc1323 = 0
pmtu_default_age = 10
pmtu_rediscover_interval = 30
udp_pmtu_discover = 1
tcp_pmtu_discover = 1
ipqmaxlen = 100
directed_broadcast = 1
ipignoreredirects = 0
ipsrcroutesend = 1
ipsrcrouterecv = 0
ipsrcrouteforward = 1
ip6srcrouteforward = 1
ip6_defttl = 64
ndpt_keep = 120
ndpt_reachable = 30
ndpt_retrans = 1
ndpt_probe = 5
ndpt_down = 3
ndp_umaxtries = 3
ndp_mmaxtries = 3
ip6_prune = 2
ip6forwarding = 0
multi_homed = 1
main_if6 = 0
main_site6 = 0
site6_index = 0
maxnip6q = 20
llsleep_timeout = 3
tcp_timewait = 1
tcp_ephemeral_low = 32768
tcp_ephemeral_high = 65535
udp_ephemeral_low = 32768
udp_ephemeral_high = 65535
delayack = 0
delayackports = {}
sack = 0
use_isno = 1
tcp_newreno = 0
tcp_nagle_limit = 65535

[Tony Rall]

It looks like your ethernet adapter is 10Mb/s half duplex. If it is
connected to a multispeed switch port, make sure the port is
configured for 10Mb/s half duplex - don't rely on auto-negotiation.

--
Tony Rall

[Jon Shelley]

That is an outrageous number of broadcasts being received! I would think you
should see this on all hosts on the subnet or vlan. Enter "tcpdump" and watch
your screen, you should see which host(s) is/are broadcasting. If that's not
coming from a bad adapter on your network, then you need some serious
segmentation. If tcpdump is too detailed to pull out the info you need, you
can run an iptrace and filter based on the destination of the broadcast
address on your network. Can't remember the syntax of this, but you can get
this off the man pages.

Jon

In article <39EB5D5E...@lsil.com>, Steve Waltner

[Jesper James]

Hi,

Are you running AIX 4.3.3 ML05??. I have similar problems on machines
(43P's
and B50's) running this ML when using the onboard NIC. Using a PCI card
NIC works fine. There were no problems like this runnig 433ML04.

Seems to only affect inbound traffic though, but makes the machines next
to useless on the network.

best regards,
Jesper James
j...@proactive.dk

[Steve Waltner]

I'll respond to the three responses I have seen so far.

Regarding the version of AIX, I can't tell for certain which ML I am running.
It appears to either be ML02 or ML04

# instfix -i | grep ML
All filesets for 4.3.1.0_AIX_ML were found.
All filesets for 4.3.2.0_AIX_ML were found.
All filesets for AIX43ML were found.
All filesets for 4.3.1.0_AIX_ML were found.
All filesets for 4.3.2.0_AIX_ML were found.
All filesets for 4.3.3.0_AIX_ML were found.
Not all filesets for 4330-02_AIX_ML were found.
All filesets for 4320-02_AIX_ML were found.
Not all filesets for 4330-03_AIX_ML were found.
Not all filesets for 4330-04_AIX_ML were found.

Regarding the auto-sensing... The system is attached to a auto-sensing port
on a switch, which had detected that it was 10Mb/half-duplex. I deactivated
the auto-sensing to lock it at 10/half, but this didn't help the performance.

Regarding the "large" amount of broadcast packets, really this isn't that
many packets. The system currently show 3,910,557 incoming broadcasts in just
under 13 days. That averages 3.5 broadcasts per second. Sure the broadcasts
account for 84% of the incoming traffic (total of 4,653,308 packets), but
this system is isolated on a switch so it only sees the broadcasts or the
packets specifically destined for it. Since the system is so slow, it isn't
doing anything besides my testing.

I did get an email about turning off the tcp_pmtu_discovery and
udp_pmtu_discovery. I tried that as well, but it didn't help. This feature
(Path MTU Discovery) is used to find the largest MTU to talk to systems on
remote networks. Since I'm talking between two systems on the same network,
this feature doesn't come into play.

One thing I noticed yesterday was that the Packets Dropped and CRC Errors
section of the output of "entstat -d ent0" have exactly the same number in
them. Any other ideas?

Jesper James wrote:

> Hi,
>
> Are you running AIX 4.3.3 ML05??. I have similar problems on machines
> (43P's
> and B50's) running this ML when using the onboard NIC. Using a PCI card
> NIC works fine. There were no problems like this runnig 433ML04.
>
> Seems to only affect inbound traffic though, but makes the machines next
> to useless on the network.
>
> best regards,
> Jesper James
> j...@proactive.dk
>
> Steve Waltner wrote:
> >

[Stephen Wylie]

Have you tried reseating the card/swapping the cable?
Steve Wylie

Steve Waltner wrote in message <39EC93A9...@lsil.com>...

[Hajo Ehlers]

Hi Walter,
for trouble shooting i would do the following:

1. How to isolate the problem:
I asume your 43P has ONLY one network adapter.

0: Check with a tool, that your adapter on your 43P is in the PHYSICAL
state it should be

a: Replace cable and check again
Fault: You have found the problem
OK: goto b:

b:get your maschine and the test maschine off your cisco and connect
your two machines together with a dump hub.
You can connect the hub to the cisco to.
43p----|HUB|---( use a new cable for this connection)----|cisco|
test---| |

Now test you performence between test machine and your 43p ( So do not
use the cisco for test data transfer)

....
let use know about the results:

regards
Hajo Ehlers

BTW: if you are using NFS in all cases ( ftp to a nfs mount ) set the
rsize and wsize to 2048.
Might be also a good idea to set "the wall" to a higher number.

Steve Waltner schrieb: