Unresponsive 7010....need diagnostic help

Vance Huntley

unread,

Feb 10, 1997, 3:00:00 AM2/10/97

to

I am baffled by our Cisco 7010 (IOS 11.1). The configuation:

Appletalk is routed on all three ethernet interfaces. IP is bridged.

The router is functioning OK; we pump almost 20 GB/day out through it.
However, many operations are impossibly slow. SNMP inquiries to the
router often time out. Pinging the router from any of the LAN's results
in about 40-60% packet loss, while pinging to locations on the internet
(i.e. _thru_ the router) have much less packet loss. It is impossible to
log into the router via telnet - simple commands take many minutes to echo
back, much less return results. "sh proc" returns ~%22 CPU utilization,
which seems high, considering that the traffic level on the T3 is hovering
around 4 Mb.

Until recently, all of this has been little more than an annoyance. Now,
Linux hosts on the 100Mb LAN are experiencing "no route to host" problems
when ftp-ing to hosts on the other LAN's and on the internet. I suspect
that arp on the Linux boxes is timing out when trying to reach the router,
just as SNMP requests do.

I earlier suspected a wiring problem. However, this seems unlikely since
the problems occur on every interface. Also, the entire 100Mb setup,
including wiring, hub, and Cisco interface has been swapped out for new
equipment.

Any suggestions? What should I investigate? I will gladly provide
additional information to anyone with a theory.....

Thanks in advance for your help--

Vance

--
------------
Vance Huntley

Director of Technology & GenesisJive Guy
WebGenesis, Inc., Ithaca, NY 14850

va...@webgenesis.com --- 607.255.8499

Check out "The Globe"! --- http://www.theglobe.com/
------------

Tony Li

unread,

Feb 11, 1997, 3:00:00 AM2/11/97

to

This smacks of an overloaded CPU. You might check out "show process" in
more detail. Yes, 22% is high, but AT is fast switched, so that's
not too surprising.

Also you wanna watch you interrupt load and look for packet drops. If
you can route IP and enable autonomous switching, you might buy back
some CPU.

Tony

Vance Huntley

unread,

Feb 13, 1997, 3:00:00 AM2/13/97

to

In article <330171...@jnx.com>, t...@jnx.com wrote:

>This smacks of an overloaded CPU. You might check out "show process" in
>more detail. Yes, 22% is high, but AT is fast switched, so that's
>not too surprising.
>
>Also you wanna watch you interrupt load and look for packet drops. If
>you can route IP and enable autonomous switching, you might buy back
>some CPU.
>
>Tony
>
>Vance Huntley wrote:
>>
>> I am baffled by our Cisco 7010 (IOS 11.1). The configuation:
>>

[ Lengthy ASCII-art description of our network]

>>
>> Until recently, all of this has been little more than an annoyance. Now,
>> Linux hosts on the 100Mb LAN are experiencing "no route to host" problems
>> when ftp-ing to hosts on the other LAN's and on the internet. I suspect
>> that arp on the Linux boxes is timing out when trying to reach the router,
>> just as SNMP requests do.
>>

OK, I have followed up on some of Tony Li's suggestions, and discovered
the following:
Interfaces:
- Less than 0.5% error rate on all interfaces
- only 195 drops ( input on 1 of our 10Mb ether interfaces, out of
more than 2 billion packets input )
- high (~20%) collision rate on same 10Mb ethernet as above

Buffers:
- Very good. Only 3 misses of any kind vs. 90 million hits

CPU: This morning, around 14%. Tony's suggestion that AT was a CPU hog
made sense to me, but the numbers dont seem to support it. Here are the
biggest CPU users
2 225882984 2120192 106538 0.00% 1.18% 1.76% 0 Check heaps
44 19366140 1296024 14942 0.00% 0.30% 0.18% 0 IP SNMP
19 15254988 12984756 1174 0.00% 0.09% 0.08% 0 IP Input
26 10704428 184207 58111 0.00% 0.08% 0.06% 0 IP Cache Ager
25 3862348 11229137 343 0.00% 0.02% 0.01% 0 IP Background
35 3196356 184335 17339 0.00% 0.02% 0.00% 0 Per-minute Jobs
1 3137600 406611 7716 0.00% 0.01% 0.00% 0 RIP Router
9 2163320 4395881 492 0.00% 0.00% 0.00% 0 ARP Input
42 1199208 2378526 504 0.00% 0.00% 0.00% 0 HyBridge Input
33 1056456 11045712 95 0.00% 0.00% 0.00% 0 TTY Background
22 761316 1294123 588 0.00% 0.00% 0.00% 0 CDP Protocol
47 400856 11063689 36 0.00% 0.00% 0.00% 0 Spanning Tree
14 365092 2355989 154 0.00% 0.00% 0.00% 0 Env Mon
34 228352 1106351 206 0.00% 0.00% 0.00% 0 Net Input
38 71100 81946 867 0.40% 0.15% 0.07% 0 AT NBP
48 69140 3229896 21 0.00% 0.00% 0.00% 0 AT Domain
20 24032 2214087 10 0.00% 0.00% 0.00% 0 TCP Timer
36 15340 47178 325 0.00% 0.03% 0.01% 0 AT Input

Interrupt level stacks:
Level Called Unused/Size Name
3 759330789 1476/2000 CXBus Interfaces
4 3 1580/2000 CXBus hot stall
5 796507 1900/2000 Console UART

I've tried disabling SNMP, since it is the only significant CPU user which
I know how to turn off. However, our symptoms (failed pings to/from
router to machines on the LAN, "no route to host" errors connecting
machines on other ethernet segments, etc.) persist. Once again, I am
stumped. Any ideas, anyone?

Thanks in advance for any suggestions you can provide.

Peter P. Morrissey

unread,

Feb 14, 1997, 3:00:00 AM2/14/97

to

In article <vance-13029...@newsstand.cit.cornell.edu> va...@webgenesis.com (Vance Huntley) writes:
...

>I've tried disabling SNMP, since it is the only significant CPU user which
>I know how to turn off. However, our symptoms (failed pings to/from
>router to machines on the LAN, "no route to host" errors connecting
>machines on other ethernet segments, etc.) persist. Once again, I am
>stumped. Any ideas, anyone?

>Thanks in advance for any suggestions you can provide.

One thing that would help a lot would be to put an analyzer on both
networks and trace say an ftp between the two hosts on different LANS,
or even trace the pings on both lans between the same two network
addresses. At least that's what I usually do in cases like this. You
can then see if packets are getting dropped, or hosts are not responding,
and in what direction the problems are happening.

Usually if the Cisco router is the cause, you will see the "drop" and/or
"ignore" counters going up.

Pete Morrissey
Syracuse University

Steve Schallehn

unread,

Feb 14, 1997, 3:00:00 AM2/14/97

to

In <330171...@jnx.com> Tony Li <t...@jnx.com> writes:

>This smacks of an overloaded CPU. You might check out "show process" in
>more detail. Yes, 22% is high, but AT is fast switched, so that's
>not too surprising.

>Also you wanna watch you interrupt load and look for packet drops. If
>you can route IP and enable autonomous switching, you might buy back
>some CPU.

>Tony

>Vance Huntley wrote:
[...]

>> Appletalk is routed on all three ethernet interfaces. IP is bridged.

Like Tony mentioned, bridging might be the source of your troubles.

Our network at Iowa State had a problem recently with a bridging
broadcast storm. During the outage, access to the routers was severly
degraded, even from the console. Cisco and I could not determine the
source of the slowdown doing 'show proc cpu'. Disconnecting
the router from the network and then shutting off bridging was the
only immediate fix.

Apparently, IOS 11.1 does not show bridging processes in the
'show proc cpu'.

-Steve Schallehn
Network Engineer, Telecommunications
Iowa State University