Router delays every 2 minutes
Patrick Cervicek
router, it responds always within 1ms. But every 2 minutes, the Router
latency is at ~800ms for nearly one second. This repeats every 2 minutes.
http://lisas.de/~patrick/temp/delay-rscs10.png
We have no paket loss. Problem is, this effects also VoIP, which will be
delayed, too.
What could be the reason for this periodically delays?
Patrick
Merv
Traffic thru the box is treated different than traffic to the box.
Ping to a device on the other side of thw switch; it is a much better
test of the switch latency.
Patrick Cervicek
> Ping to the router are treated as a low priority task.
>
> Traffic thru the box is treated different than traffic to the box.
Sombody told me that already. But ...
> Ping to a device on the other side of thw switch; it is a much better
> test of the switch latency.
... this doesn't help in this case. When I do VoIP to an other Vlan over
this router, VoIP is ugly in the same second.
anyb...@hotmail.com
This is not how network latency is usually defined.
What matters is the delay that packets passing through
the router are subjected to. This test is not
measuring that delay.
The 6500 (in general) forwards traffic in /hardware/
and the CPU is used for management tasks only.
When you ping the router you are pinging the CPU.
This is not necessarily a problem.
The delay of 200ms or so is however quit large and I
would want to try to get some idea of what was going on.
We see delays of a few 10s of ms when we ping
our 4500s (SE 4, 5).
This is /no indication whatsoever/ that you voice traffic
is at risk.
"What is the reason?" This could be hard to figure out.
Start by checking router CPU.
What happens on /your/ network every 2 mins?
Patrick Cervicek
>>We have one Cisco Router 6506. When I ping the local interface of the
>>router, it responds always within 1ms. But every 2 minutes, the Router
>>latency is at ~800ms for nearly one second. This repeats every 2 minutes.
>>http://lisas.de/~patrick/temp/delay-rscs10.png
>>We have no paket loss. Problem is, this effects also VoIP, which will be
>>delayed, too.
>>What could be the reason for this periodically delays?
>
>
> This is not how network latency is usually defined.
> What matters is the delay that packets passing through
> the router are subjected to. This test is not
> measuring that delay.
I pinged other devices from other vlans. The delay is the same every 2
minutes. Hosts in the same vlan are not affected.
> The 6500 (in general) forwards traffic in /hardware/
> and the CPU is used for management tasks only.
> When you ping the router you are pinging the CPU.
> This is not necessarily a problem.
> The delay of 200ms or so is however quit large and I
> would want to try to get some idea of what was going on.
> We see delays of a few 10s of ms when we ping
> our 4500s (SE 4, 5).
>
> This is /no indication whatsoever/ that you voice traffic
> is at risk.
Actually, it does. VoIP was the reason why I found this bug with our
router. As this is our only realtime-application, we didn't notice it
earlier with other TCP/UDP-Applications.
> "What is the reason?" This could be hard to figure out.
> Start by checking router CPU.
>
> What happens on /your/ network every 2 mins?
I don't know as I am not the network admin, I report only errors to him ;-)
I did a check with ethereal but I could not find anything exceptional
(broadcast, Multicast).
Merv
If pings thru the switch are delayed every 2 minutes then something is
amiss.
If you are not the network administrator then it is going to be hard to
help you as responders would need some output from the router in
question in order to be able to assist. Perhaps you can get you
netadmin to post required info
Required output for troubleshooting:
sh version
sh process cpu
sh ip cef
sh ip arp sum
sh running-config
sh int status
sh int
Phillip Remaker
I can't think of any normal periodic activity that happens every *two*
minutes.
Need more clues. Does the traffic level also spike at that time?
HeadsetAdapter.com
message news:e47kn6$4q$1...@news.BelWue.DE...
As it was told before, it'd be hard to troubleshoot since you are not
Network Admin. I would recommend to connect to the Console on the switch,
and see if you see any messages at the time when you have your switch
delays. Somehting MUST be in the log - errors, warnings, activity reported,
etc. It could be your starting point.
Good luck,
Mike
----
"Headset Adapters for a Cisco IP Phone" www.headsetadapter.com
Patrick Cervicek
> As it was told before, it'd be hard to troubleshoot since you are not
> Network Admin. I would recommend to connect to the Console on the switch,
> and see if you see any messages at the time when you have your switch
> delays. Somehting MUST be in the log - errors, warnings, activity reported,
> etc. It could be your starting point.
>
Today I informed the network admin. He will handle this problem and
contact this group within the next days.
As far as we saw, there were nothing exceptional in the logs. Do we need
a certain debug-/log-flag set?
Patrick
Patrick Cervicek
> Are there any ugly spikes in CPU or "CPUHOG" messages in syslog?
We didn't see any 'CPU' messages.
> I can't think of any normal periodic activity that happens every *two*
> minutes.
>
> Need more clues. Does the traffic level also spike at that time?
No. Nothing in syslog and nothing in mrtg.
Patrick Cervicek
Pings to other vlans (Esslingen,same location) are not affected - but
one. We have a BSD-VPN-Router connected to this interface/vlan, which
is connected to an other router (B) at an other location. All Hosts
after B are delayed after 2 minutes for one second
Ping_Host <-> Router_A <-> BSD-VPN-Router <-> VPN_Router_B <->
$any_HOST_at_B
The interesting part is, that pings to the BSD-Router and Router B are
never affected.
Here you have an mtr-graph from Ping_Host (look at Wrst)
> mtr -i 0.1 rglx00xx ($any_HOST_at_B) My traceroute [v0.67]
> rhlxx (Host_at_A) (0.0.0.0)(tos=0x0 psize=64 bitpattern=0x00) Mon May 15 17:32:09 2006
> Keys: Help Display mode Restart statistics Order of fields quit
> Packets Pings
> Host Loss% Snt Last Avg Best Wrst StDev
> 1. 134.108.56.xxx (Router_A) 0.0% 891 0.4 3.1 0.3 348.1 24.4
> 2. 134.108.73.xxx (BSD-VPN) 0.0% 891 0.5 0.5 0.3 2.3 0.2
> 3. 129.143.118.xxx (Router_B) 0.0% 891 2.5 3.0 2.2 7.4 0.4
> 4. 134.108.100.xxx ($any_HOST_at_B) 0.0% 891 7.5 6.3 2.5 313.8 21.2
I want to do VoIP with Location_B, but you see that everything is
delayed after 2 minutes (at the same time, when Route_A's Latency is
high , $any_HOST_at_B is too)
We will check that tomorrow,
Patrick
Patrick Cervicek
incoming-Accesslist. This Reflexive-ACL is very big.
We put the
"permit ip host $VPN_Router_B host $BSD-VPN !Allow VPN-Router_B to us"
*before* that statement in our Incoming-ACL, and now we have very good
ping results to location B. VoIP works!
> mtr -i 0.1 rglx00xx My traceroute [v0.67]
> rhlxxx (0.0.0.0)(tos=0x0 psize=64 bitpattern=0x00) Tue May 16 18:04:52 2006
> Keys: Help Display mode Restart statistics Order of fields quit
> Packets Pings
> Host Loss% Snt Last Avg Best Wrst StDev
> 1. 134.108.56.xxx (Router_A) 0.0% 805 5.9 6.2 0.4 327.6 28.3
> 2. 134.108.73.xxx (BSD-VPN) 0.0% 805 1.1 1.4 0.3 16.3 1.7
> 3. 129.143.118.xxx (VPN_Router_B) 0.0% 804 6.2 4.3 2.1 24.1 2.1
> 4. 134.108.100.xxx ($any_HOST_at_B) 0.0% 804 7.4 5.3 2.4 17.0 2.0
>
We think, that (efficient) Hardware-Routing was impossible as the
Hardware had always to lookup the dynamic ACL before passing traffic
through.
Thank you for your hints!
Patrick