Catalyst 3750G drops packets with IPv4 options
Spoon
I have a problem with a Cisco Catalyst 3750G switch/router.
In my tests, I could not get the 3750G to reliably route more than 3400
IPv4 packets per second when these packets contained IP options.
The packets were UDP/IPv4. I tried both small (payload = 100 bytes)
and large (payload = 1450 bytes) packets.
The IP options consisted of 4 or 8 NOPs (option type 1). Thus there
were no alignment issues, and nothing extra for the router to process.
3570 pps && payload= 100 : 1.1% packet loss
3570 pps && payload=1450 : 2.3% packet loss
5000 pps && payload=1450 : 30.2% packet loss
The following page does not mention my situation:
http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml
I ran show process cpu:
CPU utilization for five seconds: 87%/29%; one min: 89%; five min: 73%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
[...]
150 7678123 5843620 1313 54.63% 53.74% 43.74% 0 IP Input
A few entries show less than 0.15% CPU usage. All other show 0%.
Where does the 87% figure come from?
Is it 87% overall with 29% IRQ servicing => 58% other than IRQ.
Why is the router dropping packets if the CPU is not maxed out?
Is this a known problem?
Has it perhaps been fixed in a more recent firmware revision?
For reference:
>show version
Cisco IOS Software, C3750 Software (C3750-IPSERVICES-M), Version
12.2(25)SEC, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Thu 14-Jul-05 21:33 by antonino
ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SE, RELEASE
SOFTWARE (fc)
cyrus uptime is 1 week, 2 days, 18 hours, 44 minutes
System returned to ROM by power-on
System image file is "flash:/c3750-ipservices-mz.122-25.SEC.bin"
cisco WS-C3750G-24TS-1U (PowerPC405) processor (revision A0) with
118784K/12280K bytes of memory.
Processor board ID FOC0902U0GY
Last reset from power-on
17 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:13:19:86:CA:00
Motherboard assembly number : 73-9637-07
Power supply part number : 341-0098-01
Motherboard serial number : FOC09020A2U
Power supply serial number : DCA084701XH
Model revision number : A0
Motherboard revision number : A0
Model number : WS-C3750G-24TS-S1U
System serial number : FOC0902U0GY
SFP Module assembly part number : 73-7757-02
SFP Module revision Number : A0
SFP Module serial number : CAT085207B5
Top Assembly Part Number : 800-25730-01
Top Assembly Revision Number : A0
Version ID : 01
CLEI Code Number : CNMWS00ARA
Hardware Board Revision Number : 0x05
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 28 WS-C3750G-24TS-1U 12.2(25)SEC
C3750-IPSERVICES-M
Configuration register is 0xF
Regards.
flamer die.spam@hotmail.com
> Hello,
>
> I have a problem with a Cisco Catalyst 3750G switch/router.
>
> In my tests, I could not get the 3750G to reliably route more than 3400
> IPv4 packets per second when these packets contained IP options.
>
> The packets were UDP/IPv4. I tried both small (payload = 100 bytes)
> and large (payload = 1450 bytes) packets.
>
> The IP options consisted of 4 or 8 NOPs (option type 1). Thus there
> were no alignment issues, and nothing extra for the router to process.
>
> 3570 pps && payload= 100 : 1.1% packet loss
> 3570 pps && payload=1450 : 2.3% packet loss
> 5000 pps && payload=1450 : 30.2% packet loss
>
check the speed and duplex of the interface
Flamer.
Bo...@hotmail.co.uk
> Hello,
>
> I have a problem with a Cisco Catalyst 3750G switch/router.
>
> In my tests, I could not get the 3750G to reliably route more than 3400
> IPv4 packets per second when these packets contained IP options.
>
> The packets were UDP/IPv4. I tried both small (payload = 100 bytes)
> and large (payload = 1450 bytes) packets.
>
> The IP options consisted of 4 or 8 NOPs (option type 1). Thus there
> were no alignment issues, and nothing extra for the router to process.
>
> 3570 pps && payload= 100 : 1.1% packet loss
> 3570 pps && payload=1450 : 2.3% packet loss
> 5000 pps && payload=1450 : 30.2% packet loss
>
The 3750 is primarily a hardware basd router but
can fall back to CPU routing under certain circumstances.
You seem to have hit thse curcumstances.
It will doo wire rate hardware routing but the software
routing performance is poor. Further there are
two kinds of software routing (fast switched and
process switched) and you are getting a lot of
process switching.
CPU utilization for five seconds: 87%/29%; one min: 89%; five min:
73%
87% total CPU of which 29% is fast switching packets.
The "IP Input" process is the one that does the process switching.
If you look for documents that cover troubleshooting
high cpu that ar relevant to you platform you may get
something that explains all of the details.
So whatever your options are doing is causing the
switch to fall back to CPU based routing.
Spoon
> Spoon wrote:
>
>> I have a problem with a Cisco Catalyst 3750G switch/router.
>>
>> In my tests, I could not get the 3750G to reliably route more than 3400
>> IPv4 packets per second when these packets contained IP options.
>>
>> The packets were UDP/IPv4. I tried both small (payload = 100 bytes)
>> and large (payload = 1450 bytes) packets.
>>
>> The IP options consisted of 4 or 8 NOPs (option type 1). Thus there
>> were no alignment issues, and nothing extra for the router to process.
>>
>> 3570 pps && payload= 100 : 1.1% packet loss
>> 3570 pps && payload=1450 : 2.3% packet loss
>> 5000 pps && payload=1450 : 30.2% packet loss
>>
> check the speed and duplex of the interface
All interfaces involved are 100 Mbit/s and full duplex.
On the sender:
# ethtool eth2
Settings for eth2:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Current message level: 0x000020c1 (8385)
Link detected: yes
That interface is connected to port 17 on the 3750:
Name : Gi1/0/17
Administrative Speed: auto
Administrative Duplex: auto
Administrative Auto-MDIX: on
Administrative Power Inline: N/A
Operational Speed: 100
Operational Duplex: full
Operational Auto-MDIX: on
On the receiver:
Settings for eth2:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Current message level: 0x000020c1 (8385)
Link detected: yes
That interface is connected to port 19 on the 3750:
Name : Gi1/0/19
Administrative Speed: auto
Administrative Duplex: auto
Administrative Auto-MDIX: on
Administrative Power Inline: N/A
Operational Speed: 100
Operational Duplex: full
Operational Auto-MDIX: on
What problem do you suspect?
I failed to mention that the router doesn't drop any packets when
packets do not carry any IP option.
Vlan17 is up, line protocol is up
Hardware is EtherSVI, address is 0013.1986.ca48 (bia 0013.1986.ca48)
Internet address is 10.1.17.254/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 10/255
Encapsulation ARPA, loopback not set
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 21/75/35000/0 (size/max/drops/flushes); Total output
drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 41465000 bits/sec, 3474 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
43540876 packets input, 3580507536 bytes, 0 no buffer
Received 0 broadcasts (364590 IP multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
52036 packets output, 3613284 bytes, 0 underruns
0 output errors, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
Vlan19 is up, line protocol is up
Hardware is EtherSVI, address is 0013.1986.ca4a (bia 0013.1986.ca4a)
Internet address is 10.1.19.254/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 10/255, rxload 1/255
Encapsulation ARPA, loopback not set
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:27:22, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 41577000 bits/sec, 3469 packets/sec
289 packets input, 17454 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
33807273 packets output, 3212600286 bytes, 0 underruns
0 output errors, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet1/0/17 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0013.1986.ca11 (bia
0013.1986.ca11)
MTU 9000 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 122/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 48059000 bits/sec, 4019 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
56540589 packets input, 382292988 bytes, 0 no buffer
Received 745135 broadcasts (0 multicast)
0 runts, 0 giants, 0 throttles
1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 745042 multicast, 0 pause input
0 input packets with dribble condition detected
434362 packets output, 35087682 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet1/0/19 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0013.1986.ca13 (bia
0013.1986.ca13)
MTU 9000 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 105/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 41515000 bits/sec, 3471 packets/sec
7177 packets input, 4052852 bytes, 0 no buffer
Received 81 broadcasts (0 multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 18 multicast, 0 pause input
0 input packets with dribble condition detected
45385038 packets output, 1476776494 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
John
probably the switch goes to software forwarding as some people have
already said. I suppose that you have not configured any of the "ip
accounting, access-lists etc)"
what you have sent shows that probably the packets are dropped in the
input queue of SVI "Vlan17" (Input queue: 21/75/35000/0 (size/max/
drops/flushes).
You could try (as a workaround):
a. raise the input queue (eg hold-queue 1000 in)
b. try using l3 ports instead of SVI. (conf t; int fa0/17 ; no
switchport; ip address 10.1.17.254 255.255.255.0)
I guess that the best thing you can do is to upgrade to latest
software and if the problems perists, open a case with TAC.
--john
>
> >> Why is the router dropping packets if the CPU is not maxed out?
>
> >> Is this a known problem?
Thrill5
only a certain amount of memory that it uses for switching and routing. By
default most of the memory is allocated for MAC addresses, and if you have
more than 2000 routes you will run out of memory for unicast routes at which
point you are software routing packets. After issuing the command you will
need to reload the switch to make it effective.
Scott
<Bo...@hotmail.co.uk> wrote in message
news:1178585871.2...@e65g2000hsc.googlegroups.com...