Utilization numbers not corresponding with switch

[Paul Manno]

Hi all,

I am monitoring an HP 5412zl with sFlowTrend.  The software is collecting data, but many of the values seem very wrong.  For example, I have two 10Gbps links trunked together.  If I log into the switch and inspect the port utilization for the ports in the trunk, the switch reports 00.50% util for both tx and rx on each of the trunk ports, while sFlowTrend shows the usage at 50% or higher.  I don't know how sFlowTrend is figuring out the utilization numbers, but something appears to be misconfigured somewhere here.  When I look at the interface in sFlowTrend, it is reporting 20 Gbps speed, which is correct.  Just wondering what I could be doing wrong here.

Thanks for the help.
Paul

[Stuart Johnston]

Hi Paul,

Where are you seeing the incorrect utilisation - on the counters tab, or top-n? Could you compare the two and let me know how they compare, and what the respective values are? Also, it would be good to compare the frames/second in the top-n with the counters frames/second.

Would you be able to take a pcap trace of the sFlow being received by sFlowTrend, so we can look at it further? If you could, we would need a minimum of 5 minutes. We would have to find somewhere for you to upload to.

thanks,

Stuart

--
You received this message because you are subscribed to the Google Groups "sFlowTrend" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sflowtrend+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Manno]

Hi Stuart,

On the switch, when I do a:

show interfaces port-utilization

I get:
                                 Rx                                      Tx
 Port      Mode        | ---------------------------            | ---------------------------
                             | Kbits/sec   Pkts/sec  Util   | Kbits/sec  Pkts/sec   Util
 -------- --------- + ---------- ---------- ----- + ---------- ---------- -----
L1-Trk10 10GigFD   | 387648     28486      03.87 | 50016      487        00.50
L2-Trk10 10GigFD   | 49960       316         00.49 | 50800      889        00.50

sFlowTrend Network tab, Interfaces Tab for Trk10 shows
Speed 20G
Util around 8K (yes K... which is 8000%?)

sFlowTrend Thresholds tab for Trk10 shows
Speed 20 Gbps
Utilization: 90%
Unicasts: 75%
Broadcasts: 90%
Multicasts: 90%

Maybe it's just a threshold thing.. my dashboard thresholds are all red, but the switch is hardly doing anything.  I've attached some screen shots.

If you need the pcap I can figure that out for you, but let me know where this info gets you.

Thanks,
Paul

[Stuart Johnston]

Hi Paul,

I think this problem is caused by a bug in the ProCurve sFlow implementation - the sFlow counters for a trunk get stuck, and then eventually are sent out. This causes a very spiky utilisation. I’ll try to find out more about the status.

regards,

Stuart

<sflowtrend.zip>

[Stuart Johnston]

Paul,

We’re not sure about the exact cause. What sFlow counter polling interval do you have configured on the switch for this interface?

If you could get us a pcap that would be great, it would at least tell us what is wrong. I saw that you registered at myinmon.com - the best way to transfer the cap would be to use the file transfer link under Support, in the left hand menu. Please let me know when you’ve uploaded it.

thanks,

Stuart

[Paul Manno]

Hi Stuart,

Sampling @ 1024, Polling @ 30

Uploading that file (dumpcap.pcap) now.

[Sonia Panchen]

Hello Paul,

Thanks for uploading the pcap trace, that was very useful. 

We have looked at the trace using sflowtool http://www.inmon.com/technology/sflowTools.php. The problem is that the counter samples for two interfaces in particular (ifIndex 299 (a trunk) and ifIndex 293) are reporting strange (incorrect) values. The counters should always increase in value (except on a wrap). However, what we observe is that every few samples, the values are reported as 0. This means that when you take deltas between consecutive counters to calculate rates and utilisation, you will get the results that you are seeing. This would appear to be a problem with the sFlow implementation in the HP 5412zl switch. I think we have seen this problem before, but don’t remember if a fix is available. Are you able to discuss this with HP support?

To look at this for yourself:

sflowtool -l -r dump cap.pcap > dumpcap.txt

grep "CNTR,10.5.0.2,299” dumpcap.txt

The 8th column is the ifInOctets column (see http://www.sflow.org/SFLOW-STRUCTS5.txt if_counters for the columns).

Best regards,

Sonia Panchen

[Paul Manno]

Hi Sonia,

Thanks for the info.  I'll open a ticket with HP and see what they have to say.  I'll keep this thread updated with findings.

Best,
Paul

[Paul Manno]

Hi Sonia,

Finally had a maintenance window to apply the updates as indicated by HP Support.  They had me upgrade the firmware in this order:

upgrade the firmware to K.15.13.0014 --- K.15.15.0006 ---K.15.16.0011----K.15.18.0011

Since doing so, the information presented by SFlowTrend corresponds exactly with the switch CLI.  Looks like HP fixed the bug in one or more of those software updates.

Thanks for your help!
Paul

[Sonia Panchen]

Paul,

Thank you for letting us know that the problem is now resolved and the HP software version that fixes the problem.

Best regards,

Sonia Panchen

[Robin Clayton]

I found this bug last week myself when using HP PCM, after hours of tweaking and hacking away trying to work out why interface usage was  through the roof I saw in PCM UDP SLOW and SNMP packets being very high.

I looked through the firmware revisions on HP and found an BUG relating to SNMP/Sflow , which lead me to .

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_kc-0118894

I applied that last night and sure enough normality was returned to the world.

Key Points.

1) It's Cosmetic

2) It only affects trunked links.

Rob