*** Failed allocating statistic counter

[Andreas Bourges]

Hi Team,

...customer had to move to a new server for their production environment. During this move, TRex version changed from v2.92 to v3.06 and network card changed from X710 to E810. After migrating to the new server, customer is facing now a similar problem I already discovered some time ago (https://groups.google.com/g/trex-tgn/c/lUQI78iXhe4).

New network card: 6c:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)

But the problem then was that I had enabled latency stats for all streams at that time. Now, latency stats are disabled, but I still can only add 127 streams - adding the 128th raises the following error:

Port 0 : *** Failed allocating statistic counter. Probably all are used for this rule type

  File "/opt/TREX/CURRENT/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
    ret = f(*args, **kwargs)
  File "/opt/TREX/CURRENT/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 48, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/TREX/CURRENT/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 850, in start
    raise TRexError(rc)
trex.common.trex_exceptions.TRexError: ESC[1mPort 0 : *** Failed allocating statistic counter. Probably all are used for this rule type.ESC[22m

Any hint on where to start tracking this down?

Thanks and best regards,

Andreas

[hanoh haim]

Hi Andres, E810 is not supported due to intel DPDK driver issues. 

I would keep the XL710 

Thanks

Hanoh

[Haklat haklat]

Hi, I have had same issue with Nvidia, even in SW mode it is limited to 127.
Now checking, I think maybe limit is set in trex_driver files. There is some comment there about reasoning for setting to 127
/scratch/trex-core-3.05/src/drivers]# grep num_counters *

trex_driver_base.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) = 0;
trex_driver_base.h:    void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) {
trex_driver_base.h:         m_real_drv->get_rx_stat_capabilities(flags, num_counters, base_ip_id);
trex_driver_bnxt.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) { }
trex_driver_i40e.cpp:void CTRexExtendedDriverBase40G::get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) {
trex_driver_i40e.cpp:        num_counters = MAX_FLOW_STATS;
trex_driver_i40e.cpp:        num_counters = MAX_FLOW_STATS_X710;
trex_driver_i40e.cpp:    m_max_flow_stats = num_counters;
trex_driver_i40e.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id);
trex_driver_ice.cpp:void CTRexExtendedDriverIce::get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) {
trex_driver_ice.cpp:    num_counters = 127; //With MAX_FLOW_STATS we saw packet failures in rx_test. Need to check.
trex_driver_ice.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id);
trex_driver_igb.cpp:void CTRexExtendedDriverBase1G::get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) {
trex_driver_igb.cpp:        num_counters = MAX_FLOW_STATS;
trex_driver_igb.cpp:        num_counters = UINT8_MAX;
trex_driver_igb.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id);
trex_driver_igc.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) { }
trex_driver_ixgbe.cpp:void CTRexExtendedDriverBase10G::get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) {
trex_driver_ixgbe.cpp:        num_counters = MAX_FLOW_STATS;
trex_driver_ixgbe.cpp:        num_counters = 127;
trex_driver_ixgbe.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id);
trex_driver_mlx5.cpp:void CTRexExtendedDriverBaseMlnx5G::get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) {
trex_driver_mlx5.cpp:    num_counters = 127; //With MAX_FLOW_STATS we saw packet failures in rx_test. Need to check.
trex_driver_mlx5.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id);
trex_driver_ntacc.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) {
trex_driver_ntacc.h:        num_counters = MAX_FLOW_STATS;
trex_driver_vic.cpp:void CTRexExtendedDriverBaseVIC::get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) {
trex_driver_vic.cpp:    num_counters = MAX_FLOW_STATS;
trex_driver_vic.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id);
trex_driver_virtual.cpp:void CTRexExtendedDriverVirtBase::get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id) {
trex_driver_virtual.cpp:    num_counters = MAX_FLOW_STATS;
trex_driver_virtual.h:    virtual void get_rx_stat_capabilities(uint16_t &flags, uint16_t &num_counters, uint16_t &base_ip_id);

[Haklat haklat]

Hi,
I finally tried compiling with:
- trex_driver_mlx5.cpp:    num_counters = 127; //With MAX_FLOW_STATS we saw packet failures in rx_test. Need to check.
+  trex_driver_mlx5.cpp:    num_counters =  MAX_FLOW_STATS  ; //With MAX_FLOW_STATS we saw packet failures in rx_test. Need to check.

Think this solved at least my specific case lacking PGID counters  when using Nvidia - not sure yet if any bad side effects with packet failures but not sending much for this specific case.

BR//Håkan