*** Failed allocating statistic counter

[Andreas Bourges]

Hi,

..I seem to cross a limit with my current setup (added another 100 streams...), but all in all I only have 256 streams. I know that I had even more streams running some time ago, so I don't understand why this happens.

Would be happy for any hint what the problem could be ...

Thanks,

Andreas

trex is running in sw mode:

./t-rex-64 --iom 0 -c 4 -i --emu --software --cfg /etc/trex_cfg.yaml --no-key

This is the log from my script:

2021-11-08 11:28:16,325 [ERROR] [trexclient.stream_manager: 753] :   Traceback (most recent call last):

  File "/opt/trex_client/trexclient/stream_manager.py", line 715, in start_dataCollector

    if self.recv_from_frontend() == 0:

  File "/opt/trex_client/trexclient/stream_manager.py", line 789, in recv_from_frontend

    return self.handle_queue_result(s)

  File "/opt/trex_client/trexclient/stream_manager.py", line 842, in handle_queue_result

    self.c.start(ports=ports, duration=self.duration)

  File "/opt/TREX/CURRENT/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2

    ret = f(*args, **kwargs)

  File "/opt/TREX/CURRENT/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 49, in wrapper

    return func(self, *args, **kwargs)

  File "/opt/TREX/CURRENT/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 796, in start

    raise TRexError(rc)

trex.common.trex_exceptions.TRexError: ESC[1mPort 0 : *** Failed allocating statistic counter. Probably all are used for this rule type.ESC[22m

ESC[1mPort 0 : *** Failed allocating statistic counter. Probably all are used for this rule type.ESC[22m

[hanoh haim]

Hi Andreas,

The issue are not with the streams it is with the flow-stats collection 

[Andreas Bourges]

Hi Hanoh,

..yes, sorry, wrong wording. I'm using a distinct pg_id per stream - but I thought the limit in sw mode was ~1024 ?

Thanks,

Andreas

[hanoh haim]

Hi, 

Yes, in software mode, there are 1024 rules. when you stop the traffic the rules does not remove immediately 

verify it by running the add/start after clean load of the server 

Thanks

Hanoh

[andyb...@googlemail.com]

Ok, seems to be a problem in my code. Will go and see there first 😉

 

Thanks,

 

Andreas

[andyb...@googlemail.com]

HI,

 

…couldn’t find anything in my code. The problem pop’s after after adding more than 128 streams (each having unique pg_id) and then call start() ?! The same code on a different hardware works without any problem (>500 streams).

 

Any hint?

 

Thanks,

 

Andreas

 

Von: hanoh haim <hhaim...@gmail.com>
Gesendet: Montag, 8. November 2021 13:42
An: Andreas Bourges <andyb...@googlemail.com>
Cc: TRex Traffic Generator <trex...@googlegroups.com>
Betreff: Re: [trex-tgn] *** Failed allocating statistic counter

 

Hi, 

[hanoh haim]

What is the hardware type that it does not work?

--

Hanoh

Sent from my iPhone

[andyb...@googlemail.com]

 

Turns out, there are similar cards involved:

 

 

Does not work:

 

5e:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

        Subsystem: Intel Corporation Ethernet 10G 2P X710 Adapter

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0

        Interrupt: pin A routed to IRQ 285

        NUMA node: 0

        Region 0: Memory at ba000000 (64-bit, prefetchable) [size=16M]

        Region 3: Memory at bb008000 (64-bit, prefetchable) [size=32K]

        Expansion ROM at b8800000 [disabled] [size=512K]

        Capabilities: <access denied>

        Kernel driver in use: igb_uio

        Kernel modules: i40e

 

 

 

Works:

 

12:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

        Subsystem: Hewlett Packard Enterprise Ethernet 10GbE 4P 563SFP+ Adapter

        Physical Slot: 1

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 64 bytes

        Interrupt: pin A routed to IRQ 26

        NUMA node: 0

        Region 0: Memory at e0000000 (64-bit, prefetchable) [size=16M]

        Region 3: Memory at e4000000 (64-bit, prefetchable) [size=32K]

        Expansion ROM at e4080000 [virtual] [disabled] [size=512K]

        Capabilities: <access denied>

        Kernel driver in use: igb_uio

        Kernel modules: i40e

[hanoh haim]

Hi Andreas,

I did a quick test on the same setup, and it seems working fine. 

I've loaded the server with --software. could you verify that you have the same server version?

[csi-trex-09]> sudo ./t-rex-64 -i -c 7 --software

Starting  TRex v2.92 please wait  ...
 set driver name net_i40e
 driver capability  : TCP_UDP_OFFLOAD  TSO
 set dpdk queues mode to MULTI_QUE
 Number of ports found: 8

trex>start -f stl/flow_stats.py -t --num_streams 1000
Removing all streams from port(s) [0._, 1._, 2._, 3._, 4._, 5._, 6._, 7._]: [SUCCESS]
Attaching 1000 streams to port(s) [0._]:                     [SUCCESS]
Attaching 1000 streams to port(s) [1._]:                     [SUCCESS]
Attaching 1000 streams to port(s) [2._]:                     [SUCCESS]
Attaching 1000 streams to port(s) [3._]:                     [SUCCESS]
Attaching 1000 streams to port(s) [4._]:                     [SUCCESS]
Attaching 1000 streams to port(s) [5._]:                     [SUCCESS]
Attaching 1000 streams to port(s) [6._]:                     [SUCCESS]
Attaching 1000 streams to port(s) [7._]:                     [SUCCESS]
Starting traffic on port(s) [0._, 1._, 2._, 3._, 4._, 5._, 6._, 7._]: [SUCCESS]


thanks

Hanoh

[Andreas Bourges]

Hi Hanoh,

sorry for the delay, but we ran into another problem where only TRex generated UDP traffic is not crossing a firewall :-( Will probably open another thread for this, once we see where the error is coming from...

Regarding the the current problem:

...trex is normally started via trex_daemon_server and process list looks like this:

root      16483  0.0  0.0 488276 36752 ?        Ssl  Nov08   2:08 python3 /opt/TREX/CURRENT/trex_daemon_server start-live

root      17763  0.0  0.0 113284  1688 ?        S<s  Nov08   0:00  \_ /bin/bash ./t-rex-64 --iom 0 -c 4 -i --emu --software --cfg /etc/trex_cfg.yaml --no-key

root      17872  491  0.0 34569708 24160 ?      S<l  Nov08 12207:38      \_ ./_t-rex-64 --iom 0 -c 4 -i --emu --software --cfg /etc/trex_cfg.yaml --no-key

When started on CLI the output is this:

[root@mndemuclxr151 CURRENT]# ./t-rex-64 -c 4 -i --emu --software --cfg /etc/trex_cfg.yaml

Starting Scapy server.... Scapy server is started

Starting Emu server.... Emu server is started

The ports are bound/configured.

Starting  TRex v2.92 please wait  ...

 set driver name net_i40e

 driver capability  : TCP_UDP_OFFLOAD  TSO

 set dpdk queues mode to MULTI_QUE

Number of ports found: 2

TRex config file:

[j0578@mndemuclxr151 ~]$ cat /etc/trex_cfg.yaml

### Config file generated by dpdk_setup_ports.py ###

- version: 2

  interfaces: ['5e:00.0', '5e:00.1']

  stack: linux_based

  port_info:

      - ip: 1.1.1.1

        default_gw: 2.2.2.2

      - ip: 2.2.2.2

        default_gw: 1.1.1.1

  platform:

      master_thread_id: 0

      latency_thread_id: 1

      dual_if:

        - socket: 0

          threads: [2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62]

Thanks,

Andreas

[Besart Dollma]

Hi Andreas, 

Please provide us with the profile if you can and how are you running so we can try exactly the same steps in our setups. 

Thanks, 

[hanoh haim]

Hi Andreas, 

this line indicates that you are in software mode. the driver does not influence on the number of rules 

"set dpdk queues mode to MULTI_QUE"

Thanks

Hanoh

[Andreas Bourges]

Hi,

...I'm still debugging this. As mentioned earlier, we have currently another problem (getting traffic through a palo alto FW). Palo Case is already open, but currently there's no hint what could be the problem. The only thing we observed was that the IP flow identification field of the generated UDP packets was always "0xFFFF".

As far as I can remember, the flow id field is used to implement the pg_id in the packets - right ? Is there any circumstance that TRex will rewrite this field to FFFF ?

We cannot reproduce that on the other installation, which uses a similar hardware/software. If I create a "normal" python-profile and load it in trex-console, everything works as expected, too.

I will have a look into my code, what's happening to the pg_id while generating streams.

Thanks,

Andreas

[Andreas Bourges]

Hi,

...so I think I found the cuplrit :-) By accident I enabled the latency stats on the streams and it looks like this set the flow-id to FFFF... Could that be? Also the limit to 128 streams makes more sense then.

The other question is, why PA FW drops that traffic... When I disable latency stats, traffic is forwarded as expected.

Thanks,

Andreas

[Andreas Bourges]

Hi,

...just had a look at the docs:

"The IPv4 identification field (or IPv6 flow label in case of IPv6 packet) of the stream is changed to a defined constant value (in the reserved range described in the "per stream statistics" section), to signal to the hardware to pass the stream to the software."

I guess this value is FFFF ? Is there any other change to the packets, that might make a firewall ignore those packets? Just had a short discussion with the customer and we need to be pretty sure if there are other changes to the packets or not, because PA support will fingerpoint TRex I guess... but from my point of view there's no reason for them to drop flow id FFFF packets?!

Thanks,

Andreas

[hanoh haim]

You can capture the packets. Yes, latency flow is marked with ipv4.Id == 0xffff and a magic in the payload 

In software mode the ipv4.id is not required it is a left over from hardware mode

Thanks

Hanoh 

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/cb34b120-47fc-43e3-abeb-71d923aeb7een%40googlegroups.com.