Encountered CFlowStatRuleMgr::update_counters issue causing connect and traffic_stats API failure.

[Divya Saxena]

Hi Team,

We have been using TRex HLTAPI (v2.32) for our traffic tests. In our test environment, we are using TRex installed on 16-port VMs with PCI passthru to the linux host and with Ethernet controller X710 for 10GbE SFP+.

During our test execution, following error was encountered while using traffic_stats API call. 

  File "/corp/sta/opt/ActivePython-2.7/lib/python2.7/site-packages/trex_stl_lib/trex_stl_hltapi.py", line 718, in traffic_stats

    stats = self.trex_client.get_stats(port_handle)

  File "/corp/sta/opt/ActivePython-2.7/lib/python2.7/site-packages/trex_stl_lib/trex_stl_client.py", line 1199, in wrap2

    ret = f(*args, **kwargs)

  File "/corp/sta/opt/ActivePython-2.7/lib/python2.7/site-packages/trex_stl_lib/trex_stl_client.py", line 1660, in get_stats

    return self.__get_stats(ports)

  File "/corp/sta/opt/ActivePython-2.7/lib/python2.7/site-packages/trex_stl_lib/trex_stl_client.py", line 1018, in __get_stats

    pgid_stats = self.get_pgid_stats()

  File "/corp/sta/opt/ActivePython-2.7/lib/python2.7/site-packages/trex_stl_lib/trex_stl_client.py", line 1199, in wrap2

    ret = f(*args, **kwargs)

  File "/corp/sta/opt/ActivePython-2.7/lib/python2.7/site-packages/trex_stl_lib/trex_stl_client.py", line 2324, in get_pgid_stats

    return self.pgid_stats.get_stats(pgid_list)

  File "/corp/sta/opt/ActivePython-2.7/lib/python2.7/site-packages/trex_stl_lib/trex_stl_stats.py", line 1762, in get_stats

    total += fs[val][int(port)]

TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'

And around same time, we observed below messages at trex server side.

May 09 00:27:54 htrex-20311.ca.stalab.ciena.com t-rex-64[25629]: CFlowStatRuleMgr::update_counters()1:Could not count p:133920698 b:0 rx packets, on port 5, because no mapping was found.

May 09 00:27:54 htrex-20311.ca.stalab.ciena.com t-rex-64[25629]: CFlowStatRuleMgr::update_counters()2:Could not count p:133928568 b:8571428352 tx packets on port 5, because no mapping was found.

May 09 00:27:54 htrex-20311.ca.stalab.ciena.com t-rex-64[25629]: CFlowStatRuleMgr::update_counters()1:Could not count p:2 b:0 rx packets, on port 6, because no mapping was found.

May 09 00:27:54 htrex-20311.ca.stalab.ciena.com t-rex-64[25629]: CFlowStatRuleMgr::update_counters()1:Could not count p:1 b:64 tx packets on port 7, because no mapping was found.

May 09 00:27:54 htrex-20311.ca.stalab.ciena.com t-rex-64[25629]: CFlowStatRuleMgr::update_counters()2:Could not count p:6696427 b:0 rx packets, on port 7, because no mapping was found.

May 09 00:27:54 htrex-20311.ca.stalab.ciena.com t-rex-64[25629]: CFlowStatRuleMgr::update_counters()2:Could not count p:133928568 b:0 rx packets, on port 12, because no mapping was found.

May 09 00:27:54 htrex-20311.ca.stalab.ciena.com t-rex-64[25629]: CFlowStatRuleMgr::update_counters()1:Could not count p:133920698 b:0 rx packets, on port 13, because no mapping was found.

May 09 00:27:54 htrex-20311.ca.stalab.ciena.com t-rex-64[25629]: CFlowStatRuleMgr::update_counters()1:Could not count p:2 b:0 rx packets, on port 14, because no mapping was found.

May 09 00:27:54 htrex-20311.ca.stalab.ciena.com t-rex-64[25629]: CFlowStatRuleMgr::update_counters()2:Could not count p:6696427 b:428571328 tx packets on port 14, because no mapping was found.

May 09 00:27:54 htrex-20311.ca.stalab.ciena.com t-rex-64[25629]: CFlowStatRuleMgr::update_counters()2:Could not count p:6696427 b:0 rx packets, on port 15, because no mapping was found.

In the test, following steps were performed:

1. Connect to TRex server and acquire 3 ports.

2. Configure tagged stream with flow stats enable on each port.

3. Send/receive 6696428 frames 64 byte each from each port.

4. Collect statistics

Post these errors, all the connect API call in other tests which were trying to acquire any of above highlighted ports started failing with same python error as mentioned above. Moreover, starting console was also failing with similar error traceback.

[htrex-20311(dsaxena)]-> TRex 1004> ./start-console

 

Using 'python' as Python interpeter

  

Connecting to RPC server on localhost:4501                   [SUCCESS]

 

Connecting to publisher server on localhost:4500             [SUCCESS]

 

Traceback (most recent call last):

  File "/corp/sta/opt/ActivePython-2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main

   "__main__", fname, loader, pkg_name)

  File "/corp/sta/opt/ActivePython-2.7/lib/python2.7/runpy.py", line 72, in _run_code

    exec code in run_globals

  File "/corp/sta/opt/TRex/v2.32/automation/trex_control_plane/stl/console/trex_console.py", line 989, in <module>

    main()

  File "/corp/sta/opt/TRex/v2.32/automation/trex_control_plane/stl/console/trex_console.py", line 942, in main

    stateless_client.connect()

  File "/corp/sta/opt/TRex/v2.32/automation/trex_control_plane/stl/trex_stl_lib/trex_stl_client.py", line 1199, in wrap2

    ret = f(*args, **kwargs)

  File "/corp/sta/opt/TRex/v2.32/automation/trex_control_plane/stl/trex_stl_lib/trex_stl_client.py", line 1768, in connect

    rc = self.__connect()

  File "/corp/sta/opt/TRex/v2.32/automation/trex_control_plane/stl/trex_stl_lib/trex_stl_client.py", line 975, in __connect

    self.get_pgid_stats()

  File "/corp/sta/opt/TRex/v2.32/automation/trex_control_plane/stl/trex_stl_lib/trex_stl_client.py", line 1199, in wrap2

    ret = f(*args, **kwargs)

  File "/corp/sta/opt/TRex/v2.32/automation/trex_control_plane/stl/trex_stl_lib/trex_stl_client.py", line 2324, in get_pgid_stats

    return self.pgid_stats.get_stats(pgid_list)

  File "/corp/sta/opt/TRex/v2.32/automation/trex_control_plane/stl/trex_stl_lib/trex_stl_stats.py", line 1762, in get_stats

    total += fs[val][int(port)]

TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'

Only solution which we found to resolve this issue was restart the trex server. Since we faced this issue a couple of times so would like to know the reason of this issue and how it should be handled so that smooth test execution can be carried out.

Is it a known issue and fixed in new releases? 

Please confirm and let me know if any other detail is required to investigate this issue.

[Yaroslav Brustinov]

Hi, Divya.

1. Is the issue reproducible or sporadic?
2. Could you share the way to recreate the issue? (Traffic sent, config commands etc.)
3. Could you start the console with -v (verbose) and send output of RPC conversation before the error?

Thanks,

Yaroslav.

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/25b4a2c4-2c89-4205-b9fb-bef7d8b15a9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yaroslav Brustinov]

Regarding traffic, it would help to get output of following trex-console command: (assuming acquired ports are 0, 1 and 2)

streams -p 0 1 2 --code

(You will need to mask out failing Python code lines in order for console to ignore the failing step(s) and continue, just to debug the issue)

[Divya Saxena]

Hi Yaroslav,

Thanks for your quick response on this query.

> 1. Is the issue reproducible or sporadic?
I have seen this issue two times a day. Its sporadic in nature but I think reproduction can be tried by running tests in loop.

> 2. Could you share the way to recreate the issue? (Traffic sent, config commands etc.)
Following are the steps performed in the test:
(please note that I am using tcl wrapper over trex python api so syntax used here are tcl based but keys are same as used in python dict)
1. Connect to server
connect -device htrex-20311 -port_list 4 5 3 -reset true -break_locks true -username concentrator/testbed/layer2Mesh
2. Make acquired port's link to UP state using 'set_port_attr' low level API.
3. Check if port state is UP using 'get_port_attr' low level API.
4. Create stream on each port. e.g. for port 4:
traffic_config -name "stream1" -mode "create" -port_handle 4 -frame_size 64 -mac_src "00:50:56:b9:de:70" -mac_dst "00:50:56:b9:de:71" -l3_protocol "ipv4" -ip_src_addr "10.0.0.1" -ip_src_mode "increment" -ip_src_count 254 -ip_dst_addr "8.0.0.1" -ip_dst_mode "increment" -ip_dst_count 254 -l4_protocol "udp" -udp_dst_port 12 -udp_src_port 1025 -rate_pps 1339285 -vlan_id 100 -disable_flow_stats "false" -flow_stats_id 102 -transmit_mode "single_burst" -pkts_per_burst 6696427
5. Clear stats on all ports
traffic_control -action "clear_stats" -port_handle 4 5 3
6. Start traffic on all ports
traffic_control -action "run" -port_handle 4 5 3
7. Poll and stop traffic on all ports
traffic_control -action "poll" -port_handle 4 5 3
traffic_control -action "stop" -port_handle 4 5 3
8. Get statistics
traffic_stats -mode "aggregate" -port_handle 4

> 3. Could you start the console with -v (verbose) and send output of RPC conversation before the error?
I will try reproducing this issue with keeping console ON with -v in a parallel session and share you the output.

[Divya Saxena]

I'll try this in my reproduction attempt.

[Divya Saxena]

Hi Yaroslav,

I successfully reproduced the issue but I couldn't see anything on console except following kind of messages.

2018-05-13 21:13:48 - [server][info]     - Port 14 job done

I started console using 'start-console -v' command in another shell while tests were started. After issue occurrence, current console session was active but when I tried another session after closing the current one, session failed as I reported earlier (attached here the output what is displayed when use verbose mode).

Please help how to move further.

[Yaroslav Brustinov]

Hi,

At this state of server, could you apply what I wrote in previous mail?

Regarding traffic, it would help to get output of following trex-console command: (assuming acquired ports are 0, 1 and 2)

streams -p 0 1 2 --code

(You will need to mask out failing Python code lines in order for console to ignore the failing step(s) and continue, just to debug the issue)

 

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/c301c1e8-e4f8-4b58-b523-812f9b43b3f9%40googlegroups.com.

[Yaroslav Brustinov]

Please ignore previous mail, I forgot that you have shown the traffic already.

[Yaroslav Brustinov]

Hi,

Could you try the following, in this file:

automation\trex_control_plane\stl\trex_stl_lib\trex_stl_client.py

function:

def stop (self, ports = None, rx_delay_ms = None):

Try increasing rx_delay_ms to higher number (1000, for example).

Could be that default delay (10ms) is not enough, and packets are still arriving after it.

Thanks,

Yaroslav.

[Divya Saxena]

I'll try with this change.

[Divya Saxena]

Hi Yaroslav,

I tried with suggested change but it didn't work. Even tried changes on both side, server (automation/trex_control_plane/stl/trex_stl_lib/trex_stl_client.py) and client (trex_client/stl/trex_stl_lib/trex_stl_client.py) side as well, that is also not working.

def stop (self, ports = None, rx_delay_ms = None):

...
        if rx_delay_ms is None:
            if self.ports[ports[0]].is_virtual(): # assume all ports have same type
                rx_delay_ms = 100
            else:
                rx_delay_ms = 1000
                #rx_delay_ms = 10
 

Can you please also try at your end and suggest me the fix? 

[Divya Saxena]

Hi Yaroslav,

Did you get the chance to try this? It would be very helpful if you can suggest fix for this issue since its impacting my tests. If further trial is needed with some code changes, I could do it at my end. Please suggest.

[Yaroslav Brustinov]

Hi, Divya.

We have tried to reconstruct the issue, without success.

We did not find what can cause "no mapping" stderr prints yet, we can, however, provide a patch that will eliminate "null" and "Infinity" from rates of bps and pps (they are causing exception in Python).

Another thing, in JSON-RPC, there are more rx packets on different ports than tx sent.

Is it expected?

It could be caused by "start" -> packets are sent -> there is high latency in network -> "stop" -> "start" again with same pg-id -> packets are arriving from the past -> more rx.

Could it be your case?

Here is the RPC for pg_id 103:

103	pkts		pps		bps
port	tx	rx	tx	rx	tx	rx
0	0	6696427	0	0	0	0
1	6696427	0	0	0	0	0
2	0	31	0	0	0	0
3	0	6696427	0	0	0	0
4	6696427	0	0	0	0	0
5	0	5266	0	0	0	0
6	0	6700573	0	Infinity	0	null
7	6696427	0	null	0	null	0
8	0	6696427	0	0	0	0
9	0	0	0	0	0	0
10	0	31	0	0	0	0
11	0	6696427	0	0	0	0
12	0	0	0	0	0	0
13	0	5266	0	0	0	0
14	0	6700573	0	Infinity	0	null
15	0	0	0	0	0	0

The best would be if you could provide a setup where the issue occurs, so that we could debug it.

You are from Cisco, right?

Thanks,

Yaroslav.

On Fri, May 18, 2018 at 7:14 AM, Divya Saxena <divya...@gmail.com> wrote:

Hi Yaroslav,

Did you get the chance to try this? It would be very helpful if you can suggest fix for this issue since its impacting my tests. If further trial is needed with some code changes, I could do it at my end. Please suggest.

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/551e54c6-36d9-4fcd-b76c-d2be4866e156%40googlegroups.com.

[Divya Saxena]

Hi Yaroslav,

The best would be if you could provide a setup where the issue occurs, so that we could debug it.
You are from Cisco, right?

No, I am not from Cisco so I think would not be able to share the setup. btw Thanks for your kind support and concern.

We did not find what can cause "no mapping" stderr prints yet, we can, however, provide a patch that will eliminate "null" and "Infinity" from rates of bps and pps (they are causing exception in Python)

That would be great. If this is the problem that causing exception and patch fix could resolve the issue, that would unblock me. 

Another thing, in JSON-RPC, there are more rx packets on different ports than tx sent.
Is it expected?
It could be caused by "start" -> packets are sent -> there is high latency in network -> "stop" -> "start" again with same pg-id -> packets are arriving from the past -> more rx.
Could it be your case?

In my environment, multiple DUTs are connected with one TRex VM which has 16 ports and 2-3 TRex ports are assigned per DUT so total 5-6 single DUT topology is created. I was running same test on these DUTs topology in batch mode to reproduce the issue that might have done something at TRex side which might caused these stats. In the test, traffic was run only once for each pg_id (102,103,203,201,301,302) and I think high latency in network was not observed. So I am not sure about these rx stats.

[Yaroslav Brustinov]

Hi,

That would be great. If this is the problem that causing exception and patch fix could resolve the issue, that would unblock me. 

 

Please try out this branch (it's based on v2.41)

https://github.com/ybrustin/trex-core/tree/flow_stats_nan_rates

In my environment, multiple DUTs are connected with one TRex VM which has 16 ports and 2-3 TRex ports are assigned per DUT so total 5-6 single DUT topology is created. I was running same test on these DUTs topology in batch mode to reproduce the issue that might have done something at TRex side which might caused these stats. In the test, traffic was run only once for each pg_id (102,103,203,201,301,302) and I think high latency in network was not observed. So I am not sure about these rx stats.

 Few questions:

1. Do you run the tests at different ports in serial way or in parallel?
2. Do you use the same pg_ids for different sets of ports? For example, is pg_id 103 used only with traffic sent from ports 1,4,7 or others too?

Thanks,

Yaroslav.

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/d0e516d9-21b5-46fa-aad4-334dde1ba650%40googlegroups.com.

[Divya Saxena]

Hi Yaroslav,

Please try out this branch (it's based on v2.41)
https://github.com/ybrustin/trex-core/tree/flow_stats_nan_rates

I will try this branch.

 

Do you run the tests at different ports in serial way or in parallel?

 

Parallel.

Do you use the same pg_ids for different sets of ports? For example, is pg_id 103 used only with traffic sent from ports 1,4,7 or others too?

Since same test case was running on different set of DUTs and TRex ports so same pg_id 103 was used with other ports too. 

[Yaroslav Brustinov]

Hi,

I will try this branch.

FYI, I've pushed another commit regarding the issue to same branch.

By the way, which hypervisor are you using?

Thanks,

Yaroslav.

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/ef3fe3ce-6e0f-423f-935c-899e2a9d3d39%40googlegroups.com.

[Divya Saxena]

Hi Yaroslav,

We are using qemu-kvm hypervisor on CentOS 7.4.

Regarding trying your branch, I checked out it but while starting the server its giving error due to unavailable file. Is there any other way to do it?

[root@htrex-20311 scripts]# ./t-rex-64 -i
Starting Scapy server.... Scapy server is started
The ports are bound/configured.
./t-rex-64: line 60: ./_t-rex-64: No such file or directory

[Yaroslav Brustinov]

Hi,

Github contains the sources, you need to compile them, e.g.:

cd linux_dpdk

./b configure

./b

cd ../scripts

<run the TRex>

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/c7aa69e4-e47a-4f62-a38b-88368a809d8c%40googlegroups.com.

[Divya Saxena]

I installed gcc/g++ on my VM and tried what you suggested but couldn't make it. Its not working on CentOS. (Attached config.log).

Is it possible for you to provide compiled package as we get from TRex release repo?

[root@htrex-20311 linux_dpdk]# ./b configure
Setting top to                           : /root/divya/trex-patch/trex-core
Setting out to                           : /root/divya/trex-patch/trex-core/linux_dpdk/build_dpdk
Checking for program 'gcc, cc'           : /usr/bin/gcc
Checking for program 'ar'                : /usr/bin/ar
Checking for program 'g++, c++'          : /usr/bin/g++
Checking for program 'ar'                : /usr/bin/ar
Checking for program 'ldd'               : /usr/bin/ldd
Checking for library z                   : not found
Could not determine Linux distribution.
Ubuntu install:
sudo apt install zlib1g-dev


Fedora install:
sudo yum install zlib-devel


The configuration failed
(complete log in /root/divya/trex-patch/trex-core/linux_dpdk/build_dpdk/config.log)

[Yaroslav Brustinov]

Hi,

Please use (it's faster/easier IMHO):

sudo yum install zlib-devel

Thanks,

Yaroslav

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/1714ab0e-7607-4e10-8fd6-f91c9bfdb982%40googlegroups.com.

[Divya Saxena]

Hi,

After installing mentioned library, it worked without any further dependency. Now I am running my test to confirm the issue resolution.

[Divya Saxena]

Hi Yaroslav,

With this patch, I haven't yet observed that python exception again so this issue seems resolved. Thanks.

I believe this fix will be available in next upcoming general release i.e 2.42 so I would like to confirm by when new release is planned. We'll use this patch for now and once new release is available, we would like to switch to that latest release. Please confirm.

[Yaroslav Brustinov]

Hi,

Yes, it is going to be part of next release.

Thanks,
Yaroslav

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/2a338458-b77b-4d81-8433-d0e1370ae283%40googlegroups.com.

[Divya Saxena]

Ok, Thanks.

[Divya Saxena]

Hi Yaroslav,

I have a query related to packet group statistics for the same test. RX statistics (mode='streams') at TRex port is showing unexpected packets. E.g. Port 6 statistics is showing packets for pg_id 103 and 203 and port 14 statistics is showing packets for pg_id 201 and 301 which is unexpected. Statistics at DUT side seems ok.

trex(read-only)>streams -p 6 7 14 --code




[verbose] Sending Request To Server:


{
    "id": "os9i34te",
    "jsonrpc": "2.0",
    "method": "get_all_streams",
    "params": {
        "api_h": "2VM8fWV7",
        "port_id": 6
    }
}






[verbose] Server Response:


[
    {
        "id": "os9i34te",
        "jsonrpc": "2.0",
        "result": {
            "streams": {
                "1": {
                    "action_count": 0,
                    "enabled": true,
                    "flags": 3,
                    "flow_stats": {
                        "enabled": true,
                        "rule_type": "stats",
                        "stream_id": 102
                    },
                    "isg": 0,
                    "mode": {
                        "rate": {
                            "type": "pps",
                            "value": 1339285
                        },
                        "total_pkts": 1,
                        "type": "single_burst"
                    },
                    "next_stream_id": -1,
                    "packet": {
                        "binary": "AFBWud5xAFBWud5wgQAwZAgARQAAKgAAAABAEWjCCgAAAQgAAAEEAQAMABY/Caqqqqqqqqqqqqqqqqqq",
                        "meta": ""
                    },
                    "self_start": true,
                    "vm": {
                        "instructions": [
                            {
                                "init_value": 2147483647,
                                "max_value": 2147483900,
                                "min_value": 2147483647,
                                "name": "inc_4_253_1",
                                "op": "inc",
                                "size": 4,
                                "step": 1,
                                "type": "flow_var"
                            },
                            {
                                "add_value": -1979711486,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 30,
                                "type": "write_flow_var"
                            },
                            {
                                "add_value": -2013265918,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 34,
                                "type": "write_flow_var"
                            },
                            {
                                "pkt_offset": 18,
                                "type": "fix_checksum_ipv4"
                            }
                        ]
                    }
                },
                "2": {
                    "action_count": 0,
                    "enabled": true,
                    "flags": 3,
                    "flow_stats": {
                        "enabled": true,
                        "rule_type": "stats",
                        "stream_id": 103
                    },
                    "isg": 0,
                    "mode": {
                        "rate": {
                            "type": "pps",
                            "value": 1339285
                        },
                        "total_pkts": 6696427,
                        "type": "single_burst"
                    },
                    "next_stream_id": -1,
                    "packet": {
                        "binary": "AFBWud5yAFBWud5wgQAwZAgARQAAKgAAAABAEWjCCgAAAQgAAAEEAQAMABY/Caqqqqqqqqqqqqqqqqqq",
                        "meta": ""
                    },
                    "self_start": true,
                    "vm": {
                        "instructions": [
                            {
                                "init_value": 2147483647,
                                "max_value": 2147483900,
                                "min_value": 2147483647,
                                "name": "inc_4_253_1",
                                "op": "inc",
                                "size": 4,
                                "step": 1,
                                "type": "flow_var"
                            },
                            {
                                "add_value": -1979711486,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 30,
                                "type": "write_flow_var"
                            },
                            {
                                "add_value": -2013265918,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 34,
                                "type": "write_flow_var"
                            },
                            {
                                "pkt_offset": 18,
                                "type": "fix_checksum_ipv4"
                            }
                        ]
                    }
                }
            }
        }
    }
]






[verbose] Sending Request To Server:


{
    "id": "f1n7nthe",
    "jsonrpc": "2.0",
    "method": "get_all_streams",
    "params": {
        "api_h": "2VM8fWV7",
        "port_id": 7
    }
}






[verbose] Server Response:


[
    {
        "id": "f1n7nthe",
        "jsonrpc": "2.0",
        "result": {
            "streams": {
                "1": {
                    "action_count": 0,
                    "enabled": true,
                    "flags": 3,
                    "flow_stats": {
                        "enabled": true,
                        "rule_type": "stats",
                        "stream_id": 203
                    },
                    "isg": 0,
                    "mode": {
                        "rate": {
                            "type": "pps",
                            "value": 1339285
                        },
                        "total_pkts": 1,
                        "type": "single_burst"
                    },
                    "next_stream_id": -1,
                    "packet": {
                        "binary": "AFBWud5yAFBWud5xgQAwZAgARQAAKgAAAABAEWjCCgAAAQgAAAEEAQAMABY/Caqqqqqqqqqqqqqqqqqq",
                        "meta": ""
                    },
                    "self_start": true,
                    "vm": {
                        "instructions": [
                            {
                                "init_value": 2147483647,
                                "max_value": 2147483900,
                                "min_value": 2147483647,
                                "name": "inc_4_253_1",
                                "op": "inc",
                                "size": 4,
                                "step": 1,
                                "type": "flow_var"
                            },
                            {
                                "add_value": -1979711486,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 30,
                                "type": "write_flow_var"
                            },
                            {
                                "add_value": -2013265918,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 34,
                                "type": "write_flow_var"
                            },
                            {
                                "pkt_offset": 18,
                                "type": "fix_checksum_ipv4"
                            }
                        ]
                    }
                },
                "2": {
                    "action_count": 0,
                    "enabled": true,
                    "flags": 3,
                    "flow_stats": {
                        "enabled": true,
                        "rule_type": "stats",
                        "stream_id": 201
                    },
                    "isg": 0,
                    "mode": {
                        "rate": {
                            "type": "pps",
                            "value": 1339285
                        },
                        "total_pkts": 6696427,
                        "type": "single_burst"
                    },
                    "next_stream_id": -1,
                    "packet": {
                        "binary": "AFBWud5wAFBWud5xgQAwZAgARQAAKgAAAABAEWjCCgAAAQgAAAEEAQAMABY/Caqqqqqqqqqqqqqqqqqq",
                        "meta": ""
                    },
                    "self_start": true,
                    "vm": {
                        "instructions": [
                            {
                                "init_value": 2147483647,
                                "max_value": 2147483900,
                                "min_value": 2147483647,
                                "name": "inc_4_253_1",
                                "op": "inc",
                                "size": 4,
                                "step": 1,
                                "type": "flow_var"
                            },
                            {
                                "add_value": -1979711486,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 30,
                                "type": "write_flow_var"
                            },
                            {
                                "add_value": -2013265918,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 34,
                                "type": "write_flow_var"
                            },
                            {
                                "pkt_offset": 18,
                                "type": "fix_checksum_ipv4"
                            }
                        ]
                    }
                }
            }
        }
    }
]






[verbose] Sending Request To Server:


{
    "id": "u6ja65bb",
    "jsonrpc": "2.0",
    "method": "get_all_streams",
    "params": {
        "api_h": "2VM8fWV7",
        "port_id": 14
    }
}






[verbose] Server Response:


[
    {
        "id": "u6ja65bb",
        "jsonrpc": "2.0",
        "result": {
            "streams": {
                "1": {
                    "action_count": 0,
                    "enabled": true,
                    "flags": 3,
                    "flow_stats": {
                        "enabled": true,
                        "rule_type": "stats",
                        "stream_id": 301
                    },
                    "isg": 0,
                    "mode": {
                        "rate": {
                            "type": "pps",
                            "value": 1339285
                        },
                        "total_pkts": 1,
                        "type": "single_burst"
                    },
                    "next_stream_id": -1,
                    "packet": {
                        "binary": "AFBWud5wAFBWud5ygQAwZAgARQAAKgAAAABAEWjCCgAAAQgAAAEEAQAMABY/Caqqqqqqqqqqqqqqqqqq",
                        "meta": ""
                    },
                    "self_start": true,
                    "vm": {
                        "instructions": [
                            {
                                "init_value": 2147483647,
                                "max_value": 2147483900,
                                "min_value": 2147483647,
                                "name": "inc_4_253_1",
                                "op": "inc",
                                "size": 4,
                                "step": 1,
                                "type": "flow_var"
                            },
                            {
                                "add_value": -1979711486,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 30,
                                "type": "write_flow_var"
                            },
                            {
                                "add_value": -2013265918,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 34,
                                "type": "write_flow_var"
                            },
                            {
                                "pkt_offset": 18,
                                "type": "fix_checksum_ipv4"
                            }
                        ]
                    }
                },
                "2": {
                    "action_count": 0,
                    "enabled": true,
                    "flags": 3,
                    "flow_stats": {
                        "enabled": true,
                        "rule_type": "stats",
                        "stream_id": 302
                    },
                    "isg": 0,
                    "mode": {
                        "rate": {
                            "type": "pps",
                            "value": 1339285
                        },
                        "total_pkts": 6696427,
                        "type": "single_burst"
                    },
                    "next_stream_id": -1,
                    "packet": {
                        "binary": "AFBWud5xAFBWud5ygQAwZAgARQAAKgAAAABAEWjCCgAAAQgAAAEEAQAMABY/Caqqqqqqqqqqqqqqqqqq",
                        "meta": ""
                    },
                    "self_start": true,
                    "vm": {
                        "instructions": [
                            {
                                "init_value": 2147483647,
                                "max_value": 2147483900,
                                "min_value": 2147483647,
                                "name": "inc_4_253_1",
                                "op": "inc",
                                "size": 4,
                                "step": 1,
                                "type": "flow_var"
                            },
                            {
                                "add_value": -1979711486,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 30,
                                "type": "write_flow_var"
                            },
                            {
                                "add_value": -2013265918,
                                "is_big_endian": true,
                                "name": "inc_4_253_1",
                                "pkt_offset": 34,
                                "type": "write_flow_var"
                            },
                            {
                                "pkt_offset": 18,
                                "type": "fix_checksum_ipv4"
                            }
                        ]
                    }
                }
            }
        }
    }
]


Port: 6


Stream ID: 1
    packet = (Ether(src='00:50:56:b9:de:70', dst='00:50:56:b9:de:71', type=33024) /
              Dot1Q(vlan=100, type=2048, id=1, prio=1) /
              IP(src='10.0.0.1', proto=17, dst='8.0.0.1', chksum=26818, len=42, ihl=5, id=0) /
              UDP(dport=12, sport=1025, len=22, chksum=16137) /
              Raw(load='\xaa' * 14))
    vm = STLVM()
    vm.var(name='inc_4_253_1', size=4, op='inc', init_value='127.255.255.255', min_value='127.255.255.255', max_value='128.0.0.252', step=1)
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.src', add_val=-1979711486, byte_order='big')
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.dst', add_val=-2013265918, byte_order='big')
    vm.fix_chksum(offset='IP')
    stream = STLStream(packet = STLPktBuilder(pkt = packet, vm = vm),
                       flow_stats = STLFlowStats(102),
                       mac_src_override_by_pkt = True,
                       mac_dst_override_mode = 1,
                       mode = STLTXSingleBurst(total_pkts = 1, pps = 1339285))


Stream ID: 2
    packet = (Ether(src='00:50:56:b9:de:70', dst='00:50:56:b9:de:72', type=33024) /
              Dot1Q(vlan=100, type=2048, id=1, prio=1) /
              IP(src='10.0.0.1', proto=17, dst='8.0.0.1', chksum=26818, len=42, ihl=5, id=0) /
              UDP(dport=12, sport=1025, len=22, chksum=16137) /
              Raw(load='\xaa' * 14))
    vm = STLVM()
    vm.var(name='inc_4_253_1', size=4, op='inc', init_value='127.255.255.255', min_value='127.255.255.255', max_value='128.0.0.252', step=1)
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.src', add_val=-1979711486, byte_order='big')
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.dst', add_val=-2013265918, byte_order='big')
    vm.fix_chksum(offset='IP')
    stream = STLStream(packet = STLPktBuilder(pkt = packet, vm = vm),
                       flow_stats = STLFlowStats(103),
                       mac_src_override_by_pkt = True,
                       mac_dst_override_mode = 1,
                       mode = STLTXSingleBurst(total_pkts = 6696427, pps = 1339285))


Port: 7


Stream ID: 1
    packet = (Ether(src='00:50:56:b9:de:71', dst='00:50:56:b9:de:72', type=33024) /
              Dot1Q(vlan=100, type=2048, id=1, prio=1) /
              IP(src='10.0.0.1', proto=17, dst='8.0.0.1', chksum=26818, len=42, ihl=5, id=0) /
              UDP(dport=12, sport=1025, len=22, chksum=16137) /
              Raw(load='\xaa' * 14))
    vm = STLVM()
    vm.var(name='inc_4_253_1', size=4, op='inc', init_value='127.255.255.255', min_value='127.255.255.255', max_value='128.0.0.252', step=1)
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.src', add_val=-1979711486, byte_order='big')
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.dst', add_val=-2013265918, byte_order='big')
    vm.fix_chksum(offset='IP')
    stream = STLStream(packet = STLPktBuilder(pkt = packet, vm = vm),
                       flow_stats = STLFlowStats(203),
                       mac_src_override_by_pkt = True,
                       mac_dst_override_mode = 1,
                       mode = STLTXSingleBurst(total_pkts = 1, pps = 1339285))


Stream ID: 2
    packet = (Ether(src='00:50:56:b9:de:71', dst='00:50:56:b9:de:70', type=33024) /
              Dot1Q(vlan=100, type=2048, id=1, prio=1) /
              IP(src='10.0.0.1', proto=17, dst='8.0.0.1', chksum=26818, len=42, ihl=5, id=0) /
              UDP(dport=12, sport=1025, len=22, chksum=16137) /
              Raw(load='\xaa' * 14))
    vm = STLVM()
    vm.var(name='inc_4_253_1', size=4, op='inc', init_value='127.255.255.255', min_value='127.255.255.255', max_value='128.0.0.252', step=1)
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.src', add_val=-1979711486, byte_order='big')
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.dst', add_val=-2013265918, byte_order='big')
    vm.fix_chksum(offset='IP')
    stream = STLStream(packet = STLPktBuilder(pkt = packet, vm = vm),
                       flow_stats = STLFlowStats(201),
                       mac_src_override_by_pkt = True,
                       mac_dst_override_mode = 1,
                       mode = STLTXSingleBurst(total_pkts = 6696427, pps = 1339285))


Port: 14


Stream ID: 1
    packet = (Ether(src='00:50:56:b9:de:72', dst='00:50:56:b9:de:70', type=33024) /
              Dot1Q(vlan=100, type=2048, id=1, prio=1) /
              IP(src='10.0.0.1', proto=17, dst='8.0.0.1', chksum=26818, len=42, ihl=5, id=0) /
              UDP(dport=12, sport=1025, len=22, chksum=16137) /
              Raw(load='\xaa' * 14))
    vm = STLVM()
    vm.var(name='inc_4_253_1', size=4, op='inc', init_value='127.255.255.255', min_value='127.255.255.255', max_value='128.0.0.252', step=1)
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.src', add_val=-1979711486, byte_order='big')
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.dst', add_val=-2013265918, byte_order='big')
    vm.fix_chksum(offset='IP')
    stream = STLStream(packet = STLPktBuilder(pkt = packet, vm = vm),
                       flow_stats = STLFlowStats(301),
                       mac_src_override_by_pkt = True,
                       mac_dst_override_mode = 1,
                       mode = STLTXSingleBurst(total_pkts = 1, pps = 1339285))


Stream ID: 2
    packet = (Ether(src='00:50:56:b9:de:72', dst='00:50:56:b9:de:71', type=33024) /
              Dot1Q(vlan=100, type=2048, id=1, prio=1) /
              IP(src='10.0.0.1', proto=17, dst='8.0.0.1', chksum=26818, len=42, ihl=5, id=0) /
              UDP(dport=12, sport=1025, len=22, chksum=16137) /
              Raw(load='\xaa' * 14))
    vm = STLVM()
    vm.var(name='inc_4_253_1', size=4, op='inc', init_value='127.255.255.255', min_value='127.255.255.255', max_value='128.0.0.252', step=1)
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.src', add_val=-1979711486, byte_order='big')
    vm.write(fv_name='inc_4_253_1', pkt_offset='IP.dst', add_val=-2013265918, byte_order='big')
    vm.fix_chksum(offset='IP')
    stream = STLStream(packet = STLPktBuilder(pkt = packet, vm = vm),
                       flow_stats = STLFlowStats(302),
                       mac_src_override_by_pkt = True,
                       mac_dst_override_mode = 1,
                       mode = STLTXSingleBurst(total_pkts = 6696427, pps = 1339285))

Console's stdout is attached. (It was executed with v2.32. However, no change was observed in the test result when executed with v2.41)

Looks like something is wrong at TRex side. Can you please help me on this? Environment is same as I explained earlier.

[Divya Saxena]

Hi Yaroslav,

Seems previous post got truncated due to long message so attaching output of 'streams' instead of direct pasting it here. (It was executed with v2.32. However, no change was observed in the test result when executed with v2.41).

[Yaroslav Brustinov]

Hi, Divya.

I'm out of office for some time.
Could you try using different pg_id for different port(s), so that they don't interfere in statistics?

Thanks,
Yaroslav

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/0fae7ba7-00e4-45ff-8102-27ca345be1e7%40googlegroups.com.

[Divya Saxena]

Hi Yaroslav,

In order to narrow down the problem, this time I have done single test execution using trex ports 6,7 and 14 only and no other ports were in use. Moreover, I restarted the trex service before test execution so that no previous data could be considered in stats calculation. So as per my understanding, in this case it shouldn't have any interfere in stats.

Also, I have a concern if using same pg_id with different set of ports would cause such statistics issues as you indicated. Isn't it a problem at TRex side?

Probably I need to wait until you are back or someone else can look into this issue, if possible.

[Divya Saxena]

In addition to details mentioned in previous post, one other point:

I tried test with 3 ports topology (6,7,14) but this time transmission is done only from first port (also I used low level API). So port 6 has transmitted two streams with pg_id 102 and 103. Following is the stream statistics. In this, ipackets for port 6 is zero whereas flow stats for pg_id 103 shows 6696427 rx packets for port 6.

stats(14,ibytes)                      = 428571328

stats(14,ierrors)                     = 0

stats(14,ipackets)                    = 6696427

stats(14,obytes)                      = 0

stats(14,oerrors)                     = 0

stats(14,opackets)                    = 0

stats(14,rx_bps)                      = 242386816.0

stats(14,rx_bps_L1)                   = 318132656.0

stats(14,rx_pps)                      = 473411.5

stats(14,rx_util)                     = 3.18132656

stats(14,tx_bps)                      = 42744.2

stats(14,tx_bps_L1)                   = 56104.2

stats(14,tx_pps)                      = 83.5

stats(14,tx_util)                     = 0.000561042

stats(6,ibytes)                       = 0

stats(6,ierrors)                      = 0

stats(6,ipackets)                     = 0

stats(6,obytes)                       = 428571392

stats(6,oerrors)                      = 0

stats(6,opackets)                     = 6696428

stats(6,rx_bps)                       = 42827.2

stats(6,rx_bps_L1)                    = 56203.2

stats(6,rx_pps)                       = 83.6

stats(6,rx_util)                      = 0.00562032

stats(6,tx_bps)                       = 242483504.0

stats(6,tx_bps_L1)                    = 318259536.0

stats(6,tx_pps)                       = 473600.2

stats(6,tx_util)                      = 31.8259536

stats(7,ibytes)                       = 64

stats(7,ierrors)                      = 0

stats(7,ipackets)                     = 1

stats(7,obytes)                       = 0

stats(7,oerrors)                      = 0

stats(7,opackets)                     = 0

stats(7,rx_bps)                       = 42769.1

stats(7,rx_bps_L1)                    = 56129.1

stats(7,rx_pps)                       = 83.5

stats(7,rx_util)                      = 0.000561291

stats(7,tx_bps)                       = 42826.0

stats(7,tx_bps_L1)                    = 56202.0

stats(7,tx_pps)                       = 83.6

stats(7,tx_util)                      = 0.00056202

stats(flow_stats,102,rx_pkts,15)      = 1
stats(flow_stats,102,rx_pkts,7)       = 1
stats(flow_stats,102,rx_pkts,total)   = 2
stats(flow_stats,102,tx_bytes,6)      = 64
stats(flow_stats,102,tx_bytes,total)  = 64
stats(flow_stats,102,tx_pkts,6)       = 1
stats(flow_stats,102,tx_pkts,total)   = 1
stats(flow_stats,103,rx_pkts,14)      = 6696427
stats(flow_stats,103,rx_pkts,6)       = 6696427
stats(flow_stats,103,rx_pkts,total)   = 13392854
stats(flow_stats,103,tx_bytes,6)      = 428571328
stats(flow_stats,103,tx_bytes,total)  = 428571328
stats(flow_stats,103,tx_pkts,6)       = 6696427
stats(flow_stats,103,tx_pkts,total)   = 6696427

stats(global,bw_per_core)             = 5.867

stats(global,cpu_util)                = 1.034

stats(global,queue_full)              = 0

stats(global,rx_bps)                  = 242472416.0

stats(global,rx_cpu_util)             = 0.0

stats(global,rx_drop_bps)             = 0.0

stats(global,rx_pps)                  = 473578.7

stats(global,tx_bps)                  = 242569072.0

stats(global,tx_pps)                  = 473767.4

stats(total,ibytes)                   = 428571392

stats(total,ierrors)                  = 0

stats(total,ipackets)                 = 6696428

stats(total,obytes)                   = 428571392

stats(total,oerrors)                  = 0

stats(total,opackets)                 = 6696428

stats(total,rx_bps)                   = 242472412.3

stats(total,rx_bps_L1)                = 318244988.3

stats(total,rx_pps)                   = 473578.6

stats(total,rx_util)                  = 3.187508171

stats(total,tx_bps)                   = 242569074.2

stats(total,tx_bps_L1)                = 318371842.2

stats(total,tx_pps)                   = 473767.3

stats(total,tx_util)                  = 31.827076662

(Flow stats parameters with value 0 or N/A are not listed here).

Please look into this once you are back.

[Yaroslav Brustinov]

Hi, Divya.

As we can't reproduce the issue, any new inputs are valuable.
Regarding latest results, could you clarify how did you run it?

1. Run the server
2. Clear stats
3. Start transmitting from single port 6
4. Stopped the traffic
5. Got shown results

Is it according to algorithm above?
Did the error appear immediately or you repeated steps 2-5?

Another question, are all interfaces of used NICs are in use by TRex? Or there are some interfaces under ESXi/Linux driver?

Thanks,
Yaroslav

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/6a0b3461-d256-42e2-b29e-c71d39efcd9f%40googlegroups.com.

[Yaroslav Brustinov]

And another question, are the stats you show were taken directly from JSON-RPC or from Python API? If it is the second, could you attach JSON-RPC response of stats?

[Divya Saxena]

Hi Yaroslav,

1. Run the server
2. Clear stats
3. Start transmitting from single port 6
4. Stopped the traffic
5. Got shown results
Is it according to algorithm above?
Did the error appear immediately or you repeated steps 2-5?

Yes, this is the algorithm but stats were captured after second iteration of steps 2-4. First iteration was done for MAC learning.

Another question, are all interfaces of used NICs are in use by TRex? Or there are some interfaces under ESXi/Linux driver?

 

All are used by TRex.

And another question, are the stats you show were taken directly from JSON-RPC or from Python API? If it is the second, could you attach JSON-RPC response of stats?

Python API was used for stats. I attached output of console (having stats rpc) and streams in the post of May 25th/28th which were taken when all 3 ports were participating in transmission. Attaching that here. Please let me know if that will work for debugging. I don't have rpc stats for execution posted last and that server has been installed with other TRex versions for some other tests trials thus not in same state so would need to run test again if stats of that case is required.

[Divya Saxena]

Hi Yaroslav,

Did you get the chance to investigate the issue?

[Yaroslav Brustinov]

Hi,

I'm back, but not yet, sorry.

Thanks,

Yaroslav

On Mon, Jun 25, 2018, 7:10 AM Divya Saxena <divya...@gmail.com> wrote:

Hi Yaroslav,

Did you get the chance to investigate the issue?

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/dff00cdc-6e4d-4651-8782-d05101c74d6b%40googlegroups.com.

[Divya Saxena]

Ok, I'll wait for your troubleshooting. This is kind of blocking for the tests, your inputs will be helpful to figure out and resolve the issue.

[Yaroslav Brustinov]

Hi, Divya.

It would be good to get RPC correspondence with the branch I've shared (and single port transmission).

Also, it would be good [in addition] to see "xstats" at beginning and end of iteration 2.

Another thing, could you try the same tests with loopback or switch?

Could you confirm or refute that in the last test two iterations occurred after fresh run of TRex server?

Thanks,

Yaroslav.

On Tue, Jun 26, 2018 at 1:05 PM, Divya Saxena <divya...@gmail.com> wrote:

Ok, I'll wait for your troubleshooting. This is kind of blocking for the tests, your inputs will be helpful to figure out and resolve the issue.

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/06cacbcd-8d8c-466a-8a4e-bdba12e8f5c3%40googlegroups.com.

[Divya Saxena]

Hi Yaroslav,

Another thing, could you try the same tests with loopback or switch?

This issue is occurring for topology having more than 2 ports.

Could you confirm or refute that in the last test two iterations occurred after fresh run of TRex server?

Test having transmission from all 3 ports were executed after fresh run of TRex server. (posted on 28th May)

It would be good to get RPC correspondence with the branch I've shared (and single port transmission).
Also, it would be good [in addition] to see "xstats" at beginning and end of iteration 2.

I have re-executed the test with single port transmission after fresh start of TRex server (using your branch) and collected RPC console log. PFA trex-console-verbose.log. (Headers are included in the log to show when those commands were executed on console).

[Yaroslav Brustinov]

Hi,

I have re-executed the test with single port transmission after fresh start of TRex server (using your branch) and collected RPC console log. PFA trex-console-verbose.log. (Headers are included in the log to show when those commands were executed on console).

Are you sure that you have used my branch?

This is from latest response you attached:

            "build_date": "May 22 2018",
            "build_time": "02:49:36",
            "built_by": "root",
            "mode": "STL",
            "version": "v2.36"

Thanks,

Yaroslav

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/42f0f5b7-0ac3-43a0-b5e0-039694824c33%40googlegroups.com.

[Divya Saxena]

Yes, it was built from package which you shared (https://github.com/ybrustin/trex-core/tree/flow_stats_nan_rates).

[Yaroslav Brustinov]

Hi, Divya.

It's just strange that version is v2.36, while it's expected to be v2.41

Did the last attachment with xstats correspond to error in the flow stats?

I've pushed another branch:

https://github.com/ybrustin/trex-core/tree/flow_stats_debug

It has two defines which would allow to debug it better by printing of TRex server:

1. All RPC conversation (request and response)

2. Flow stats entry to functions.

Please run the server with "--iom 0" so that there will be no usual statistics prints, only the debug ones.

Please attach output of server, and erroneous flow stats.

Also, please share the command you run the server.

I can't reproduce your error.

If you could send single Python file to run and reproduce the error, it would help much.

Thanks,

Yaroslav.

On Fri, Jun 29, 2018 at 6:02 AM, Divya Saxena <divya...@gmail.com> wrote:

Yes, it was built from package which you shared (https://github.com/ybrustin/trex-core/tree/flow_stats_nan_rates).

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/81b86083-4736-47c3-a220-3270bed331eb%40googlegroups.com.

[Divya Saxena]

Hi Yaroslav,

Did the last attachment with xstats correspond to error in the flow stats?

 

I captured xstats just after traffic stop and after some delay just in case it is required to see if anything abnormal happened due to timing. I thought it might be helpful for you to debug. What stats I saw at python HLTAPI side are mentioned below. (format: port,<mode of stats>,pg_id,<rx/tx>,<stats name>)

secIter(14,stream,102,rx,line_rate_percentage) = 0.0
secIter(14,stream,102,rx,total_pkt_bit_rate)   = 0
secIter(14,stream,102,rx,total_pkt_bytes)      = N/A
secIter(14,stream,102,rx,total_pkt_rate)       = 0
secIter(14,stream,102,rx,total_pkts)           = 0
secIter(14,stream,102,rx,total_pkts_bytes)     = N/A
secIter(14,stream,102,tx,line_rate_percentage) = 0.0
secIter(14,stream,102,tx,total_pkt_bit_rate)   = 0
secIter(14,stream,102,tx,total_pkt_bytes)      = 0
secIter(14,stream,102,tx,total_pkt_rate)       = 0
secIter(14,stream,102,tx,total_pkts)           = 0
secIter(14,stream,102,tx,total_pkts_bytes)     = 0
secIter(14,stream,103,rx,line_rate_percentage) = 0.0
secIter(14,stream,103,rx,total_pkt_bit_rate)   = 0
secIter(14,stream,103,rx,total_pkt_bytes)      = N/A
secIter(14,stream,103,rx,total_pkt_rate)       = 0
secIter(14,stream,103,rx,total_pkts)           = 6696427
secIter(14,stream,103,rx,total_pkts_bytes)     = N/A
secIter(14,stream,103,tx,line_rate_percentage) = 0.0
secIter(14,stream,103,tx,total_pkt_bit_rate)   = 0
secIter(14,stream,103,tx,total_pkt_bytes)      = 0
secIter(14,stream,103,tx,total_pkt_rate)       = 0
secIter(14,stream,103,tx,total_pkts)           = 0
secIter(14,stream,103,tx,total_pkts_bytes)     = 0
secIter(14,stream,201,rx,line_rate_percentage) = 0.0
secIter(14,stream,201,rx,total_pkt_bit_rate)   = 0
secIter(14,stream,201,rx,total_pkt_bytes)      = N/A
secIter(14,stream,201,rx,total_pkt_rate)       = 0
secIter(14,stream,201,rx,total_pkts)           = 0
secIter(14,stream,201,rx,total_pkts_bytes)     = N/A
secIter(14,stream,201,tx,line_rate_percentage) = 0.0
secIter(14,stream,201,tx,total_pkt_bit_rate)   = 0
secIter(14,stream,201,tx,total_pkt_bytes)      = 0
secIter(14,stream,201,tx,total_pkt_rate)       = 0
secIter(14,stream,201,tx,total_pkts)           = 0
secIter(14,stream,201,tx,total_pkts_bytes)     = 0
secIter(14,stream,203,rx,line_rate_percentage) = 0.0
secIter(14,stream,203,rx,total_pkt_bit_rate)   = 0
secIter(14,stream,203,rx,total_pkt_bytes)      = N/A
secIter(14,stream,203,rx,total_pkt_rate)       = 0
secIter(14,stream,203,rx,total_pkts)           = 0
secIter(14,stream,203,rx,total_pkts_bytes)     = N/A
secIter(14,stream,203,tx,line_rate_percentage) = 0.0
secIter(14,stream,203,tx,total_pkt_bit_rate)   = 0
secIter(14,stream,203,tx,total_pkt_bytes)      = 0
secIter(14,stream,203,tx,total_pkt_rate)       = 0
secIter(14,stream,203,tx,total_pkts)           = 0
secIter(14,stream,203,tx,total_pkts_bytes)     = 0
secIter(14,stream,301,rx,line_rate_percentage) = 0.0
secIter(14,stream,301,rx,total_pkt_bit_rate)   = 0
secIter(14,stream,301,rx,total_pkt_bytes)      = N/A
secIter(14,stream,301,rx,total_pkt_rate)       = 0
secIter(14,stream,301,rx,total_pkts)           = 0
secIter(14,stream,301,rx,total_pkts_bytes)     = N/A
secIter(14,stream,301,tx,line_rate_percentage) = 0.0
secIter(14,stream,301,tx,total_pkt_bit_rate)   = 0
secIter(14,stream,301,tx,total_pkt_bytes)      = 0
secIter(14,stream,301,tx,total_pkt_rate)       = 0
secIter(14,stream,301,tx,total_pkts)           = 0
secIter(14,stream,301,tx,total_pkts_bytes)     = 0
secIter(14,stream,302,rx,line_rate_percentage) = 0.0
secIter(14,stream,302,rx,total_pkt_bit_rate)   = 0
secIter(14,stream,302,rx,total_pkt_bytes)      = N/A
secIter(14,stream,302,rx,total_pkt_rate)       = 0
secIter(14,stream,302,rx,total_pkts)           = 0
secIter(14,stream,302,rx,total_pkts_bytes)     = N/A
secIter(14,stream,302,tx,line_rate_percentage) = 0.0
secIter(14,stream,302,tx,total_pkt_bit_rate)   = 0
secIter(14,stream,302,tx,total_pkt_bytes)      = 0
secIter(14,stream,302,tx,total_pkt_rate)       = 0
secIter(14,stream,302,tx,total_pkts)           = 0
secIter(14,stream,302,tx,total_pkts_bytes)     = 0
secIter(6,stream,102,rx,line_rate_percentage)  = 0.0
secIter(6,stream,102,rx,total_pkt_bit_rate)    = 0
secIter(6,stream,102,rx,total_pkt_bytes)       = N/A
secIter(6,stream,102,rx,total_pkt_rate)        = 0
secIter(6,stream,102,rx,total_pkts)            = 0
secIter(6,stream,102,rx,total_pkts_bytes)      = N/A
secIter(6,stream,102,tx,line_rate_percentage)  = 0.0
secIter(6,stream,102,tx,total_pkt_bit_rate)    = 0
secIter(6,stream,102,tx,total_pkt_bytes)       = 64
secIter(6,stream,102,tx,total_pkt_rate)        = 0
secIter(6,stream,102,tx,total_pkts)            = 1
secIter(6,stream,102,tx,total_pkts_bytes)      = 64
secIter(6,stream,103,rx,line_rate_percentage)  = 0.0
secIter(6,stream,103,rx,total_pkt_bit_rate)    = 0
secIter(6,stream,103,rx,total_pkt_bytes)       = N/A
secIter(6,stream,103,rx,total_pkt_rate)        = 0
secIter(6,stream,103,rx,total_pkts)            = 6696427
secIter(6,stream,103,rx,total_pkts_bytes)      = N/A
secIter(6,stream,103,tx,line_rate_percentage)  = 0.0
secIter(6,stream,103,tx,total_pkt_bit_rate)    = 0
secIter(6,stream,103,tx,total_pkt_bytes)       = 428571328
secIter(6,stream,103,tx,total_pkt_rate)        = 0
secIter(6,stream,103,tx,total_pkts)            = 6696427
secIter(6,stream,103,tx,total_pkts_bytes)      = 428571328
secIter(6,stream,201,rx,line_rate_percentage)  = 0.0
secIter(6,stream,201,rx,total_pkt_bit_rate)    = 0
secIter(6,stream,201,rx,total_pkt_bytes)       = N/A
secIter(6,stream,201,rx,total_pkt_rate)        = 0
secIter(6,stream,201,rx,total_pkts)            = 0
secIter(6,stream,201,rx,total_pkts_bytes)      = N/A
secIter(6,stream,201,tx,line_rate_percentage)  = 0.0
secIter(6,stream,201,tx,total_pkt_bit_rate)    = 0
secIter(6,stream,201,tx,total_pkt_bytes)       = 0
secIter(6,stream,201,tx,total_pkt_rate)        = 0
secIter(6,stream,201,tx,total_pkts)            = 0
secIter(6,stream,201,tx,total_pkts_bytes)      = 0
secIter(6,stream,203,rx,line_rate_percentage)  = 0.0
secIter(6,stream,203,rx,total_pkt_bit_rate)    = 0
secIter(6,stream,203,rx,total_pkt_bytes)       = N/A
secIter(6,stream,203,rx,total_pkt_rate)        = 0
secIter(6,stream,203,rx,total_pkts)            = 0
secIter(6,stream,203,rx,total_pkts_bytes)      = N/A
secIter(6,stream,203,tx,line_rate_percentage)  = 0.0
secIter(6,stream,203,tx,total_pkt_bit_rate)    = 0
secIter(6,stream,203,tx,total_pkt_bytes)       = 0
secIter(6,stream,203,tx,total_pkt_rate)        = 0
secIter(6,stream,203,tx,total_pkts)            = 0
secIter(6,stream,203,tx,total_pkts_bytes)      = 0
secIter(6,stream,301,rx,line_rate_percentage)  = 0.0
secIter(6,stream,301,rx,total_pkt_bit_rate)    = 0
secIter(6,stream,301,rx,total_pkt_bytes)       = N/A
secIter(6,stream,301,rx,total_pkt_rate)        = 0
secIter(6,stream,301,rx,total_pkts)            = 0
secIter(6,stream,301,rx,total_pkts_bytes)      = N/A
secIter(6,stream,301,tx,line_rate_percentage)  = 0.0
secIter(6,stream,301,tx,total_pkt_bit_rate)    = 0
secIter(6,stream,301,tx,total_pkt_bytes)       = 0
secIter(6,stream,301,tx,total_pkt_rate)        = 0
secIter(6,stream,301,tx,total_pkts)            = 0
secIter(6,stream,301,tx,total_pkts_bytes)      = 0
secIter(6,stream,302,rx,line_rate_percentage)  = 0.0
secIter(6,stream,302,rx,total_pkt_bit_rate)    = 0
secIter(6,stream,302,rx,total_pkt_bytes)       = N/A
secIter(6,stream,302,rx,total_pkt_rate)        = 0
secIter(6,stream,302,rx,total_pkts)            = 0
secIter(6,stream,302,rx,total_pkts_bytes)      = N/A
secIter(6,stream,302,tx,line_rate_percentage)  = 0.0
secIter(6,stream,302,tx,total_pkt_bit_rate)    = 0
secIter(6,stream,302,tx,total_pkt_bytes)       = 0
secIter(6,stream,302,tx,total_pkt_rate)        = 0
secIter(6,stream,302,tx,total_pkts)            = 0
secIter(6,stream,302,tx,total_pkts_bytes)      = 0
secIter(7,stream,102,rx,line_rate_percentage)  = 0.0
secIter(7,stream,102,rx,total_pkt_bit_rate)    = 0
secIter(7,stream,102,rx,total_pkt_bytes)       = N/A
secIter(7,stream,102,rx,total_pkt_rate)        = 0
secIter(7,stream,102,rx,total_pkts)            = 1
secIter(7,stream,102,rx,total_pkts_bytes)      = N/A
secIter(7,stream,102,tx,line_rate_percentage)  = 0.0
secIter(7,stream,102,tx,total_pkt_bit_rate)    = 0
secIter(7,stream,102,tx,total_pkt_bytes)       = 0
secIter(7,stream,102,tx,total_pkt_rate)        = 0
secIter(7,stream,102,tx,total_pkts)            = 0
secIter(7,stream,102,tx,total_pkts_bytes)      = 0
secIter(7,stream,103,rx,line_rate_percentage)  = 0.0
secIter(7,stream,103,rx,total_pkt_bit_rate)    = 0
secIter(7,stream,103,rx,total_pkt_bytes)       = N/A
secIter(7,stream,103,rx,total_pkt_rate)        = 0
secIter(7,stream,103,rx,total_pkts)            = 0
secIter(7,stream,103,rx,total_pkts_bytes)      = N/A
secIter(7,stream,103,tx,line_rate_percentage)  = 0.0
secIter(7,stream,103,tx,total_pkt_bit_rate)    = 0
secIter(7,stream,103,tx,total_pkt_bytes)       = 0
secIter(7,stream,103,tx,total_pkt_rate)        = 0
secIter(7,stream,103,tx,total_pkts)            = 0
secIter(7,stream,103,tx,total_pkts_bytes)      = 0
secIter(7,stream,201,rx,line_rate_percentage)  = 0.0
secIter(7,stream,201,rx,total_pkt_bit_rate)    = 0
secIter(7,stream,201,rx,total_pkt_bytes)       = N/A
secIter(7,stream,201,rx,total_pkt_rate)        = 0
secIter(7,stream,201,rx,total_pkts)            = 0
secIter(7,stream,201,rx,total_pkts_bytes)      = N/A
secIter(7,stream,201,tx,line_rate_percentage)  = 0.0
secIter(7,stream,201,tx,total_pkt_bit_rate)    = 0
secIter(7,stream,201,tx,total_pkt_bytes)       = 0
secIter(7,stream,201,tx,total_pkt_rate)        = 0
secIter(7,stream,201,tx,total_pkts)            = 0
secIter(7,stream,201,tx,total_pkts_bytes)      = 0
secIter(7,stream,203,rx,line_rate_percentage)  = 0.0
secIter(7,stream,203,rx,total_pkt_bit_rate)    = 0
secIter(7,stream,203,rx,total_pkt_bytes)       = N/A
secIter(7,stream,203,rx,total_pkt_rate)        = 0
secIter(7,stream,203,rx,total_pkts)            = 0
secIter(7,stream,203,rx,total_pkts_bytes)      = N/A
secIter(7,stream,203,tx,line_rate_percentage)  = 0.0
secIter(7,stream,203,tx,total_pkt_bit_rate)    = 0
secIter(7,stream,203,tx,total_pkt_bytes)       = 0
secIter(7,stream,203,tx,total_pkt_rate)        = 0
secIter(7,stream,203,tx,total_pkts)            = 0
secIter(7,stream,203,tx,total_pkts_bytes)      = 0
secIter(7,stream,301,rx,line_rate_percentage)  = 0.0
secIter(7,stream,301,rx,total_pkt_bit_rate)    = 0
secIter(7,stream,301,rx,total_pkt_bytes)       = N/A
secIter(7,stream,301,rx,total_pkt_rate)        = 0
secIter(7,stream,301,rx,total_pkts)            = 0
secIter(7,stream,301,rx,total_pkts_bytes)      = N/A
secIter(7,stream,301,tx,line_rate_percentage)  = 0.0
secIter(7,stream,301,tx,total_pkt_bit_rate)    = 0
secIter(7,stream,301,tx,total_pkt_bytes)       = 0
secIter(7,stream,301,tx,total_pkt_rate)        = 0
secIter(7,stream,301,tx,total_pkts)            = 0
secIter(7,stream,301,tx,total_pkts_bytes)      = 0
secIter(7,stream,302,rx,line_rate_percentage)  = 0.0
secIter(7,stream,302,rx,total_pkt_bit_rate)    = 0
secIter(7,stream,302,rx,total_pkt_bytes)       = N/A
secIter(7,stream,302,rx,total_pkt_rate)        = 0
secIter(7,stream,302,rx,total_pkts)            = 0
secIter(7,stream,302,rx,total_pkts_bytes)      = N/A
secIter(7,stream,302,tx,line_rate_percentage)  = 0.0
secIter(7,stream,302,tx,total_pkt_bit_rate)    = 0
secIter(7,stream,302,tx,total_pkt_bytes)       = 0
secIter(7,stream,302,tx,total_pkt_rate)        = 0
secIter(7,stream,302,tx,total_pkts)            = 0
secIter(7,stream,302,tx,total_pkts_bytes)      = 0
secIter(arp)                                   = -
secIter(description)                           = Ethernet Controller X710 for 10GbE SFP+
secIter(dest)                                  = 00:00:00:00:1f:00
secIter(driver)                                = net_i40e
secIter(fc)                                    = none
secIter(fc_supported)                          = no
secIter(grat_arp)                              = off
secIter(hw_mac)                                = 00:90:0b:6d:37:32
secIter(index)                                 = 14
secIter(is_fc_supported)                       = 0
secIter(is_led_supported)                      = 1
secIter(is_link_supported)                     = 1
secIter(is_virtual)                            = no
secIter(layer_mode)                            = Ethernet
secIter(led_change_supported)                  = yes
secIter(link)                                  = UP
secIter(link_change_supported)                 = yes
secIter(log)                                   =
secIter(mult)                                  = off
secIter(numa)                                  = 0
secIter(pci_addr)                              = 0000:00:1e.0
secIter(port_handle,14)                        = 14
secIter(port_handle,6)                         = 6
secIter(port_handle,7)                         = 7
secIter(prom)                                  = on
secIter(prom_supported)                        = N/A
secIter(rx,caps)                               = flow_stats latency
secIter(rx,counters)                           = 127
secIter(rx_filter_mode)                        = hardware match
secIter(rx_queue)                              = off
secIter(speed)                                 = 10
secIter(src_ipv4)                              = -
secIter(src_mac)                               = 00:00:00:00:1e:00
secIter(status)                                = 1
secIter(stopped)                               = 1
secIter(stream_id)                             = 302
secIter(supp_speeds)                           = 1000 10000
secIter(vlan)                                  = -

I've pushed another branch:
https://github.com/ybrustin/trex-core/tree/flow_stats_debug
It has two defines which would allow to debug it better by printing of TRex server:
1. All RPC conversation (request and response)
2. Flow stats entry to functions.
Please run the server with "--iom 0" so that there will be no usual statistics prints, only the debug ones.
Please attach output of server, and erroneous flow stats.

I'll do that.

Also, please share the command you run the server.

Similar to other services, a service is created for trex and started service as 'systemctl start trex'.

[root@htrex-20311 ~]# cat /usr/lib/systemd/system/trex.service

[Unit]

Description=TRex Server Service

[Service]

ExecStartPre=/bin/sleep 30

WorkingDirectory=/corp/sta/opt/TRex/trex-core-patch

ExecStart=/corp/sta/opt/TRex/trex-core-patch/t-rex-64 -i

StandardOutput=null

StandardError=syslog

Restart=always

RestartSec=30

[Install]

WantedBy=multi-user.target

If you could send single Python file to run and reproduce the error, it would help much.

I'll try to create it.

[Yaroslav Brustinov]

Hi,

Just to extract relevant stats from your last print of stats:

secIter(6,stream,102,tx,total_pkts)            = 1

secIter(7,stream,102,rx,total_pkts)            = 1

secIter(6,stream,103,rx,total_pkts)            = 6696427

secIter(6,stream,103,tx,total_pkts)            = 6696427

secIter(14,stream,103,rx,total_pkts)           = 6696427

I've added just now another commit to repo https://github.com/ybrustin/trex-core/tree/flow_stats_debug

Please update if you have cloned already.

Thanks,

Yaroslav.

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/3279933b-7d6e-4add-b23d-a9b4e9b60b74%40googlegroups.com.

[Yaroslav Brustinov]

Hi, Divya.

We have found the issue.

I've opened defect in youtrack with fuller explanation and pictures:

https://trex-tgn.cisco.com/youtrack/issue/trex-531

The plan for now is to detect this situation, print message to user and exit.

Thanks,

Yaroslav

...

[Message clipped]  

[Divya Saxena]

Oh, that's nice. Thanks for your support.

btw I executed test with your new branch and also created a python script to reproduce issue but couldn't share with you yesterday as I was out of office. 

Since you got the root cause so I think its not needed now.

I have gone through the ticket. So as you said that it'll not be fixed now just notify user about the error situation, any idea by which release it could be fixed? 

[Yaroslav Brustinov]

Hi,

So as you said that it'll not be fixed now just notify user about the error situation, any idea by which release it could be fixed? 

We have found elegant solution, to use PF id of interface, which is persistent cross host-guest machines:

​

Will be used in next version (v2.42)

Thanks,

Yaroslav.

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/637d117d-1b94-4681-bc28-91734a62e669%40googlegroups.com.

[Divya Saxena]

Hi Yaroslav,

Will be used in next version (v2.42)

Thanks. When this release will be available to users?

[hanoh haim]

Hi Divya,

TRex version v2.43 was released with a few bug fixes. 

WARNING: this version includes the consolidation of Client Python ASTF so a few things was changed in Python side, read the release notes for more info.

 

thanks,

Hanoh

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/edc1242c-8204-47fc-9980-ef1c82cea9c8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

[Divya Saxena]

Ok. Thanks.