stream tx-stats empty

[Andreas Bourges]

Hi,

...me again :(

After fixing the VIC1225 problem on our side, I got the first streams up and running again. TUI show that packets
on both ports are sent and received. But if I look at stream-stats, I only see "RX packets" - no TX ?!

I know, I had a similar problem a while ago, when the traffic was actually not reaching the rx-port, but now I did verify,
that traffic makes its way through the test-network and is actually delivered to the trex-rx-port.

How can I provide more information or have a deeper look myself, on where the problem might be located?

Thanks,

Andreas

[hanoh haim]

Andreas,

It looks like a bug. Tx is taken from DP cores. TUI was changed for scale 

Could you check the API?

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/5e42d294-9437-410b-b78d-2a1a09ef647f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

[Andreas Bourges]

Hi Hanoh,

On Freitag, 31. März 2017 09:26:31 CEST hanoh haim wrote:
> It looks like a bug. Tx is taken from DP cores. TUI was changed for scale
> Could you check the API?

...oh - didn't realize 2.23 is already out. I'm currently downloading v2.23
and perform the tests again. I'll send an update in the afternoon.

regards,

Andreas

[Andreas Bourges]

Hi Hanoh,

...did a quick check and the problem still persists. I'm running 2 streams in
this test (one for port0 -> port 1 and the other from port1 -> port0, each
approx 1000 pps). Please see the statistics below.

Regarding the relatively high rx packets - they're most likely caused by the
HSRP that's running in the vlans. But this shouldn't cause any trouble, right
(because they have no pg_id/flow_id set)?

regards,

Andreas


{0: {'ibytes': 2909746,
'ierrors': 0,
'ipackets': 23682,
'obytes': 635072,
'oerrors': 0,
'opackets': 6108,
'rx_bps': 2366770.8,
'rx_bps_L1': 2764530.8,
'rx_pps': 2486.0,
'rx_util': 0.027645307999999997,
'tx_bps': 811243.6,
'tx_bps_L1': 967259.6,
'tx_pps': 975.1,
'tx_util': 0.009672595999999999},
1: {'ibytes': 2909622,
'ierrors': 0,
'ipackets': 23678,
'obytes': 635072,
'oerrors': 0,
'opackets': 6108,
'rx_bps': 2366345.0,
'rx_bps_L1': 2764009.0,
'rx_pps': 2485.4,
'rx_util': 0.02764009,
'tx_bps': 811168.8,
'tx_bps_L1': 967168.7999999999,
'tx_pps': 975.0,
'tx_util': 0.009671688},
'flow_stats': {46: {'rx_bps': {1: 831791.3647601862,
'total': 831791.3647601862},
'rx_bps_L1': {1: 991751.2425986835,
'total': 991751.2425986835},
'rx_bytes': {1: 634816, 'total': 634816},
'rx_pkts': {1: 6104, 'total': 6104},
'rx_pps': {1: 999.7492364906085,
'total': 999.7492364906085},
'tx_bps': {'total': 'N/A'},
'tx_bps_L1': {'total': 'N/A'},
'tx_bytes': {'total': 'N/A'},
'tx_pkts': {'total': 'N/A'},
'tx_pps': {'total': 'N/A'}},
204: {'rx_bps': {0: 831791.3647601862,
'total': 831791.3647601862},
'rx_bps_L1': {0: 991751.2425986835,
'total': 991751.2425986835},
'rx_bytes': {0: 634816, 'total': 634816},
'rx_pkts': {0: 6104, 'total': 6104},
'rx_pps': {0: 999.7492364906085,
'total': 999.7492364906085},
'tx_bps': {'total': 'N/A'},
'tx_bps_L1': {'total': 'N/A'},
'tx_bytes': {'total': 'N/A'},
'tx_pkts': {'total': 'N/A'},
'tx_pps': {'total': 'N/A'}},
'global': {'rx_err': {}, 'tx_err': {}}},
'global': {'bw_per_core': 15.9,
'cpu_util': 0.02035,
'queue_full': 0,
'rx_bps': 4733115.5,
'rx_cpu_util': 0.126,
'rx_drop_bps': 0.0,
'rx_pps': 4971.4,
'tx_bps': 1622412.4,
'tx_pps': 1950.0},
'latency': {'global': {'bad_hdr': 0, 'old_flow': 0}},
'total': {'ibytes': 5819368,
'ierrors': 0,
'ipackets': 47360,
'obytes': 1270144,
'oerrors': 0,
'opackets': 12216,
'rx_bps': 4733115.8,
'rx_bps_L1': 5528539.8,
'rx_pps': 4971.4,
'rx_util': 0.055285398,
'tx_bps': 1622412.4,
'tx_bps_L1': 1934428.4,
'tx_pps': 1950.1,
'tx_util': 0.019344283999999996}}

On Freitag, 31. März 2017 09:26:31 CEST hanoh haim wrote:

[hanoh haim]

In both software and hardware modes those counters are taken from software.

Probably related to new --software mode (wonder how our regression work?)

Ido will have a look 

Hanoh 

[hanoh haim]

Another interesting thing from your setup

Total Tx=1.7Mbps

Total Rx=4Mbps 

Is it expected ?

[Andreas Bourges]

I think yes, but need to do some math first :) hsrp for a lot of vlans tuned for fast failover might result in high Rx counters...
-- sent from mobile

[Andreas Bourges]

Hi Hanoh,

...instead of using math, I saved a capture from TRex. As expected there's
roughly 1.5 mbps of HSRP received per interface. Since we're trunking all vlans
to the qinq switch and trex, that's expected.

What might be a problem - the current setup sends the packets to both trex-ports.
So, we're sending traffic on port-0 and it's injected into the lab-network. On the
other side of the network, the traffic gets delivered to the qinq switch and sent to
both ports of trex. Could that cause the statistics problem?

I've been aware of the issue for quite a while, but hoped, that the sending trex port would just
ignore the packets, that should be received on the other port?!

thanks and regards,

Andreas

[ido barnea]

Hi Andreas,
I’ll have a look on the issues you described first thing Sunday morning.
Regarding the below. This is how the flow statistics work. We count packets reaching from all interfaces.
We thought this is better like that, since user can have all kind of setups, and we can’t anticipate which interface the traffic will come in from.
Do you have better idea how to implement?

Thanks,
Ido

> --
> You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
> To post to this group, send email to trex...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/9210f885-bca2-4626-9e1e-46f7f9de00b5%40googlegroups.com.

[Andreas Bourges]

Hi Ido,

On Freitag, 31. März 2017 17:16:30 CEST ido barnea wrote:
> I’ll have a look on the issues you described first thing Sunday morning.

That would really be great. I'm still not sure if TRex is to be blamed, but
somehow we have to work out, what the problem is (might still be in the setup,
but I double-checked with my colleague michael today once more...).

> Regarding the below. This is how the flow statistics work. We count packets
> reaching from all interfaces. We thought this is better like that, since
> user can have all kind of setups, and we can’t anticipate which interface
> the traffic will come in from. Do you have better idea how to implement?

From the docs I got the impression that it's always a tx -> rx relationship,
because only pairs of ports can be used with TRex. That's not really a
problem, as long as we know how TRex behaves. We've been thinking today
again about duplicate packets, that might arrive at the transmitting port, but
we now have the strong opinion, that this is not possible with the current
setup. So we should be fine from that perspective and don't have to change
anything here in the setup.

If I can assist you in debugging the issue, please let me know. I just
confirmed, that packets *are* received by TRex - after ~1500s the arp cache on
the device timed out and statistics stopped increasing (RX packets stopped
increasing while TX was still N/A).

One side note - we've roughly 4 weeks left until we go into the CPOC. If we
would decide, that we can't use TRex with our use-case, it would mean that we
would have to prepare for the spirent device, which in turn means additional
work. So the pressure on us to have this up-and-running is steadily
increasing. That's not meant as an offence (I think you're really doing a
great work and super-fast and friendly support), but maybe TRex is not (yet)
as evolved as we would need it. From my opinion we have at most another week,
before we must be *very* sure, that things work out with TRex or we sadly have
to switch to spirent. I don't want to put additional pressure on you, but I
just want to be clear about our options.

Thanks for your support,

Andreas

[Itay Marom]

Hi Andreas,

As Ido guessed, you were running under service mode and using RX stats.

we have an open bug for this that was submitted few weeks ago:

https://trex-tgn.cisco.com/youtrack/issue/trex-371 - Latency stats does not work with service mode on

Anyway,

I've working on a fix currently and already validated it to be working.

i'll provide you a patch in half an hour or so for validation.

Thanks,

Itay

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/5566039.6vKMihxkcr%40hal.

For more options, visit https://groups.google.com/d/optout.

--

Itay Marom

[Itay Marom]

Hi Andreas,

Can you please try the attached patch ?

Thanks,

Itay

--

Itay Marom

[Andreas Bourges]

Hi Itay,

...thanks for your assistance. I'm currently compiling and will try to test
the patch tomorrow. However - I made a mistake by not stopping service-mode
for the ports. It was a one-liner to fix this in our code.

Since we are experiencing heavy time pressure, I have to put emphasis on the
open topics this week, so we can confidently plan with TRex . I will certainly
test the patch (even if it is not really relevant for us), but this is
currently no prio-1 topic. I'll let you know if it would work for us, once I
tested the patch.

Thank you for your efforts!!!

regards,

Andreas

> >> email to trex-tgn+u...@googlegroups.com.

[Itay Marom]

Sure....just one last thing:

Under service mode, trying to activate traffic will generate a warning...unless suppressed with force.

Did you use force as well ?

Just trying to understand the scenario.

Thanks,

Itay

> >> email to trex-tgn+unsubscribe@googlegroups.com.

[Andreas Bourges]

Hi Itay,

On Montag, 3. April 2017 20:27:40 CEST Itay Marom wrote:
> Under service mode, trying to activate traffic will generate a
> warning...unless suppressed with force.
> Did you use force as well ?
> Just trying to understand the scenario.

Yes - this is the line :(

self.c.start(ports=self.getStreamPorts(), duration = duration
,force=True)


Thanks,

Andreas

> > > >> email to trex-tgn+u...@googlegroups.com.

[Itay Marom]

Well, if I was a lawyer I could say that using 'force' gives us a free pass :)...

But i'm not :)...so we will fix the issue in the next version...

Thanks for finding this.

Itay

> > > >> email to trex-tgn+unsubscribe@googlegroups.com.

> > > >> To post to this group, send email to trex...@googlegroups.com.
> > > >> To view this discussion on the web visit
> >
> > https://groups.google.com/d/ms
> >
> > > >> gid/trex-tgn/5566039.6vKMihxkcr%40hal.
> > > >> For more options, visit https://groups.google.com/d/optout.
> > > >
> > > > --
> > > > Itay Marom

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/6040304.1UrkCQTfTD%40hal.

[Andreas Bourges]

Itay,

On Montag, 3. April 2017 20:44:06 CEST Itay Marom wrote:
> Well, if I was a lawyer I could say that using 'force' gives us a free pass
> :)...

...I know - and I'm not complaining ;-) However, I'm glad you all helped me
out here, because I've been reading the code over and over again and didn't
spot the issue :(

> But i'm not :)...so we will fix the issue in the next version...
> Thanks for finding this.

...thanks for your great help!

Andreas

> > > > > >> email to trex-tgn+u...@googlegroups.com.

> > > > > >> To post to this group, send email to trex...@googlegroups.com.
> > > > > >> To view this discussion on the web visit
> > > >
> > > > https://groups.google.com/d/ms
> > > >
> > > > > >> gid/trex-tgn/5566039.6vKMihxkcr%40hal.
> > > > > >> For more options, visit https://groups.google.com/d/optout.
> > > > > >
> > > > > > --
> > > > > > Itay Marom
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "TRex Traffic Generator" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to trex-tgn+u...@googlegroups.com.

> > To post to this group, send email to trex...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/

> > msgid/trex-tgn/6040304.1UrkCQTfTD%40hal.

[Andreas Bourges]

Good Morning!

..I applied the patch and started traffic while service-mode was still on - it
worked! Got TX and RX stats.

Tried to disable force when starting the streams, but then I knew why I had to
use it in the first place:

Since we're doing our own ARPs and not using the ip addresses on the TRex
ports itself, TRex gives a warning and refuses to start, because the ARP
resolution for the ports has not been performed!

"trex_stl_lib.trex_stl_exceptions.STLError: Port(s) [0, 1] have unresolved
destination addresses - please resolve them or specify 'force'"

> Well, if I was a lawyer I could say that using 'force' gives us a free pass
> :)...

Yep - but I think my chances got better now to win the law suit ;-)

Anyway, patch works!

Have a nice day!

Andreas

On Montag, 3. April 2017 20:44:06 CEST Itay Marom wrote:

> > > > > >> email to trex-tgn+u...@googlegroups.com.

> > > > > >> To post to this group, send email to trex...@googlegroups.com.
> > > > > >> To view this discussion on the web visit
> > > >
> > > > https://groups.google.com/d/ms
> > > >
> > > > > >> gid/trex-tgn/5566039.6vKMihxkcr%40hal.
> > > > > >> For more options, visit https://groups.google.com/d/optout.
> > > > > >
> > > > > > --
> > > > > > Itay Marom
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "TRex Traffic Generator" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to trex-tgn+u...@googlegroups.com.

> > To post to this group, send email to trex...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/

> > msgid/trex-tgn/6040304.1UrkCQTfTD%40hal.