t-rex update mac address only at startup, then loose it

Lorenzo Lolli

unread,

Feb 15, 2018, 9:46:37 AM2/15/18

to TRex Traffic Generator

Greetings,

I'm a new user of t-rex, and I find a very interesting tools. Let me thank you for developing it.

I've setup a small lab, with a t-rex server with 2 intel NIC, directly connected to a DUT (a router in my case). Please note that I've followed the documentation http://trex-tgn.cisco.com/trex/doc/trex_config_guide.html

I've found a strange behaviour, it seems that t-rex in my env does learn gw mac address only on startup, then it forgive it.

This is how I can reproduce the problem.

T-rex version 2.36
Trex interfaces:

| 2 | -1 | 04:00.0 | 00:1b:21:48:33:30 | 82576 Gigabit Network Connection | igb_uio | | |
| 3 | -1 | 04:00.1 | 00:1b:21:48:33:31 | 82576 Gigabit Network Connection | igb_uio | | |

trex-cfg.yaml

### Config file generated by dpdk_setup_ports.py ###

- port_limit: 2
version: 2
interfaces: ['04:00.0', '04:00.1']
port_info:
- ip: 192.168.1.10
default_gw: 192.168.1.1
- ip: 192.168.2.10
default_gw: 192.168.2.1

platform:
master_thread_id: 0
latency_thread_id: 1
dual_if:
- socket: 0
threads: [2,3,4,5,6,7,8,9,10,11,12,13,14,15]

DUT Interfaces:
eth0 00:D0:D6:55:5A:3D 192.168.1.1
lan-ptp 00:D0:D6:55:5A:41 192.168.2.1

the connections is as follow:

eth0 00:D0:D6:55:5A:3D 192.168.1.1 <---> 192.168.1.10 00:1b:21:48:33:30
lan-ptp 00:D0:D6:55:5A:41 192.168.2.1 <---> 192.168.2.10 00:1b:21:48:33:31

Then, I startup trex in interactive mode:

./t-rex-64 -i -c 2

and I run the Console

./trex-console

Using 'python' as Python interpeter
Connecting to RPC server on localhost:4501 [SUCCESS]
Connecting to publisher server on localhost:4500 [SUCCESS]
Acquiring ports [0, 1]: [SUCCESS]

Server Info:
Server version: v2.36 @ STL
Server mode: Stateless
Server CPU: 2 x Intel(R) Xeon(R) CPU E5630 @ 2.53GHz
Ports count: 2 x 1Gbps @ 82576 Gigabit Network Connection
-=TRex Console v2.0=-

after that, I check the port status with portattr

trex>portattr
Port Status

everything looks OK. BUT, if I try to resolve mac address again:

trex>service

Enabling service mode on port(s) [0, 1]: [SUCCESS]

13.11 [ms]

trex(service)>resolve

Resolving destination on port(s) [0, 1]: [FAILED]

Port 0 - *** Failed to receive ARP response from 192.168.1.1
Port 1 - *** Failed to receive ARP response from 192.168.2.1

6.03 [sec]

I get an error.
Please note that if I clear arp on my DUT before running resolve command on t-rex, the arp table will be populated again as expected.
Only on t-rex side I cannot resolve arp.

And of course, I cannot ping:

trex(service)>ping -p 0 -n 1 -d 192.168.1.1

Pinging 192.168.1.1 from port 0 with 64 bytes of data:
Request timed out.
trex(service)>ping -p 1 -n 1 -d 192.168.2.1

Pinging 192.168.2.1 from port 1 with 64 bytes of data:
Request timed out.

this happen if I try to force resolve/update mac address or not.

If I run a loopback tests, everything works fine.

Can you please help me?

hanoh haim

unread,

Feb 15, 2018, 9:52:44 AM2/15/18

to Lorenzo Lolli, TRex Traffic Generator

Hi,

There is an ability to capture the traffic in service mode.

Could you capture in both cases?

thanks,

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/af031d5e-4781-4de9-aa83-0fa81bc66244%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

Lorenzo Lolli

unread,

Feb 15, 2018, 10:05:31 AM2/15/18

to TRex Traffic Generator

I've tryed to capture traffic, but nothing showed up:

trex>service

Enabling service mode on port(s) [0, 1]: [SUCCESS]

14.23 [ms]

trex(service)>capture monitor start --rx 0 -v

Starting stdout capture monitor - verbose: 'high' [SUCCESS]

*** use 'capture monitor stop' to abort capturing... ***

trex(service)>arp -p 0

Resolving destination on port(s) [0]: [FAILED]

Port 0 - *** Failed to receive ARP response from 192.168.1.1

3.01 [sec]

trex(service)>

While I was expecting the ARP request at least.
Am I missing something?

Yaroslav Brustinov

unread,

Feb 15, 2018, 10:10:41 AM2/15/18

to Lorenzo Lolli, TRex Traffic Generator

Hi,

add --tx 0 for request:

capture monitor start --rx 0 --tx 0 -v

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/ea7befdc-95b6-4d67-a689-56e76d5004a1%40googlegroups.com.

Lorenzo Lolli

unread,

Feb 15, 2018, 10:19:12 AM2/15/18

to TRex Traffic Generator

Thank you for prompt answer both of you :)

With the correct flag I'm able to see the request, but I cannot see the answer.

trex(service)>capture monitor start --rx 0 --tx 0 -v

Starting stdout capture monitor - verbose: 'high' [SUCCESS]

*** use 'capture monitor stop' to abort capturing... ***

trex(service)>arp -p 0

Resolving destination on port(s) [0]: [FAILED]

Port 0 - *** Failed to receive ARP response from 192.168.1.1

3.01 [sec]

trex(service)>

#1 Port: 0 âââ¶ TX

Type: ARP, Size: 42 B, TS: 2.91 [sec]

###[ Ethernet ]###
dst = ff:ff:ff:ff:ff:ff
src = 00:1b:21:48:33:30
type = 0x806
###[ ARP ]###
hwtype = 0x1
ptype = 0x800
hwlen = 6
plen = 4
op = who-has
hwsrc = 00:1b:21:48:33:30
psrc = 192.168.1.10
hwdst = 00:00:00:00:00:00
pdst = 192.168.1.1

Anyway, I'm sure that it should work.
On the same machine running t-rex, with the standard kernel module and manual ip configuration I'm able to both arp and ping.

Lorenzo Lolli

unread,

Feb 15, 2018, 10:28:34 AM2/15/18

to TRex Traffic Generator

Is it possible that the arp request is malformed?

In the capture I can see:

hwdst = 00:00:00:00:00:00

shouldn't be

hwdst = FF:FF:FF:FF:FF:FF

?

Lorenzo Lolli

unread,

Feb 15, 2018, 11:02:21 AM2/15/18

to TRex Traffic Generator

I've done one more test:

running a ping, from the DUT to T-REX, while capture is active, I cannot see nothing.

running a resolve, from T-REX after cleaning ARP table, create the following entries:

VALID ENTRIES
IP ADDRESS FLAGS MAC ADDRESS DEVICE VRF
192.168.2.10 dynamic 00-1B-21-48-33-31 eth0 global
192.168.1.10 dynamic 00-1B-21-48-33-30 lan-ptp global

INVALID ENTRIES
IP ADDRESS STATUS
192.168.1.10 unknown

Yaroslav Brustinov

unread,

Feb 15, 2018, 11:09:56 AM2/15/18

to Lorenzo Lolli, TRex Traffic Generator

shouldn't be hwdst = FF:FF:FF:FF:FF:FF ?

https://en.wikipedia.org/wiki/Address_Resolution_Protocol

Target hardware address (THA)
Media address of the intended receiver. In an ARP request this field is ignored. In an ARP reply this field is used to indicate the address of the host that originated the ARP request.

Lorenzo Lolli

unread,

Feb 15, 2018, 11:14:34 AM2/15/18

to TRex Traffic Generator

Agree, I've checked and found the same.

So, from DUT to T-REX, I cannot ping nor arp. If I run t-rex for the first time I can arp, if I force a resolve, I cannot arp and I get an invalid entry on the mac address table on the DUT.

Any idea?

hanoh haim

unread,

Feb 15, 2018, 11:18:09 AM2/15/18

to Lorenzo Lolli, TRex Traffic Generator

Try to move TRex to promiscuous , maybe DUT send to the wrong MAC

Try to capture in DUT

Compare working and non-working scenarios

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/9e3f1a65-5536-4c41-8772-83458c62f604%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Lorenzo Lolli

unread,

Feb 15, 2018, 11:23:23 AM2/15/18

to TRex Traffic Generator

Again, thank you for your support :)

trex(service)>portattr -a --prom on

Applying attributes on port(s) [0, 1]: [SUCCESS]

trex(service)>portattr

promiscuous | on | on

multicast | off | off
flow ctrl | none | none
-- | |
layer mode | IPv4 | IPv4
src IPv4 | 192.168.1.10 | 192.168.2.10
src MAC | 00:1b:21:48:33:30 | 00:1b:21:48:33:31
--- | |
Destination | 192.168.1.1 | 192.168.2.1

ARP Resolution | unresolved | unresolved

---- | |
VLAN | - | -
----- | |
PCI Address | 0000:04:00.0 | 0000:04:00.1
NUMA Node | 0 | 0

RX Filter Mode | fetch all | fetch all
RX Queueing | off | off
Grat ARP | off | off

trex(service)>resolve

Resolving destination on port(s) [0, 1]: [FAILED]

Port 0 - *** Failed to receive ARP response from 192.168.1.1
Port 1 - *** Failed to receive ARP response from 192.168.2.1

6.02 [sec]

trex(service)>

#4 Port: 0 âââ¶ TX

Type: ARP, Size: 42 B, TS: 3862.58 [sec]

###[ Ethernet ]###
dst = ff:ff:ff:ff:ff:ff
src = 00:1b:21:48:33:30
type = 0x806
###[ ARP ]###
hwtype = 0x1
ptype = 0x800
hwlen = 6
plen = 4
op = who-has

hwsrc = 00:1b:21:48:33:30
psrc = 192.168.1.10

hwdst = 00:00:00:00:00:00

pdst = 192.168.1.1

trex(service)>

it seems that even with promiscous mode the problem is the same

hanoh haim

unread,

Feb 15, 2018, 11:25:12 AM2/15/18

to Lorenzo Lolli, TRex Traffic Generator

Try to understand why the DUT does not answer.

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/0c14cf04-ce47-4e98-bbc7-a2a3f245f764%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Lorenzo Lolli

unread,

Feb 15, 2018, 11:41:06 AM2/15/18

to TRex Traffic Generator

I've done one more tests with Cisco ASR920. The results is the same. Tomorrow I will try one more tests with a linux server as DUT, and I will try to figure out by differences using t-rex driver and standard kernel driver, what is wrong.

Lorenzo Lolli

unread,

Feb 16, 2018, 7:56:14 AM2/16/18

to TRex Traffic Generator

Greetings,

I've made another test. I've used a linux machine as DUT, with no firewall rules and tcpdump.
I've connected each port of t-rex to the linux DUT.

When I start t-rex service, t-rex does send 2 arp request for each port. The first is for itself, and the second is for the gateway ip address.
The DUT does answer and T-rex save the information. It is able to display it in portattr.

Then in trex-console/service I run a "resolve", and the command fail (as it was with a router and with Cisco ASR920).
On the DUT side, I can see the ARP request AND answer.
On the T-rex side, there is NO answer, even with capture mode enabled.

But there is more. Since the DUT get the request from t-rex, it is able to write in arp table mac address from t-rex. So it is able to send a ping to t-rex.
But, if I ping from DUT to t-rex, I can see ICMP packets on DUT side, but I CANNOT see anything on t-rex side

So, it seems that arp mechanism does work only when t-rex is started, then the information is lost and it does not work anymore, no matter what I do.

Please note that on my T-rex server, the hardware is an INTEL 82576.
In the tab.5 in t-rex installation documentation it is NOT listed. Is it possible that it is not supported?

Best regards,
Lorenzo Lolli

Anyway, great speech at Cisco Live, and thank you for your work.

Lorenzo Lolli

unread,

Feb 16, 2018, 7:58:04 AM2/16/18

to TRex Traffic Generator

Last but not least, in the dpdk documentation http://dpdk.org/doc/nics , intel 82576 IS supported/listed.

hanoh haim

unread,

Feb 16, 2018, 8:01:19 AM2/16/18

to Lorenzo Lolli, TRex Traffic Generator

Ok , this explains the issue. This NIC is not supported and this is expected.

To workaround this (in expense of performance) try using “—software”. In this case the smart HW filters won’t be activated (does not work with this NIC)

Could you validate?

Hanoh

On Fri, 16 Feb 2018 at 14:58 Lorenzo Lolli <lorenzo...@gmail.com> wrote:

Last but not least, in the dpdk documentation http://dpdk.org/doc/nics , intel 82576 IS supported/listed.

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/7acd736b-b113-4130-990c-1fc20b943681%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Lorenzo Lolli

unread,

Feb 16, 2018, 8:23:31 AM2/16/18

to TRex Traffic Generator

Greetings,

thank you for your tip. I've found the same in ticket https://groups.google.com/forum/#!msg/trex-tgn/bwm7ywIllO8/hp_gA0xMAwAJ;context-place=forum/trex-tgn and I've already tested it. With software mode it does work!

Thank you so much for your great support and patience

Mario Z

unread,

Mar 11, 2018, 1:01:00 PM3/11/18

to TRex Traffic Generator

Hi,

i was having the same issue on ESX,

i found that adding "--arp-refresh-period 5" in the command-line solved the issue for me

Reply all

Reply to author

Forward