Trex traffic is not reaching the DUT - IP connectivity lost between DUT and Trex VM after configuring DPDK ports

Manvitha Karre

unread,

Oct 4, 2021, 6:41:22 PM10/4/21

to TRex Traffic Generator

Our scenerio: DUT is palo alto firewall which is connected to a switch connected to ESXI host where we have CentOS VM running Trex application

Before we assigned interfaces to Trex, we were able to ping from DUT to the two IPs assigned on VM (to use for trex) . Once interfaces are added into DPDK, we lost the IP reachability between DUT and VM on those two IPs

Steps performed:

1. Initially, I configured two dpdk interfaces on CentOS VM running Trex with command "sudo ./dpdk_setup_ports.py -i". It generated below config file

### Config file generated by dpdk_setup_ports.py ###

- version: 2

interfaces: ['13:00.0', '1b:00.0']

port_info:

- ip: 10.152.55.5

default_gw: 10.152.55.4

- ip: 10.152.56.5

default_gw: 10.152.56.4

platform:

master_thread_id: 0

latency_thread_id: 1

dual_if:

- socket: 0

threads: [2]

Save the config to file? (Y/n)y

Default filename is /etc/trex_cfg.yaml

Press ENTER to confirm or enter new file:

File /etc/trex_cfg.yaml already exist, overwrite? (y/N)y

Saved to /etc/trex_cfg.yaml.

Note: If I understood right, first port with IP 10.152.55.5 corr to clients and second port IP 10.152.56.5 corr to servers. Correct me if I am wrong?

2. Configured the static ARP entries on the DUT for the Trex interface IPs

Configured static routes on DUT for default client and server pools on VM

3. Ran Trex command "sudo ./t-rex-64 --cfg /etc/trex_cfg.yaml -f ./cap2/http_simple.yaml -c 1 -d 100"

Output was as below:

[xxx@ v2.92]$ sudo ./t-rex-64 -f ./cap2/http_simple.yaml -c 1 -d 100

[sudo] password for xxx:

The ports are bound/configured.

Starting TRex v2.92 please wait ...

set driver name net_vmxnet3

driver capability : TSO

set dpdk queues mode to ONE_QUE

Number of ports found: 2

zmq publisher at: tcp://*:4500

wait 1 sec .

port : 0

------------

link : link : Link Up - speed 10000 Mbps - full-duplex

promiscuous : 0

port : 1

------------

link : link : Link Up - speed 10000 Mbps - full-duplex

promiscuous : 0

number of ports : 2

max cores for 2 ports : 1

tx queues per port : 3

no client generator pool configured, using default pool

no server generator pool configured, using default pool

-------------------------------

RX core uses TX queue number 65535 on all ports

core, c-port, c-queue, s-port, s-queue, lat-queue

------------------------------------------

1 0 0 1 0 0

-------------------------------

no client generator pool configured, using default pool

no server generator pool configured, using default pool

-- loading cap file avl/delay_10_http_browsing_0.pcap

Failed resolving dest MAC for default gateway:10.152.55.4 on port 0

4. Then I configured source and dest mac addresses in the config yaml file as below:

### Config file generated by dpdk_setup_ports.py ###

- version: 2

interfaces: ['13:00.0', '1b:00.0']

port_info:

- ip: 10.152.55.5

default_gw: 10.152.55.4

src_mac: 00:50:56:a4:c1:38

dest_mac: b4:0c:25:e5:40:4a

- ip: 10.152.56.5

default_gw: 10.152.56.4

src_mac: 00:50:56:a4:6f:3e

dest_mac: b4:0c:25:e5:40:4a

platform:

master_thread_id: 0

latency_thread_id: 1

dual_if:

- socket: 0

threads: [2]

Note: On palo alto we had an sub interface with default gateway for VM initially, now we configured two new subinterfaces for the trex, so it has got one mac address for all 3 interfaces. Is this causing the application to fail? correct me.

5. Re-run Trex command: "sudo ./t-rex-64 --cfg /etc/trex_cfg.yaml -f ./cap2/http_simple.yaml -c 1 -d 100". Below is the output, ARP resolution error did not happen, but we see no traffic hitting DUT.

summary stats

--------------

Total-pkt-drop : 10244 pkts

Total-tx-bytes : 9576721 bytes

Total-tx-sw-bytes : 0 bytes

Total-rx-bytes : 320 byte

Total-tx-pkt : 10249 pkts

Total-rx-pkt : 5 pkts

Total-sw-tx-pkt : 0 pkts

Total-sw-err : 0 pkts

Total ARP sent : 2 pkts

Total ARP received : 0 pkts

Please help me identify the cause of the application testing failure.

Thanks & Regards,

Manvitha

Besart Dollma

unread,

Oct 8, 2021, 2:21:54 AM10/8/21

to TRex Traffic Generator

Hi Manvitha,

Setting static ARP in onside is fine, but TRex can resolve ARP. The fact that you received this Failed resolving dest MAC for default gateway:10.152.55.4 on port 0 means that your Palo Alto is not responding to ARP requests.

You don't need to configure complicated stuff, let ARP do its magic.

From what I understand, the setup should be something like this:

TRexPort 0 (10.152.55.5) -> Palo Alto Interface(10.152.55.4) -> Palo Alto Interface(10.152.56.4) -> TRex Port (1) (10.152.56.4).

It is your responsibility to route the packets in between Palo Alto interfaces.

You can verify that the setup is correct using portattr -a.

You can capture the transmitted/received traffic in TRex using the console to verify that indeed traffic is being generated as expected and received.

Thanks,

Manvitha Karre

unread,

Oct 14, 2021, 12:44:04 AM10/14/21

to TRex Traffic Generator

Hi Besi,

Thank you for your response. I rechecked the application and below are my observations:

For testing we moved the IPs from DUT paloalto to a cisco switch which is directly connected to the server running trex vm application and even then, the switch isn't learning any mac addresses from the vm, and also it cant ping the vm’s nics in the same vlan - are we running into any basic fundamental issue?

These ips were pingable from switch before we installed trex application on the centos vm, and as soon as we installed it and configured dpdk ports, immediately we see this mac address and arp not learning issue.

Could you please assist us to troubleshoot it further.

Thanks,

Manvitha

Besart Dollma

unread,

Oct 14, 2021, 3:06:53 AM10/14/21

to TRex Traffic Generator

Hi,

Of course, TRex binds the port so they are not available to the CentOs anymore.

Have you started TRex? Try starting it in interactive mode and performing an ARP request.

Thanks,

Manvitha Karre

unread,

Oct 18, 2021, 2:53:38 PM10/18/21

to TRex Traffic Generator

[xxxx@yyyy v2.92]$ ls

astf avl cfg emu generated run_functional_tests _t-rex-64 _t-rex-64-debug-o trex_client_v2.92.tar.gz x710_ddp

astf_schema.json bird daemon_server exp ko run_regression t-rex-64 t-rex-64-debug-o trex-console

astf-sim bp-sim-64 doc_process.py external_libs master_daemon.py so _t-rex-64-debug _t-rex-64-o trex_daemon_server

astf-sim-utl bp-sim-64-debug dpdk_nic_bind.py find_python.sh ndr stl t-rex-64-debug t-rex-64-o trex_emu

automation cap2 dpdk_setup_ports.py general_daemon_server __pycache__ stl-sim t-rex-64-debug-gdb trex-cfg trex-emu

[xxxx@yyyy v2.92]$ sudo ./t-rex-64 -i c 14

[sudo] password for xxxx:

Error: current path is not readable by user "nobody" (starting at /export).

Two possible solutions:

1. (Recommended)

Copy TRex to some public location (/tmp or /scratch, assuming it has proper permissions (chmod 777 etc.))

2. (Not recommended)

Change permissions of current path. (Starting from directory /export).

chmod 777 /export -R

Could not start scapy daemon server, which is needed by GUI to create packets.

If you don't need it, use --no-scapy-server flag.

ERROR encountered while configuring TRex system

Reading through the documentation, I found " sudo ./t-rex-64 -i c 14 " to be command to run trex in interactive mode. Above is the errror I am getting while I do so. Please assist. Thank you!

Besart Dollma

unread,

Oct 19, 2021, 3:34:48 AM10/19/21

to TRex Traffic Generator

Hi,

Seems to be a problem with your filesystem and file permissions.

Thanks,

Ted Wang

unread,

Nov 29, 2023, 4:17:38 AM11/29/23

to TRex Traffic Generator

I meet the same issue with you, after investigate, I found the root cause is, the acl of my home dir is set to an unusual value, the 'rx' permissions of "other" are cancelled, I don't know why.