Ports destination mac are unresolved (SRIOV)

deep23...@gmail.com

unread,

Mar 11, 2017, 7:33:05 AM3/11/17

to TRex Traffic Generator

Hi,

I am trying to run trex v2.20 in openstack.
Guest OS: Ubuntu 14.04 x86_64 with kernel 4.4.0-59-generic.
Host OS: Ubuntu 14.04 x86_64 with kernel 4.4.0-63-generic.

The host has Intel Corporation Ethernet 10G 2P X520 Adapter for SRIOV.
Guest VM has 3 interfaces connected to 3 networks (44.0.0.0/24, 45.0.0.0/24, 46.0.0.0/24).

Steps performed:
1. Run dpdk_setup_ports to generate config file with interfaces 1 2.
2. Start server stateless ./t-rex64 -i
3. Start trex-console in separate terminal
4. print portattr

I faced similar thing when I was doing it for normal 'virtio' nics. It got solved with pinging gateway 45.0.0.1 and 46.0.0.1 and populating arp cache. Is there any additional step that I am missing?

Also, can you tell from where this mac address for destination is picked?

Thanks
Deepanshu

ido barnea

unread,

Mar 11, 2017, 11:52:49 AM3/11/17

to deep23...@gmail.com, TRex Traffic Generator

Hi Deepanshu,
Can you send:
1. The /etc/trex_cfg.yaml file you are using.
2. Output of ./t-rex64 -i - - iom 0 -v 3
In there, you can see the ARP packets we send and receive.

Just to make sure I understand. After pinging the 45.0.0.1 and 46.0.0.1, when you run “portattr”, you see the MAC addresses?

Finding Destination MAC is done like in any network device, by sending ARP request for the appropriate IP address.

Thanks,
Ido

> --
> You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
> To post to this group, send email to trex...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/61952a67-ae0c-4208-8687-152d954d69f0%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

hanoh haim

unread,

Mar 11, 2017, 12:02:22 PM3/11/17

to deep23...@gmail.com, ido barnea, TRex Traffic Generator

One more thing, in case of L3 mode and link is going down/up you will need to resolve again.

Hanoh

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/545811E7-42A7-4C0B-B72F-96730D63D15E%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

deep23...@gmail.com

unread,

Mar 12, 2017, 4:38:18 AM3/12/17

to TRex Traffic Generator, deep23...@gmail.com, ido....@gmail.com

Hi,

Trex Config file
======================================

### Config file generated by dpdk_setup_ports.py ###

- port_limit: 2
version: 2
interfaces: ['00:06.0', '00:07.0']
port_info:
- ip: 45.0.0.3
default_gw: 45.0.0.1
- ip: 46.0.0.3
default_gw: 46.0.0.1

platform:
master_thread_id: 0
latency_thread_id: 1
dual_if:
- socket: 0
threads: [2,3]

Output of ./t-rex64 -i - - iom 0 -v 3

======================================

Creating huge node
Loading kernel drivers for the first time
ERROR: We don't have precompiled igb_uio.ko module for your kernel version
Will try compiling automatically.
Success.
/usr/bin/python dpdk_nic_bind.py --bind=igb_uio 0000:00:06.0
/usr/bin/python dpdk_nic_bind.py --bind=igb_uio 0000:00:07.0
Starting Scapy server..... Scapy server is started
Starting TRex v2.20 please wait ...
Using configuration file /etc/trex_cfg.yaml
port limit : 2
port_bandwidth_gb : 10
if_mask : None
thread_per_dual_if : 1
if : 00:06.0, 00:07.0,
enable_zmq_pub : 1
zmq_pub_port : 4500
m_zmq_rpc_port : 4501
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
memory per 2x10G ports
MBUF_64 : 16380
MBUF_128 : 8190
MBUF_256 : 8190
MBUF_512 : 8190
MBUF_1024 : 8190
MBUF_2048 : 4095
MBUF_4096 : 128
MBUF_9K : 512
TRAFFIC_MBUF_64 : 65520
TRAFFIC_MBUF_128 : 32760
TRAFFIC_MBUF_256 : 8190
TRAFFIC_MBUF_512 : 8190
TRAFFIC_MBUF_1024 : 8190
TRAFFIC_MBUF_2048 : 65520
TRAFFIC_MBUF_4096 : 128
TRAFFIC_MBUF_9K : 512
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
master thread : 0
rx thread : 1
dual_if : 0
socket : 0
[ 2 3 ]
CTimerWheelYamlInfo does not exist
flags : 8010b00
write_file : 0
verbose : 3
realtime : 1
flip : 0
cores : 1
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
vm mode : 0
cfg file :
mac file :
out file :
client cfg file :
duration : 0
factor : 1
mbuf_factor : 1
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 2
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 1 dst:00:00:00:00:00:00 src:00:00:00:00:00:00
port : 2 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 3 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 4 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 5 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 6 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 7 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 8 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 9 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 10 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 11 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 12 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 13 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 14 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 15 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 81900
MBUF_128 : 40950
MBUF_256 : 16380
MBUF_512 : 16380
MBUF_1024 : 16380
MBUF_2048 : 69615
MBUF_4096 : 256
MBUF_9K : 1024
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
get_each_core_dp_flows : 524288
Total memory : 104.00 Mbytes
core_mask 7
sockets : 0
active sockets : 1
ports_sockets : 1
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
phy | virt
2 1
args
xx
-c
0x7
-n
4
--log-level
4
--master-lcore
0
-w
00:06.0
-w
00:07.0
set driver name net_ixgbe_vf
TRex cfg port id: 0 <-> DPDK port id: 0
TRex cfg port id: 1 <-> DPDK port id: 1
zmq publisher at: tcp://*:4500
Number of ports found: 2

if_index : 0
driver name : net_ixgbe_vf
min_rx_bufsize : 1024
max_rx_pktlen : 9728
max_rx_queues : 4
max_tx_queues : 4
max_mac_addrs : 128
rx_offload_capa : f
tx_offload_capa : 3f
port 0 desc: 82599 Ethernet Controller Virtual Function
port 1 desc: 82599 Ethernet Controller Virtual Function
wait 1 sec .
port : 0
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 1
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
-------------------------------
RX core uses TX queue number 0 on all ports
core, c-port, c-queue, s-port, s-queue, lat-queue
------------------------------------------
1 0 0 1 0 0
-------------------------------
number of ports : 2
max cores for 2 ports : 1
max queue per port : 3
TX grat ARP on port 0 - ip: 45.0.0.3 mac: fa:16:3e:14:7d:9a
TX grat ARP on port 1 - ip: 46.0.0.3 mac: fa:16:3e:e3:a3:f9
TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown
TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown
TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown
TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown
TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown
TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown
TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown
TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown
TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown
TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown
*******Pretest after resolving ********
Pre test info start ===================
Port 0:
Sources:
ip: 45.0.0.3 mac: fa:16:3e:14:7d:9a
Destinations:
ip: 45.0.0.1 mac: Unknown
Port 1:
Sources:
ip: 46.0.0.3 mac: fa:16:3e:e3:a3:f9
Destinations:
ip: 46.0.0.1 mac: Unknown
Pre test info end ===================
Failed resolving dest MAC for default gateway:45.0.0.1 on port 0
Failed resolving dest MAC for default gateway:46.0.0.1 on port 1

Logs suggest that the gateway IPs are not reachable but they were reachable from kernel space before dpdk nic binding.

@Ido

After pinging the 45.0.0.1 and 46.0.0.1, when you run “portattr”, you see the MAC addresses?

I thought that trex uses kernel's ARP cache to auto populate the gateway's mac address but I was wrong. That's why I pinged the gateways from kernel ip stack to populate its arp cache. But trex populates it itself using ARP so that action has no effect whatsoever.

@Hanoch

in case of L3 mode and link is going down/up you will need to resolve again.

Didn't understood this. Are you talking about the arp resolution done by trex?

I tried these commands on trex-console but no success there.
Command: ping -p 0 -d 45.0.0.1

Output:
Pinging 45.0.0.1 from port 0 with 64 bytes of data:

Action has failed with the following error:

Port 0 : *** ping - port has an unresolved destination, cannot determine next hop MAC address

Command: arp
Output:
Resolving destination on port(s) [0, 1]: [FAILED]

Action has failed with the following error:

Port 0 : *** failed to receive ARP response (0 retries)
Port 1 : *** failed to receive ARP response (0 retries)

One thing I observed when I was pinging the gateway from kernel space is continuous packet loss (25-50%). Does this signifies anything?

Also, I saw the doc. It says MTU of host should be increased 9000. In my host, it is 1500. Can it affect arp resolution?

Thanks
Deepanshu

hanoh haim

unread,

Mar 12, 2017, 4:52:56 AM3/12/17

to TRex Traffic Generator, deep23...@gmail.com, ido....@gmail.com

Yes, should set PF mtu,

see manual.

sudo ifconfig eth3 up mtu 9000
sudo ifconfig eth2 up mtu 9000

Hanoh

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/662e6c7e-4fdd-421e-af42-0b7a15fea244%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

ido barnea

unread,

Mar 12, 2017, 5:23:53 AM3/12/17

to hanoh haim, TRex Traffic Generator, deep23...@gmail.com

Hi Deepanshu,

Please report is enlarging MTU works for you.

We will try to see how we can automate this, or give good error message.

Thanks,

Ido

deep23...@gmail.com

unread,

Mar 12, 2017, 6:06:43 AM3/12/17

to TRex Traffic Generator, hhaim...@gmail.com, deep23...@gmail.com

Yes it worked but initially I got this. It is a new VM so IPs got changed a little bit.

TX grat ARP on port 0 - ip: 45.0.0.3 mac: fa:16:3e:e7:bc:b0
TX grat ARP on port 1 - ip: 46.0.0.2 mac: fa:16:3e:79:81:60

TX ARP request on port 0 - ip: 45.0.0.1 mac: Unknown
TX ARP request on port 1 - ip: 46.0.0.1 mac: Unknown