TREX SRIOV - DPDK driver adding or not stripping VLAN on ingress packets

[hakl...@gmail.com]

Hi,
I am using TREX VM in SRIOV (Openstack). VM is Ubuntu 16.04 and trex version is 2.27.

I can send statless traffic and get statistics between ports, but on ingress VLAN tag is never stripped of (TREX still accepts packet as correct). But because of this ARP does not resolve (reply gets back with the segmentaion ID VLAN attached). If I use normal Ubuntu driver (non dpdk) all works fine VLAN tag stripped on ingress and ARP resolves.

Any ideas what could be wrong? below some more info.

BR/Håkan

trex(service)>portattr
Port Status

port | 0 | 1
-------------------------------------------------------------
driver | net_ixgbe_vf | net_ixgbe_vf
description | 82599 Ethernet Con | 82599 Ethernet Con
link status | UP | UP
link speed | 10 Gb/s | 10 Gb/s
port status | IDLE | IDLE
promiscuous | off | off
multicast | off | off
flow ctrl | N/A | N/A
-- | |
layer mode | IPv4 | Ethernet
src IPv4 | 172.16.0.8 | -
src MAC | fa:16:3e:e6:c5:d2 | fa:16:3e:c6:0b:73
--- | |
Destination | 172.16.0.4 | fa:16:3e:e6:c5:d2
ARP Resolution | unresolved | -
---- | |
VLAN | - | -
----- | |
PCI Address | 0000:00:05.0 | 0000:00:06.0
NUMA Node | -1 | -1
RX Filter Mode | fetch all | fetch all
RX Queueing | off | off
Grat ARP | off | off


root@etxhlat-trex-vm-1:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial

root@etxhlat-trex-vm-1:~# uname -a
Linux etxhlat-trex-vm-1 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux


Capture:
#5 Port: 0 ──▶ TX

Type: ARP, Size: 60 B, TS: 236.99 [sec]

Ether(type=2054, dst='ff:ff:ff:ff:ff:ff', src='fa:16:3e:e6:c5:d2')/ARP(hwle\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')oad=b'\x00\x00\x00\


#6 Port: 0 ◀── RX

Type: ARP, Size: 46 B, TS: 236.99 [sec]

Ether(type=33024, dst='fa:16:3e:e6:c5:d2', src='fa:16:3e:27:90:9b')/Dot1Q(t:3e:e6:c5:d2')27:90:9b', plen=4, op=2, hwtype=1, psrc='172.16.0.7', hwdst='fa:16:

[Itay Marom]

Hi,

You can set the port VLAN using the console command 'vlan'

Or the L3 command.

This will cause the port to accept vlan ARP packets with the correct tagging

Itay

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/6ef38585-9b0d-4db4-b3dd-727fd2ec703f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[hanoh haim]

TRex does not strip the VLAN from ingress.

You can configure Dot1Q to TRex port 0 (same value as 172.16.0.7) and TRex will respond.

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/6ef38585-9b0d-4db4-b3dd-727fd2ec703f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

[hakl...@gmail.com]

Thanks, for quick answer.
Hi,
unfortunatelly this does not help as VLAN tag on egress does not seem to go out. The two packets below are untagged ARPs. When ARPing with vlan opackets does not increase. Anyway there should not be tag added on ingress for this case (some dpdk/SRIOV driver issue, using kernel driver does not have same behavior):



-Per port stats table
ports | 0 | 1
-----------------------------------------------------------------------------------------
opackets | 2 | 0


Monitoring on TX and RX port 0 and 1 only TX shown:


trex(service)>l3 --port 1 --src 172.16.0.9 --dst 17.16.0.8 --vlan 440

Setting port(s) [1] with VLAN '440': [SUCCESS]


Setting port 1 in L3 mode: [SUCCESS]


Resolving destination on port(s) [1]: [FAILED]

Port 1 - *** Failed to receive ARP response from 17.16.0.8

3.06 [sec]

trex(service)>

#4 Port: 1 ──▶ TX

Type: ARP, Size: 60 B, TS: 547.26 [sec]

Ether(dst='ff:ff:ff:ff:ff:ff', type=33024, src='fa:16:3e:c6:0b:73')/Dot1Q(i0\x00')ype=1)/Padding(load=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00

trex(service)>l3 --port 0 --src 172.16.0.8 --dst 17.16.0.9 --vlan 440

Setting port(s) [0] with VLAN '440': [SUCCESS]


Setting port 0 in L3 mode: [SUCCESS]


Resolving destination on port(s) [0]: [FAILED]

Port 0 - *** Failed to receive ARP response from 17.16.0.9

3.06 [sec]

trex(service)>

#5 Port: 0 ──▶ TX

Type: ARP, Size: 60 B, TS: 560.80 [sec]

Ether(dst='ff:ff:ff:ff:ff:ff', type=33024, src='fa:16:3e:e6:c5:d2')/Dot1Q(i0\x00')ype=1)/Padding(load=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00

trex(service)>

-Per port stats table
ports | 0 | 1
-----------------------------------------------------------------------------------------
opackets | 2 | 0
obytes | 136 | 0


BR & Thanks/Håkan

[hanoh haim]

Hi 

Did you add the VF with explicit VLAN? Maybe removing the VLAN would help.

Our answer would have worked with direct PF.

Thanks,

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/c786e85c-338e-48d4-86ce-296d5f96a11e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[hanoh haim]

Hi Hakan, 

it seems that VF or switch add VLAN in case of egress and don't remove it from ingress. 

I've found this document, it seems this could be configure in low level not sure how to do that in high level 

https://www.intel.com/content/www/us/en/embedded/products/networking/82599-sr-iov-driver-companion-guide.html

if you would be able to configure it not to ignore VLAN it should work 

thanks,

Hanoh

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/c786e85c-338e-48d4-86ce-296d5f96a11e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

[hakl...@gmail.com]

Hi,
again and thanks for all the answers.

I have got the following answer from DUT (openstack implementation of SRIOV):

"""
the scenario that has been proposed in description is an unsupported one. In case of SR-IOV VLAN segmentation the VLAN stripping is done by the guest driver (in most of the cases ixgbevf). This come from community as design decision. If VF is attached to DPDK driver than the DPDK driver must strip it, otherwise the connection does not work properly.
"""

An alternative would be to use provider type flat in Openstack, but this is unfortunatelly not an option in my case :-(

I guess this stripping on the guest driver then would need to be done in the DPDK driver part of the TREX distibution in order to work for provider network type VLAN segmentation? or is the dpdk driver part of Linux distribution itself, and I would somehow need to make the VLAN strip change to that source code?

Thanks/Håkan

[hanoh haim]

Interesting, probably there is a good reason to do so. We could add this asymmetric VLAN support. Thanks for asking in openStack forum 

Thanks,

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/32debfd0-2124-4f08-90d1-bfb89f0e738a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[eng...@gmail.com]

Hi,
one last thing on this topic. Found that in VPP, there is a configuration for VLAN strip offload options in startup.config file

https://wiki.fd.io/view/VPP/Command-line_Arguments

under the section; "dpdk" parameters, there is something mentioning this. Thought this was maybe worth mentioning as I was thinking there are probably some relations between VPP and TREX DPDK implementations.

Thanks again for brilliant project/Håkan

[hanoh haim]

Hi 

VPP and TRex both originated from Cisco and share DPDK library. 

TRex is used in fd.io CSIT project as a TGN for performance VPP testing.

The VLAN code is not shared. Actually we are going to implement the negative of that not_add_vlan 

Thanks,

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/f88d1a5c-64d5-47b0-bc85-2e37036f83f5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Itay Marom]

Hi Hakan,

Just remembered that I've added an explicit VLAN support for the console ARP command.

So you can simply configure the port for VLAN 440, but when executing ARP override the port VLAN config with your own:

trex> arp -p 0 -v 

-v provides an override mechanism for VLAN configuration when executing ARP.

when you provide -v with nothing it will send the ARP without any VLAN tagging.

This way, the TX pkt will be without VLAN but the response will be excepting the VLAN you have configured with 'l3' command.

Please try it out and update if this solves your problem.

Thanks,

Itay

On Wed, Jul 5, 2017 at 8:06 PM, hanoh haim <hhaim...@gmail.com> wrote:

Hi 

VPP and TRex both originated from Cisco and share DPDK library. 

TRex is used in fd.io CSIT project as a TGN for performance VPP testing.

The VLAN code is not shared. Actually we are going to implement the negative of that not_add_vlan 

Thanks,

Hanoh

On Wed, 5 Jul 2017 at 19:45 <eng...@gmail.com> wrote:

Hi,
one last thing on this topic. Found that in VPP, there is a configuration for VLAN strip offload options in startup.config file

https://wiki.fd.io/view/VPP/Command-line_Arguments

under the section; "dpdk" parameters, there is something mentioning this. Thought this was maybe worth mentioning as I was thinking there are probably some relations between VPP and TREX DPDK implementations.

Thanks again for brilliant project/Håkan

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/f88d1a5c-64d5-47b0-bc85-2e37036f83f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

--

You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/CA%2BYxBo%2BGeLjR4vcKwCGjf0c9ocx23iMkiPrU1WcFwrMnsoaMNA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Itay Marom

[Itay Marom]

However,

If the solution above does not work here is patch that should solve the problem.

It simply modifies ARP service to ignore VLANs and treat any received packet as relevant.

Apply it on your workspace and it should allow you to ARP with no VLAN at all and accept the response regardless if it is tagged or not.

--

Itay Marom

[hakl...@gmail.com]

Thanks alot, will try it out.
BR/Håkan

[hakl...@gmail.com]

Hi,
now I got time to test. Here is the feedback.

Using the python patch worked for me but not an option to set VLAN on port and use the arp -p 0 -v:

With patch:

trex(service)>capture monitor start --rx 0
trex(service)>l3 --port 0 --src 172.16.0.7 --dst 172.16.0.10

Setting port 0 in L3 mode: [SUCCESS]

Resolving destination on port(s) [0]: [SUCCESS]

Port 0 - Recieved ARP reply from: 172.16.0.10, hw: fa:16:3e:9a:6f:56

113.36 [ms]

trex(service)>

#2 Port: 0 ◀── RX

Type: ARP, Size: 64 B, TS: 535.85 [sec]

Ether(type=33024, dst='fa:16:3e:27:90:9b', src='fa:16:3e:9a:6f:56')/Dot1Q(type=2054, vlan=440, prio=0, id=0)/ARP(op=2, hwlen=6, hwsrc='fa:16:3e:9a:6f:56', hwtype=1, plen=4, ptype=2048, psrc='172.16.0.10', pdst='172.16.0.7', hwdst='fa:16:3e:27:90:9b')/Padding(load=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')

Side note:
I used VPP on other SRIOV node (the 172.16.0.10 from above), and managed to use the VLAN strip offset with the driver to strip VLAN on ingress in that one:

root@ubuntu-vpp:~# vppctl show trace
------------------- Start of thread 0 vpp_main -------------------
Packet 1

00:02:50:745214: dpdk-input
TenGigabitEthernet0/5/0 rx queue 0
buffer 0x4e03: current data 0, length 60, free-list 0, totlen-nifb 0, trace 0x0
PKT MBUF: port 0, nb_segs 1, pkt_len 60
buf_len 2176, data_len 60, ol_flags 0x1c1, data_off 128, phys_addr 0x9a238100
packet_type 0x0
Packet Offload Flags
PKT_RX_VLAN_PKT (0x0001) RX packet is a 802.1q VLAN packet
PKT_RX_VLAN_STRIPPED (0x0040) RX packet VLAN tag stripped
PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
ARP: fa:16:3e:27:90:9b -> ff:ff:ff:ff:ff:ff
request, type ethernet/IP4, address size 6/4
fa:16:3e:27:90:9b/172.16.0.7 -> 00:00:00:00:00:00/172.16.0.10
00:02:50:745226: ethernet-input
ARP: fa:16:3e:27:90:9b -> ff:ff:ff:ff:ff:ff
00:02:50:745232: arp-input
request, type ethernet/IP4, address size 6/4
fa:16:3e:27:90:9b/172.16.0.7 -> 00:00:00:00:00:00/172.16.0.10
00:02:50:745245: TenGigabitEthernet0/5/0-output
TenGigabitEthernet0/5/0
ARP: fa:16:3e:9a:6f:56 -> fa:16:3e:27:90:9b
reply, type ethernet/IP4, address size 6/4
fa:16:3e:9a:6f:56/172.16.0.10 -> fa:16:3e:27:90:9b/172.16.0.7
00:02:50:745247: TenGigabitEthernet0/5/0-tx
TenGigabitEthernet0/5/0 tx queue 0
buffer 0x4e03: current data 0, length 60, free-list 0, totlen-nifb 0, trace 0x0
ARP: fa:16:3e:9a:6f:56 -> fa:16:3e:27:90:9b
reply, type ethernet/IP4, address size 6/4
fa:16:3e:9a:6f:56/172.16.0.10 -> fa:16:3e:27:90:9b/172.16.0.7

root@ubuntu-vpp:~# more /etc/vpp/startup.conf

unix {
nodaemon
log /tmp/vpp.log
full-coredump
}

api-trace {
on
}

api-segment {
gid vpp
}

dpdk {
socket-mem 1024,1024
dev 0000:00:05.0 {vlan-strip-offload on}
dev 0000:00:06.0 {vlan-strip-offload on}
}
root@ubuntu-vpp:~#


Thanks alot/Håkan