Netmap ixgbe stripping Vlan tags

[Harika Tandra]

Hi all,

I am running Netmap with "intel 10G 82598EB" card in promiscuous mode.
While capturing packets via Netmap the driver is stripping off Vlan tags.
I tested my setup, I am able to see Vlan tags when the same card is in promiscuous
mode without Netmap.

This maybe due to the netmap related changes to the device driver code.
But I don't know much about drivers. I would appreciate any help or pointer
regarding this.

Thanks,
Htandra.




_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net...@freebsd.org"

[Andre Oppermann]

On 23.08.2013 00:36, Harika Tandra wrote:
> Hi all,
>
> I am running Netmap with "intel 10G 82598EB" card in promiscuous mode.
> While capturing packets via Netmap the driver is stripping off Vlan tags.
> I tested my setup, I am able to see Vlan tags when the same card is in promiscuous
> mode without Netmap.
>
> This maybe due to the netmap related changes to the device driver code.
> But I don't know much about drivers. I would appreciate any help or pointer
> regarding this.

This is standard behavior in FreeBSD drivers where the vlan header is removed
and the vlan tag is placed in an mbuf field. Together with netmap it doesn't
make sense of course and the driver should disable it when switching to netmap
mode.

The general usefulness of hardware vlan tag stripping/insertion is debatable as
it doesn't gain much, if anything, and was intended for an entirely different
purpose, namely to help the windows kernel over its total lack of vlan awareness.
There an interface can be put into vlan mode where it simply listens to just one
vlan as if it was native. The driver then delivers normal ethernet frames with
vlan tags removed to the kernel and the other way around on send. It makes a
lot of sense when the kernel isn't vlan aware because inserting and removing the
vlan tag is painful then.

In FreeBSD we don't have that problem as the kernel is fully vlan aware and can
easily parse and insert the vlan tag itself at the same time it copies in the
normal ethernet header. Packets coming down the stack have sufficient reserved
leading space to add a couple of (stacked) vlan tags. Packets going up the stack
can just as easily be parsed for vlan tags as well.

It can be argued that in FreeBSD hardware vlan tag insert/removal is an unnecessary
misfeature and only complicates things. Especially in the context of multi-vlan
stacking. Doing it in software would be in fact just as fast remove a bit of code
from the drivers.

--
Andre

[Juli Mallett]

On Fri, Aug 23, 2013 at 12:02 AM, Andre Oppermann <an...@freebsd.org> wrote:

> On 23.08.2013 00:36, Harika Tandra wrote:
>
>> Hi all,
>>
>> I am running Netmap with "intel 10G 82598EB" card in promiscuous mode.
>> While capturing packets via Netmap the driver is stripping off Vlan tags.
>> I tested my setup, I am able to see Vlan tags when the same card is in
>> promiscuous
>> mode without Netmap.
>>
>> This maybe due to the netmap related changes to the device driver code.
>> But I don't know much about drivers. I would appreciate any help or
>> pointer
>> regarding this.
>>
>
> This is standard behavior in FreeBSD drivers where the vlan header is
> removed
> and the vlan tag is placed in an mbuf field. Together with netmap it
> doesn't
> make sense of course and the driver should disable it when switching to
> netmap
> mode.
>

I think this runs counter to the netmap ethos some (as I understand it.)
In the same way that netmap doesn't enable promiscuous mode since you may
be doing non-promiscuous processing with netmap, it also should leave
whether VLAN_HWFILTER (or whatever) is enabled up to the program or the
user in question. It would be nice if it could do the state management for
these things (to ensure the interface goes back to its original state if
the program crashes), but as yet it can't. There are lots of passive
capture applications where you might not care about having the VLAN tags
intact and so would prefer to have them stripped. If there's a bug here,
it's that packet metadata is lost going into netmap entirely, not that the
VLAN tags aren't in the frame.

> The general usefulness of hardware vlan tag stripping/insertion is
> debatable as
> it doesn't gain much, if anything, and was intended for an entirely
> different
> purpose, namely to help the windows kernel over its total lack of vlan
> awareness.
>

It also helps (slightly) reduce overhead in packet processing where you
just want to move data from card to host as fast as possible.

> It can be argued that in FreeBSD hardware vlan tag insert/removal is an
> unnecessary
> misfeature and only complicates things. Especially in the context of
> multi-vlan
> stacking. Doing it in software would be in fact just as fast remove a bit
> of code
> from the drivers.

Are you sure? While that may hold up for ixgbe (though I'd say it doesn't,
at least without packets going straight into cache so there's no penalty
for accessing parts of the packet you don't care about, again looking at
the netmap case really) it doesn't hold up for low-powered network
processors with very fast offload engines. (NB: I'd rather see it gone; I
think it's an annoying complication when writing network drivers [which
have way too many fiddly bits; I was bound to dislike some aspect of 'tools
not policy' eventually, I guess], or if it stays I'd rather see it be a
mandatory option on hardware where there's no penalty for using it. Some
hardware's performance falls off spectacularly with this kind of offloading
enabled, losing whatever gains stripping unnecessary data provided.)

Thanks,
Juli.

[Andre Oppermann]

On 23.08.2013 09:13, Juli Mallett wrote:

> On Fri, Aug 23, 2013 at 12:02 AM, Andre Oppermann <an...@freebsd.org <mailto:an...@freebsd.org>> wrote:
>
> On 23.08.2013 00:36, Harika Tandra wrote:
>
> Hi all,
>
> I am running Netmap with "intel 10G 82598EB" card in promiscuous mode.
> While capturing packets via Netmap the driver is stripping off Vlan tags.
> I tested my setup, I am able to see Vlan tags when the same card is in promiscuous
> mode without Netmap.
>
> This maybe due to the netmap related changes to the device driver code.
> But I don't know much about drivers. I would appreciate any help or pointer
> regarding this.
>
>
> This is standard behavior in FreeBSD drivers where the vlan header is removed
> and the vlan tag is placed in an mbuf field. Together with netmap it doesn't
> make sense of course and the driver should disable it when switching to netmap
> mode.
>
>
> I think this runs counter to the netmap ethos some (as I understand it.) In the same way that
> netmap doesn't enable promiscuous mode since you may be doing non-promiscuous processing with
> netmap, it also should leave whether VLAN_HWFILTER (or whatever) is enabled up to the program or the
> user in question. It would be nice if it could do the state management for these things (to ensure
> the interface goes back to its original state if the program crashes), but as yet it can't. There
> are lots of passive capture applications where you might not care about having the VLAN tags intact
> and so would prefer to have them stripped. If there's a bug here, it's that packet metadata is lost
> going into netmap entirely, not that the VLAN tags aren't in the frame.

I don't think vlan tag removal and promiscuous mode are comparable. The former
mangles the packet whereas the latter determines which packets are deliverd in
the first place. Running netmap w/o promiscuous mode makes a lot of sense when
you only want to receive packets destined for this host. A netmap consumer
typically doesn't expect packets be mangled at all, mostly likely netmap is
expressly used to get the packet exactly as they were seen on the wire.

> The general usefulness of hardware vlan tag stripping/insertion is debatable as
> it doesn't gain much, if anything, and was intended for an entirely different
> purpose, namely to help the windows kernel over its total lack of vlan awareness.
>
>
> It also helps (slightly) reduce overhead in packet processing where you just want to move data from
> card to host as fast as possible.

There isn't much of a difference really. Whether you do m_adj(m, -14) or m_adj(m, -18)
is the same. Parsing the tag out of the header is about the same as putting it in and
reading it from the mbuf pkthdr. The main overhead of passing it to the per-vlan ifnet
is the same. On the way out the same applies. Prepending 14 vs. 18 bytes for the
ethernet header isn't more overhead than what each driver has to do to for the vlan
insertion descriptor setup.

> It can be argued that in FreeBSD hardware vlan tag insert/removal is an unnecessary
> misfeature and only complicates things. Especially in the context of multi-vlan
> stacking. Doing it in software would be in fact just as fast remove a bit of code
> from the drivers.
>
> Are you sure? While that may hold up for ixgbe (though I'd say it doesn't, at least without packets
> going straight into cache so there's no penalty for accessing parts of the packet you don't care
> about, again looking at the netmap case really) it doesn't hold up for low-powered network
> processors with very fast offload engines.

The stack has to look at the ethernet header anyway to determine what kind of
packet it is. The 4 additional bytes of vlan tag are in the same cache line.
In the low-powered network processors with fast packet engines you try not to
touch the packet on the CPU at all. Even things like NAT are done in the engine.
So you are limited to the capabilities of the hardware to retain full speed.

> (NB: I'd rather see it gone; I think it's an annoying
> complication when writing network drivers [which have way too many fiddly bits; I was bound to
> dislike some aspect of 'tools not policy' eventually, I guess],

Fully agreed here. It only complicates things for no real benefit. The way our
stack works makes it totally unnecessary actually.

> or if it stays I'd rather see it be
> a mandatory option on hardware where there's no penalty for using it. Some hardware's performance
> falls off spectacularly with this kind of offloading enabled, losing whatever gains stripping
> unnecessary data provided.)

Having two path's makes things only more complicated. While some NICs support
QinQ stacking nowadays it makes integration into the stack yet more complex.
I'd rather have it removed and do the magic in ether_input() at once and same
for all. Much less chance for bugs or surprising differences in behavior.
Also allows for arbitrary deep QinQ stacking as has become all the rage in
metro ethernet.

--
Andre

[Harika Tandra]

Hi all,

I agree with Andre's statement

> A netmap consumer
> typically doesn't expect packets be mangled at all, mostly likely netmap is
> expressly used to get the packet exactly as they were seen on the wire.

For my application I want to see the whole packet as is (as seen on the wire).
I am sure it is important for many users who are interested in
using netmap for speedup of packet capture in network security/monitoring applications.

When I disable "vlanhwfilter" flag on the interface. It is behaving as expected and is
not stripping the Vlan tags when placed in promiscuous mode. Netmap seems to be ignoring
his setting or is resetting this option someplace (??). Any suggestion on where in Netmap
code this maybe ?

Thanks for all your help.

Thanks,
Harika.

[Andre Oppermann]

On 23.08.2013 15:12, Harika Tandra wrote:
> Hi all,
>

> I agree with Andre's statement
>> A netmap consumer
>> typically doesn't expect packets be mangled at all, mostly likely netmap is
>> expressly used to get the packet exactly as they were seen on the wire.
>
> For my application I want to see the whole packet as is (as seen on the wire).
> I am sure it is important for many users who are interested in
> using netmap for speedup of packet capture in network security/monitoring applications.
>
> When I disable "vlanhwfilter" flag on the interface. It is behaving as expected and is
> not stripping the Vlan tags when placed in promiscuous mode. Netmap seems to be ignoring
> his setting or is resetting this option someplace (??). Any suggestion on where in Netmap
> code this maybe ?

When you switch an interface to netmap mode it does a soft-reset first.
That reverts the vlanhwfilter configuration to default on. It's not
netmap that does it but the driver. It seems to happen in or around
ixgbe_setup_vlan_hw_support().

[Luigi Rizzo]

responding to the first email in the thread:

netmap by design does not modify the setting of the card
(vlan, hwcsum, tso/lro, promisc, ...).

The main practical reasons behind this choice are that
i) in many cases both settings make sense when in netmap mode,
ii) we can use existing OS tools/ioctl to modify the settings
without having to replicate them within the netmap code.

vlan tag insertion/removal is probably the one feature which
should be always disabled in netmap mode. For others, the case
is not so clear.

As an example, having tso and hwcsum enabled is useful
to postpone those expensive ops to the very last moment;
the upcoming version of netmap supports scatter-gather I/O,
and when used to interconnect virtual machines through
the VALE switch you don't even need to do the segmentation
or checksumming.

cheers
luigi

On Fri, Aug 23, 2013 at 12:36 AM, Harika Tandra <hta...@gloriad.org> wrote:

> Hi all,
>
> I am running Netmap with "intel 10G 82598EB" card in promiscuous mode.
> While capturing packets via Netmap the driver is stripping off Vlan tags.
> I tested my setup, I am able to see Vlan tags when the same card is in
> promiscuous
> mode without Netmap.
>
> This maybe due to the netmap related changes to the device driver code.
> But I don't know much about drivers. I would appreciate any help or pointer
> regarding this.
>

> Thanks,
> Htandra.

>
>
>
>
> _______________________________________________
> freeb...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>

--
-----------------------------------------+-------------------------------
Prof. Luigi RIZZO, ri...@iet.unipi.it . Dip. di Ing. dell'Informazione
http://www.iet.unipi.it/~luigi/ . Universita` di Pisa
TEL +39-050-2211611 . via Diotisalvi 2
Mobile +39-338-6809875 . 56122 PISA (Italy)
-----------------------------------------+-------------------------------

[Harika Tandra]

Hi Luigi,

Thank you so much for the reply. It makes it clear that Netmap
doesn't change vlanhwfilter option or any other option set on the card.
It is very good news for my application !!

Currently I have the "vlanhwfilter" option disabled on the NIC. But not seeing
the vlan tags when opened in netmap mode. This is from my /etc/rc.conf file :

ifconfig_ix0="inet 192.168.10.10 netmask 255.255.255.0 -vlanhwfilter "

Perhaps I am making a mistake when opening the device in netmap mode.
Do I need to set the flags separately via ioctl call when opening interface in netmap
mode ? I would appreciate any pointers on this.

I am following the pkt-gen example and this is the code I am using to open
the interface :

int fd = open("/dev/netmap", O_RDWR);
if( fd == -1) {
ArgusLog(LOG_ERR,"Error : Cannot open netmap device");

} else {
struct nmreq tnmr;
void *mmap_addr;
struct netmap_if *tnifp;
struct NetMapArgs *targs = NULL;
bzero(&tnmr, sizeof(tnmr));
tnmr.nr_version = NETMAP_API;

strncpy(tnmr.nr_name, device->name, sizeof(tnmr.nr_name));
if ((ioctl(fd, NIOCGINFO, &tnmr)) == -1) {
ArgusDebug(4,"Unable to get if info for %s", device->name);
}else{
mmap_addr = (struct netmap_d *) mmap(0, tnmr.nr_memsize,
PROT_WRITE | PROT_READ,
MAP_SHARED, fd, 0);
if (mmap_addr != MAP_FAILED) {
if (ioctl(fd, NIOCREGIF, &tnmr) == -1) {
ArgusDebug(4,"Unable to register interface %s", tnmr.nr_name);
} else {
tnifp = NETMAP_IF(mmap_addr, tnmr.nr_offset);
targs = (struct NetMapArgs *)ArgusCalloc(1, sizeof(*targs));
targs->nmr = tnmr;
targs->fd = fd;
targs->nifp = tnifp;
inf->nmargs = targs;
}
}

Thanks,
Harika.