Fresh SO install 12.04.03 vs VLAN tagged capture interfaces

[Justin Knox]

Hi all,
I'm in the process of migrating from my tried and true FreeBSD 9.1 sensors to SO. Through the course of this, I'm having bumping into what I consider to be pretty odd behavior with my new SO install and I'm hoping someone might have some insight.

The hardware:
IBM X3650 M3 server
72GB RAM
RAID1 146GB for /
RAID5 ~1TB for /nsm
4 Broadcom BCM5709 interfaces
4 Intel 82580 interfaces (on an i350-T4 card).

According to Ubuntu:
eth0 - eth3 are the Broadcom NICs
eth4 - eth7 are the Intel NICs

At present:
eth0 = mgmt
eth1-eth3 are unconfigured
eth4 = capture interface.
receiving dot1q tagged traffic fed by an aggregation switch. only two vlans, let's call them 10 and 20.
eth5-eth7 are unconfigured

I built the sensor using the SO 12.04.3 iso, and then ran updates via sudo soup. I ran sosetup and had it configure eth4, and rebooted.

I doublechecked what ethtool -k had to say, and it showed rxvlan and txvlan offloading still enabled - I disabled with:

ethtool -K eth4 rxvlan off

before continuing, I wanted to make sure I was seeing what I expected:

tcpdump -nei eth4 -c 10 gives me a few packets from each vlan - as expected.
tcpdump -nei eth4 -c 10 'vlan 10' just sits and spins. setting the bpf to 'vlan 20' does as well.

So, I re-enabled rxvlan:
ethtool -K eth4 rxvlan on

and got the same results...

Going back to my FreeBSD sensor, on identical hardware - plumbed up to the same agg switch (pardon the interface naming, but igb0 = eth4 in this case).

tcpdump -nei igb0 -c 10 operates as expected and gives a few packets from each vlan
tcpdump -nei igb0 -c 10 'vlan 10' returns 10 packets from vlan 10 ONLY.
tcpdump -nei igb0 -c 10 'vlan 20' returns 10 packets from vlan 20 ONLY.

So above I document my experiences without 8021q and the two vlans added via vconfig, but I get the same results even after going through those config steps.

What really boggles is that if I do go through the vlan configs and do something like:
tcpdump -nei eth4.10 -c 10

... I get 10 unencapsulated packets from that vlan as I'd expect.

So the kernel is popping the dot1q tag as expected when I do that config work...

For hahas, I went through sosetup and wanted to see what netsniff-ng put to disk. I'm not seeing any dot1q tags there.

I'm hoping that I'm missing something ridiculously simple here. Any thoughts are greatly appreciated. I can have the aggregation switch config changed to feed untagged traffic, but I'd like to keep that as a last resort.

Any thoughts?

Thanks,
Justin

[Doug Burks]

Hi Justin,

I'm not sure I understand. It sounds like "tcpdump -nei eth4 -c 10"
is showing you all the traffic you expect to see, so this is what I
would expect Snort/Bro/netsniff-ng to see since they wouldn't be
applying any bpf filters like "vlan 10" (unless specifically
configured to do so). Are you expecting to see something different?


--
Doug Burks

[Justin Knox]

Hi Doug,

Thanks for the reply. Sorry if my post made little sense, I'll try to clarify. When doing a wide doing a wide open tcpdump like I'm talking about - I expect to see the tagged traffic. So, for the wide open capture case: check, I see what i expect to see.

now, if I were to look for vlan 10 traffic on that same interface (where the wide open capture shows it to be): 

tcpdump -nei eth4 -c 10 'vlan 10'

^^^ that never actually displays any packets captured. Whereas on my FreeBSD sensors, the same command captures and prints out the 10 packets with the ethernet header as I'd expect - all from vlan 10 (only vlan 10).

So, taking it a step further - if a wide open capture is showing me what I expect to see, then I should see it on disk from netsniff-ng. Except when I load up the pcaps from my sensor's dailylogs, I don't see any dot1q tags at all. Not a single one.

This makes me ask: where are the tags getting peeled off?

In my current aggregation arrangement, most of my sensor interfaces are receiving tagged traffic (only tagged traffic, not mixed with untagged). On the bsd sensors, I've got virtual interfaces defined for each of the vlans I'm watching, and have snort sniffing off of those for ease of configuration. As noted previously, when I go through the motions and build out the same virtual interface arrangement using the 8021q module and some copy/paste in /etc/network/interfaces - this works as expected, and if I tcpdump on the virtual interfaces I see the correct, untagged traffic.

I could mimic the same by tuning individual snort instances on the physical interface in SO to watch a specific vlan via bpf, but that doesn't seem to be working as I expect.

Since I know the wide open capture case works fine, I may tweak the aggregation side and see if we can remove the dot1q tags there to simplify the config. I just rather hope to find an answer to this since it's boggled me so.

Maybe it's a driver thing I'm missing? 

--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/groups/opt_out.

[Doug Burks]

On Fri, Feb 21, 2014 at 9:30 PM, Justin Knox <jk...@indexzero.org> wrote:
> Hi Doug,
> Thanks for the reply. Sorry if my post made little sense, I'll try to
> clarify. When doing a wide doing a wide open tcpdump like I'm talking about
> - I expect to see the tagged traffic. So, for the wide open capture case:
> check, I see what i expect to see.
>
> now, if I were to look for vlan 10 traffic on that same interface (where the
> wide open capture shows it to be):
> tcpdump -nei eth4 -c 10 'vlan 10'
>
> ^^^ that never actually displays any packets captured. Whereas on my FreeBSD
> sensors, the same command captures and prints out the 10 packets with the
> ethernet header as I'd expect - all from vlan 10 (only vlan 10).
>
> So, taking it a step further - if a wide open capture is showing me what I
> expect to see, then I should see it on disk from netsniff-ng. Except when I
> load up the pcaps from my sensor's dailylogs, I don't see any dot1q tags at
> all. Not a single one.

So when you run tcpdump, you see vlan tags in the console output, but
there are no vlan tags in the netsniff-ng pcaps?

And netsniff-ng is recording the actual packets just fine, it's just
not recording the vlan tags?

When you put this system into production and you pivot from an IDS
alert or Bro log to pcap, do you need to be able to see the vlan tags?

--
Doug Burks

[Justin Knox]

So when you run tcpdump, you see vlan tags in the console output, but

there are no vlan tags in the netsniff-ng pcaps?

Correct - wide open tcpdump does see vlan tags.

What's bizarre is that the bpf filtering for the vlan keyword looks broken. badly. There's more to it though. A coworker tagged me out and found this gem:

http://seclists.org/tcpdump/2013/q1/35

^^ based upon that thread, I'm lead to believe that the libpcap folks are baffled too.

 

And netsniff-ng is recording the actual packets just fine, it's just
not recording the vlan tags?

Correct - netsniff-ng is apparently recording fine, just no vlan tags.

But - if I were to apply a vlan-based bpf to it in bpf.conf, I get the same inconsistent results I see with tcpdump. I've confirmed the same at the CLI.

 

When you put this system into production and you pivot from an IDS
alert or Bro log to pcap, do you need to be able to see the vlan tags?

This question turned out to be much more philosophical than I think you meant it to be :)

My initial reaction is: yes. Really though, this comes from some pre-existing environmental configuration baggage from my existing sensor deployment.

The two vlans in question represent two different paths, but in both cases $HOME_NET would be the same.

As such, while I'd normally use two separate ids instances for tuning purposes, I can probably merge them and leverage all the other tools just fine.

In fact, I've been whiteboarding what happens if I strip dot1q from the mix entirely and break up what vlans go where via our aggregation switches - we've got plenty of physical interfaces to play with, after all.

I hope that thread above helps. I can imagine there will be others trying to monitor dot1q trunks at some point, and they could get tripped up as I did.

--Justin

[Doug Burks]

On Mon, Feb 24, 2014 at 10:06 AM, Justin Knox <jk...@indexzero.org> wrote:
>> So when you run tcpdump, you see vlan tags in the console output, but
>> there are no vlan tags in the netsniff-ng pcaps?
>>
>
> Correct - wide open tcpdump does see vlan tags.
> What's bizarre is that the bpf filtering for the vlan keyword looks broken.
> badly. There's more to it though. A coworker tagged me out and found this
> gem:
> http://seclists.org/tcpdump/2013/q1/35
>
> ^^ based upon that thread, I'm lead to believe that the libpcap folks are
> baffled too.

This recent discussion on the netsniff-ng mailing list may also be related:
https://groups.google.com/d/topic/netsniff-ng/L2Ozks6DqAE/discussion

I've updated our BPF page:
https://code.google.com/p/security-onion/wiki/BPF#VLAN

[Justin Knox]

On Monday, February 24, 2014 10:20:02 AM UTC-5, Doug Burks wrote:
>
> This recent discussion on the netsniff-ng mailing list may also be related:
>
> https://groups.google.com/d/topic/netsniff-ng/L2Ozks6DqAE/discussion
>
>
>
> I've updated our BPF page:
>
> https://code.google.com/p/security-onion/wiki/BPF#VLAN

Thanks, Doug. For myself, my workaround is to have things distributed to my sensor interfaces differently. This seems to be the easiest way to deal with it for now. I still see the dot1q tags, but now I don't need to worry about bpf filters to tune instances...