Some Snort alerts not working after 14.04 update

[Jeff H]

I noticed today that I am no longer getting alerts for executable downloads in Snort. Specifically I used to get alerts for ET POLICY PE EXE or DLL Windows file download (sid 2000419) when downloading executables without SSL.

Looking at past Snort alerts I think there may be other rules that are no longer firing, but this is the only one I am sure of.

It looks to me like the last alerts I got for those were the day before updating to 14.04.

I have EXE downloads listed in ELSA that should have alerted on this Snort rule.

To test I downloaded an executable, found it listed in ELSA under Sites hosting EXEs. Pivot to capme to download pcap and open it in Wireshark and can export the executable.

I have another system with Snort installed that I use for analysis and I ran the pcap through Snort there with the rule loaded and it alerted as expected.

sostat-redacted attached (I know I have a lot of Snort rules enabled, but this is a home lab setup and the hardware seems to be able to handle the amount of rules. I had the same amount of rules running prior to the 14.04 update and this rule alerted just fine.)

Verifying the rule is not disabled:

$ grep 2000419 /etc/nsm/rules/downloaded.rules
alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download"; flow:established,to_client; content:"MZ"; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2000419; rev:18;)
alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2018959; rev:2;)

I recently disabled eth2 as a monitoring interface, but that was done a couple days ago, and it looks like this issue started when I upgraded to 14.04

Any help tracking this down would be appreciated.

Thanks

[Doug Burks]

Hi Jeff,

Do you have any suppressions or autocats enabled?

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Jeff H]

On Tue, Feb 9, 2016 at 10:38 AM, Doug Burks <doug....@gmail.com> wrote:

Hi Jeff,

Do you have any suppressions or autocats enabled?

--
Doug Burks

Hi Doug, I do have some suppressions in place, but nothing for sid 2000419. I don't have any autocats enabled.

Jeff

[Jeff H]

On Tue, Feb 9, 2016 at 10:51 AM, Jeff H <jeff...@gmail.com> wrote:

Hi Doug, I do have some suppressions in place, but nothing for sid 2000419. I don't have any autocats enabled.

Jeff

Here's a minor update, and just wondering if anyone has any additional thoughts on why this might not be working or further testing to try to narrow it down.

I noticed that Snort VRT has a rule (16425) for EXE downloads which has flowbits:noalert set. I copied it, removed the flowbit, changed the SID and put it in my local.rules.

This rule does alert on exe downloads, though it is looking for the exe download request (Home_Net -> External-Net), where as the ET rule is looking for the exe download (External_Net -> Home_Net)

Looking at the pcap in capme traffic in both directions is being seen, and the pcap does alert on the ET rule when run through Snort on another system.

Are there any Snort logs I can check to see if something isn't working properly? Or a way to run Snort in more verbose mode?

Jeff

[Doug Burks]

What do you have HOME_NET set to?

What do you have EXTERNAL_NET set to?

Can you provide the CapMe transcript?

[Jeff H]

On Wed, Feb 10, 2016 at 5:50 AM, Doug Burks <doug....@gmail.com> wrote:

What do you have HOME_NET set to?

What do you have EXTERNAL_NET set to?

Can you provide the CapMe transcript?

As far as I can remember I have not modified HOME_NET or EXTERNAL_NET. My monitoring interface is eth1. /etc/nsm/[hostname]-eth1/snort.conf lists:

ipvar HOME_NET [192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]

ipvar EXTERNAL_NET any

To further test I copied the ET rule (2000419) to my local.rules, changing the SID and changing EXTERNAL_NET any -> HOME_NET any to any any -> any any. When downloading an exe that should trigger this rule (putty.exe) this modified rule does not alert either. But the Snort VRT rule mentioned previously still does.

I will send the pcap from capme off list.

Thanks

Jeff

[Doug Burks]

I tcpreplay'd your pcap in a 14.04 VM and Snort definitely fired an
alert for ET rule 2000419 as it should. I then increased my Snort
instances from 1 to 2 and Snort still alerted properly.

Have you tried rebooting?

Have you tried lowering your Snort instances from 2 to 1 (IDS_LB_PROCS
in sensor.conf)?

[Jeff H]

On Wed, Feb 10, 2016 at 9:31 AM, Doug Burks <doug....@gmail.com> wrote:

I tcpreplay'd your pcap in a 14.04 VM and Snort definitely fired an
alert for ET rule 2000419 as it should.  I then increased my Snort
instances from 1 to 2 and Snort still alerted properly.

Have you tried rebooting?

Have you tried lowering your Snort instances from 2 to 1 (IDS_LB_PROCS
in sensor.conf)?

Hi Doug,

Thanks for testing in a new 14.04 VM, that was one of my next troubleshooting steps, but I hadn't gotten it setup yet.

I have rebooted.

I lowered my Snort instances from 2 to 1 and it still does not alert on the ET rule (the VRT rule still alerts just like before)

I'm running out of ideas, but decided to look at NIC offloading (http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html) and may be a bit closer.

I believe I went through this and disabled offloading per the blog post above when I first setup my sensor about a year ago (with SO 12.04)

Running ethtool -k eth1 I see a few things related to VLANs are listed as on. I don't know if these have always been on or if something changed with the upgrade (but the rule/alert was working properly before the upgrade). Most of my traffic is on VLANs and eth1 on my Security Onion system is getting a span of the traffic to/from all VLANs. All of the systems I was testing from were on VLANs. I connected a system to the LAN with no vlan tag and downloaded an EXE and it alerted on the rules properly. So I'm wondering if one of the features still on is causing the issues. The output of ethtool -k is:

$ ethtool -k eth1

Features for eth1:

rx-checksumming: off

tx-checksumming: off

tx-checksum-ipv4: off [fixed]

tx-checksum-ip-generic: off

tx-checksum-ipv6: off [fixed]

tx-checksum-fcoe-crc: off [fixed]

tx-checksum-sctp: off [fixed]

scatter-gather: off

tx-scatter-gather: off

tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: off

tx-tcp-segmentation: off

tx-tcp-ecn-segmentation: off [fixed]

tx-tcp6-segmentation: off

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: off

generic-receive-offload: off

large-receive-offload: off

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on

rx-vlan-filter: on [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-ipip-segmentation: off [fixed]

tx-sit-segmentation: off [fixed]

tx-udp_tnl-segmentation: off [fixed]

tx-mpls-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: on

loopback: off [fixed]

rx-fcs: off [fixed]

rx-all: off [fixed]

tx-vlan-stag-hw-insert: off [fixed]

rx-vlan-stag-hw-parse: off [fixed]

rx-vlan-stag-filter: off [fixed]

l2-fwd-offload: off [fixed]

I was able to disable rx-vlan-offload, tx-vlan-offload, receive-hashing and highdma by running:

sudo ethtool -K eth1 rxvlan off 

sudo ethtool -K eth1 txvlan off

sudo ethtool -K eth1 rxhash off

sudo ethtool -K eth1 highdma off

But I am unable to disable rx-vlan-filter and tx-nocache-copy as I can't find the commands. I looked through the help documentation and tried a few guesses, but couldn't find anything concrete and my guesses result in:

ethtool: bad command line argument(s)

For more information run ethtool -h

These specific features weren't mentioned in the Security Onion blog post above, so I am just guessing if they need to be on or off. Are you able to Give guidance on them? 

This Security Onion system is running in ESXi and I haven't changed any NIC offloading settings in ESXi (I'm not sure if that is necessary). But I do know these alerts were working properly before the upgrade and my VLAN/network hasn't changed.

Jeff

[Jeff H]

Doing more troubleshooting, I still haven't narrowed it down any further. Non VLAN tagged traffic seems to alert fine on this rule, but VLAN traffic does not alert on this rule (i am getting some alerts on traffic with VLAN tags, so it is not broken completely)

When I replay the pcap retrieved from capme of the traffic that should trigger this alert on the same system it alerts as expected.

Narrowing in on the VLAN tagged vs non VLAN tagged traffic just a guess, but I can't think of much else to look at.

The network setup is:

ESXi Host with pfSense Router, Security Onion and a few other VMs.

Two physical interfaces are in use on the ESXi host, one for WAN on the pfSense, one for LAN and all VLANs. The VLANs are configured on the pfSense and on a Ubiquiti ToughSwitch which connects the pfSense LAN interface to the rest of the physical network.

On the ESXi host the vSwitch for the LAN network (with all of the VLANs configured on it) is set to allow promiscious mode and the VLAN ID for the LAN port group is 4095 (all VLANs). The Security Onion promiscuous mode interface is on this LAN port group. As far as I can tell with tcpdump on security onion it is seeing all the traffic it should be.

This setup allows the monitor interface in Security Onion to see all traffic to/from my physical internal network and any of the VMs on the ESXi host, including the router, so I am seeing all traffic to/from the internet. 

I am open to making networking changes if need be as I've pieced this setup together while learning about Security Onion and testing networking tools. I'm sure there is room for improvement but prior to the Security Onion 14.04 upgrade I was getting alerts on the ET 2000419 rule.

Any help would be appreciated.

[Jeff H]

One more bit of info...

I installed AlienVault OSSIM on this same ESXi host to test and configured it to use the same virtual NIC for monitoring that Security Onion does. It alerts as expected on ET 2000419 when I download an exe (putty.exe). I believe it is running Suricata, but its been a while since I've used OSSIM so I'm not 100% sure.

I much prefer Security Onion, so I hope we can get this figured out.

[Doug Burks]

Have you tried a new VM with a fresh installation of Security Onion
14.04 to see if it works there?

[Jeff H]

On Thu, Feb 11, 2016 at 3:18 AM, Doug Burks <doug....@gmail.com> wrote:

Have you tried a new VM with a fresh installation of Security Onion
14.04 to see if it works there?

I setup a fresh install of Security Onion 14.04 (installed from the latest ISO with all updates applied) on the same hardware running Snort. It does not alert on ET rule 2000419.

I took a snapshot before running sosetup so I was able to revert it and configure it to run Suricata instead of Snort. It also does not alert on rule 2000419.

Is anyone else running Security Onion 14.04 on ESXi monitoring a physical interface that has VLAN tagged traffic able to confirm if they are able to see alerts for 2000419 on the VLAN tagged networks?

I'm still narrowed in on the VLANs as traffic with VLAN tags doesn't alert but traffic without VLAN tags does alert ok. At this point I'm in over my head with VLANs I'm afraid. Would there be something in the networking configuration or pfring config I could look at on the OSSIM system (which is alerting properly on the VLAN tagged traffic) to compare settings? I imagine it will get tricky fast as it's a different OS and probably different version of pfring (assuming they use pfring at all).

I am going to try to setup a similar ESXi host with Security Onion in a lab at work, but thats going to take some time to get going.

As always I appreciate the help.

Jeff

[Heine Lysemose]

This part is from the suricata configuration file …

 

# This option controls the use of vlan ids in the flow (and defrag)

# hashing. Normally this should be enabled, but in some (broken)

# setups where both sides of a flow are not tagged with the same vlan

# tag, we can ignore the vlan id's in the flow hashing.

vlan:

  use-for-tracking: true

 

Could that maybe help you with the detection?

 

Regards,

Lysemose

--

[Jeff H]

On Thu, Feb 11, 2016 at 10:14 AM, Heine Lysemose <lyse...@gmail.com> wrote:

This part is from the suricata configuration file …

 

# This option controls the use of vlan ids in the flow (and defrag)

# hashing. Normally this should be enabled, but in some (broken)

# setups where both sides of a flow are not tagged with the same vlan

# tag, we can ignore the vlan id's in the flow hashing.

vlan:

  use-for-tracking: true

 

Could that maybe help you with the detection?

 

Regards,

Lysemose

Thanks Lysemose, but that didn't help either.

Really starting to bang my head against the wall on this one. 

Can anyone advise how I can see the VLAN tags on traffic captured by Security Onion?

Jeff 

[Jeff H]

On Thu, Feb 11, 2016 at 3:25 PM, Jeff H <jeff...@gmail.com> wrote:

Thanks Lysemose, but that didn't help either.

Really starting to bang my head against the wall on this one. 

Can anyone advise how I can see the VLAN tags on traffic captured by Security Onion?

Jeff 

I was able to see the vlan tags in tcpdump by running sudo tcpdump -i eth1 -n -e host 46.43.34.31

This is a sample of the output of the above tcpdump command when downloading putty.exe. VLAN tags are there for traffic in both directions. Does anything stand out that might point to the cause of this problem?

00:43:05.377555 10:b9:c0:89:f0:13 > 01:2c:c9:dc:2d:a8, ethertype 802.1Q (0x8100), length 82: vlan 1003, p 0, ethertype IPv4, 192.168.1.10.62064 > 46.43.34.31.80: Flags [S], seq 1470028050, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 589529095 ecr 0,sackOK,eol], length 0

00:43:05.545126 01:2c:c9:dc:2d:a8 > 10:b9:c0:89:f0:13, ethertype 802.1Q (0x8100), length 78: vlan 1003, p 0, ethertype IPv4, 46.43.34.31.80 > 192.168.1.10.62064: Flags [S.], seq 920133069, ack 1470028051, win 28960, options [mss 1460,sackOK,TS val 502713337 ecr 589529095,nop,wscale 7], length 0

00:43:05.547041 10:b9:c0:89:f0:13 > 01:2c:c9:dc:2d:a8, ethertype 802.1Q (0x8100), length 70: vlan 1003, p 0, ethertype IPv4, 192.168.1.10.62064 > 46.43.34.31.80: Flags [.], ack 1, win 4117, options [nop,nop,TS val 589529267 ecr 502713337], length 0

00:43:05.547374 10:b9:c0:89:f0:13 > 01:2c:c9:dc:2d:a8, ethertype 802.1Q (0x8100), length 570: vlan 1003, p 0, ethertype IPv4, 192.168.1.10.62064 > 46.43.34.31.80: Flags [P.], seq 1:501, ack 1, win 4117, options [nop,nop,TS val 589529267 ecr 502713337], length 500

00:43:05.719947 01:2c:c9:dc:2d:a8 > 10:b9:c0:89:f0:13, ethertype 802.1Q (0x8100), length 70: vlan 1003, p 0, ethertype IPv4, 46.43.34.31.80 > 192.168.1.10.62064: Flags [.], ack 501, win 235, options [nop,nop,TS val 502713381 ecr 589529267], length 0

00:43:05.720588 01:2c:c9:dc:2d:a8 > 10:b9:c0:89:f0:13, ethertype 802.1Q (0x8100), length 633: vlan 1003, p 0, ethertype IPv4, 46.43.34.31.80 > 192.168.1.10.62064: Flags [P.], seq 1:564, ack 501, win 235, options [nop,nop,TS val 502713381 ecr 589529267], length 563

00:43:05.722102 10:b9:c0:89:f0:13 > 01:2c:c9:dc:2d:a8, ethertype 802.1Q (0x8100), length 70: vlan 1003, p 0, ethertype IPv4, 192.168.1.10.62064 > 46.43.34.31.80: Flags [.], ack 564, win 4100, options [nop,nop,TS val 589529439 ecr 502713381], length 0

00:43:05.734731 10:b9:c0:89:f0:13 > 01:2c:c9:dc:2d:a8, ethertype 802.1Q (0x8100), length 568: vlan 1003, p 0, ethertype IPv4, 192.168.1.10.62064 > 46.43.34.31.80: Flags [P.], seq 501:999, ack 564, win 4100, options [nop,nop,TS val 589529451 ecr 502713381], length 498

00:43:05.908282 01:2c:c9:dc:2d:a8 > 10:b9:c0:89:f0:13, ethertype 802.1Q (0x8100), length 1518: vlan 1003, p 0, ethertype IPv4, 46.43.34.31.80 > 192.168.1.10.62064: Flags [.], seq 564:2012, ack 999, win 243, options [nop,nop,TS val 502713428 ecr 589529451], length 1448

00:43:05.908353 01:2c:c9:dc:2d:a8 > 10:b9:c0:89:f0:13, ethertype 802.1Q (0x8100), length 1518: vlan 1003, p 0, ethertype IPv4, 46.43.34.31.80 > 192.168.1.10.62064: Flags [.], seq 2012:3460, ack 999, win 243, options [nop,nop,TS val 502713428 ecr 589529451], length 1448

[Doug Burks]

I did the following test:

- started with a VM with the Security Onion 14.04.3.1 ISO image

- downloaded vlan.cap.gz from Wireshark sample captures:
https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=vlan.cap.gz

- gunzip vlan.cap.gz

- used tcpdump to find an IP address in the pcap that only occurred in
vlan tagged traffic

- wrote a Snort rule to alert on traffic from that IP address and
added to /etc/nsm/rules/local.rules

- ran "sudo rule-update"

- used tcpreplay to replay vlan.cap to sniffing interface eth1

- verified that Snort alerted properly on the vlan tagged traffic (see
attached screenshot)


Based on this test, Snort appears to be alerting properly on vlan
tagged traffic.

Are you able to duplicate this test and perhaps compare to your live
vlan tagged traffic looking for any differences?

Just to confirm, when you run Security Onion Setup, are you allowing
it to configure /etc/network/interfaces? Can you send a copy of your
/etc/network/interfaces file?

> --
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

[Jeff H]

On Fri, Feb 12, 2016 at 10:18 AM, Doug Burks <doug....@gmail.com> wrote:

I did the following test:

- started with a VM with the Security Onion 14.04.3.1 ISO image

- downloaded vlan.cap.gz from Wireshark sample captures:
https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=vlan.cap.gz

- gunzip vlan.cap.gz

- used tcpdump to find an IP address in the pcap that only occurred in
vlan tagged traffic

- wrote a Snort rule to alert on traffic from that IP address and
added to /etc/nsm/rules/local.rules

- ran "sudo rule-update"

- used tcpreplay to replay vlan.cap to sniffing interface eth1

- verified that Snort alerted properly on the vlan tagged traffic (see
attached screenshot)


Based on this test, Snort appears to be alerting properly on vlan
tagged traffic.

Are you able to duplicate this test and perhaps compare to your live
vlan tagged traffic looking for any differences?

Just to confirm, when you run Security Onion Setup, are you allowing
it to configure /etc/network/interfaces?  Can you send a copy of your
/etc/network/interfaces file?

Hi Doug,

I'm definitely seeing Snort alerts on some VLAN tagged traffic. Just not all that I should be and I can't narrow down exactly what the cause is. What is so frustrating is that I can't figure out the cause, all the while Suricata in OSSIM works fine when Snort and Suricata in Security Onion does not. And other aspects (Bro, full packet capture, etc.) of Security Onion also seem to work fine.

I've been meaning to make some major networking changes to my lab so I think now is a good time to stop banging my head against the desk, tear things apart, and hope it works better when I set it back up again.

But it does leaving me scratching my head as to what happened, so I can prevent it from happening again.

Thanks for all your help on this. I'll report back with my new setup when it is complete.

Jeff

 

[Doug Burks]

I didn't see a response to this part, so let me try again:

Just to confirm, when you run Security Onion Setup, are you allowing
it to configure /etc/network/interfaces? Can you send a copy of your
/etc/network/interfaces file?

[Heine Lysemose]

Any applied BPF filters?

 

Regards,

Lysemose

 

From: Doug Burks
Sent: 12. februar 2016 23:01
To: securit...@googlegroups.com
Subject: Re: [security-onion] Some Snort alerts not working after 14.04 update

 

I didn't see a response to this part, so let me try again:

[Jeff H]

Yes, allowing Security Onion to set interfaces. The only modifications I have made were to disable offloading per the blog post and change RX buffer size per Wiki (I believe someone on this list suggested I set that a while back, it was not a recent change. The disabling of rxvlan txvlan on eth1 were added recently while troubleshooting this issue. 

And no BPF filters.

$ ethtool -g eth1

Ring parameters for eth1:

Pre-set maximums:

RX: 4096

RX Mini: 0

RX Jumbo: 0

TX: 4096

Current hardware settings:

RX: 4096

RX Mini: 0

RX Jumbo: 0

TX: 256

$ cat /etc/network/interfaces 

# This configuration was created by the Security Onion setup script.

#

# The original network interface configuration file was backed up to:

# /etc/network/interfaces.bak.

#   

# This file describes the network interfaces available on your system

# and how to activate them. For more information, see interfaces(5).

# loopback network interface

auto lo

iface lo inet loopback

# Management network interface

auto eth0

iface eth0 inet static

  address 192.168.1.4

  gateway 192.168.1.1

  netmask 255.255.255.0

  dns-nameservers 192.168.1.1

  dns-domain localdomain

auto eth1

iface eth1 inet manual

  up ip link set $IFACE promisc on arp off up

  down ip link set $IFACE promisc off down

  post-up ethtool -G $IFACE rx 4096; for i in rx tx sg tso ufo gso gro lro rxvlan txvlan; do ethtool -K $IFACE $i off; done

  post-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6

auto eth2

iface eth2 inet manual

  up ip link set $IFACE promisc on arp off up

  down ip link set $IFACE promisc off down

  post-up ethtool -G $IFACE rx 4096; for i in rx tx sg tso ufo gso gro lro; do ethtool -K $IFACE $i off; done

  post-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6

auto eth3

iface eth3 inet manual

  up ip link set $IFACE promisc on arp off up

  down ip link set $IFACE promisc off down

  post-up ethtool -G $IFACE rx 4096; for i in rx tx sg tso ufo gso gro lro; do ethtool -K $IFACE $i off; done

  post-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6

[Jeff H]

It's going to take a week or so for my new switch and other hardware to get here. So I'll be running this system for another week (or longer) most likely.

I'll probably continue to poke around at it. If I find anything new I'll report back.

If anyone thinks of anything they would like me to test I'm happy to do it. Also willing to check the OSSIM configurations to compare if someone knows of something that may be promising, but I am out of ideas. I did check the suricata.yaml on OSSIM and they also have vlan: use-for-tracking: true

I'd be interested to hear from others running Security Onion 14.04 on ESXi with VLANs to hear if they are having any issues.

Thanks again for all the help.

Jeff

[Jeff H]

I just thought of another way to test. On my Security Onion system I captured traffic of a system on one of my VLANs downloading putty.exe, using tcpdump to preserve the VLAN tags (sudo tcpdump -i eth1 -n -e -w putty.cap host 46.43.34.31)

In the tcpdump output I can see the VLAN tags on traffic in both directions. As well as in Wireshark. In Wireshark I can export the exe and compare the checksum to cleanly downloaded copy of putty.exe and they match.

I see the exe download in Bro logs, but nothing in Squil from Snort.

When I replay the pcap on the same Security Onion box I again see the exe download in Bro, but nothing from Snort.

When I run the pcap through a standalone Snort instance on another system I get an alert for ET 2000419

When I replay the pcap in a brand new Security Onion 14.04 system running Suricata running on the same hardware (I still had it setup from earlier testing) I do not get any alerts in Squil.

Doug, I'll send the pcap off list. If anyone else wants to see it I can send it privately as well. I'm going to try to setup a new Security Onion system running Snort on different hardware to further test, but that will take me some time.

Interested to see if this alerts properly for you.

Jeff

[Doug Burks]

Notes from my testing thus far:

Security Onion 14.04
replayed pcap
Snort does not alert (note that Snort log file shows TCP Gaps = 2)
Suricata does not alert
Bro logs EXE

Security Onion 12.04
same results as above

If I switch Suricata from PF_RING to AF_PACKET, it will alert properly.

This would seem to point to PF_RING, but I know we have lots of folks
using PF_RING with VLANs and I don't remember seeing or hearing about
this issue before.

Looking at the pcap in Wireshark, I see the following:
duplicate acks
TCP out-of-order

I wonder if something about your network is causing these issues which
in turn causes PF_RING to drop packets which results in Snort showing
TCP Gaps = 2 and not alerting?

[Jeff H]

On Sat, Feb 13, 2016 at 6:50 AM, Doug Burks <doug....@gmail.com> wrote:

Notes from my testing thus far:

Security Onion 14.04
replayed pcap
Snort does not alert (note that Snort log file shows TCP Gaps = 2)
Suricata does not alert
Bro logs EXE

Security Onion 12.04
same results as above

If I switch Suricata from PF_RING to AF_PACKET, it will alert properly.

This would seem to point to PF_RING, but I know we have lots of folks
using PF_RING with VLANs and I don't remember seeing or hearing about
this issue before.

Looking at the pcap in Wireshark, I see the following:
duplicate acks
TCP out-of-order

I wonder if something about your network is causing these issues which
in turn causes PF_RING to drop packets which results in Snort showing
TCP Gaps = 2 and not alerting?

It looks to me like OSSIM is running Suricata with AF_PACKET, so that explains why it is alerting on this traffic when Security Onion is not.

I'm wracking my brain trying to think of other changes around the same time that I upgraded from Security Onion 12.04 to 14.04 as the last alerts I have for ET 2000419 were the morning before I upgraded to 14.04. I can't think of any changes at the same time.

I did make some networking changes starting a week after the update, but I would have expected alerts for this rule during that week -- not sure if any of these would effect things, but i swapped out my cable modem, updated the VMware tools on my virtual router, added a new VLAN for guest wifi on switch and router, as well as added the physical access point for the new guest wifi vlan. (But this new vlan is not being used here).

But all of this was done a week or more after upgrading Security Onion and when the alerts for this SID stopped.

Since noticing this alert not firing earlier this week I've been testing various settings on the switch and ESXi host that could also be effecting things now.

When I get a chance I will do a tcpdump on the traffic of a system with no vlan tagged traffic so we can see if it has the duplicate acks and TCP out-of-order as well. 

Jeff

[Jeff H]

On Sat, Feb 13, 2016 at 8:18 AM, Jeff H <jeff...@gmail.com> wrote:

When I get a chance I will do a tcpdump on the traffic of a system with no vlan tagged traffic so we can see if it has the duplicate acks and TCP out-of-order as well. 

Jeff

Doug, I think you're on to something regarding the duplicate acks and TCP out-of-order.

I ran the same tcpdump command on Security Onion (sudo tcpdump -i eth1 -n -e -w putty-no-vlan.cap host 46.43.34.31) and downloaded putty.exe from a system without any VLAN tags. It alerted properly in Snort as expected and looking at the capture in Wireshark I don't see the duplicate acks or TCP out-of-order.

I also ran the same tcp dump command on the AlienVault OSSIM monitor interface while downloading from a system with VLAN tags. The traffic doesn't alert in Snort (as expected) but the packet capture does not have the duplicate acks or TCP out-of-order, instead it has some TCP ACKed unseen segments.

While testing yesterday I disabled Flow Control and Spanning Tree Protocol on my switch. I undid those changes and generated a new packet capture on Security Onion. It has one duplicate ACK but still does not alert in Snort.

I'll send you all three new packet captures so you can look at them if you'd like.

So now I'm left wondering:

Why the sudden change and was the timing with the Security Onion upgrade just coincidental?

Can any changes be made to get Snort and Suricata to alert properly if this cannot be fixed on the network side?

Like I said I just ordered a bunch of new equipment to make some major changes, but it won't be here for a week or so. So I can do further testing this week. I imagine I can't be the only person running into problems like this. 

Jeff

[Doug Burks]

I was doing some testing and noticed that PF_RING would report the
following Bucket Len (AKA Snap Length):

Snort - 1514
Suricata - 1516
Bro - 8192

I increased Snort's snaplen to 1518 in snort.conf, restarted Snort,
and it alerted on the traffic successfully.

For Suricata, I don't see a snaplen option in the pfring section of
suricata.yaml, so I increased the MTU of my sniffing interface as
follows:
sudo ifconfig eth1 mtu 1502

I then restarted Suricata, verified that PF_RING showed it with a
Bucket Len of 1518, replayed the traffic, and got an alert.

Can you test and confirm this on your installation?

[Jeff H]

On Sun, Feb 14, 2016 at 11:46 AM, Doug Burks <doug....@gmail.com> wrote:

I was doing some testing and noticed that PF_RING would report the
following Bucket Len (AKA Snap Length):

Snort - 1514
Suricata - 1516
Bro - 8192

I increased Snort's snaplen to 1518 in snort.conf, restarted Snort,
and it alerted on the traffic successfully.

For Suricata, I don't see a snaplen option in the pfring section of
suricata.yaml, so I increased the MTU of my sniffing interface as
follows:
sudo ifconfig eth1 mtu 1502

I then restarted Suricata, verified that PF_RING showed it with a
Bucket Len of 1518, replayed the traffic, and got an alert.

Can you test and confirm this on your installation?

Hi Doug,

I can confirm that changing Snort's snaplen to 1518 causes it to alert properly on ET 2000419 on my production/lab system on both new live traffic and the pcaps I captured previously.

Changing the MTU on the sniffing interface on the Suricata test system still does not get it to alert. Where do I look to see the PF_RING Bucket Len after changing the MTU?

Commands I ran: 

sudo ifconfig eth1 mtu 1502

sudo nsm_sensor_ps-restart

ip link show (to verify 1502 mtu on eth1)

[download putty.exe, check Sguil]

I'll reconfigure this test system later today or tomorrow just to verify that nothing I did in testing previously is effecting it (I can't think of anything, but want to be sure.)

Looking at the snort.conf to suricata.yaml documentation (https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Snortconf_to_Suricatayaml) it says: "Suricata always works at full snap length to provide full traffic visibility." 

I have CC'd Victor Julien here to see if he has any input on the Suricata settings as he contacted me off list for a copy of the pcap that Suricata wasn't alerting properly on. 

Thanks for all your help on this Doug. I'll report back when I've setup and tested a new Suricata test system.

Jeff

 

[Doug Burks]

Bucket Len is reported by PF_RING in /proc/net/pf_ring/, so you can
search for the Bucket Len of Suricata workers as follows:

grep -A20 "Suricata" /proc/net/pf_ring/*eth* |grep "Bucket Len"

My testing showed that Suricata would add 16 to the MTU of the
interface which is why I set my MTU to 1502 and then verified Bucket
Len of 1518. If that's not working for you, you might try setting MTU
to 1518.

[Jeff H]

I setup a new Suricata test system and can confirm that changing the monitor interface mtu to 1502 and restarting Suricata caused the pf_ring Bucket Len to change to 1518. Downloading an exe and replaying the previous pcaps gave me alerts on ET 2000419.

I haven't closely compared the alerts I started getting after making the changes to Snort's snaplen yesterday with those I was getting the past few weeks, but looking at the amount and variety I do think there were other rules effected by this as well.

So I have to admit some of this is a bit over my head. I haven't gotten this far down in the weeds with the networking side of things (because Security Onion does such a good job of packaging it up!).
What should the take away be from all of this? 

Should this somehow be distilled down as something that users should check when setting up new systems? 

Are there any drawbacks to manually setting Snort's snaplen or changing the MTU on the interface for Suricata?

Thanks for all your help Doug.

Jeff

[Doug Burks]

Now that you've confirmed this resolved this issue for you, I'll start
a new thread for further discussion and feedback.

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---

[Doug Burks]

New thread for further discussion:
https://groups.google.com/d/topic/security-onion/94s7beFDMU0/discussion

--
Doug Burks

[Browner 87]

Hi Doug, I'm hoping maybe you have further suggestions on this topic. I'm in almost *exactly* the same situation - SO 14.04 running in a VM on ESXi with a SPAN that has VLAN traffic (only VLAN traffic, no non-VLAN traffic). I'm quite sure it's a pf_ring issue because if I capture traffic from eth1 with tcpdump and scan that PCAP offline, Suricata catches all everything reliably.

Currently - rules that don't require a stream work fine (testmyids.com triggers an alert every time without fail), but rules that require a proper stream do not fire (I get no alerts for downloading putty, I'm also hitting a website that should fire a "cloned site" rule or two, and I'm hitting a page that should trigger a MassScan alert that doesn't).

I'll say that I've already checked the obvious things too - the rules are for sure enabled, the engine is running and detects other things (like some P2P triggers).

I upped my MTU from 1500 to 1502, then to 1518, then to 9000, then to 1400 (just to see), restarting with "nsm_sensor_ps-restart" each time and none of those changes made any difference. I've also validated that "grep -A20 "Suricata" /proc/net/pf_ring/*eth* | grep "Bucket Len"" returns a number 16 higher than my MTU.

Is there anything else I can do to try to solve this? Otherwise I'll be forced to go back to 12.04 :(

In case it matters, I don't know if this is normal, I get a lot of SURICATA STREAM Packet with invalid ack and SURICATA STREAM SHUTDOWN RST invalid ack when I rapidly load/reload pages. I normally leave Suricata STREAM events turned off so I don't know if this is expected.

[Doug Burks]

Please start a new thread and include your sostat-redacted output. 

[Doug Burks]

I've created the following page:
https://github.com/Security-Onion-Solutions/security-onion/wiki/VLAN-Traffic

Please feel free to add or modify as necessary.

--
Doug Burks