Suricata not alerting on test alert

[Joe Lane]

I'm trying to figure out why suricata is not generating an alert from visiting testmyids.com. I have had this problem before and thought I had fixed it by disabling vlan tracking in the suricata config. Since the last suricata update the config reverted that option to true so I turned it back off expecting to fix my problem again however it has not. I am receiving other alerts as expected, but i can't get the testmyids alert to trigger. I have also verified I am actually capturing the traffic. To do that I used elsa to find the bro conn log connection, then used capME to inspect the capture and sure enough it's there plain as day. Also, i almost forgot, i have verified the rule is actually activated by: "grep uid=0 /etc/nsm/rules/downloaded.rules'

In conclusion what could cause suricata to not alert when:
a. The alert is indeed activated.
b. The traffic to generate the rule is present on the monitored interface.
c. Suricata is generating alerts for many other rules, but not this one.

[Doug Burks]

Hi Joe,

What's the output of the following?
grep 2100498 /etc/nsm/pulledpork/* /etc/nsm/rules/*

Can you also send the capME transcript?

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Joe Lane]

user@hostname:~$ grep 2100498 /etc/nsm/pulledpork/* /etc/nsm/rules/*

/etc/nsm/rules/downloaded.rules:alert ip any any -> any any (msg:"GPL ATTACK_RESPONSE id check returned root"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7;)

/etc/nsm/rules/sid-msg.map:2100498 || GPL ATTACK_RESPONSE id check returned root

--
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/XUUNgIGqsv4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

[Doug Burks]

Which suricata config file did you update?

After updating your suricata config, did you restart Suricata?

Have you tried restarting Suricata again?

Have you tried rebooting the box?

Is Suricata dropping packets?

[Joe Lane]

Which suricata config file did you update?

/etc/nsm/hostname-eth2/suricata.yaml

After updating your suricata config, did you restart Suricata?

Yes

Have you tried restarting Suricata again?

Yes

Have you tried rebooting the box?

No, I will try that now.

Is Suricata dropping packets?

=========================================================================

IDS Engine (suricata) packet drops

=========================================================================

/nsm/sensor_data/hostname-eth2/stats.log

tcp.ssn_memcap_drop       | RxPFReth28                | 0

tcp.segment_memcap_drop   | RxPFReth28                | 0

[Doug Burks]

Please try the following from your Security Onion box or some other Linux box:

for i in {1..10}; do curl testmyids.com; done

Do you get any alerts at all?

Are you able to find all 10 connections in the Bro conn.log?

[Joe Lane]

Just tested it from a completely different internal host, I see all 10 conns and 0 alerts.

[Doug Burks]

Can you provide the relevant entries from Bro's http.log?

[Joe Lane]

This is a sanitized elsa export.

[Doug Burks]

Please send your Suricata log (/var/log/nsm/hostname-eth2/suricata.log).

[Joe Lane]

It was actually very short, so I will paste it.

Executing: suricata --user sguil --group sguil -c /etc/nsm/hostname-eth2/suricata.yaml --pfring=eth2 -F /etc/nsm/hostname/bpf-ids.conf -l /nsm/sensor_data/hostname-eth2 

12/1/2015 -- 16:19:09 - <Notice> - This is Suricata version 2.0.5 RELEASE

12/1/2015 -- 16:19:09 - <Warning> - [ERRCODE: SC_ERR_NO_RULES(42)] - No rules loaded from /etc/nsm/rules/local.rules

12/1/2015 -- 16:19:17 - <Warning> - [ERRCODE: SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2200038, gid 1: unknown rule

12/1/2015 -- 16:19:17 - <Notice> - all 8 packet processing threads, 3 management threads initialized, engine started.

13/1/2015 -- 07:06:08 - <Notice> - Signal Received.  Stopping engine.

13/1/2015 -- 07:06:09 - <Notice> - Stats for 'eth2':  pkts: 3971950829, drop: 0 (0.00%), invalid chksum: 0

Today's log is the same minus the signal and stop.

[Doug Burks]

Please send your /etc/nsm/hostname-eth2/threshold.conf.

Please also run the following command:

sudo sostat-redacted

There will be a lot of output, so you may need to increase your
terminal's scroll buffer OR redirect the output of the command to a
file:

sudo sostat-redacted > sostat-redacted.txt 2>&1

sostat-redacted will automatically redact any IPv4/IPv6/MAC addresses,
but there may be additional sensitive info that you still need to
redact manually.

Attach the output to your email in plain text format (.txt) OR use a
service like http://pastebin.com.

[Joe Lane]

the only active line in the threshold.conf is:

suppress gen_id 1, sig_id 2200038

Here is the sostat: http://pastebin.com/P3bv0P6H

[Doug Burks]

Since Suricata is complaining about 2200038, have you tried removing
it from threshold.conf and restarting Suricata?

12/1/2015 -- 16:19:17 - <Warning> - [ERRCODE:
SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2200038, gid 1: unknown
rule

Have you tried decreasing your number of Suricata PF_RING instances to
see if that makes any difference?
https://code.google.com/p/security-onion/wiki/PF_RING

[Joe Lane]

Okay, I took out the threshold entry and restarted suri. I'm not sure if it's related or not but I ran curl to testmyids 10 times on 3 different hosts. Each 10 connections are available in both the http and conn log of bro, however i only got 1 alert (count 1) of GPL ATTACK_RESPONSE id check returned root from the first host I ran it on. 

[Doug Burks]

If you repeat that test (restart Suricata, curl testmyids.com on 3
different hosts), are the results consistent (alert only from first
host)?

Is there anything different about that first host (different VLAN perhaps?)?

[Joe Lane]

Just redid the test, now I didn't get any alerts, still getting other alerts or other stuff though. Bro logs show all connections. It's like bro gets all the traffic but suri only gets some, but i'm not sure how that would be possible. 

All hosts tested were on the same VLAN. I tested with other hosts from different VLANs with no change in results. 

[Doug Burks]

Are you running any BPFs?
cat /etc/nsm/*/bpf.conf

Have you tried decreasing your number of Suricata PF_RING instances to
see if that makes any difference?
https://code.google.com/p/security-onion/wiki/PF_RING

[Joe Lane]

No bpf.

[Doug Burks]

I've asked this question a couple of times now, but haven't seen a
response...have you tried decreasing your number of Suricata PF_RING

instances to see if that makes any difference?
https://code.google.com/p/security-onion/wiki/PF_RING

[Joe Lane]

Indeed you did I'm, sorry , I missed it. 

I have decreased the rings for suri from 8 to 4. After the restart, I received 1 alert, which happened to be the 3rd connection from the 10 that I ran on the first host. 

I restarted, ran it again, got no alerts.

Restarted a 3rd time, ran it again, alerted on the 4th connection.

[Doug Burks]

Just out of curiosity, could you try lowering IDS_LB_PROCS all the way
down to 1 to see if that makes any difference?

[Joe Lane]

Brought it down to 1. Ran curl about 100 times, got it to alert 12 of those. There didn't seem to be any pattern. For a second I thought it was alternating between the hosts I was trying, then it alerted twice on the same host. I tried waiting 1-2 minutes between running curl, and spamming it. Nothing seemed to make a difference. When I brought it down to one though, I did start losing packets. 

/proc/net/pf_ring/29653-eth2.82

Appl. Name         : Suricata

Tot Packets        : 138256586

Tot Pkt Lost       : 595102

TX: Send Errors    : 0

Reflect: Fwd Errors: 0

Min Num Slots      : 65538

Num Free Slots     : 65370

[Joe Lane]

I have been working with Victor from the Suri project and we have concluded that Suri is not receiving all of the traffic. The PF_RING stats for the Suri processes only account for a small portion (as in 800M vs 4B) of the total traffic bro and netsniff get. Disabling both bro and netsniff does not increase the traffic Suri gets either. Anyone have any other ideas?

BTW, no BPF going on in either the cmd line specified file or the Suri yaml itself.

[Victor Julien]

On 01/14/2015 06:30 PM, Joe Lane wrote:
> I have been working with Victor from the Suri project and we have concluded that Suri is not receiving all of the traffic. The PF_RING stats for the Suri processes only account for a small portion (as in 800M vs 4B) of the total traffic bro and netsniff get. Disabling both bro and netsniff does not increase the traffic Suri gets either. Anyone have any other ideas?
>
> BTW, no BPF going on in either the cmd line specified file or the Suri yaml itself.
>

The ~4B vs ~800M numbers I got from the sostat pfring numbers posted in
this thread earlier: http://pastebin.com/P3bv0P6H

The sum of the bro numbers is just over 4B, the num of the suri numbers
just under 800M

--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------

[Doug Burks]

Keep in mind that those PF_RING numbers are only for the lifetime of
the process itself. Since Joe has restarted Suricata multiple times,
it's possible for the numbers to be lower simply because the Suricata
processes haven't been running as long as the Bro processes.

Let's try a new test:

- reboot and let Bro/Suricata come up naturally

- Bro will start slightly before Suricata, so its number will be
*slightly* higher

- keep an eye on the stats for an hour or two without manually
restarting Bro/Suricata

Are the numbers tracking fairly closely, or are the Suricata PF_RING
numbers falling behind more and more over time?

[Joe Lane]

Well, it looks like it might not be that after all. Here are the numbers. 

minutes, bro, suricata

10, 56990753, 52613763

20, 118788361, 114411540

30, 181235149, 176858219

40, 240861334, 236485101

50, 302052231, 297675461

60, 368850998, 364474682

I have attached a plot of this: blue is Bro, green is Suricata

​

--
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/XUUNgIGqsv4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

[Doug Burks]

OK, so it sounds like Suricata is definitely getting the traffic from PF_RING properly.

As another data point, I fired up a VM and ran the following tests:

- 1 Suricata thread, 100 "curl testmyids.com", 100 alerts

- 4 Suricata threads, 100 "curl testmyids.com", 100 alerts

- 8 Suricata threads, 100 "curl testmyids.com", 100 alerts

So I'm not able to duplicate your issue.

Are you able to duplicate your issue on a fresh installation?

[Joe Lane]

That was my next step once I was confidant I could do no further testing. I will do a fresh install next.

[Joe Lane]

I have an update on this. About 5 months ago we acquired brand new hardware for our sensor. We have also changed the network topology to make it easier to monitor. All interesting traffic is now coming into one 10G interface that the sensor monitors. At the moment our average peak is about 600Mbps with minimal loss running Bro, Full Cap, Snort. After a while i decided to give Suricata another shot since i had brand new hardware, a fresh install, and a better network topology. Unfortunently, yet again, running an identical ruleset as snort (of course with the pulled pork -T option) i have relatively no alerts.

With our previous setup we suspected that 2 things could be going on. 1, our network topology generated duplicate frames that could have caused monitoring to be confused. 2, a strange hardware incompatibility. Now that both of those things are different, i'm really scratching my head here.

Looking at a PCAP of a connection to www.testmyids.com the connection is absolutely perfect, no dups, no retransmissions, nothing. I'm really at a loss as to whats going on here.

[Victor Julien]

Do you have vlans in there? We've seen some cases where parts of the
traffic were tagged, other parts weren't.

If so try disabling vlan tracking. In your YAML:

vlan:
use-for-tracking: false

[Joe Lane]

That was the first thing I tried. Both ways i'm not getting alerts. Our SPAN port monitors a single VLAN. Looking at the PCAPS from CapME, there is no tagging information.

[Victor Julien]

On 18-09-15 17:21, Joe Lane wrote:
> That was the first thing I tried. Both ways i'm not getting alerts. Our
> SPAN port monitors a single VLAN. Looking at the PCAPS from CapME, there
> is no tagging information.

Can you (privately) share the testmyids.com pcap?

Cheers,
Victor

>
> On Fri, Sep 18, 2015 at 10:08 AM, Victor Julien <li...@inliniac.net
> <mailto:li...@inliniac.net>> wrote:
>
> On 18-09-15 15:47, Joe Lane wrote:
>
> I have an update on this. About 5 months ago we acquired brand
> new hardware for our sensor. We have also changed the network
> topology to make it easier to monitor. All interesting traffic
> is now coming into one 10G interface that the sensor monitors.
> At the moment our average peak is about 600Mbps with minimal
> loss running Bro, Full Cap, Snort. After a while i decided to
> give Suricata another shot since i had brand new hardware, a
> fresh install, and a better network topology. Unfortunently, yet
> again, running an identical ruleset as snort (of course with the
> pulled pork -T option) i have relatively no alerts.
>
> With our previous setup we suspected that 2 things could be
> going on. 1, our network topology generated duplicate frames
> that could have caused monitoring to be confused. 2, a strange
> hardware incompatibility. Now that both of those things are
> different, i'm really scratching my head here.
>
> Looking at a PCAP of a connection to www.testmyids.com

> <http://www.testmyids.com> the connection is absolutely perfect,

> no dups, no retransmissions, nothing. I'm really at a loss as to
> whats going on here.
>
>
> Do you have vlans in there? We've seen some cases where parts of the
> traffic were tagged, other parts weren't.
>
> If so try disabling vlan tracking. In your YAML:
>
> vlan:
> use-for-tracking: false
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "security-onion" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/security-onion/XUUNgIGqsv4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> security-onio...@googlegroups.com

> <mailto:security-onion%2Bunsu...@googlegroups.com>.

> To post to this group, send email to securit...@googlegroups.com

> <mailto:securit...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "security-onion" group.

> To unsubscribe from this group and stop receiving emails from it, send
> an email to security-onio...@googlegroups.com

> <mailto:security-onio...@googlegroups.com>.

> To post to this group, send email to securit...@googlegroups.com

> <mailto:securit...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

[Justin Hussey]

Was there ever any follow-up to this?

I am having the same problem, but haven't gone through all the aforementioned troubleshooting steps.  I am running traffic from a SPAN port monitoring a vlan, and set up security onion in production mode (chose Suricata+ET-rules off the bat)...   I have not been seeing the alerts in sguil/squert or the suricata logs.  

>     securit...@googlegroups.com

>     <mailto:security-onion%2Bunsu...@googlegroups.com>.
>     To post to this group, send email to securit...@googlegroups.com
>     <mailto:securit...@googlegroups.com>.
>     Visit this group at http://groups.google.com/group/security-onion.
>     For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to securit...@googlegroups.com
> <mailto:security-onion+unsub...@googlegroups.com>.

[Wes Lambert]

Hi Justin,

Please start a new thread for your issue.

Thanks,

Wes

> <mailto:security-onio...@googlegroups.com>.

> To post to this group, send email to securit...@googlegroups.com
> <mailto:securit...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.


--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/de8d55b0-0928-4bf4-8e76-eadbfbd40989%40googlegroups.com.

--

https://twitter.com/therealwlambert

https://securityonion.net/