Squert and Sguil not showing current alerts

[gary boyce]

My alerts take a day to show up. I have cleared /nsm/sensor_data/*/*unified* and rebooted but it doesn't seem to make any difference. It was suggested that Barnyard2 is backed up. I have an octo-core 1.6GHz with 24 Gigs of ram and a 750 GB hard drive. I have 4 cores for suricata and 2 for bro. I can make any changes necessary to make this work. I am open to all suggestions.

My sostat - http://pastebin.com/embed_js.php?i=mnqc8b3L
My /nsm/sensor-data - http://pastebin.com/embed_js.php?i=HKv7GUjq

[Doug Burks]

From your sostat output:

=========================================================================
Sguil events summary for yesterday
=========================================================================
+---------+-------------+---------------------------------------------------------------------+
| Totals | GenID:SigID | Signature
|
+---------+-------------+---------------------------------------------------------------------+
| 3990371 | 1:2210045 | SURICATA STREAM Packet with invalid ack
|
| 14905 | 1:9001511 | ICMP TO GOOGLES DNS
|
| 5201 | 1:2210008 | SURICATA STREAM 3way handshake SYN resend
different seq on SYN recv |
| 5192 | 1:2210004 | SURICATA STREAM 3way handshake SYNACK resend
with different ack |
| 4882 | 1:2210017 | SURICATA STREAM CLOSEWAIT invalid ACK
|
| 2586 | 1:2210023 | SURICATA STREAM ESTABLISHED SYNACK resend
with different ACK |
| 2285 | 1:2210030 | SURICATA STREAM FIN invalid ack
|
| 2164 | 1:2210038 | SURICATA STREAM FIN out of window
|
| 2093 | 1:2210016 | SURICATA STREAM CLOSEWAIT FIN out of window
|
| 1849 | 1:2210042 | SURICATA STREAM TIMEWAIT ACK with wrong seq
|
| 1543 | 1:2230003 | SURICATA TLS invalid handshake message
|
| 1352 | 1:2210039 | SURICATA STREAM Last ACK with wrong seq
|
| 1322 | 1:2210002 | SURICATA STREAM 3way handshake right seq
wrong ack evasion |
| 1063 | 1:2210036 | SURICATA STREAM FIN2 invalid ack
|
| 845 | 1:2210024 | SURICATA STREAM ESTABLISHED SYNACK resend
with different seq |
| 548 | 1:2210012 | SURICATA STREAM 4way handshake SYNACK with
wrong SYN |
| 498 | 1:2210032 | SURICATA STREAM FIN1 FIN with wrong seq
|
| 258 | 1:2210026 | SURICATA STREAM ESTABLISHED SYN resend
|
| 251 | 1:2200034 | SURICATA TCP header length too small
|
| 191 | 1:2200029 | SURICATA ICMPv6 unknown type
|

Yesterday, you had 3.9 million alerts for "SURICATA STREAM Packet with
invalid ack". That is most likely what's causing barnyard2 to get
backed up. If you don't care about this alert, then you should
disable it altogether so that barnyard2 won't have to process that
many events. Note that if you just autocat it, barnyard2 will still
have to process it, resulting in the backlog. Please see the section
entitled "Disable the sid":
https://github.com/Security-Onion-Solutions/security-onion/wiki/ManagingAlerts

If you don't care about the other SURICATA STREAM events, then you
should disable (not just autocat) those as well.

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Matt .]

What stands out to me is the number of rules enabled, 18,000. It needs to be more like 5,000, maybe as high as 7,000 (I run about 6,200).

[gary boyce]

Okay, Iv'e gotten my rules down to 7539 and I am still 24 hours behind. I only have 2 unified2 files.
ls -alh /nsm/sensor_data/*/*unified*
-rw-r--r-- 1 sguil sguil 9.0K Dec 9 07:38 /nsm/sensor_data/Telpage-IDS-eth1/snort.unified2.1449621413
-rw-r--r-- 1 sguil sguil 31K Dec 9 13:05 /nsm/sensor_data/Telpage-IDS-eth1/snort.unified2.1449647137

Just so that I am clear on the matter.. The unified2 files are the backlog right?

[Doug Burks]

Have you tried removing all unified2 files and rebooting?

[gary boyce]

After removing all unified2 files and rebooting I am able see alerts coming in but I am still seing Suricata-Stream events. I thought I took care of them and I have reviewed https://github.com/Security-Onion-Solutions/security-onion/wiki/ManagingAlerts

Here is my disabled.conf Will you let me know how to properly disable the suricata stream events?

# example disablesid.conf V3.1

# Example of modifying state for individual rules
# 1:1034,1:9837,1:1270,1:3390,1:710,1:1249,3:13010
1:2101411,1:2210020,1:22100:45,1:2210029,1:2210000,1:2210010,1:2210044,1:2210021,1:2210046,1:2210015,1:2210003

# Example of modifying state for rule ranges
# 1:220-1:3264,3:13010-3:13013

# Example of modifying state for MS and cve rules, note the use of the :
# in cve. This will modify MS09-008, cve 2009-0233, bugtraq 21301,
# and all MS00 and all cve 2000 related sids! These support regular expression
# matching only after you have specified what you are looking for, i.e.
# MS00-<regex> or cve:<regex>, the first section CANNOT contain a regular
# expression (MS\d{2}-\d+) will NOT work, use the pcre: keyword (below)
# for this.
# MS09-008,cve:2009-0233,bugtraq:21301,MS00-\d+,cve:2000-\d+

# Example of using the pcre: keyword to modify rulestate. the pcre keyword
# allows for full use of regular expression syntax, you do not need to designate
# with / and all pcre searches are treated as case insensitive. For more information
# about regular expression syntax: http://www.regular-expressions.info/
# The following example modifies state for all MS07 through MS10
# pcre:MS(0[7-9]|10)-\d+
pcre:stream,pcre:ICMP,pcre:web_specific_apps,pcre:activex,pcre:netbios,pcre:info,pcre:chat,pcre:snmp,pcre:tftp,pcre:telnet,pcre:voip,pcre:current_events,pcr$

# Example of modifying state for specific categories entirely (see README.CATEGORIES)
# VRT-web-iis,ET-shellcode,ET-emergingthreats-smtp,Custom-shellcode,Custom-emergingthreats-smtp
Suricata-Stream,WEB_SPECIFIC_APPS,ACTIVEX,NETBIOS,INFO,SQL,WEB_CLIENT,CNC,RPC,FTP,WEB_SERVER,ICMP_INFO,NETBIOS,ICMP,IMAP,CHAT,POLICY,WEB_CLIENT,SNMP,TFTP,TE$

# Any of the above values can be on a single line or multiple lines, when
# on a single line they simply need to be separated by a ,
# 1:9837,1:220-1:3264,3:13010-3:13013,pcre:MS(0[0-7])-\d+,MS09-008,cve:2009-0233

# The modifications in this file are for sample/example purposes only and
# should not actively be used, you need to modify this file to fit your
# environment.

[Brian Kellogg]

I had issues getting them all disabled so I ended up using the below in my disabled.conf.

pcre:SURICATA

I've not noticed this disabling anything that I want enabled.

[gary boyce]

I gave that a go cleared my unified files updated the rules and rebooted and I am still seeing suricata stream and other suricata events.

[Doug Burks]

What is the output of the following?
grep SURICATA /etc/nsm/rules/downloaded.rules

[gary boyce]

http://pastebin.com/embed_js.php?i=See7UvB1

[Doug Burks]

OK, looks like all SURICATA alerts are commented out.

Please try the following:

sudo nsm_sensor_ps-stop

log into Sguil/Squert and F8 any existing SURICATA alerts

remove any unified2 files

sudo reboot

[gary boyce]

Ill give that a try. Thanks. Also, I saw in another post that I can disable all rules by putting pcre:sid in the disabled.conf. My question is... Does that disable the custom rules as well? I am still trying to get some of my custom rules to alert with out success. It would be nice for the time being if the only alerts I was seeing was my custom ones.

[Doug Burks]

disablesid.conf should only apply to the rules that are downloaded and
end up in downloaded.rules.

[gary boyce]

Excellent. I just go a hit on one of my rules. There is definately no backlog anymore. Thanks for all your help. I will keep plugging away. Great job on this incredible tools and your support is awesome!
Keep up the good work.