Sguil uncategorized events
rex warnert
I was wondering if anyone else is seeing a disconnect in the categorization of events in sguil.
If I log into Sguil or Squert everything looks normal maybe 12,000 or so events uncategorized on a daily basis. But when you run SOSTAT or sguil-db-purge it shows 129,333 uncategorized events.
if I manually and/or autocat events in Sguil or Squert why are they not reflected in sostat or when I do sguil-db-purge.....?
If you need output of files let me know
***though I am in the middle of a db purge now and it is taking FOREVER to categorize the oldest 29,333 events.***
Sguil Uncategorized Events
=========================================================================
COUNT(*)
100000 (I just finished a dbpurge)
=========================================================================
Sguil events summary for yesterday
=========================================================================
Totals GenID:SigID Signature
3815 120:7 http_inspect: CHUNKED ENCODING - EXCESSIVE CONSECUTIVE SMALL CHUNKS
2280 137:1 ssp_ssl: Invalid Client HELLO after Server HELLO Detected
1633 3:30881 MALWARE-OTHER dns request with long host name segment - possible data exfiltration attempt
1611 119:15 http_inspect: OVERSIZE REQUEST-URI DIRECTORY
1162 3:19187 PROTOCOL-DNS TMG Firewall Client long host entry exploit attempt
670 129:5 stream5: Bad segment, overlap adjusted size less than/equal 0
274 129:3 stream5: Data sent on stream not accepting data
216 129:17 stream5: ACK number is greater than prior FIN
166 120:9 http_inspect: JAVASCRIPT OBFUSCATION LEVELS EXCEEDS 1
87 3:15912 OS-WINDOWS TCP window closed before receiving data
81 1:31978 OS-OTHER Bash CGI environment variable injection attempt
37 120:11 http_inspect: MULTIPLE ENCODINGS WITHIN JAVASCRIPT OBFUSCATED DATA
36 120:10 http_inspect: JAVASCRIPT WHITESPACES EXCEEDS MAX ALLOWED
34 128:1 ssh: Gobbles exploit
19 129:19 stream5: TCP window closed before receiving data
13 1:19438 SQL url ending in comment characters - possible sql injection attempt
12 140:3 sip: URI is too long
12 1:30524 SERVER-OTHER OpenSSL TLSv1.1 heartbeat read overrun attempt
11 124:10 smtp: Base64 Decoding failed
7 123:8 frag3: Fragmentation overlap
6 120:4 http_inspect: HTTP RESPONSE HAS UTF CHARSET WHICH FAILED TO NORMALIZE
3 119:28 http_inspect: POST W/O CONTENT-LENGTH OR CHUNKS
3 1:34194 SERVER-WEBAPP RevSlider information disclosure attempt
1 3:32111 SERVER-OTHER CSO-user ASA IKEv2 denial of service attempt
1 119:3 http_inspect: U ENCODING
Total
12190
=========================================================================
Top 50 All time Sguil Events
=========================================================================
Totals GenID:SigID Signature
82614 129:19 stream5: TCP window closed before receiving data
82070 120:7 http_inspect: CHUNKED ENCODING - EXCESSIVE CONSECUTIVE SMALL CHUNKS
66190 119:15 http_inspect: OVERSIZE REQUEST-URI DIRECTORY
54305 3:30881 MALWARE-OTHER dns request with long host name segment - possible data exfiltration attempt
19890 129:5 stream5: Bad segment, overlap adjusted size less than/equal 0
18687 1:1000985 Monitor external request to RDP port
17560 120:9 http_inspect: JAVASCRIPT OBFUSCATION LEVELS EXCEEDS 1
17478 137:1 ssp_ssl: Invalid Client HELLO after Server HELLO Detected
11936 3:19187 PROTOCOL-DNS TMG Firewall Client long host entry exploit attempt
10081 129:3 stream5: Data sent on stream not accepting data
9372 140:27 sip: Maximum dialogs in a session reached
8827 1:2000419 ET POLICY PE EXE or DLL Windows file download
8220 120:10 http_inspect: JAVASCRIPT WHITESPACES EXCEEDS MAX ALLOWED
2704 3:30881 Snort Alert [1:30881:4]
1488 1:2015561 ET INFO PDF Using CCITTFax Filter
1415 1:2014819 ET INFO Packed Executable Download
1155 1:2019401 ET POLICY Vulnerable Java Version 1.8.x Detected
1131 1:2011802 ET DNS DNS Lookup for localhost.DOMAIN.TLD
964 125:1 ftp_pp: Telnet command on FTP command channel
939 1:2012252 ET SHELLCODE Common 0a0a0a0a Heap Spray String
924 1:34463 APP-DETECT TeamViewer remote administration tool outbound connection attempt
879 124:10 smtp: Base64 Decoding failed
875 129:17 stream5: ACK number is greater than prior FIN
842 1:2006380 ET POLICY Outgoing Basic Auth Base64 HTTP Password detected unencrypted
803 1:2013031 ET POLICY Python-urllib/ Suspicious User Agent
637 120:11 http_inspect: MULTIPLE ENCODINGS WITHIN JAVASCRIPT OBFUSCATED DATA
585 3:15912 OS-WINDOWS TCP window closed before receiving data
571 1:25459 FILE-PDF Adobe Acrobat Reader incomplete JP2K image geometry exploit attempt
537 1:2001219 ET SCAN Potential SSH Scan
509 1:2016360 ET INFO JAVA - ClassID
505 1:2013222 ET SHELLCODE Excessive Use of HeapLib Objects Likely Malicious Heap Spray Attempt
489 1:2014518 ET INFO EXE - OSX Disk Image Download
470 3:20825 SERVER-WEBAPP generic web server hashing collision attack
444 1:2018359 ET INFO GENERIC SUSPICIOUS POST to Dotted Quad with Fake Browser 2
416 1:2014920 ET POLICY Microsoft Online Storage Client Hello TLSv1 Possible SkyDrive (2)
416 1:2014919 ET POLICY Microsoft Online Storage Client Hello TLSv1 Possible SkyDrive (1)
333 1:2010066 ET POLICY Data POST to an image file (gif)
297 1:2012843 ET POLICY Cleartext WordPress Login
295 124:1 smtp: Attempted command buffer overflow
290 1:2011507 ET WEB_CLIENT PDF With Embedded File
268 128:1 ssh: Gobbles exploit
237 1:2010908 ET MALWARE Mozilla User-Agent (Mozilla/5.0) Inbound Likely Fake
233 1:31977 OS-OTHER Bash CGI environment variable injection attempt
230 124:2 smtp: Attempted data header buffer overflow
224 1:32670 MALWARE-CNC Win.Dropper.Ch variant outbound connection
223 1:2013410 ET POLICY Outbound MSSQL Connection to Standard port (1433)
216 1:2013936 ET POLICY SSH banner detected on TCP 443 likely proxy evasion
206 1:2402000 ET DROP Dshield Block Listed Source group 1
195 1:2016877 ET POLICY Unsupported/Fake FireFox Version 2.
191 1:2012647 ET POLICY Dropbox.com Offsite File Backup in Use
Total
437279
=========================================================================
=========================================================================
Last update
=========================================================================
=========================================================================
ELSA
=========================================================================
Syslog-ng
Checking for process:
25592 supervising syslog-ng
25593 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
Checking for connection:
Connection to localhost 514 port [tcp/shell] succeeded!
MySQL
Checking for process:
2453 /usr/sbin/mysqld
49083 /usr/bin/mysql -uroot -BN -e DELETE FROM history WHERE timestamp < DATE_SUB(NOW(), INTERVAL 1 DAY); -D securityonion_db
Checking for connection:
Connection to localhost 3306 port [tcp/mysql] succeeded!
Sphinx
Checking for process:
1338 /usr/bin/indexer --config /etc/sphinxsearch/sphinx.conf --rotate perm_4
2347 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach
Checking for connection:
Connection to localhost 9306 port [tcp/*] succeeded!
ELSA Buffers in Queue:
6
If this number is consistently higher than 20, please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/FAQ#why-does-sostat-show-a-high-number-of-elsa-buffers-in-queue
ELSA Directory Sizes:
1.4T /nsm/elsa/data
22M /var/lib/mysql/syslog
1.8M /var/lib/mysql/syslog_data
ELSA Index Date Range:
MIN(start) MAX(end)
2015-10-27 05:56:13 2015-11-02 16:30:33
Wes
This recently happened to me (if I am interpreting your issue correctly). I had to manually categorize the items in securityonion_db because sostat output did not correlate what I was seeing within the Sguil console.
After stopping sguild (sudo nsm_server_ps-stop), I directly updated securityonion_db:
(Example, substituting last modified, signature, status, etc.)
UPDATE event SET status=1, last_modified='2015-10-22 13:46:00', last_uid='sguil' WHERE event.status=0 and event.signature LIKE '%';
More instructions here:
http://taosecurity.blogspot.com/2013/02/recovering-from-suricata-gone-wild.html
I'm not quite sure why this happened in the first place, but I hope this helps.
If this is not relative to your issue, please disregard.
Thanks,
Wes
rex warnert
Thank you for the quick reply. I had a moment of clarity where I think I worked out why this is happening. I am testing my theory now. If it doesnt work I will try what you posted. if it does work I will post what i did and why I think it is happening.
Rex
rex warnert
When you read the messages properly it give you the clues. We stood up the server recently and were in the process of tuning. So in the beginning we had 47K+ rules and lots of alerts. As we tuned, I would do a sguil-db-purge after setting the DAYS TO KEEP=1 and DAYSTOREPAIR=1 in the securityonion.conf. it would categorize everything down to 100,000. When it did that it always said doing the oldest events...
So what happened was everytime I changed something reduced the rule set, manually categorizing rules etc. it was on NEW events. When I setup auto.cat rules it only effected new stuff coming in not those oldest 100,000 that was still sitting in the DB somewhere uncategorized. Even though they did not show up in sguil or Squert as needing to be categorized.
So to test my Theory out I set the DAYS TO KEEP=1 and DAYSTOREPAIR=1 in the securityonion.conf again but also set the UNCAT_MAX=1. Then ran sguil-sb-purge again and it finally categorized "THE OLDEST" 100,000 events that had been given me the problem.
Now when I run sostat it gives me the correct amount.
Hope that helps someone else that this has been bugging.
Regards,
Rex
On Monday, November 2, 2015 at 11:49:50 AM UTC-5, Wes wrote: