Sguil stops showing events

635 views
Skip to first unread message

Jerry Shenk

unread,
Mar 23, 2012, 11:08:31 AM3/23/12
to security-onion
OK, I've got something strange going on - my SO installation has been
up and running for 6 weeks or so. Yesterday morning, I noticed that
the event query tab had stopped updating at around 7AM. I checked a
few things and noticed that port 7736 wasn't listening (netstat -an |
grep LIST). I tried restarting a few times, rebooted and then thought
that maybe I should just upgrade to the latest...might fix some bug.
After that, the event counter jumps up every time I restart...like it
just processes all the stuff that was "stuck" in a queue someplace.
It takes awhile for everything to get fully started but one tclsh
stops taking 100% of the CPU, port 7736 is listening. I did that a
few times yesterday - mostly just restarting nsm but a few times, I
rebooted the whole box. This morning, the last events are again from
6:59 but this time, port 7736 is running but, the "barnyard (spooler,
unified2 format" process was in a FAIL state (service nsm status). A
restart cause the event list to be current and right now the tclsh
process is taking 100% of the CPU so I don't know yet wether it will
be processing events...probably not.

Any ideas what my problem might be?

scott runnels

unread,
Mar 23, 2012, 11:18:25 AM3/23/12
to securit...@googlegroups.com
Hi Jerry,
Can you run sostat and send us the output? Please make sure to scrub any sensitive information from the outputs.

v/r
Scott

Jerry Shenk

unread,
Mar 23, 2012, 11:19:39 AM3/23/12
to security-onion
I should probably wait till tclsh drops down below 90% before I send
that, right?

scott runnels

unread,
Mar 23, 2012, 11:22:56 AM3/23/12
to securit...@googlegroups.com
That's up to you. The field I'm most interested in is generated by:

mysql -uroot securityonion_db -e 'SELECT COUNT(*) FROM event WHERE status=0'

You can probably run just that command and send it back to the list.

v/r
Scott


On Mar 23, 2012, at 11:08 AM, Jerry Shenk wrote:

Jerry Shenk

unread,
Mar 23, 2012, 11:38:10 AM3/23/12
to security-onion
root@ids-desktop:~# sostat
=========================================================================
Service Status
=========================================================================
Status: securityonion
* sguil server [ OK ]
Status: ids-desktop-eth2
* pcap_agent (sguil) [ OK ]
* sancp_agent (sguil) [ OK ]
* snort_agent (sguil) [ OK ]
* pads_agent (sguil) [ OK ]
* snort (alert data) [ OK ]
* barnyard2 (spooler, unified2 format) [ OK ]
* sancp (session data) [ OK ]
* pads (asset info) [ OK ]
* daemonlogger (full packet data) [ OK ]
* argus [ OK ]
* httpry [ OK ]
* httpry_agent (sguil) [ OK ]
Status: HIDS
* ossec_agent (sguil) [ OK ]
Status: Bro
Name Type Host Status Pid Peers Started
bro standalone localhost running 2827 0 23 Mar
11:05:26

=========================================================================
Interface Status
=========================================================================
eth0 Link encap:Ethernet HWaddr e4:11:5b:12:ec:ea
inet addr:192.168.16.9 Bcast:192.168.16.255 Mask:
255.255.255.0
inet6 addr: fe80::e611:5bff:fe12:ecea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:504928 errors:0 dropped:0 overruns:0 frame:0
TX packets:165665 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:97087076 (97.0 MB) TX bytes:160397377 (160.3 MB)
Memory:fbde0000-fbe00000

eth1 Link encap:Ethernet HWaddr e4:11:5b:12:ec:eb
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Memory:fbee0000-fbf00000

eth2 Link encap:Ethernet HWaddr 00:26:55:ec:74:ce
inet addr:1.1.1.1 Bcast:1.255.255.255 Mask:255.255.255.255
inet6 addr: fe80::226:55ff:feec:74ce/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4469880 errors:0 dropped:0 overruns:0 frame:0
TX packets:400 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1890406861 (1.8 GB) TX bytes:35160 (35.1 KB)
Memory:fbfe0000-fc000000

eth3 Link encap:Ethernet HWaddr 00:26:55:ec:74:cf
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Memory:fbfa0000-fbfc0000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:228856 errors:0 dropped:0 overruns:0 frame:0
TX packets:228856 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:60277068 (60.2 MB) TX bytes:60277068 (60.2 MB)


=========================================================================
Disk Usage
=========================================================================
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 220G 107G 103G 51% /
none 1.9G 216K 1.9G 1% /dev
none 1.9G 0 1.9G 0% /dev/shm
none 1.9G 152K 1.9G 1% /var/run
none 1.9G 0 1.9G 0% /var/lock
none 1.9G 0 1.9G 0% /lib/init/rw
none 220G 107G 103G 51% /var/lib/ureadahead/
debugfs

=========================================================================
IDS Rules Update
=========================================================================
Fri Mar 23 07:01:01 EDT 2012
Backing up current downloaded.rules file before it gets overwritten.
Cleaning up downloaded.rules backup files older than 30 days.
Running PulledPork.
http://code.google.com/p/pulledpork/
_____ ____
`----,\ )
`--==\\ / PulledPork v0.5.0 The Drowning Rat
`--==\\/
.-~~~~-.Y|\\_ Copyright (C) 2009-2010 JJ Cummings
@_/ / 66\_ cumm...@gmail.com
| \ \ _(")
\ /-| ||'--' Rules give me wings!
\_\ \_\\
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Checking latest MD5 for snortrules-snapshot-2920.tar.gz....
Rules tarball download of snortrules-snapshot-2920.tar.gz....
They Match
Done!
Prepping rules from snortrules-snapshot-2920.tar.gz for work....
Done!
Checking latest MD5 for emerging.rules.tar.gz....
Rules tarball download of emerging.rules.tar.gz....
They Match
Done!
Prepping rules from emerging.rules.tar.gz for work....
Done!
Reading rules...
Generating Stub Rules....
Done
Reading rules...
Reading rules...
Reading rules...
Processing /etc/pulledpork/enablesid.conf....
Modified 1 rules
Done
Processing /etc/pulledpork/dropsid.conf....
Modified 0 rules
Done
Processing /etc/pulledpork/disablesid.conf....
Modified 0 rules
Done
Modifying Sids....
Done!
Setting Flowbit State....
Enabled 55 flowbits
Enabled 36 flowbits
Done
Writing /etc/nsm/rules/downloaded.rules....
Done
Writing /etc/nsm/rules/so_rules.rules....
Done
Generating sid-msg.map....
Done
Writing /etc/snort/sid-msg.map....
Done
Writing /var/log/sid_changes.log....
Done
Rule Stats....
New:-------8
Deleted:---10
Enabled Rules:----17121
Dropped Rules:----0
Disabled Rules:---11664
Total Rules:------28785
Done
Please review /var/log/sid_changes.log for additional details
Fly Piggy Fly!
Restarting Barnyard2.
Restarting: ids-desktop-eth2
* stopping: barnyard2 (spooler, unified2 format) [ OK ]
* starting: barnyard2 (spooler, unified2 format) [ OK ]
Restarting IDS Engine.
Restarting: ids-desktop-eth2
* stopping: snort (alert data) [ OK ]
* starting: snort (alert data) [ OK ]

=========================================================================
CPU Usage
=========================================================================
top - 11:34:50 up 21:30, 1 user, load average: 1.95, 1.99, 1.78
Tasks: 174 total, 2 running, 172 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.0%us, 3.4%sy, 0.7%ni, 89.9%id, 1.0%wa, 0.0%hi,
0.0%si, 0.0%st
Mem: 3903556k total, 3785044k used, 118512k free, 126616k
buffers
Swap: 11888660k total, 15536k used, 11873124k free, 1984852k
cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2131 root 20 0 972m 964m 2920 R 101 25.3 28:42.87 tclsh
2827 root 20 0 28536 21m 9.8m S 6 0.6 2:07.71 bro
2836 root 25 5 24680 14m 4532 S 4 0.4 1:35.87 bro
2318 sguil 20 0 555m 303m 135m S 2 8.0 0:43.29 snort
1 root 20 0 2888 1652 1216 S 0 0.0 0:00.85 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/
0
4 root 20 0 0 0 0 S 0 0.0 0:00.47 ksoftirqd/
0
5 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/
0
6 root RT 0 0 0 0 S 0 0.0 0:00.01 migration/
1
7 root 20 0 0 0 0 S 0 0.0 0:00.49 ksoftirqd/
1
8 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/
1
9 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/
2
10 root 20 0 0 0 0 S 0 0.0 0:01.04 ksoftirqd/
2
11 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/
2
12 root RT 0 0 0 0 S 0 0.0 0:00.02 migration/
3
13 root 20 0 0 0 0 S 0 0.0 0:00.56 ksoftirqd/
3
14 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/
3
15 root 20 0 0 0 0 S 0 0.0 0:09.40 events/0
16 root 20 0 0 0 0 S 0 0.0 0:00.52 events/1
17 root 20 0 0 0 0 S 0 0.0 0:04.08 events/2
18 root 20 0 0 0 0 S 0 0.0 0:00.55 events/3
19 root 20 0 0 0 0 S 0 0.0 0:00.00 cpuset
20 root 20 0 0 0 0 S 0 0.0 0:00.00 khelper
21 root 20 0 0 0 0 S 0 0.0 0:00.00 async/mgr
22 root 20 0 0 0 0 S 0 0.0 0:00.00 pm
24 root 20 0 0 0 0 S 0 0.0 0:00.01
sync_supers
25 root 20 0 0 0 0 S 0 0.0 0:00.02 bdi-
default
26 root 20 0 0 0 0 S 0 0.0 0:00.00
kintegrityd/0
27 root 20 0 0 0 0 S 0 0.0 0:00.00
kintegrityd/1
28 root 20 0 0 0 0 S 0 0.0 0:00.00
kintegrityd/2
29 root 20 0 0 0 0 S 0 0.0 0:00.00
kintegrityd/3
30 root 20 0 0 0 0 S 0 0.0 0:00.05 kblockd/0
31 root 20 0 0 0 0 S 0 0.0 0:00.28 kblockd/1
32 root 20 0 0 0 0 S 0 0.0 0:00.02 kblockd/2
33 root 20 0 0 0 0 S 0 0.0 0:00.02 kblockd/3
34 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpid
35 root 20 0 0 0 0 S 0 0.0 0:00.00
kacpi_notify
36 root 20 0 0 0 0 S 0 0.0 0:00.00
kacpi_hotplug
37 root 20 0 0 0 0 S 0 0.0 0:00.00 ata/0
38 root 20 0 0 0 0 S 0 0.0 0:05.17 ata/1
39 root 20 0 0 0 0 S 0 0.0 0:00.00 ata/2
40 root 20 0 0 0 0 S 0 0.0 0:04.93 ata/3
41 root 20 0 0 0 0 S 0 0.0 0:00.00 ata_aux
42 root 20 0 0 0 0 S 0 0.0 0:00.00
ksuspend_usbd
43 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
44 root 20 0 0 0 0 S 0 0.0 0:00.00 kseriod
45 root 20 0 0 0 0 S 0 0.0 0:00.00 kmmcd
50 root 20 0 0 0 0 S 0 0.0 0:00.01
khungtaskd
51 root 20 0 0 0 0 S 0 0.0 0:01.44 kswapd0
52 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
53 root 20 0 0 0 0 S 0 0.0 0:00.00 aio/0
54 root 20 0 0 0 0 S 0 0.0 0:00.00 aio/1
55 root 20 0 0 0 0 S 0 0.0 0:00.00 aio/2
56 root 20 0 0 0 0 S 0 0.0 0:00.00 aio/3
57 root 20 0 0 0 0 S 0 0.0 0:00.00 ecryptfs-
kthrea
58 root 20 0 0 0 0 S 0 0.0 0:00.00 crypto/0
59 root 20 0 0 0 0 S 0 0.0 0:00.00 crypto/1
60 root 20 0 0 0 0 S 0 0.0 0:00.00 crypto/2
61 root 20 0 0 0 0 S 0 0.0 0:00.00 crypto/3
77 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
78 root 20 0 0 0 0 S 0 0.0 0:00.01 scsi_eh_1
81 root 20 0 0 0 0 S 0 0.0 0:00.01 scsi_eh_2
82 root 20 0 0 0 0 S 0 0.0 0:16.94 scsi_eh_3
85 root 20 0 0 0 0 S 0 0.0 0:00.00 kstriped
86 root 20 0 0 0 0 S 0 0.0 0:00.00 kmpathd/0
87 root 20 0 0 0 0 S 0 0.0 0:00.00 kmpathd/1
88 root 20 0 0 0 0 S 0 0.0 0:00.00 kmpathd/2
89 root 20 0 0 0 0 S 0 0.0 0:00.00 kmpathd/3
90 root 20 0 0 0 0 S 0 0.0 0:00.00
kmpath_handlerd
91 root 20 0 0 0 0 S 0 0.0 0:00.00 ksnapd
92 root 20 0 0 0 0 S 0 0.0 0:00.00 kondemand/
0
93 root 20 0 0 0 0 S 0 0.0 0:00.00 kondemand/
1
94 root 20 0 0 0 0 S 0 0.0 0:00.00 kondemand/
2
95 root 20 0 0 0 0 S 0 0.0 0:00.00 kondemand/
3
96 root 20 0 0 0 0 S 0 0.0 0:00.00
kconservative/0
97 root 20 0 0 0 0 S 0 0.0 0:00.00
kconservative/1
98 root 20 0 0 0 0 S 0 0.0 0:00.00
kconservative/2
99 root 20 0 0 0 0 S 0 0.0 0:00.00
kconservative/3
315 root 20 0 0 0 0 S 0 0.0 0:00.00
usbhid_resumer
367 root 20 0 0 0 0 S 0 0.0 0:06.91 jbd2/
sda1-8
368 root 20 0 0 0 0 S 0 0.0 0:00.00 ext4-dio-
unwrit
369 root 20 0 0 0 0 S 0 0.0 0:00.00 ext4-dio-
unwrit
370 root 20 0 0 0 0 S 0 0.0 0:00.00 ext4-dio-
unwrit
371 root 20 0 0 0 0 S 0 0.0 0:00.00 ext4-dio-
unwrit
397 root 20 0 0 0 0 S 0 0.0 0:06.13 flush-8:0
432 root 20 0 2316 752 668 S 0 0.0 0:00.08 upstart-
udev-br
436 root 16 -4 2688 336 332 S 0 0.0 0:00.04 udevd
558 root 18 -2 2556 324 320 S 0 0.0 0:00.00 udevd
567 root 18 -2 2684 316 312 S 0 0.0 0:00.00 udevd
789 root 20 0 0 0 0 S 0 0.0 0:00.00 kpsmoused
983 messageb 20 0 2920 1048 684 S 0 0.0 0:00.06 dbus-
daemon
1002 root 20 0 18784 2856 2680 S 0 0.1 0:00.02 gdm-
binary
1005 syslog 20 0 45836 1516 1156 S 0 0.0 0:37.86 rsyslogd
1015 avahi 20 0 3044 1424 1300 S 0 0.0 0:00.17 avahi-
daemon
1016 avahi 20 0 2928 332 320 S 0 0.0 0:00.00 avahi-
daemon
1032 root 20 0 9516 4016 3804 S 0 0.1 0:00.04
NetworkManager
1034 root 20 0 4172 2024 1856 S 0 0.1 0:00.02 modem-
manager
1050 root 20 0 20472 2476 2240 S 0 0.1 0:00.03 console-
kit-dae
1115 root 20 0 20372 2828 2788 S 0 0.1 0:00.01 gdm-
simple-slav
1130 root 20 0 54012 15m 4464 S 0 0.4 0:05.85 Xorg
1157 root 20 0 4836 1468 1464 S 0 0.0 0:00.00
wpa_supplicant
1264 root 20 0 1792 484 480 S 0 0.0 0:00.00 getty
1268 root 20 0 1792 484 480 S 0 0.0 0:00.00 getty
1287 root 20 0 1792 488 484 S 0 0.0 0:00.00 getty
1288 root 20 0 1792 488 484 S 0 0.0 0:00.00 getty
1291 root 20 0 1792 488 484 S 0 0.0 0:00.00 getty
1295 root 20 0 2048 508 504 S 0 0.0 0:00.00 acpid
1302 root 20 0 2376 772 704 S 0 0.0 0:00.10 cron
1303 daemon 20 0 2248 312 296 S 0 0.0 0:00.00 atd
1312 root 20 0 2828 532 468 S 0 0.0 0:03.57
irqbalance
1373 mysql 20 0 151m 49m 6528 S 0 1.3 7:21.51 mysqld
1385 gdm 20 0 3384 516 512 S 0 0.0 0:00.00 dbus-
launch
1386 gdm 20 0 2664 764 628 S 0 0.0 0:00.01 dbus-
daemon
1388 gdm 20 0 27284 5164 4816 S 0 0.1 0:00.05 gnome-
session
1412 gdm 20 0 6512 3148 2208 S 0 0.1 0:00.22 gconfd-2
1413 root 20 0 39704 10m 6604 S 0 0.3 0:01.14 apache2
1486 gdm 20 0 34036 9m 8008 S 0 0.3 0:01.38 gnome-
settings-
1535 gdm 20 0 6168 2044 1752 S 0 0.1 0:00.00 gvfsd
1541 root 20 0 6700 2420 1816 S 0 0.1 0:00.02 cupsd
1577 gdm 20 0 28560 7824 6140 S 0 0.2 0:00.30 metacity
1599 gdm 20 0 32652 12m 9980 S 0 0.3 0:02.62 gdm-
simple-gree
1605 root 20 0 7932 2384 2024 S 0 0.1 0:00.00 gdm-
session-wor
1611 gdm 20 0 3856 1924 1672 S 0 0.0 0:00.00 xfconfd
1615 ossec 20 0 3528 2172 688 S 0 0.1 0:14.32 ossec-
analysisd
1625 root 20 0 1964 512 396 S 0 0.0 0:05.45 ossec-
logcollec
1655 root 20 0 3024 1892 620 S 0 0.0 0:29.46 ossec-
syscheckd
1664 ossec 20 0 2240 764 516 S 0 0.0 0:00.55 ossec-
monitord
1681 gdm 20 0 16580 3200 2256 S 0 0.1 0:00.01 xfce4-
power-man
1700 haldaemo 20 0 16512 3824 3028 S 0 0.1 0:00.39 hald
1701 root 20 0 3536 1276 1068 S 0 0.0 0:00.00 hald-
runner
1816 root 20 0 5552 2124 1716 S 0 0.1 0:00.00 sshd
1839 root 20 0 3612 1224 1048 S 0 0.0 0:00.00 hald-
addon-inpu
1895 root 20 0 3616 1232 1052 S 0 0.0 0:20.11 hald-
addon-stor
1897 haldaemo 20 0 3420 1168 992 S 0 0.0 0:00.00 hald-
addon-acpi
2188 root 20 0 5692 3040 2216 S 0 0.1 0:00.07 tclsh
2227 root 20 0 5680 3024 2216 S 0 0.1 0:00.07 tclsh
2266 root 20 0 5956 3324 2252 S 0 0.1 0:00.10 tclsh
2268 root 20 0 3260 668 568 S 0 0.0 0:00.01 tail
2357 root 20 0 14312 8396 1428 S 0 0.2 0:13.28 barnyard2
2406 sguil 20 0 7728 5572 5172 S 0 0.1 0:01.34 sancp
2451 sguil 20 0 7324 6032 4980 S 0 0.2 0:01.11 pads
2490 root 20 0 5720 3048 2228 S 0 0.1 0:00.10 tclsh
2492 root 20 0 3256 616 524 S 0 0.0 0:00.00 cat
2516 sguil 20 0 6204 4928 4788 S 0 0.1 0:03.62
daemonlogger
2550 root 20 0 9216 2948 932 S 0 0.1 0:00.03 tclsh
2551 root 20 0 9216 2792 776 S 0 0.1 0:00.00 tclsh
2568 sguil 20 0 31404 8316 3184 S 0 0.2 0:02.74 argus
2621 sguil 20 0 6500 5116 4964 S 0 0.1 0:00.00 httpry
2660 root 20 0 5680 3024 2216 S 0 0.1 0:00.07 tclsh
2698 root 20 0 6224 3096 1800 S 0 0.1 0:00.03 tclsh
2818 root 20 0 4448 1536 1288 S 0 0.0 0:00.01 bash
2989 Debian-e 20 0 6728 964 604 S 0 0.0 0:00.00 exim4
3024 root 20 0 1792 564 484 S 0 0.0 0:00.00 getty
3077 ntp 20 0 4424 1384 1044 S 0 0.0 0:01.68 ntpd
3106 root 20 0 4216 1332 1136 S 0 0.0 0:00.00 sostat
3307 root 20 0 2544 1096 800 R 0 0.0 0:00.00 top
3798 root 20 0 10900 3556 2728 S 0 0.1 0:00.41 sshd
3803 www-data 20 0 77436 62m 3524 S 0 1.6 2:06.57 ruby
3888 ids 20 0 10900 1920 1032 S 0 0.0 0:08.84 sshd
3889 ids 20 0 4548 1856 1492 S 0 0.0 0:00.00 bash
3891 root 20 0 6588 2120 1600 S 0 0.1 0:00.08 su
3903 root 20 0 4688 2008 1500 S 0 0.1 0:00.04 bash
8861 root 20 0 9216 2788 776 S 0 0.1 0:00.00 tclsh
29895 root 20 0 5404 1748 1528 S 0 0.0 0:00.00
PassengerWatchd
29899 root 20 0 15180 2000 1776 S 0 0.1 0:00.05
PassengerHelper
29901 root 20 0 10988 6620 2240 S 0 0.2 0:09.09 ruby
29906 nobody 20 0 9580 3136 2576 S 0 0.1 0:00.01
PassengerLoggin
29913 www-data 20 0 41648 9644 4040 S 0 0.2 0:00.05 apache2
29914 www-data 20 0 41048 9372 4084 S 0 0.2 0:00.28 apache2
29915 www-data 20 0 39932 5628 1536 S 0 0.1 0:00.03 apache2
29916 www-data 20 0 41048 9360 4072 S 0 0.2 0:00.31 apache2
29917 www-data 20 0 41636 9936 4032 S 0 0.3 0:00.12 apache2
30576 www-data 20 0 39948 5596 1496 S 0 0.1 0:00.01 apache2


=========================================================================
Log Archive
=========================================================================
/nsm/bro/logs/:
2012-01-13 2012-01-17 2012-01-21 2012-01-25 2012-01-29
2012-02-02 2012-02-06 2012-02-10 2012-02-14 2012-02-18
2012-02-22 2012-02-26 2012-03-01 2012-03-05 2012-03-09
2012-03-13 2012-03-17 2012-03-21 stats
2012-01-14 2012-01-18 2012-01-22 2012-01-26 2012-01-30
2012-02-03 2012-02-07 2012-02-11 2012-02-15 2012-02-19
2012-02-23 2012-02-27 2012-03-02 2012-03-06 2012-03-10
2012-03-14 2012-03-18 2012-03-22
2012-01-15 2012-01-19 2012-01-23 2012-01-27 2012-01-31
2012-02-04 2012-02-08 2012-02-12 2012-02-16 2012-02-20
2012-02-24 2012-02-28 2012-03-03 2012-03-07 2012-03-11
2012-03-15 2012-03-19 2012-03-23
2012-01-16 2012-01-20 2012-01-24 2012-01-28 2012-02-01
2012-02-05 2012-02-09 2012-02-13 2012-02-17 2012-02-21
2012-02-25 2012-02-29 2012-03-04 2012-03-08 2012-03-12
2012-03-16 2012-03-20 current

/nsm/sensor_data/ids-desktop-eth2/dailylogs/:
2012-01-11 2012-01-15 2012-01-19 2012-01-23 2012-01-27
2012-01-31 2012-02-04 2012-02-08 2012-02-12 2012-02-16
2012-02-20 2012-02-24 2012-02-28 2012-03-03 2012-03-07
2012-03-11 2012-03-15 2012-03-19 2012-03-23
2012-01-12 2012-01-16 2012-01-20 2012-01-24 2012-01-28
2012-02-01 2012-02-05 2012-02-09 2012-02-13 2012-02-17
2012-02-21 2012-02-25 2012-02-29 2012-03-04 2012-03-08
2012-03-12 2012-03-16 2012-03-20
2012-01-13 2012-01-17 2012-01-21 2012-01-25 2012-01-29
2012-02-02 2012-02-06 2012-02-10 2012-02-14 2012-02-18
2012-02-22 2012-02-26 2012-03-01 2012-03-05 2012-03-09
2012-03-13 2012-03-17 2012-03-21
2012-01-14 2012-01-18 2012-01-22 2012-01-26 2012-01-30
2012-02-03 2012-02-07 2012-02-11 2012-02-15 2012-02-19
2012-02-23 2012-02-27 2012-03-02 2012-03-06 2012-03-10
2012-03-14 2012-03-18 2012-03-22

=========================================================================
IDS Engine (snort) packet drops
=========================================================================
/nsm/sensor_data/ids-desktop-eth2/snort.stats last reported
pkt_drop_percent as 0.000

=========================================================================
Sguil Uncategorized Events
=========================================================================
+----------+
| COUNT(*) |
+----------+
| 802005 |
+----------+

=========================================================================
Snorby Events Summary for yesterday
=========================================================================
+--------+-------------
+------------------------------------------------------------------------------------
+
| Totals | SignatureID |
SignatureName
|
+--------+-------------
+------------------------------------------------------------------------------------
+
| 3264 | 3 | Snort Alert
[142:3:0]
|
| 2568 | 5 | sensitive_data: sensitive data - eMail
addresses |
| 1129 | 2 | Snort Alert
[142:2:0]
|
| 1100 | 1 | Snort Alert
[142:1:0]
|
| 921 | 1 | sensitive_data: sensitive data global
threshold exceeded |
| 907 | 5 | stream5: Bad segment, overlap adjusted size
less than/equal 0 |
| 221 | 2002192 | ET CHAT MSN status
change |
| 214 | 2100540 | GPL CHAT MSN
message
|
| 169 | 9 | Snort Alert
[124:9:0]
|
| 152 | 2009375 | ET CHAT General MSN Chat
Activity |
| 98 | 2009376 | ET CHAT MSN User-Agent
Activity |
| 67 | 2101990 | GPL CHAT MSN user
search |
| 64 | 8 | Snort Alert
[120:8:0]
|
| 55 | 2013504 | ET POLICY GNU/Linux APT User-Agent Outbound
likely related to package management |
| 37 | 1 | ssp_ssl: Invalid Client HELLO after Server
HELLO Detected |
| 28 | 3 | http_inspect: NO CONTENT-LENGTH OR TRANSFER-
ENCODING IN HTTP RESPONSE |
| 23 | 2101991 | GPL CHAT MSN login
attempt |
| 11 | 2001682 | ET CHAT MSN IM Poll via
HTTP |
| 8 | 4 | http_inspect: HTTP RESPONSE HAS UTF CHARSET
WHICH FAILED TO NORMALIZE |
| 8 | 1390 | GPL SHELLCODE x86 inc ebx
NOOP |
| 8 | 6 | Snort Alert
[120:6:0]
|
| 6 | 3 | sensitive_data: sensitive data - U.S. social
security numbers with dashes |
| 5 | 2013028 | ET POLICY curl User-Agent
Outbound |
| 4 | 6 | sensitive_data: sensitive data - U.S. phone
numbers |
| 4 | 2100511 | GPL MISC Invalid PCAnywhere
Login |
| 3 | 2101986 | GPL CHAT MSN outbound file transfer
request |
| 3 | 2101988 | GPL CHAT MSN outbound file transfer
accept |
| 3 | 2011582 | ET POLICY Vulnerable Java Version 1.6.x
Detected |
| 3 | 15306 | FILE-IDENTIFY Portable Executable binary file
magic detection |
| 3 | 2 | sensitive_data: sensitive data - Credit card
numbers |
| 2 | 4 | stream5: TCP Timestamp is outside of PAWS
window |
| 1 | 14 | stream5: TCP Timestamp is
missing |
| 1 | 16482 | WEB-CLIENT Microsoft Internet Explorer
userdata behavior memory corruption attempt |
| 1 | 19 | http_inspect: LONG
HEADER |
| 1 | 2013031 | ET POLICY Python-urllib/ Suspicious User
Agent |
| 1 | 2100498 | GPL ATTACK_RESPONSE id check returned
root |
| 1 | 498 | ATTACK-RESPONSES id check returned
root |
| 1 | 8375 | WEB-ACTIVEX QuickTime Object ActiveX clsid
access |
| 1 | 4156 | WEB-ACTIVEX Windows Media Player 7+ ActiveX
object access |
| 1 | 2103134 | GPL WEB_CLIENT PNG large colour depth
download attempt |
| 1 | 15517 | WEB-CLIENT Microsoft Windows AVI DirectShow
QuickTime parsing overflow attempt |
| 1 | 16313 | FILE-IDENTIFY download of executable content
- x-header |
| 1 | 2003068 | ET SCAN Potential SSH Scan
OUTBOUND |
+--------+-------------
+------------------------------------------------------------------------------------
+
+-------+
| Total |
+-------+
| 11100 |
+-------+

=========================================================================
Top 50 All Time Snorby Events
=========================================================================
+--------+-------------
+----------------------------------------------------------------------------------
+
| Totals | SignatureID |
SignatureName
|
+--------+-------------
+----------------------------------------------------------------------------------
+
| 198031 | 3 | Snort Alert
[142:3:0] |
| 166097 | 408 | ICMP Echo
Reply
|
| 115173 | 5 | sensitive_data: sensitive data - eMail
addresses |
| 70034 | 2009375 | ET CHAT General MSN Chat
Activity |
| 53870 | 2 | Snort Alert
[142:2:0] |
| 51327 | 1 | Snort Alert
[142:1:0] |
| 47531 | 1 | sensitive_data: sensitive data global
threshold exceeded |
| 42365 | 12 | stream5: TCP Small Segment Threshold
Exceeded |
| 22385 | 5 | stream5: Bad segment, overlap adjusted size
less than/equal 0 |
| 20680 | 7 | stream5: Limit on number of overlapping TCP
packets reached |
| 14145 | 384 | ICMP
PING
|
| 13984 | 382 | ICMP PING
Windows
|
| 13800 | 9 | Snort Alert
[124:9:0] |
| 13078 | 4 | ssh: Protocol
mismatch |
| 9270 | 2100540 | GPL CHAT MSN
message |
| 7555 | 2002192 | ET CHAT MSN status
change |
| 7504 | 540 | CHAT MSN
message
|
| 5940 | 399 | ICMP Destination Unreachable Host
Unreachable |
| 4734 | 8 | Snort Alert
[120:8:0] |
| 3796 | 19 | http_inspect: LONG
HEADER |
| 3266 | 3 | http_inspect: NO CONTENT-LENGTH OR TRANSFER-
ENCODING IN HTTP RESPONSE |
| 2772 | 12800 | SHELLCODE base64 x86
NOOP |
| 2717 | 2101990 | GPL CHAT MSN user
search |
| 2691 | 2 | ftp_pp: Invalid FTP
command |
| 2573 | 19280 | POLICY attempted download of a PDF with
embedded Flash over pop3 |
| 2433 | 1394 | SHELLCODE x86 inc ecx
NOOP |
| 2398 | 2003068 | ET SCAN Potential SSH Scan
OUTBOUND |
| 2271 | 2003494 | ET MALWARE AskSearch Toolbar Spyware User-
Agent (AskTBar) |
| 2187 | 1990 | CHAT MSN user
search |
| 2057 | 1 | ssp_ssl: Invalid Client HELLO after Server
HELLO Detected |
| 1979 | 2010140 | ET P2P Vuze BT UDP
Connection |
| 1939 | 402 | ICMP Destination Unreachable Port
Unreachable |
| 1920 | 15 | stream5: Reset outside
window |
| 1548 | 2013504 | ET POLICY GNU/Linux APT User-Agent Outbound
likely related to package management |
| 1150 | 12801 | SHELLCODE base64 x86
NOOP |
| 1064 | 2010785 | ET CHAT Facebook Chat (buddy
list) |
| 923 | 2009376 | ET CHAT MSN User-Agent
Activity |
| 746 | 1390 | GPL SHELLCODE x86 inc ebx
NOOP |
| 715 | 2100368 | GPL ICMP_INFO PING
BSDtype |
| 715 | 2100366 | GPL ICMP_INFO PING
*NIX |
| 715 | 368 | ICMP PING
BSDtype
|
| 715 | 2003626 | ET MALWARE Double User-Agent (User-Agent User-
Agent) |
| 668 | 2000419 | ET POLICY PE EXE or DLL Windows file
download |
| 618 | 2101991 | GPL CHAT MSN login
attempt |
| 328 | 2001682 | ET CHAT MSN IM Poll via
HTTP |
| 316 | 15147 | WEB-CLIENT Microsoft Internet Explorer
malformed iframe buffer overflow attempt |
| 313 | 2100511 | GPL MISC Invalid PCAnywhere
Login |
| 216 | 2 | sensitive_data: sensitive data - Credit card
numbers |
| 188 | 3 | sensitive_data: sensitive data - U.S. social
security numbers with dashes |
| 179 | 20584 | WEB-CLIENT Mozilla multiple content-type
headers malicious redirect attempt |
+--------+-------------
+----------------------------------------------------------------------------------
+
+--------+
| Total |
+--------+
| 926310 |
+--------+
root@ids-desktop:~#

Jerry Shenk

unread,
Mar 23, 2012, 11:41:11 AM3/23/12
to security-onion
update - tclsh is now below 100% and events seem to be coming in
normally.

The uncatagorized event count is incrimenting also:
root@ids-desktop:~# mysql -uroot securityonion_db -e 'SELECT COUNT(*)
FROM event WHERE status=0'
+----------+
| COUNT(*) |
+----------+
| 804822 |
+----------+

scott runnels

unread,
Mar 23, 2012, 11:43:26 AM3/23/12
to securit...@googlegroups.com
Yup!  Need to either do some tuning or some categorizing!  http://code.google.com/p/security-onion/wiki/ManagingAlerts

Jerry Shenk

unread,
Mar 23, 2012, 11:56:41 AM3/23/12
to security-onion
The main thing I'd like to tune is this one:
198031 | 3 | Snort Alert [142:3:0]

That's getting a hit every time somebody checks mail from the off-site
mail server.

I've tried doing that in threshold.conf with this:
suppress gen_id 105, sig_id 3, track by_src, ip xx.xx.xx.xx


I had also initially done a bunch of categorizing but found that I've
I categorized something, then it would never be available for alerting
on so, I removed the categories so that I'd be sure to get the
alerts. I guess I should categorize a little more preceisely.

scott runnels

unread,
Mar 23, 2012, 12:06:01 PM3/23/12
to securit...@googlegroups.com
Hi Jerry, 

After you applied your suppression, did you see any new alerts for that signature?  Adding the suppression will stop future events, but you'd still need to clear them out of the DB either by categorizing within sguil or manually updating the entries.  

You can try this:

mysql -uroot securityonion_db -e 'SELECT COUNT(*) AS cnt, signature, signature_id FROM event WHERE status=0 GROUP BY signature ORDER BY cnt DESC LIMIT 10;' 

that will show you the top 10 alert producing signatures.

v/r
Scott Runnels


On Mar 23, 2012, at 11:38 AM, Jerry Shenk wrote:

Jerry Shenk

unread,
Mar 23, 2012, 12:11:13 PM3/23/12
to security-onion
Yes, I did see those same events continuing. I suppose it's possible
that something was sitting in a queue that hadn't been processed yet.
I'll try it again definitely let it run quite awhile before "giving
up".

scott runnels

unread,
Mar 23, 2012, 12:19:58 PM3/23/12
to securit...@googlegroups.com
You can modify that SQL statement to show you the most recent timestamp on events as well:

mysql -uroot securityonion_db -e 'SELECT COUNT(*), signature, signature_id, MAX(timestamp) FROM event WHERE status=0 GROUP BY signature ORDER BY cnt DESC LIMIT 10;'


v/r
Scott

Doug Burks

unread,
Mar 23, 2012, 12:36:29 PM3/23/12
to securit...@googlegroups.com
Hi Jerry,

Replies inline.

On Fri, Mar 23, 2012 at 11:56 AM, Jerry Shenk <jerry...@gmail.com> wrote:
> The main thing I'd like to tune is this one:
>  198031 |           3 | Snort Alert [142:3:0]
>
> That's getting a hit every time somebody checks mail from the off-site
> mail server.
>
> I've tried doing that in threshold.conf with this:
> suppress gen_id 105, sig_id 3, track by_src, ip xx.xx.xx.xx

Why are you suppressing gen_id 105? Shouldn't that be 142?

> I had also initially done a bunch of categorizing but found that I've
> I categorized something, then it would never be available for alerting
> on so, I removed the categories so that I'd be sure to get the
> alerts.  I guess I should categorize a little more preceisely.

What do you mean by categorizing? Using autocat.conf? Yes, it's
designed to automatically categorize any new events you've specified
and remove them from the RealTime Events tab, leaving them in the
database so that you can still query for them when necessary. If you
don't want the event to ever be in the database at all, you should
disable the rule in /etc/pulledpork/disablesid.conf or suppress it in
/etc/nsm/HOSTNAME-INTERFACE/threshold.conf.

Hope that helps!

Thanks,
--
Doug Burks | http://securityonion.blogspot.com
Don't miss SANS SEC503 Intrusion Detection In-Depth in
Augusta GA 6/11 - 6/16 | 10% discount for ISSA Members!
http://augusta.issa.org/drupal/SANS-Augusta-2012

Jerry Shenk

unread,
Mar 23, 2012, 12:42:51 PM3/23/12
to security-onion
Why, yes, I suppose it should be 142. When I click on the signature,
it shows sid: 3, gid: 105 which is a BO rule....that never really made
sense to me.

About categorizing, yes, I was doing that in autocat.conf. My idea
was that the events would be in categories...I guess that's right but,
that caused them to not be available for alerting on using
sguild.email. I do want them in the database and categorizing AND e-
mail alerts would be nice if that's possible.

Doug Burks

unread,
Mar 23, 2012, 1:11:05 PM3/23/12
to securit...@googlegroups.com
Replies inline.

On Fri, Mar 23, 2012 at 12:42 PM, Jerry Shenk <jerry...@gmail.com> wrote:
> Why, yes, I suppose it should be 142.  When I click on the signature,
> it shows sid: 3, gid: 105 which is a BO rule....that never really made
> sense to me.

Interesting. Could you send a screenshot?

> About categorizing, yes, I was doing that in autocat.conf.  My idea
> was that the events would be in categories...I guess that's right but,
> that caused them to not be available for alerting on using
> sguild.email.  I do want them in the database and categorizing AND e-
> mail alerts would be nice if that's possible.

When you add an event to autocat.conf, you're telling Sguil that you
don't want to be alerted in realtime, either by the RealTime Events
tab or by email. Think of it this way: if it's important enough to
generate an email, then it's important enough to appear in the
RealTime Events tab.

Jerry Shenk

unread,
Mar 23, 2012, 1:59:04 PM3/23/12
to security-onion
I'm not sure how to attach a picture but, I think the URL pretty well
explains the problem: https://192.168.16.9/squert/.inc/rule.php?sigID=3.
If you want the screenshot, just give me an e-mail to send it to.

So, it's simply pulling up sidID3 which is a BO alert.

Here is the text from the signature:
BO_SERVER_TRAFFIC_DETECT

alert ( msg: "BO_SERVER_TRAFFIC_DETECT"; sid: 3; gid: 105; rev: 1;
metadata: rule-type preproc, policy balanced-ips drop, policy security-
ips drop ; classtype:trojan-activity; reference:cve,1999-0660;)

downloaded.rules, line 27992.

Doug Burks

unread,
Mar 23, 2012, 3:17:37 PM3/23/12
to securit...@googlegroups.com
Ahh, you're experiencing this in Squert. If you look at one of these
events in the Sguil client, you should see something like "Rules and
signatures are not available for the generator ID 142."

PaulH, can you confirm this issue in Squert?

Thanks,
Doug

--

Jerry Shenk

unread,
Mar 23, 2012, 3:42:30 PM3/23/12
to security-onion
Now that I understand it, it makes perfect sense;) Not only that, it
works too.

Paul Halliday

unread,
Mar 23, 2012, 6:07:03 PM3/23/12
to securit...@googlegroups.com
On Fri, Mar 23, 2012 at 2:59 PM, Jerry Shenk <jerry...@gmail.com> wrote:
> I'm not sure how to attach a picture but, I think the URL pretty well
> explains the problem: https://192.168.16.9/squert/.inc/rule.php?sigID=3.
> If you want the screenshot, just give me an e-mail to send it to.

I am a visual kinda guy, could you send it to me please? :)

Thanks.

>
> So, it's simply pulling up sidID3 which is a BO alert.
>
> Here is the text from the signature:
> BO_SERVER_TRAFFIC_DETECT
>
> alert ( msg: "BO_SERVER_TRAFFIC_DETECT"; sid: 3; gid: 105; rev: 1;
> metadata: rule-type preproc, policy balanced-ips drop, policy security-
> ips drop ; classtype:trojan-activity; reference:cve,1999-0660;)
>
> downloaded.rules, line 27992.

--
Paul Halliday
http://www.squertproject.org/

Jerry Shenk

unread,
Mar 23, 2012, 6:39:57 PM3/23/12
to security-onion
done
Reply all
Reply to author
Forward
0 new messages