Security Onion Kernel Panic

949 views
Skip to first unread message

Thanh Le

unread,
Nov 11, 2013, 5:29:16 PM11/11/13
to securit...@googlegroups.com
Hi Doug,

I have followed your instructions to deploy Security Onion in my production network and the problem that I'm running into right now is the "Kernel Panic" message.

When the kernel panic message happens the server freezes up and I have to reboot it. The kernel panic message appears approximately 10 or 15 minutes after the server comes online.

I have done some research on the kernel panic message and I believed it has to do with the network card. I've been trying to use different type of network cards which didn't help. The network card which I have installed on the server right now is the Intel 82576.

I'm totally lost right now and If you can guide me in the right direction to fix this issue I really appreciated.

BTW....If I can get this to work in my production environment, I'm very confident that I can get my boss to contribute some money to this project.


Thanks

kernal panic.jpg

Doug Burks

unread,
Nov 11, 2013, 6:46:36 PM11/11/13
to securit...@googlegroups.com
Have you tried our ISO image?  It uses the original series of 12.04 kernels and may be more stable. 

For more info, please see:
--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/groups/opt_out.


--
Doug Burks
http://securityonion.blogspot.com

Thanh Le

unread,
Nov 12, 2013, 3:27:32 PM11/12/13
to securit...@googlegroups.com
Doug,

If I try your ISO image, then I can't partition the hard disks the way that you recommended for production.

Thanks,

Matt Gregory

unread,
Nov 12, 2013, 4:07:45 PM11/12/13
to securit...@googlegroups.com

You should have the option for custom partioning during the Ubuntu installation. If I remember correctly,  you have to select the "Advanced" link on one of the dialog screens.

Matt

Thanh Le

unread,
Nov 12, 2013, 5:27:36 PM11/12/13
to securit...@googlegroups.com
I believed it does not support LVM with your ISO image.

Doug Burks

unread,
Nov 12, 2013, 6:23:46 PM11/12/13
to securit...@googlegroups.com
On Tue, Nov 12, 2013 at 5:27 PM, Thanh Le <tle...@gmail.com> wrote:
> I believed it does not support LVM with your ISO image.

LVM is not required.


--
Doug Burks
http://securityonion.net

Thanh Le

unread,
Nov 12, 2013, 7:17:58 PM11/12/13
to securit...@googlegroups.com

Doug,

I have two 30 TB disk space which I need to use LVM to create one big partition to store /nsm log.

Do you have any recommendation for that?

Thanks,

Doug Burks

unread,
Nov 12, 2013, 8:57:50 PM11/12/13
to securit...@googlegroups.com
A few options:

- install our ISO image on the first drive/array leaving free space for /nsm to be allocated later. Reboot into your new installation. Add LVM and add the remaining space. Run Setup. 

- boot our ISO image into the Live environment and configure LVM manually:

- use an older Ubuntu server 12.04 image that has the original kernel series

- it may be possible to use the Ubuntu server 12.04.3 image and revert back to the original kernel series

--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/groups/opt_out.

Pietro Delsante

unread,
Nov 15, 2013, 9:25:47 AM11/15/13
to securit...@googlegroups.com
Hi,

I see you are using kernel 3.8.0-29. I never tried it, but I remember that I was having kernel panics with 3.8.0-30 a couple of months ago, and the problem was resolved when I updated to 3.8.0-31. Now I am happily running 3.8.0-33 that was released a couple of days ago.

I do not know if this is the same problem you're seeing, as I never figured out what was the cause, however before rebuilding your box from scratch, did you try to update the kernel?

Regards,
Pietro

Thanh Le

unread,
Nov 15, 2013, 1:16:53 PM11/15/13
to securit...@googlegroups.com

Pietro,

I will try that and post the update later.

Thank you.

Thanh Le

unread,
Nov 15, 2013, 1:34:02 PM11/15/13
to securit...@googlegroups.com

Pietro,

I have updated the kernel to 3.8.0-33 and the pf_ring still crashes.

Thanh Le

unread,
Nov 15, 2013, 6:18:33 PM11/15/13
to securit...@googlegroups.com
Doug,

I used your Security Onion image and I'm still getting kernel panic.

Doug Burks

unread,
Nov 15, 2013, 6:48:13 PM11/15/13
to securit...@googlegroups.com
Have you installed the latest kernel?

Have you run diagnostics on your server to rule out hardware issues?

Thanh Le

unread,
Nov 18, 2013, 5:00:47 PM11/18/13
to securit...@googlegroups.com
Yes,

Installed the latest kernel and the kernel panic still exist.

BTW....I just installed Security Onion on a different box and I still get the kernel panic message.

Any other suggestions?

Thanks,

Doug Burks

unread,
Nov 18, 2013, 5:02:44 PM11/18/13
to securit...@googlegroups.com
Have you run full hardware diagnostics on your server to rule out
hardware issues?

Thanh Le

unread,
Nov 18, 2013, 5:37:57 PM11/18/13
to securit...@googlegroups.com

I did run the full hardware diagnostic and didn't see any problem.

I installed Security Onion on an HP Server box and I still get the kernel panic message.

Ronny Vaningh

unread,
Nov 19, 2013, 1:14:35 AM11/19/13
to securit...@googlegroups.com

Please check the md5sum of your iso

Doug Burks

unread,
Nov 19, 2013, 6:23:01 AM11/19/13
to securit...@googlegroups.com
Please send the output of the following:
sudo sostat-redacted

It will redact IPv4 addresses, but there may be additional data that
you need to manually redact.

If you don't have sostat-redacted, you can either install all
available updates or do "sudo sostat" and manually redact.

Thanh Le

unread,
Nov 19, 2013, 1:41:19 PM11/19/13
to securit...@googlegroups.com
I checked the md5sum of my iso and it is good.

Thanh Le

unread,
Nov 19, 2013, 2:04:21 PM11/19/13
to securit...@googlegroups.com

Doug, here is the output of the sostat-redacted file.

=========================================================================
Service Status
=========================================================================
Status: securityonion
* sguil server[ OK ]
Status: HIDS
* ossec_agent (sguil)[ OK ]
Status: Bro
Name Type Host Status Pid Peers Started
bro standalone localhost running 3899 0 19 Nov 18:45:04
Status: WA-TEST-IDS-eth1
* netsniff-ng (full packet data)[ OK ]
* pcap_agent (sguil)[ OK ]
* snort_agent-1 (sguil)[ OK ]
* snort-1 (alert data)[ OK ]
* barnyard2-1 (spooler, unified2 format)[ FAIL ]
* stale PID file found, process will be restarted at the next 5-minute interval!
* prads (sessions/assets)[ OK ]
* sancp_agent (sguil)[ OK ]
* pads_agent (sguil)[ OK ]
* argus[ OK ]
* http_agent (sguil)[ OK ]

=========================================================================
Interface Status
=========================================================================
eth0 Link encap:Ethernet HWaddr 00:1b:21:72:23:d4
inet addr:X.X.X.X Bcast:X.X.X.X Mask:X.X.X.X
inet6 addr: fe80::21b:21ff:fe72:23d4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:100827 errors:0 dropped:0 overruns:0 frame:0
TX packets:2303 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7879602 (7.8 MB) TX bytes:333967 (333.9 KB)
Memory:fcee0000-fcf00000

eth1 Link encap:Ethernet HWaddr 00:1b:21:72:23:d5
UP BROADCAST NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Memory:fc7c0000-fc7e0000

lo Link encap:Local Loopback
inet addr:X.X.X.X Mask:X.X.X.X
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2253 errors:0 dropped:0 overruns:0 frame:0
TX packets:2253 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2777212 (2.7 MB) TX bytes:2777212 (2.7 MB)


=========================================================================
Disk Usage
=========================================================================
Filesystem Size Used Avail Use% Mounted on
/dev/cciss/c0d0p1 321G 12G 293G 4% /
udev 7.9G 4.0K 7.9G 1% /dev
tmpfs 3.2G 352K 3.2G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.9G 0 7.9G 0% /run/shm

=========================================================================
Network Sockets
=========================================================================
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 930 root 3u IPv4 12529 0t0 TCP *:22 (LISTEN)
sshd 930 root 4u IPv6 12531 0t0 TCP *:22 (LISTEN)
mysqld 1103 mysql 10u IPv4 2035 0t0 TCP X.X.X.X:3306 (LISTEN)
syslog-ng 1110 root 16u IPv4 9164 0t0 TCP *:514 (LISTEN)
syslog-ng 1110 root 17u IPv4 9165 0t0 UDP *:514
searchd 1125 sphinxsearch 7u IPv4 1686 0t0 TCP *:9306 (LISTEN)
searchd 1125 sphinxsearch 8u IPv4 1687 0t0 TCP *:9312 (LISTEN)
ossec-csy 1202 ossecm 5u IPv4 11518 0t0 UDP X.X.X.X:33877->X.X.X.X:514
/usr/sbin 1507 root 4u IPv4 12645 0t0 TCP *:443 (LISTEN)
/usr/sbin 1507 root 5u IPv4 12648 0t0 TCP *:9876 (LISTEN)
/usr/sbin 1507 root 6u IPv4 12650 0t0 TCP *:3154 (LISTEN)
/usr/sbin 1507 root 7u IPv4 12654 0t0 TCP *:444 (LISTEN)
/usr/sbin 1589 www-data 4u IPv4 12645 0t0 TCP *:443 (LISTEN)
/usr/sbin 1589 www-data 5u IPv4 12648 0t0 TCP *:9876 (LISTEN)
/usr/sbin 1589 www-data 6u IPv4 12650 0t0 TCP *:3154 (LISTEN)
/usr/sbin 1589 www-data 7u IPv4 12654 0t0 TCP *:444 (LISTEN)
/usr/sbin 1590 www-data 4u IPv4 12645 0t0 TCP *:443 (LISTEN)
/usr/sbin 1590 www-data 5u IPv4 12648 0t0 TCP *:9876 (LISTEN)
/usr/sbin 1590 www-data 6u IPv4 12650 0t0 TCP *:3154 (LISTEN)
/usr/sbin 1590 www-data 7u IPv4 12654 0t0 TCP *:444 (LISTEN)
/usr/sbin 1591 www-data 4u IPv4 12645 0t0 TCP *:443 (LISTEN)
/usr/sbin 1591 www-data 5u IPv4 12648 0t0 TCP *:9876 (LISTEN)
/usr/sbin 1591 www-data 6u IPv4 12650 0t0 TCP *:3154 (LISTEN)
/usr/sbin 1591 www-data 7u IPv4 12654 0t0 TCP *:444 (LISTEN)
/usr/sbin 1592 www-data 4u IPv4 12645 0t0 TCP *:443 (LISTEN)
/usr/sbin 1592 www-data 5u IPv4 12648 0t0 TCP *:9876 (LISTEN)
/usr/sbin 1592 www-data 6u IPv4 12650 0t0 TCP *:3154 (LISTEN)
/usr/sbin 1592 www-data 7u IPv4 12654 0t0 TCP *:444 (LISTEN)
/usr/sbin 1593 www-data 4u IPv4 12645 0t0 TCP *:443 (LISTEN)
/usr/sbin 1593 www-data 5u IPv4 12648 0t0 TCP *:9876 (LISTEN)
/usr/sbin 1593 www-data 6u IPv4 12650 0t0 TCP *:3154 (LISTEN)
/usr/sbin 1593 www-data 7u IPv4 12654 0t0 TCP *:444 (LISTEN)
ntpd 2491 ntp 16u IPv4 17730 0t0 UDP *:123
ntpd 2491 ntp 17u IPv6 17731 0t0 UDP *:123
ntpd 2491 ntp 18u IPv4 17737 0t0 UDP X.X.X.X:123
ntpd 2491 ntp 19u IPv4 17738 0t0 UDP X.X.X.X:123
ntpd 2491 ntp 20u IPv6 17739 0t0 UDP [::1]:123
ntpd 2491 ntp 21u IPv6 17740 0t0 UDP [fe80::21b:21ff:fe72:23d4]:123
bro 3899 root 4u IPv4 20663 0t0 UDP X.X.X.X:60208->X.X.X.X:53
bro 3926 root 0u IPv4 12277 0t0 TCP *:47760 (LISTEN)
bro 3926 root 1u IPv6 12278 0t0 TCP *:47760 (LISTEN)
bro 3926 root 4u IPv4 20663 0t0 UDP X.X.X.X:60208->X.X.X.X:53
tclsh 3980 root 13u IPv4 18851 0t0 TCP *:7734 (LISTEN)
tclsh 3980 root 14u IPv4 18852 0t0 TCP *:7736 (LISTEN)
tclsh 3980 root 15u IPv4 18871 0t0 TCP X.X.X.X:7736->X.X.X.X:48921 (ESTABLISHED)
tclsh 3980 root 16u IPv4 20776 0t0 TCP X.X.X.X:7736->X.X.X.X:48922 (ESTABLISHED)
tclsh 3980 root 17u IPv4 18066 0t0 TCP X.X.X.X:7736->X.X.X.X:48923 (ESTABLISHED)
tclsh 3980 root 18u IPv4 20860 0t0 TCP X.X.X.X:7736->X.X.X.X:48926 (ESTABLISHED)
tclsh 3980 root 19u IPv4 18091 0t0 TCP X.X.X.X:7736->X.X.X.X:48927 (ESTABLISHED)
tclsh 3980 root 20u IPv4 20905 0t0 TCP X.X.X.X:7736->X.X.X.X:48928 (ESTABLISHED)
tclsh 3980 root 21u IPv4 26540 0t0 TCP X.X.X.X:7736->X.X.X.X:48961 (ESTABLISHED)
tclsh 4051 root 3u IPv4 21609 0t0 TCP X.X.X.X:48923->X.X.X.X:7736 (ESTABLISHED)
tclsh 4051 root 7u IPv4 26539 0t0 TCP X.X.X.X:48961->X.X.X.X:7736 (ESTABLISHED)
tclsh 4313 root 3u IPv4 18031 0t0 TCP X.X.X.X:48921->X.X.X.X:7736 (ESTABLISHED)
tclsh 4333 root 3u IPv4 18039 0t0 TCP X.X.X.X:48922->X.X.X.X:7736 (ESTABLISHED)
tclsh 4333 root 4u IPv4 18040 0t0 TCP X.X.X.X:8001 (LISTEN)
tclsh 4427 root 3u IPv4 18081 0t0 TCP X.X.X.X:48926->X.X.X.X:7736 (ESTABLISHED)
tclsh 4445 root 3u IPv4 18090 0t0 TCP X.X.X.X:48927->X.X.X.X:7736 (ESTABLISHED)
tclsh 4485 root 3u IPv4 18113 0t0 TCP X.X.X.X:48928->X.X.X.X:7736 (ESTABLISHED)
sshd 4841 root 3u IPv4 17078 0t0 TCP X.X.X.X:22->X.X.X.X:29450 (ESTABLISHED)
sshd 4980 tle 3u IPv4 17078 0t0 TCP X.X.X.X:22->X.X.X.X:29450 (ESTABLISHED)

=========================================================================
IDS Rules Update
=========================================================================

=========================================================================
CPU Usage
=========================================================================
top - 18:59:03 up 15 min, 2 users, load average: 0.31, 0.35, 0.32
Tasks: 173 total, 2 running, 168 sleeping, 0 stopped, 3 zombie
Cpu(s): 3.5%us, 1.7%sy, 0.1%ni, 92.7%id, 1.8%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 16433044k total, 1838576k used, 14594468k free, 85112k buffers
Swap: 16774140k total, 0k used, 16774140k free, 572260k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7730 root 20 0 268m 81m 4112 S 95 0.5 0:01.48 perl
3899 root 20 0 699m 84m 68m S 16 0.5 2:11.53 bro
3926 root 25 5 410m 81m 64m R 14 0.5 1:53.77 bro
1352 root 20 0 5456 1672 652 S 2 0.0 0:08.32 ossec-syscheckd
7831 root 20 0 0 0 0 Z 2 0.0 0:00.01 perl <defunct>
1 root 20 0 24476 2380 1332 S 0 0.0 0:01.11 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.20 ksoftirqd/0
5 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:0
7 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/u:0H
8 root RT 0 0 0 0 S 0 0.0 0:00.08 migration/0
9 root 20 0 0 0 0 S 0 0.0 0:00.00 rcu_bh
10 root 20 0 0 0 0 S 0 0.0 0:00.66 rcu_sched
11 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
12 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
13 root 20 0 0 0 0 S 0 0.0 0:00.12 ksoftirqd/1
14 root RT 0 0 0 0 S 0 0.0 0:00.08 migration/1
16 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/1:0H
17 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/2
18 root 20 0 0 0 0 S 0 0.0 0:00.14 ksoftirqd/2
19 root RT 0 0 0 0 S 0 0.0 0:00.08 migration/2
20 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/2:0
21 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/2:0H
22 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/3
23 root 20 0 0 0 0 S 0 0.0 0:00.16 ksoftirqd/3
24 root RT 0 0 0 0 S 0 0.0 0:00.08 migration/3
25 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
26 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/3:0H
27 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/4
28 root 20 0 0 0 0 S 0 0.0 0:00.21 ksoftirqd/4
29 root RT 0 0 0 0 S 0 0.0 0:00.09 migration/4
30 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/4:0
31 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/4:0H
32 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/5
33 root 20 0 0 0 0 S 0 0.0 0:00.15 ksoftirqd/5
34 root RT 0 0 0 0 S 0 0.0 0:00.09 migration/5
36 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/5:0H
37 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/6
38 root 20 0 0 0 0 S 0 0.0 0:00.14 ksoftirqd/6
39 root RT 0 0 0 0 S 0 0.0 0:00.09 migration/6
40 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/6:0
41 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/6:0H
42 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/7
43 root 20 0 0 0 0 S 0 0.0 0:00.18 ksoftirqd/7
44 root RT 0 0 0 0 S 0 0.0 0:00.09 migration/7
46 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/7:0H
47 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
48 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
49 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
50 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
51 root 20 0 0 0 0 S 0 0.0 0:00.00 bdi-default
52 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
53 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
54 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
55 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
56 root 0 -20 0 0 0 S 0 0.0 0:00.00 md
57 root 0 -20 0 0 0 S 0 0.0 0:00.00 devfreq_wq
58 root 20 0 0 0 0 S 0 0.0 0:00.01 kworker/0:1
59 root 20 0 0 0 0 S 0 0.0 0:00.04 kworker/1:1
60 root 20 0 0 0 0 S 0 0.0 0:00.04 kworker/2:1
61 root 20 0 0 0 0 S 0 0.0 0:00.08 kworker/3:1
62 root 20 0 0 0 0 S 0 0.0 0:00.07 kworker/4:1
63 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/5:1
64 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/6:1
65 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/7:1
66 root 20 0 0 0 0 S 0 0.0 0:00.00 khungtaskd
67 root 20 0 0 0 0 S 0 0.0 0:00.00 kswapd0
68 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
69 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
70 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
71 root 20 0 0 0 0 S 0 0.0 0:00.00 ecryptfs-kthrea
72 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
83 root 0 -20 0 0 0 S 0 0.0 0:00.00 kthrotld
84 root 20 0 0 0 0 S 0 0.0 0:00.02 kworker/u:1
85 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
86 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
88 root 0 -20 0 0 0 S 0 0.0 0:00.00 binder
108 root 0 -20 0 0 0 S 0 0.0 0:00.00 deferwq
109 root 0 -20 0 0 0 S 0 0.0 0:00.00 charger_manager
111 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/5:2
149 root 20 0 0 0 0 S 0 0.0 0:00.01 kworker/7:2
209 root 20 0 0 0 0 S 0 0.0 0:00.00 cciss_scan
269 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
293 root 20 0 0 0 0 D 0 0.0 0:00.06 jbd2/cciss!c0d0
294 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
387 root 20 0 18032 1428 528 S 0 0.0 0:00.08 upstart-udev-br
394 root 20 0 21868 1604 808 S 0 0.0 0:00.07 udevd
525 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:2
580 root 20 0 21864 1152 352 S 0 0.0 0:00.00 udevd
585 root 20 0 21864 1160 356 S 0 0.0 0:00.00 udevd
640 root 0 -20 0 0 0 S 0 0.0 0:00.00 kpsmoused
643 root 0 -20 0 0 0 S 0 0.0 0:00.00 edac-poller
645 root 0 -20 0 0 0 S 0 0.0 0:00.00 kvm-irqfd-clean
662 messageb 20 0 23824 928 640 S 0 0.0 0:00.02 dbus-daemon
667 root 0 -20 0 0 0 S 0 0.0 0:00.00 ttm_swap
930 root 20 0 50040 2904 2296 S 0 0.0 0:00.00 sshd
967 root 20 0 15196 408 196 S 0 0.0 0:00.00 upstart-socket-
1039 root 20 0 15792 976 812 S 0 0.0 0:00.00 getty
1052 root 20 0 15792 980 812 S 0 0.0 0:00.00 getty
1057 root 20 0 0 0 0 S 0 0.0 0:00.06 flush-104:0
1062 root 20 0 15792 972 812 S 0 0.0 0:00.00 getty
1063 root 20 0 15792 968 812 S 0 0.0 0:00.00 getty
1066 root 20 0 15792 976 812 S 0 0.0 0:00.00 getty
1071 root 20 0 4336 688 556 S 0 0.0 0:00.00 acpid
1101 root 20 0 19120 1036 792 S 0 0.0 0:00.00 cron
1102 daemon 20 0 16916 376 216 S 0 0.0 0:00.00 atd
1103 mysql 20 0 1442m 74m 8356 S 0 0.5 0:04.71 mysqld
1108 root 20 0 15988 732 544 S 0 0.0 0:00.20 irqbalance
1109 root 20 0 26788 440 200 S 0 0.0 0:00.00 syslog-ng
1110 root 20 0 72188 5772 2904 S 0 0.0 0:00.18 syslog-ng
1123 sphinxse 20 0 42088 1344 1020 S 0 0.0 0:00.00 su
1125 sphinxse 20 0 323m 40m 22m S 0 0.3 0:02.64 searchd
1126 root 20 0 4408 612 508 S 0 0.0 0:00.00 sh
1128 root 20 0 205m 37m 3844 S 0 0.2 0:02.48 perl
1164 whoopsie 20 0 195m 5080 3736 S 0 0.0 0:00.02 whoopsie
1202 ossecm 20 0 12924 616 428 S 0 0.0 0:00.00 ossec-csyslogd
1211 root 20 0 12812 532 352 S 0 0.0 0:00.00 ossec-execd
1215 ossec 20 0 14512 2352 792 S 0 0.0 0:02.49 ossec-analysisd
1220 root 20 0 4536 548 408 S 0 0.0 0:00.00 ossec-logcollec
1356 ossec 20 0 13068 544 360 S 0 0.0 0:00.00 ossec-monitord
1507 root 20 0 176m 12m 6624 S 0 0.1 0:00.06 /usr/sbin/apach
1520 root 20 0 215m 2060 1776 S 0 0.0 0:00.00 PassengerWatchd
1523 root 20 0 288m 2284 2000 S 0 0.0 0:00.00 PassengerHelper
1527 root 20 0 108m 8180 2152 S 0 0.0 0:00.06 ruby1.9.1
1530 nobody 20 0 165m 4660 3636 S 0 0.0 0:00.00 PassengerLoggin
1557 root 20 0 58588 1632 1192 S 0 0.0 0:00.02 login
1589 www-data 20 0 176m 6932 672 S 0 0.0 0:00.00 /usr/sbin/apach
1590 www-data 20 0 176m 6932 672 S 0 0.0 0:00.00 /usr/sbin/apach
1591 www-data 20 0 176m 6932 672 S 0 0.0 0:00.00 /usr/sbin/apach
1592 www-data 20 0 176m 6932 672 S 0 0.0 0:00.00 /usr/sbin/apach
1593 www-data 20 0 176m 6932 672 S 0 0.0 0:00.00 /usr/sbin/apach
2491 ntp 20 0 37780 2220 1592 S 0 0.0 0:00.02 ntpd
2628 tle 20 0 27404 8696 1752 S 0 0.1 0:00.41 bash
2747 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/0:1H
2991 root 20 0 12332 1528 1296 S 0 0.0 0:00.00 bash
3921 root 20 0 0 0 0 Z 0 0.0 0:00.00 sh <defunct>
3923 root 20 0 0 0 0 Z 0 0.0 0:00.00 sh <defunct>
3980 root 20 0 124m 12m 3760 S 0 0.1 0:00.55 tclsh
4051 root 20 0 36116 5880 3016 S 0 0.0 0:00.08 tclsh
4168 root 20 0 118m 3428 764 S 0 0.0 0:00.01 tclsh
4169 root 20 0 118m 3228 548 S 0 0.0 0:00.00 tclsh
4292 sguil 20 0 95652 67m 64m S 0 0.4 0:00.10 netsniff-ng
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted
4313 root 20 0 32916 4508 2948 S 0 0.0 0:00.03 tclsh
4333 root 20 0 32840 4536 2964 S 0 0.0 0:00.03 tclsh
4335 root 20 0 4352 352 276 S 0 0.0 0:00.00 tail
4364 sguil 20 0 538m 218m 10m S 0 1.4 0:12.61 snort
4394 root 20 0 4352 356 276 S 0 0.0 0:00.00 tail
4409 sguil 20 0 25736 6864 3656 S 0 0.0 0:00.04 prads
4427 root 20 0 32400 4232 2944 S 0 0.0 0:00.02 tclsh
4429 root 20 0 4336 352 276 S 0 0.0 0:00.00 cat
4445 root 20 0 32524 4256 2944 S 0 0.0 0:00.03 tclsh
4464 sguil 20 0 111m 6280 1232 S 0 0.0 0:01.14 argus
4485 root 20 0 32532 4260 2940 S 0 0.0 0:00.03 tclsh
4487 root 20 0 4348 608 508 S 0 0.0 0:00.00 tail
4634 root 19 -1 14896 1932 304 S 0 0.0 0:00.02 dema
4841 root 20 0 77576 3632 2812 S 0 0.0 0:00.02 sshd
4980 tle 20 0 77576 1752 932 S 0 0.0 0:00.03 sshd
4981 tle 20 0 27380 8672 1752 S 0 0.1 0:00.41 bash
5102 www-data 20 0 416m 91m 3804 S 0 0.6 0:03.17 ruby
6007 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/1:0
6388 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/3:1H
6398 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/1:2
6404 root 20 0 4408 612 508 S 0 0.0 0:00.00 sh
6407 root 20 0 4408 324 220 S 0 0.0 0:00.00 sh
6412 root 20 0 4316 352 272 S 0 0.0 0:00.00 sleep
7619 root 20 0 43084 1700 1324 S 0 0.0 0:00.00 sudo
7620 root 20 0 12308 1344 1140 S 0 0.0 0:00.00 sostat-redacted
7621 root 20 0 12336 1488 1256 S 0 0.0 0:00.00 sostat
7622 root 20 0 11508 820 696 S 0 0.0 0:00.00 sed
7728 root 20 0 37980 1244 896 S 0 0.0 0:00.00 cron
7729 root 20 0 4408 608 508 S 0 0.0 0:00.00 sh
7828 root 20 0 17340 1288 916 R 0 0.0 0:00.00 top


=========================================================================
Log Archive
=========================================================================
/nsm/sensor_data/WA-TEST-IDS-eth1/dailylogs/ - 1 days
8.3G .
8.3G ./2013-11-19

/nsm/bro/logs/ - 1 days
4.4M .
4.4M ./2013-11-19
28K ./stats

=========================================================================
Bro netstats
=========================================================================
Average packet loss as percent across all Bro workers:
bro: 1384887543.554201 recvd=0 dropped=0 link=0

=========================================================================
IDS Engine (snort) packet drops
=========================================================================
/nsm/sensor_data/WA-TEST-IDS-eth1/snort-1.stats last reported pkt_drop_percent as 0.000

=========================================================================
pf_ring stats
=========================================================================
Appl. Name : <unknown>
Tot Packets : 0
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : snort-cluster-51-socket-0
Tot Packets : 0
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

=========================================================================
Netsniff-NG - Reported Packet Loss (per interval)
=========================================================================
0 Loss

=========================================================================
Sguil Uncategorized Events
=========================================================================
COUNT(*)
2555

=========================================================================
Sguil events summary for yesterday
=========================================================================
Total
0

=========================================================================
Top 50 All time Sguil Events
=========================================================================
Totals GenID:SigID Signature
929 1:2101411 GPL SNMP public access udp
415 1:2100376 GPL ICMP_INFO PING Microsoft Windows
267 1:2100538 GPL NETBIOS SMB IPC$ unicode share access
112 10000:1 PADS New Asset - unknown @snmp
107 10000:2 PADS Changed Asset - ssl TLS 1.0 Client Hello
93 1:2013410 ET POLICY Outbound MSSQL Connection to Standard port (1433)
91 10000:2 PADS Changed Asset - unknown @https
67 1:2102466 GPL NETBIOS SMB-DS IPC$ unicode share access
65 1:2012889 ET POLICY Http Client Body contains pw= in cleartext
45 1:2010935 ET POLICY Suspicious inbound to MSSQL port 1433
37 10000:1 PADS New Asset - unknown @microsoft-ds
29 1:2103003 GPL NETBIOS SMB-DS Session Setup NTMLSSP unicode asn1 overflow attempt
21 10000:1 PADS New Asset - unknown @ldap
19 1:2012296 ET VOIP Modified Sipvicious Asterisk PBX User-Agent
15 10000:2 PADS Changed Asset - smb Windows SMB
12 1:2000419 ET POLICY PE EXE or DLL Windows file download
12 10000:1 PADS New Asset - smb Windows SMB
11 10000:2 PADS Changed Asset - http Microsoft-IIS 7.5
11 10000:2 PADS Changed Asset - ssl Generic TLS 1.0 SSL
11 10000:2 PADS Changed Asset - ssl SSL 2.0 Client Hello
10 10000:2 PADS Changed Asset - unknown @microsoft-ds
7 10000:1 PADS New Asset - ssl TLS 1.0 Client Hello
7 10000:2 PADS Changed Asset - unknown @www
6 10000:1 PADS New Asset - unknown @www
5 10000:2 PADS Changed Asset - domain DNS SQR No Error
5 10000:2 PADS Changed Asset - unknown @domain
5 10000:1 PADS New Asset - unknown @https
4 1:2101390 GPL SHELLCODE x86 inc ebx NOOP
4 1:2101201 GPL WEB_SERVER 403 Forbidden
4 10000:1 PADS New Asset - unknown @ntp
4 10000:1 PADS New Asset - unknown @syslog
3 10000:2 PADS Changed Asset - http Windows-Update (Agent)
3 10000:1 PADS New Asset - ssl Generic TLS 1.0 SSL
3 10000:1 PADS New Asset - unknown @domain
3 10000:2 PADS Changed Asset - http Apache 2.2.16 (Debian)
2 10000:2 PADS Changed Asset - http Canon HTTP Client Ver3.0
2 10000:2 PADS Changed Asset - http Apache
2 1:2001329 ET POLICY RDP connection request
2 10000:1 PADS New Asset - http Microsoft WinRM Client
2 1:2002087 ET POLICY Inbound Frequent Emails - Possible Spambot Inbound
2 1:2103000 GPL NETBIOS SMB Session Setup NTMLSSP unicode asn1 overflow attempt
2 10000:1 PADS New Asset - dns TCP DNS Server
2 10000:2 PADS Changed Asset - http Microsoft Office/14.0 (Windows NT 6.1; Microsoft Outlook 14.0.6129; Pro)
2 1:2012063 ET NETBIOS Microsoft SRV2.SYS SMB Negotiate ProcessID Function Table Dereference
2 10000:2 PADS Changed Asset - http Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)
2 10000:2 PADS Changed Asset - http WhatsUp/1.0
2 1:2001330 ET POLICY RDP connection confirm
2 10000:1 PADS New Asset - http WhatsUp/1.0
1 10000:1 PADS New Asset - http Server: 4eaa
1 10000:1 PADS New Asset - http Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36
Total
2516

=========================================================================
Top 50 URLs for yesterday
=========================================================================
Total
0

=========================================================================
Snorby Events Summary for yesterday
=========================================================================
Total
0

=========================================================================
Top 50 All Time Snorby Events
=========================================================================
Totals GenID:SigID SignatureName
988 1:2101411 GPL SNMP public access udp
428 1:2100376 GPL ICMP_INFO PING Microsoft Windows
269 1:2100538 GPL NETBIOS SMB IPC$ unicode share access
196 1:2013410 ET POLICY Outbound MSSQL Connection to Standard port (1433)
69 1:2102466 GPL NETBIOS SMB-DS IPC$ unicode share access
65 1:2012889 ET POLICY Http Client Body contains pw= in cleartext
56 1:2010935 ET POLICY Suspicious inbound to MSSQL port 1433
29 1:2103003 GPL NETBIOS SMB-DS Session Setup NTMLSSP unicode asn1 overflow attempt
21 1:2012296 ET VOIP Modified Sipvicious Asterisk PBX User-Agent
12 1:2000419 ET POLICY PE EXE or DLL Windows file download
4 1:2101201 GPL WEB_SERVER 403 Forbidden
4 1:2101390 GPL SHELLCODE x86 inc ebx NOOP
2 1:2012063 ET NETBIOS Microsoft SRV2.SYS SMB Negotiate ProcessID Function Table Dereference
2 1:2103000 GPL NETBIOS SMB Session Setup NTMLSSP unicode asn1 overflow attempt
2 1:2002087 ET POLICY Inbound Frequent Emails - Possible Spambot Inbound
2 1:2001329 ET POLICY RDP connection request
2 1:2001330 ET POLICY RDP connection confirm
2 1:2001583 ET SCAN Behavioral Unusual Port 1433 traffic, Potential Scan or Infection
1 1:2009700 ET VOIP Multiple Unauthorized SIP Responses UDP
1 1:2102472 GPL NETBIOS SMB-DS C$ unicode share access
1 1:2012086 ET SHELLCODE Possible Call with No Offset TCP Shellcode
1 1:648 GPL SHELLCODE x86 NOOP
1 1:2014297 ET POLICY Vulnerable Java Version 1.7.x Detected
Total
2158

Thanh Le

unread,
Nov 19, 2013, 2:21:08 PM11/19/13
to securit...@googlegroups.com
On Tuesday, November 19, 2013 3:23:01 AM UTC-8, Doug Burks wrote:

Doug,

I also have the following three warnings when my system boots up.

WARNING: Configuration file format is too old, please update it to use the 3.3 format as some constructs might operate inefficiently;
WARNING: global: the default value of log_fifo_size() has changed to 10000 in version 3.3 to reflect log_iw_size() changes for tcp()/udp() window size changes;
* Starting SphinxSearch Daemon^[[122G[ OK ]
WARNING: The default behaviour for injecting messages in db-parser() has changed in version 3.3 from internal to pass-through, use an explicit inject-mode(internal) option $
* Starting system logging syslog-ng ^[[128G
^[[122G[ OK ]

Doug Burks

unread,
Nov 19, 2013, 4:21:18 PM11/19/13
to securit...@googlegroups.com
Observations/recommendations inline.

On Tue, Nov 19, 2013 at 2:04 PM, Thanh Le <tle...@gmail.com> wrote:
> Status: Bro
> Name Type Host Status Pid Peers Started
> bro standalone localhost running 3899 0 19 Nov 18:45:04

Looks like when you ran Setup, you selected Quick Setup. Make sure
you select Advanced Setup when you're doing a production deployment.

> Status: WA-TEST-IDS-eth1
> * netsniff-ng (full packet data)[ OK ]
> * pcap_agent (sguil)[ OK ]
> * snort_agent-1 (sguil)[ OK ]
> * snort-1 (alert data)[ OK ]
> * barnyard2-1 (spooler, unified2 format)[ FAIL ]
> * stale PID file found, process will be restarted at the next 5-minute interval!

Not sure why barnyard is showing FAIL. This shouldn't be related to
the kernel panic, but please check the Barnyard log file:
cat /var/log/nsm/WA-TEST-IDS-eth1/barnyard2-1.log

> * prads (sessions/assets)[ OK ]
> * sancp_agent (sguil)[ OK ]
> * pads_agent (sguil)[ OK ]
> * argus[ OK ]
> * http_agent (sguil)[ OK ]

When you run Advanced Setup, try disabling all these services.

You may want to try disabling all services but Snort, or all services
but Bro to see if you can pinpoint one service that's causing the
kernel panic.

Also try switching from Snort to Suricata to see if that makes any difference.

<snip>
> awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted

Based on this error, it looks like you're running an older version of
sostat. Make sure that you apply all updates.

Doug Burks

unread,
Nov 19, 2013, 4:21:58 PM11/19/13
to securit...@googlegroups.com
On Tue, Nov 19, 2013 at 2:21 PM, Thanh Le <tle...@gmail.com> wrote:
> I also have the following three warnings when my system boots up.
>
> WARNING: Configuration file format is too old, please update it to use the 3.3 format as some constructs might operate inefficiently;
> WARNING: global: the default value of log_fifo_size() has changed to 10000 in version 3.3 to reflect log_iw_size() changes for tcp()/udp() window size changes;
> * Starting SphinxSearch Daemon^[[122G[ OK ]
> WARNING: The default behaviour for injecting messages in db-parser() has changed in version 3.3 from internal to pass-through, use an explicit inject-mode(internal) option $
> * Starting system logging syslog-ng ^[[128G
> ^[[122G[ OK ]

These messages are normal and would not cause a kernel panic.

Thanh Le

unread,
Nov 19, 2013, 6:11:55 PM11/19/13
to securit...@googlegroups.com

Doug,

The service which caused the kernel panic is * snort-1 (alert data). If the service fail everything run just fine.

I don't know why that service could caused the kernel panic.

Doug Burks

unread,
Nov 19, 2013, 7:35:08 PM11/19/13
to securit...@googlegroups.com
Have you tried switching from Snort to Suricata?
> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/groups/opt_out.



Thanh Le

unread,
Nov 21, 2013, 3:51:15 PM11/21/13
to securit...@googlegroups.com

Switching from Snort to Suritcata got the same result.

Could it be I have too much network traffics and Security Onion couldn't handle it?

Thanks,

Doug Burks

unread,
Nov 21, 2013, 3:56:11 PM11/21/13
to securit...@googlegroups.com
If your system is underpowered for the amount of traffic you're
monitoring, it will drop packets, but it should not result in a kernel
panic.

How much traffic are you monitoring?

Thanh Le

unread,
Nov 21, 2013, 4:09:06 PM11/21/13
to securit...@googlegroups.com

The system has 2 quad core CPU and 16GB of RAM.

How much traffic are you monitoring? more than 2GB in 5 minutes.

Doug Burks

unread,
Nov 22, 2013, 9:03:09 AM11/22/13
to securit...@googlegroups.com
Any chance you can increase your RAM?

Please try the following:

- perform a fresh installation with our ISO image using default
partitioning (no LVM)
- install all updates and reboot
- run Setup
- choose Advanced Setup
- Enable only Bro (disable Snort/Suricata and all other sniffing processes)
- choose multiple Bro workers

Thanh Le

unread,
Nov 26, 2013, 8:26:03 PM11/26/13
to securit...@googlegroups.com

I tried your suggestions and still couldn't get Security Onion up for 10 minutes.

Doug Burks

unread,
Nov 30, 2013, 10:10:29 PM11/30/13
to securit...@googlegroups.com
If you then disable Bro (no sniffing processes running at all), is it
more stable?

Thanh Le

unread,
Dec 2, 2013, 1:00:38 PM12/2/13
to securit...@googlegroups.com

Yes, it is stable if no niffing processes running.

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/PL2CfRNqBfY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

Doug Burks

unread,
Dec 3, 2013, 9:46:12 AM12/3/13
to securit...@googlegroups.com
Here are some other things to try to see if they make any difference:

- monitor less traffic

- update all firmware on the box

- update NIC driver, disable irqbalance, set irq affiinity as described here:
http://pevma.blogspot.com/2013/12/suricata-and-grand-slam-of-open-source.html

Thanh Le

unread,
Dec 5, 2013, 7:02:33 PM12/5/13
to securit...@googlegroups.com
No go. I gave up.

Doug Burks

unread,
Dec 6, 2013, 6:50:13 AM12/6/13
to securit...@googlegroups.com
Sorry to hear that. I haven't heard of any other users who have had
chronic kernel panics like this. If you ever have time to revisit the
issue and are able to resolve, I'd appreciate any feedback.

Thanks,
Doug

On Thu, Dec 5, 2013 at 7:02 PM, Thanh Le <tle...@gmail.com> wrote:
> No go. I gave up.

Vaha

unread,
Jan 24, 2014, 3:42:44 PM1/24/14
to securit...@googlegroups.com
Hello Doug,

I work directly with Thanh and I want to take over this project. I have been reading a lot about how to set it correctly based on your suggestions.

My first process is to install a fresh copy of Ubuntu (KUbubtu 12.04.3) and install the latest updates and kernel on the system. Once this is completed I will run the advanced setup and make sure only to select the options you have given Thanh earlier.

Hopefully me and you can work on this together since I have quite an in-depth knowledge of Linux kernels and we may find the culprit on what's going on.

Thanks,

Vaha

Reply all
Reply to author
Forward
0 new messages