Re: [security-onion] ELSA - can't search the Index (but archive works fine)

Heine Lysemose

unread,

May 13, 2013, 3:49:54 PM5/13/13

to securit...@googlegroups.com

Hi

Couple of things you can look at.

Your load is pretty high, over 10.
You have 18000 rules enabled, try to bring the down.
Have you renamed the server? You have both /nsm/sensor_data/securityonion-eth4 and /nsm/sensor_data/securityonionFW-eth1 in the sostat output...

Regards,
Lysemose

On May 13, 2013 9:23 PM, "Matthew Thacker" <matthewaa...@gmail.com> wrote:

Having a problem where no matter what I try and search for in ELSA if I have "Index" selected I get this error message:
Warnings: node 127.0.0.1 got error $VAR1 = undef;

Searching the archive works fine. I've googled and seen a few problems that seem similar but none of the fixes I've found work for me. I've rebooted, rotated the indexes with 'sudo indexer --rotate --all', and restarted sphinxsearch and don't know what to try next.

Thanks!
matthew

sostat output:
=========================================================================
Service Status
=========================================================================
Status: securityonion
* sguil server[ OK ]
Status: HIDS
* ossec_agent (sguil)[ OK ]
Status: Bro
Name Type Host Status Pid Peers Started
manager manager 192.168.x.x running 4674 5 13 May 18:52:51
proxy proxy 192.168.x.x running 4964 5 13 May 18:52:54
securityonion-eth4-1 worker 192.168.x.x running 5767 2 13 May 18:52:59
securityonion-eth4-2 worker 192.168.x.x running 5765 2 13 May 18:52:59
securityonion-eth4-3 worker 192.168.x.x running 5766 2 13 May 18:52:59
securityonion-eth4-4 worker 192.168.x.x running 5768 2 13 May 18:52:59
Status: securityonion-eth4
* netsniff-ng (full packet data)[ OK ]
* pcap_agent (sguil)[ OK ]
* snort_agent-1 (sguil)[ OK ]
* snort_agent-2 (sguil)[ OK ]
* snort_agent-3 (sguil)[ OK ]
* snort_agent-4 (sguil)[ OK ]
* snort_agent-5 (sguil)[ OK ]
* snort_agent-6 (sguil)[ OK ]
* snort_agent-7 (sguil)[ OK ]
* snort-1 (alert data)[ OK ]
* snort-2 (alert data)[ OK ]
* snort-3 (alert data)[ OK ]
* snort-4 (alert data)[ OK ]
* snort-5 (alert data)[ OK ]
* snort-6 (alert data)[ OK ]
* snort-7 (alert data)[ OK ]
* barnyard2-1 (spooler, unified2 format)[ OK ]
* barnyard2-2 (spooler, unified2 format)[ OK ]
* barnyard2-3 (spooler, unified2 format)[ OK ]
* barnyard2-4 (spooler, unified2 format)[ OK ]
* barnyard2-5 (spooler, unified2 format)[ OK ]
* barnyard2-6 (spooler, unified2 format)[ OK ]
* barnyard2-7 (spooler, unified2 format)[ OK ]
* prads (sessions/assets)[ OK ]
* sancp_agent (sguil)[ OK ]
* pads_agent (sguil)[ OK ]
* argus[ OK ]
* http_agent (sguil)[ OK ]

=========================================================================
Interface Status
=========================================================================
eth0 Link encap:Ethernet HWaddr 00:1d:09:67:58:64
inet addr:192.168.x.x Bcast:192.168.x.x Mask:255.255.255.0
inet6 addr: fe80::21d:9ff:fe67:5864/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:276794 errors:0 dropped:0 overruns:0 frame:0
TX packets:1679387 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:183689691 (183.6 MB) TX bytes:2406503596 (2.4 GB)
Interrupt:16 Memory:f8000000-f8012800

eth2 Link encap:Ethernet HWaddr 00:e0:4c:68:6a:a6
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:52 Base address:0x4000

eth3 Link encap:Ethernet HWaddr 00:e0:4c:68:6a:a7
UP BROADCAST NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:53 Base address:0xe000

eth4 Link encap:Ethernet HWaddr 00:e0:4c:68:6a:a8
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:3315052 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2022202579 (2.0 GB) TX bytes:0 (0.0 B)
Interrupt:54 Base address:0x4000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:57675 errors:0 dropped:0 overruns:0 frame:0
TX packets:57675 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:117818494 (117.8 MB) TX bytes:117818494 (117.8 MB)

=========================================================================
Disk Usage
=========================================================================
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 261G 134G 115G 54% /
udev 6.9G 4.0K 6.9G 1% /dev
tmpfs 2.8G 1.1M 2.8G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 6.9G 0 6.9G 0% /run/shm
/dev/sdc1 1000G 872G 128G 88% /nsm

=========================================================================
Network Sockets
=========================================================================
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
smbd 974 root 27u IPv4 12655 0t0 TCP 192.168.x.x:445 (LISTEN)
smbd 974 root 28u IPv4 12656 0t0 TCP 192.168.x.x:139 (LISTEN)
avahi-dae 1068 avahi 12u IPv4 12559 0t0 UDP *:5353
avahi-dae 1068 avahi 13u IPv6 12560 0t0 UDP *:5353
avahi-dae 1068 avahi 14u IPv4 12561 0t0 UDP *:35999
avahi-dae 1068 avahi 15u IPv6 12562 0t0 UDP *:56972
nmbd 1148 root 9u IPv4 10505 0t0 UDP *:137
nmbd 1148 root 10u IPv4 10506 0t0 UDP *:138
nmbd 1148 root 11u IPv4 10512 0t0 UDP 192.168.x.x:137
nmbd 1148 root 12u IPv4 10513 0t0 UDP 192.168.x.x:137
nmbd 1148 root 13u IPv4 10514 0t0 UDP 192.168.x.x:138
nmbd 1148 root 14u IPv4 10515 0t0 UDP 192.168.x.x:138
iscsid 1484 root 9u IPv4 8874 0t0 TCP 192.168.x.x:45436->192.168.x.x:3260 (ESTABLISHED)
sshd 1606 root 3r IPv4 8820 0t0 TCP 192.168.x.x:22->192.168.x.x:41787 (ESTABLISHED)
sshd 1713 root 3r IPv4 8902 0t0 TCP *:22 (LISTEN)
sshd 1713 root 4u IPv6 8904 0t0 TCP *:22 (LISTEN)
sshd 2042 admin 3u IPv4 8820 0t0 TCP 192.168.x.x:22->192.168.x.x:41787 (ESTABLISHED)
sshd 2042 admin 8u IPv6 1863 0t0 TCP [::1]:50000 (LISTEN)
sshd 2042 admin 9u IPv4 1864 0t0 TCP 127.0.0.1:50000 (LISTEN)
sshd 2042 admin 10u IPv6 1867 0t0 TCP [::1]:50001 (LISTEN)
sshd 2042 admin 11u IPv4 1868 0t0 TCP 127.0.0.1:50001 (LISTEN)
sshd 2042 admin 12u IPv4 57923 0t0 TCP 127.0.0.1:36606->127.0.0.1:3306 (ESTABLISHED)
sshd 2042 admin 13u IPv4 58976 0t0 TCP 127.0.0.1:36607->127.0.0.1:3306 (ESTABLISHED)
syslog-ng 2076 root 9u IPv4 7617 0t0 TCP *:514 (LISTEN)
syslog-ng 2076 root 10u IPv4 7618 0t0 UDP *:514
mysqld 2142 mysql 10u IPv4 17476 0t0 TCP 127.0.0.1:3306 (LISTEN)
mysqld 2142 mysql 20u IPv4 55905 0t0 TCP 127.0.0.1:3306->127.0.0.1:36589 (ESTABLISHED)
mysqld 2142 mysql 30u IPv4 59621 0t0 TCP 127.0.0.1:3306->127.0.0.1:36602 (ESTABLISHED)
mysqld 2142 mysql 31u IPv4 55906 0t0 TCP 127.0.0.1:3306->127.0.0.1:36590 (ESTABLISHED)
mysqld 2142 mysql 34u IPv4 55909 0t0 TCP 127.0.0.1:3306->127.0.0.1:36592 (ESTABLISHED)
mysqld 2142 mysql 54u IPv4 53170 0t0 TCP 127.0.0.1:3306->127.0.0.1:36597 (ESTABLISHED)
mysqld 2142 mysql 129u IPv4 59577 0t0 TCP 127.0.0.1:3306->127.0.0.1:36598 (ESTABLISHED)
mysqld 2142 mysql 140u IPv4 53881 0t0 TCP 127.0.0.1:3306->127.0.0.1:36599 (ESTABLISHED)
mysqld 2142 mysql 173u IPv4 57924 0t0 TCP 127.0.0.1:3306->127.0.0.1:36606 (ESTABLISHED)
mysqld 2142 mysql 178u IPv4 57925 0t0 TCP 127.0.0.1:3306->127.0.0.1:36607 (ESTABLISHED)
searchd 2143 sphinxsearch 7u IPv4 14441 0t0 TCP *:9306 (LISTEN)
searchd 2143 sphinxsearch 8u IPv4 14442 0t0 TCP *:9312 (LISTEN)
/usr/sbin 2737 root 4u IPv4 7932 0t0 TCP *:443 (LISTEN)
/usr/sbin 2737 root 5u IPv4 7935 0t0 TCP *:9876 (LISTEN)
/usr/sbin 2737 root 6u IPv4 7937 0t0 TCP *:3154 (LISTEN)
/usr/sbin 2737 root 7u IPv4 7941 0t0 TCP *:444 (LISTEN)
/usr/sbin 2787 www-data 4u IPv4 7932 0t0 TCP *:443 (LISTEN)
/usr/sbin 2787 www-data 5u IPv4 7935 0t0 TCP *:9876 (LISTEN)
/usr/sbin 2787 www-data 6u IPv4 7937 0t0 TCP *:3154 (LISTEN)
/usr/sbin 2787 www-data 7u IPv4 7941 0t0 TCP *:444 (LISTEN)
/usr/sbin 2788 www-data 4u IPv4 7932 0t0 TCP *:443 (LISTEN)
/usr/sbin 2788 www-data 5u IPv4 7935 0t0 TCP *:9876 (LISTEN)
/usr/sbin 2788 www-data 6u IPv4 7937 0t0 TCP *:3154 (LISTEN)
/usr/sbin 2788 www-data 7u IPv4 7941 0t0 TCP *:444 (LISTEN)
/usr/sbin 2789 www-data 4u IPv4 7932 0t0 TCP *:443 (LISTEN)
/usr/sbin 2789 www-data 5u IPv4 7935 0t0 TCP *:9876 (LISTEN)
/usr/sbin 2789 www-data 6u IPv4 7937 0t0 TCP *:3154 (LISTEN)
/usr/sbin 2789 www-data 7u IPv4 7941 0t0 TCP *:444 (LISTEN)
/usr/sbin 2790 www-data 4u IPv4 7932 0t0 TCP *:443 (LISTEN)
/usr/sbin 2790 www-data 5u IPv4 7935 0t0 TCP *:9876 (LISTEN)
/usr/sbin 2790 www-data 6u IPv4 7937 0t0 TCP *:3154 (LISTEN)
/usr/sbin 2790 www-data 7u IPv4 7941 0t0 TCP *:444 (LISTEN)
/usr/sbin 2791 www-data 4u IPv4 7932 0t0 TCP *:443 (LISTEN)
/usr/sbin 2791 www-data 5u IPv4 7935 0t0 TCP *:9876 (LISTEN)
/usr/sbin 2791 www-data 6u IPv4 7937 0t0 TCP *:3154 (LISTEN)
/usr/sbin 2791 www-data 7u IPv4 7941 0t0 TCP *:444 (LISTEN)
sshd 3932 root 3r IPv4 18038 0t0 TCP 192.168.x.x:22->192.168.x.x:50436 (ESTABLISHED)
tclsh 4013 root 13u IPv4 29533 0t0 TCP *:7734 (LISTEN)
tclsh 4013 root 14u IPv4 29534 0t0 TCP *:7736 (LISTEN)
tclsh 4013 root 15u IPv4 37903 0t0 TCP 127.0.0.1:7736->127.0.0.1:37502 (ESTABLISHED)
tclsh 4013 root 16u IPv4 34754 0t0 TCP 127.0.0.1:7736->127.0.0.1:37505 (ESTABLISHED)
tclsh 4013 root 17u IPv4 31474 0t0 TCP 127.0.0.1:7736->127.0.0.1:37506 (ESTABLISHED)
tclsh 4013 root 18u IPv4 32309 0t0 TCP 127.0.0.1:7736->127.0.0.1:37507 (ESTABLISHED)
tclsh 4013 root 19u IPv4 32433 0t0 TCP 127.0.0.1:7736->127.0.0.1:37508 (ESTABLISHED)
tclsh 4013 root 20u IPv4 39086 0t0 TCP 127.0.0.1:7736->127.0.0.1:37509 (ESTABLISHED)
tclsh 4013 root 21u IPv4 35623 0t0 TCP 127.0.0.1:7736->127.0.0.1:37510 (ESTABLISHED)
tclsh 4013 root 22u IPv4 40050 0t0 TCP 127.0.0.1:7736->127.0.0.1:37511 (ESTABLISHED)
tclsh 4013 root 23u IPv4 37402 0t0 TCP 127.0.0.1:7736->127.0.0.1:37512 (ESTABLISHED)
tclsh 4013 root 24u IPv4 40155 0t0 TCP 192.168.x.x:7736->192.168.x.x:55448 (ESTABLISHED)
tclsh 4013 root 25u IPv4 40156 0t0 TCP 192.168.x.x:7736->192.168.x.x:55449 (ESTABLISHED)
tclsh 4013 root 26u IPv4 37450 0t0 TCP 192.168.x.x:7736->192.168.x.x:55450 (ESTABLISHED)
tclsh 4013 root 27u IPv4 41079 0t0 TCP 192.168.x.x:7736->192.168.x.x:55452 (ESTABLISHED)
tclsh 4013 root 28u IPv4 32762 0t0 TCP 192.168.x.x:7736->192.168.x.x:55451 (ESTABLISHED)
tclsh 4013 root 29u IPv4 42020 0t0 TCP 192.168.x.x:7736->192.168.x.x:55454 (ESTABLISHED)
tclsh 4013 root 30u IPv4 42242 0t0 TCP 127.0.0.1:7736->127.0.0.1:37513 (ESTABLISHED)
tclsh 4013 root 31u IPv4 38822 0t0 TCP 127.0.0.1:7736->127.0.0.1:37518 (ESTABLISHED)
tclsh 4013 root 34u IPv4 42344 0t0 TCP 127.0.0.1:7736->127.0.0.1:37514 (ESTABLISHED)
tclsh 4013 root 37u IPv4 53891 0t0 TCP 127.0.0.1:7736->127.0.0.1:37548 (ESTABLISHED)
tclsh 4013 root 38u IPv4 57710 0t0 TCP 127.0.0.1:7736->127.0.0.1:37551 (ESTABLISHED)
tclsh 4098 root 3u IPv4 36977 0t0 TCP 127.0.0.1:37502->127.0.0.1:7736 (ESTABLISHED)
tclsh 4098 root 7u IPv4 53180 0t0 TCP 127.0.0.1:37548->127.0.0.1:7736 (ESTABLISHED)
tclsh 4098 root 8u IPv4 53897 0t0 TCP 127.0.0.1:37551->127.0.0.1:7736 (ESTABLISHED)
sshd 4259 admin 3u IPv4 18038 0t0 TCP 192.168.x.x:22->192.168.x.x:50436 (ESTABLISHED)
bro 4674 root 4u IPv4 21865 0t0 UDP 192.168.x.x:43301->192.168.x.x:53
bro 4724 root 0u IPv4 23689 0t0 TCP *:47761 (LISTEN)
bro 4724 root 1u IPv6 23690 0t0 TCP *:47761 (LISTEN)
bro 4724 root 2u IPv4 22002 0t0 TCP 192.168.x.x:47761->192.168.x.x:33614 (ESTABLISHED)
bro 4724 root 4u IPv4 21865 0t0 UDP 192.168.x.x:43301->192.168.x.x:53
bro 4724 root 8u IPv4 23021 0t0 TCP 192.168.x.x:47761->192.168.x.x:33618 (ESTABLISHED)
bro 4724 root 10u IPv4 25072 0t0 TCP 192.168.x.x:47761->192.168.x.x:33620 (ESTABLISHED)
bro 4724 root 11u IPv4 25907 0t0 TCP 192.168.x.x:47761->192.168.x.x:33622 (ESTABLISHED)
bro 4724 root 12u IPv4 25908 0t0 TCP 192.168.x.x:47761->192.168.x.x:33623 (ESTABLISHED)
bro 4964 root 4u IPv4 22001 0t0 UDP 192.168.x.x:35293->192.168.x.x:53
bro 4990 root 0u IPv4 22672 0t0 TCP 192.168.x.x:33614->192.168.x.x:47761 (ESTABLISHED)
bro 4990 root 1u IPv4 22675 0t0 TCP *:47762 (LISTEN)
bro 4990 root 2u IPv6 22676 0t0 TCP *:47762 (LISTEN)
bro 4990 root 4u IPv4 22001 0t0 UDP 192.168.x.x:35293->192.168.x.x:53
bro 4990 root 7u IPv4 25900 0t0 TCP 192.168.x.x:47762->192.168.x.x:41749 (ESTABLISHED)
bro 4990 root 9u IPv4 21450 0t0 TCP 192.168.x.x:47762->192.168.x.x:41751 (ESTABLISHED)
bro 4990 root 10u IPv4 19400 0t0 TCP 192.168.x.x:47762->192.168.x.x:41753 (ESTABLISHED)
bro 4990 root 11u IPv4 22130 0t0 TCP 192.168.x.x:47762->192.168.x.x:41756 (ESTABLISHED)
bro 5765 root 4u IPv4 22876 0t0 UDP 192.168.x.x:58629->192.168.x.x:53
bro 5766 root 4u IPv4 22086 0t0 UDP 192.168.x.x:50433->192.168.x.x:53
bro 5767 root 4u IPv4 22875 0t0 UDP 192.168.x.x:58509->192.168.x.x:53
bro 5768 root 4u IPv4 25790 0t0 UDP 192.168.x.x:39316->192.168.x.x:53
ntpd 5823 ntp 16u IPv4 25826 0t0 UDP *:123
ntpd 5823 ntp 17u IPv6 25827 0t0 UDP *:123
ntpd 5823 ntp 18u IPv4 25833 0t0 UDP 127.0.0.1:123
ntpd 5823 ntp 19u IPv4 25834 0t0 UDP 192.168.x.x:123
ntpd 5823 ntp 20u IPv6 25835 0t0 UDP [fe80::21d:9ff:fe67:5864]:123
ntpd 5823 ntp 21u IPv6 25836 0t0 UDP [::1]:123
bro 5911 root 0u IPv4 23019 0t0 TCP 192.168.x.x:41749->192.168.x.x:47762 (ESTABLISHED)
bro 5911 root 1u IPv4 23020 0t0 TCP 192.168.x.x:33618->192.168.x.x:47761 (ESTABLISHED)
bro 5911 root 2u IPv4 23024 0t0 TCP *:47763 (LISTEN)
bro 5911 root 4u IPv4 22875 0t0 UDP 192.168.x.x:58509->192.168.x.x:53
bro 5911 root 8u IPv6 23025 0t0 TCP *:47763 (LISTEN)
bro 5919 root 0u IPv4 25901 0t0 TCP 192.168.x.x:41751->192.168.x.x:47762 (ESTABLISHED)
bro 5919 root 1u IPv4 25902 0t0 TCP 192.168.x.x:33620->192.168.x.x:47761 (ESTABLISHED)
bro 5919 root 2u IPv4 25905 0t0 TCP *:47766 (LISTEN)
bro 5919 root 4u IPv4 25790 0t0 UDP 192.168.x.x:39316->192.168.x.x:53
bro 5919 root 8u IPv6 25906 0t0 TCP *:47766 (LISTEN)
bro 5925 root 0u IPv4 22124 0t0 TCP 192.168.x.x:41753->192.168.x.x:47762 (ESTABLISHED)
bro 5925 root 1u IPv4 22125 0t0 TCP 192.168.x.x:33622->192.168.x.x:47761 (ESTABLISHED)
bro 5925 root 2u IPv4 22128 0t0 TCP *:47765 (LISTEN)
bro 5925 root 4u IPv4 22086 0t0 UDP 192.168.x.x:50433->192.168.x.x:53
bro 5925 root 8u IPv6 22129 0t0 TCP *:47765 (LISTEN)
bro 5927 root 0u IPv4 23026 0t0 TCP 192.168.x.x:33623->192.168.x.x:47761 (ESTABLISHED)
bro 5927 root 1u IPv4 23027 0t0 TCP 192.168.x.x:41756->192.168.x.x:47762 (ESTABLISHED)
bro 5927 root 2u IPv4 23030 0t0 TCP *:47764 (LISTEN)
bro 5927 root 4u IPv4 22876 0t0 UDP 192.168.x.x:58629->192.168.x.x:53
bro 5927 root 8u IPv6 23031 0t0 TCP *:47764 (LISTEN)
tclsh 6212 root 3u IPv4 33524 0t0 TCP 127.0.0.1:37505->127.0.0.1:7736 (ESTABLISHED)
tclsh 6525 root 3u IPv4 28849 0t0 TCP 127.0.0.1:8001 (LISTEN)
tclsh 6525 root 5u IPv4 34770 0t0 TCP 127.0.0.1:37506->127.0.0.1:7736 (ESTABLISHED)
tclsh 6525 root 7u IPv4 58504 0t0 TCP 127.0.0.1:8001->127.0.0.1:43629 (ESTABLISHED)
tclsh 6859 root 3u IPv4 26160 0t0 TCP 127.0.0.1:8002 (LISTEN)
tclsh 6859 root 5u IPv4 35487 0t0 TCP 127.0.0.1:37507->127.0.0.1:7736 (ESTABLISHED)
tclsh 6859 root 7u IPv4 58467 0t0 TCP 127.0.0.1:8002->127.0.0.1:52037 (ESTABLISHED)
tclsh 7178 root 3u IPv4 26279 0t0 TCP 127.0.0.1:8003 (LISTEN)
tclsh 7178 root 5u IPv4 31512 0t0 TCP 127.0.0.1:37508->127.0.0.1:7736 (ESTABLISHED)
tclsh 7178 root 7u IPv4 56829 0t0 TCP 127.0.0.1:8003->127.0.0.1:38682 (ESTABLISHED)
tclsh 7496 root 3u IPv4 27483 0t0 TCP 127.0.0.1:8004 (LISTEN)
tclsh 7496 root 5u IPv4 39085 0t0 TCP 127.0.0.1:37509->127.0.0.1:7736 (ESTABLISHED)
tclsh 7496 root 7u IPv4 51134 0t0 TCP 127.0.0.1:8004->127.0.0.1:44003 (ESTABLISHED)
tclsh 7845 root 3u IPv4 27555 0t0 TCP 127.0.0.1:8005 (LISTEN)
tclsh 7845 root 5u IPv4 35622 0t0 TCP 127.0.0.1:37510->127.0.0.1:7736 (ESTABLISHED)
tclsh 7845 root 7u IPv4 58431 0t0 TCP 127.0.0.1:8005->127.0.0.1:53389 (ESTABLISHED)
tclsh 8210 root 3u IPv4 27624 0t0 TCP 127.0.0.1:8006 (LISTEN)
tclsh 8210 root 5u IPv4 39257 0t0 TCP 127.0.0.1:37511->127.0.0.1:7736 (ESTABLISHED)
tclsh 8210 root 7u IPv4 51136 0t0 TCP 127.0.0.1:8006->127.0.0.1:60906 (ESTABLISHED)
tclsh 8563 root 3u IPv4 31979 0t0 TCP 127.0.0.1:8007 (LISTEN)
tclsh 8563 root 5u IPv4 35662 0t0 TCP 127.0.0.1:37512->127.0.0.1:7736 (ESTABLISHED)
tclsh 8563 root 7u IPv4 56771 0t0 TCP 127.0.0.1:8007->127.0.0.1:56413 (ESTABLISHED)
/usr/sbin 9351 www-data 4u IPv4 7932 0t0 TCP *:443 (LISTEN)
/usr/sbin 9351 www-data 5u IPv4 7935 0t0 TCP *:9876 (LISTEN)
/usr/sbin 9351 www-data 6u IPv4 7937 0t0 TCP *:3154 (LISTEN)
/usr/sbin 9351 www-data 7u IPv4 7941 0t0 TCP *:444 (LISTEN)
barnyard2 10947 root 3u IPv4 54742 0t0 TCP 127.0.0.1:43629->127.0.0.1:8001 (ESTABLISHED)
barnyard2 10947 root 4u IPv4 53880 0t0 TCP 127.0.0.1:36599->127.0.0.1:3306 (ESTABLISHED)
barnyard2 11155 root 3u IPv4 58466 0t0 TCP 127.0.0.1:52037->127.0.0.1:8002 (ESTABLISHED)
barnyard2 11155 root 4u IPv4 59576 0t0 TCP 127.0.0.1:36598->127.0.0.1:3306 (ESTABLISHED)
barnyard2 11343 root 3u IPv4 56828 0t0 TCP 127.0.0.1:38682->127.0.0.1:8003 (ESTABLISHED)
barnyard2 11343 root 4u IPv4 56848 0t0 TCP 127.0.0.1:36602->127.0.0.1:3306 (ESTABLISHED)
barnyard2 11581 root 3u IPv4 51133 0t0 TCP 127.0.0.1:44003->127.0.0.1:8004 (ESTABLISHED)
barnyard2 11581 root 4u IPv4 53043 0t0 TCP 127.0.0.1:36592->127.0.0.1:3306 (ESTABLISHED)
barnyard2 11815 root 3u IPv4 58430 0t0 TCP 127.0.0.1:53389->127.0.0.1:8005 (ESTABLISHED)
barnyard2 11815 root 4u IPv4 58443 0t0 TCP 127.0.0.1:36590->127.0.0.1:3306 (ESTABLISHED)
barnyard2 12042 root 3u IPv4 51135 0t0 TCP 127.0.0.1:60906->127.0.0.1:8006 (ESTABLISHED)
barnyard2 12042 root 4u IPv4 58440 0t0 TCP 127.0.0.1:36589->127.0.0.1:3306 (ESTABLISHED)
barnyard2 12244 root 3u IPv4 59395 0t0 TCP 127.0.0.1:56413->127.0.0.1:8007 (ESTABLISHED)
barnyard2 12244 root 4u IPv4 54817 0t0 TCP 127.0.0.1:36597->127.0.0.1:3306 (ESTABLISHED)
tclsh 12690 root 3u IPv4 38484 0t0 TCP 127.0.0.1:37513->127.0.0.1:7736 (ESTABLISHED)
/usr/sbin 12811 www-data 4u IPv4 7932 0t0 TCP *:443 (LISTEN)
/usr/sbin 12811 www-data 5u IPv4 7935 0t0 TCP *:9876 (LISTEN)
/usr/sbin 12811 www-data 6u IPv4 7937 0t0 TCP *:3154 (LISTEN)
/usr/sbin 12811 www-data 7u IPv4 7941 0t0 TCP *:444 (LISTEN)
tclsh 12966 root 3u IPv4 41218 0t0 TCP 127.0.0.1:37514->127.0.0.1:7736 (ESTABLISHED)
tclsh 13498 root 3u IPv4 45083 0t0 TCP 127.0.0.1:37518->127.0.0.1:7736 (ESTABLISHED)
/usr/sbin 15951 www-data 4u IPv4 7932 0t0 TCP *:443 (LISTEN)
/usr/sbin 15951 www-data 5u IPv4 7935 0t0 TCP *:9876 (LISTEN)
/usr/sbin 15951 www-data 6u IPv4 7937 0t0 TCP *:3154 (LISTEN)
/usr/sbin 15951 www-data 7u IPv4 7941 0t0 TCP *:444 (LISTEN)

=========================================================================
IDS Rules Update
=========================================================================
Mon May 13 13:01:01 UTC 2013
Backing up current downloaded.rules file before it gets overwritten.
Cleaning up downloaded.rules backup files older than 30 days.
Running PulledPork.
http://code.google.com/p/pulledpork/
_____ ____
`----,\ )
`--==\\ / PulledPork v0.6.1 the Smoking Pig <////~
`--==\\/
.-~~~~-.Y|\\_ Copyright (C) 2009-2011 JJ Cummings
@_/ / 66\_ cumm...@gmail.com
| \ \ _(")
\ /-| ||'--' Rules give me wings!
\_\ \_\\
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Checking latest MD5 for snortrules-snapshot-2941.tar.gz....
They Match
Done!
Prepping rules from snortrules-snapshot-2941.tar.gz for work....
Done!
Checking latest MD5 for emerging.rules.tar.gz....
They Match
Done!
Prepping rules from emerging.rules.tar.gz for work....
Done!
Reading rules...
Generating Stub Rules....
Done
Reading rules...
Reading rules...
Reading rules...
Processing /etc/nsm/pulledpork/enablesid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/dropsid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/disablesid.conf....
Modified 10 rules
Done
Modifying Sids....
Done!
Setting Flowbit State....
Enabled 47 flowbits
Done
Writing /etc/nsm/rules/downloaded.rules....
Done
Writing /etc/nsm/rules/so_rules.rules....
Done
Generating sid-msg.map....
Done
Writing /etc/nsm/rules/sid-msg.map....
Done
Writing /var/log/sid_changes.log....
Done
Rule Stats....
New:-------1
Deleted:---1
Enabled Rules:----18258
Dropped Rules:----0
Disabled Rules:---16453
Total Rules:------34711
Done
Please review /var/log/sid_changes.log for additional details
Fly Piggy Fly!
Restarting Barnyard2.
Restarting: securityonion-eth4
* stopping: barnyard2-1 (spooler, unified2 format)[ OK ]
* starting: barnyard2-1 (spooler, unified2 format)[ OK ]
* stopping: barnyard2-2 (spooler, unified2 format)[ OK ]
* starting: barnyard2-2 (spooler, unified2 format)[ OK ]
* stopping: barnyard2-3 (spooler, unified2 format)[ OK ]
* starting: barnyard2-3 (spooler, unified2 format)[ OK ]
* stopping: barnyard2-4 (spooler, unified2 format)[ OK ]
* starting: barnyard2-4 (spooler, unified2 format)[ OK ]
* stopping: barnyard2-5 (spooler, unified2 format)[ OK ]
* starting: barnyard2-5 (spooler, unified2 format)[ OK ]
* stopping: barnyard2-6 (spooler, unified2 format)[ OK ]
* starting: barnyard2-6 (spooler, unified2 format)[ OK ]
* stopping: barnyard2-7 (spooler, unified2 format)[ OK ]
* starting: barnyard2-7 (spooler, unified2 format)[ OK ]
Restarting IDS Engine.
Restarting: securityonion-eth4
* stopping: snort-1 (alert data)[ OK ]
* starting: snort-1 (alert data)[ OK ]
* stopping: snort-2 (alert data)[ OK ]
* starting: snort-2 (alert data)[ OK ]
* stopping: snort-3 (alert data)[ OK ]
* starting: snort-3 (alert data)[ OK ]
* stopping: snort-4 (alert data)[ OK ]
* starting: snort-4 (alert data)[ OK ]
* stopping: snort-5 (alert data)[ OK ]
* starting: snort-5 (alert data)[ OK ]
* stopping: snort-6 (alert data) (not running)[ WARN ]
- stale PID file found, deleting!
* starting: snort-6 (alert data)[ OK ]
* stopping: snort-7 (alert data)[ OK ]
* starting: snort-7 (alert data)[ OK ]

=========================================================================
CPU Usage
=========================================================================
top - 19:13:35 up 22 min, 1 user, load average: 12.84, 13.15, 10.53
Tasks: 246 total, 12 running, 228 sleeping, 0 stopped, 6 zombie
Cpu(s): 63.0%us, 17.2%sy, 1.6%ni, 15.0%id, 2.5%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 14366316k total, 12097944k used, 2268372k free, 197884k buffers
Swap: 21857072k total, 0k used, 21857072k free, 6178488k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12042 root 20 0 241m 142m 1776 R 72 1.0 10:06.99 barnyard2
11343 root 20 0 241m 142m 1776 R 65 1.0 10:24.55 barnyard2
10947 root 20 0 241m 142m 1776 R 49 1.0 10:00.84 barnyard2
12244 root 20 0 241m 142m 1776 R 47 1.0 9:51.28 barnyard2
11815 root 20 0 241m 142m 1776 R 42 1.0 10:15.80 barnyard2
11581 root 20 0 241m 142m 1776 R 40 1.0 10:22.11 barnyard2
11155 root 20 0 241m 142m 1776 R 29 1.0 10:03.88 barnyard2
4013 root 20 0 135m 22m 3868 R 24 0.2 1:34.56 tclsh
12966 root 20 0 34172 6144 3068 S 22 0.0 0:05.77 tclsh
4724 root 25 5 143m 20m 928 R 20 0.1 3:56.48 bro
5765 root 20 0 229m 119m 75m S 18 0.9 3:56.18 bro
5767 root 20 0 300m 119m 75m S 18 0.8 3:57.55 bro
4990 root 25 5 72868 19m 936 S 16 0.1 3:55.43 bro
5766 root 20 0 227m 118m 75m S 16 0.8 3:56.86 bro
5768 root 20 0 299m 118m 75m S 16 0.8 3:53.05 bro
5919 root 25 5 129m 84m 64m S 14 0.6 2:44.44 bro
5925 root 25 5 129m 84m 64m S 14 0.6 2:44.39 bro
5927 root 25 5 129m 84m 64m S 14 0.6 2:44.75 bro
2142 mysql 20 0 2790m 274m 8276 S 13 2.0 9:07.30 mysqld
5911 root 25 5 129m 84m 64m R 13 0.6 2:44.06 bro
2604 root 20 0 200m 36m 3624 R 5 0.3 0:03.09 perl
13202 sguil 20 0 111m 14m 1148 S 5 0.1 0:41.02 argus
2327 root 20 0 0 0 0 S 4 0.0 0:01.72 flush-8:32
3616 root 20 0 17468 1340 896 R 4 0.0 0:00.02 top
12461 sguil 20 0 28632 9.8m 3780 S 4 0.1 0:49.04 prads
2326 lightdm 20 0 244m 13m 10m S 2 0.1 0:03.58 lightdm-gtk-gre
2916 root 20 0 0 0 0 S 2 0.0 0:00.01 kworker/1:0
4964 root 20 0 282m 27m 4008 S 2 0.2 0:18.98 bro
6006 sguil 20 0 281m 255m 239m D 2 1.8 0:20.88 netsniff-ng
10244 sguil 20 0 711m 339m 11m S 2 2.4 0:47.29 snort
10502 sguil 20 0 711m 339m 11m S 2 2.4 0:50.02 snort
10734 sguil 20 0 712m 340m 11m S 2 2.4 0:45.17 snort
18723 root 20 0 0 0 0 S 2 0.0 0:01.03 kworker/u:2
1 root 20 0 24680 2632 1372 S 0 0.0 0:05.06 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.27 ksoftirqd/0
6 root RT 0 0 0 0 S 0 0.0 0:00.05 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
8 root RT 0 0 0 0 S 0 0.0 0:00.08 migration/1
10 root 20 0 0 0 0 S 0 0.0 0:00.24 ksoftirqd/1
12 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
13 root RT 0 0 0 0 S 0 0.0 0:00.20 migration/2
15 root 20 0 0 0 0 S 0 0.0 0:00.24 ksoftirqd/2
16 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/2
17 root RT 0 0 0 0 S 0 0.0 0:00.06 migration/3
19 root 20 0 0 0 0 S 0 0.0 0:00.23 ksoftirqd/3
20 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/3
21 root RT 0 0 0 0 S 0 0.0 0:00.04 migration/4
23 root 20 0 0 0 0 S 0 0.0 0:00.26 ksoftirqd/4
24 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/4
25 root RT 0 0 0 0 S 0 0.0 0:00.05 migration/5
27 root 20 0 0 0 0 S 0 0.0 0:00.19 ksoftirqd/5
28 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/5
29 root RT 0 0 0 0 S 0 0.0 0:00.06 migration/6
31 root 20 0 0 0 0 S 0 0.0 0:00.30 ksoftirqd/6
32 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/6
33 root RT 0 0 0 0 S 0 0.0 0:00.06 migration/7
35 root 20 0 0 0 0 S 0 0.0 0:00.22 ksoftirqd/7
36 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/7
37 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
38 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
39 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
40 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
42 root 20 0 0 0 0 S 0 0.0 0:00.00 sync_supers
43 root 20 0 0 0 0 S 0 0.0 0:00.00 bdi-default
44 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
45 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
46 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
47 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
48 root 0 -20 0 0 0 S 0 0.0 0:00.00 md
50 root 20 0 0 0 0 S 0 0.0 0:00.17 kworker/2:1
51 root 20 0 0 0 0 S 0 0.0 0:00.03 kworker/3:1
52 root 20 0 0 0 0 S 0 0.0 0:00.36 kworker/4:1
53 root 20 0 0 0 0 S 0 0.0 0:00.32 kworker/5:1
55 root 20 0 0 0 0 S 0 0.0 0:00.01 kworker/7:1
56 root 20 0 0 0 0 S 0 0.0 0:00.00 khungtaskd
57 root 20 0 0 0 0 S 0 0.0 0:00.00 kswapd0
58 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
59 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
60 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
61 root 20 0 0 0 0 S 0 0.0 0:00.00 ecryptfs-kthrea
62 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
70 root 0 -20 0 0 0 S 0 0.0 0:00.00 kthrotld
71 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
72 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
93 root 0 -20 0 0 0 S 0 0.0 0:00.00 devfreq_wq
94 root 20 0 0 0 0 S 0 0.0 0:00.54 kworker/0:2
153 root 20 0 0 0 0 S 0 0.0 0:00.09 kworker/7:2
203 root 20 0 0 0 0 S 0 0.0 0:00.20 kworker/2:2
229 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
236 root 20 0 0 0 0 S 0 0.0 0:00.16 kworker/6:2
272 root 0 -20 0 0 0 S 0 0.0 0:00.00 ttm_swap
296 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_3
297 root 20 0 0 0 0 S 0 0.0 0:00.11 usb-storage
298 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_4
299 root 20 0 0 0 0 S 0 0.0 0:00.08 usb-storage
371 root 20 0 0 0 0 S 0 0.0 0:00.36 kworker/5:2
379 root 20 0 0 0 0 S 0 0.0 0:06.78 jbd2/sda1-8
380 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
547 root 20 0 17760 1160 532 S 0 0.0 0:00.13 upstart-udev-br
596 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfs_mru_cache
597 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfslogd
598 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfsdatad
599 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfsconvertd
600 root 20 0 21912 1748 824 S 0 0.0 0:00.13 udevd
870 root 0 -20 0 0 0 S 0 0.0 0:00.00 kmpathd
871 root 0 -20 0 0 0 S 0 0.0 0:00.00 kmpath_handlerd
874 root 0 -20 0 0 0 S 0 0.0 0:00.00 kpsmoused
973 root 0 -20 0 0 0 S 0 0.0 0:00.00 edac-poller
974 root 20 0 119m 5744 4632 S 0 0.0 0:00.02 smbd
1003 messageb 20 0 24268 1416 800 S 0 0.0 0:00.10 dbus-daemon
1008 root 20 0 78400 2544 1816 S 0 0.0 0:00.02 sudo
1038 root 20 0 28144 5144 1692 S 0 0.0 0:00.33 bash
1068 avahi 20 0 32312 1516 1244 S 0 0.0 0:00.02 avahi-daemon
1069 avahi 20 0 32180 468 216 S 0 0.0 0:00.00 avahi-daemon
1090 root 20 0 101m 4012 2996 S 0 0.0 0:00.01 cupsd
1096 root 20 0 21188 1708 1428 S 0 0.0 0:00.00 bluetoothd
1125 root 10 -10 0 0 0 S 0 0.0 0:00.00 krfcommd
1142 root 20 0 0 0 0 S 0 0.0 0:00.74 flush-8:0
1148 root 20 0 91260 2064 1268 S 0 0.0 0:00.26 nmbd
1150 root 20 0 0 0 0 S 0 0.0 0:00.08 kworker/3:2
1333 root 0 -20 0 0 0 S 0 0.0 0:00.00 iscsi_eh
1341 root 0 -20 0 0 0 S 0 0.0 0:00.00 ib_addr
1371 root 0 -20 0 0 0 S 0 0.0 0:00.00 ib_mcast
1372 root 0 -20 0 0 0 S 0 0.0 0:00.00 iw_cm_wq
1373 root 0 -20 0 0 0 S 0 0.0 0:00.00 ib_cm
1440 root 0 -20 0 0 0 S 0 0.0 0:00.00 rdma_cm
1484 root 10 -10 13700 3492 2336 S 0 0.0 0:00.12 iscsid
1540 root 20 0 15188 392 192 S 0 0.0 0:00.00 upstart-socket-
1554 root 20 0 119m 1644 532 S 0 0.0 0:00.00 smbd
1606 root 20 0 101m 4368 3328 S 0 0.0 0:00.01 sshd
1685 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_5
1686 root 0 -20 0 0 0 S 0 0.0 0:00.00 iscsi_q_5
1687 root 0 -20 0 0 0 S 0 0.0 0:00.00 scsi_wq_5
1688 root 20 0 22220 1604 388 S 0 0.0 0:00.00 udevd
1690 root 20 0 21908 1300 376 S 0 0.0 0:00.00 udevd
1713 root 20 0 50032 2912 2304 S 0 0.0 0:00.01 sshd
1723 root 20 0 0 0 0 S 0 0.0 0:00.08 xfsbufd/sdc1
1725 root 20 0 0 0 0 S 0 0.0 0:00.63 xfsaild/sdc1
1763 root 20 0 4090m 3948 2840 S 0 0.0 0:00.06 console-kit-dae
1830 root 20 0 207m 4828 3624 S 0 0.0 0:00.05 polkitd
1837 root 20 0 4400 612 508 S 0 0.0 0:00.00 sh
1840 root 20 0 4400 324 220 S 0 0.0 0:00.00 sh
1845 root 20 0 4308 352 276 S 0 0.0 0:00.00 sleep
2042 admi 20 0 108m 8812 932 S 0 0.1 0:00.74 sshd
2044 root 20 0 20024 968 804 S 0 0.0 0:00.00 getty
2050 root 20 0 20024 968 804 S 0 0.0 0:00.00 getty
2068 root 20 0 20024 964 804 S 0 0.0 0:00.00 getty
2069 root 20 0 20024 964 804 S 0 0.0 0:00.00 getty
2072 root 20 0 20024 964 804 S 0 0.0 0:00.00 getty
2075 root 20 0 26780 436 200 S 0 0.0 0:00.00 syslog-ng
2076 root 20 0 89256 18m 2984 S 0 0.1 0:06.30 syslog-ng
2086 root 20 0 4460 812 556 S 0 0.0 0:00.00 acpid
2089 root 20 0 19112 1024 780 S 0 0.0 0:00.01 cron
2090 daemon 20 0 16908 376 216 S 0 0.0 0:00.00 atd
2097 root 20 0 15980 712 528 S 0 0.0 0:00.30 irqbalance
2107 sphinxse 20 0 72916 2040 1468 S 0 0.0 0:00.00 su
2108 root 20 0 280m 4276 3508 S 0 0.0 0:00.02 lightdm
2120 root 20 0 202m 19m 9852 S 0 0.1 0:02.38 Xorg
2143 sphinxse 20 0 447m 160m 124m S 0 1.1 0:53.39 searchd
2211 mail 20 0 12588 648 464 S 0 0.0 0:00.00 nullmailer-send
2252 root 20 0 185m 4720 3720 S 0 0.0 0:00.02 lightdm
2257 root 20 0 132m 4328 3660 S 0 0.0 0:00.05 accounts-daemon
2260 root 20 0 12804 540 352 S 0 0.0 0:00.00 ossec-execd
2277 ossec 20 0 14508 2368 796 S 0 0.0 0:09.45 ossec-analysisd
2294 root 20 0 4528 532 400 S 0 0.0 0:00.00 ossec-logcollec
2320 lightdm 20 0 4400 616 508 S 0 0.0 0:00.00 lightdm-greeter
2325 lightdm 20 0 23952 688 448 S 0 0.0 0:00.00 dbus-daemon
2335 root 20 0 5852 2100 644 S 0 0.0 0:32.50 ossec-syscheckd
2339 ossec 20 0 13060 548 364 S 0 0.0 0:00.00 ossec-monitord
2353 lightdm 20 0 52420 2388 1992 S 0 0.0 0:00.00 gvfsd
2355 lightdm 20 0 215m 3612 2996 S 0 0.0 0:00.00 gvfs-fuse-daemo
2370 root 20 0 214m 4300 3340 S 0 0.0 0:00.10 upowerd
2442 root 20 0 94656 2588 1900 S 0 0.0 0:00.00 lightdm
2603 root 20 0 4400 612 512 S 0 0.0 0:00.00 sh
2737 root 20 0 176m 12m 6604 S 0 0.1 0:00.18 /usr/sbin/apach
2743 root 20 0 215m 1940 1684 S 0 0.0 0:00.00 PassengerWatchd
2746 root 20 0 288m 2296 2004 S 0 0.0 0:00.02 PassengerHelper
2748 root 20 0 108m 8184 2148 S 0 0.1 0:00.09 ruby1.9.1
2752 nobody 20 0 165m 4664 3640 S 0 0.0 0:00.01 PassengerLoggin
2787 www-data 20 0 373m 100m 6040 S 0 0.7 0:03.75 /usr/sbin/apach
2788 www-data 20 0 373m 100m 5820 S 0 0.7 0:03.50 /usr/sbin/apach
2789 www-data 20 0 373m 100m 5832 S 0 0.7 0:03.39 /usr/sbin/apach
2790 www-data 20 0 373m 100m 5820 S 0 0.7 0:03.98 /usr/sbin/apach
2791 www-data 20 0 176m 7636 1304 S 0 0.1 0:00.00 /usr/sbin/apach
2805 root 20 0 20024 968 800 S 0 0.0 0:00.00 getty
3018 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/6:1
3134 root 20 0 78148 2368 1772 S 0 0.0 0:00.00 sudo
3135 root 20 0 16556 1472 1248 S 0 0.0 0:00.00 sostat
3932 root 20 0 101m 4404 3352 S 0 0.0 0:00.03 sshd
4098 root 20 0 40196 7528 2796 S 0 0.1 0:00.07 tclsh
4259 admi 20 0 101m 1992 940 S 0 0.0 0:00.03 sshd
4262 admi 20 0 31960 8956 1688 S 0 0.1 0:00.56 bash
4653 root 20 0 17884 1588 1312 S 0 0.0 0:00.00 bash
4674 root 20 0 1792m 25m 4004 S 0 0.2 0:25.14 bro
4912 root 20 0 17884 1592 1312 S 0 0.0 0:00.00 bash
4941 root 20 0 120m 5320 888 S 0 0.0 0:00.09 tclsh
4942 root 20 0 120m 5160 728 S 0 0.0 0:00.00 tclsh
5694 root 20 0 17884 1592 1312 S 0 0.0 0:00.00 bash
5697 root 20 0 17884 1592 1312 S 0 0.0 0:00.00 bash
5699 root 20 0 17884 1592 1312 S 0 0.0 0:00.00 bash
5701 root 20 0 17884 1592 1312 S 0 0.0 0:00.00 bash
5823 ntp 20 0 37772 2244 1616 S 0 0.0 0:00.10 ntpd
6212 root 20 0 33500 5168 3024 S 0 0.0 0:00.07 tclsh
6525 root 20 0 33456 5212 3032 S 0 0.0 0:00.06 tclsh
6542 root 20 0 4344 360 280 S 0 0.0 0:00.00 tail
6859 root 20 0 33456 5212 3032 S 0 0.0 0:00.07 tclsh
6879 root 20 0 4344 356 280 S 0 0.0 0:00.00 tail
7178 root 20 0 33456 5208 3032 S 0 0.0 0:00.06 tclsh
7191 root 20 0 4344 360 280 S 0 0.0 0:00.00 tail
7496 root 20 0 33460 5212 3032 S 0 0.0 0:00.06 tclsh
7543 root 20 0 4344 356 280 S 0 0.0 0:00.00 tail
7845 root 20 0 33456 5212 3032 S 0 0.0 0:00.06 tclsh
7904 root 20 0 4344 360 280 S 0 0.0 0:00.00 tail
8210 root 20 0 33456 5212 3032 S 0 0.0 0:00.06 tclsh
8255 root 20 0 4344 360 280 S 0 0.0 0:00.00 tail
8563 root 20 0 33460 5212 3032 S 0 0.0 0:00.06 tclsh
8610 root 20 0 4344 360 280 S 0 0.0 0:00.00 tail
8996 sguil 20 0 711m 339m 11m S 0 2.4 0:43.31 snort
9351 www-data 20 0 176m 7636 1304 S 0 0.1 0:00.00 /usr/sbin/apach
9360 sguil 20 0 711m 339m 11m S 0 2.4 0:46.73 snort
9668 sguil 20 0 711m 338m 11m S 0 2.4 0:47.63 snort
9972 sguil 20 0 712m 339m 11m S 0 2.4 0:46.41 snort
10458 www-data 20 0 0 0 0 Z 0 0.0 0:00.04 /usr/sbin/apach <defunct>
10771 root 20 0 4344 356 280 S 0 0.0 0:00.00 tail
11304 root 20 0 0 0 0 S 0 0.0 0:00.41 kworker/4:2
12690 root 20 0 33048 4928 3016 S 0 0.0 0:00.59 tclsh
12756 root 20 0 4328 360 280 S 0 0.0 0:00.02 cat
12811 www-data 20 0 176m 6920 660 S 0 0.0 0:00.00 /usr/sbin/apach
13466 www-data 20 0 0 0 0 Z 0 0.0 0:00.03 /usr/sbin/apach <defunct>
13471 www-data 20 0 0 0 0 Z 0 0.0 0:00.03 /usr/sbin/apach <defunct>
13498 root 20 0 33588 5416 3028 S 0 0.0 0:35.18 tclsh
13633 root 20 0 4344 608 508 S 0 0.0 0:00.08 tail
15426 www-data 20 0 0 0 0 Z 0 0.0 0:00.04 /usr/sbin/apach <defunct>
15951 www-data 20 0 176m 6920 660 S 0 0.0 0:00.00 /usr/sbin/apach
15991 www-data 20 0 0 0 0 Z 0 0.0 0:00.03 /usr/sbin/apach <defunct>
15992 www-data 20 0 0 0 0 Z 0 0.0 0:00.04 /usr/sbin/apach <defunct>
18265 root 20 0 0 0 0 S 0 0.0 0:00.26 kworker/0:3
18756 root 20 0 0 0 0 S 0 0.0 0:00.03 kworker/6:0
18814 www-data 20 0 425m 90m 3812 S 0 0.6 0:10.29 ruby
18966 root 20 0 0 0 0 S 0 0.0 0:00.30 kworker/1:1
24591 root 20 0 0 0 0 S 0 0.0 0:00.12 kworker/0:0
24706 root 20 0 0 0 0 S 0 0.0 0:00.18 kworker/1:2
25187 root 20 0 0 0 0 S 0 0.0 0:00.31 kworker/u:0
30427 root 20 0 0 0 0 S 0 0.0 0:00.23 kworker/u:1
32620 root 20 0 0 0 0 S 0 0.0 0:00.02 kworker/0:1

=========================================================================
Log Archive
=========================================================================
/nsm/sensor_data/securityonion-eth4/dailylogs/
790G .
31G ./2013-05-04
31G ./2013-05-05
101G ./2013-05-06
80G ./2013-05-07
117G ./2013-05-08
125G ./2013-05-09
120G ./2013-05-10
79G ./2013-05-11
38G ./2013-05-12
74G ./2013-05-13

/nsm/sensor_data/securityonionFW-eth1/dailylogs/
/usr/bin/sostat: line 69: cd: /nsm/sensor_data/securityonionFW-eth1/dailylogs: No such file or directory
72M .
12K ./.config
4.0K ./.pulse
8.0K ./.ssh

/nsm/bro/logs/
2.4G .
169M ./2013-04-30
177M ./2013-05-01
167M ./2013-05-02
173M ./2013-05-03
134M ./2013-05-04
139M ./2013-05-05
193M ./2013-05-06
178M ./2013-05-07
172M ./2013-05-08
170M ./2013-05-09
165M ./2013-05-10
118M ./2013-05-11
119M ./2013-05-12
141M ./2013-05-13
165M ./stats

=========================================================================
IDS Engine (snort) packet drops
=========================================================================
/nsm/sensor_data/securityonion-eth4/snort-1.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/securityonion-eth4/snort-2.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/securityonion-eth4/snort-3.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/securityonion-eth4/snort-4.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/securityonion-eth4/snort-5.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/securityonion-eth4/snort-6.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/securityonion-eth4/snort-7.stats last reported pkt_drop_percent as 0.000

=========================================================================
pf_ring stats
=========================================================================
Appl. Name : snort-cluster-51-socket-0
Tot Packets : 352075
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : snort-cluster-51-socket-0
Tot Packets : 707454
Tot Pkt Lost : 6588
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : snort-cluster-51-socket-0
Tot Packets : 325760
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : <unknown>
Tot Packets : 749450
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : <unknown>
Tot Packets : 886882
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : <unknown>
Tot Packets : 890746
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : <unknown>
Tot Packets : 634371
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : snort-cluster-51-socket-0
Tot Packets : 378689
Tot Pkt Lost : 30982
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : snort-cluster-51-socket-0
Tot Packets : 337339
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : snort-cluster-51-socket-0
Tot Packets : 289811
Tot Pkt Lost : 12985
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : snort-cluster-51-socket-0
Tot Packets : 464782
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.

Doug Burks

unread,

May 14, 2013, 7:58:36 AM5/14/13

to securit...@googlegroups.com

Hi Matthew,

Can you confirm that sphinx is listening with the following?
nc localhost 9306

Thanks,
Doug

On Mon, May 13, 2013 at 3:53 PM, Matthew Thacker
<matthewaa...@gmail.com> wrote:
> FW-eth1 (and theres another one FW2) are remote sensors.
> I'll try and knock down the number of enabled rules, but would too high a load keep the index from working at all? It used to work, but somewhere in the last couple of weeks it just stopped.

>
> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
Doug Burks
http://securityonion.blogspot.com

Martin Holste

unread,

May 17, 2013, 1:25:26 AM5/17/13

to security-onion

If you want to take Sphinx for a test drive, here's how you would do some searches to verify that something is in there:

mysql -h0 -P9306 -e "select * from distributed_local where match('tcp')"

That, of course, assumes that the word 'tcp' appears somewhere, which is usually a safe bet.

On Thu, May 16, 2013 at 7:19 AM, Matthew Thacker <matthewaa...@gmail.com> wrote:

Does anyone have any ideas on how to debug or troubleshoot? I don't know whether the problem lies in sphinx or mysql but I don't have any idea how to troubleshoot either of those programs so I'm kind of at a loss. Any help would be appreciated.

Karolis

unread,

May 17, 2013, 2:20:44 AM5/17/13

to securit...@googlegroups.com

I had same problem. In my case where were no /var/run/sphinxsearch folder. I solved it by re-creating the directory with appropriate permissions.

mkdir -p /var/run/sphinxsearch

chown sphinxsearch:root /var/run/sphinxsearch

debug info:

dpkg -l | grep sphinx

ii sphinxsearch 2.0.7-rel20-3736-0ubuntu10securityonion24 Fast standalone full-text SQL search engine

sudo start-stop-daemon --start --pidfile /var/run/sphinxsearch/searchd.pid --chuid sphinxsearch --exec /usr/bin/searchd

Sphinx 2.0.7-id64-dev (rel20-r3736)

snip

FATAL: failed to create pid file '/var/run/sphinxsearch/searchd.pid': No such file or directory

Karolis

Martin Holste

unread,

May 17, 2013, 6:04:23 PM5/17/13

to security-onion

That error suggests you edited /etc/elsa_node.conf and raised the num_indexes from its default of 200, which is fine, but you have to recreate the sphinx.conf file to match. You can do this (in stock ELSA, anyway, Scott please correct me if this is wrong in SO) by moving the /usr/local/etc/sphinx.conf file and then running echo "" | perl /opt/elsa/node/elsa.pl -o to do a dry-run and generate the file anew.

On Fri, May 17, 2013 at 7:19 AM, Matthew Thacker <matthewaa...@gmail.com> wrote:

All that seemed to be fine, but it did lead me to the log files which were helpful. I'm seeing this, could the "unknown local index" be my problem and if so, how do I fix that?

* TRACE [2013/05/17 12:13:45] /opt/elsa/web/lib/AsyncDB.pm (237) AsyncDB::sphinx 2787 [undef]
ran query
* DEBUG [2013/05/17 12:13:45] /opt/elsa/web/lib/API.pm (2017) API::__ANON__ 2787 [undef]
Sphinx query for node 127.0.0.1 finished in 0.00601696968078613
* ERROR [2013/05/17 12:13:45] /opt/elsa/web/lib/API.pm (2023) API::__ANON__ 2787 [undef]

node 127.0.0.1 got error $VAR1 = undef;

* ERROR [2013/05/17 12:13:45] /opt/elsa/web/lib/AsyncDB.pm (99) AsyncDB::__ANON__ 2787 [undef]
unknown local index 'temp_378' in search request at /opt/elsa/web/lib/AsyncDB.pm 236
* TRACE [2013/05/17 12:13:45] /opt/elsa/web/lib/API.pm (771) API::__ANON__ 2787 [undef]
connected to 127.0.0.1 on 9306 in 0.0276920795440674
* DEBUG [2013/05/17 12:13:45] /opt/elsa/web/lib/API.pm (2336) API::_sphinx_query 2787 [undef]
completed query in 0.040687084197998 with 0 rows
* INFO [2013/05/17 12:13:45] /opt/elsa/web/lib/API.pm (1740) API::query 2787 [undef]
Query 339 returned 0 rows

Reply all

Reply to author

Forward