Inconsistent ELSA queries and Index dates disappearing from the calendar

Martin Paszkiewicz

unread,

Jan 9, 2014, 4:00:10 PM1/9/14

to securit...@googlegroups.com

I have been battling a strange issue....and it may only be an issue because I think it is.

I have the latest updates via SOUP. The new ELSA page is VERY convenient and I greatly appreciate that; however, I find that if I do the same query over again just maybe a second or two apart, my results are significantly different every time. For example, if I choose "Top DST IPs" then the top IP has a count of 7777. If I submit the query right after it gives me the first results; within my second results for the same IP the count is 39138. If I submit it again the count jumps to 51283. Shouldn't I get a closer count number for the same start date? Sometimes my totals are less than the previous query...which perplexes me.

The ELSA calendar is also dropping days and I am lucky if I get more than 1 days worth to query from index. I adjusted elsa_node.conf and that has increased the amount of logs in index but does not increase how many days I can query the index from.

My system is a standalone box with 3 sensors. I originally installed the system from ISO early last year.

Am I missing something?

Kind regards,
Martin Paszkiewicz

Doug Burks

unread,

Jan 9, 2014, 5:31:07 PM1/9/14

to securit...@googlegroups.com

Hi Martin,

Replies inline.

On Thu, Jan 9, 2014 at 4:00 PM, Martin Paszkiewicz
<zulun...@gmail.com> wrote:
> I have been battling a strange issue....and it may only be an issue because I think it is.
>
> I have the latest updates via SOUP. The new ELSA page is VERY convenient and I greatly appreciate that; however, I find that if I do the same query over again just maybe a second or two apart, my results are significantly different every time. For example, if I choose "Top DST IPs" then the top IP has a count of 7777. If I submit the query right after it gives me the first results; within my second results for the same IP the count is 39138. If I submit it again the count jumps to 51283. Shouldn't I get a closer count number for the same start date? Sometimes my totals are less than the previous query...which perplexes me.

Have you tried increasing the query_timeout value in
/etc/elsa_web.conf and restarting Apache?

> The ELSA calendar is also dropping days and I am lucky if I get more than 1 days worth to query from index. I adjusted elsa_node.conf and that has increased the amount of logs in index but does not increase how many days I can query the index from.

What changes did you make to elsa_node.conf?

Martin Paszkiewicz

unread,

Jan 9, 2014, 6:03:21 PM1/9/14

to securit...@googlegroups.com

I may have answered some of my questions. I started using ELSA with BRO recently and just noticed the -from field- date appears to auto populate with the current time. So, every time I re-ran the the query it was from the pre-populated start date? This is what I am figuring.

The query_timeout in /etc/elsa_web.conf is currently set to 10 but I will test with a higher value, thank you.

For /etc/elsa_node.conf

>>Under "archive"<<

"days": 90,
"percentage": 10.
"table_size": 10000000
},

------

"log_size_limit" : 1200000000000,

------

# Uncomment to establish a retention period in days for indexed logs
"days": 90,

------

"allowed_temp_percent" : 80,

------

"allowed_mem_percent": 60,

------

#How many concurrent log readers can run. 1 should be fine for up to 50k logs/sec on the same node.
"num_log_readers" : 3

------

# Stats retention settings
"retention_days": 365

Thank you,
Martin Paszkiewicz

Martin Paszkiewicz

unread,

Jan 10, 2014, 1:35:30 PM1/10/14

to securit...@googlegroups.com

Doug,

I have adjusted /etc/elsa_web.conf , query_timeout , from 10 to 60 and this has greatly reduced the timeout errors that seemed to randomly happen when running a query. THANKS!!

As for losing calendar dates to query from. I ended up making some more adjustments to elsa_node.conf and used the following link as a guide:

https://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation

I adjusted the log_size_limit to 4300000000000

adjusted perm_index_size from 10000000 to 100000000

adjusted index_interval from 60 to 30

So far with these settings I am getting a wider date/time range to query the index from. I will monitor over the next few days but until then do you see any pitfalls that may arise with those adjustments?

Kind regards,
Martin Paszkiewicz

Doug Burks

unread,

Jan 11, 2014, 7:15:34 AM1/11/14

to securit...@googlegroups.com

log_size_limit should be fine assuming you have that much disk space
to devote to ELSA.

I'm not sure if there are pitfalls for perm_index_size and
index_interval. Martin Holste may be able to give you better guidance
for those.

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Doug Burks

Martin Paszkiewicz

unread,

Jan 13, 2014, 12:21:41 PM1/13/14

to securit...@googlegroups.com

I have monitored my system after making adjustments to elsa_node.conf and elsa_web.conf. Happily, I have a wider date range to query indexes from. Unfortunately, I still seem to experience inconsistencies when making queries. Please see attached screen shots as examples. Sometimes when I run a query, I get a syntax error right off the bat but if I re-run the same query most of the time it will go through without complaining.

The screen shots also show that for a set date range the number counts in the query do not come back with consistent information. Are my search queries too broad? Shouldn't I get the same counts for the same query?

Anyone else run into the same issue?

Kind regards,
Martin Paszkiewicz

Query_ERROR.JPG

Query1.JPG

Query2.JPG

Query3.JPG

Query4.JPG

Martin Paszkiewicz

unread,

Jan 13, 2014, 12:25:12 PM1/13/14

to securit...@googlegroups.com

sostat-redacted

=========================================================================
Service Status
=========================================================================
Status: securityonion
* sguil server[ OK ]
Status: HIDS
* ossec_agent (sguil)[ OK ]
Status: Bro
Name Type Host Status Pid Peers Started
manager manager X.X.X.X running 5401 9 10 Jan 14:56:21
proxy proxy X.X.X.X running 5566 9 10 Jan 14:56:23
NETMON-eth2-1 worker X.X.X.X running 6710 2 10 Jan 14:56:26
NETMON-eth2-2 worker X.X.X.X running 6705 2 10 Jan 14:56:26
NETMON-eth2-3 worker X.X.X.X running 6708 2 10 Jan 14:56:26
NETMON-eth3-1 worker X.X.X.X running 6712 2 10 Jan 14:56:26
NETMON-eth4-1 worker X.X.X.X running 6711 2 10 Jan 14:56:26
NETMON-eth4-2 worker X.X.X.X running 6709 2 10 Jan 14:56:26
NETMON-eth4-3 worker X.X.X.X running 6706 2 10 Jan 14:56:26
NETMON-eth4-4 worker X.X.X.X running 6707 2 10 Jan 14:56:26
Status: NETMON-eth2
* netsniff-ng (full packet data)[ OK ]
* pcap_agent (sguil)[ OK ]
* snort_agent (sguil)[ OK ]
* suricata (alert data)[ OK ]
* barnyard2 (spooler, unified2 format)[ OK ]
Status: NETMON-eth3
* netsniff-ng (full packet data)[ OK ]
* pcap_agent (sguil)[ OK ]
* snort_agent (sguil)[ OK ]
* suricata (alert data)[ OK ]
* barnyard2 (spooler, unified2 format)[ OK ]
Status: NETMON-eth4
* netsniff-ng (full packet data)[ OK ]
* pcap_agent (sguil)[ OK ]
* snort_agent (sguil)[ OK ]
* suricata (alert data)[ OK ]
* barnyard2 (spooler, unified2 format)[ OK ]

=========================================================================
Interface Status
=========================================================================
eth1 Link encap:Ethernet HWaddr 5c:f3:fc:3e:2d:c0
inet addr:X.X.X.X Bcast:X.X.X.X Mask:X.X.X.X
inet6 addr: fe80::5ef3:fcff:fe3e:2dc0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3041021 errors:0 dropped:0 overruns:0 frame:0
TX packets:2239040 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:301796692 (301.7 MB) TX bytes:1350542134 (1.3 GB)
Memory:91a00000-91a20000

eth2 Link encap:Ethernet HWaddr 00:0a:f7:02:f7:4d
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:2082918649 errors:0 dropped:115532 overruns:0 frame:1663
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1254413438270 (1.2 TB) TX bytes:0 (0.0 B)
Interrupt:45

eth3 Link encap:Ethernet HWaddr 00:0a:f7:02:f7:4e
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:171545233 errors:0 dropped:76 overruns:0 frame:3
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:98663863695 (98.6 GB) TX bytes:0 (0.0 B)
Interrupt:42

eth4 Link encap:Ethernet HWaddr 00:0a:f7:02:f7:4f
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:3230084537 errors:0 dropped:2468236 overruns:0 frame:24391
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1449678579082 (1.4 TB) TX bytes:0 (0.0 B)
Interrupt:45

lo Link encap:Local Loopback
inet addr:X.X.X.X Mask:X.X.X.X
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:133896915 errors:0 dropped:0 overruns:0 frame:0
TX packets:133896915 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:212118527634 (212.1 GB) TX bytes:212118527634 (212.1 GB)

=========================================================================
Link Statistics
=========================================================================
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
RX: bytes packets errors dropped overrun mcast
212118649835 133897126 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
212118649835 133897126 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:0a:f7:02:f7:4c brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
0 0 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 5c:f3:fc:3e:2d:c0 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
301796692 3041021 0 0 0 4463
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
1350542134 2239040 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
4: eth2: <BROADCAST,MULTICAST,NOARP,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:0a:f7:02:f7:4d brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
1254413438270 2082918649 0 0 1663 1
RX errors: length crc frame fifo missed
0 0 0 0 115532
TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
5: eth3: <BROADCAST,MULTICAST,NOARP,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:0a:f7:02:f7:4e brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
98663863695 171545233 0 0 3 196052
RX errors: length crc frame fifo missed
0 0 0 0 76
TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
6: eth4: <BROADCAST,MULTICAST,NOARP,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:0a:f7:02:f7:4f brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
1449678579082 3230084537 0 0 24391 197041
RX errors: length crc frame fifo missed
0 0 0 0 2468236
TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
7: usb0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 5e:f3:fc:3e:2d:c7 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
0 0 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0

=========================================================================
Disk Usage
=========================================================================
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 83G 9.4G 69G 12% /
udev 32G 4.0K 32G 1% /dev
tmpfs 13G 820K 13G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 32G 0 32G 0% /run/shm
/dev/sdb1 2.8T 91G 2.7T 4% /var
/dev/sdb2 17T 14T 3.0T 82% /nsm

=========================================================================
Network Sockets
=========================================================================
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
avahi-dae 1563 avahi 12u IPv4 30236 0t0 UDP *:5353
avahi-dae 1563 avahi 13u IPv6 30237 0t0 UDP *:5353
avahi-dae 1563 avahi 14u IPv4 30238 0t0 UDP *:57500
avahi-dae 1563 avahi 15u IPv6 30239 0t0 UDP *:55328
cupsd 1615 root 8u IPv6 4544007 0t0 TCP [::1]:631 (LISTEN)
cupsd 1615 root 9u IPv4 4544008 0t0 TCP X.X.X.X:631 (LISTEN)
dhclient3 1758 root 5u IPv4 22658 0t0 UDP *:68
sshd 1843 root 3u IPv4 1719 0t0 TCP *:22 (LISTEN)
sshd 1843 root 4u IPv6 1721 0t0 TCP *:22 (LISTEN)
mysqld 2000 mysql 10u IPv4 31386 0t0 TCP X.X.X.X:3306 (LISTEN)
mysqld 2000 mysql 38u IPv4 4507602 0t0 TCP X.X.X.X:3306->X.X.X.X:37602 (ESTABLISHED)
mysqld 2000 mysql 70u IPv4 4505490 0t0 TCP X.X.X.X:3306->X.X.X.X:37587 (ESTABLISHED)
mysqld 2000 mysql 176u IPv4 4509122 0t0 TCP X.X.X.X:3306->X.X.X.X:37589 (ESTABLISHED)
searchd 2150 sphinxsearch 7u IPv4 8388 0t0 TCP *:9306 (LISTEN)
searchd 2150 sphinxsearch 8u IPv4 8389 0t0 TCP *:9312 (LISTEN)
ossec-csy 2171 ossecm 5u IPv4 453 0t0 UDP X.X.X.X:38154->X.X.X.X:514
xrdp 2761 xrdp 6u IPv4 555 0t0 TCP *:3389 (LISTEN)
xrdp-sesm 2763 root 6u IPv4 9401 0t0 TCP X.X.X.X:3350 (LISTEN)
/usr/sbin 2845 root 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 2845 root 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 2845 root 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 2845 root 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
ntpd 4201 ntp 16u IPv4 20550 0t0 UDP *:123
ntpd 4201 ntp 17u IPv6 20551 0t0 UDP *:123
ntpd 4201 ntp 18u IPv4 20557 0t0 UDP X.X.X.X:123
ntpd 4201 ntp 19u IPv4 20558 0t0 UDP X.X.X.X:123
ntpd 4201 ntp 20u IPv6 20559 0t0 UDP [fe80::5ef3:fcff:fe3e:2dc0]:123
ntpd 4201 ntp 21u IPv6 20560 0t0 UDP [::1]:123
barnyard2 4563 root 3u IPv4 4517969 0t0 TCP X.X.X.X:60153->X.X.X.X:8000 (ESTABLISHED)
barnyard2 4563 root 4u IPv4 4517972 0t0 TCP X.X.X.X:37587->X.X.X.X:3306 (ESTABLISHED)
barnyard2 4636 root 3u IPv4 4497395 0t0 TCP X.X.X.X:58415->X.X.X.X:8100 (ESTABLISHED)
barnyard2 4636 root 4u IPv4 4501313 0t0 TCP X.X.X.X:37589->X.X.X.X:3306 (ESTABLISHED)
tclsh 4642 root 13u IPv4 4409499 0t0 TCP *:7734 (LISTEN)
tclsh 4642 root 14u IPv4 4409500 0t0 TCP *:7736 (LISTEN)
tclsh 4642 root 15u IPv4 4759319 0t0 TCP X.X.X.X:7736->X.X.X.X:34357 (ESTABLISHED)
tclsh 4642 root 16u IPv4 4752205 0t0 TCP X.X.X.X:7736->X.X.X.X:34358 (ESTABLISHED)
tclsh 4642 root 17u IPv4 4399666 0t0 TCP X.X.X.X:7736->X.X.X.X:57384 (ESTABLISHED)
tclsh 4642 root 18u IPv4 4399667 0t0 TCP X.X.X.X:7736->X.X.X.X:57385 (ESTABLISHED)
tclsh 4642 root 19u IPv4 4754009 0t0 TCP X.X.X.X:7736->X.X.X.X:34356 (ESTABLISHED)
tclsh 4642 root 20u IPv4 4417647 0t0 TCP X.X.X.X:7736->X.X.X.X:57387 (ESTABLISHED)
tclsh 4642 root 21u IPv4 4738043 0t0 TCP X.X.X.X:7736->X.X.X.X:34331 (ESTABLISHED)
barnyard2 4716 root 3u IPv4 4505502 0t0 TCP X.X.X.X:55571->X.X.X.X:8200 (ESTABLISHED)
barnyard2 4716 root 4u IPv4 4505505 0t0 TCP X.X.X.X:37602->X.X.X.X:3306 (ESTABLISHED)
bro 5401 root 4u IPv4 17612 0t0 UDP X.X.X.X:53476->X.X.X.X:53
bro 5411 root 0u IPv4 22397 0t0 TCP *:47761 (LISTEN)
bro 5411 root 1u IPv6 22398 0t0 TCP *:47761 (LISTEN)
bro 5411 root 2u IPv4 28148 0t0 TCP X.X.X.X:47761->X.X.X.X:36398 (ESTABLISHED)
bro 5411 root 4u IPv4 17612 0t0 UDP X.X.X.X:53476->X.X.X.X:53
bro 5411 root 21u IPv4 36674 0t0 TCP X.X.X.X:47761->X.X.X.X:36400 (ESTABLISHED)
bro 5411 root 23u IPv4 15722 0t0 TCP X.X.X.X:47761->X.X.X.X:36401 (ESTABLISHED)
bro 5411 root 24u IPv4 38175 0t0 TCP X.X.X.X:47761->X.X.X.X:36404 (ESTABLISHED)
bro 5411 root 25u IPv4 14988 0t0 TCP X.X.X.X:47761->X.X.X.X:36405 (ESTABLISHED)
bro 5411 root 26u IPv4 25255 0t0 TCP X.X.X.X:47761->X.X.X.X:36407 (ESTABLISHED)
bro 5411 root 27u IPv4 18776 0t0 TCP X.X.X.X:47761->X.X.X.X:36410 (ESTABLISHED)
bro 5411 root 28u IPv4 18777 0t0 TCP X.X.X.X:47761->X.X.X.X:36411 (ESTABLISHED)
bro 5411 root 29u IPv4 18778 0t0 TCP X.X.X.X:47761->X.X.X.X:36414 (ESTABLISHED)
bro 5566 root 4u IPv4 38991 0t0 UDP X.X.X.X:40267->X.X.X.X:53
bro 5574 root 0u IPv4 15593 0t0 TCP X.X.X.X:36398->X.X.X.X:47761 (ESTABLISHED)
bro 5574 root 1u IPv4 15598 0t0 TCP *:47762 (LISTEN)
bro 5574 root 2u IPv6 15599 0t0 TCP *:47762 (LISTEN)
bro 5574 root 4u IPv4 38991 0t0 UDP X.X.X.X:40267->X.X.X.X:53
bro 5574 root 19u IPv4 8969 0t0 TCP X.X.X.X:47762->X.X.X.X:34761 (ESTABLISHED)
bro 5574 root 21u IPv4 34450 0t0 TCP X.X.X.X:47762->X.X.X.X:34764 (ESTABLISHED)
bro 5574 root 22u IPv4 35026 0t0 TCP X.X.X.X:47762->X.X.X.X:34765 (ESTABLISHED)
bro 5574 root 23u IPv4 25254 0t0 TCP X.X.X.X:47762->X.X.X.X:34768 (ESTABLISHED)
bro 5574 root 24u IPv4 25256 0t0 TCP X.X.X.X:47762->X.X.X.X:34770 (ESTABLISHED)
bro 5574 root 25u IPv4 25257 0t0 TCP X.X.X.X:47762->X.X.X.X:34771 (ESTABLISHED)
bro 5574 root 26u IPv4 23449 0t0 TCP X.X.X.X:47762->X.X.X.X:34774 (ESTABLISHED)
bro 5574 root 27u IPv4 25258 0t0 TCP X.X.X.X:47762->X.X.X.X:34775 (ESTABLISHED)
bro 6705 root 4u IPv4 18750 0t0 UDP X.X.X.X:43366->X.X.X.X:53
bro 6706 root 4u IPv4 39091 0t0 UDP X.X.X.X:48470->X.X.X.X:53
bro 6707 root 4u IPv4 20636 0t0 UDP X.X.X.X:48176->X.X.X.X:53
bro 6708 root 4u IPv4 26745 0t0 UDP X.X.X.X:54928->X.X.X.X:53
bro 6709 root 4u IPv4 12638 0t0 UDP X.X.X.X:42920->X.X.X.X:53
bro 6710 root 4u IPv4 17069 0t0 UDP X.X.X.X:55746->X.X.X.X:53
bro 6711 root 4u IPv4 32695 0t0 UDP X.X.X.X:52696->X.X.X.X:53
bro 6712 root 4u IPv4 15717 0t0 UDP X.X.X.X:50866->X.X.X.X:53
bro 6715 root 0u IPv4 14975 0t0 TCP X.X.X.X:34761->X.X.X.X:47762 (ESTABLISHED)
bro 6715 root 1u IPv4 14978 0t0 TCP X.X.X.X:36400->X.X.X.X:47761 (ESTABLISHED)
bro 6715 root 2u IPv4 25240 0t0 TCP *:47768 (LISTEN)
bro 6715 root 4u IPv4 12638 0t0 UDP X.X.X.X:42920->X.X.X.X:53
bro 6715 root 20u IPv6 25241 0t0 TCP *:47768 (LISTEN)
bro 6721 root 0u IPv4 28832 0t0 TCP X.X.X.X:36401->X.X.X.X:47761 (ESTABLISHED)
bro 6721 root 1u IPv4 28835 0t0 TCP X.X.X.X:34764->X.X.X.X:47762 (ESTABLISHED)
bro 6721 root 2u IPv4 28838 0t0 TCP *:47763 (LISTEN)
bro 6721 root 4u IPv4 17069 0t0 UDP X.X.X.X:55746->X.X.X.X:53
bro 6721 root 20u IPv6 28839 0t0 TCP *:47763 (LISTEN)
bro 6766 root 0u IPv4 28866 0t0 TCP X.X.X.X:34765->X.X.X.X:47762 (ESTABLISHED)
bro 6766 root 1u IPv4 28869 0t0 TCP X.X.X.X:36404->X.X.X.X:47761 (ESTABLISHED)
bro 6766 root 2u IPv4 28872 0t0 TCP *:47765 (LISTEN)
bro 6766 root 4u IPv4 26745 0t0 UDP X.X.X.X:54928->X.X.X.X:53
bro 6766 root 20u IPv6 28873 0t0 TCP *:47765 (LISTEN)
bro 6817 root 0u IPv4 18768 0t0 TCP X.X.X.X:36405->X.X.X.X:47761 (ESTABLISHED)
bro 6817 root 1u IPv4 18771 0t0 TCP X.X.X.X:34768->X.X.X.X:47762 (ESTABLISHED)
bro 6817 root 2u IPv4 18774 0t0 TCP *:47767 (LISTEN)
bro 6817 root 4u IPv4 32695 0t0 UDP X.X.X.X:52696->X.X.X.X:53
bro 6817 root 20u IPv6 18775 0t0 TCP *:47767 (LISTEN)
bro 6822 root 0u IPv4 17070 0t0 TCP X.X.X.X:36407->X.X.X.X:47761 (ESTABLISHED)
bro 6822 root 1u IPv4 17073 0t0 TCP X.X.X.X:34770->X.X.X.X:47762 (ESTABLISHED)
bro 6822 root 2u IPv4 17076 0t0 TCP *:47769 (LISTEN)
bro 6822 root 4u IPv4 39091 0t0 UDP X.X.X.X:48470->X.X.X.X:53
bro 6822 root 20u IPv6 17077 0t0 TCP *:47769 (LISTEN)
bro 6828 root 0u IPv4 32705 0t0 TCP X.X.X.X:34771->X.X.X.X:47762 (ESTABLISHED)
bro 6828 root 1u IPv4 32708 0t0 TCP X.X.X.X:36410->X.X.X.X:47761 (ESTABLISHED)
bro 6828 root 2u IPv4 32711 0t0 TCP *:47770 (LISTEN)
bro 6828 root 4u IPv4 20636 0t0 UDP X.X.X.X:48176->X.X.X.X:53
bro 6828 root 20u IPv6 32712 0t0 TCP *:47770 (LISTEN)
bro 6831 root 0u IPv4 25888 0t0 TCP X.X.X.X:36411->X.X.X.X:47761 (ESTABLISHED)
bro 6831 root 1u IPv4 25891 0t0 TCP X.X.X.X:34774->X.X.X.X:47762 (ESTABLISHED)
bro 6831 root 2u IPv4 25894 0t0 TCP *:47766 (LISTEN)
bro 6831 root 4u IPv4 15717 0t0 UDP X.X.X.X:50866->X.X.X.X:53
bro 6831 root 20u IPv6 25895 0t0 TCP *:47766 (LISTEN)
bro 6834 root 0u IPv4 22478 0t0 TCP X.X.X.X:34775->X.X.X.X:47762 (ESTABLISHED)
bro 6834 root 1u IPv4 22481 0t0 TCP X.X.X.X:36414->X.X.X.X:47761 (ESTABLISHED)
bro 6834 root 2u IPv4 22484 0t0 TCP *:47764 (LISTEN)
bro 6834 root 4u IPv4 18750 0t0 UDP X.X.X.X:43366->X.X.X.X:53
bro 6834 root 20u IPv6 22485 0t0 TCP *:47764 (LISTEN)
/usr/sbin 9864 www-data 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 9864 www-data 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 9864 www-data 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 9864 www-data 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
/usr/sbin 9864 www-data 34u IPv4 5160326 0t0 TCP X.X.X.X:53163->X.X.X.X:3154 (CLOSE_WAIT)
/usr/sbin 13873 www-data 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 13873 www-data 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 13873 www-data 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 13873 www-data 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
tclsh 18661 root 3u IPv4 4753985 0t0 TCP X.X.X.X:34331->X.X.X.X:7736 (ESTABLISHED)
syslog-ng 18696 root 23u IPv4 4005963 0t0 TCP *:514 (LISTEN)
syslog-ng 18696 root 24u IPv4 4005964 0t0 UDP *:514
tclsh 18983 root 3u IPv4 4765778 0t0 TCP X.X.X.X:34356->X.X.X.X:7736 (ESTABLISHED)
tclsh 19056 root 3u IPv4 4766745 0t0 TCP X.X.X.X:34357->X.X.X.X:7736 (ESTABLISHED)
tclsh 19128 root 3u IPv4 4766751 0t0 TCP X.X.X.X:34358->X.X.X.X:7736 (ESTABLISHED)
/usr/sbin 19259 www-data 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 19259 www-data 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 19259 www-data 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 19259 www-data 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
/usr/sbin 19354 www-data 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 19354 www-data 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 19354 www-data 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 19354 www-data 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
tclsh 19420 root 3u IPv4 4324224 0t0 TCP X.X.X.X:57385->X.X.X.X:7736 (ESTABLISHED)
tclsh 19420 root 4u IPv4 3365651 0t0 TCP X.X.X.X:8000 (LISTEN)
tclsh 19420 root 6u IPv4 4509116 0t0 TCP X.X.X.X:8000->X.X.X.X:60153 (ESTABLISHED)
/usr/sbin 19425 www-data 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 19425 www-data 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 19425 www-data 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 19425 www-data 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
ruby1.9.1 19441 www-data 12u IPv4 3014211 0t0 TCP X.X.X.X:35224 (LISTEN)
tclsh 19845 root 3u IPv4 4399665 0t0 TCP X.X.X.X:57387->X.X.X.X:7736 (ESTABLISHED)
tclsh 19845 root 4u IPv4 3371945 0t0 TCP X.X.X.X:8100 (LISTEN)
tclsh 19845 root 6u IPv4 4508073 0t0 TCP X.X.X.X:8100->X.X.X.X:58415 (ESTABLISHED)
tclsh 20583 root 3u IPv4 4412598 0t0 TCP X.X.X.X:57384->X.X.X.X:7736 (ESTABLISHED)
tclsh 20583 root 4u IPv4 3365713 0t0 TCP X.X.X.X:8200 (LISTEN)
tclsh 20583 root 6u IPv4 4508075 0t0 TCP X.X.X.X:8200->X.X.X.X:55571 (ESTABLISHED)
sshd 21565 root 3u IPv4 4854610 0t0 TCP X.X.X.X:22->X.X.X.X:40610 (ESTABLISHED)
/usr/sbin 23032 www-data 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 23032 www-data 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 23032 www-data 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 23032 www-data 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
/usr/sbin 25219 www-data 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 25219 www-data 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 25219 www-data 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 25219 www-data 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
/usr/sbin 25219 www-data 33u IPv4 5151574 0t0 TCP X.X.X.X:53211->X.X.X.X:3154 (CLOSE_WAIT)
/usr/sbin 25221 www-data 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 25221 www-data 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 25221 www-data 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 25221 www-data 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
/usr/sbin 25222 www-data 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 25222 www-data 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 25222 www-data 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 25222 www-data 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
/usr/sbin 27319 www-data 4u IPv4 31492 0t0 TCP *:443 (LISTEN)
/usr/sbin 27319 www-data 5u IPv4 31495 0t0 TCP *:9876 (LISTEN)
/usr/sbin 27319 www-data 6u IPv4 31497 0t0 TCP *:3154 (LISTEN)
/usr/sbin 27319 www-data 7u IPv4 31501 0t0 TCP *:444 (LISTEN)
ruby1.9.1 28456 www-data 12u IPv4 4897216 0t0 TCP X.X.X.X:50908 (LISTEN)

=========================================================================
IDS Rules Update
=========================================================================
Mon Jan 13 07:01:01 UTC 2014
Backing up current local_rules.xml file.
Cleaning up local_rules.xml backup files older than 30 days.
Backing up current downloaded.rules file before it gets overwritten.
Cleaning up downloaded.rules backup files older than 30 days.
Backing up current local.rules file before it gets overwritten.
Cleaning up local.rules backup files older than 30 days.
Running PulledPork.
http://code.google.com/p/pulledpork/
_____ ____
`----,\ )
`--==\\ / PulledPork v0.6.1 the Smoking Pig <////~
`--==\\/
.-~~~~-.Y|\\_ Copyright (C) 2009-2011 JJ Cummings
@_/ / 66\_ cumm...@gmail.com
| \ \ _(")
\ /-| ||'--' Rules give me wings!
\_\ \_\\
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Checking latest MD5 for etpro.rules.tar.gz....
They Match
Done!
Prepping rules from etpro.rules.tar.gz for work....
Done!
Reading rules...
Generating Stub Rules....
Done
Reading rules...
Reading rules...
Reading rules...
Processing /etc/nsm/pulledpork/enablesid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/dropsid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/disablesid.conf....
Modified 2 rules
Done
Modifying Sids....
Done!
Setting Flowbit State....
Done
Writing /etc/nsm/rules/downloaded.rules....
Done
Writing /etc/nsm/rules/so_rules.rules....
Done
Generating sid-msg.map....
Done
Writing /etc/nsm/rules/sid-msg.map....
Done
Writing /var/log/sid_changes.log....
Done
Rule Stats....
New:-------0
Deleted:---0
Enabled Rules:----8852
Dropped Rules:----0
Disabled Rules:---775
Total Rules:------9627
Done
Please review /var/log/sid_changes.log for additional details
Fly Piggy Fly!
Restarting Barnyard2.
Restarting: NETMON-eth2
* stopping: barnyard2 (spooler, unified2 format)[ OK ]
* starting: barnyard2 (spooler, unified2 format)[ OK ]
Restarting: NETMON-eth3
* stopping: barnyard2 (spooler, unified2 format)[ OK ]
* starting: barnyard2 (spooler, unified2 format)[ OK ]
Restarting: NETMON-eth4
* stopping: barnyard2 (spooler, unified2 format)[ OK ]
* starting: barnyard2 (spooler, unified2 format)[ OK ]
Restarting IDS Engine.
Restarting: NETMON-eth2
* stopping: suricata (alert data)[ OK ]
* starting: suricata (alert data)[ OK ]
Restarting: NETMON-eth3
* stopping: suricata (alert data)[ OK ]
* starting: suricata (alert data)[ OK ]
Restarting: NETMON-eth4
* stopping: suricata (alert data)[ OK ]
* starting: suricata (alert data)[ OK ]

=========================================================================
CPU Usage
=========================================================================
top - 17:19:25 up 3 days, 2:24, 1 user, load average: 22.21, 20.87, 20.47
Tasks: 319 total, 18 running, 301 sleeping, 0 stopped, 0 zombie
Cpu(s): 15.7%us, 8.7%sy, 1.1%ni, 73.6%id, 0.3%wa, 0.0%hi, 0.5%si, 0.0%st
Mem: 65953156k total, 65603176k used, 349980k free, 3456k buffers
Swap: 47482556k total, 4182220k used, 43300336k free, 24946228k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4998 sguil 20 0 6044m 4.5g 898m S 302 7.2 707:38.63 Suricata-Main
4826 sguil 20 0 5856m 4.5g 898m S 202 7.2 536:19.24 Suricata-Main
6705 root 20 0 1934m 1.8g 1.0g R 101 2.8 1755:46 bro
6706 root 20 0 3925m 3.1g 1.0g R 101 5.0 1719:37 bro
6707 root 20 0 1856m 1.7g 1.0g R 101 2.7 1668:22 bro
6710 root 20 0 3996m 2.4g 1.0g R 101 3.8 1744:20 bro
480 root 20 0 524m 351m 3732 R 99 0.5 70:59.81 indexer
6708 root 20 0 1871m 1.7g 1.0g R 99 2.8 1727:25 bro
6709 root 20 0 1871m 1.7g 1.0g R 99 2.8 1680:14 bro
6711 root 20 0 1867m 1.7g 1.0g R 99 2.8 1675:22 bro
20089 root 20 0 519m 192m 3768 R 99 0.3 0:03.16 indexer
2000 mysql 20 0 6192m 204m 5012 S 45 0.3 387:34.27 mysqld
18713 root 20 0 276m 36m 3884 S 43 0.1 143:48.52 perl
6712 root 20 0 1404m 1.3g 1.0g R 37 2.0 1073:38 bro
15551 sguil 20 0 627m 565m 561m R 35 0.9 95:01.57 netsniff-ng
18696 root 20 0 6341m 5.8g 1592 S 33 9.3 95:09.14 syslog-ng
5401 root 20 0 394m 88m 3060 S 27 0.1 326:15.11 bro
5566 root 20 0 303m 214m 2976 S 27 0.3 179:37.25 bro
15679 sguil 20 0 504m 447m 443m R 25 0.7 114:27.22 netsniff-ng
5411 root 25 5 122m 13m 900 S 19 0.0 1169:40 bro
5574 root 25 5 105m 10m 908 S 19 0.0 1159:16 bro
6817 root 25 5 1108m 1.0g 1.0g R 16 1.6 685:10.05 bro
6766 root 25 5 1108m 1.0g 1.0g S 14 1.6 685:53.42 bro
6715 root 25 5 1108m 1.0g 1.0g S 12 1.6 683:59.79 bro
6721 root 25 5 1108m 1.0g 1.0g S 12 1.6 684:46.84 bro
6822 root 25 5 1108m 1.0g 1.0g R 12 1.6 679:34.38 bro
6831 root 25 5 1108m 1.0g 1.0g R 12 1.6 687:55.01 bro
6834 root 25 5 1108m 1.0g 1.0g S 10 1.6 687:58.89 bro
117 root 20 0 0 0 0 S 8 0.0 13:56.84 kswapd1
6828 root 25 5 1108m 1.0g 1.0g S 8 1.6 682:48.36 bro
4903 sguil 20 0 1006m 530m 130m S 4 0.8 17:58.35 Suricata-Main
86 root 20 0 0 0 0 S 2 0.0 1:01.19 kworker/20:0
4151 root 20 0 0 0 0 S 2 0.0 1:06.00 kworker/8:2
15615 sguil 20 0 819m 313m 308m R 2 0.5 13:13.57 netsniff-ng
19780 root 20 0 0 0 0 S 2 0.0 0:04.91 kworker/0:0
20250 root 20 0 17468 1448 944 R 2 0.0 0:00.02 top
1 root 20 0 24740 2084 1164 S 0 0.0 0:13.23 init
2 root 20 0 0 0 0 S 0 0.0 0:00.02 kthreadd
3 root 20 0 0 0 0 S 0 0.0 3:06.33 ksoftirqd/0
6 root RT 0 0 0 0 S 0 0.0 0:18.71 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:17.64 watchdog/0
8 root RT 0 0 0 0 S 0 0.0 0:08.64 migration/1
9 root 20 0 0 0 0 S 0 0.0 1:31.63 kworker/1:0
10 root 20 0 0 0 0 S 0 0.0 0:06.72 ksoftirqd/1
12 root RT 0 0 0 0 S 0 0.0 0:00.64 watchdog/1
13 root RT 0 0 0 0 S 0 0.0 0:06.14 migration/2
14 root 20 0 0 0 0 S 0 0.0 1:28.55 kworker/2:0
15 root 20 0 0 0 0 S 0 0.0 0:06.19 ksoftirqd/2
16 root RT 0 0 0 0 S 0 0.0 0:00.62 watchdog/2
17 root RT 0 0 0 0 S 0 0.0 0:06.67 migration/3
19 root 20 0 0 0 0 S 0 0.0 0:06.29 ksoftirqd/3
20 root RT 0 0 0 0 S 0 0.0 0:00.56 watchdog/3
21 root RT 0 0 0 0 S 0 0.0 0:06.38 migration/4
23 root 20 0 0 0 0 S 0 0.0 0:06.25 ksoftirqd/4
24 root RT 0 0 0 0 S 0 0.0 0:00.69 watchdog/4
25 root RT 0 0 0 0 S 0 0.0 0:06.38 migration/5
26 root 20 0 0 0 0 S 0 0.0 1:22.11 kworker/5:0
27 root 20 0 0 0 0 S 0 0.0 0:06.26 ksoftirqd/5
28 root RT 0 0 0 0 S 0 0.0 0:00.57 watchdog/5
29 root RT 0 0 0 0 S 0 0.0 0:05.72 migration/6
31 root 20 0 0 0 0 S 0 0.0 0:09.76 ksoftirqd/6
32 root RT 0 0 0 0 S 0 0.0 0:02.68 watchdog/6
33 root RT 0 0 0 0 S 0 0.0 0:05.13 migration/7
35 root 20 0 0 0 0 S 0 0.0 0:10.91 ksoftirqd/7
36 root RT 0 0 0 0 S 0 0.0 0:00.80 watchdog/7
37 root RT 0 0 0 0 S 0 0.0 0:06.00 migration/8
39 root 20 0 0 0 0 S 0 0.0 0:07.30 ksoftirqd/8
40 root RT 0 0 0 0 S 0 0.0 0:00.89 watchdog/8
41 root RT 0 0 0 0 S 0 0.0 0:04.65 migration/9
43 root 20 0 0 0 0 S 0 0.0 0:07.37 ksoftirqd/9
44 root RT 0 0 0 0 S 0 0.0 0:00.69 watchdog/9
45 root RT 0 0 0 0 S 0 0.0 0:05.05 migration/10
46 root 20 0 0 0 0 S 0 0.0 1:16.65 kworker/10:0
47 root 20 0 0 0 0 S 0 0.0 0:07.35 ksoftirqd/10
48 root RT 0 0 0 0 S 0 0.0 0:00.64 watchdog/10
49 root RT 0 0 0 0 S 0 0.0 0:04.16 migration/11
50 root 20 0 0 0 0 S 0 0.0 1:25.36 kworker/11:0
51 root 20 0 0 0 0 S 0 0.0 0:07.38 ksoftirqd/11
52 root RT 0 0 0 0 S 0 0.0 0:00.90 watchdog/11
53 root RT 0 0 0 0 S 0 0.0 0:06.17 migration/12
54 root 20 0 0 0 0 S 0 0.0 1:14.47 kworker/12:0
55 root 20 0 0 0 0 S 0 0.0 0:03.40 ksoftirqd/12
56 root RT 0 0 0 0 S 0 0.0 0:00.63 watchdog/12
57 root RT 0 0 0 0 S 0 0.0 0:05.89 migration/13
59 root 20 0 0 0 0 S 0 0.0 0:04.30 ksoftirqd/13
60 root RT 0 0 0 0 S 0 0.0 0:00.72 watchdog/13
61 root RT 0 0 0 0 S 0 0.0 0:05.51 migration/14
62 root 20 0 0 0 0 S 0 0.0 1:16.46 kworker/14:0
63 root 20 0 0 0 0 S 0 0.0 0:04.49 ksoftirqd/14
64 root RT 0 0 0 0 S 0 0.0 0:00.62 watchdog/14
65 root RT 0 0 0 0 S 0 0.0 0:04.84 migration/15
66 root 20 0 0 0 0 S 0 0.0 1:08.25 kworker/15:0
67 root 20 0 0 0 0 S 0 0.0 0:04.15 ksoftirqd/15
68 root RT 0 0 0 0 S 0 0.0 0:00.52 watchdog/15
69 root RT 0 0 0 0 S 0 0.0 0:05.90 migration/16
70 root 20 0 0 0 0 S 0 0.0 1:17.73 kworker/16:0
71 root 20 0 0 0 0 S 0 0.0 0:03.93 ksoftirqd/16
72 root RT 0 0 0 0 S 0 0.0 0:00.52 watchdog/16
73 root RT 0 0 0 0 S 0 0.0 0:04.88 migration/17
75 root 20 0 0 0 0 S 0 0.0 0:03.76 ksoftirqd/17
76 root RT 0 0 0 0 S 0 0.0 0:00.57 watchdog/17
77 root RT 0 0 0 0 S 0 0.0 0:05.35 migration/18
79 root 20 0 0 0 0 S 0 0.0 0:04.00 ksoftirqd/18
80 root RT 0 0 0 0 S 0 0.0 0:00.68 watchdog/18
81 root RT 0 0 0 0 S 0 0.0 0:05.23 migration/19
82 root 20 0 0 0 0 S 0 0.0 1:01.11 kworker/19:0
83 root 20 0 0 0 0 S 0 0.0 0:05.53 ksoftirqd/19
84 root RT 0 0 0 0 S 0 0.0 0:00.76 watchdog/19
85 root RT 0 0 0 0 S 0 0.0 0:04.15 migration/20
87 root 20 0 0 0 0 S 0 0.0 0:04.16 ksoftirqd/20
88 root RT 0 0 0 0 S 0 0.0 0:00.59 watchdog/20
89 root RT 0 0 0 0 S 0 0.0 0:04.19 migration/21
90 root 20 0 0 0 0 S 0 0.0 0:59.11 kworker/21:0
91 root 20 0 0 0 0 S 0 0.0 0:04.05 ksoftirqd/21
92 root RT 0 0 0 0 S 0 0.0 0:00.70 watchdog/21
93 root RT 0 0 0 0 S 0 0.0 0:04.48 migration/22
95 root 20 0 0 0 0 S 0 0.0 0:03.89 ksoftirqd/22
96 root RT 0 0 0 0 S 0 0.0 0:00.59 watchdog/22
97 root RT 0 0 0 0 S 0 0.0 0:03.12 migration/23
99 root 20 0 0 0 0 S 0 0.0 0:03.90 ksoftirqd/23
100 root RT 0 0 0 0 S 0 0.0 0:00.61 watchdog/23
101 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
102 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
103 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
104 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
106 root 20 0 0 0 0 S 0 0.0 0:00.61 sync_supers
107 root 20 0 0 0 0 S 0 0.0 0:00.01 bdi-default
108 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
109 root 0 -20 0 0 0 S 0 0.0 0:00.09 kblockd
110 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
111 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
112 root 0 -20 0 0 0 S 0 0.0 0:00.00 md
113 root 20 0 0 0 0 S 0 0.0 1:22.80 kworker/11:1
115 root 20 0 0 0 0 S 0 0.0 0:00.21 khungtaskd
116 root 20 0 0 0 0 S 0 0.0 10:18.10 kswapd0
118 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
119 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
120 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
121 root 20 0 0 0 0 S 0 0.0 0:00.00 ecryptfs-kthrea
122 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
130 root 0 -20 0 0 0 S 0 0.0 0:00.00 kthrotld
132 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
133 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
134 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
135 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_3
136 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_4
137 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_5
140 root 20 0 0 0 0 S 0 0.0 0:01.25 kworker/u:4
141 root 20 0 0 0 0 S 0 0.0 0:01.79 kworker/u:5
163 root 0 -20 0 0 0 S 0 0.0 0:00.00 devfreq_wq
164 root 20 0 0 0 0 S 0 0.0 0:59.41 kworker/19:1
172 root 20 0 0 0 0 S 0 0.0 0:52.41 kworker/13:1
191 root 20 0 0 0 0 S 0 0.0 1:21.43 kworker/3:1
199 root 20 0 0 0 0 S 0 0.0 1:09.01 kworker/14:1
200 root 20 0 0 0 0 S 0 0.0 1:03.44 kworker/21:1
238 root 20 0 0 0 0 R 0 0.0 1:09.94 kworker/17:1
240 root 20 0 0 0 0 S 0 0.0 1:12.52 kworker/12:1
242 root 20 0 0 0 0 S 0 0.0 1:01.02 kworker/22:1
245 root 20 0 0 0 0 S 0 0.0 0:56.13 kworker/18:1
315 root 20 0 11436 444 444 S 0 0.0 0:00.00 tail
324 root 20 0 0 0 0 S 0 0.0 1:24.98 kworker/1:1
412 root 20 0 0 0 0 S 0 0.0 1:08.43 kworker/4:2
456 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_6
500 root 20 0 11436 444 444 S 0 0.0 0:00.00 tail
573 root 20 0 11436 444 444 S 0 0.0 0:00.00 tail
582 root 20 0 0 0 0 S 0 0.0 0:07.12 jbd2/sda1-8
583 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
688 root 20 0 17236 540 468 S 0 0.0 0:00.07 upstart-udev-br
690 root 20 0 22464 624 624 S 0 0.0 0:00.08 udevd
962 root 20 0 22356 280 276 S 0 0.0 0:00.00 udevd
963 root 20 0 22356 248 244 S 0 0.0 0:00.00 udevd
1038 root 0 -20 0 0 0 S 0 0.0 0:00.00 edac-poller
1072 root 0 -20 0 0 0 S 0 0.0 0:00.00 kmpathd
1073 root 0 -20 0 0 0 S 0 0.0 0:00.00 kmpath_handlerd
1266 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfs_mru_cache
1267 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfslogd
1268 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfsdatad
1269 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfsconvertd
1270 root 20 0 0 0 0 S 0 0.0 0:03.70 xfsbufd/sdb1
1271 root 20 0 0 0 0 S 0 0.0 0:38.40 xfsaild/sdb1
1292 root 20 0 0 0 0 S 0 0.0 0:08.67 xfsbufd/sdb2
1294 root 20 0 0 0 0 S 0 0.0 0:41.83 xfsaild/sdb2
1523 messageb 20 0 24268 772 336 S 0 0.0 0:00.07 dbus-daemon
1563 avahi 20 0 32316 944 864 S 0 0.0 0:00.01 avahi-daemon
1565 avahi 20 0 32184 148 128 S 0 0.0 0:00.00 avahi-daemon
1606 root 20 0 21192 856 856 S 0 0.0 0:00.00 bluetoothd
1610 root 20 0 15192 388 316 S 0 0.0 0:00.00 upstart-socket-
1615 root 20 0 101m 1248 1244 S 0 0.0 0:00.03 cupsd
1616 root 10 -10 0 0 0 S 0 0.0 0:00.00 krfcommd
1652 root 20 0 0 0 0 S 0 0.0 0:03.46 flush-8:0
1653 root 20 0 0 0 0 S 0 0.0 12:16.27 flush-8:16
1758 root 20 0 7268 664 516 S 0 0.0 0:00.97 dhclient3
1843 root 20 0 50036 1360 1324 S 0 0.0 0:00.00 sshd
1930 root 20 0 20028 692 688 S 0 0.0 0:00.00 getty
1935 root 20 0 20028 692 688 S 0 0.0 0:00.00 getty
1944 root 20 0 20028 692 688 S 0 0.0 0:00.00 getty
1945 root 20 0 20028 692 688 S 0 0.0 0:00.00 getty
1948 root 20 0 20028 692 688 S 0 0.0 0:00.00 getty
1964 root 20 0 4464 536 532 S 0 0.0 0:00.00 acpid
1965 root 20 0 19116 800 672 S 0 0.0 0:03.29 cron
1966 daemon 20 0 16912 164 160 S 0 0.0 0:00.00 atd
1974 root 20 0 15984 628 504 S 0 0.0 3:04.00 irqbalance
1976 root 20 0 264m 1156 1156 S 0 0.0 0:00.01 lightdm
2001 root 20 0 225m 2464 2040 S 0 0.0 4:22.52 Xorg
2005 sphinxse 20 0 79236 1296 1296 S 0 0.0 0:00.00 su
2031 root 20 0 4154m 1188 1188 S 0 0.0 0:00.11 console-kit-dae
2123 root 20 0 190m 1568 1068 S 0 0.0 0:00.05 polkitd
2150 sphinxse 20 0 15.4g 11g 11g S 0 18.8 15:54.12 searchd
2171 ossecm 20 0 12920 532 432 S 0 0.0 0:01.76 ossec-csyslogd
2176 root 20 0 12808 344 324 S 0 0.0 0:00.04 ossec-execd
2180 ossec 20 0 14640 2084 680 S 0 0.0 0:09.11 ossec-analysisd
2192 root 20 0 4516 492 400 S 0 0.0 0:00.06 ossec-logcollec
2198 root 20 0 177m 1632 1632 S 0 0.0 0:00.00 lightdm
2208 root 20 0 118m 1664 1320 S 0 0.0 0:03.70 accounts-daemon
2239 lightdm 20 0 4404 492 488 S 0 0.0 0:00.00 lightdm-greeter
2245 root 20 0 6116 1188 544 S 0 0.0 3:11.04 ossec-syscheckd
2257 ossec 20 0 13072 412 408 S 0 0.0 0:00.18 ossec-monitord
2259 lightdm 20 0 23956 272 272 S 0 0.0 0:00.00 dbus-daemon
2260 lightdm 20 0 235m 3896 2532 S 0 0.0 9:58.25 lightdm-gtk-gre
2273 lightdm 20 0 52424 1184 1180 S 0 0.0 0:00.00 gvfsd
2275 lightdm 20 0 203m 1168 1168 S 0 0.0 0:00.00 gvfs-fuse-daemo
2294 root 20 0 214m 1640 1260 S 0 0.0 0:00.03 upowerd
2363 root 20 0 98.6m 1320 1316 S 0 0.0 0:00.00 lightdm
2573 root 20 0 0 0 0 S 0 0.0 0:32.64 kworker/23:0
2738 root 20 0 101m 1168 1048 S 0 0.0 0:00.17 winbindd
2761 xrdp 20 0 18920 140 140 S 0 0.0 0:00.00 xrdp
2763 root 20 0 29384 292 292 S 0 0.0 0:00.00 xrdp-sesman
2826 root 20 0 101m 832 700 S 0 0.0 0:00.21 winbindd
2845 root 20 0 176m 6152 3396 S 0 0.0 0:07.29 /usr/sbin/apach
2913 root 20 0 20028 692 688 S 0 0.0 0:00.00 getty
3652 root 20 0 0 0 0 S 0 0.0 0:29.98 kworker/0:1
4201 ntp 20 0 37776 1248 1128 S 0 0.0 0:12.75 ntpd
4563 root 20 0 146m 47m 1512 S 0 0.1 0:16.47 barnyard2
4565 root 20 0 0 0 0 S 0 0.0 0:16.70 kworker/17:2
4636 root 20 0 146m 46m 1536 S 0 0.1 0:15.27 barnyard2
4642 root 20 0 126m 5608 2436 S 0 0.0 0:04.94 tclsh
4657 root 20 0 126m 2688 548 S 0 0.0 0:01.64 tclsh
4658 root 20 0 125m 2452 332 S 0 0.0 0:00.00 tclsh
4716 root 20 0 146m 47m 1512 S 0 0.1 0:17.86 barnyard2
4875 root 20 0 0 0 0 S 0 0.0 0:13.74 kworker/6:0
5212 root 20 0 17896 1232 1228 S 0 0.0 0:00.00 bash
5468 root 20 0 17896 1232 1228 S 0 0.0 0:00.00 bash
6631 root 20 0 17896 1232 1228 S 0 0.0 0:00.00 bash
6633 root 20 0 17896 1232 1228 S 0 0.0 0:00.00 bash
6635 root 20 0 17896 1232 1228 S 0 0.0 0:00.00 bash
6636 root 20 0 17896 1232 1228 S 0 0.0 0:00.00 bash
6639 root 20 0 17896 1232 1228 S 0 0.0 0:00.00 bash
6641 root 20 0 17896 1232 1228 S 0 0.0 0:00.00 bash
6645 root 20 0 17896 1232 1228 S 0 0.0 0:00.00 bash
6647 root 20 0 17896 1232 1228 S 0 0.0 0:00.00 bash
6815 root 20 0 0 0 0 S 0 0.0 0:29.00 kworker/2:2
6932 root 20 0 4344 232 232 S 0 0.0 0:00.00 tail
7061 root 20 0 4344 312 232 S 0 0.0 0:00.00 tail
7200 root 20 0 4344 232 232 S 0 0.0 0:00.00 tail
7338 www-data 20 0 432m 101m 3140 S 0 0.2 17:47.38 ruby
9864 www-data 20 0 398m 119m 8072 S 0 0.2 0:03.75 /usr/sbin/apach
11295 root 20 0 0 0 0 S 0 0.0 0:38.70 kworker/5:2
11366 root 20 0 0 0 0 S 0 0.0 0:05.38 kworker/6:1
12432 root 20 0 0 0 0 S 0 0.0 0:05.46 kworker/3:0
12483 root 20 0 0 0 0 S 0 0.0 0:36.95 kworker/7:0
12492 root 20 0 0 0 0 S 0 0.0 0:03.24 kworker/18:0
12618 root 20 0 0 0 0 S 0 0.0 0:15.84 kworker/0:2
13873 www-data 20 0 392m 114m 7588 S 0 0.2 0:03.20 /usr/sbin/apach
15911 root 20 0 0 0 0 S 0 0.0 0:06.80 kworker/10:1
16026 root 20 0 0 0 0 S 0 0.0 0:36.42 kworker/4:1
16033 root 10 -10 25576 12m 3396 S 0 0.0 0:04.52 atop
16035 root 20 0 0 0 0 S 0 0.0 0:54.27 kworker/8:0
17192 root 20 0 4404 608 508 S 0 0.0 0:00.00 sh
17195 root 20 0 4404 428 320 S 0 0.0 0:00.00 sh
17402 root 20 0 0 0 0 S 0 0.0 0:12.50 kworker/0:3
17428 root 20 0 0 0 0 S 0 0.0 0:01.68 kworker/6:3
17777 root 20 0 0 0 0 S 0 0.0 1:00.24 kworker/15:2
17928 root 20 0 0 0 0 S 0 0.0 0:58.34 kworker/9:3
18661 root 20 0 43136 4296 1892 S 0 0.0 0:00.20 tclsh
18664 root 20 0 11440 508 436 S 0 0.0 0:00.00 tail
18695 root 20 0 26784 400 160 S 0 0.0 0:00.00 syslog-ng
18711 root 20 0 4404 596 492 S 0 0.0 0:00.00 sh
18983 root 20 0 40284 3592 1908 S 0 0.0 0:00.23 tclsh
19056 root 20 0 40020 3320 1912 S 0 0.0 0:00.12 tclsh
19128 root 20 0 40284 3644 1912 S 0 0.0 0:00.23 tclsh
19242 root 20 0 215m 1288 1228 S 0 0.0 0:00.00 PassengerWatchd
19245 root 20 0 865m 2436 1716 S 0 0.0 3:02.55 PassengerHelper
19247 root 20 0 108m 2760 1908 S 0 0.0 0:00.11 ruby1.9.1
19251 nobody 20 0 165m 2780 1752 S 0 0.0 0:00.21 PassengerLoggin
19259 www-data 20 0 408m 66m 7588 S 0 0.1 0:18.86 /usr/sbin/apach
19354 www-data 20 0 399m 118m 7420 S 0 0.2 0:20.19 /usr/sbin/apach
19420 root 20 0 40020 2576 1716 S 0 0.0 0:00.47 tclsh
19425 www-data 20 0 408m 66m 7420 S 0 0.1 0:19.37 /usr/sbin/apach
19441 www-data 20 0 353m 90m 2780 S 0 0.1 7:03.71 ruby1.9.1
19504 root 20 0 11436 444 444 S 0 0.0 0:00.00 tail
19776 root 20 0 4312 348 272 S 0 0.0 0:00.00 sleep
19845 root 20 0 40020 2528 1780 S 0 0.0 0:00.18 tclsh
19847 root 20 0 11436 540 444 S 0 0.0 0:00.00 tail
19864 root 20 0 75244 2012 1424 S 0 0.0 0:00.00 cron
19866 root 20 0 4404 612 508 S 0 0.0 0:00.00 sh
19869 root 20 0 233m 53m 3812 S 0 0.1 0:01.96 perl
19874 root 20 0 16544 1348 1140 S 0 0.0 0:00.00 sostat-redacted
19875 root 20 0 16568 1492 1264 S 0 0.0 0:00.00 sostat
19876 root 20 0 15744 820 696 S 0 0.0 0:00.00 sed
20583 root 20 0 40020 2548 1776 S 0 0.0 0:00.70 tclsh
20585 root 20 0 11436 444 444 S 0 0.0 0:00.00 tail
20605 root 20 0 0 0 0 S 0 0.0 0:16.15 kworker/9:1
21565 root 20 0 154m 4348 3016 S 0 0.0 0:03.31 sshd
21725 root 20 0 34104 8728 1568 S 0 0.0 0:01.10 bash
23032 www-data 20 0 408m 121m 7664 S 0 0.2 0:06.12 /usr/sbin/apach
24157 root 20 0 0 0 0 S 0 0.0 0:13.42 kworker/16:1
25219 www-data 20 0 410m 69m 7612 S 0 0.1 0:15.95 /usr/sbin/apach
25221 www-data 20 0 416m 72m 7608 S 0 0.1 0:15.54 /usr/sbin/apach
25222 www-data 20 0 417m 112m 7592 S 0 0.2 0:16.18 /usr/sbin/apach
26078 root 20 0 0 0 0 S 0 0.0 1:07.94 kworker/23:1
27298 root 20 0 75244 1840 1276 S 0 0.0 0:00.00 cron
27301 root 20 0 4404 608 508 S 0 0.0 0:00.00 sh
27302 root 20 0 234m 53m 3460 S 0 0.1 0:08.34 perl
27319 www-data 20 0 410m 122m 7856 S 0 0.2 0:05.51 /usr/sbin/apach
28446 root 20 0 0 0 0 S 0 0.0 0:07.57 kworker/13:2
28456 www-data 20 0 348m 85m 2708 S 0 0.1 0:05.59 ruby1.9.1
28515 root 20 0 0 0 0 S 0 0.0 0:10.41 kworker/7:2
28903 root 20 0 0 0 0 S 0 0.0 0:09.65 kworker/22:0
29212 root 20 0 0 0 0 S 0 0.0 0:58.03 kworker/20:1

=========================================================================
Log Archive
=========================================================================
/nsm/sensor_data/NETMON-eth2/dailylogs/ - 26 days
5.5T .
192G ./2013-12-19
758G ./2013-12-20
30G ./2013-12-21
27G ./2013-12-22
29G ./2013-12-23
25G ./2013-12-24
25G ./2013-12-25
48G ./2013-12-26
52G ./2013-12-27
23G ./2013-12-28
25G ./2013-12-29
25G ./2013-12-30
24G ./2013-12-31
23G ./2014-01-01
47G ./2014-01-02
49G ./2014-01-03
30G ./2014-01-04
32G ./2014-01-05
581G ./2014-01-06
660G ./2014-01-07
713G ./2014-01-08
768G ./2014-01-09
795G ./2014-01-10
91G ./2014-01-11
77G ./2014-01-12
473G ./2014-01-13

/nsm/sensor_data/NETMON-eth3/dailylogs/ - 26 days
571G .
11G ./2013-12-19
36G ./2013-12-20
16G ./2013-12-21
14G ./2013-12-22
14G ./2013-12-23
11G ./2013-12-24
13G ./2013-12-25
35G ./2013-12-26
40G ./2013-12-27
11G ./2013-12-28
11G ./2013-12-29
12G ./2013-12-30
13G ./2013-12-31
13G ./2014-01-01
16G ./2014-01-02
16G ./2014-01-03
20G ./2014-01-04
21G ./2014-01-05
29G ./2014-01-06
36G ./2014-01-07
36G ./2014-01-08
44G ./2014-01-09
41G ./2014-01-10
18G ./2014-01-11
24G ./2014-01-12
33G ./2014-01-13

/nsm/sensor_data/NETMON-eth4/dailylogs/ - 26 days
6.1T .
274G ./2013-12-19
862G ./2013-12-20
27G ./2013-12-21
24G ./2013-12-22
27G ./2013-12-23
23G ./2013-12-24
21G ./2013-12-25
23G ./2013-12-26
22G ./2013-12-27
22G ./2013-12-28
24G ./2013-12-29
23G ./2013-12-30
22G ./2013-12-31
21G ./2014-01-01
47G ./2014-01-02
50G ./2014-01-03
27G ./2014-01-04
29G ./2014-01-05
660G ./2014-01-06
743G ./2014-01-07
785G ./2014-01-08
856G ./2014-01-09
924G ./2014-01-10
96G ./2014-01-11
75G ./2014-01-12
541G ./2014-01-13

/nsm/bro/logs/ - 30 days
106G .
8.0K ./2013-12-13
6.7G ./2013-12-16
11G ./2013-12-17
7.9G ./2013-12-18
8.1G ./2013-12-19
9.0G ./2013-12-20
420M ./2013-12-21
377M ./2013-12-22
443M ./2013-12-23
398M ./2013-12-24
394M ./2013-12-25
420M ./2013-12-26
421M ./2013-12-27
393M ./2013-12-28
380M ./2013-12-29
421M ./2013-12-30
417M ./2013-12-31
372M ./2014-01-01
736M ./2014-01-02
724M ./2014-01-03
392M ./2014-01-04
380M ./2014-01-05
9.4G ./2014-01-06
9.0G ./2014-01-07
9.2G ./2014-01-08
12G ./2014-01-09
9.5G ./2014-01-10
1.2G ./2014-01-11
1.3G ./2014-01-12
6.1G ./2014-01-13
607M ./stats

=========================================================================
Bro netstats
=========================================================================
Average packet loss as percent across all Bro workers: 0.836684

NETMON-eth2-1: 1389633495.290578 recvd=662672061 dropped=6325881 link=662672061
NETMON-eth2-2: 1389633468.291372 recvd=673384218 dropped=5290840 link=673384218
NETMON-eth2-3: 1389633515.214098 recvd=667891825 dropped=5397966 link=667891825
NETMON-eth3-1: 1389633569.336963 recvd=171408400 dropped=45 link=171408400
NETMON-eth4-1: 1389633468.506979 recvd=764985703 dropped=5895803 link=764985703
NETMON-eth4-2: 1389633493.407962 recvd=757433456 dropped=5146677 link=757433456
NETMON-eth4-3: 1389633459.685881 recvd=849532971 dropped=9938905 link=849532971
NETMON-eth4-4: 1389633468.920683 recvd=754339305 dropped=6361940 link=754339305

=========================================================================
IDS Engine (suricata) packet drops
=========================================================================
/nsm/sensor_data/NETMON-eth2/stats.log
tcp.ssn_memcap_drop | RxPFReth27 | 0
tcp.segment_memcap_drop | RxPFReth27 | 16269945

/nsm/sensor_data/NETMON-eth3/stats.log
tcp.ssn_memcap_drop | RxPFReth31 | 0
tcp.segment_memcap_drop | RxPFReth31 | 0

/nsm/sensor_data/NETMON-eth4/stats.log
tcp.ssn_memcap_drop | RxPFReth47 | 0
tcp.segment_memcap_drop | RxPFReth47 | 14910273

=========================================================================
pf_ring stats
=========================================================================
PF_RING Version : 5.6.1 ($Revision: exported$)
Total rings : 37

Standard (non DNA) Options
Ring slots : 65534
Slot version : 15
Capture TX : Yes [RX+TX]
IP Defragment : No
Socket Mode : Standard
Transparent mode : Yes [mode 0]
Total plugins : 0
Cluster Fragment Queue : 1056
Cluster Fragment Discard : 92532000

/proc/net/pf_ring/5016-eth2.2379
Appl. Name : Suricata
Tot Packets : 107240144
Tot Pkt Lost : 251709
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 77923

/proc/net/pf_ring/5017-eth2.2380
Appl. Name : Suricata
Tot Packets : 111013466
Tot Pkt Lost : 389855
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 77933

/proc/net/pf_ring/5018-eth2.2381
Appl. Name : Suricata
Tot Packets : 104416778
Tot Pkt Lost : 14203
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 78023

/proc/net/pf_ring/5019-eth2.2382
Appl. Name : Suricata
Tot Packets : 102806042
Tot Pkt Lost : 1187869
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 77964

/proc/net/pf_ring/5020-eth2.2383
Appl. Name : Suricata
Tot Packets : 103933291
Tot Pkt Lost : 91351
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 77901

/proc/net/pf_ring/5021-eth2.2384
Appl. Name : Suricata
Tot Packets : 108272374
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 77998

/proc/net/pf_ring/5022-eth2.2385
Appl. Name : Suricata
Tot Packets : 107080559
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 77999

/proc/net/pf_ring/5026-eth3.2386
Appl. Name : Suricata
Tot Packets : 33470483
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 78018

/proc/net/pf_ring/5031-eth4.2387
Appl. Name : Suricata
Tot Packets : 187840689
Tot Pkt Lost : 162783
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 78019

/proc/net/pf_ring/5032-eth4.2388
Appl. Name : Suricata
Tot Packets : 198806006
Tot Pkt Lost : 5323125
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 77901

/proc/net/pf_ring/5033-eth4.2389
Appl. Name : Suricata
Tot Packets : 133719389
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 77919

/proc/net/pf_ring/5034-eth4.2390
Appl. Name : Suricata
Tot Packets : 134674426
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 77997

/proc/net/pf_ring/5035-eth4.2391
Appl. Name : Suricata
Tot Packets : 141266600
Tot Pkt Lost : 1653770
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 78010

/proc/net/pf_ring/5036-eth4.2392
Appl. Name : Suricata
Tot Packets : 190415421
Tot Pkt Lost : 66443
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 77903

/proc/net/pf_ring/5037-eth4.2393
Appl. Name : Suricata
Tot Packets : 191249959
Tot Pkt Lost : 509998
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 78028
Num Free Slots : 78016

/proc/net/pf_ring/6705-eth2.1
Appl. Name : <unknown>
Tot Packets : 680230727
Tot Pkt Lost : 5290840
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 130434
Num Free Slots : 0

/proc/net/pf_ring/6706-eth4.2
Appl. Name : <unknown>
Tot Packets : 861421916
Tot Pkt Lost : 9938905
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 130434
Num Free Slots : 0

/proc/net/pf_ring/6707-eth4.7
Appl. Name : <unknown>
Tot Packets : 762570112
Tot Pkt Lost : 6361940
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 130434
Num Free Slots : 0

/proc/net/pf_ring/6708-eth2.8
Appl. Name : <unknown>
Tot Packets : 674063632
Tot Pkt Lost : 5397966
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 130434
Num Free Slots : 0

/proc/net/pf_ring/6709-eth4.4
Appl. Name : <unknown>
Tot Packets : 763905871
Tot Pkt Lost : 5146677
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 130434
Num Free Slots : 0

/proc/net/pf_ring/6710-eth2.6
Appl. Name : <unknown>
Tot Packets : 670107653
Tot Pkt Lost : 6325881
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 130434
Num Free Slots : 0

/proc/net/pf_ring/6711-eth4.5
Appl. Name : <unknown>
Tot Packets : 772775269
Tot Pkt Lost : 5895803
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 130434
Num Free Slots : 0

/proc/net/pf_ring/6712-eth3.3
Appl. Name : <unknown>
Tot Packets : 171410464
Tot Pkt Lost : 45
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 130434
Num Free Slots : 130434

=========================================================================
Netsniff-NG - Reported Packet Loss (per interval)
=========================================================================
File: /var/log/nsm/NETMON-eth2/netsniff-ng.log Processed: +837953 Lost: -15058
File: /var/log/nsm/NETMON-eth2/netsniff-ng.log Processed: +924337 Lost: -19664
File: /var/log/nsm/NETMON-eth2/netsniff-ng.log Processed: +579032 Lost: -1
File: /var/log/nsm/NETMON-eth2/netsniff-ng.log Processed: +942138 Lost: -656220
File: /var/log/nsm/NETMON-eth2/netsniff-ng.log Processed: +714451 Lost: -63863
File: /var/log/nsm/NETMON-eth2/netsniff-ng.log Processed: +970687 Lost: -83339
File: /var/log/nsm/NETMON-eth2/netsniff-ng.log Processed: +973642 Lost: -165995
File: /var/log/nsm/NETMON-eth2/netsniff-ng.log Processed: +761123 Lost: -1
File: /var/log/nsm/NETMON-eth2/netsniff-ng.log Processed: +919151 Lost: -47289
File: /var/log/nsm/NETMON-eth2/netsniff-ng.log Processed: +843355 Lost: -823147
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1212755 Lost: -11323
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1069373 Lost: -58004
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1074910 Lost: -4877
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1073569 Lost: -2
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1199588 Lost: -279331
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1238152 Lost: -198290
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1088321 Lost: -1329429
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1096423 Lost: -25870
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1168059 Lost: -9220
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1184541 Lost: -37282
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1115281 Lost: -2
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1200809 Lost: -53505
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1231208 Lost: -42853
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1076700 Lost: -3
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1298858 Lost: -12181
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1073716 Lost: -5
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +1112189 Lost: -11281
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +983458 Lost: -1427010
File: /var/log/nsm/NETMON-eth4/netsniff-ng.log Processed: +3412792 Lost: -56465

=========================================================================
Sguil Uncategorized Events
=========================================================================
COUNT(*)
35

=========================================================================
Sguil events summary for yesterday
=========================================================================
Totals GenID:SigID Signature
18 1:100000232 GPL CHAT Google Talk Logon
18 1:100000230 GPL CHAT MISC Jabber/Google Talk Outgoing Traffic
18 1:2002334 ET CHAT Google IM traffic Jabber client sign-on
14 1:2500080 ET COMPROMISED Known Compromised or Hostile Host Traffic group 41
12 1:2522528 ET TOR Known Tor Relay/Router (Not Exit) Node Traffic group 265
7 1:2500028 ET COMPROMISED Known Compromised or Hostile Host Traffic group 15
5 1:2500036 ET COMPROMISED Known Compromised or Hostile Host Traffic group 19
4 1:2500016 ET COMPROMISED Known Compromised or Hostile Host Traffic group 9
4 1:2101616 GPL DNS named version attempt
3 1:2500004 ET COMPROMISED Known Compromised or Hostile Host Traffic group 3
3 1:2500050 ET COMPROMISED Known Compromised or Hostile Host Traffic group 26
3 1:2500032 ET COMPROMISED Known Compromised or Hostile Host Traffic group 17
2 1:2500000 ET COMPROMISED Known Compromised or Hostile Host Traffic group 1
2 1:2500066 ET COMPROMISED Known Compromised or Hostile Host Traffic group 34
2 1:2002400 ET USER_AGENTS Suspicious User Agent (Microsoft Internet Explorer)
1 1:2500008 ET COMPROMISED Known Compromised or Hostile Host Traffic group 5
1 1:2500048 ET COMPROMISED Known Compromised or Hostile Host Traffic group 25
1 1:2500056 ET COMPROMISED Known Compromised or Hostile Host Traffic group 29
Total
118

=========================================================================
Top 50 All time Sguil Events
=========================================================================
Totals GenID:SigID Signature
13235 1:2006402 ET POLICY Incoming Basic Auth Base64 HTTP Password detected unencrypted
10241 1:2017884 ET INFO SUSPICIOUS SMTP EXE - ZIP file with .exe filename inside (Inbound)
9364 1:2013504 ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management
9342 1:2101201 GPL WEB_SERVER 403 Forbidden
6218 1:2000419 ET POLICY PE EXE or DLL Windows file download
6147 1:2011582 ET POLICY Vulnerable Java Version 1.6.x Detected
4866 1:2013030 ET POLICY libwww-perl User-Agent
4825 1:2000328 ET POLICY Outbound Multiple Non-SMTP Server Emails
4562 1:2013028 ET POLICY curl User-Agent Outbound
3947 1:2803167 ETPRO POLICY MOBILE Android Device User-Agent
3381 1:2014520 ET INFO EXE - Served Attached HTTP
3300 1:2012936 ET SCAN ZmEu Scanner User-Agent Inbound
3161 1:2001219 ET SCAN Potential SSH Scan
3003 1:2002997 ET WEB_SERVER PHP Remote File Inclusion (monster list http)
2960 1:2014819 ET INFO Packed Executable Download
2736 1:2014095 ET POLICY Kindle Fire Browser User-Agent Outbound
2561 1:2010144 ET P2P Vuze BT UDP Connection (5)
2472 1:2013936 ET POLICY SSH banner detected on TCP 443 likely proxy evasion
2285 1:2015744 ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)
2154 1:2013505 ET POLICY GNU/Linux YUM User-Agent Outbound likely related to package management
2020 1:2002825 ET POLICY POSSIBLE Web Crawl using Curl
1923 1:2014527 ET CURRENT_EVENTS Exploit Kit Delivering Compressed Flash Content to Client
1770 1:2014472 ET INFO JAVA - Java Archive Download
1677 1:2013290 ET POLICY MOBILE Apple device leaking UDID from SpringBoard via GET
1564 1:2012889 ET POLICY Http Client Body contains pw= in cleartext
1552 1:2013410 ET POLICY Outbound MSSQL Connection to Standard port (1433)
1380 1:2016981 ET WEB_SERVER open_basedir PHP config option in uri
1380 1:2016977 ET WEB_SERVER allow_url_include PHP config option in uri
1380 1:2016979 ET WEB_SERVER suhosin.simulation PHP config option in uri
1380 1:2016982 ET WEB_SERVER auto_prepend_file PHP config option in uri
1380 1:2016978 ET WEB_SERVER safe_mode PHP config option in uri
1271 1:2016980 ET WEB_SERVER disable_functions PHP config option in uri
1263 1:2011768 ET WEB_SERVER PHP tags in HTTP POST
1215 1:2014297 ET POLICY Vulnerable Java Version 1.7.x Detected
1213 1:2012843 ET POLICY Cleartext WordPress Login
1174 1:2002945 ET POLICY Java Url Lib User Agent Web Crawl
1145 1:2016683 ET WEB_SERVER WebShell Generic - wget http - POST
1142 1:2001855 ET MALWARE Fun Web Products Spyware User-Agent (FunWebProducts)
1128 1:2008066 ET MALWARE Blank User-Agent (descriptor but no string)
1114 1:2002087 ET POLICY Inbound Frequent Emails - Possible Spambot Inbound
1063 1:2010935 ET POLICY Suspicious inbound to MSSQL port 1433
991 1:2017910 ET INFO suspicious - gzipped file via JAVA - could be pack200-ed JAR
970 1:2002911 ET SCAN Potential VNC Scan 5900-5920
966 1:2012888 ET POLICY Http Client Body contains pwd= in cleartext
939 1:2012887 ET POLICY Http Client Body contains pass= in cleartext
875 1:100000230 GPL CHAT MISC Jabber/Google Talk Outgoing Traffic
864 1:2007994 ET MALWARE Suspicious User-Agent (1 space)
831 1:2803491 ETPRO TROJAN Suspicious HTTP STOP Return - Trojan.Win32.FakeAV.cfty or Related Controller
792 1:2002334 ET CHAT Google IM traffic Jabber client sign-on
791 1:100000232 GPL CHAT Google Talk Logon
Total
175460

=========================================================================
Top 50 URLs for yesterday
=========================================================================
Total
0

=========================================================================
Snorby Events Summary for yesterday
=========================================================================
Totals GenID:SigID SignatureName
18 1:2002334 ET CHAT Google IM traffic Jabber client sign-on
18 1:100000232 GPL CHAT Google Talk Logon
18 1:100000230 GPL CHAT MISC Jabber/Google Talk Outgoing Traffic
14 1:2500080 ET COMPROMISED Known Compromised or Hostile Host Traffic group 41
12 1:2522528 ET TOR Known Tor Relay/Router (Not Exit) Node Traffic group 265
7 1:2500028 ET COMPROMISED Known Compromised or Hostile Host Traffic group 15
5 1:2500036 ET COMPROMISED Known Compromised or Hostile Host Traffic group 19
4 1:2500016 ET COMPROMISED Known Compromised or Hostile Host Traffic group 9
4 1:2101616 GPL DNS named version attempt
3 1:2500050 ET COMPROMISED Known Compromised or Hostile Host Traffic group 26
3 1:2500032 ET COMPROMISED Known Compromised or Hostile Host Traffic group 17
3 1:2500004 ET COMPROMISED Known Compromised or Hostile Host Traffic group 3
2 1:2500066 ET COMPROMISED Known Compromised or Hostile Host Traffic group 34
2 1:2002400 ET USER_AGENTS Suspicious User Agent (Microsoft Internet Explorer)
2 1:2500000 ET COMPROMISED Known Compromised or Hostile Host Traffic group 1
1 1:2500008 ET COMPROMISED Known Compromised or Hostile Host Traffic group 5
1 1:2500056 ET COMPROMISED Known Compromised or Hostile Host Traffic group 29
1 1:2500048 ET COMPROMISED Known Compromised or Hostile Host Traffic group 25
Total
118

=========================================================================
Top 50 All Time Snorby Events
=========================================================================
Totals GenID:SigID SignatureName
13236 1:2006402 ET POLICY Incoming Basic Auth Base64 HTTP Password detected unencrypted
10241 1:2017884 ET INFO SUSPICIOUS SMTP EXE - ZIP file with .exe filename inside (Inbound)
9364 1:2013504 ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management
9342 1:2101201 GPL WEB_SERVER 403 Forbidden
6218 1:2000419 ET POLICY PE EXE or DLL Windows file download
6147 1:2011582 ET POLICY Vulnerable Java Version 1.6.x Detected
4866 1:2013030 ET POLICY libwww-perl User-Agent
4825 1:2000328 ET POLICY Outbound Multiple Non-SMTP Server Emails
4562 1:2013028 ET POLICY curl User-Agent Outbound
3948 1:2803167 ETPRO POLICY MOBILE Android Device User-Agent
3381 1:2014520 ET INFO EXE - Served Attached HTTP
3300 1:2012936 ET SCAN ZmEu Scanner User-Agent Inbound
3161 1:2001219 ET SCAN Potential SSH Scan
3003 1:2002997 ET WEB_SERVER PHP Remote File Inclusion (monster list http)
2960 1:2014819 ET INFO Packed Executable Download
2736 1:2014095 ET POLICY Kindle Fire Browser User-Agent Outbound
2561 1:2010144 ET P2P Vuze BT UDP Connection (5)
2472 1:2013936 ET POLICY SSH banner detected on TCP 443 likely proxy evasion
2285 1:2015744 ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)
2154 1:2013505 ET POLICY GNU/Linux YUM User-Agent Outbound likely related to package management
2020 1:2002825 ET POLICY POSSIBLE Web Crawl using Curl
1923 1:2014527 ET CURRENT_EVENTS Exploit Kit Delivering Compressed Flash Content to Client
1770 1:2014472 ET INFO JAVA - Java Archive Download
1677 1:2013290 ET POLICY MOBILE Apple device leaking UDID from SpringBoard via GET
1574 1:2013410 ET POLICY Outbound MSSQL Connection to Standard port (1433)
1564 1:2012889 ET POLICY Http Client Body contains pw= in cleartext
1380 1:2016982 ET WEB_SERVER auto_prepend_file PHP config option in uri
1380 1:2016981 ET WEB_SERVER open_basedir PHP config option in uri
1380 1:2016979 ET WEB_SERVER suhosin.simulation PHP config option in uri
1380 1:2016978 ET WEB_SERVER safe_mode PHP config option in uri
1380 1:2016977 ET WEB_SERVER allow_url_include PHP config option in uri
1271 1:2016980 ET WEB_SERVER disable_functions PHP config option in uri
1263 1:2011768 ET WEB_SERVER PHP tags in HTTP POST
1215 1:2014297 ET POLICY Vulnerable Java Version 1.7.x Detected
1213 1:2012843 ET POLICY Cleartext WordPress Login
1174 1:2002945 ET POLICY Java Url Lib User Agent Web Crawl
1145 1:2016683 ET WEB_SERVER WebShell Generic - wget http - POST
1142 1:2001855 ET MALWARE Fun Web Products Spyware User-Agent (FunWebProducts)
1129 1:2008066 ET MALWARE Blank User-Agent (descriptor but no string)
1118 1:2002087 ET POLICY Inbound Frequent Emails - Possible Spambot Inbound
1063 1:2010935 ET POLICY Suspicious inbound to MSSQL port 1433
991 1:2017910 ET INFO suspicious - gzipped file via JAVA - could be pack200-ed JAR
970 1:2002911 ET SCAN Potential VNC Scan 5900-5920
966 1:2012888 ET POLICY Http Client Body contains pwd= in cleartext
941 1:2012887 ET POLICY Http Client Body contains pass= in cleartext
875 1:100000230 GPL CHAT MISC Jabber/Google Talk Outgoing Traffic
864 1:2007994 ET MALWARE Suspicious User-Agent (1 space)
832 1:2803491 ETPRO TROJAN Suspicious HTTP STOP Return - Trojan.Win32.FakeAV.cfty or Related Controller
792 1:2002334 ET CHAT Google IM traffic Jabber client sign-on
791 1:100000232 GPL CHAT Google Talk Logon
Total
175474

Doug Burks

unread,

Jan 13, 2014, 2:50:42 PM1/13/14

to securit...@googlegroups.com

On Mon, Jan 13, 2014 at 12:25 PM, Martin Paszkiewicz
<zulun...@gmail.com> wrote:
> top - 17:19:25 up 3 days, 2:24, 1 user, load average: 22.21, 20.87, 20.47

Load average of 22 is high. How many CPU cores do you have? If your
CPU cores are busy monitoring traffic, they may have trouble pulling
data from ELSA within the query_timeout value.

Looking at other parts of your sostat output, you're dropping lots of
packets. You may need to increase your hardware and/or build separate
sensors and have a dedicated master server.

--
Doug Burks

Martin Paszkiewicz

unread,

Jan 13, 2014, 3:50:53 PM1/13/14

to securit...@googlegroups.com

Doug,
Thank you kindly for your advice and time. I have 24 cores with 64GB ram. My thought on load averages is a value of 1 for a single cpu system is being fully utilized and anything over 1 would be processes waiting in queue. Please correct me if I am wrong, but with a 24 core system wouldn't a load average of higher than 24 indicate that there are processes waiting in the run queue? So, assuming that is correct would a load average of 22 indicate a 24 core system still has room to breathe?

I will look into applying a more robust BPF to clean up the noise being captured unless there is a better method in determining the cause of loss?

As an FYI...the inconsistent ELSA query results also occur during off-peak traffic periods (late in the evening) where monitored traffic hovers between 3 to 12 Mb/s and load averages are in the single digits.

Doug Burks

unread,

Jan 13, 2014, 4:00:38 PM1/13/14

to securit...@googlegroups.com

Replies inline.

On Mon, Jan 13, 2014 at 3:50 PM, Martin Paszkiewicz
<zulun...@gmail.com> wrote:
> Doug,
> Thank you kindly for your advice and time. I have 24 cores with 64GB ram. My thought on load averages is a value of 1 for a single cpu system is being fully utilized and anything over 1 would be processes waiting in queue. Please correct me if I am wrong, but with a 24 core system wouldn't a load average of higher than 24 indicate that there are processes waiting in the run queue? So, assuming that is correct would a load average of 22 indicate a 24 core system still has room to breathe?

That's generally correct, but if you're averaging 90% usage on each
CPU core and you have a traffic spike, your cores will quickly reach
100% usage and start dropping packets.

> I will look into applying a more robust BPF to clean up the noise being captured unless there is a better method in determining the cause of loss?

How much traffic are you monitoring?

> As an FYI...the inconsistent ELSA query results also occur during off-peak traffic periods (late in the evening) where monitored traffic hovers between 3 to 12 Mb/s and load averages are in the single digits.

Are you still running query_timeout 60? Have you tried increasing to
a higher number like 1000?

--
Doug Burks

Martin Paszkiewicz

unread,

Jan 13, 2014, 6:37:58 PM1/13/14

to securit...@googlegroups.com

Beautiful! Changing it to 1000 produced far more consistent results. I was thinking the timeout was in seconds, my error. I ended up setting it to 3000 and so far all the query counts are consistent! Syntax errors also seemed to have disappeared.

My environment has a 1Gbps link to the internet. I have 3 sensors on a standalone box monitoring outside of the firewall, behind the firewall + web content filters and one sensor for the DMZ. Traffic averages to about 300Mbps throughout the workday and rarely I have seen it spike to 600Mbps. The above sostat-redacted was taken during the highest avg peak of the day, 11am - 1pm.

Would this mean that I could potentially be monitoring up to 3Gbps aggregated? If so, perhaps my box is under powered or sorely needs a good tune.

Kind regards,
Martin Paszkiewicz

Martin Paszkiewicz

unread,

Jan 13, 2014, 7:09:55 PM1/13/14

to securit...@googlegroups.com

https://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation

Oh, I guess it is in seconds after all.. weird why 1000 seconds vs 3000 seconds makes more of a difference. I certainly do not wait that long for results. 20 to 30 seconds max for my box.

Doug Burks

unread,

Jan 13, 2014, 9:13:22 PM1/13/14

to securit...@googlegroups.com

I wonder if that Documentation page is referring to "timeout" being
passed in the query itself, but perhaps "query_timeout" set in
/etc/elsa_web.conf is measured in ms? Perhaps Martin Holste can
clarify this for us.

As far as tuning your box, here are some things you can try:

- top shows your system is swapping, so max out the RAM in the box

- trim your Suricata ruleset, eliminating all unnecessary rules

- purchase dedicated sensor box(es)

Matthew Thacker

unread,

Jan 15, 2014, 11:05:53 AM1/15/14

to securit...@googlegroups.com

Martin/Doug,
I'm having the same issues, especially the little red "Query syntax error" in Martin's screenshot. I also frequently get a mostly blank page with "0 node(s) with undefined logs..." pretty frequently when loading the page. I upped the query_timeout as you suggested about and it seems to have helped a little but it's still a pain. Any ideas?

sostat-redacted below for reference:

=========================================================================
Service Status
=========================================================================
Status: securityonion
* sguil server[ OK ]
Status: HIDS
* ossec_agent (sguil)[ OK ]
Status: Bro
Name Type Host Status Pid Peers Started

manager manager X.X.X.X running 14451 5 14 Jan 14:42:04
proxy proxy X.X.X.X running 14493 5 14 Jan 14:42:06
securityonion-eth1-1 worker X.X.X.X running 14579 2 14 Jan 14:42:08
securityonion-eth1-2 worker X.X.X.X running 14578 2 14 Jan 14:42:08
securityonion-eth1-3 worker X.X.X.X running 14580 2 14 Jan 14:42:08
securityonion-eth1-4 worker X.X.X.X running 14581 2 14 Jan 14:42:08
Status: securityonion-eth1

* netsniff-ng (full packet data)[ OK ]
* pcap_agent (sguil)[ OK ]
* snort_agent (sguil)[ OK ]
* suricata (alert data)[ OK ]
* barnyard2 (spooler, unified2 format)[ OK ]

* prads (sessions/assets)[ OK ]
* sancp_agent (sguil)[ OK ]
* pads_agent (sguil)[ OK ]
* argus[ OK ]
* http_agent (sguil)[ OK ]

=========================================================================
Interface Status
=========================================================================
eth0 Link encap:Ethernet HWaddr b8:ac:6f:16:19:9b

inet addr:X.X.X.X Bcast:X.X.X.X Mask:X.X.X.X

inet6 addr: fe80::baac:6fff:fe16:199b/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:122282236 errors:0 dropped:0 overruns:0 frame:0
TX packets:1160735 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:39590873041 (39.5 GB) TX bytes:417199831 (417.1 MB)
Interrupt:37 Memory:ec000000-ec012800

eth1 Link encap:Ethernet HWaddr b8:ac:6f:16:19:9d

UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1

RX packets:2440890325 errors:5610 dropped:1843 overruns:0 frame:5610

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000

RX bytes:1304735735030 (1.3 TB) TX bytes:0 (0.0 B)
Interrupt:37 Memory:ea000000-ea012800

lo Link encap:Local Loopback
inet addr:X.X.X.X Mask:X.X.X.X
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:55181529 errors:0 dropped:0 overruns:0 frame:0
TX packets:55181529 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:293199076880 (293.1 GB) TX bytes:293199076880 (293.1 GB)

=========================================================================
Link Statistics
=========================================================================
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
RX: bytes packets errors dropped overrun mcast

293199076880 55181529 0 0 0 0

RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns

293199076880 55181529 0 0 0 0

TX errors: aborted fifo window heartbeat
0 0 0 0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether b8:ac:6f:16:19:9b brd ff:ff:ff:ff:ff:ff

RX: bytes packets errors dropped overrun mcast

39590873041 122282236 0 0 0 3062

RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns

417199831 1160735 0 0 0 0

TX errors: aborted fifo window heartbeat
0 0 0 0

3: eth1: <BROADCAST,MULTICAST,NOARP,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether b8:ac:6f:16:19:9d brd ff:ff:ff:ff:ff:ff

RX: bytes packets errors dropped overrun mcast

1304735735030 2440890325 5610 1843 0 1650901

RX errors: length crc frame fifo missed

5610 0 0 0 0

TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0

=========================================================================
Disk Usage
=========================================================================
Filesystem Size Used Avail Use% Mounted on

/dev/sdb1 237G 42G 184G 19% /
udev 16G 4.0K 16G 1% /dev
tmpfs 6.3G 1.8M 6.3G 1% /run

none 5.0M 0 5.0M 0% /run/lock

none 16G 0 16G 0% /run/shm
/dev/sda1 14T 1.7T 13T 12% /nsm

=========================================================================
Network Sockets
=========================================================================
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

smbd 1257 root 27u IPv4 14346 0t0 TCP *:445 (LISTEN)
smbd 1257 root 28u IPv4 14348 0t0 TCP *:139 (LISTEN)
nmbd 1266 root 9u IPv4 7822 0t0 UDP *:137
nmbd 1266 root 10u IPv4 7823 0t0 UDP *:138
nmbd 1266 root 11u IPv4 7825 0t0 UDP X.X.X.X:137
nmbd 1266 root 12u IPv4 7826 0t0 UDP X.X.X.X:137
nmbd 1266 root 13u IPv4 7827 0t0 UDP X.X.X.X:138
nmbd 1266 root 14u IPv4 7828 0t0 UDP X.X.X.X:138
avahi-dae 1294 avahi 12u IPv4 7855 0t0 UDP *:5353
avahi-dae 1294 avahi 13u IPv6 7856 0t0 UDP *:5353
avahi-dae 1294 avahi 14u IPv4 7857 0t0 UDP *:56283
avahi-dae 1294 avahi 15u IPv6 7858 0t0 UDP *:34067
cupsd 1309 root 8u IPv6 7741005 0t0 TCP [::1]:631 (LISTEN)
cupsd 1309 root 9u IPv4 7741006 0t0 TCP X.X.X.X:631 (LISTEN)
sshd 1427 root 3u IPv4 8441 0t0 TCP *:22 (LISTEN)
sshd 1427 root 4u IPv6 8443 0t0 TCP *:22 (LISTEN)
xinetd 1571 root 5u IPv4 10597 0t0 UDP *:69
mysqld 1645 mysql 10u IPv4 14071 0t0 TCP X.X.X.X:3306 (LISTEN)
mysqld 1645 mysql 80u IPv4 7722275 0t0 TCP X.X.X.X:3306->X.X.X.X:34269 (ESTABLISHED)
ossec-csy 1664 ossecm 5u IPv4 14426 0t0 UDP X.X.X.X:50603->X.X.X.X:514
/usr/sbin 1767 www-data 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 1767 www-data 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 1767 www-data 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 1767 www-data 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
/usr/sbin 1770 www-data 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 1770 www-data 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 1770 www-data 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 1770 www-data 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
master 2164 root 12u IPv4 1992 0t0 TCP *:25 (LISTEN)
master 2164 root 13u IPv6 1993 0t0 TCP *:25 (LISTEN)
ntpd 2657 ntp 16u IPv4 18532 0t0 UDP *:123
ntpd 2657 ntp 17u IPv6 18533 0t0 UDP *:123
ntpd 2657 ntp 18u IPv4 18539 0t0 UDP X.X.X.X:123
ntpd 2657 ntp 19u IPv4 18540 0t0 UDP X.X.X.X:123
ntpd 2657 ntp 20u IPv6 18541 0t0 UDP [fe80::baac:6fff:fe16:199b]:123
ntpd 2657 ntp 21u IPv6 18542 0t0 UDP [::1]:123
/usr/sbin 8684 www-data 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 8684 www-data 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 8684 www-data 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 8684 www-data 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
/usr/sbin 8684 www-data 31u IPv4 8208257 0t0 TCP X.X.X.X:38887->X.X.X.X:3154 (CLOSE_WAIT)
/usr/sbin 8685 www-data 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 8685 www-data 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 8685 www-data 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 8685 www-data 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
/usr/sbin 8687 www-data 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 8687 www-data 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 8687 www-data 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 8687 www-data 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
barnyard2 9183 root 3u IPv4 8057879 0t0 TCP X.X.X.X:60595->X.X.X.X:8000 (ESTABLISHED)
barnyard2 9183 root 4u IPv4 7716827 0t0 TCP X.X.X.X:34269->X.X.X.X:3306 (ESTABLISHED)
/usr/sbin 10330 www-data 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 10330 www-data 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 10330 www-data 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 10330 www-data 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
/usr/sbin 10331 www-data 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 10331 www-data 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 10331 www-data 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 10331 www-data 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
/usr/sbin 10332 www-data 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 10332 www-data 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 10332 www-data 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 10332 www-data 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
/usr/sbin 11566 www-data 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 11566 www-data 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 11566 www-data 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 11566 www-data 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
/usr/sbin 11567 www-data 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 11567 www-data 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 11567 www-data 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 11567 www-data 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
searchd 12307 sphinxsearch 7u IPv4 6904804 0t0 TCP *:9306 (LISTEN)
searchd 12307 sphinxsearch 8u IPv4 6904805 0t0 TCP *:9312 (LISTEN)
/usr/sbin 12363 root 4u IPv4 6910263 0t0 TCP *:443 (LISTEN)
/usr/sbin 12363 root 5u IPv4 6910266 0t0 TCP *:9876 (LISTEN)
/usr/sbin 12363 root 6u IPv4 6910268 0t0 TCP *:3154 (LISTEN)
/usr/sbin 12363 root 7u IPv4 6910272 0t0 TCP *:444 (LISTEN)
bro 14451 root 4u IPv4 6913358 0t0 UDP X.X.X.X:43485->X.X.X.X:53
bro 14460 root 0u IPv4 6916064 0t0 TCP *:47761 (LISTEN)
bro 14460 root 1u IPv6 6916065 0t0 TCP *:47761 (LISTEN)
bro 14460 root 2u IPv4 6918357 0t0 TCP X.X.X.X:47761->X.X.X.X:45364 (ESTABLISHED)
bro 14460 root 4u IPv4 6913358 0t0 UDP X.X.X.X:43485->X.X.X.X:53
bro 14460 root 19u IPv4 6921223 0t0 TCP X.X.X.X:47761->X.X.X.X:45375 (ESTABLISHED)
bro 14460 root 21u IPv4 6919245 0t0 TCP X.X.X.X:47761->X.X.X.X:45376 (ESTABLISHED)
bro 14460 root 22u IPv4 6919251 0t0 TCP X.X.X.X:47761->X.X.X.X:45380 (ESTABLISHED)
bro 14460 root 23u IPv4 6919252 0t0 TCP X.X.X.X:47761->X.X.X.X:45381 (ESTABLISHED)
bro 14493 root 4u IPv4 6914854 0t0 UDP X.X.X.X:41927->X.X.X.X:53
bro 14502 root 0u IPv4 6916072 0t0 TCP X.X.X.X:45364->X.X.X.X:47761 (ESTABLISHED)
bro 14502 root 1u IPv4 6916075 0t0 TCP *:47762 (LISTEN)
bro 14502 root 2u IPv6 6916076 0t0 TCP *:47762 (LISTEN)
bro 14502 root 4u IPv4 6914854 0t0 UDP X.X.X.X:41927->X.X.X.X:53
bro 14502 root 19u IPv4 6913420 0t0 TCP X.X.X.X:47762->X.X.X.X:34654 (ESTABLISHED)
bro 14502 root 21u IPv4 6914949 0t0 TCP X.X.X.X:47762->X.X.X.X:34655 (ESTABLISHED)
bro 14502 root 22u IPv4 6914957 0t0 TCP X.X.X.X:47762->X.X.X.X:34656 (ESTABLISHED)
bro 14502 root 23u IPv4 6913441 0t0 TCP X.X.X.X:47762->X.X.X.X:34659 (ESTABLISHED)
bro 14578 root 4u IPv4 6916723 0t0 UDP X.X.X.X:41697->X.X.X.X:53
bro 14579 root 4u IPv4 6914938 0t0 UDP X.X.X.X:51395->X.X.X.X:53
bro 14580 root 4u IPv4 6917584 0t0 UDP X.X.X.X:46964->X.X.X.X:53
bro 14581 root 4u IPv4 6918375 0t0 UDP X.X.X.X:50607->X.X.X.X:53
bro 14608 root 0u IPv4 6917593 0t0 TCP X.X.X.X:45376->X.X.X.X:47761 (ESTABLISHED)
bro 14608 root 1u IPv4 6921224 0t0 TCP X.X.X.X:34655->X.X.X.X:47762 (ESTABLISHED)
bro 14608 root 2u IPv4 6921227 0t0 TCP *:47763 (LISTEN)
bro 14608 root 4u IPv4 6914938 0t0 UDP X.X.X.X:51395->X.X.X.X:53
bro 14608 root 19u IPv6 6921228 0t0 TCP *:47763 (LISTEN)
bro 14612 root 0u IPv4 6919242 0t0 TCP X.X.X.X:45375->X.X.X.X:47761 (ESTABLISHED)
bro 14612 root 1u IPv4 6919246 0t0 TCP X.X.X.X:34654->X.X.X.X:47762 (ESTABLISHED)
bro 14612 root 2u IPv4 6919249 0t0 TCP *:47764 (LISTEN)
bro 14612 root 4u IPv4 6916723 0t0 UDP X.X.X.X:41697->X.X.X.X:53
bro 14612 root 20u IPv6 6919250 0t0 TCP *:47764 (LISTEN)
bro 14615 root 0u IPv4 6914956 0t0 TCP X.X.X.X:34656->X.X.X.X:47762 (ESTABLISHED)
bro 14615 root 1u IPv4 6914959 0t0 TCP X.X.X.X:45381->X.X.X.X:47761 (ESTABLISHED)
bro 14615 root 2u IPv4 6914962 0t0 TCP *:47766 (LISTEN)
bro 14615 root 4u IPv4 6918375 0t0 UDP X.X.X.X:50607->X.X.X.X:53
bro 14615 root 20u IPv6 6914963 0t0 TCP *:47766 (LISTEN)
bro 14628 root 0u IPv4 6918398 0t0 TCP X.X.X.X:45380->X.X.X.X:47761 (ESTABLISHED)
bro 14628 root 1u IPv4 6918400 0t0 TCP X.X.X.X:34659->X.X.X.X:47762 (ESTABLISHED)
bro 14628 root 2u IPv4 6918403 0t0 TCP *:47765 (LISTEN)
bro 14628 root 4u IPv4 6917584 0t0 UDP X.X.X.X:46964->X.X.X.X:53
bro 14628 root 20u IPv6 6918404 0t0 TCP *:47765 (LISTEN)
tclsh 18649 root 13u IPv4 7639844 0t0 TCP *:7734 (LISTEN)
tclsh 18649 root 14u IPv4 7639845 0t0 TCP *:7736 (LISTEN)
tclsh 18649 root 15u IPv4 7918653 0t0 TCP X.X.X.X:7736->X.X.X.X:43819 (ESTABLISHED)
tclsh 18649 root 16u IPv4 7919023 0t0 TCP X.X.X.X:7736->X.X.X.X:43895 (ESTABLISHED)
tclsh 18649 root 17u IPv4 8047862 0t0 TCP X.X.X.X:7736->X.X.X.X:43924 (ESTABLISHED)
tclsh 18649 root 18u IPv4 7994472 0t0 TCP X.X.X.X:7736->X.X.X.X:43909 (ESTABLISHED)
tclsh 18649 root 19u IPv4 7911049 0t0 TCP X.X.X.X:7736->X.X.X.X:43867 (ESTABLISHED)
tclsh 18649 root 20u IPv4 7920548 0t0 TCP X.X.X.X:7736->X.X.X.X:43881 (ESTABLISHED)
tclsh 18649 root 21u IPv4 8162115 0t0 TCP X.X.X.X:7734->X.X.X.X:49314 (ESTABLISHED)
sshd 26103 root 3u IPv4 8146092 0t0 TCP X.X.X.X:22->X.X.X.X:53559 (ESTABLISHED)
sshd 26268 username 3u IPv4 8146092 0t0 TCP X.X.X.X:22->X.X.X.X:53559 (ESTABLISHED)
syslog-ng 27309 root 22u IPv4 7446874 0t0 TCP *:514 (LISTEN)
syslog-ng 27309 root 23u IPv4 7446875 0t0 UDP *:514
tclsh 30376 root 3u IPv4 7916747 0t0 TCP X.X.X.X:43819->X.X.X.X:7736 (ESTABLISHED)
tclsh 31154 root 3u IPv4 7916525 0t0 TCP X.X.X.X:43867->X.X.X.X:7736 (ESTABLISHED)
tclsh 31280 root 3u IPv4 7911076 0t0 TCP X.X.X.X:43881->X.X.X.X:7736 (ESTABLISHED)
tclsh 31460 root 3u IPv4 7922894 0t0 TCP X.X.X.X:43895->X.X.X.X:7736 (ESTABLISHED)
tclsh 31853 root 3u IPv4 8041667 0t0 TCP X.X.X.X:43909->X.X.X.X:7736 (ESTABLISHED)
tclsh 32040 root 3u IPv4 8031773 0t0 TCP X.X.X.X:43924->X.X.X.X:7736 (ESTABLISHED)
tclsh 32040 root 4u IPv4 8031779 0t0 TCP X.X.X.X:8000 (LISTEN)
tclsh 32040 root 6u IPv4 8031928 0t0 TCP X.X.X.X:8000->X.X.X.X:60595 (ESTABLISHED)

=========================================================================
IDS Rules Update
=========================================================================

Wed Jan 15 07:01:01 UTC 2014

Backing up current local_rules.xml file.
Cleaning up local_rules.xml backup files older than 30 days.
Backing up current downloaded.rules file before it gets overwritten.
Cleaning up downloaded.rules backup files older than 30 days.
Backing up current local.rules file before it gets overwritten.
Cleaning up local.rules backup files older than 30 days.
Running PulledPork.
http://code.google.com/p/pulledpork/
_____ ____
`----,\ )
`--==\\ / PulledPork v0.6.1 the Smoking Pig <////~
`--==\\/
.-~~~~-.Y|\\_ Copyright (C) 2009-2011 JJ Cummings
@_/ / 66\_ cumm...@gmail.com
| \ \ _(")
\ /-| ||'--' Rules give me wings!
\_\ \_\\
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Checking latest MD5 for snortrules-snapshot-2955.tar.gz....
They Match
Done!
Prepping rules from snortrules-snapshot-2955.tar.gz for work....
Done!
Checking latest MD5 for emerging.rules.tar.gz....
No Match
Done
Rules tarball download of emerging.rules.tar.gz....
They Match
Done!
Prepping rules from emerging.rules.tar.gz for work....

Done!
Reading rules...
Generating Stub Rules....
Done
Reading rules...
Reading rules...
Reading rules...
Processing /etc/nsm/pulledpork/enablesid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/dropsid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/disablesid.conf....

Modified 663 rules

Done
Modifying Sids....
Done!
Setting Flowbit State....

Enabled 110 flowbits
Enabled 1 flowbits
Enabled 1 flowbits

Done
Writing /etc/nsm/rules/downloaded.rules....
Done
Writing /etc/nsm/rules/so_rules.rules....
Done
Generating sid-msg.map....
Done
Writing /etc/nsm/rules/sid-msg.map....
Done
Writing /var/log/sid_changes.log....
Done
Rule Stats....

New:-------327
Deleted:---2
Enabled Rules:----17780
Dropped Rules:----0
Disabled Rules:---18762
Total Rules:------36542

Done
Please review /var/log/sid_changes.log for additional details
Fly Piggy Fly!
Restarting Barnyard2.

Restarting: securityonion-eth1

* stopping: barnyard2 (spooler, unified2 format)[ OK ]
* starting: barnyard2 (spooler, unified2 format)[ OK ]
Restarting IDS Engine.

Restarting: securityonion-eth1

* stopping: suricata (alert data)[ OK ]
* starting: suricata (alert data)[ OK ]

=========================================================================
CPU Usage
=========================================================================
top - 16:01:33 up 7 days, 19:20, 1 user, load average: 3.89, 2.97, 2.89
Tasks: 229 total, 3 running, 225 sleeping, 0 stopped, 1 zombie
Cpu(s): 15.4%us, 7.0%sy, 1.3%ni, 72.0%id, 4.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 33010160k total, 32659712k used, 350448k free, 148160k buffers
Swap: 50051312k total, 5231436k used, 44819876k free, 21392916k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

11564 root 20 0 510m 281m 3716 R 96 0.9 1:23.91 indexer
1645 mysql 20 0 4388m 203m 4984 S 24 0.6 1092:56 mysqld
14502 root 25 5 99864 22m 960 S 24 0.1 222:02.58 bro
14460 root 25 5 99864 25m 988 S 22 0.1 266:44.94 bro
14579 root 20 0 676m 609m 518m S 22 1.9 302:46.58 bro
14581 root 20 0 682m 613m 518m S 22 1.9 301:28.28 bro
14580 root 20 0 670m 604m 518m S 20 1.9 300:55.62 bro
14608 root 25 5 595m 536m 512m S 16 1.7 190:21.73 bro
14628 root 25 5 595m 535m 512m S 16 1.7 193:58.84 bro
14615 root 25 5 595m 535m 512m R 14 1.7 189:20.52 bro
9284 sguil 20 0 2688m 2.2g 387m S 12 7.1 57:19.79 Suricata-Main
14578 root 20 0 675m 609m 518m S 12 1.9 304:07.64 bro
4844 sguil 20 0 5044m 124m 3392 S 8 0.4 416:02.35 prads
14612 root 25 5 595m 535m 512m S 8 1.7 191:39.61 bro
23241 sguil 20 0 111m 20m 1120 S 4 0.1 21:14.53 argus
8712 root 20 0 0 0 0 S 2 0.0 0:00.34 kworker/2:1
12282 root 20 0 17336 1316 888 R 2 0.0 0:00.01 top
14451 root 20 0 318m 56m 3688 S 2 0.2 26:44.88 bro
23423 sguil 20 0 326m 292m 288m S 2 0.9 23:40.29 netsniff-ng
27309 root 20 0 897m 828m 2864 S 2 2.6 8:09.68 syslog-ng
1 root 20 0 24704 2200 1100 S 0 0.0 0:39.75 init
2 root 20 0 0 0 0 S 0 0.0 0:00.12 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:13.56 ksoftirqd/0
6 root RT 0 0 0 0 S 0 0.0 0:01.21 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:02.15 watchdog/0
8 root RT 0 0 0 0 S 0 0.0 0:01.60 migration/1
10 root 20 0 0 0 0 S 0 0.0 0:12.76 ksoftirqd/1
12 root RT 0 0 0 0 S 0 0.0 0:01.69 watchdog/1
13 root RT 0 0 0 0 S 0 0.0 0:02.05 migration/2
15 root 20 0 0 0 0 S 0 0.0 0:10.55 ksoftirqd/2
16 root RT 0 0 0 0 S 0 0.0 0:02.10 watchdog/2
17 root RT 0 0 0 0 S 0 0.0 0:02.06 migration/3
19 root 20 0 0 0 0 S 0 0.0 0:12.65 ksoftirqd/3
20 root RT 0 0 0 0 S 0 0.0 0:01.42 watchdog/3
21 root RT 0 0 0 0 S 0 0.0 0:01.85 migration/4
23 root 20 0 0 0 0 S 0 0.0 0:11.29 ksoftirqd/4
24 root RT 0 0 0 0 S 0 0.0 0:01.42 watchdog/4
25 root RT 0 0 0 0 S 0 0.0 0:01.76 migration/5
27 root 20 0 0 0 0 S 0 0.0 0:13.17 ksoftirqd/5
28 root RT 0 0 0 0 S 0 0.0 0:01.39 watchdog/5
29 root RT 0 0 0 0 S 0 0.0 0:01.41 migration/6
31 root 20 0 0 0 0 S 0 0.0 0:12.33 ksoftirqd/6
32 root RT 0 0 0 0 S 0 0.0 0:01.45 watchdog/6
33 root RT 0 0 0 0 S 0 0.0 0:01.64 migration/7
35 root 20 0 0 0 0 S 0 0.0 0:13.19 ksoftirqd/7
36 root RT 0 0 0 0 S 0 0.0 0:01.49 watchdog/7
37 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
38 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
39 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
40 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
42 root 20 0 0 0 0 S 0 0.0 0:00.95 sync_supers
43 root 20 0 0 0 0 S 0 0.0 0:00.03 bdi-default
44 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
45 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
46 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
47 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
48 root 0 -20 0 0 0 S 0 0.0 0:00.00 md
53 root 20 0 0 0 0 S 0 0.0 0:08.52 kworker/5:1
54 root 20 0 0 0 0 S 0 0.0 0:10.04 kworker/6:1
55 root 20 0 0 0 0 S 0 0.0 0:11.56 kworker/7:1
56 root 20 0 0 0 0 S 0 0.0 0:00.39 khungtaskd
57 root 20 0 0 0 0 S 0 0.0 25:37.14 kswapd0
58 root 20 0 0 0 0 S 0 0.0 15:05.32 kswapd1
59 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
60 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
61 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
62 root 20 0 0 0 0 S 0 0.0 0:00.00 ecryptfs-kthrea
63 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
71 root 0 -20 0 0 0 S 0 0.0 0:00.00 kthrotld
91 root 0 -20 0 0 0 S 0 0.0 0:00.00 devfreq_wq
236 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
246 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
251 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
255 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_3
259 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:2
260 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_4
261 root 20 0 0 0 0 S 0 0.0 0:06.22 kworker/u:3
306 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_5
332 root 0 -20 0 0 0 S 0 0.0 0:00.00 ttm_swap
405 root 20 0 0 0 0 S 0 0.0 0:10.06 kworker/3:2
413 root 20 0 0 0 0 S 0 0.0 10:19.14 jbd2/sdb1-8
414 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
607 root 20 0 0 0 0 S 0 0.0 0:03.50 kworker/1:3
636 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfs_mru_cache
637 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfslogd
638 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfsdatad
639 root 0 -20 0 0 0 S 0 0.0 0:00.00 xfsconvertd
640 root 20 0 0 0 0 S 0 0.0 0:15.07 xfsbufd/sda1
641 root 20 0 0 0 0 S 0 0.0 1:58.05 xfsaild/sda1
657 root 20 0 17236 376 376 S 0 0.0 0:00.07 upstart-udev-br
659 root 20 0 21956 584 584 S 0 0.0 0:00.06 udevd
769 root 0 -20 0 0 0 S 0 0.0 0:00.00 edac-poller
781 root 0 -20 0 0 0 S 0 0.0 0:00.00 kpsmoused
819 root 20 0 21960 224 224 S 0 0.0 0:00.00 udevd
826 root 0 -20 0 0 0 S 0 0.0 0:00.00 kmpathd
832 root 0 -20 0 0 0 S 0 0.0 0:00.00 kmpath_handlerd
945 root 20 0 15192 360 288 S 0 0.0 0:00.01 upstart-socket-
1257 root 20 0 119m 2536 2316 S 0 0.0 0:01.39 smbd
1263 messageb 20 0 24400 1240 748 S 0 0.0 0:00.15 dbus-daemon
1265 root 20 0 119m 476 380 S 0 0.0 0:00.06 smbd
1266 root 20 0 91264 948 836 S 0 0.0 0:05.36 nmbd
1278 root 20 0 21192 756 756 S 0 0.0 0:00.00 bluetoothd
1290 root 10 -10 0 0 0 S 0 0.0 0:00.00 krfcommd
1294 avahi 20 0 32308 1132 992 S 0 0.0 0:00.31 avahi-daemon
1295 avahi 20 0 32184 152 136 S 0 0.0 0:00.00 avahi-daemon
1309 root 20 0 101m 2668 2224 S 0 0.0 0:00.46 cupsd
1336 root 20 0 0 0 0 S 0 0.0 1:27.13 flush-8:16
1427 root 20 0 50036 1544 1436 S 0 0.0 0:00.01 sshd
1467 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/7:0
1524 root 20 0 20012 636 632 S 0 0.0 0:00.00 getty
1530 root 20 0 20012 636 632 S 0 0.0 0:00.00 getty
1548 root 20 0 20012 636 632 S 0 0.0 0:00.00 getty
1549 root 20 0 20012 636 632 S 0 0.0 0:00.00 getty
1552 root 20 0 20012 636 632 S 0 0.0 0:00.00 getty
1571 root 20 0 14972 624 620 S 0 0.0 0:00.00 xinetd
1580 root 20 0 4464 528 524 S 0 0.0 0:00.00 acpid
1584 root 20 0 264m 1068 1068 S 0 0.0 0:00.01 lightdm
1591 root 20 0 15984 584 484 S 0 0.0 1:55.68 irqbalance
1593 root 20 0 19116 824 688 S 0 0.0 0:07.56 cron
1594 daemon 20 0 16912 188 172 S 0 0.0 0:00.00 atd
1606 root 20 0 203m 2488 2016 S 0 0.0 10:48.37 Xorg
1664 ossecm 20 0 13052 536 436 S 0 0.0 0:03.52 ossec-csyslogd
1673 root 20 0 12808 340 320 S 0 0.0 0:00.10 ossec-execd
1687 root 20 0 4090m 2100 1868 S 0 0.0 0:00.23 console-kit-dae
1756 root 20 0 190m 2108 1512 S 0 0.0 0:00.11 polkitd
1766 ossec 20 0 14904 2396 664 S 0 0.0 0:06.47 ossec-analysisd
1767 www-data 20 0 410m 133m 7920 S 0 0.4 0:04.31 /usr/sbin/apach
1770 www-data 20 0 400m 122m 7816 S 0 0.4 0:04.13 /usr/sbin/apach
1771 root 20 0 4532 456 388 S 0 0.0 0:00.44 ossec-logcollec
1784 root 20 0 170m 1460 1460 S 0 0.0 0:00.00 lightdm
1787 root 20 0 118m 1928 1500 S 0 0.0 0:08.81 accounts-daemon
1817 lightdm 20 0 4404 504 500 S 0 0.0 0:00.00 lightdm-greeter
1825 lightdm 20 0 23956 156 156 S 0 0.0 0:00.00 dbus-daemon
1826 lightdm 20 0 235m 4344 2948 S 0 0.0 18:08.75 lightdm-gtk-gre
1828 lightdm 20 0 52408 1164 1164 S 0 0.0 0:00.00 gvfsd
1830 lightdm 20 0 203m 1068 1068 S 0 0.0 0:00.00 gvfs-fuse-daemo
1846 root 20 0 214m 1884 1580 S 0 0.0 0:00.08 upowerd
1872 root 20 0 5968 1980 604 S 0 0.0 9:32.52 ossec-syscheckd
1877 ossec 20 0 13072 580 408 S 0 0.0 0:00.45 ossec-monitord
2033 root 20 0 94664 1240 1236 S 0 0.0 0:00.00 lightdm
2164 root 20 0 25112 1256 1140 S 0 0.0 0:03.76 master
2170 postfix 20 0 27340 1332 1224 S 0 0.0 0:00.99 qmgr
2349 root 20 0 20012 636 632 S 0 0.0 0:00.00 getty
2359 root 20 0 0 0 0 S 0 0.0 24:49.88 flush-8:0
2657 ntp 20 0 37776 1376 1200 S 0 0.0 0:24.49 ntpd
3031 root 20 0 31652 852 680 S 0 0.0 0:01.25 screen
3032 root 20 0 25388 1684 1568 S 0 0.0 0:00.08 bash
3074 root 20 0 25388 1676 1564 S 0 0.0 0:00.06 bash
3257 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
4781 root 20 0 4344 256 256 S 0 0.0 0:00.00 tail
4957 root 19 -1 14892 360 300 S 0 0.0 0:46.02 dema
5016 www-data 20 0 567m 150m 3728 S 0 0.5 28:09.17 ruby
5053 postfix 20 0 38236 1472 1304 S 0 0.0 0:00.51 tlsmgr
5455 root 20 0 0 0 0 S 0 0.0 0:44.28 kworker/1:1
5480 root 20 0 0 0 0 S 0 0.0 0:00.45 kworker/0:1
6675 root 20 0 0 0 0 S 0 0.0 0:02.39 kworker/4:2
7504 root 20 0 11420 492 492 S 0 0.0 0:00.00 tail
7780 root 20 0 0 0 0 S 0 0.0 0:00.29 kworker/2:2
7816 root 20 0 4404 604 500 S 0 0.0 0:00.01 sh
7817 root 20 0 221m 42m 3876 S 0 0.1 37:27.58 perl
7949 root 20 0 0 0 0 S 0 0.0 0:04.00 kworker/1:2
8684 www-data 20 0 408m 131m 7780 S 0 0.4 0:03.20 /usr/sbin/apach
8685 www-data 20 0 409m 132m 7776 S 0 0.4 0:03.08 /usr/sbin/apach
8687 www-data 20 0 384m 112m 6772 S 0 0.3 0:02.77 /usr/sbin/apach
9183 root 20 0 208m 109m 1716 S 0 0.3 4:11.28 barnyard2
9530 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:0
10230 root 20 0 4404 596 492 S 0 0.0 0:00.00 sh
10233 root 20 0 4404 324 220 S 0 0.0 0:00.00 sh
10238 root 20 0 4312 352 272 S 0 0.0 0:00.00 sleep
10261 root 20 0 0 0 0 S 0 0.0 0:00.12 kworker/2:0
10262 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:2
10330 www-data 20 0 384m 111m 5488 S 0 0.3 0:02.69 /usr/sbin/apach
10331 www-data 20 0 387m 112m 5976 S 0 0.3 0:02.83 /usr/sbin/apach
10332 www-data 20 0 177m 10m 3556 S 0 0.0 0:00.00 /usr/sbin/apach
11192 root 20 0 68920 1844 1292 S 0 0.0 0:00.00 cron
11196 root 20 0 4404 604 500 S 0 0.0 0:00.00 sh
11200 root 20 0 234m 53m 3788 S 0 0.2 0:03.08 perl
11443 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/5:0
11566 www-data 20 0 177m 9m 3428 S 0 0.0 0:00.00 /usr/sbin/apach
11567 www-data 20 0 176m 7604 1312 S 0 0.0 0:00.00 /usr/sbin/apach
11961 postfix 20 0 27280 1712 1392 S 0 0.0 0:00.00 cleanup
11991 postfix 20 0 27188 1488 1212 S 0 0.0 0:00.00 trivial-rewrite
12003 postfix 20 0 27428 2244 1712 S 0 0.0 0:00.00 local
12020 root 20 0 78156 2316 1716 S 0 0.0 0:00.00 sudo
12021 root 20 0 16544 1172 984 S 0 0.0 0:00.00 sostat-redacted
12022 root 20 0 16568 1440 1212 S 0 0.0 0:00.00 sostat
12023 root 20 0 15744 816 688 S 0 0.0 0:00.00 sed
12237 root 20 0 11420 492 492 S 0 0.0 0:00.00 tail
12300 sphinxse 20 0 72924 1944 1396 S 0 0.0 0:00.00 su
12307 sphinxse 20 0 2484m 1.2g 953m S 0 3.7 8:51.20 searchd
12363 root 20 0 176m 11m 5860 S 0 0.0 0:02.38 /usr/sbin/apach
12365 root 20 0 215m 1496 1244 S 0 0.0 0:00.00 PassengerWatchd
12368 root 20 0 288m 1716 1432 S 0 0.0 0:00.56 PassengerHelper
12370 root 20 0 108m 7756 1880 S 0 0.0 0:00.07 ruby1.9.1
12373 nobody 20 0 165m 4028 3024 S 0 0.0 0:00.13 PassengerLoggin
14435 root 20 0 16580 1472 1240 S 0 0.0 0:00.00 bash
14481 root 20 0 16580 1476 1240 S 0 0.0 0:00.00 bash
14493 root 20 0 106m 38m 3632 S 0 0.1 14:24.74 bro
14529 root 20 0 16584 1480 1240 S 0 0.0 0:00.00 bash
14531 root 20 0 16584 1488 1240 S 0 0.0 0:00.00 bash
14537 root 20 0 16584 1484 1240 S 0 0.0 0:00.00 bash
14546 root 20 0 16584 1488 1240 S 0 0.0 0:00.00 bash
14604 root 20 0 0 0 0 Z 0 0.0 0:00.02 sh <defunct>
15640 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/4:1
16196 root 20 0 31652 568 564 S 0 0.0 0:00.03 screen
16197 root 20 0 25384 1412 1036 S 0 0.0 0:00.00 bash
16207 root 20 0 25388 1136 1132 S 0 0.0 0:00.03 bash
17142 root 20 0 11428 592 592 S 0 0.0 0:00.96 tail
18649 root 20 0 132m 9348 3884 S 0 0.0 5:28.34 tclsh
18664 root 20 0 126m 4376 880 S 0 0.0 0:01.64 tclsh
18665 root 20 0 126m 4036 548 S 0 0.0 0:00.00 tclsh
22490 root 20 0 21696 232 208 S 0 0.0 0:00.00 udevd
22513 root 20 0 0 0 0 S 0 0.0 0:02.31 kworker/6:0
26103 root 20 0 101m 4324 3268 S 0 0.0 0:00.01 sshd
26268 twsgadmi 20 0 101m 1824 768 S 0 0.0 0:00.00 sshd
26269 twsgadmi 20 0 32276 9252 1672 S 0 0.0 0:00.40 bash
27308 root 20 0 26784 428 192 S 0 0.0 0:00.00 syslog-ng
28644 postfix 20 0 27176 1568 1268 S 0 0.0 0:00.00 pickup
29929 root 20 0 11420 492 492 S 0 0.0 0:00.00 tail
30376 root 20 0 40600 5356 2972 S 0 0.0 0:35.09 tclsh
31154 root 20 0 40128 4856 2916 S 0 0.0 4:13.15 tclsh
31156 root 20 0 11444 712 600 S 0 0.0 0:01.55 tail
31280 root 20 0 43264 5872 2980 S 0 0.0 0:00.11 tclsh
31282 root 20 0 11440 612 516 S 0 0.0 0:00.00 tail
31460 root 20 0 39488 4212 2912 S 0 0.0 0:00.12 tclsh
31462 root 20 0 11424 356 276 S 0 0.0 0:00.00 cat
31853 root 20 0 41072 5820 2972 S 0 0.0 0:00.24 tclsh
32040 root 20 0 39620 4240 2916 S 0 0.0 0:01.34 tclsh
32059 root 20 0 11436 612 516 S 0 0.0 0:00.00 tail

=========================================================================
Log Archive
=========================================================================
/nsm/sensor_data/securityonion-eth1/dailylogs/ - 10 days
1.4T .
172G ./2014-01-06
44G ./2014-01-07
169G ./2014-01-08
169G ./2014-01-09
192G ./2014-01-10
75G ./2014-01-11
68G ./2014-01-12
203G ./2014-01-13
212G ./2014-01-14
107G ./2014-01-15

/nsm/bro/logs/ - 80 days
27G .
53M ./2013-10-28
197M ./2013-10-29
235M ./2013-10-30
227M ./2013-10-31
246M ./2013-11-01
175M ./2013-11-02
171M ./2013-11-03
262M ./2013-11-04
266M ./2013-11-05
331M ./2013-11-06
270M ./2013-11-07
283M ./2013-11-08
202M ./2013-11-09
200M ./2013-11-10
275M ./2013-11-11
259M ./2013-11-12
295M ./2013-11-13
278M ./2013-11-14
264M ./2013-11-15
194M ./2013-11-16
193M ./2013-11-17
275M ./2013-11-18
282M ./2013-11-19
265M ./2013-11-20
257M ./2013-11-21
262M ./2013-11-22
198M ./2013-11-23
193M ./2013-11-24
307M ./2013-11-25
276M ./2013-11-26
247M ./2013-11-27
181M ./2013-11-28
183M ./2013-11-29
171M ./2013-11-30
171M ./2013-12-01
260M ./2013-12-02
269M ./2013-12-03
279M ./2013-12-04
289M ./2013-12-05
278M ./2013-12-06
206M ./2013-12-07
187M ./2013-12-08
274M ./2013-12-09
260M ./2013-12-10
377M ./2013-12-11
369M ./2013-12-12
356M ./2013-12-13
250M ./2013-12-14
254M ./2013-12-15
364M ./2013-12-16
410M ./2013-12-17
381M ./2013-12-18
413M ./2013-12-19
373M ./2013-12-20
235M ./2013-12-21
231M ./2013-12-22
376M ./2013-12-23
249M ./2013-12-24
244M ./2013-12-25
414M ./2013-12-26
437M ./2013-12-27
246M ./2013-12-28
256M ./2013-12-29
325M ./2013-12-30
320M ./2013-12-31
268M ./2014-01-01
471M ./2014-01-02
380M ./2014-01-03
260M ./2014-01-04
258M ./2014-01-05
461M ./2014-01-06
745M ./2014-01-07
1.5G ./2014-01-08
1.5G ./2014-01-09
796M ./2014-01-10
677M ./2014-01-11
670M ./2014-01-12
865M ./2014-01-13
587M ./2014-01-14
226M ./2014-01-15
163M ./stats

=========================================================================
Bro netstats
=========================================================================
Average packet loss as percent across all Bro workers: 0.000000

securityonion-eth1-1: 1389801696.741410 recvd=100057905 dropped=0 link=100057905
securityonion-eth1-2: 1389801696.935205 recvd=104950650 dropped=0 link=104950650
securityonion-eth1-3: 1389801697.141240 recvd=104719640 dropped=0 link=104719640
securityonion-eth1-4: 1389801697.345302 recvd=97393710 dropped=0 link=97393710

=========================================================================
IDS Engine (suricata) packet drops
=========================================================================

/nsm/sensor_data/securityonion-eth1/stats.log
tcp.ssn_memcap_drop | RxPFReth16 | 0
tcp.segment_memcap_drop | RxPFReth16 | 0

=========================================================================
pf_ring stats
=========================================================================
PF_RING Version : 5.6.1 ($Revision: $)
Total rings : 10

Standard (non DNA) Options
Ring slots : 32768

Slot version : 15
Capture TX : Yes [RX+TX]
IP Defragment : No
Socket Mode : Standard
Transparent mode : Yes [mode 0]
Total plugins : 0

Cluster Fragment Queue : 296
Cluster Fragment Discard : 5976779

/proc/net/pf_ring/14578-eth1.2092
Appl. Name : <unknown>
Tot Packets : 104952513

Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

Min Num Slots : 65216
Num Free Slots : 65216

/proc/net/pf_ring/14579-eth1.2094
Appl. Name : <unknown>
Tot Packets : 100066557

Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

Min Num Slots : 65216
Num Free Slots : 65216

/proc/net/pf_ring/14580-eth1.2091
Appl. Name : <unknown>
Tot Packets : 104720927

Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

Min Num Slots : 65216
Num Free Slots : 65216

/proc/net/pf_ring/14581-eth1.2093
Appl. Name : <unknown>
Tot Packets : 97396022

Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

Min Num Slots : 65216
Num Free Slots : 65216

/proc/net/pf_ring/9323-eth1.2296
Appl. Name : Suricata
Tot Packets : 16630626

Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

Min Num Slots : 39012
Num Free Slots : 38960

/proc/net/pf_ring/9324-eth1.2297
Appl. Name : Suricata
Tot Packets : 17283725

Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

Min Num Slots : 39012
Num Free Slots : 38900

/proc/net/pf_ring/9325-eth1.2298
Appl. Name : Suricata
Tot Packets : 13138411

Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

Min Num Slots : 39012
Num Free Slots : 38924

/proc/net/pf_ring/9326-eth1.2299
Appl. Name : Suricata
Tot Packets : 13661294

Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

Min Num Slots : 39012
Num Free Slots : 39010

/proc/net/pf_ring/9327-eth1.2300
Appl. Name : Suricata
Tot Packets : 27964812
Tot Pkt Lost : 6416

TX: Send Errors : 0
Reflect: Fwd Errors: 0

Min Num Slots : 39012
Num Free Slots : 38992

/proc/net/pf_ring/9328-eth1.2301
Appl. Name : Suricata
Tot Packets : 15775189

Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0

Min Num Slots : 39012
Num Free Slots : 38966

=========================================================================
Netsniff-NG - Reported Packet Loss (per interval)
=========================================================================

0 Loss

Message has been deleted

Martin Paszkiewicz

unread,

Jan 15, 2014, 12:45:13 PM1/15/14

to securit...@googlegroups.com

Matthew,
I can try to assist until Doug, Martin Holste or someone else that has a deeper understanding of what is going on can chime in.

When you are running your queries are they within the index dates and/or archive date ranges? Verify by hovering your mouse over the from date field for a second or two. I was having the 0 node no query results problem but my problem was that I was trying to search outside of the available query ranges.

How many days are available to query from your index opposed to your archive?

Kind regards,
Martin Paszkiewicz

Matthew Thacker

unread,

Jan 15, 2014, 1:16:02 PM1/15/14

to securit...@googlegroups.com

Thanks for the reply!

The query syntax errors most frequently pop up without changing the dates from what they default to (48 hours ago) and it looks like the earliest indexed record is about 30 hours ago so that could be part of the problem. The time is well within the archived dates though, which seem to be 10 days ago (when I rebuilt this server). What do I change to keep them indexed for a longer period?

It also seems strange to me how quickly that little query syntax error box pops up. It's almost immediately there. If I keep hitting submit without changing anything it will work after a time or two and when it works the query always takes a few seconds but the failures are instant.

Have you managed to solve the problems you were having?

Martin Paszkiewicz

unread,

Jan 15, 2014, 1:56:14 PM1/15/14

to securit...@googlegroups.com

Matthew,
With Doug's guidance I reduced my ELSA problems so far (fingers-X-crossed) to nearly nil.

I don't want to steer you wrong because I am certainly no expert on the subject(please take that into consideration) but I used the following link as a guide:

http://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation

Adjust your elsa_node.conf accordingly to your box's available resources/HDD space.

You can compare your current elsa_node.conf to the following settings that I made. My settings may not be appropriate for your box and you could possibly fubar your box to a worse of state than it was before. Just saying...

------------

For /etc/elsa_node.conf

>>Under "archive"<<

"days": 90,
"percentage": 10.
"table_size": 10000000
},

------

"log_size_limit" : 1200000000000,

------

# Uncomment to establish a retention period in days for indexed logs
"days": 90,

------

"allowed_temp_percent" : 80,

------

"allowed_mem_percent": 60,

------

#How many concurrent log readers can run. 1 should be fine for up to 50k logs/sec on the same node.
"num_log_readers" : 3

------

# Stats retention settings
"retention_days": 365

------

I adjusted the log_size_limit to 4300000000000

adjusted perm_index_size from 10000000 to 100000000

adjusted index_interval from 60 to 30

------

REBOOT and Pray.

Kind regards,
Martin Paszkiewicz

Martin Paszkiewicz

unread,

Jan 15, 2014, 2:00:10 PM1/15/14

to securit...@googlegroups.com

What did you set your query_timeout to in /etc/elsa_web.conf
??

Matthew Thacker

unread,

Jan 15, 2014, 2:51:43 PM1/15/14

to securit...@googlegroups.com

Martin,
Ok I've bumped all those settings up as much as I dare. My box isn't quite as tricked out as your so I didn't go quite as high with my numbers. Will see if there's any change and reply back.

In reply to your next question - I set it to 1000 sometime yesterday. I saw shortly thereafter that you found the setting was in seconds not ms, but I never changed it back.

Martin Paszkiewicz

unread,

Jan 15, 2014, 3:38:35 PM1/15/14

to securit...@googlegroups.com

I am probably wrong that it is in seconds as Doug pointed out what I was referencing was a query syntax parameter in the documentation. I am still uncertain but think it would be beneficial to clarify.

I bumped mine to 3000 since setting it to 1000 was still giving me a bit of inconsistency with my query's total count results.

So far though after all the changes, ELSA has been solid for me. Still testing though...

Kind regards,
Martin Paszkiewicz

Matthew Thacker

unread,

Jan 15, 2014, 3:45:35 PM1/15/14

to securit...@googlegroups.com

Cool, I'll bump mine up then too.
Thanks for all your help!

Reply all

Reply to author

Forward