Receiving message "stale PID file found, process will be restarted at the next 5-minute interval!"
1,869 views
Skip to first unread message
epson...@verizon.net
Aug 12, 2014, 9:26:27 AM8/12/14
to securit...@googlegroups.com
I have been running Security Onion for several months but am finding that I have to re-install it periodically because it just stops working. Currently, when I run a "sudo service nsm status," the two Snort processes indicate "stale PID file found, process will be restarted at the next 5-minute interval!" I saw a post on this from last year, but there didn't seem to be any resolution. I ran the command "tail -100 /var/log/nsm/securityonion/sguild.log" but all that it shows is a lot of archived alerts such as:
2014-08-12 13:12:18 pid(2357) Archived Alert: 0 2 bad-unknown SecurityOnion-eth1-1 {2014-08-10 18:40:30} 3 127456 {ET POLICY Outbound MSSQL Connection to Standard port (1433)} <source ip> <dest ip> 6 63587 1433 1 2013410 4 1645 1645
Heine Lysemose
Aug 12, 2014, 9:56:09 AM8/12/14
to securit...@googlegroups.com
Hi
Please include, sudo sostat-redacted, and lets have a look at your system.
Regards,
Lysemose
On Tue, Aug 12, 2014 at 3:26 PM, <epson...@verizon.net> wrote:
I have been running Security Onion for several months but am finding that I have to re-install it periodically because it just stops working. Currently, when I run a "sudo service nsm status," the two Snort processes indicate "stale PID file found, process will be restarted at the next 5-minute interval!" I saw a post on this from last year, but there didn't seem to be any resolution. I ran the command "tail -100 /var/log/nsm/securityonion/sguild.log" but all that it shows is a lot of archived alerts such as:
2014-08-12 13:12:18 pid(2357) Archived Alert: 0 2 bad-unknown SecurityOnion-eth1-1 {2014-08-10 18:40:30} 3 127456 {ET POLICY Outbound MSSQL Connection to Standard port (1433)} <source ip> <dest ip> 6 63587 1433 1 2013410 4 1645 1645
--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
Shane Castle
Aug 12, 2014, 10:02:11 AM8/12/14
to securit...@googlegroups.com
Yes, and look in /var/log/nsm/<sensor>/snortu-[1-2].log for the reasons
your Snorts died.
In the past, one reason that happened to me was a change in the rules
that caused the Snorts to barf, sometimes owing to a fault on my part (a
change meant that a rule update failed and left bad syntax in the rule)
and sometimes owing to a failure in the downloaded rules.
Another thing that can trigger this is poor disk space maintenance.
--
Mit besten Grüßen
Shane Castle
On 12.08.2014 15:56, Heine Lysemose wrote:
> Hi
>
> Please include, sudo sostat-redacted, and lets have a look at your system.
>
> Regards,
> Lysemose
>
>
> On Tue, Aug 12, 2014 at 3:26 PM, <epson...@verizon.net
> <mailto:epson...@verizon.net>> wrote:
>
> I have been running Security Onion for several months but am finding
> that I have to re-install it periodically because it just stops
> working. Currently, when I run a "sudo service nsm status," the two
> Snort processes indicate "stale PID file found, process will be
> restarted at the next 5-minute interval!" I saw a post on this from
> last year, but there didn't seem to be any resolution. I ran the
> command "tail -100 /var/log/nsm/securityonion/sguild.log" but all
> that it shows is a lot of archived alerts such as:
>
> 2014-08-12 13:12:18 pid(2357) Archived Alert: 0 2 bad-unknown
> SecurityOnion-eth1-1 {2014-08-10 18:40:30} 3 127456 {ET POLICY
> Outbound MSSQL Connection to Standard port (1433)} <source ip> <dest
> ip> 6 63587 1433 1 2013410 4 1645 1645
>
> --
> You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to security-onio...@googlegroups.com
> <mailto:security-onion%2Bunsu...@googlegroups.com>.
your Snorts died.
In the past, one reason that happened to me was a change in the rules
that caused the Snorts to barf, sometimes owing to a fault on my part (a
change meant that a rule update failed and left bad syntax in the rule)
and sometimes owing to a failure in the downloaded rules.
Another thing that can trigger this is poor disk space maintenance.
--
Mit besten Grüßen
Shane Castle
On 12.08.2014 15:56, Heine Lysemose wrote:
> Hi
>
> Please include, sudo sostat-redacted, and lets have a look at your system.
>
> Regards,
> Lysemose
>
>
> On Tue, Aug 12, 2014 at 3:26 PM, <epson...@verizon.net
> <mailto:epson...@verizon.net>> wrote:
>
> I have been running Security Onion for several months but am finding
> that I have to re-install it periodically because it just stops
> working. Currently, when I run a "sudo service nsm status," the two
> Snort processes indicate "stale PID file found, process will be
> restarted at the next 5-minute interval!" I saw a post on this from
> last year, but there didn't seem to be any resolution. I ran the
> command "tail -100 /var/log/nsm/securityonion/sguild.log" but all
> that it shows is a lot of archived alerts such as:
>
> 2014-08-12 13:12:18 pid(2357) Archived Alert: 0 2 bad-unknown
> SecurityOnion-eth1-1 {2014-08-10 18:40:30} 3 127456 {ET POLICY
> Outbound MSSQL Connection to Standard port (1433)} <source ip> <dest
> ip> 6 63587 1433 1 2013410 4 1645 1645
>
> --
> You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to security-onio...@googlegroups.com
> To post to this group, send email to securit...@googlegroups.com
> <mailto:securit...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.
>
>
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to security-onio...@googlegroups.com
> <mailto:security-onio...@googlegroups.com>.
> You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to security-onio...@googlegroups.com
> To post to this group, send email to securit...@googlegroups.com
> <mailto:securit...@googlegroups.com>.
epson...@verizon.net
Aug 12, 2014, 10:12:44 AM8/12/14
to securit...@googlegroups.com
At the end of the most recent snortu-1.log, I see:
[ Port Based Pattern Matching Memory ]
+- [ Aho-Corasick Summary ] -------------------------------------
| Storage Format : Full-Q
| Finite Automaton : DFA
| Alphabet Size : 256 Chars
| Sizeof State : Variable (1,2,4 bytes)
| Instances : 200
| 1 byte states : 182
| 2 byte states : 18
| 4 byte states : 0
| Characters : 205105
| States : 90611
| Transitions : 5282478
| State Density : 22.8%
| Patterns : 14135
| Match States : 11201
| Memory (MB) : 50.46
| Patterns : 1.47
| Match Lists : 4.58
| DFA
| 1 byte states : 1.34
| 2 byte states : 42.65
| 4 byte states : 0.00
+----------------------------------------------------------------
[ Number of patterns truncated to 20 bytes: 3560 ]
pfring DAQ configured to passive.
ERROR: Can't initialize DAQ pfring (-1) -
Fatal Error, Quitting..
[ Port Based Pattern Matching Memory ]
+- [ Aho-Corasick Summary ] -------------------------------------
| Storage Format : Full-Q
| Finite Automaton : DFA
| Alphabet Size : 256 Chars
| Sizeof State : Variable (1,2,4 bytes)
| Instances : 200
| 1 byte states : 182
| 2 byte states : 18
| 4 byte states : 0
| Characters : 205105
| States : 90611
| Transitions : 5282478
| State Density : 22.8%
| Patterns : 14135
| Match States : 11201
| Memory (MB) : 50.46
| Patterns : 1.47
| Match Lists : 4.58
| DFA
| 1 byte states : 1.34
| 2 byte states : 42.65
| 4 byte states : 0.00
+----------------------------------------------------------------
[ Number of patterns truncated to 20 bytes: 3560 ]
pfring DAQ configured to passive.
ERROR: Can't initialize DAQ pfring (-1) -
Fatal Error, Quitting..
Heine Lysemose
Aug 12, 2014, 10:27:11 AM8/12/14
to securit...@googlegroups.com
Try this, https://groups.google.com/forum/m/#!topic/security-onion/iUx-bWpfOQU
And please remember to update Security Onion with sudo soup to avoid similar errors.
Regards,
Lysemose
--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
epson...@verizon.net
Aug 12, 2014, 10:38:54 AM8/12/14
to securit...@googlegroups.com
Could this be a result of manually upgrading the HWE stack? I did that because I kept seeing messages indicating that the HWE I had was no longer supported. When I ran the proposed fix in the link you provided, here is what happened:
sudo apt-get install --reinstall securityonion-pfring-module
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/83.8 kB of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 183560 files and directories currently installed.)
Preparing to replace securityonion-pfring-module 20121107-0ubuntu0securityonion10 (using .../securityonion-pfring-module_20121107-0ubuntu0securityonion10_all.deb) ...
Stopping: HIDS
* stopping: ossec_agent (sguil) [ OK ]
Stopping: SecurityOnion-eth1
* stopping: netsniff-ng (full packet data) [ OK ]
* stopping: pcap_agent (sguil) [ OK ]
* stopping: snort_agent-1 (sguil) [ OK ]
* stopping: snort_agent-2 (sguil) [ OK ]
* stopping: snort-1 (alert data) (not running) [ WARN ]
- stale PID file found, deleting!
* stopping: snort-2 (alert data) (not running) [ WARN ]
- stale PID file found, deleting!
* stopping: barnyard2-1 (spooler, unified2 format) [ OK ]
* stopping: barnyard2-2 (spooler, unified2 format) [ OK ]
* stopping: prads (sessions/assets) [ OK ]
* stopping: sancp_agent (sguil) [ OK ]
* stopping: pads_agent (sguil) [ OK ]
Waiting 5 seconds for processes to terminate gracefully.....
Killing any remaining processes using pf_ring.
Removing pf_ring from /etc/modules...done.
Removing pf_ring from running kernel...ERROR: Module pf_ring does not exist in /proc/modules
done.
Removing pf_ring from DKMS...done.
Unpacking replacement securityonion-pfring-module ...
Setting up securityonion-pfring-module (20121107-0ubuntu0securityonion10) ...
sudo apt-get install --reinstall securityonion-pfring-module
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/83.8 kB of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 183560 files and directories currently installed.)
Preparing to replace securityonion-pfring-module 20121107-0ubuntu0securityonion10 (using .../securityonion-pfring-module_20121107-0ubuntu0securityonion10_all.deb) ...
Stopping: HIDS
* stopping: ossec_agent (sguil) [ OK ]
Stopping: SecurityOnion-eth1
* stopping: netsniff-ng (full packet data) [ OK ]
* stopping: pcap_agent (sguil) [ OK ]
* stopping: snort_agent-1 (sguil) [ OK ]
* stopping: snort_agent-2 (sguil) [ OK ]
* stopping: snort-1 (alert data) (not running) [ WARN ]
- stale PID file found, deleting!
* stopping: snort-2 (alert data) (not running) [ WARN ]
- stale PID file found, deleting!
* stopping: barnyard2-1 (spooler, unified2 format) [ OK ]
* stopping: barnyard2-2 (spooler, unified2 format) [ OK ]
* stopping: prads (sessions/assets) [ OK ]
* stopping: sancp_agent (sguil) [ OK ]
* stopping: pads_agent (sguil) [ OK ]
Waiting 5 seconds for processes to terminate gracefully.....
Killing any remaining processes using pf_ring.
Removing pf_ring from /etc/modules...done.
Removing pf_ring from running kernel...ERROR: Module pf_ring does not exist in /proc/modules
done.
Removing pf_ring from DKMS...done.
Unpacking replacement securityonion-pfring-module ...
Setting up securityonion-pfring-module (20121107-0ubuntu0securityonion10) ...
Creating symlink /var/lib/dkms/pf_ring/5/source ->
/usr/src/pf_ring-5
DKMS: add completed.
Kernel preparation unnecessary for this kernel. Skipping...
Building module:
cleaning build area....
make KERNELRELEASE=3.13.0-33-generic -C /lib/modules/3.13.0-33-generic/build M=/var/lib/dkms/pf_ring/5/build.....(bad exit status: 2)
Error! Bad return status for module build on kernel: 3.13.0-33-generic (x86_64)
Consult /var/lib/dkms/pf_ring/5/build/make.log for more information.
FATAL: Module pf_ring not found.
dpkg: error processing securityonion-pfring-module (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
securityonion-pfring-module
E: Sub-process /usr/bin/dpkg returned an error code (1)
Shane Castle
Aug 12, 2014, 10:41:28 AM8/12/14
to securit...@googlegroups.com
Also, your allocated disk space seems very low if you are going to get
an average of ~80BG of data per day. And you need to remember to
categorize your sguil alerts.
--
Mit besten Grüßen
Shane Castle
> <mailto:security-onion%2Bunsu...@googlegroups.com>.
an average of ~80BG of data per day. And you need to remember to
categorize your sguil alerts.
--
Mit besten Grüßen
Shane Castle
On 12.08.2014 16:27, Heine Lysemose wrote:
> Try this,
> https://groups.google.com/forum/m/#!topic/security-onion/iUx-bWpfOQU
>
> And please remember to update Security Onion with sudo soup to avoid
> similar errors.
>
> Regards,
> Lysemose
>
> On Aug 12, 2014 4:18 PM, <epson...@verizon.net
> Try this,
> https://groups.google.com/forum/m/#!topic/security-onion/iUx-bWpfOQU
>
> And please remember to update Security Onion with sudo soup to avoid
> similar errors.
>
> Regards,
> Lysemose
>
> On Aug 12, 2014 4:18 PM, <epson...@verizon.net
> To post to this group, send email to securit...@googlegroups.com
> <mailto:securit...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.
>
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to security-onio...@googlegroups.com
> <mailto:security-onio...@googlegroups.com>.
> You received this message because you are subscribed to the Google
> Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to security-onio...@googlegroups.com
> To post to this group, send email to securit...@googlegroups.com
> <mailto:securit...@googlegroups.com>.
Heine Lysemose
Aug 12, 2014, 10:47:41 AM8/12/14
to securit...@googlegroups.com
Yes the current pfring module is not compatible with newer kernels.
There's currently testing in progress with a version of pfring, snort and suricata and the new HTC stack. The new packages are in RC-state.
You should revert to get things back on track.
Also follow Shane's recommendations.
Regards,
Lysemose
epson...@verizon.net
Aug 12, 2014, 11:48:09 AM8/12/14
to securit...@googlegroups.com
Is there a way to revert to the original HWE stack without having to re-install Ubuntu?
Heine Lysemose
Aug 12, 2014, 12:43:23 PM8/12/14
to securit...@googlegroups.com
I actually don't know. If I was you I would try Google for an answer...
/Lysemose
On Aug 12, 2014 5:52 PM, <epson...@verizon.net> wrote:
Is there a way to revert to the original HWE stack without having to re-install Ubuntu?
epson...@verizon.net
Aug 12, 2014, 12:56:08 PM8/12/14
to securit...@googlegroups.com
Okay. Thank you for your help!
Reply all
Reply to author
Forward
0 new messages