Disappearing Suracata stats.log files

[Kevin Branch]

Hi,

I was just trying to get a fix on how many packets were being dropped by Suricata on an up to date SO standalone system when I could not find the stats.log file even though it is configured to be populated by suricata.yaml.  Upon closer inspection, after restarting Suricata the stats file would appear, but a while later it would disappear.   I then looked more broadly across all Suricata instances of a couple of other modern SO systems and stats.log was missing in all cases.    Even my older pre-Elastic SO systems exhibit this behavior.

However, when I check with lsof on all of these systems I see every one of the missing stats.log files are accounted for but marked "(deleted)".  I do not know what might be deleting them.

Is anyone else seeing this on their SO sensors?  Specifically, for those using Suricata under SO, on your SO sensor(s) do you get zero results from this command

ls -alh /nsm/sensor_data/*-*/stats.log

but also do see a list of one or more stats.log files accounted for in the output of this command?

lsof | grep stats.log | grep sensor_data | grep deleted

Thanks for your feedback,

Kevin

[Wes Lambert]

Hi Kevin,

I haven't noticed this in particular, but I will keep an eye out.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Doug Burks]

Hi Kevin,

I think this may have been due to nsm_sensor_ps-start and nsm_sensor_ps-restart doing an rm on the stats.log.  I'm updating those scripts to use truncate instead:

https://github.com/Security-Onion-Solutions/security-onion/issues/1456

On Sat, Feb 16, 2019 at 8:43 PM Kevin Branch <ke...@branchnetconsulting.com> wrote:

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Doug Burks
CEO
Security Onion Solutions, LLC

[Kevin Branch]

I noticed that rm in  nsm_sensor_ps-start and nsm_sensor_ps-restart but I wonder if that could be it, because when I first start Suricata with "/usr/sbin/nsm_sensor_ps-start --only-snort-alert", within a few seconds the expected stats.log files appear.  It isn't till some time later that those files are then deleted inexplicably.  I think it happens within an hour but I will have to watch more closely to get a feel for the specific timing.  I just restarted Suricata and the stats.log appeared and they are still there 5 minutes later.  I wonder if they will disappear at the top of the hour.  Will let you know.

Kevin

[Doug Burks]

I did find at least one way in which this issue could manifest.  If Suricata is currently running and then you run "so-nids-start" (this is the new wrapper for "/usr/sbin/nsm_sensor_ps-start --only-snort-alert"), then it would rm the stats.log but not start a new Suricata, so the current Suricata is still trying to write to the now-removed stats.log and no new stats.log is ever created.  Changing to truncate fixes at least this one condition.