Using syscheck in production

[Mike Lissner]

I'm looking for advice about improving the signal/noise ratio for syscheck alerts. I just installed OSSEC and I'm loving it, but I know that if I can't improve the signal to noise ratio of syscheck, I'll have to turn it off.

As an example, yesterday I got an alert that sudoedit had changed. This is definitely from a OS update, and all the other alerts I've gotten from syscheck have been too. I know I'm going to start ignoring these alerts. At the same time, even if I'm vigilant, I'm concerned that once the OS updates this file three times, it'll auto-ignore itself, effectively disabling the system. Maybe that's OK, but it seems bad.

I want to pay attention to syscheck alerts, I think they're an important part of OSSEC (maybe not?), but I won't pay attention for long with this level of noise. How do folks deal with this so that it's a useful feature they don't just ignore in practice? Maybe the idea is to just keep a log of the changes and rely on other things to alert you of an intruder?

Mike

[Yana Zaeva]

Hi Mike,

The syscheck module can be kind of noisy, especially when you have loads of agents registered. However, you can play with the rules a little bit in order to adapt this module to your necessities and be alerted of the events that are of greater importance for you. You can ignore some files that you know that change quite a lot and monitor in realtime the ones that do not. 

Also, if you are concerned about not being alerted when the file was changed more than three times, you can change this option by changing <auto_ignore>yes</auto_ignore>  for <auto_ignore>no</auto_ignore>. If you are unable to find this option in the <syscheck> module, add it, as this option is set to yes by default. 

I will leave you some information about File Integrity Monitoring for further information:

- Syscheck configuration options: https://www.ossec.net/docs/manual/syscheck/index.html

- How syscheck works: https://documentation.wazuh.com/current/user-manual/capabilities/file-integrity/how-it-works.html

Let me know if you have any doubts. 

Regards, 

Yana.

[Mike Lissner]

Thanks Yana. I guess I should have mentioned I took a look at those settings and read the docs and that I'm sort of seeing this as a UX problem. Right now, I think the default UX of the syscheck module is bad enough (many false positives leading to ignored true positives) that the syscheck module isn't all that useful — which is truly a shame!

What I was thinking about was some way to:

1. Stop false positives (maybe by integrating with updating software somehow? maybe by disabling emails in OSSEC during the daily update scripts? I'm surprised there aren't some recipes here.)

2. Keep true positives (maybe stop ignoring alerts after the third time except on a few boring files? Or maybe stopping false positives is all that's needed to make this OK?)

Has any thought been put into this area? Seems really important to making the syscheck module trustworthy and useful instead of ignored and self-ignoring.

Thanks for everything,

Mike

--

---
You received this message because you are subscribed to a topic in the Google Groups "ossec-list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ossec-list/9WdcRoc4kto/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ossec-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ossec-list/163d61fb-9fa3-48e3-8c0b-ef3b8827f27cn%40googlegroups.com.

--

Mike Lissner

Executive Director

Free Law Project

https://free.law

[Scott Wozny]

My experience when doing tuning of a different file integrity checker a number of years ago led me to 2 approaches to maintaining my operational sanity.

1) The files showing changes on an average day I almost always turned off reporting for without guilt.  There were only so many cycles in a day and almost any file mod that happened as a matter of course that COULD have been from a bad actor would ALSO have required other modifications to produce a successful compromise AND a bunch of log data I was also using the tool to watch so while the security purist in me said, "That program shouldn't change that file in that way that prevents me from seeing a real compromise that would involve a change in that file" so much of the software we all use is out of our control so I would write the exception to stop the reporting, make a note on my "exceptions I hate" list and revisit the decision as part of my annual reviews.  Sometimes the thing got fixed that allowed me to turn monitoring back on for that file, but that was uncommon.  I just needed to come to peace with it.

2) The files showing changes during system updates I, too, wanted an automagic way to have the system differentiate for me, but I was never able to find a way and I ended up being OK with that.  For file changes during system patching, I knew when patch day was coming so I could "safely" ignore massive numbers of file change alerts at the time and then went through my summary report the next day, scrolled through it next to the list of software that had been updated (found on the change control form) and ended up being pretty confident no one had snuck anything in that wasn't expected.  You might call that "ignoring" messages, but operational reality is seldom so black and white and people need to make choices of where to expend their time in a sea of information.  The other reason that helped me sleep well with this approach is that it meant when someone did a change that did NOT go through the change control process on an average Tuesday, my tightly wound file integrity checker would spring into action and I could immediately track down whether it was a breach (it never ended up being a breach) or a tech who violated policy by skipping over the change control process (almost as big a threat as a breach). Either way, it was a win, but took a while of reminding people we were watching to stop them from skipping over the change control process they previously thought they could get away with.  Management support was absolutely necessary to make this happen.

So once I got past the psychological barrier of "I have to check every file integrity monitoring tool alert" after patch day the tool became really useful in a completely unexpected way.  Obviously, your situation will be different, but you asked how people deal with this and my answer was I used it to get patching and software installations under control and it worked great for that.  Perhaps OS updates occurring at unpredictable times (setting off OSSEC in unpredictable ways) is the issue you may want to address.

My 2 cents,

Scott

You received this message because you are subscribed to the Google Groups "ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ossec-list/CAKs1xOHT1mPFFyZwyZTKwP4oZxkfZ9kBm%3DDu5b1nKZbx%3DThEeA%40mail.gmail.com.

[Mike Lissner]

This is really helpful. So I guess it's a matter of identifying files that can just be ignored by ignore rules, letting things get auto-ignored, and scheduling system updates for a time you know they're happening.

One solution could be to use hooks before/after apt. Seems like this isn't totally crazy and apt has a solution for this:

https://unix.stackexchange.com/questions/401126/run-a-command-before-after-ubuntu-apt-upgrade-unattended-upgrades

If we had some basic scripts for this, would it be something that OSSEC maintainers would want to merge?

There's some talk about this problem over here: https://github.com/freedomofpress/securedrop/issues/1712, but no solutions. They point out that the solution of stopping and restarting OSSEC before/after apt would work, but would leave a window of vulnerability. I think that's an OK tradeoff. In fact, I think it's an improvement. Right now, regular updates + the auto_ignore rule slowly chip away at syscheck while spamming users. Eventually important files are auto-ignored, and users stop paying attention to alerts. If we did the above, we'd have a window of vulnerability, but we'd be in a much better place.

Thoughts?

Mike

To view this discussion on the web visit https://groups.google.com/d/msgid/ossec-list/CACUKT_qd2Zj97KtjyL7nykUjg2q_XScv1uGz%3D8gjQNcAUdiYOQ%40mail.gmail.com.