Squert and Sguil magically categorizing events

[Matt .]

Sorry in advance for the wordy post...

I've observed the following behavior on both of my standalone instances of SO, and after redoing installs with new images. One instance has fairly heavy traffic (300Gb a day of logs), the other is light traffic (5GB a day of logs). I keep both instances current. I've noticed this behavior since day one.

I've hoped the issues is somehow related to my servers having only 4GB of ram. I'll be replacing them with newer servers with 16GB of ram before long. But after today's frustration I decided to post something vs wait to see the behavior after the servers are replaced.

Sguil often groups many unrelated "404" events. So every day I end up clicking on "view correlated events" so that I can properly categorize them. This opens a new tab listing the individual events. If I close that tab before I categorize them all, several times the system magically categorizes the rest that were on the tab (incorrectly). I've worked around this in Sguil simply by not closing the tab until I've categorized them all.

I've also discovered with Squert if event grouping is left on, and I expand a grouped set of events, sometimes categorizing a sub group will result in the rest being magically categorized (incorrectly). I've worked around this by turning off event grouping.

Both of these issues have workarounds, but today something went wrong while categorizing via squert, the workaround apparently failed. So I ended up with thousands of incorrectly categorized events. It's too time consuming to sift through them all and fix, so last nights events will have to remain incorrectly categorized.

Any ideas?

Thanks,
Matt

[Bamm Visscher]

Alert aggregation/correlation is difficult because the state of a session doesn't manifest itself in the actual alert. Alerts are asymmetric and the source IP of a "to client" alert will actually be the address of the server. This messes with our aggregation based on the source IP within the Sguil client. Turning of aggregation completely doesn't sound like a good idea, but I can see if it's feasible to make it a server config option. The aggregation is done at the server side, so doing so will turn it off for all clients.

When you categorize a main alert, all events aggregated under it are catted also. Since the server doesn't know that you did the categorization outside of the realtime pane, it happily updates all alerts under it. You found the work around using "view correlated events". The key to this work around from the  is to understand that you are actually categorizing each aggregated alert twice. Once when the "main" event is catted, and then again when you update each alert individually. Hopefully understanding the magic helps.

Bamm

--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
sguil - The Analyst Console for NSM
http://sguil.sf.net

[Matt .]

So the moral of the story is, don't close any window/tab before all events are categorized.

Thanks,
Matt

[BBCan177]

On Monday, April 14, 2014 9:44:09 PM UTC-4, Bamm wrote:
> Alert aggregation/correlation is difficult because the state of a session doesn't manifest itself in the actual alert. Alerts are asymmetric and the source IP of a "to client" alert will actually be the address of the server. This messes with our aggregation based on the source IP within the Sguil client. Turning of aggregation completely doesn't sound like a good idea, but I can see if it's feasible to make it a server config option. The aggregation is done at the server side, so doing so will turn it off for all clients.
>
> When you categorize a main alert, all events aggregated under it are catted also. Since the server doesn't know that you did the categorization outside of the realtime pane, it happily updates all alerts under it. You found the work around using "view correlated events". The key to this work around from the  is to understand that you are actually categorizing each aggregated alert twice. Once when the "main" event is catted, and then again when you update each alert individually. Hopefully understanding the magic helps.

> Bamm

> sguil - The Analyst Console for NSM
> http://sguil.sf.net

"but I can see if it's feasible to make it a server config option."

SGUIL version 0.9.0

We can only ask... lol.... we would be lost without this great tool...

[Bamm Visscher]

Too late. 0.9.0 was released earlier this month. 

Bamm

--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

[Matt .]

While Sguil doesn't, Squert does sub-aggregate on IP. I wouldn't want to disable aggregation. I still make use of it's benefits daily closing dozens or hundreds of events at once all belonging to the same IP.

[BBCan177]

On Tuesday, April 15, 2014 5:34:16 PM UTC-4, Bamm wrote:
> Too late. 0.9.0 was released earlier this month. 

> Bamm
>

Great News... Can't wait to try it in Security Onion.