ELSA tuning for large storage

[Brian Kellogg]

We have a bare metal box with a lot of storage dedicated to OSSEC and ELSA as a sensor to a SecurityOnion deployment.  Total disk is around 100TB.  Should I increase num_indexes or any other settings to ensure I use most of the disk?  Re-indexing all the data later to increase num_indexes will be out of the question.  I'm still fuzzy on what the setting "perm_index_size" does.  Thanks for any expert advice.  

I've read a lot on this but still a unsure what the best settings are for this box for ELSA and Sphinxsearch.

[Martin Holste]

100 TB is a large deployment, 10 TB would be pretty normal. So, if you have a lot of RAM (> 100 GB), you can probably get away with upping the default of 10 million for "perm_index_size" to something like 50 million. At the default 10 million and 200 permanent indexes ("num_indexes"), you get a max size per node of 2 billion events. I usually figure on an average event size of 800 bytes and a 2:1 inflation of the index to the original data, meaning you'd expect to max out at 2 billion * 800 * 2 = 3.2 TB. I have run with 400 as num_indexes as standard in production, so I know that value is ok. If you upped perm_index_size to 50 million, that would move your estimate up to 20 billion events on the node, which would be about 32 TB using the above model. I think some on this list have successfully gotten a num_indexes of 800 to work, but some activities will slow down a bit, and it's possible you could run into memory usage problems with some queries. If you tried 800 @ 50 million for the perm_index_size, you're up to 40 billion events at an estimated size of 64 TB. So, I don't think it's likely that you can fully use 100 TB of index on a single node and expect not to get weirdness on indexed queries. I would recommend ways of trying to split that disk amongst physical boxes or at least multiple VM's to keep the num_indexes reasonable.

Keep in mind that archive is not affected by any of this, it will just grow forever (up until the prescribed size), so you can let the remaining TB be archive (keep in mind that has an 8:1 compression ratio), so we're talking probably decades of storage, even at fairly high event rates.

On Fri, May 22, 2015 at 9:00 AM, Brian Kellogg <thef...@gmail.com> wrote:

We have a bare metal box with a lot of storage dedicated to OSSEC and ELSA as a sensor to a SecurityOnion deployment.  Total disk is around 100TB.  Should I increase num_indexes or any other settings to ensure I use most of the disk?  Re-indexing all the data later to increase num_indexes will be out of the question.  I'm still fuzzy on what the setting "perm_index_size" does.  Thanks for any expert advice.  

I've read a lot on this but still a unsure what the best settings are for this box for ELSA and Sphinxsearch.

--
You received this message because you are subscribed to the Google Groups "enterprise-log-search-and-archive" group.
To unsubscribe from this group and stop receiving emails from it, send an email to enterprise-log-search-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Kellogg]

Thanks for the great explanation.  Really appreciate it.

I have run 800 indexes without issue for about six months on a test box with 32GB RAM.  This box has 256GB of RAM.  We're trying to log to one box mainly so that we can write OSSEC event correlation rules against all of our agents and syslogs.

So if I want a years worth of logs indexed I can set the retention to 365 for sphinx and then set my archive percentage to what?  Also I upped the allowed_mem_percent to 50, should I go higher.  OSSEC uses very little RAM.

--
You received this message because you are subscribed to a topic in the Google Groups "enterprise-log-search-and-archive" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/enterprise-log-search-and-archive/Z-6YrCD_FkU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to enterprise-log-search-...@googlegroups.com.

[Martin Holste]

Archive percentage is of the log_size_limit, which is what the total disk footprint shares between archive and index. So, if you don't want any archive, set it to 0, otherwise, set log_size_limit to as much of the disk as you're willing to allocate to ELSA and set the archive to what you want. Setting a higher perm_index_size will give your 800 indexes more longevity, but they will probably still roll well before you hit 100 TB.