Question about Elastic Stack disk management / archive data
Tom Wood
I've been noticing elastic-search filling the disk in the few test sensors I have deployed. As far as I can see there isn't any equivalent to log_size_limit in the config files. How is the storage management performed? currently it just seems to stop processing new data.
From what I can see curator is supposed to handle this but something doesn't seem to be working. Before I rebuild does this mean anything?:
2017-11-10 13:04:01,912 ERROR Schema error: Configuration: filter: Location: Action ID "1", action "delete_indices", filter #1: {'source': 'creation_date', 'use_age': True, 'filtertype': 'space'}: Bad Value: "(could not determine)", required key not provided @ data['disk_space']. Check configuration file.
Similarly with ELSA we had the concept of archives to store compressed unindexed data. Is there something similar with Elastic? If so can we expect similar compression ratios? From what I can see with the elastic documentation curator snapshots are the closest.
Thanks,
Tom
Wes
Tom,
What version of Security Onion on Elastic are you running?
If you are still running Alpha, you may want to try installing Beta:
http://blog.securityonion.net/2017/11/elastic-stack-beta-release-and-security.html
If you are already running Beta:
What is the output of the following?
cat /etc/curator/action/delete.yml
In regard to your question about archives -- Elastic archives have to be closed and then later re-opened -- I do not believe the ability to do this from within Kibana is currently available. Since we like to have as much information immediately searchable as possible, we are currently only maintaining the indices and deleting as space limitations require. You could certainly utilize the close.yml (to close indices) as you desire, but it is not currently supported/recommended configuration.
We will likely investigate a more robust approach to this in the future.
Thanks,
Wes
Tom Wood
Hi Wes,
Upgraded from Alpha -> Beta so that might be the issue.
cat /etc/curator/action/delete.yml
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True. If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
1:
action: delete_indices
description: >-
Delete indices when $disk_space value (in GB) is exceeded.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: logstash-
- filtertype: space
source: creation_date
use_age: True
disk_space:
Looks like the disk space variable might just be missing.
I will take a look at using close.yml thanks. For us it's important to have a balance of searchable data and preserving historical data for a set period.
Doug Burks
When you upgraded from Alpha to Beta, did you run "sudo so-elastic-configure"?
For more information, please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Elastic-Alpha-to-Beta
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.
--
Doug Burks
Tom Wood
> Hi Tom,
>
> When you upgraded from Alpha to Beta, did you run "sudo so-elastic-configure"?
>
> For more information, please see:
> https://github.com/Security-Onion-Solutions/security-onion/wiki/Elastic-Alpha-to-Beta
>
Hi Doug,
I thought I did but I've just reran it and now curator is now functioning.
Thanks,
Tom