Elasticsearch curator

[Francois Lachance]

Hello all!

According to the SO wiki (https://github.com/Security-Onion-Solutions/security-onion/wiki/Curator), the curator should remove older indexes based on two parameters in the /etc/nsm/securityonion.conf configuration file:
- CURATOR_CLOSE_DAYS which is set at 30
- LOG_SIZE_LIMIT which is set at 183GB

And of course, the CURATOR_ENABLED="yes" is also in the configuration file.

Unfortunately, today I seem to have run out of disk space.

> du /nsm/elasticsearch -h -s
300G /nsm/elasticsearch

Alrigthy, let's take a look at the curator log in /var/log/curator/curator.log:

> cat //var/log/curator/curator.log
2018-06-04 21:14:01,657 INFO Preparing Action ID: 1, "delete_indices"
2018-06-04 21:14:01,673 INFO Trying Action ID: 1, "delete_indices": Delete indices when $disk_space value (in GB) is exceeded.
2018-06-04 21:14:01,707 INFO Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2018-06-04 21:14:01,707 INFO Action ID: 1, "delete_indices" completed.
2018-06-04 21:14:01,707 INFO Job completed.
2018-06-04 21:14:01,754 INFO Preparing Action ID: 1, "close"
2018-06-04 21:14:01,766 INFO Trying Action ID: 1, "close": Close indices older than 30 days (based on index name), for logstash- prefixed indices.
2018-06-04 21:14:01,792 INFO Skipping action "close" due to empty list: <class 'curator.exceptions.NoIndices'>
2018-06-04 21:14:01,792 INFO Action ID: 1, "close" completed.
2018-06-04 21:14:01,793 INFO Job completed.
2018-06-04 21:15:02,145 INFO Preparing Action ID: 1, "close"
2

<many many more lines later>

2018-07-24 21:56:01,435 INFO Preparing Action ID: 1, "close"
2018-07-24 21:56:01,442 INFO Trying Action ID: 1, "close": Close indices older than 30 days (based on index name), for logstash- prefixed indices.
2018-07-24 21:56:01,495 INFO Preparing Action ID: 1, "delete_indices"
2018-07-24 21:56:01,502 INFO Trying Action ID: 1, "delete_indices": Delete indices when $disk_space value (in GB) is exceeded.
2018-07-24 21:56:02,344 INFO Skipping action "close" due to empty list: <class 'curator.exceptions.NoIndices'>
2018-07-24 21:56:02,344 INFO Action ID: 1, "close" completed.
2018-07-24 21:56:02,344 INFO Job completed.
2018-07-24 21:56:02,438 INFO Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2018-07-24 21:56:02,438 INFO Action ID: 1, "delete_indices" completed.
2018-07-24 21:56:02,438 INFO Job completed.
2018-07-24 21:57:01,780 INFO Preparing Action ID: 1, "close"
2018-07-24 21:57:01,787 INFO Trying Action ID: 1, "close": Close indices older than 30 days (based on index name), for logstash- prefixed indices.
2018-07-24 21:57:01,841 INFO Preparing Action ID: 1, "delete_indices"
2018-07-24 21:57:01,848 INFO Trying Action ID: 1, "delete_indices": Delete indices when $disk_space value (in GB) is exceeded.
2018-07-24 21:57:02,634 INFO Skipping action "close" due to empty list: <class 'curator.exceptions.NoIndices'>
2018-07-24 21:57:02,634 INFO Action ID: 1, "close" completed.
2018-07-24 21:57:02,634 INFO Job completed.
2018-07-24 21:57:02,728 INFO Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>
2018-07-24 21:57:02,728 INFO Action ID: 1, "delete_indices" completed.
2018-07-24 21:57:02,728 INFO Job completed.

Looks like the Curator has never been able to delete any indices! Any idea what's going on? The empty list appears to be the reason why nothing gets deleted, but I don't know enough about docker to look into the Curator.

Thanks,

Francois

[Wes Lambert]

Francois,

What is the output of the following (from storage node, or box housing the indices)?

curl localhost:9200/_cat/indices?pretty

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Francois Lachance]

Wes,

See the attached file for the results.

Thanks,
Francois

[Wes Lambert]

Hi Francois,

From the output above, it looks like Curator is at least closing indices based on the 30 day threshold.  How is your disk setup?  Please provide the output of sostat redacted for the affected storage node/heavy node/standalone.

Thanks,

Wes

[Francois Lachance]

Wes,

Does closed indices actually free up the space on the hard drive?

I have attached the output of the sostat-redacted command.

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/7H_kfBGeFyI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

[Wes Lambert]

I don't believe they free up space.  If so, likely not much.  It's just that they are no longer actively searched when querying for data in Kibana, etc.

I was moreso referring to Curator itself behaving somewhat appropriately.  If you look in /etc/curator/action/delete.yml, what is the value for disk_space?  That value should be the threshold that is reached on the disk (and should match LOG_SIZE_LIMIT) for Elasticsearch data, before Curator deletes old indices.  

Thanks,

Wes

[Francois Lachance]

Does anyone know how I can invoke the curator manually with the --dry-run command line option? I need to figure out why it is not deleting indices to reduce my total disk space usage to the setting in securityonion.yml. It does appear that indices have been marked as closed, but for some reason they are still hanging around.

Thanks,

Francois

[Wes Lambert]

Hi Francois,

Try taking a look at /etc/cron.d/curator-delete.

Thanks,

Wes

[Tony Butt]

Wes, Francois,
I am seeing exactly the same situation.
Indices are being closed, but not deleted, and are showing the same message in curator.log
I have resorted to writing a script and manually running it monthly to delete older closed indexes.
Tony

[Wes]

Hi Francois,

Please see:

https://github.com/Security-Onion-Solutions/security-onion/issues/1340

We have a fix in the pipeline.

Thanks,
Wes

[Wes]

s/Francois/Tony/ .. sorry :)

Thanks!
Wes