Redis logstash Queue keeps increasing

[Ryan Hall]

I am not very familiar with REDIS but I believe this is the bottle neck causing my problems.

I have a distributed deployment with 2 forward nodes, 1 master, and 2 storage nodes. When I watch the Redis queue on the master it keeps climbing slowly. I'm currently at about 600,000 and rising. Every once in awhile I stop seeing nodes in Kibana and when I look at the /var/log/logstash/logstash.log file I see errors that contain: "OOM command not allowed when used memory > ‘maxmemory’". I increased the maxmemory to 8GB, which I'm not sure if I should have done that but that seems to help as I don't lose nodes in Kibana as often, but I do still lose them sometimes.

I was wondering if you have any ideas about what might be the problem? I have followed the commands on the Security Onion Redis wiki page about draining the queue and that seem to get things back in order but doesnt solve the greater problem. Should I create another storage node? Would that help drain the queue faster?

Thanks for any assistance!

Ryan

[Wes Lambert]

Hi Ryan,

A couple of things you could look at before adding a new storage node:

-Trying to get the storage nodes to pull from the queue faster

-Trying to get Redis to push out more records on each pull

Try taking a look at the redis input on the storage node(s) and the redis output on the master and adjusting the options there.

Elastic has some great documentation here for said config:

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-redis.html

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Ryan Hall]

Hi Wes,

Thanks for the Links!

Off the top of your head, do you know what commands I might need to run to pull from the queue faster or push out more? Since I'm not familiar with Redis, I'm not even sure how to get that information. Are the fields in the links suppose to go in the redis.conf file or set via redis-cli?

Thanks!

Ryan

[Wes Lambert]

Hi Ryan,

These settings will live in the Redis output config on the master server and the input config on the storage node(s).  The links reference various settings you could use to tune.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Ryan Hall]

Thanks Wes!

I believe I found the config files:
- master: /etc/logstash/conf.d.redis.output/9999_output_redis.conf
- Storage: /etc/logstash/conf.d/0900_input_redis.conf

It looks like both files support Threads and Batch_count Options. Would these be the options you would change?

Do you have a recommendation on which one you would prefer to use over the other? Also, any guidance for how many threads or what Batch_count value would be appropriate to try?

Thanks so much!

Ryan

[Wes Lambert]

Hi Ryan,

I would try experimenting with several options to see what works best for you.

The documentation should provide some background on each:

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-redis.html

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[dan....@gmail.com]

Ryan, did you have any luck with re-configuring Redis? I believe I'm having a similar problem, however their documentation isn't very explicit :|.

[Ryan Hall]

Hi Dan,

I have been playing with threads and batch_count on the storage nodes. I am currently up to 10 threads a piece for my 2 storage nodes and 250 batch count which I believe is double the default. This has definitely helped some but I'm still have problems during heaving load.

I also found out that when my storage nodes were on there own VM data store I didnt have nearly as many problems. So I might be having an I/O problem with my storage, which we are looking at as well.

Here is my /etc/logstash/conf.d/0900_input_redis.conf at the moment:

input {
redis {
host => '172.18.0.1'
data_type => 'list'
key => 'logstash:redis'
type => 'redis-input'
threads => 10
batch_count => 250
}
}

Hope this helps and good luck!

Ryan