Logstash slow to restart

[Sam Wallace]

I've given logstash 10gb of memory. More workers then comes. I've tried using rngtools to increase the entropy. Verified I have the dev random as the entropy object. My logstash container still takes about ten minutes to restart the service and start receiving logs again.

I've reviewed the logstash tuning performance documentation, but not sure.

What are additional troubleshooting steps they I can preform?

[Mark W. Jeanmougin]

What does top look like while you're waiting for it to start?

I assume "sudo so-status" shows OK for everything except so-logstash which is WAIT?

If you run "time host www.google.com", what's the real time it takes to run?

MJ

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Wes Lambert]

Hi Sam,

How long does it take for Logstash to start in your case?

We are seeking to improve start time for Logstash and have attempted techniques like you have described (without any observable improvement), but given the large amount of mutates and other actions that occur in the config files, it still may take 5-10 minutes to fully initialize.  

This may improve in the future, as we consider transitioning to multiple pipelines, and as other options provided by the Elastic team become available

Thanks,

Wes

[Sam Wallace]

Logstash says that it is initializing. The cpu for this process sits at 100% while it tries to start. Once it is running the cpu sits at 20-30% while processing logs.

The real time to process is 0.258s. 

Wes - are you suggesting that if I remove templates and confs that I am not using that may potentially speed up loading logstash?

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/BBzjD5dDC9M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

[Wes Lambert]

Hi Sam,

Again, how long overall (minutes/seconds) is it taking for Logstash to fully initialize?

Yes, you may improve load time by removing some of the config files , but these may be replaced when running updates.

Thanks,

Wes

[Sam Wallace]

So I timed it and it takes about 9-10 minutes. This is from restarting to seeing the starting on 9600 message. Shortly after logs start to get processed again. 

On my other elk stack I can restart logstash in under thirty seconds

[Wes Lambert]

Hi Sam,

Comparatively, how many config files are you using with your other stack, and on average, what is the number of actions performed within each?

You can remove the symbolic links for config files you don't want (in /etc/logstash/conf.d) and reduce initialization time that way:

Ex.

~45 seconds...

Only running the following (14) config files vs the ~100 original files:

0000_input_syslogng.conf

1001_preprocess_syslogng.conf

1004_preprocess_syslog_types.conf

1115_preprocess_bro_ssh.conf

1116_preprocess_bro_ssl.conf

6000_bro.conf

8000_postprocess_bro_cleanup.conf

8001_postprocess_common_ip_augmentation.conf

8006_postprocess_dns.conf

8007_postprocess_http.conf

8200_postprocess_tagging.conf

8998_postprocess_log_elapsed.conf

8999_postprocess_rename_type.conf

9000_output_bro.conf

Thanks,

Wes

https://twitter.com/therealwlambert

https://securityonion.net/

[Sam Wallace]

The other stack has one giant config file with all three input, filter and output. Were you able to reproduce the long restart time with all of the configs enabled?

Sam

[Wes Lambert]

Hi Sam,

As mentioned before it is typical for Logstash in our case (Security Onion default config) to take 5-10 minutes to fully initialize.

Thanks,

Wes