Problem with elastic search, stops after 2 days

[Bart Van Hees]

Hello all,

I use security onion in a small network and the security onion is a virtual machine on vmware esxi with 4 cores, 16 GB ram and 1 TB disk space.

It receives from 8 devices logs and always after two days the Kibana is in status red for elastic search.

I already changed in securityonion.conf  log limit size for elasticsearch to 790 GB but this did not help.

What is strange is that 16 GB memory is always fully used

the so-stat-redacted file is in annex 

Can someone help me?

thanks in advance

Bart

[Wes Lambert]

Are there any clues in the Elasticsearch log?

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/8bd411de-4056-41d1-bc6b-41f65b42b045%40googlegroups.com.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[bartv...@gmail.com]

Hello Wes,

 

I see a lot of warnings regarding the failed to update shard information for clusterinfoupdatejob, but I am not sure if this is the cause.

But also this is in the time frame that elasticsearch stopped.

 

I placed the log in annex

 

Regards Bart

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/OEP7Iu2bekQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAHjBB6HS4sbDY3LcaHGLRXeQMmEUOwj2HmjYZNnnGDeE5bBz0Q%40mail.gmail.com.

[Bart Van Hees]

Hello Wes

I see a lot of warnings regarding the failed to update shard information for clusterinfoupdatejob, but I am not sure if this is the cause.

But also this is in the time frame that elasticsearch stopped.

regards

Bart

Op vrijdag 22 mei 2020 16:25:19 UTC+2 schreef Wes:

Are there any clues in the Elasticsearch log?

Thanks,

Wes

On Thu, May 21, 2020 at 2:38 AM Bart Van Hees <bartv...@gmail.com> wrote:

Hello all,

I use security onion in a small network and the security onion is a virtual machine on vmware esxi with 4 cores, 16 GB ram and 1 TB disk space.

It receives from 8 devices logs and always after two days the Kibana is in status red for elastic search.

I already changed in securityonion.conf  log limit size for elasticsearch to 790 GB but this did not help.

What is strange is that 16 GB memory is always fully used

the so-stat-redacted file is in annex 

Can someone help me?

thanks in advance

Bart

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to securit...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/8bd411de-4056-41d1-bc6b-41f65b42b045%40googlegroups.com.

[Bart Van Hees]

Hello Wes,

I think i found the problem, in the logstash.yml i placed the pipeline workers on 2 instead of default.

I placed it now back default and the problem has not appeared.

I wil see in a day or two

Op donderdag 21 mei 2020 08:38:18 UTC+2 schreef Bart Van Hees:

[Wes Lambert]

Hi Bart,

Yes, Logstash will try to use all available cores unless you limit those worker processes -- as you can see, it can be an issue affecting overall operation or other services sometimes.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/25ebaa80-fc84-4eae-a54e-3d729ffe166c%40googlegroups.com.