kibana shows 0 logs

[packetsmacker]

I have been testing kibana and noticed Squert has current alerts but kibana is missing a few weeks. How do I troubleshoot this?

[Wes Lambert]

packetsmacker,

Please provide the output of sostat-redacted, attaching as a plain text file, or using a service like Pastebin.com

Also, try checking the logs for Elasticsearch and Logstash:

/var/log/elasticsearch/

/var/log/logstash/

Also see:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Logstash

https://github.com/Security-Onion-Solutions/security-onion/wiki/Elasticsearch

Thanks,

Wes

On Thu, Oct 12, 2017 at 4:41 PM, packetsmacker <ott....@gmail.com> wrote:

I have been testing kibana and noticed Squert has current alerts but kibana is missing a few weeks. How do I troubleshoot this?

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[packetsmacker]

On Thursday, October 12, 2017 at 4:50:52 PM UTC-4, Wes wrote:
> packetsmacker,
>
>
> Please provide the output of sostat-redacted, attaching as a plain text file, or using a service like Pastebin.com
>
>
> Also, try checking the logs for Elasticsearch and Logstash:
>
>
> /var/log/elasticsearch/
> /var/log/logstash/
>
>
> Also see:
> https://github.com/Security-Onion-Solutions/security-onion/wiki/Logstash
>
> https://github.com/Security-Onion-Solutions/security-onion/wiki/Elasticsearch
>
>
>
>
>
> Thanks,
> Wes
>
>
> On Thu, Oct 12, 2017 at 4:41 PM, packetsmacker <ott....@gmail.com> wrote:
> I have been testing kibana and noticed Squert has current alerts but kibana is missing a few weeks. How do I troubleshoot this?
>
>
>
> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to the Google Groups "security-onion" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
>
> To post to this group, send email to securit...@googlegroups.com.

>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.

Looks like disk space issue. /nsm is at 91% I can't seem to recall were I change that in the config. This box is for testing so I don't really care if I only have a day or half day of logs in /nsm. I am just blanking out on were to change it. Thanks for the help.

[Wes Lambert]

CRIT_DISK_USAGE can be adjusted in /etc/nsm/securityonion.conf.  The cron job that runs to purge data clears the oldest data first.

However, since you are saying you are not receiving current alerts in Kibana, but are receiving current alerts in Squert, I would check to ensure logs are still getting through Logstash to Elasticsearch.

Have you checked the Elastic logs?

Thanks,

Wes

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[packetsmacker]

Trying to run sostat. It has seems to hang at the top 50 all time squil events. I am going to let it run for a while and see if it finishes. If it finishes I will upload it. Cleaning up space didn't seem to help. The warning went away but still not working correctly.

[packetsmacker]

This is the errors in logstash.log after running sudo so-elastic-start


[2017-10-18T20:46:22,819][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method `>' for nil:NilClass>, :backtrace=>["(eval):984802:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):984800:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):984852:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):984842:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):60584:in `filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:383:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:316:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:315:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:382:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:363:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:330:in `start_workers'"]}





[2017-10-18T20:46:22,972][ERROR][logstash.inputs.tcp ] An error occurred. Closing connection {:client=>"172.17.0.1:55390", :exception=>#<Errno::EBADF: Bad file descriptor - Bad file descriptor>, :backtrace=>["org/jruby/RubyIO.java:3019:in `sysread'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-4.1.2/lib/logstash/inputs/tcp.rb:282:in `read'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-4.1.2/lib/logstash/inputs/tcp.rb:206:in `handle_socket'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-4.1.2/lib/logstash/inputs/tcp.rb:195:in `server_connection_thread'"]}

[Wes Lambert]

packetsmacker,

Here are some things you could try:

-Clear the queue in /nsm/logstash/queue/main/ (sudo rm /nsm/logstash/queue/main/*); Restart Logstash (sudo docker stop logstash && sudo so-elastic-start)

-Disable DomainStats in /etc/nsm/securityonion.conf by setting "yes" to "no" for DOMAINSTATS_ENABLED; then run sudo so-elastic-restart

-Increase the size of the queue in /etc/logstash/logstash.yml (queue.max_bytes); then run sudo docker stop so-logstash , then sudo so-elastic-start

-Increase the number of logstash workers or heap size;  then run sudo docker stop so-logstash , then sudo so-elastic-start

https://github.com/Security-Onion-Solutions/security-onion/wiki/Logstash#configuration

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[packetsmacker]

So here is what I did.

Clear queue
Set DomainStats to no
Increased queue size to 2gb
Increased workers to 2

Restarted with so-elastic-start and now I have some logs showing up. I wasn't sure what to set the queue and workers to but I figured small increase was better then a large one. My heap size it set at the max per the wiki link you posted.

Thanks for the help!