SO RC1 - Missing Data in Kibana

[Josh Silvestro]

Hello!

I'm having a weird issue on two different boxes. I received alerts from sguil over the past 2 days but in kibana looking for event_type: snort returns only returns STUN alerts. But no other type of alert? I know that doesn't make sense, but that's what's happening. I look at Squert and I can see a few alerts all non-OSSEC, and when I search event_type: snort and look for the SID or IPs, nada. I even tried pivoting from Squert and there's no data returned. I also expanded my time frame to 7 days.

Last time I had an issue like this back in Alpha, it was due to the HDD filling up. However, just looked and I'm sitting at 29%, so doesn't appear to be the issue.

Thoughts? I can do a re-install, but I have two boxes experiencing the same issue.

[Philip Robson]

Are you seeing any events in Kibana?

I had an issue in which our backups were causing a high load on the server, the logstash queue would build up and logs would stop. Looking at Timelion the logs would spike then stop a little later. I found the traffic and blocked it using the pcap bpf conf.

To get it up and running again i had to:
sudo rm /nsm/logstash/queue/main/*
(make sure it is empty after running)
sudo docker stop so-logstash
sudo so-elastic-restart

Increasing the workers and queue in /etc/logstash/logstash.yml helped a little but once more backups ran it happened again.

If its snort in kibana thats stopped then i found it was an issue with my local rules or threshold (type)

In all instances sguil would keep on working.

[Josh Silvestro]

I am seeing events in Kibana, just not Snort. I get an alert I can go in and see bro & firewall data (piped in via syslog) for that IP. Look for a snort event for that IP or SID, nada.

I did just pull down the latest docker images, maybe that will have an affect?

In regards to your snort in kibana, I'm in the same boat. What do you mean in regards to issues in your threshold? As context, it's the same threshold that I had weeks ago pre-RC1 without issue.

[Philip Robson]

Sometimes it was caused by a type in threshold.conf, sometimes i found that restarting the service with sudo nsm_sensor_ps-restart --only-snort-alert would create a warning, found i had to do a rule update and restart all the nsm sensor services and it started populating again.

I am new to this but i seem to be working it out.

[Josh Silvestro]

So to touch back on this, I'm still getting time outs and bad gateways no matter what I change. I even bumped up memory and CPU and still get the issues. Luckily, I'm running RC1 next to a legacy ELSA installation so I can failback. But with the current RC1 issues I'm experiencing it's been unusable. I have two clients who were interested in seeing elastic capabilities so I stood up an RC1 box at each location for demo. But last night 1 client experienced a massive attack and trying to "event_type: firewall AND source_ip: "X.X.X.X" over 12 hours causes a timeout and no data returned. So I went to elsa and was able to find the data I needed. I stood up a Beta 3 box and have not had these issues. So I'm not sure if it's the change in elastic version or something else. But unfortunately if RC1 was the primary version currently I don't think I'd be using it.

It's quite possible it's something I'm doing, but again, in all Beta version I had not had these issues. :\

[Doug Burks]

Hi Josh,

Please keep in mind all the warnings and disclaimers that have applied
to our entire Elastic release cycle including RC1:
http://blog.securityonion.net/2018/01/security-onion-elastic-stack-release.html

Experimental Setup is BLEEDING EDGE and TOTALLY UNSUPPORTED!
If this breaks your system, you get to keep both pieces!
This is a work in progress and is in constant flux.
This is intended to build a quick prototype proof of concept so you
can see what our ultimate Elastic configuration might look like. This
configuration will change drastically over time leading up to the
final release.
Do NOT run this on a system that you care about!
Do NOT run this on a system that has data that you care about!
This should only be run on a TEST box with TEST data!
Experimental Setup may result in nausea, vomiting, or a burning sensation.

As I mentioned to you over on the following thread, I have not been
able to duplicate your issue:
https://groups.google.com/d/topic/security-onion/xxmIitQBcPQ/discussion

Can you provide more information about your installation?

Please run the following command:

sudo sostat-redacted

There will be a lot of output, so you may need to increase your
terminal's scroll buffer OR redirect the output of the command to a
file:

sudo sostat-redacted > sostat-redacted.txt 2>&1

sostat-redacted will automatically redact any IPv4/IPv6/MAC addresses,
but there may be additional sensitive info that you still need to
redact manually.

Attach the output to your email in plain text format (.txt) OR use a
service like http://pastebin.com.

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks

[Josh Silvestro]

Doug,

Thanks for the response, and I do understand it's bleeding edge. I apologize if it came off as snarky or angry, I'm not! Was just noting that in Beta releases I did not have this issue, and as a "Release Candidate" it's currently unusable for me. Thanks for all the work you do on this.

Please see the attached file.

[Doug Burks]

Looking at your sostat output, I don't see much out of the ordinary
other than this:

Total Documents: 61838976
Total Size: 28171MB

Those numbers seem high for the amount of network data this box is
seeing. Are you sending other types of data to the box?

What is the output of the following?
curl http://localhost:9200/_cat/indices

[Josh Silvestro]

On top of network captures, there are hosts sending in OSSEC data. One host sending beats data.

yellow open elastalert_status_silence vuQDke2MRrKv2xhQ5EeCQQ 5 1 1755 0 260.7kb 260.7kb
green open logstash-bro-2018.02.08 ki2dCxpFQgWbjsNKOwGYUQ 1 0 3369822 0 2.1gb 2.1gb
green open logstash-bro-2018.02.11 zo6V8GPNT4SqFY9eDQgkMQ 1 0 482839 0 333.6mb 333.6mb
green open logstash-ids-2018.02.12 mRN8eipTTn-NiDyONEp62g 1 0 1 0 23.3kb 23.3kb
green open logstash-bro-2018.02.12 XIGzlYtkTU2OHblgdHLgpg 1 0 400301 0 280.9mb 280.9mb
green open logstash-syslog-2018.02.07 CgGsAesiSmyLXmAvZP3frQ 1 0 4232010 0 1.4gb 1.4gb
green open logstash-syslog-2018.02.10 casSYnthT_qxdC-ZX51ucw 1 0 4755269 0 1.9gb 1.9gb
green open logstash-ids-2018.02.07 kf1TFnm8TSec6LDEQOVAWA 1 0 220 0 256.5kb 256.5kb
green open logstash-firewall-2018.02.12 ZLcwJLpbQjeW14TYciRNMg 1 0 145755 0 99.9mb 99.9mb
green open logstash-ids-2018.02.09 wlcBJhTcQIODayF8xvXI4Q 1 0 79 0 191kb 191kb
yellow open elastalert_status_status iwTQGe1qSRWDWP9VvJmSlw 5 1 154316 0 40.9mb 40.9mb
green open logstash-ids-2018.02.08 iy47P3ZOR1-5QvQlHmtZpQ 1 0 289 0 232.9kb 232.9kb
green open logstash-beats-2018.02.12 eHzfn5EURgu8irwSjDFh5Q 1 0 995095 0 497.9mb 497.9mb
green open logstash-firewall-2018.02.10 -J-9tAfiR_a3JiOjowB9QQ 1 0 244946 0 164.8mb 164.8mb
green open logstash-firewall-2018.02.08 a-ugsRt4Tx231X8PG2tJLg 1 0 830507 0 611.6mb 611.6mb
green open logstash-bro-2018.02.07 rfBNU6awTXmp1jBtzO0npA 1 0 1153124 0 765.7mb 765.7mb
yellow open elastalert_status i18xO3AeSIyc4LUPt6YSXg 5 1 1754 0 422.1kb 422.1kb
green open logstash-beats-2018.02.07 9mfhSZdgSSmkXRD-qv1VpQ 1 0 475282 0 115.3mb 115.3mb
yellow open elastalert_status_error S2jKIlAZT0G7v6yMFPiVKw 5 1 6 0 45.6kb 45.6kb
green open logstash-beats-2018.02.11 OTQcmCdXTQ6Q2bwcWpZwEg 1 0 1083487 0 503.1mb 503.1mb
green open logstash-bro-2018.02.10 dH1o9dPNQXKvMTnZs-8MYw 1 0 714008 0 462.6mb 462.6mb
green open logstash-firewall-2018.02.07 BKcEnrBpTLOWgMOMh0ZGNA 1 0 267100 0 214.2mb 214.2mb
green open logstash-beats-2018.02.10 VotlNpsbRAm0zATs49HWhA 1 0 1811513 0 824.1mb 824.1mb
green open logstash-ids-2018.02.11 3zy43wuIRHGShoeCJG3xEg 1 0 56 0 188.6kb 188.6kb
yellow open elastalert_status_past 2Jl3QT5wS9KM9mAXKW6fFQ 5 1 0 0 1.2kb 1.2kb
green open logstash-firewall-2018.02.09 zhPN_4XeQgiuFejz-OefPQ 1 0 791948 0 578.7mb 578.7mb
green open logstash-bro-2018.02.09 cChlV2YhTGaPAvZNg2_lEA 1 0 3078774 0 1.9gb 1.9gb
yellow open .kibana t-WGwJjDQwyAIyaa5rHpIg 1 1 491 42 1.1mb 1.1mb
green open logstash-syslog-2018.02.09 583g4uJYSBSbOEaPKaSyuA 1 0 12932159 0 4.5gb 4.5gb
green open logstash-beats-2018.02.08 kvrNqt7tTRGjYY_4FeXaTQ 1 0 805739 0 196.9mb 196.9mb
green open logstash-beats-2018.02.09 8uzB21yDQxe6RDx-H4S4hg 1 0 2616379 0 1.2gb 1.2gb
green open logstash-syslog-2018.02.08 vp0ZT6wcTYSPY-5NbI6YVA 1 0 14633753 0 5gb 5gb
green open logstash-firewall-2018.02.11 bdPuWaMjSLGS4LO4UqFipA 1 0 251420 0 155.3mb 155.3mb
green open logstash-ids-2018.02.10 5L5g6LehQNqes1Yox81s3Q 1 0 2 0 56.6kb 56.6kb
green open logstash-syslog-2018.02.12 jybduVabTzK7wUrBlSIJ-Q 1 0 2830022 0 1.1gb 1.1gb
green open logstash-syslog-2018.02.11 nPww-PmEQTGyHDp8c2yfLw 1 0 3765353 0 1.4gb 1.4gb

[Doug Burks]

Have you tried closing the large indices to see if that makes any difference?

[Josh Silvestro]

I have not. I see that you can curl them closed, is that the suggested method?

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
> To post to this group, send email to security-onion@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/IZChj7b3BcU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thank You,

Joshua Silvestro

[Doug Burks]

Yes.

>> > an email to security-onio...@googlegroups.com.
>> > To post to this group, send email to securit...@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/security-onion.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Doug Burks
>>
>> --
>> Follow Security Onion on Twitter!
>> https://twitter.com/securityonion
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "security-onion" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/security-onion/IZChj7b3BcU/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> security-onio...@googlegroups.com.
>> To post to this group, send email to securit...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Thank You,
> Joshua Silvestro
>
>

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.