SO RC1 - Multiple logstash-* listed in Index Patterns

[Josh Silvestro]

New RC1 looks great! Following initial setup I went in to management, because personally I set sample to 250, popular to 0, and default time period to 8 hours.

Regardless, I noticed that under Index Patterns, I have 4 *:logstash-* index patterns and one set as default. I've done nothing at this point but run the so-setup and add one logstash conf file, which I'd used in the past SO releases.

Should this happen? Can I safely delete the other 3?

[Josh Silvestro]

Also, for almost every search I run I get

Discover: Request Timeout after 30000ms
Error: Request Timeout after 30000ms
at https://X.X.X.X/bundles/kibana.bundle.js?v=16363:61:163257
at https://X.X.X.X/bundles/kibana.bundle.js?v=16363:61:163678

[Wes Lambert]

Josh,

I'm not sure why there are mutliple *:logstash-* index patterns in Kibana, but you should be able to remove the non-default one(s) without issue.  In regard to the timeout issue, have you tried simply restarting with 'sudo so-elastic-restart'?

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Josh Silvestro]

OK, removed multiple indexes, seems to be OK still. I have already done an so-elastic-restart and a full reboot of the machine. I just gave it another go and still get the same timeout issue. 

On Tue, Jan 30, 2018 at 9:48 AM, Wes Lambert <wlamb...@gmail.com> wrote:

Josh,

I'm not sure why there are mutliple *:logstash-* index patterns in Kibana, but you should be able to remove the non-default one(s) without issue.  In regard to the timeout issue, have you tried simply restarting with 'sudo so-elastic-restart'?

Thanks,

Wes

On Tue, Jan 30, 2018 at 9:30 AM, 'Josh Silvestro' via security-onion <security-onion@googlegroups.com> wrote:

Also, for almost every search I run I get

Discover: Request Timeout after 30000ms
Error: Request Timeout after 30000ms
    at https://X.X.X.X/bundles/kibana.bundle.js?v=16363:61:163257
    at https://X.X.X.X/bundles/kibana.bundle.js?v=16363:61:163678

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/xxmIitQBcPQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thank You,

Joshua Silvestro

[Wes Lambert]

Josh,

Try tailing the following to look for clues:

/var/log/logstash/logstash.log

/var/log/elasticsearch/hostname.log

Also, try taking a look for errors in /var/log/nsm/sosetup.log.

Thanks,

Wes

--

Thank You,

Joshua Silvestro

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.

[Josh Silvestro]

OK, so on a what if it's me hunch. I went and set the sample size down to 150 from 250 and it appears to have fixed the issue. 250 always worked for me in prior releases, but I'll keep an eye on this and see if it continues to work.

[Josh Silvestro]

So I'm still getting timeout issues. I've taken the sample size down to 50 now and still having time outs. Unless I'm improperly using it, a default sample size of 10 seems not always helpful. If a port scan or FTP brute force is attempted and I can only view 10 out of 100 events, I don't truly get a good grasp of the situation and if additional action is needed.

Not sure if it's just related to the latest version of Elasticsearch, because I fired up the Beta 3 and it ran fine no issues still with 250 sample size. Back to RC1 and a sample size of 50 yells at me.

[Josh Silvestro]

On Wednesday, January 31, 2018 at 12:58:47 PM UTC-5, Josh Silvestro wrote:
> So I'm still getting timeout issues. I've taken the sample size down to 50 now and still having time outs. Unless I'm improperly using it, a default sample size of 10 seems not always helpful. If a port scan or FTP brute force is attempted and I can only view 10 out of 100 events, I don't truly get a good grasp of the situation and if additional action is needed.
>
> Not sure if it's just related to the latest version of Elasticsearch, because I fired up the Beta 3 and it ran fine no issues still with 250 sample size. Back to RC1 and a sample size of 50 yells at me.

I've confirmed I can duplicate the issue consistently with "Sample Size" changes. If I have 100+ it will time out. If I have (at this time) 50 or less it's fine. But again, with the data I've testing with, 300 events, 50 or less show blocked, but I can't really know about the other 200+ without modifying down to finite time frames.

Or am I just going about it the wrong way?

[Doug Burks]

Hi Josh,

I just tested in a VM with discover:sampleSize 500 and I'm not getting
any timeout errors. Are you able to duplicate this on a fresh
installation?

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--

Doug Burks

[Josh Silvestro]

Doug - I can try a fresh install, but this is actually a second install separate from the original that's having the issues. 

On Wed, Jan 31, 2018 at 1:18 PM, Doug Burks <doug....@gmail.com> wrote:

Hi Josh,

I just tested in a VM with discover:sampleSize 500 and I'm not getting
any timeout errors.  Are you able to duplicate this on a fresh
installation?

On Wed, Jan 31, 2018 at 1:08 PM, 'Josh Silvestro' via security-onion
<security-onion@googlegroups.com> wrote:
> On Wednesday, January 31, 2018 at 12:58:47 PM UTC-5, Josh Silvestro wrote:
>> So I'm still getting timeout issues. I've taken the sample size down to 50 now and still having time outs. Unless I'm improperly using it, a default sample size of 10 seems not always helpful. If a port scan or FTP brute force is attempted and I can only view 10 out of 100 events, I don't truly get a good grasp of the situation and if additional action is needed.
>>
>> Not sure if it's just related to the latest version of Elasticsearch, because I fired up the Beta 3 and it ran fine no issues still with 250 sample size. Back to RC1 and a sample size of 50 yells at me.
>
> I've confirmed I can duplicate the issue consistently with "Sample Size" changes. If I have 100+ it will time out. If I have (at this time) 50 or less it's fine. But again, with the data I've testing with, 300 events, 50 or less show blocked, but I can't really know about the other 200+ without modifying down to finite time frames.
>
> Or am I just going about it the wrong way?
>
> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
> To post to this group, send email to security-onion@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/xxmIitQBcPQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thank You,

Joshua Silvestro

[Doug Burks]

Hi Josh,

We tracked down the multiple logstash-* issue today so RC2 installs
should only have one logstash-* Index Pattern. RC2 should fix some
other issues as well. Stay tuned!

>> > an email to security-onio...@googlegroups.com.
>> > To post to this group, send email to securit...@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/security-onion.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Doug Burks
>>
>> --
>> Follow Security Onion on Twitter!
>> https://twitter.com/securityonion
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "security-onion" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/security-onion/xxmIitQBcPQ/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> security-onio...@googlegroups.com.
>> To post to this group, send email to securit...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Thank You,
> Joshua Silvestro
>
>

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.

[Josh Silvestro]

Awesome, glad to hear it! 

>> > an email to security-onion+unsubscribe@googlegroups.com.
>> > To post to this group, send email to security-onion@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/security-onion.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Doug Burks
>>
>> --
>> Follow Security Onion on Twitter!
>> https://twitter.com/securityonion
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "security-onion" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/security-onion/xxmIitQBcPQ/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> security-onion+unsubscribe@googlegroups.com.
>> To post to this group, send email to security-onion@googlegroups.com.

>> Visit this group at https://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Thank You,
> Joshua Silvestro
>
>
> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to security-onion+unsubscribe@googlegroups.com.
> To post to this group, send email to security-onion@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/xxmIitQBcPQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Josh Silvestro]

Wasn't sure if I should open a new thread, but I seem to do that often. However, in RC2, I still have multiple *:logstash-* listed under index.

[Wes Lambert]

Hi Josh,

Did you upgrade, or install from the new ISO?

Thanks,

Wes

On Tue, Feb 20, 2018 at 5:36 PM, 'Josh Silvestro' via security-onion <securit...@googlegroups.com> wrote:

Wasn't sure if I should open a new thread, but I seem to do that often. However, in RC2, I still have multiple *:logstash-* listed under index.

[Josh Silvestro]

Clean install. Blew the old RC1 installation away. 

On Tue, Feb 20, 2018 at 5:37 PM, Wes Lambert <wlamb...@gmail.com> wrote:

Hi Josh,

Did you upgrade, or install from the new ISO?

Thanks,

Wes

On Tue, Feb 20, 2018 at 5:36 PM, 'Josh Silvestro' via security-onion <security-onion@googlegroups.com> wrote:

Wasn't sure if I should open a new thread, but I seem to do that often. However, in RC2, I still have multiple *:logstash-* listed under index.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/xxmIitQBcPQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Wes Lambert]

Would you be able to provide the log found in /var/log/nsm/sosetup.log?

Thanks,

Wes

Thank You,

Joshua Silvestro

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.