logstash wont initialize

Kris Secinfo

unread,

Mar 29, 2018, 3:06:57 PM3/29/18

to securit...@googlegroups.com

sudo so-status output is below. Note that Logstash is in a WARN state, and will eventually go to a FAIL state, but I dont know why. Also see the output of sudo sostat-redacted (attached)

Status: securityonion

* sguil server [ OK ]

Status: HIDS

* ossec_agent (sguil) [ OK ]

Status: Bro

Name Type Host Status Pid Started

manager manager localhost running 21126 29 Mar 18:52:48

proxy proxy localhost running 21367 29 Mar 18:52:51

seconion-eth0-1 worker localhost running 21455 29 Mar 18:52:53

seconion-eth0-2 worker localhost running 21459 29 Mar 18:52:53

seconion-eth0-3 worker localhost running 21460 29 Mar 18:52:53

Status: seconion-eth0

* netsniff-ng (full packet data) [ OK ]

* pcap_agent (sguil) [ OK ]

* snort_agent (sguil) [ OK ]

* suricata (alert data) [ OK ]

* barnyard2 (spooler, unified2 format) [ OK ]

Status: Elastic stack

* so-elasticsearch [ OK ] * so-logstash -- Logstash has started, but is still initializing... [ WARN ] * so-kibana [ OK ] * so-freqserver [ OK ] * so-domainstats [ OK ] * so-curator [ OK ] * so-elastalert [ OK ]

sostat-redacted.txt

Kris Secinfo

unread,

Mar 29, 2018, 3:40:55 PM3/29/18

to securit...@googlegroups.com

more info:

I ran sudo so-stop, then rebooted the server, then ran sudo so-start. When trying to start things up, I got some interesting output:

so-freqserver: 7a2745d638f50b48732a91fa6d84579ca5478e0b71ad5aa698f2ff8426aabf92

so-domainstats: docker: Error response from daemon: Conflict. The container name "/so-domainstats" is already in use by container "6552c474fd403d813f0ef640b11d356b1cc66d5332eda0848dccfb11be3509b7". You have to remove (or rename) that container to be able to reuse that name.

See 'docker run --help'.

Error response from daemon: No such container: so-domainstats

so-elasticsearch: 1bf28698bf4a69085f120108d170f58b386ebee8cd7a4b3e69e3948d30e3272f

so-logstash: 6c49038b123a984d1a3718f86d6a50bc046e325f0c88c6f4414ec72185041e1a

Error response from daemon: endpoint with name so-logstash already exists in network so-elastic-net

Waiting for ElasticSearch................................connected!

so-kibana: e39306916e15e4ad4a53873597abfd52ee5e798c8002fbbcb3aad606b8febbb4

so-elastalert: docker: Error response from daemon: Conflict. The container name "/so-elastalert" is already in use by container "d0912b5f35e2f394943b50d2bed4aa4beef0d303bf8412aa1fbb6093f356d0a4". You have to remove (or rename) that container to be able to reuse that name.

See 'docker run --help'.

Error response from daemon: endpoint with name so-elastalert already exists in network so-elastic-net

so-curator: 66da7ba9e93b3d79578e5c485a39c3ea1f683164a816f14a7dbc999ef11cc5d8

Error response from daemon: endpoint with name so-curator already exists in network so-elastic-net

Interestingly enough everything shows a status of "OK", again with exception to Logstash:

sudo so-status

Status: securityonion

* sguil server [ OK ]

Status: HIDS

* ossec_agent (sguil) [ OK ]

Status: Bro

Name Type Host Status Pid Started

manager manager localhost running 4652 29 Mar 19:33:58

proxy proxy localhost running 4721 29 Mar 19:34:00

seconion-eth0-1 worker localhost running 4993 29 Mar 19:34:01

seconion-eth0-2 worker localhost running 4992 29 Mar 19:34:01

seconion-eth0-3 worker localhost running 4996 29 Mar 19:34:01

Status: seconion-eth0

* netsniff-ng (full packet data) [ OK ]

* pcap_agent (sguil) [ OK ]

* snort_agent (sguil) [ OK ]

* suricata (alert data) [ OK ]

* barnyard2 (spooler, unified2 format) [ OK ]

Status: Elastic stack

* so-elasticsearch [ OK ] * so-logstash -- Logstash has started, but is still initializing... [ WARN ] * so-kibana [ OK ] * so-freqserver [ OK ] * so-domainstats [ OK ] * so-curator [ OK ] * so-elastalert [ OK ]

Kris Secinfo

unread,

Mar 30, 2018, 8:39:40 AM3/30/18

to securit...@googlegroups.com

Here is some more information below, if anyone has any suggestions, I would really appreciate it!

output from /var/log/logstash/logstash.log:

[2018-03-29T20:09:08,796][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"undefined method `tr' for #<BigDecimal:5c250a3f,'0.3E2',1(4)>", "backtrace"=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:344:in `convert_float'", "org/jruby/RubyMethod.java:115:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:in `block in convert'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:309:in `block in convert'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:299:in `convert'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:252:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `block in multi_filter'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47:in `multi_filter'", "(eval):1575668:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):1575662:in `block in initialize'", "(eval):1575684:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):1575679:in `block in initialize'", "(eval):81009:in `block in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:447:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385:in `block in start_workers'"], :thread=>"#<Thread:0x4e287c07 sleep>"}

[2018-03-29T20:09:08,851][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method `tr' for #<BigDecimal:332c887a,'0.1728E-2',4(8)>>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:344:in `convert_float'", "org/jruby/RubyMethod.java:127:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:313:in `block in convert'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:299:in `convert'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.3.1/lib/logstash/filters/mutate.rb:252:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `block in multi_filter'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47:in `multi_filter'", "(eval):1575647:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):1575643:in `block in initialize'", "(eval):1575682:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):1575679:in `block in initialize'", "(eval):81009:in `block in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:447:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385:in `block in start_workers'"]}

[2018-03-29T20:09:08,993][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `tr' for #<BigDecimal:428f42ad,'0.3E2',1(4)>

On Thu, Mar 29, 2018 at 2:06 PM, Kris Secinfo <kriss...@gmail.com> wrote:

Wes Lambert

unread,

Mar 30, 2018, 9:01:09 AM3/30/18

to securit...@googlegroups.com

Sounds like you may need to follow the steps here:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Elastic-RC4

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

Kris Secinfo

unread,

Mar 30, 2018, 9:59:08 AM3/30/18

to securit...@googlegroups.com

Thanks Wes. I ran through all of those steps, and it is still failing.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Kris Secinfo

unread,

Mar 30, 2018, 10:11:27 AM3/30/18

to securit...@googlegroups.com

[2018-03-30T14:06:57,914][WARN ][logstash.filters.rest ] You are using a deprecated config setting "sprintf" set in rest. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"sprintf", :plugin=><LogStash::Filters::Rest request=>{"url"=>"http://freqserver:10004/measure/%{freq_common_name}"}, sprintf=>true, json=>false, target=>"certificate_common_name_frequency_score", id=>"c92147d95594d43042989b6aa01500370da5051cd39c0f1ba27192610f69fbec", enable_metric=>true, periodic_flush=>false, request_timeout=>60, socket_timeout=>10, connect_timeout=>10, follow_redirects=>true, pool_max=>50, pool_max_per_route=>25, keepalive=>true, automatic_retries=>1, retry_non_idempotent=>false, validate_after_inactivity=>200, keystore_type=>"JKS", truststore_type=>"JKS", cookies=>true, tag_on_rest_failure=>["_restfailure"], tag_on_json_failure=>["_jsonparsefailure"]>}

[2018-03-30T14:06:58,054][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}

[2018-03-30T14:06:58,364][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x60b365fb run>"}

[2018-03-30T14:06:58,511][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}

[2018-03-30T14:06:58,692][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x60b365fb run>"}

To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.

Doug Burks

unread,

Mar 30, 2018, 1:33:55 PM3/30/18

to securit...@googlegroups.com

Please try the following:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Elastic-RC4#logstash-failures

To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Doug Burks

Kris Secinfo

unread,

Mar 30, 2018, 3:49:32 PM3/30/18

to securit...@googlegroups.com

Worked great Doug! thank you!

Doug Burks

amrita...@gmail.com

unread,

Apr 12, 2018, 6:17:20 AM4/12/18

to security-onion

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

>
> To post to this group, send email to securit...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
>
> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to the Google Groups "security-onion" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

>
> To post to this group, send email to securit...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
>
>
>
>
> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to the Google Groups "security-onion" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

>
> To post to this group, send email to securit...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
> --
>

> Doug Burks

Still Warn, i just follow the logstash failures

Wes Lambert

unread,

Apr 12, 2018, 6:55:11 AM4/12/18

to securit...@googlegroups.com

Please start a new thread instead of replying to an old one.

Thanks,

Wes

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

Reply all

Reply to author

Forward