Trouble starting logstash on host-only network.

[infos...@roadflares.org]

I was having an issue with getting logstash started on an SO installation last night -- I tweeted it to SO (https://twitter.com/InfosecGoon/status/1029899144457084935) and I wanted to follow up with more details.

The installation was on a Virtualbox host-only network, IP range 172.16.0.0/16.

The IP scheme:

172.16.0.1 - VB host
172.16.0.5 - Win2008 Domain Controller / DNS
172.16.0.10 - Win 2008
172.16.0.15 - Win 2008
172.16.0.20 - Win 7
172.16.0.25 - SO box

I noticed in Kibana that no logs were being ingested. When I ran so-status, I saw that Logstash was not running. Running so-logstash-restart returns an error "so-logstash: docker: Error: error contacting notary server" with the IP address 172.16.0.5.

Initially, I thought this was a DNS issue because the IP listed was the DNS server for this isolated network. On further reflection, I wonder if this is an IP conflict between the 172.16.0.0/16 space I'm using for the network and the IP addresses used by the docker containers. Is that possible? If so, can the docker IP configuration be changed or do I need to re-address the network?

Thanks for your time,

--Matt

[Wes Lambert]

Have you tried changing the bridge IP/range?

https://github.com/Security-Onion-Solutions/security-onion/wiki/Docker#bridge

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[infos...@roadflares.org]

On Friday, August 17, 2018 at 12:31:53 PM UTC-4, Wes wrote:
> Have you tried changing the bridge IP/range?
>
>
> https://github.com/Security-Onion-Solutions/security-onion/wiki/Docker#bridge
>
>
>
> Thanks,
> Wes

I changed the bridge range to 10.17.0.1/24, logstash still won't start properly.

Attempting to start it manually throws this error:

root@SO:~# so-logstash-start
so-logstash: docker: Error: error contacting notary server: dial tcp: lookup notary.docker.io on 172.16.0.5:53: read udp 172.16.0.25:57019->172.16.0.5:53: i/o timeout.
See 'docker run --help'.
Error response from daemon: No such container: so-logstash

If I remove the DNS server from /etc/network/interfaces, the error changes to this:

root@SO:~# so-logstash-start
so-logstash: docker: Error: error contacting notary server: dial tcp: lookup notary.docker.io on [::1]:53: read udp [::1]:38061->[::1]:53: read: connection refused.
See 'docker run --help'.
Error response from daemon: No such container: so-logstash



Here's the contents of the logstash log file, if it helps shed any light:

[2018-08-18T12:43:17,645][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-08-18T12:43:17,697][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-08-18T12:43:19,290][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-18T12:43:20,972][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-08-18T12:43:22,228][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-18T12:43:34,426][ERROR][logstash.agent ] Internal API server error {:status=>500, :request_method=>"GET", :path_info=>"/_node/stats", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*", :error=>"Unexpected Internal Error", :class=>"LogStash::Instrument::MetricStore::MetricNotFound", :message=>"For path: events. Map keys: [:reloads]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:225:in `block in get_recursively'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:224:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:235:in `block in get_recursively'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:224:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:95:in `block in get'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:94:in `get'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:108:in `get_shallow'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:157:in `block in extract_metrics'", "org/jruby/RubyArray.java:1734:in `each'", "org/jruby/RubyEnumerable.java:936:in `inject'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:133:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/service.rb:30:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/commands/base.rb:22:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/commands/stats.rb:42:in `events'", "/usr/share/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:35:in `events_payload'", "/usr/share/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:21:in `block in GET /?:filter?'", "org/jruby/RubyMethod.java:111:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `block in compile!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block in route!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:994:in `route_eval'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block in route!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1015:in `block in process_route'", "org/jruby/RubyKernel.java:1114:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `process_route'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:973:in `block in route!'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `route!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1085:in `block in dispatch!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'", "org/jruby/RubyKernel.java:1114:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1082:in `dispatch!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `block in call!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'", "org/jruby/RubyKernel.java:1114:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `call!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:895:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/xss_header.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/path_traversal.rb:16:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/json_csrf.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/base.rb:49:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/base.rb:49:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/frame_options.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.6/lib/rack/nulllogger.rb:9:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.6/lib/rack/head.rb:13:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.6/lib/rack/urlmap.rb:66:in `block in call'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.6/lib/rack/urlmap.rb:50:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:57:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.6/lib/rack/builder.rb:153:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:557:in `handle_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:404:in `process_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:270:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:106:in `block in spawn_thread'"]}
[2018-08-18T12:43:35,325][ERROR][logstash.agent ] API HTTP Request {:status=>500, :request_method=>"GET", :path_info=>"/_node/stats", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}

[infos...@roadflares.org]

Update: I reIPed the network from 172.16.0.0/16 to 10.10.10.0/16. Same error. Blew away the VM and rebuilt it with the SO 14.04.5.12 ISO instead. Same error.

I don't know what could be causing this. It's literally a completely vanilla install straight from the ISO. Some kind of weird interaction with Virtualbox?

[infos...@roadflares.org]

On Saturday, August 18, 2018 at 11:45:55 AM UTC-4, infos...@roadflares.org wrote:
> Update: I reIPed the network from 172.16.0.0/16 to 10.10.10.0/16. Same error. Blew away the VM and rebuilt it with the SO 14.04.5.12 ISO instead. Same error.
>
> I don't know what could be causing this. It's literally a completely vanilla install straight from the ISO. Some kind of weird interaction with Virtualbox?

Added another network interface (eth1) bridged to my normal network, relaunched with "so-stop" and "so-start". The script was able to download a bunch of container updates, but it's still throwing the same errors when I try to launch logstash.

[Wes Lambert]

Hi infosecgoon,

Do you get the same result(s) if you use something like VMWare player?

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[infos...@roadflares.org]

On Monday, August 20, 2018 at 2:11:56 PM UTC-4, Wes wrote:
> Hi infosecgoon,
>
>
> Do you get the same result(s) if you use something like VMWare player?
>
>
> Thanks,
> Wes

Thanks for the reply, Wes.

I ended up building a fresh SO VM with two interfaces, one bridged to my network and one on the host-only network. It started working properly after I did the install and ran "soup" a couple of times.

I have no idea what the root cause was, but it's working now. Thanks for the suggestions, I appreciate it.

[Adam Clark]

Does anyone have a solution for this? It seems that the ELK docker images NEED an internet connection in order to restart (once running). Weird that if I reboot the server everything comes up fine. Pretty big problem for deployments on networks not connected to the internet. I feel like I have to be missing something simple. Any way to disable the docker notary check?

- Adam

[Adam Clark]

Gonna answer my own question here... add "disable-trust-content=true" option to the "docker run..." command line in each startup script (i.e. so-logstash-start, so-kibana-start, etc.). Hope this helps someone else.

- Adam