Docker containers fail to start, iptables forwarding rules failing

[Eric Appelboom]

Hi,

Added the experimental features on Ubuntu 14.04.5 LTS overnight, initially I was able to login to Kibana but ran so-allow an IPtables locked me out.

Since rebooting I can't get ELK docker containers to start. It appears that forwarding rules are added to filter chain but they don't exist.

I have removed the the config and recreated the docker containers with no luck. I am unsure what the correct iptables rules at this point, I assume they are not installed by the container starting but are required when starting?

Eric

Nothign in /var/log/elasticsearch/docker-cluster.log

root@dhost:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
377239b49753 securityonionsolutions/so-logstash "/usr/local/bin/do..." 2 minutes ago Created so-logstash
dcf9e73993a9 securityonionsolutions/so-elasticsearch "/bin/bash bin/es-..." 2 minutes ago Created so-elasticsearch
16497bf8556a securityonionsolutions/so-domainstats "/bin/sh -c '/usr/..." 2 minutes ago Created so-domainstats
37669b890eb6 securityonionsolutions/so-freqserver "/bin/sh -c '/usr/..." 2 minutes ago Created so-freqserver



root@host:~# /usr/sbin/so-elastic-restart

Stopping containers:

so-elastalert
so-curator

Removing containers:

so-elasticsearch
so-logstash
so-kibana
so-elastalert
so-curator
so-freqserver
so-domainstats


Starting containers:

so-freqserver: e8d58a1d2fdf20619aa407605517faa5192dbcccb090b512d6b6527abf4e09d1
docker: Error response from daemon: driver failed programming external connectivity on endpoint so-freqserver (9906c3e868f4785c810d42ee6a0e95797c02f4f17b8549dfbf35be6c1e7843d8): (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 10004 -j ACCEPT: iptables: No chain/target/match by that name.
(exit status 1)).
so-domainstats: c8a54dbff4b32a72599200c0009528e1b1bf8d774cc65775a2c864f81f0fc683
docker: Error response from daemon: driver failed programming external connectivity on endpoint so-domainstats (af83a79e08292dcb6e1ebed03dd781d87f033a70d88f985dc7a764aed8cc9b5f): (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 20000 -j ACCEPT: iptables: No chain/target/match by that name.
(exit status 1)).
so-elasticsearch: 4348dc614ef99b8960ea692d1c474431ae252d64900c404a517c397e25c9d5cb
docker: Error response from daemon: driver failed programming external connectivity on endpoint so-elasticsearch (c0bda1899decd922a067a31216b96b52ba69dfdda263486155e7b52dc17e68cd): (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 9300 -j ACCEPT: iptables: No chain/target/match by that name.
(exit status 1)).
so-logstash: 9ebc1bd3ca6553a4ebf58f01cf3a60c9cb0cbcce0174b47907b1f587ba58fea0
docker: Error response from daemon: driver failed programming external connectivity on endpoint so-logstash (9fa75bb3cd81f4110f6ef08a59d55e22fb38f12378f4c5a61a2b312a01c20cba): (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 6053 -j ACCEPT: iptables: No chain/target/match by that name.
(exit status 1)).
Waiting for ElasticSearch................................................................
Connection attempt timed out. Unable to connect to ElasticSearch.
Please try:
-checking log(s) in /var/log/elasticsearch/
-running 'sudo docker ps'
-running 'sudo so-elastic-restart'

so-kibana: fe102a5c2bdc558f1b164dfd064322b939fc3b639bf312df9db7363b627c5a7b
docker: Error response from daemon: driver failed programming external connectivity on endpoint so-kibana (fc73d3e0bd17d4f8adc56683964ade16f0ca7caf4fdb642a09628d25e5cae888): (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 5601 -j ACCEPT: iptables: No chain/target/match by that name.
(exit status 1)).
so-elastalert: 3679eee99ca9e1bc2b8c9673d6bef6d0d671e0751ddbecbb719917696096d5f8
so-curator: a8dbd4fadac6a30d1564b59419d96b9c13df833b8823af760d2788665f096237


root@dc1monppa01:~# iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
MASQUERADE all -- 172.18.0.0/16 0.0.0.0/0

Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:10004 to:172.17.0.2:10004
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:20000 to:172.17.0.2:20000
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9300 to:172.17.0.2:9300
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:6053 to:172.17.0.2:6053
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:5601 to:172.17.0.2:5601
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:20000

[Doug Burks]

Hi Eric,

When you say that you added the features on Ubuntu 14.04.5 LTS, I'm
assuming you mean that you didn't use our ISO image?

Is there any other software installed that would modify iptables?

Have you tried the following?
sudo iptables -t filter -N DOCKER

Have you tried googling the error message?
https://github.com/moby/moby/issues/16816

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks

[Eric Appelboom]

Hi Doug, Yes this is a older securityonion upgrade, not the ISO.
Thanks, I figured the chain had to be created just after I posted; it may be worth adding a check to install script as it may catch others. I also noted that every time you run tso-elastic-configure it appends the following line, even if it exists in syslog-ng.conf.

destination d_logstash { tcp("127.0.0.1" port(6050) template("$(format-json --scope selected_macros --scope nv_pairs --exclude DATE --key ISODATE)\n")); };

So far working well. I get some periods where Kibana appears to be missing periods of time, but my hits counter is constantly increasing (3.4 million in 5 hours). I have tuned out Bro syslog and will see how it goes.

it would be nice to have a link to /app/kibana#/dashboard/-guid-/Overview or at least default it.

Cheers

[Doug Burks]

Hi Eric,

Replies inline.

On Mon, Oct 9, 2017 at 7:23 AM, Eric Appelboom <eappe...@gmail.com> wrote:
> Hi Doug, Yes this is a older securityonion upgrade, not the ISO.
> Thanks, I figured the chain had to be created just after I posted; it may be worth adding a check to install script as it may catch others.

I think Docker should do this automatically. I don't believe we've
had any other reports of this issue. Do you have any other software
installed that would perhaps modify iptables?

> I also noted that every time you run tso-elastic-configure it appends the following line, even if it exists in syslog-ng.conf.
>
> destination d_logstash { tcp("127.0.0.1" port(6050) template("$(format-json --scope selected_macros --scope nv_pairs --exclude DATE --key ISODATE)\n")); };

Yes, we've already fixed this for the upcoming Beta release:
https://github.com/dougburks/elastic-test/commit/79782cc7fe80d716b2ebc4516672db8b28fe4080

> So far working well. I get some periods where Kibana appears to be missing periods of time, but my hits counter is constantly increasing (3.4 million in 5 hours). I have tuned out Bro syslog and will see how it goes.
>
> it would be nice to have a link to /app/kibana#/dashboard/-guid-/Overview or at least default it.

I'm not sure I understand. When you go to
https://your.security.onion.box/app/kibana, it should default to the
Overview dashboard. If you then navigate to another dashboard, you
should be able to return to the Overview dashboard by clicking the
Home link on the Navigation panel.

[Eric Appelboom]

Thanks Doug, My error, you are correct it defaults to the overview.

I have bro picking up ASA syslogs that have quotes in them, do you manage the logstash.filters.csv ? Can they be escaped?

[2017-10-09T11:48:28,709][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"1507544135.886109\tCVIDYR2lvKdykqvO71\t192.168.138.38\t514\t192.168.54.35\t514\tudp\tLOCAL4\tWARNING\tOct 09 2017 18:15:34: %ASA-4-106023: Deny tcp src MSFC_FW_VLAN:192.168.192.31/59652 dst VLAN:192.168.128.11/2230 by access-group \"MSFC_FW_VLAN_IN\" [0x3977410a, 0x0]\\x0a", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}

[Wes]

Eric,

You may want to see:

https://discuss.elastic.co/t/how-to-escape-the-double-quotes-and-r-of-csv-field-from-longstash/55884/2

Thanks
Wes

[Eric Appelboom]

Thanks Wes, that fixed it. For others the solution is edit /etc/logstash/conf.d/1117_preprocess_bro_syslog.conf and add two lines below after the separator line and do so-elastic-restart

skip_empty_columns => true
quote_char => "$"

Now just to figure out why Kibana is a hour behind in processing, it seems to be rolling forward very slowly, CPU is low (40 cores)
Installed elastichq and connected to docker-cluster, green as well.
logstash-bro is largest at 4GB with 4,875,426 docs. Nothing timed out.

New to ELK so just figuring out.

[Doug Burks]

On Mon, Oct 9, 2017 at 8:55 AM, Eric Appelboom <eappe...@gmail.com> wrote:
> Now just to figure out why Kibana is a hour behind in processing, it seems to be rolling forward very slowly, CPU is low (40 cores)

Hi Eric,

Since we're resolved your original issue of "Docker containers fail to
start, iptables forwarding rules failing", please start a new thread
with appropriate subject for any other questions or problems that you
may have.

Thanks!


--
Doug Burks