Getting started: Connect - no healthy instances

[Oļegs Vasiļjevs]

Hello!

I have been following "Getting started" guide and completely stuck on "Connect" step (https://learn.hashicorp.com/consul/getting-started/connect). If anybody could give some kind of hint on where to dig, I would be extremely grateful.

Consul version:

root@consul1:~/install# ./consul --version
Consul v1.3.0
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Agent is running with the following command:

./consul agent -dev -config-dir=/etc/consul.d -log-level=trace

Proxy is running with the following command:

./consul connect proxy -service web -upstream socat:9191 -log-level=trace

Socat is being run with the following command:

socat -v tcp-l:8181,fork exec:"/bin/cat"

/etc/consul.d/ contents are:

socat.json:
{
  "service": {
    "name": "socat",
    "port": 8181,
    "connect": { "proxy": {} }
  }
}
web.json:
{"service": {"name": "web", "tags": ["rails"], "port": 80}}

Socat binary is accepting connections, I can connect to it both locally and remotely via 8181 port:

root@consul1:~/install# nc localhost 8181
hello world
hello world

Consul is indeed listening on port 9191:

root@consul1:~/install# lsof -i
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
dhclient  634 root    6u  IPv4  11079      0t0  UDP *:bootpc
sshd      687 root    3u  IPv4  12035      0t0  TCP *:ssh (LISTEN)
sshd      687 root    4u  IPv6  12044      0t0  TCP *:ssh (LISTEN)
sshd     1101 root    3u  IPv4  13193      0t0  TCP 192.168.11.228:ssh->192.168.11.198:50502 (ESTABLISHED)
sshd     1291 root    3u  IPv4  14585      0t0  TCP 192.168.11.228:ssh->192.168.11.198:50653 (ESTABLISHED)
sshd     1763 root    3u  IPv4  15426      0t0  TCP 192.168.11.228:ssh->192.168.11.198:50876 (ESTABLISHED)
sshd     2060 root    3u  IPv4  17010      0t0  TCP 192.168.11.228:ssh->192.168.11.198:51109 (ESTABLISHED)
consul   3231 root    3u  IPv4  20512      0t0  TCP localhost:8300 (LISTEN)
consul   3231 root    5u  IPv4  20513      0t0  TCP localhost:8302 (LISTEN)
consul   3231 root    6u  IPv4  20514      0t0  UDP localhost:8302
consul   3231 root    7u  IPv4  19196      0t0  TCP localhost:8301 (LISTEN)
consul   3231 root    8u  IPv4  19197      0t0  UDP localhost:8301
consul   3231 root    9u  IPv4  20516      0t0  UDP localhost:8600
consul   3231 root   10u  IPv4  19199      0t0  TCP localhost:8600 (LISTEN)
consul   3231 root   11u  IPv4  19201      0t0  TCP localhost:8500 (LISTEN)
consul   3231 root   12u  IPv4  19203      0t0  TCP localhost:8502 (LISTEN)
consul   3231 root   13u  IPv4  20518      0t0  TCP localhost:52567->localhost:8300 (ESTABLISHED)
consul   3231 root   14u  IPv4  20519      0t0  TCP localhost:8300->localhost:52567 (ESTABLISHED)
consul   3231 root   15u  IPv4  20529      0t0  TCP localhost:8500->localhost:41804 (ESTABLISHED)
consul   3231 root   16u  IPv4  20531      0t0  TCP localhost:8500->localhost:41806 (ESTABLISHED)
consul   3249 root    3u  IPv4  19206      0t0  TCP localhost:41804->localhost:8500 (ESTABLISHED)
consul   3249 root    5u  IPv4  19207      0t0  TCP localhost:41806->localhost:8500 (ESTABLISHED)
consul   3249 root    6u  IPv4  19208      0t0  TCP localhost:9191 (LISTEN)
socat    3258 root    5u  IPv4  20542      0t0  TCP *:8181 (LISTEN)

When trying to connect to 9191 port, nothing happens. Meaning, connection to socat is not opened.
Agent log:

2018/11/07 14:12:55 [WARN] agent: Check "service:socat-proxy" socket connection failed: dial tcp 127.0.0.1:20241: connect: connection refused

Proxy log:

2018/11/07 14:12:18 [ERR] failed to dial: no healthy instances found

There is no firewall installed:

root@consul1:~/install# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

This is completely fresh Debian 9 box with the latest updates:

root@consul1:~/install# uname -a
Linux consul1 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x86_64 GNU/Linux

Consul binary was donwloaded from: https://releases.hashicorp.com/consul/1.3.0/consul_1.3.0_linux_amd64.zip

[Paul Banks]

Hi Olegs

Sorry you ran into an issue with this, I think there are a couple of things going on here...

Firstly, I noticed that the guide we recently just launched on learn.hc.com is actually the older one from before the 1.3.0 release which uses "managed proxies". Now these were deprecated but not removed in 1.3.0 so it _should_ still work but I just wanted to clarify that. I've notified our team who published that and hopefully will get it updated soon.

I know exactly why this isn't working for you - since the Consul agent is running as root, we disable starting managed proxies by default since it is a security risk that any service can register a process that could run as root. There is an option to enable it: https://www.consul.io/docs/agent/options.html#connect_proxy_allow_managed_root or you can run the agent as non-root.

As I mentioned though we've deprecated the idea of the agent actually running the proxy for you so this is not a problem with the newer guide, that just unfortunately got temporaily lost in the transition between consul.io and learn.hashicorp.com.

You can follow it along in the source if you want to try the newer guide before the fix is up but I hope that will be soon anyway: https://github.com/hashicorp/consul/blob/master/website/source/intro/getting-started/connect.html.md

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/22138e34-b0de-499d-a773-ede58f7d4e0e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oļegs Vasiļjevs]

Hello Paul,

Thank you very much for a such an open and fast reply!

I will indeed try out the new instructions available at Github, thank you for pointing at the right direction.