Vault and Consul Agent (Client)

walker.h...@gmail.com

unread,

Oct 19, 2017, 7:17:20 PM10/19/17

to Vault

Please pardon me, Im new to this group. Im sharing all my info even if Im uncertain if relevant or not. Ive had about 3 days learning Consul and vault.

Problem: Though I know that Vault is supposed to interact with the Consul Client to ensure connectivity to an active node I seem unable to do so. My NETSTATs show port 8600 only open on the Consul Client agents, 8500 is specific to the Consul Server agent. I can configure Vault to go directly to IP:8500 on the Consul Server agents and vault behaves properly. When I point it at 127.0.0.1:8600 it fails to work, and 8500 is not avail (see netstat). At this time I have only configured vault on 1 of the Consul Agent Servers until I figure out the communication issue I am having with my config and understanding. So What am I missing for vault to use the consul "Client" to communicate to the cluster? (details below)

Image of ny basic understanding of the architecture

Stage one:POC: completed
I setup a POC that was a single server with Vaul and Consul installed together
Setup MySQL dynamic backend
Setup PKI backend
setup aws key backend

setup MSSQL backend
So I moved onto the HA implementation

Stage two:production base setup first datacenter: in progress
I have 3 Servers configured as Consul "server" agents
I have 3 Servers Configured as Consul "Client" agents which also contain my Vault configurations

6 Ec2 images in total

my OSs are the latest Amazon Linux.

Stage Three; Add in TLS and other security refinements
no progress at this time

Stage Four: Roll out to other datacenters regionally
no progress at this time

Consul-Server-Agent Configuration (config in purple)

Consule Members:

Node Address Status Type Build Protocol DC Segment

consul_server1 X.X.49.200:8301 alive server 1.0.0 2 us-east-1 <all>

consul_server2 X.X.51.200:8301 alive server 1.0.0 2 us-east-1 <all>

consul_server3 X.X.52.200:8301 alive server 1.0.0 2 us-east-1 <all>

consul_client-vault1 X.X.48.210:8301 alive client 1.0.0 2 us-east-1 <default>

consul_client-vault2 X.X.49.210:8301 alive client 1.0.0 2 us-east-1 <default>

consul_client-vault3 X.X.51.210:8301 alive client 1.0.0 2 us-east-1 <default>

example of consul "Server" agent (3 Ec2 images, variations are in the join IPs and node)

{

"datacenter": "us-east-1",

"data_dir": "/opt/consul/data/",

"encrypt": "**********************==",

"log_level": "INFO",

"enable_syslog": true,

"node_name": "consul_server01",

"bootstrap": false,

"server": true,

"bootstrap_expect": 3,

"log_level": "INFO",

"enable_syslog": true,

"start_join": ["X.X.51.200", "X.X.52.200"],

"retry_join": ["X.X.51.200:8301", "X.X.52.200:8301"],

"rejoin_after_leave": true,

"retry_interval": "30s"

}

the execution of my init/consul.conf for upstart contains the below for starting the consul "server" agent

exec /usr/local/bin/consul agent \

-config-dir="/opt/consul/server" \

-bind=$BIND \

-client=0.0.0.0 \

${CONSUL_FLAGS} \

>>/var/log/consul.log 2>&1

example of my netstat on my consul server agent server

Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN

tcp 0 0 X.X.49.200:8300 0.0.0.0:* LISTEN

tcp 0 0 X.X.49.200:8301 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:35341 0.0.0.0:* LISTEN

tcp 0 0 X.X.49.200:8302 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN

tcp 0 0 X.X.49.200:8300 X.X.51.200:41777 ESTABLISHED

tcp 0 0 X.X.49.200:8300 X.X.49.200:49263 ESTABLISHED

tcp 0 0 X.X.49.200:48154 X.X.51.210:8301 TIME_WAIT

tcp 0 0 X.X.49.200:8300 X.X.52.200:54321 ESTABLISHED

tcp 0 0 X.X.49.200:49263 X.X.49.200:8300 ESTABLISHED

tcp 0 0 X.X.49.200:46696 X.X.49.210:8301 TIME_WAIT

tcp 0 0 X.X.49.200:8300 X.X.52.200:49831 ESTABLISHED

tcp 0 0 X.X.49.200:49041 X.X.51.200:8300 ESTABLISHED

tcp 0 64 X.X.49.200:22 1.22.101.104:63430 ESTABLISHED

tcp 0 0 X.X.49.200:8300 X.X.52.200:35723 ESTABLISHED

tcp 0 0 X.X.49.200:8301 X.X.49.210:37380 TIME_WAIT

tcp 0 0 X.X.49.200:22 1.22.101.104:63485 ESTABLISHED

tcp 0 0 X.X.49.200:8302 X.X.52.200:47048 TIME_WAIT

tcp 0 0 X.X.49.200:55285 X.X.52.200:8300 ESTABLISHED

tcp 0 0 X.X.49.200:8301 X.X.49.210:37382 TIME_WAIT

tcp 0 0 X.X.49.200:8302 X.X.51.200:52958 TIME_WAIT

tcp 0 0 :::58129 :::* LISTEN

tcp 0 0 :::8500 :::* LISTEN

tcp 0 0 :::22 :::* LISTEN

tcp 0 0 :::8600 :::* LISTEN

tcp 0 0 :::111 :::* LISTEN

udp 0 0 X.X.49.200:46998 X.X.48.10:53 ESTABLISHED

udp 0 0 0.0.0.0:68 0.0.0.0:*

udp 0 0 X.X.49.200:47206 X.X.48.10:53 ESTABLISHED

udp 0 0 X.X.49.200:8301 0.0.0.0:*

udp 0 0 X.X.49.200:8302 0.0.0.0:*

udp 0 0 0.0.0.0:111 0.0.0.0:*

udp 0 0 X.X.49.200:123 0.0.0.0:*

udp 0 0 127.0.0.1:123 0.0.0.0:*

udp 0 0 0.0.0.0:123 0.0.0.0:*

udp 0 0 X.X.49.200:49401 X.X.48.10:53 ESTABLISHED

udp 0 0 0.0.0.0:884 0.0.0.0:*

udp 0 0 0.0.0.0:59259 0.0.0.0:*

udp 0 0 X.X.49.200:49023 X.X.48.10:53 ESTABLISHED

udp 0 0 127.0.0.1:906 0.0.0.0:*

udp 0 0 :::8600 :::*

udp 0 0 fe80::10dc:8bff:fe8c:641:546 :::*

udp 0 0 :::111 :::*

udp 0 0 :::53914 :::*

udp 0 0 :::884 :::*

VAULT and CONSUL-CLIENT-AGENT configuration (config in blue)

Vault is a Cohabitant with Consul "Client" agent. The Vault configuration is below

storage "consul" {

address = "127.0.0.1:8500" (I have swapped out 8600 as well as trying to make a socket connection)

path = "vault-us-east-1/"

redirect_addr = "http://consul_client01.domain.net:8500" (this is the DNS name of the server it is on)

}

ha_storage "consul" {

address = "127.0.0.1:8500" (I have swapped out 8600 as well)

check_timeout = "10s"

path = "ha-vault-us-east-1/"

scheme = "http"

redirect_addr = "http://consul_client01.domain.net:8500" (this is the DNS name of the server it is on)

}

listener "tcp" {

address = "127.0.0.1:8200"

tls_disable = 1

}

cluster_name = "vault_cluster"

disable_mlock = false

my Consul "Client" config on the consul/client and vault server Im using to test

{

"server": false,

"addresses": {

"http": "0.0.0.0"

},

"ports": {

"http": 8080

},

"datacenter": "us-east-1",

"data_dir": "/opt/consul/data/",

"encrypt": "*************************==",

"log_level": "INFO",

"enable_syslog": true,

"ui": true,

"node_name": "consul_client01",

"start_join": ["X.X.49.200", "X.X.51.200", "X.X.52.200"]

}

This is the netstat on my Vault-Consul/Client Server. Note the absence of the 8500 port.

[ec2-user@ip-172-19-48-210 ~]$ sudo vi netstat.log

tcp 0 0 :::22 :::* LISTEN

tcp 0 0 :::58849 :::* LISTEN

tcp 0 0 :::111 :::* LISTEN

tcp 0 0 :::8080 :::* LISTEN

udp 0 0 127.0.0.1:8600 0.0.0.0:*

udp 0 0 0.0.0.0:45025 0.0.0.0:*

udp 0 0 0.0.0.0:68 0.0.0.0:*

udp 0 0 X.X.48.210:8301 0.0.0.0:*

udp 0 0 0.0.0.0:111 0.0.0.0:*

udp 0 0 X.X.48.210:123 0.0.0.0:*

udp 0 0 127.0.0.1:123 0.0.0.0:*

udp 0 0 0.0.0.0:123 0.0.0.0:*

udp 0 0 0.0.0.0:795 0.0.0.0:*

udp 0 0 127.0.0.1:824 0.0.0.0:*

udp 0 0 fe80::99:63ff:fe5c:b9f0:546 :::*

udp 0 0 :::111 :::*

udp 0 0 :::47240 :::*

udp 0 0 :::795 :::*

Thank you in advance for educating me on my mistake.

Cheers,

Walker Hutchinson

Chris Hoffman

unread,

Oct 19, 2017, 7:32:24 PM10/19/17

to vault...@googlegroups.com

Welcome to the group!

From looking at your configuration, it looks like you are pointing your Vault server to the wrong port on the Consul client. Vault interacts with Consul's HTTP api and it appears that the port is overridden to 8080 instead of the default port of 8500. I'm not sure the requirements for your setup but you likely don't need to expose your client's listener at all and have it just listen on localhost:8500 since that will be the location most things look for Consul.

Additionally, since Consul supports HA on Vault, you do not need the "ha_storage" section in the configuration since it is implied by your storage backend. This is only needed when you use a storage backend that does not support HA and want to use a HA backend for leader election.

Hope this helps!

Chris

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/40a11add-37f2-4c10-acf7-c98269aeb4ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

walker.h...@gmail.com

unread,

Oct 20, 2017, 10:48:50 AM10/20/17

to Vault

Thank you Very Much Chris. I thought the 8080 was specifically for the UI, it didnt dawn on me that that could be the API though in retrospect its easy to go "duh"

I have removed the HA _Storage, thanks for the info. I misread that as well evidently.

If i want the UI for consul to be accessible from anything other than 127.0.0.1:8500 do I merely change that to 0.0.0.0:8500 or is there another config for the UI access that i am missing?

I was able to initialize my vault server this morning and Im going to bring up the other two then figure out how sliding an ELB in front of the vault-client servers.

thank you again for your time and knowledge.

cheers,

-Walker

Chris Hoffman

unread,

Oct 20, 2017, 10:59:30 AM10/20/17

to vault...@googlegroups.com

You may want to check in the Consul mailing list since I am not fully versed on typical usage for Consul. In the past, I have put the UI on the server nodes only but I can't really say if that would be a recommended setup.

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/0c94673b-8b31-4f8f-ad21-7ba0cd1c4425%40googlegroups.com.

walker.h...@gmail.com

unread,

Oct 20, 2017, 1:09:23 PM10/20/17

to Vault

if I can trouble you for one more vault question

So its my understanding that 8501 is the "active cluster port" that only the active node has avail and standbys use that port for info.

[ec2-user@ip-X-X-48-210 ~]$ vault status

Sealed: false

Key Shares: 5

Key Threshold: 3

Unseal Progress: 0

Unseal Nonce:

Version: 0.8.3

Cluster Name: vault_cluster

Cluster ID: **********************************

High-Availability Enabled: true

Mode: active

Leader Cluster Address: https://consul_client-vault1-48-210.domain.local:8501

both the standby servers are receiving this error.

grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp X.X.48.210:8501: getsockopt: connection refused"; Reconnecting to {consul_client-vault1-48-210.domain.localt:8501 0

netstat -an | grep 8501 on 48.210 shows no ports

netstat below, I dont see this port being open. thoughts?

Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:8600 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:56573 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:8200 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:8201 0.0.0.0:* LISTEN

tcp 0 0 X.X.48.210:8301 0.0.0.0:* LISTEN

tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN

tcp 0 0 X.X.48.210:38232 X.X.51.200:8301 TIME_WAIT

tcp 0 0 X.X.48.210:22 1.22.101.104:63687 ESTABLISHED

tcp 0 0 127.0.0.1:37844 127.0.0.1:8500 ESTABLISHED

tcp 0 0 X.X.48.210:8301 X.X.49.210:42400 TIME_WAIT

tcp 0 0 127.0.0.1:37846 127.0.0.1:8500 ESTABLISHED

tcp 0 0 X.X.48.210:8301 X.X.49.210:42398 TIME_WAIT

tcp 0 0 X.X.48.210:58253 X.X.52.200:8300 ESTABLISHED

tcp 0 64 X.X.48.210:22 1.22.101.104:63434 ESTABLISHED

tcp 0 0 X.X.48.210:8301 X.X.51.210:49954 TIME_WAIT

tcp 0 0 X.X.48.210:22 1.22.101.104:55910 ESTABLISHED

tcp 0 0 X.X.48.210:51193 X.X.49.200:8300 ESTABLISHED

tcp 0 0 X.X.48.210:8301 X.X.51.200:40040 TIME_WAIT

tcp 0 0 :::8500 :::* LISTEN

tcp 0 0 :::22 :::* LISTEN

tcp 0 0 :::58849 :::* LISTEN

tcp 0 0 :::111 :::* LISTEN

tcp 0 0 ::ffff:127.0.0.1:8500 ::ffff:127.0.0.1:37844 ESTABLISHED

tcp 0 0 ::ffff:127.0.0.1:8500 ::ffff:127.0.0.1:37846 ESTABLISHED

udp 0 0 127.0.0.1:8600 0.0.0.0:*

udp 0 0 0.0.0.0:45025 0.0.0.0:*

udp 0 0 0.0.0.0:68 0.0.0.0:*

udp 0 0 X.X.48.210:8301 0.0.0.0:*

udp 0 0 0.0.0.0:111 0.0.0.0:*

udp 0 0 X.X.48.210:123 0.0.0.0:*

udp 0 0 127.0.0.1:123 0.0.0.0:*

udp 0 0 0.0.0.0:123 0.0.0.0:*

udp 0 0 0.0.0.0:795 0.0.0.0:*

udp 0 0 127.0.0.1:824 0.0.0.0:*

udp 0 0 fe80::99:63ff:fe5c:b9f0:546 :::*

udp 0 0 :::111 :::*

udp 0 0 :::47240 :::*

udp 0 0 :::795 :::*

again, my thanks! I really appreciate this.

cheers,

-Walker

walker.h...@gmail.com

unread,

Oct 20, 2017, 1:11:17 PM10/20/17

to Vault

or is that supposed to be 8201 and Ive screwed up the config somewhere?

Chris

udp 0 0 <a href="http://127.0.0.1:824/" rel="nofollow" target="_blank" onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A824%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCN

Walker Hutchinson

unread,

Oct 20, 2017, 2:16:41 PM10/20/17

to vault...@googlegroups.com

I found the issue. it should be 8201 and I did have 8500 in it =)

thank you again for everything.

cheers,

-Walker

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---

You received this message because you are subscribed to a topic in the Google Groups "Vault" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vault-tool/0ChELRwCNFs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vault-tool+unsubscribe@googlegroups.com.

Chris Hoffman

unread,

Oct 20, 2017, 2:22:40 PM10/20/17

to vault...@googlegroups.com

Cool! I looked at it briefly and knew there was a config issue but didn't get a chance to respond. The default cluster listener port for inter-server communication is vault listener port plus one.

Chris

You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAGSav2g27a-YZPOALyx%2Bx2T%3DU5-6Ho20C6jASFNhv6Ta7%2BPWZw%40mail.gmail.com.

Reply all

Reply to author

Forward