Vault not connecting to Consul back end when run as root

1,797 views
Skip to first unread message

David Knudsen

unread,
Jun 12, 2019, 3:53:05 PM6/12/19
to Vault
Good afternoon,

I am having issues setting up a vault server with a consul back end, if I set the back end as "inmem" it works fine as root.

Running as me, yields the following results:

dknudsen@gms-st-npd-consul-c-475v:~$ /usr/local/bin/vault server -config=/etc/vault.d/vault.hcl -log-level=debug
==> Vault server configuration:

             Api Address: http://172.16.1.243:8200
                     Cgo: disabled
         Cluster Address: https://172.16.1.243:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "172.16.1.243:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: debug
                   Mlock: supported: true, enabled: true
                 Storage: consul (HA available)
                 Version: Vault v1.1.2
             Version Sha: 0082501623c0b704b87b1fbc84c2d725994bac54

==> Vault server started! Log data will stream in below:

2019-06-12T19:48:43.737Z [DEBUG] storage.consul: config path set: path=vault/
2019-06-12T19:48:43.737Z [DEBUG] storage.consul: config disable_registration set: disable_registration=false
2019-06-12T19:48:43.737Z [DEBUG] storage.consul: config service set: service=vault
2019-06-12T19:48:43.737Z [DEBUG] storage.consul: config service_tags set: service_tags=
2019-06-12T19:48:43.737Z [DEBUG] storage.consul: config service_address set: service_address=<nil>
2019-06-12T19:48:43.737Z [DEBUG] storage.consul: config address set: address=127.0.0.1:8500
2019-06-12T19:48:43.737Z [DEBUG] storage.consul: config scheme set: scheme=http
2019-06-12T19:48:43.741Z [DEBUG] storage.cache: creating LRU cache: size=0
2019-06-12T19:48:43.792Z [DEBUG] cluster listener addresses synthesized: cluster_addresses=[172.16.1.243:8201]
^C==> Vault shutdown triggered
2019-06-12T19:48:46.289Z [DEBUG] core: shutdown called
2019-06-12T19:48:46.289Z [INFO]  storage.consul: shutting down consul backend

Running as root, yields me this:

dknudsen@gms-st-npd-consul-c-475v:~$ sudo /usr/local/bin/vault server -config=/etc/vault.d/vault.hcl -log-level=debug

WARNING! Unable to read storage migration status.
2019-06-12T19:49:38.634Z [DEBUG] storage.consul: config path set: path=vault/
2019-06-12T19:49:38.634Z [DEBUG] storage.consul: config disable_registration set: disable_registration=false
2019-06-12T19:49:38.634Z [DEBUG] storage.consul: config service set: service=vault
2019-06-12T19:49:38.634Z [DEBUG] storage.consul: config service_tags set: service_tags=
2019-06-12T19:49:38.634Z [DEBUG] storage.consul: config service_address set: service_address=<nil>
2019-06-12T19:49:38.634Z [DEBUG] storage.consul: config address set: address=127.0.0.1:8500
2019-06-12T19:49:38.634Z [DEBUG] storage.consul: config scheme set: scheme=http
2019-06-12T19:49:38.641Z [WARN]  storage migration check error: error="Unexpected response code: 403"
2019-06-12T19:49:40.646Z [WARN]  storage migration check error: error="Unexpected response code: 403"
2019-06-12T19:49:42.651Z [WARN]  storage migration check error: error="Unexpected response code: 403"

dknudsen@gms-st-npd-consul-c-475v:~$ whoami
dknudsen
dknudsen@gms-st-npd-consul-c-475v:~$ sudo whoami
root

I am happy to provide any further information you need, but this has me stumped.

TIA,

David

Michel Vocks

unread,
Jun 13, 2019, 4:56:33 AM6/13/19
to Vault
Hi David,

it seems like Vault is not able to query Consul when it is started as root user.
Could you try to send a manual request (as root user) to Consul to see if that works?

$ sudo curl http://127.0.0.1:8500/v1/catalog/datacenters
[
    "dc1"
]

It would also help to see your used Vault configuration file.

Cheers,
Michel

David Knudsen

unread,
Jun 13, 2019, 7:16:04 AM6/13/19
to Vault
Michael,

Thanks for the response, here is the results from the query:

dknudsen@gms-st-npd-consul-c-475v:~$ sudo curl http://127.0.0.1:8500/v1/catalog/datacenters
["npd-us-central"]

And here is the contents of my vault.hcl:

listener "tcp" {
  address          = "0.0.0.0:8200"
  cluster_address  = "172.16.1.243:8201"
  tls_disable      = "true"
}

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
  scheme = "http"
}

cluster_addr = "https://172.16.1.243:8201"

Additionally, the contents of vault.service:

[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Requires=network-online.target
After=network-online.target

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl -log-level=debug
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Thanks again!

David
Reply all
Reply to author
Forward
0 new messages