Vault fail connect in status and seal

[mat...@b-sensory.com]

Hello,

I have an inexplicit error with my use of Vault (with Consul and api in Node).

I run in debian vps (ovh) foreach (1 for Node / 1 for Vault / 1 for Consul)

Version of Vault : 0.4.0

Consul is launch as service in his vps.

Here, file config for Vault :

backend “consul” {
  address = “127.0.0.1:8500"
  path = “vault”
}
listener “tcp” {
 address = “127.0.0.1:8200”
 tls_disable = 1
}

When I would start my api, error out is this :

Error: connect ETIMEDOUT xxx.xxx.xxx.xxx:8260

When I would check vault server, error out is this : 

Error checking seal status: Get https://127.0.0.1:8200/v1/sys/seal-status: dial tcp 127.0.0.1:8200: getsockopt: connection refused

When I would start vault server, error out is this : 

Error detecting advertise address: Get http://127.0.0.1:8500/v1/agent/self: dial tcp 127.0.0.1:8500: getsockopt: connection refused

This architecture has already functioning but without reason, since 2 days, Vault fail and fail.

Have you an issue or some idea ?

[Jeff Mitchell]

Hi,

Can you paste the configuration output that Vault prints to stdout
when you start it? Does running a netstat -tln show anything useful?

Also, you really, really, really should upgrade from 0.4 as soon as
possible. Many serious bugs have been fixed, and there have been a
number of fixed security issues as well.

Best,
Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/af4fb601-4171-4ad6-a5eb-191bd045c676%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[mat...@b-sensory.com]

Thx Jeff.

But what is it "the configuration output that Vault prints to stdout" ?

What do you mean ?

Result of netsat in Vault vps (for me all is correct but ...)

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN

tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN

tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN

tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN

tcp        0      0 0.0.0.0:1603            0.0.0.0:*               LISTEN

tcp        0      0 0.0.0.0:8260            0.0.0.0:*               LISTEN

tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN

tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN

tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN

tcp6       0      0 :::80                   :::*                    LISTEN

tcp6       0      0 ::1:25                  :::*                    LISTEN

tcp6       0      0 :::1603                 :::*                    LISTEN

tcp6       0      0 :::5672                 :::*                    LISTEN

Yes an upgrade is needed but not for the moment

[mat...@b-sensory.com]

Ok, I upgraded Vault in 0.8.3 but I have same errors.

Error checking seal status: Get https://127.0.0.1:8200/v1/sys/seal-status: dial tcp 127.0.0.1:8200: getsockopt: connection refused

[Jeff Mitchell]

Hi Mathieu,

When you start Vault it prints a lot of information to stdout. That
information would be useful to see.

In addition it looks like neither Vault nor Consul are running --
ports 8200 and 8500 are both not showing as listening in your netstat.
Do your Vault and Consul logs show anything useful?

Best,
Jeff

> https://groups.google.com/d/msgid/vault-tool/d005378c-54ba-4d4d-8991-5cde61184e94%40googlegroups.com.

[mat...@b-sensory.com]

Ok Jeff

But apparently is not necessary to open 8200 because is only for use it in localhost. And api use port 8260 (https)

On Consul vps, service is ok with this output :

service consul status
● consul.service - LSB: Consul service discovery framework
   Loaded: loaded (/etc/init.d/consul)
   Active: active (running) since Mon 2017-10-09 12:24:26 UTC; 1min 33s ago
  Process: 496 ExecStart=/etc/init.d/consul start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/consul.service
           └─529 /usr/local/bin/consul agent -config-dir /etc/consul

On Vault vps, output is :

vault server -config=/etc/vault/config.json
Error detecting redirect address: Get http://127.0.0.1:8500/v1/agent/self: dial tcp 127.0.0.1:8500: getsockopt: connection refused
Error initializing core: missing redirect address

On Consul vps, netstat is : (8500 is open)

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 10.8.0.1:8300           0.0.0.0:*               LISTEN      529/consul
tcp        0      0 10.8.0.1:8301           0.0.0.0:*               LISTEN      529/consul
tcp        0      0 10.8.0.1:8302           0.0.0.0:*               LISTEN      529/consul
tcp        0      0 127.0.0.1:8400          0.0.0.0:*               LISTEN      529/consul
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      566/nginx -g daemon
tcp        0      0 127.0.0.1:8500          0.0.0.0:*               LISTEN      529/consul
tcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      529/consul
tcp        0      0 0.0.0.0:1603            0.0.0.0:*               LISTEN      683/sshd
tcp        0      0 0.0.0.0:1604            0.0.0.0:*               LISTEN      542/openvpn
tcp6       0      0 :::1603                 :::*                    LISTEN      683/sshd
udp        0      0 127.0.0.1:8600          0.0.0.0:*                           529/consul
udp        0      0 0.0.0.0:68              0.0.0.0:*                           421/dhclient
udp        0      0 10.8.0.1:8301           0.0.0.0:*                           529/consul
udp        0      0 10.8.0.1:8302           0.0.0.0:*                           529/consul
udp        0      0 0.0.0.0:48302           0.0.0.0:*                           421/dhclient
udp6       0      0 :::9817                 :::*                                421/dhclient

[Jeff Mitchell]

Hi Mathieu,

By "Consul VPS" and "Vault VPS" it sounds like you're running them on
different nodes. In that case you'd need to set the Consul address to
be the actual address to your Consul agent.

Best,
Jeff

> https://groups.google.com/d/msgid/vault-tool/8c2c191c-6849-49e4-995a-db5804accfa9%40googlegroups.com.

[Pedro Melo]

Hi,

You mention that you run Vault and Consul on separate servers, but your configuration for vault uses 127.0.0.1:8500 to access consul. This will not work, given that Consul is not running on the same host as Vault.

This causes the error you are seeing: Vault is trying to connect to 127.0.0.1 on the Vault VPS but there is no Consul running there. Consul is on the Consul VPS...

There are two things you can do to solve this:

* run a consul agent (not server) on the Vault VPS;

* update the Consul configuration to have port 8500 listen on all interfaces (currently looking at the netstat you sent, consul port 8500 is only listening at 127.0.0.1 but on the Consul VPS, not the Vault VPS), and then update the Vault server with the IP of the Consul VPS.

Bye,

[mat...@b-sensory.com]

Thx Jeff and Pedro for your guidelines.

Now, my Vault (on vps Vault) config json is : 

backend "consul" {
  redirect_addr = "https://1.vault.xxx.net:8260"
  address = "consul.xxx.net.:8500"

  path = "vault"
}
listener "tcp" {


 address = "1.vault.xxx.net:8260"
 tls_disable = 1
}

When I show listen tcp on ths vps, I am this :

netstat -pltn


Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program

name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      32440/nginx -g daem
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      910/epmd
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      751/postgres
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1611/exim4
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      32440/nginx -g daem
tcp        0      0 0.0.0.0:1603            0.0.0.0:*               LISTEN      1708/sshd
tcp        0      0 0.0.0.0:8260            0.0.0.0:*               LISTEN      32440/nginx -g daem
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      1258/beam
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      548/mongod
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      507/redis-server 12
tcp6       0      0 :::80                   :::*                    LISTEN      32440/nginx -g daem
tcp6       0      0 ::1:25                  :::*                    LISTEN      1611/exim4
tcp6       0      0 :::1603                 :::*                    LISTEN      1708/sshd
tcp6       0      0 :::5672                 :::*                    LISTEN      1258/beam

When I kill this PID and do free this tcp, I have this error :

903460 [WARN ] physical/consul: reconcile unable to talk with Consul backend: error=service registration failed: Puthttp://consul.xxx.net.:8500/v1/agent/service/register: dial tcp 167.114.238.95:8500: getsockopt: connection refused

On vps Consul, I have this about Consul server

ss -antp | grep -i consul
LISTEN     0      128                10.8.0.1:8300                     *:*      users:(("consul",pid=513,fd=3))
LISTEN     0      128                10.8.0.1:8301                     *:*      users:(("consul",pid=513,fd=6))
LISTEN     0      128                10.8.0.1:8302                     *:*      users:(("consul",pid=513,fd=13))
LISTEN     0      128               127.0.0.1:8400                     *:*      users:(("consul",pid=513,fd=15))
LISTEN     0      128               127.0.0.1:8500                     *:*      users:(("consul",pid=513,fd=16))
LISTEN     0      128               127.0.0.1:8600                     *:*      users:(("consul",pid=513,fd=20))

netstat -paunt
Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program

name
tcp        0      0 10.8.0.1:8300           0.0.0.0:*               LISTEN      513/consul
tcp        0      0 10.8.0.1:8301           0.0.0.0:*               LISTEN      513/consul
tcp        0      0 10.8.0.1:8302           0.0.0.0:*               LISTEN      513/consul
tcp        0      0 127.0.0.1:8400          0.0.0.0:*               LISTEN      513/consul
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      564/nginx -g daemon
tcp        0      0 127.0.0.1:8500          0.0.0.0:*               LISTEN      513/consul
tcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      513/consul
tcp        0      0 0.0.0.0:1603            0.0.0.0:*               LISTEN      678/sshd
tcp        0      0 0.0.0.0:1604            0.0.0.0:*               LISTEN      538/openvpn
tcp        0    332 167.114.238.95:1603     90.66.211.60:50434      ESTABLISHED 696/sshd: mathieu [
tcp        0      0 167.114.238.95:1604     167.114.229.240:41954   ESTABLISHED 538/openvpn
tcp        0      0 167.114.238.95:1604     167.114.227.88:34869    ESTABLISHED 538/openvpn
tcp6       0      0 :::1603                 :::*                    LISTEN      678/sshd
udp        0      0 127.0.0.1:8600          0.0.0.0:*                           513/consul
udp        0      0 0.0.0.0:17195           0.0.0.0:*                           408/dhclient
udp        0      0 0.0.0.0:68              0.0.0.0:*                           408/dhclient
udp        0      0 10.8.0.1:8301           0.0.0.0:*                           513/consul
udp        0      0 10.8.0.1:8302           0.0.0.0:*                           513/consul
udp6       0      0 :::65402                :::*                                408/dhclient

Have you a next issue ?

Thx a lot

[Jeff Mitchell]

Hi Mathieu,

It looks like your Vault VPS can't talk to your Consul VPS -- the
connection is being denied.

Best,
Jeff

> https://groups.google.com/d/msgid/vault-tool/933d7d2f-51d8-43eb-a778-830d2132f609%40googlegroups.com.

[lk k]

Hi Team,

Its necessary that for run vault production mode require consul install

[Jeff Mitchell]

Hi,

This is not a correct statement. Dev mode just provides various configuration for you to get you up and running to test things out.

When not running in dev mode you can use any of the storage engines. Consul is not a requirement.

Best,

Jeff