Vault servers behind Haproxy does not show client IP in audit.log

[Robert Gomes]

Our organisation is promoting vault for secret management, our current architecture consist of three vault servers (one active two standby) behind Haproxy. Our Haproxy and relevant vault config is as follows.

frontend vault

    mode tcp

    bind *:8200

    log global

    option tcplog

    use_backend vault

backend vault

    mode        tcp

    balance     roundrobin

    option httpchk HEAD /v1/sys/health

    http-check expect status 200

    server vault1:8200 check check-ssl verify none inter 5000

    server vault2:8200 check check-ssl verify none inter 5000

    server vault3:8200 check check-ssl verify none inter 5000

listener "tcp" {

  address = "0.0.0.0:8200"

  cluster_address = "0.0.0.0:8201"

  tls_cert_file = "/certs/vault.cert"

  tls_key_file = "/certs/vault.key"

  proxy_protocol_behaviour = "use_always"

  proxy_protocol_authorized_addrs = "0.0.0.0"

}

storage "consul" {

  address = "127.0.0.1:8500"

  path = "vault/"

  token = "token"

}

api_addr = "https://vault1:8200"

cluster_addr = "https://vault1:8201"

ui = true

This configuration works fine, but the remote_address is overwritten by the haproxy server IP address in the audit.log. We have tried appending send-proxy to the server segment (server vault1:8200 check check-ssl verify none inter 5000 send-proxy) but in that case I get an error  Server vault/vault1 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure".

Is there a way to pass the client IP to vault backend without terminarting SSL?

[Igor Cicimov]

Remove check-ssl from server line.

[Robert Gomes]

Thanks a lot for your help. I thought this option is required for the healthcheck to work over ssl. If we remove it (check-ssl ) we see a new error.
reason: Layer7 invalid response, info: "<15><03><01>"

On Tuesday, July 23, 2019 at 6:24:52 PM UTC+7, Igor Cicimov wrote:

Remove check-ssl from server line.

[Stephen J. Butler]

Shouldn't proxy_protocol_authorized_addrs be "0.0.0.0/0"? The default with a CIDR is "/32" when the mask bits aren't specified.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/9adf3846-8e80-4c4a-8e64-3d291be44125%40googlegroups.com.

[Igor Cicimov]

You are also missing SSL on the server line.

[Jeff LOMBARDO]

Should you also set X-Forwarded-for? That is the only way to get the origin IP.

Le mar. 23 juill. 2019 18:45, Igor Cicimov <ig...@encompasscorporation.com> a écrit :

You are also missing SSL on the server line.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/213e3a94-a015-40b1-8e03-05710c14128b%40googlegroups.com.

[Igor Cicimov]

You can't do that in tcp mode hence the use of PROXY protocol.

On Wednesday, July 24, 2019 at 9:21:53 AM UTC+10, Jeff LOMBARDO wrote:

Should you also set X-Forwarded-for? That is the only way to get the origin IP.

Le mar. 23 juill. 2019 18:45, Igor Cicimov <ig...@encompasscorporation.com> a écrit :

You are also missing SSL on the server line.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault...@googlegroups.com.

[Robert Gomes]

It is increasingly looking like using send-proxy (and hence haproxy) with our configuration might not be feasible

check ssl verify none works, 

send-proxy check ssl verify none throws reason: Layer6 invalid response, info: "SSL handshake failure"

and

check check-ssl verify none works, 

send-proxy check check-ssl verify none throws reason: Layer6 invalid response, info: "SSL handshake failure"

If we remove all checks ( send-proxy ), haproxy does not throw any error, but attempting to connect to vault throws an SSL error

$ curl -v https://vault:8200/v1/sys/health

*   Trying ip...

* TCP_NODELAY set

* Connected to vault (IP) port 8200 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH

* successfully set certificate verify locations:

*   CAfile: /etc/ssl/cert.pem

  CApath: none

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

* TLSv1.2 (IN), TLS alert, Server hello (2):

* error:140043FE:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert record overflow

* stopped the pause stream!

* Closing connection 0

curl: (35) error:140043FE:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert record overflow

[Jeff Mitchell]

Hi Robert,

Make sure you are using proxy protocol v1. Vault does not currently support v2.

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/a6686842-56e2-4fa7-a2b2-91bfff00155d%40googlegroups.com.

[Igor Cicimov]

This should work:

ssl check check-ssl verify none send-proxy inter 5000

[Robert Gomes]

Unfortunately looks like same issue still
ssl check check-ssl verify none inter 5000 works ssl check check-ssl verify none send-proxy inter 5000 throws Health check for server vault/vault1 failed, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 123ms, status: 0/2 DOWN.

So the issue might be with the healthcheck, I will try to use tcpcheck instead of httpcheck. Any suggestion is really appreciated.

[Igor Cicimov]

Hmmm ok let's start from beginning. You want haproxy to just pass the SSL connections between the client and backend so you use mode tcp. I might have adviced you wrongly regarding the ssl on the server line I have to go back to the docks and check. Any way good idea about stripping down your config. Start like this:

backend vault
mode tcp
balance roundrobin

server vault1:8200 check check-ssl verify none inter 5000
server vault2:8200 check check-ssl verify none inter 5000
server vault3:8200 check send-proxy inter 5000

to confirm if proxy protocol is working at all and let hap do tcp checks only. Once you get it working, i.e. haproxy reports one of the servers as UP, start adding the rest and see where that takes you.

[Igor Cicimov]

Ummm make all 3 server lines look like the vault3 line. Bloody phone typing :-/

[Igor Cicimov]

Also curl supports proxy protocol if you want to test vault from the command line.

https://github.com/curl/curl/commit/6baeb6df35d24740c55239f24b5fc4ce86f375a5

[Robert Gomes]

Thank you so much for this suggestion. Found something very interesting

$ curl -v -XGET -k --haproxy-protocol https://vault1:8200/v1/sys/health

Note: Unnecessary use of -X or --request, GET is already inferred.

*   Trying ip:8200...

* TCP_NODELAY set

* Connected to vault1 (ip) port 8200 (#0)

> PROXY TCP4 client_ip ip 49898 8200

* ALPN, offering http/1.1

* WARNING: disabling hostname validation also disables SNI.

* A record overflow occurred

* Closing connection 0

curl: (35) A record overflow occurred

I don't know much about this error, but it might be something on the vault end (configuration or ssl certificate)

[Robert Gomes]

What's even more interesting, I set up a cault (https://github.com/tolitius/cault) cluster from scratch with only two nodes, and the following vault config.

backend "consul" {

  address = "consul:8500"

  advertise_addr = "http://127.0.0.1:8200"

  path = "vault"

  scheme = "http"

}

listener "tcp" {

  address = "0.0.0.0:8200"

  cluster_address = "0.0.0.0:8201"

  tls_cert_file = "/config/certs/agoda.cer"

  tls_key_file = "/config/certs/agoda.key"

  proxy_protocol_behaviour = "use_always"

  proxy_protocol_authorized_addrs = "0.0.0.0/0"

}

telemetry {

  dogstatsd_addr   = "localhost:8125"

  disable_hostname = true

}

disable_mlock = true

api_addr = "https://vault4:8200"

cluster_addr = "https://vault4:8201"

ui=true

$ curl -v --haproxy-protocol https://vault4:8200/v1/sys/health

Note: Unnecessary use of -X or --request, GET is already inferred.

*   Trying ip:8200...

* TCP_NODELAY set

* Connected to vault4(ip) port 8200 (#0)

> PROXY TCP4 client_ip ip 50196 8200

* ALPN, offering http/1.1

* A record overflow occurred

* Closing connection 0

curl: (35) A record overflow occurred

Vault log 

vault_1   | 2019-07-24T10:44:56.050233749Z 2019-07-24T10:44:56.050Z [INFO]  http: TLS handshake error from client_ip:50214: tls: oversized record received with length 22617

Same error as our main vault cluster, so we might need to look into vault configurations. Thanks a lot to @Igor for pointing at the curl option.

[Igor Cicimov]

The tls error suggests the SSL in vault is broken and the server does not speak SSL atall. Try the calls with http instead https and it should probably work.

[Robert Gomes]

For now I found a less than elegant solution

frontend vault

    mode http

    bind *:8200 ssl crt /certs/vault.pem

    log global

    option httplog

    option forwardfor

    use_backend vault

backend vault

    mode http

    log global

    option httpchk HEAD /v1/sys/health

    http-check expect status 200

    balance roundrobin

    

    server vault1 vault1:8200 ssl check ca-file /certs/ca-and-intermediate.cer inter 5000

    server vault2 vault2:8200  ssl check ca-file /certs/ca-and-intermediate.cer inter 5000

    server vault3 vault3:8200 ssl check ca-file /certs/ca-and-intermediate.cer inter 5000

This works and we can see the client IP in the x-forwarded-for header, I however am still inclined to making the proxy protocol work as that would ensure we are not decrypting packets at the LB. Should I try and create an issue in the vault github repo? this looks like it could be a bug.

And yes http also works fine. Only proxy protocol seems to break stuff.