TLS handshake error --- tls: first record does not look like a TLS handshake

[Michael S]

I am having trouble configuring Vault to use HTTPS.  During development, I had been using strictly HTTP, but am now trying to enable HTTPS for testing before going to production.  I want to have two listeners configured:  One using localhost over HTTP, and one for the server address, using HTTPS.

 

I am experiencing errors when I try to start the vault:

TLS handshake error from <ip address>:53568: tls: first record does not look like a TLS handshake

 

Can anyone see what I’m doing wrong?  Is this a vault configuration issue, or maybe a certificate issue?

 

Here is my configuration information:

 

Environment variable defined :

export VAULT_ADDR="http://127.0.0.1:8200"

 

Real (not self-signed) Certificates are installed as:

  /etc/ssl/certs/vault-itg.zz8.us.crt

  /etc/ssl/private/vault-itg.zz8.us.key

 

I start the vault from a shell script, using the following command line:

nohup vault server -config=config_s3_storage.hcl > $VAULTSTARTLOGFILE 2>&1 &

 

The contents of my config file (config_s3_storage.hcl) is as follows:

 

disable_mlock = true

 

storage "s3" {

  bucket = "storage-vault-itg"

  region = "us-west-2"

}

 

listener "tcp" {

  address     = "127.0.0.1:8200"

  tls_disable = 1

}

 

listener "tcp" {

  address = "172.20.209.92:8200"

  tls_disable = 0

  tls_cert_file = "/etc/ssl/certs/vault-itg.zz8.us.crt"

  tls_key_file = "/etc/ssl/private/vault-itg.zz8.us.key"

}

 

Output received from vault: 

 

==> Vault server configuration:

 

                     Cgo: disabled

              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", tls: "disabled")

              Listener 2: tcp (addr: "172.20.209.92:8200", cluster address: "172.20.209.92:8201", tls: "enabled")

               Log Level: info

                   Mlock: supported: true, enabled: false

                 Storage: s3

                 Version: Vault v0.7.0

             Version Sha: 614deacfca3f3b7162bbf30a36d6fc7362cd47f0

 

==> Vault server started! Log data will stream in below:

 

2017/05/18 18:27:41 http: TLS handshake error from 172.20.209.92:50698: tls: first record does not look like a TLS handshake

2017/05/18 18:27:41 http: TLS handshake error from 172.20.209.92:50700: tls: first record does not look like a TLS handshake

2017/05/18 18:27:41 http: TLS handshake error from 172.20.209.92:50702: tls: first record does not look like a TLS handshake

2017/05/18 18:27:41 http: TLS handshake error from 172.20.209.92:50708: tls: first record does not look like a TLS handshake

2017/05/18 18:27:41 http: TLS handshake error from 172.20.209.92:50710: tls: first record does not look like a TLS handshake

 

 

Note:  I still get the same error if I remove the line from the config file:  "tls_disable = 0" (from the second tcp listener.)

For reference, here is the contents of the configuration file that I used when I was configuring for HTTPS only:

 

disable_mlock = true

 

storage "s3" {

  bucket = "storage-vault-itg"

  region = "us-west-2"

}

 

listener "tcp" {

  address     = "127.0.0.1:8200"

  tls_disable = 1

}

 

listener "tcp" {

  address = "172.20.209.92:8200"

  tls_disable = 1

}

 

 

[Brian Kassouf]

Hi Michael,

It might be because your client is not accessing vault with https.

Try setting your environment variable to:
export VAULT_ADDR="https://127.0.0.1:8200"

Hope that helps,
Brian

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/96ce8d9e-a136-48ad-8b88-8f18b66f6fca%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Michael S]

Hi Brian,

Thanks for the reply.  

Is setting the VAULT_ADDR to https appropriate, since I have "tls_disable = 1" in the listener for 127.0.0.1?  

My thought was to keep the listener on localhost (127.0.0.1) configured for http, and have the server address configured for https.  

Thanks

---Mike

[Brian Kassouf]

Hey Mike,

My mistake, I meant to reference the other IP address. How are you
accessing vault when that error is produced? Is it with an https
address (e.g. https://172.20.209.92:8200)?

Best,
Brian

> https://groups.google.com/d/msgid/vault-tool/385fc7f4-9a33-4b3d-984b-b30293a9fac9%40googlegroups.com.

[Michael S]

Brian,

I am accessing the vault through the following command line:

vault server -config=config_s3_storage.hcl

It is my understanding that the vault command line will use the address defined in VAULT_ADDR, correct?

If that is true, that means the error occurs when the client is attempting to use "http://127.0.0.1:8200"

Since the listener for 127.0.0.1 is configured to have TLS disabled ("tls_disable = 1"), I would expect that the commands would go to the server via HTTP.  However, the error indicates that something involving TLS is happening.

I will try changing VAULT_ADDR to "https://172.20.209.92" and see what it does...

[Michael S]

I set VAULT_ADDR = https://172.20.209.92:8200

I used the same config file shown in my original post, which has TLS enabled, and includes the paths to the certificate & key.

I run the command:

vault server -config=config_s3_storage.hcl

I still get the error:

http: TLS handshake error from 172.20.209.92:36034: tls: first record does not look like a TLS handshake

So I am still stuck, and looking for a solution.

[Michael S]

Here is my latest unsuccessful test:

I am trying to configure both localhost and the server ip address to use HTTPS.

I set VAULT_ADDR = https://127.0.0.1:8200

I used the following config, which has TLS enabled for both listeners, and includes the paths to the certificate & key.

disable_mlock = true

 

storage "s3" {

  bucket = "storage-vault-itg"

  region = "us-west-2"

}

 

listener "tcp" {

  address     = "127.0.0.1:8200"

  tls_disable = 0

  tls_cert_file = "/etc/ssl/certs/vault-itg.zz8.us.crt"

  tls_key_file = "/etc/ssl/private/vault-itg.zz8.us.key"

}

 

listener "tcp" {

  address = "172.20.209.92:8200"

  tls_disable = 0

  tls_cert_file = "/etc/ssl/certs/vault-itg.zz8.us.crt"

  tls_key_file = "/etc/ssl/private/vault-itg.zz8.us.key"

}

I run the command:

vault server -config=config_s3_storage.hcl

Now I get the following error:

TLS handshake error from 127.0.0.1:42410: remote error: tls: bad certificate

The cert CA is Verisign.  I checked the certificate by running "openssl x509 -text -in <certname>" and everything looks OK in that output.

I'm at a loss why I can't get TLS to work.

[Randy Fay]

I don't believe that anybody issues certs for IP addresses, and am sure that nobody issues certs for 127.0.0.1. You'll need to access the endpoint by its FQDN (the same FQDN/hostname as the cert carries). You can test that by putting the hostname temporarily into /etc/hosts.

An easy way to experiment is to hit your endpoint with Chrome and look at the cert results. Hit https://127.0.0.1:8200 with chrome and you'll see it's insecure, but better than that, you can "Inspect", and then on the "Security" tab, you can see everything about the cert. But before you do that, get the correct hostname on your URL (with a temporary /etc/hosts hack).

-Randy

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/5490a6ce-5026-4d76-9193-e853a59cad6b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Randy Fay

ra...@randyfay.com

+1  970.462.7450

[Vishal Nayak]

Hi Michael,

If you are noticing the errors after starting Vault server and before
initializing it, then it is unlikely that this is related to
VAULT_ADDR. Please confirm if you are seeing server error messages
before initializing Vault. VAULT_ADDR is used to access the API for an
already running Vault and not to start one.

I tested the following configuration on latest Vault and it is working fine.

```
backend "file" {
path = "/Users/vishal/go/src/github.com/hashicorp/vault/filedata/"
}

disable_mlock = true

listener "tcp" {
address = "0.0.0.0:9200"

tls_disable = 0
tls_cert_file =

"/Users/vishal/go/src/github.com/hashicorp/vault/vault.crt"
tls_key_file =
"/Users/vishal/go/src/github.com/hashicorp/vault/vault.key"
}

listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = "1"
}
```

I don't yet have any clue about the errors you are seeing. Is it by
any chance related to the file permissions of the cert and key?

Regards,
Vishal

>> https://groups.google.com/d/msgid/vault-tool/5490a6ce-5026-4d76-9193-e853a59cad6b%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Randy Fay
> ra...@randyfay.com
> +1 970.462.7450
>

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vault-tool/CAGxQj1Fmc5%3DAQKgdr3%3DB%2BppoOrC3r6U2j8Dd-yMG%3DwCBcsNg1A%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.



--

vn

[Michael S]

Hi Vishal,

We were able to get everything to work, after understanding the permission issues.  

You asked:  "Please confirm if you are seeing server error messages before initializing Vault"

Upon further examination, I did determine that the errors were not happening before initializing the Vault.

The problem ended up not being related to VAULT_ADDR, and we used the value: "http://127.0.0.1:8200"

I have the setup operation scripted, and it appears that not everything was being run at the proper permissions.  At first I was running the scripts using the "sudo" command, which resulted in the failures.  I discovered that the permissions for the certificate key were restricted and the file could not be accessed by my user.  There may have been other permission issues as well.  But once I switched user to root, and ran the script, everything behaved correctly.

Thanks

---Mike