// , [vault] Certificate Woes: x509 http: TLS handshake error from ... EOF

[Nathan Basanese]

  // , 

I continue to get errors like the following in the servers in my Vault cluster with every external call to its API: 

Aug 31 18:37:38 host1.vaultcluster.mysite.com vault[37672]: 2018-08-31T18:37:38.558-0400 [INFO ] http: TLS handshake error from 172.63.13.37:54436: EOF

And, on clients of Vault, although curl is able to connect without issues, all browsers but FireFox prompt for client TLS authentication. 

I've done some research already, but I think the results I've gotten are not very relevant: 

https://github.com/golang/go/issues/13523

Ideas for what might be the issue

I have several hypotheses. It's either a problem with the certificate itself, a problem with the way that Vault uses the certificate, a problem with GoLang, a combination of those, or something else. 

1. Certificate 

I've tried several different formats for the certificate, everything from .pb7 to .cer and stacked .pem files. 

2. Vault

I haven't run Vault in debug mode yet. But with some formats for the certificate, Vault's logs would say that parts of the file needed to be skipped. With other formats (including the latest, a .cer file), Vault produced no errors. 

3. Golang

I'm assuming https://github.com/golang/go/issues/10685 is irrelevant, since Vault uses a later version of Go. 

Vault Configuration

vault.hcl

# cat /opt/vault/vault.hcl
backend "consul" {

 address = "127.0.0.1:8500"

 path    = "vault/"

}

listener "tcp" {

 address     = "0.0.0.0:8200"

 tls_disable = "false"

 tls_cert_file = "/opt/vault/tls/host1_x509_certificate_chain_stacked.cer"

 tls_key_file = "/opt/vault/tls/host1.key"

}

plugin_directory = "/opt/vault/plugin"

ui = true

Certificate Issuer and Subject Names

# cat host1_x509_certificate_chain_stacked.cer | egrep 'subj|issuer'
subject=/serialNumber=1151337/1.2.2.5.4.1.311.62.3.1.4=US/1.2.2.5.4.1.311.62.3.1.4=Delaware/businessCategory=Private Organization/C=US/postalCode=95050/ST=CA/L=My City/street=My Street/O=My Org, Inc./OU=My Inc./OU=COMODO EV Multi-Domain SSL/CN=host1.vaultcluster.mysite.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Extended Validation Secure Server CA
subject=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Extended Validation Secure Server CA
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
subject=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
issuer=/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
subject=/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
issuer=/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

Log Output

Vault Application Logs via SystemD Service

Aug 31 18:37:08 host1.vaultcluster.mysite.com vault[33627]: 2018-08-31T18:37:08.554-0400 [INFO ] http: TLS handshake error from 172.63.13.37:54051: EOF
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[33627]: ==> Vault shutdown triggered
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[33627]: 2018-08-31T18:37:09.258-0400 [INFO ] storage.consul: shutting down consul backend
Aug 31 18:37:09 host1.vaultcluster.mysite.com systemd[1]: Stopping SystemD Vault Service...
Aug 31 18:37:09 host1.vaultcluster.mysite.com systemd[1]: Starting SystemD Vault Service...
Aug 31 18:37:09 host1.vaultcluster.mysite.com systemd[1]: Started SystemD Vault Service.
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: ==> Vault server configuration:
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: Api Address: https://10.166.13.37:8200
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: Cgo: disabled
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: Cluster Address: https://10.166.13.37:8201
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled"
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: Log Level: info
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: Mlock: supported: true, enabled: true
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: Storage: consul (HA available)
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: Version: Vault v0.11.0
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: Version Sha: 87492f9258e0227f3717e3883c6a8be5716bf564
Aug 31 18:37:09 host1.vaultcluster.mysite.com vault[37672]: ==> Vault server started! Log data will stream in below:
Aug 31 18:37:20 host1.vaultcluster.mysite.com vault[37672]: 2018-08-31T18:37:20.556-0400 [INFO ] http: TLS handshake error from 172.63.13.37:54190: EOF
Aug 31 18:37:26 host1.vaultcluster.mysite.com vault[37672]: 2018-08-31T18:37:26.556-0400 [INFO ] http: TLS handshake error from 172.63.13.37:54295: EOF
Aug 31 18:37:32 host1.vaultcluster.mysite.com vault[37672]: 2018-08-31T18:37:32.557-0400 [INFO ] http: TLS handshake error from 172.63.13.37:54381: EOF
Aug 31 18:37:38 host1.vaultcluster.mysite.com vault[37672]: 2018-08-31T18:37:38.558-0400 [INFO ] http: TLS handshake error from 172.63.13.37:54436: EOF

Previous Research

History

1. I generated a certificate via COMODO, with a widely trusted root issuer AddTrust.

2. I took the binary PKCS#7 .pb7 file produced by that, and successfully converted it to a .cer file in x509 format.

3. I installed that certificate to a location on the Vault server. 

4. I changed the location of the TLS certificate in the Vault configuration file. 

5. I restarted Vault

6. I tested it with Chrome, Firefox, Safari, and Internet Explorer. 

[Jeff Mitchell]

Hi Nathan,

Try setting https://www.vaultproject.io/docs/configuration/listener/tcp.html#tls_disable_client_certs to true.

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/0687c85c-a432-40fa-a9b0-6d352e2ca0ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Theophilus Rakesh]

Try setting VAULT_SKIP_VERIFY = true in your env to skip x.509 verification.

Thanks,

Theo

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAORe8GFaQeh3pLxC0AUbYMtTk0rD-tf04CC6CNi_P3J7Wb3nrQ%40mail.gmail.com.

[Nathan Basanese]

Elvin, that seems more like a work-around than a true solution, but I thank you for offering it. 

Jeff, your solution was just the ticket, and completely resolved the issue. 

I wish I'd searched through the TCP Listener configuration docs before going down the certificate rabbit hole.