consul API via https

[Tim Sales]

I cant seem to get communication to work properly using TLS to the consul API. I have followed the documention to properly enable https in the config as can be seen below. (or at least what my understanding was) So all the datacenter communication is working correctly and there are no TLS errors. However, if I try to talk to the API via https , I get this error below. It does not matter if I explicitly give the CApath/CAfile (or use curl with comparable arguments), the error is the same. Since I have spent some time trying to understand why this is not working, I thought I would trry to post here and see if either 1) I am doing something wrong in the config (or missing something in the config) or 2) I mis-understood the documentation completely with regards to using HTTPS to talk to the API...

Can anyone point me in the right direction here? I have searched everywhere and so far I have not found a single example of anyone using https to make an API call.



thanks

curl: (35) error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate

{
  "verify_server_hostname": false,
  "verify_outgoing": true,
  "verify_incoming": true,
  "ui_dir": "/opt/consul/ui",
  "server": true,
  "recursors": [
    "10.0.0.1",
    "10.0.0.2"
  ],
  "ports": {
    "server": 8305,
    "https": 8900
  },
  "client_addr": "0.0.0.0",
  "cert_file": "/etc/consul/ssl/test.crt",
  "ca_file": "/etc/consul/ssl/ca.crt",
  "bootstrap_expect": 3,
  "addresses": {
    "https": "0.0.0.0"
  },
  "acl_master_token": "master_token",
  "acl_default_policy": "allow",
  "acl_datacenter": "test",
  "data_dir": "/opt/consul",
  "datacenter": "test",
  "dns_config": {
    "allow_stale": true
  },
  "enable_syslog": true,
  "encrypt": "encrypt token",
  "key_file": "/etc/consul/ssl/test.key",
  "log_level": "INFO",
  "node_name": "test"
}

[Brian Lalor]

There’s likely a mismatch between the certificate you’ve configured the agent with and how you’re making the connection with curl.  Introspect the cert you’re using with openssl:

openssl s_client -showcerts -connect $CONSUL_HOST:8900 < /dev/null 2>/dev/null |  openssl x509 -text -noout

Look at the Subject, X509v3 Subject Alternative Name, and one of them should match up with $CONSUL_HOST above.  If it’s a self-signed cert, or not signed by a well-known CA, you will need to make sure that curl and openssl can find the cert chain used to sign the certificate.  You can ignore cert validation problems with curl by using the -k option.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/b08303b2-cd4b-4f22-adaa-79c6012c8342%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

— 

Brian Lalor

bla...@bravo5.org

[Tim Sales]

The subject is a wildcard for the domain I am using so it matches. If there were any problems with these self-signed certificates, the internal consul communication would not be working and none of my clusters would be able to talk to each other. This is not the case so I know there is not an issue with the certificates.

This is the process I followed to create the self signed certs:

https://www.digitalocean.com/community/tutorials/how-to-secure-consul-with-tls-encryption-on-ubuntu-14-04

This is an example of a curl I am doing (but have done the same with openssl client:

curl -X PUT -d @- https://<hostname that matches the wildcard domain in cert subject:8900/v1/kv/example <<< FooValue --capath /etc/consul/ssl  --cacert /etc/consul/ssl/ca.crt

 curl: (35) error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate

[Brian Lalor]

Try adding —tlsv1 or —tlsv1.2 to the curl command.

To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/3a8f031c-7532-462f-bc7b-ab389a6aa360%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

— 

Brian Lalor

bla...@bravo5.org

[Tim Sales]

No change, however an interesting data point that proves its not certificates..  Since I have vault on the same server, I copy the same certificates (servername.crt servername.key) over to /etc/vault/ssl and then restarted the vault service. I used the openssl client and explicitly gave the same CApaht and CAfile arguments that live in consul and are used by consul. It works fine and does NOT return the "bad cert" error. This means there has to be something within the consul config that is wrong or I am missing something because these certs work fine and all the backend consul communication (RPC) is working with the certs...

Very bizzare!!

openssl   s_client    -connect vault-server:8200  -CApath /etc/consul/ssl -CAfile /etc/consul/ssl/ca.crt

[Tim Sales]

For the record, I have literally gone and installed a real certificate that was purchased through commodo that is in use all over production so I know it is valid. I literally still cant get this to work and still have the same errors. I also tried to disable the HTTP port 8500 and only allow https over 8900. Once I did that the conul logs filled with members having a bade certificate:

2016/11/26 00:48:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.122.48.138:8305 10.122.48.138:8305}: remote error: tls: bad certificate

So clearly I have a complete lack of understanding around how consul works with regards to TLS. Looking at the docs and the config I posted above, it is clear to me that all communication requires a valid certificate that has been signed by the cafile in the config. Nothing is working as I would expect related to TLS and how internal communication works along with TLS to the API..

Is there anywhere else to ask question around consul ? I will have to go back and see if there is something I missed in the docs (again)...

[Tim Sales]

I have found an issue on github which confirms this is a known issue:

https://github.com/hashicorp/consul/issues/1986#issuecomment-263038517

The last comment was a couple months back but the original issue was on 0.6.4. I am now on 0.7.1 and still see this problem.

consul v0.7.1
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

So I guess as far as the API/UI goes, I will be putting it behind nginx.