Client certificate not sent to server

7,560 views
Skip to first unread message

Dots Connected

unread,
Mar 17, 2015, 1:19:01 PM3/17/15
to golan...@googlegroups.com
Hello!

I am trying to access an API served by nginx (with a self-signed SSL cert) that requires client certificate authentication, and while I can successfully auth with curl, I unable to accomplish the same with golang.

Example using curl:

# Will not work

# Works
curl -k -E example_key+cert.pem https://127.0.0.1/api


I found an example from this list that I've tried to adapt:
And another from github:

Despite this, golang gets the same result as the first curl with no client cert.
Am I missing something, or is this a regression in go?

I'm on go version go1.4.2 linux/amd64. My code (or via http://pastebin.com/LGiXZpgx):

package main

import (
"crypto/tls"
"io/ioutil"
"log"
"net/http"
)

func main() {

certFile := "example_cert.pem"
keyFile := "example_key.pem"

// Load client cert
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
log.Fatal(err)
}

// Setup HTTPS client
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{cert},
ClientAuth: tls.RequireAndVerifyClientCert,
InsecureSkipVerify: true,
}
tlsConfig.BuildNameToCertificate()

transport := &http.Transport{
TLSClientConfig: tlsConfig,
}
client := &http.Client{
Transport: transport,
}

// Do GET something
resp, err := client.Get("https://localhost/api/")
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()

// Dump response
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
log.Println(string(data))
}

Nick Craig-Wood

unread,
Mar 18, 2015, 10:27:27 AM3/18/15
to Dots Connected, golan...@googlegroups.com
Here is a working example of a client and a server with client side
certificates.

Hope that helps!

https://gist.github.com/ncw/9253562
> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com
> <mailto:golang-nuts...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.


--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick

wat bat

unread,
Mar 20, 2015, 4:17:54 PM3/20/15
to Nick Craig-Wood, golan...@googlegroups.com
Nick,

It looks like your client example code does not differ from my example at all, other than using the raw TCP dial, rather than the http package.

When trying your example, I can indeed connect your client to its server counterpart, but nginx will reject your client code with the same error I mentioned above. This should be reproducible in any nginx install with the setting "ssl_verify_client" set to "on".

Would a test server to hit help? It's pretty clear that curl is doing the right thing and that the golang client is behaving incorrectly, or at least differently, in a way that nginx does not accept.

James Bardin

unread,
Mar 20, 2015, 4:53:15 PM3/20/15
to golan...@googlegroups.com, ni...@craig-wood.com


On Friday, March 20, 2015 at 4:17:54 PM UTC-4, Dots Connected wrote:

When trying your example, I can indeed connect your client to its server counterpart, but nginx will reject your client code with the same error I mentioned above. 


Maybe I missed it, but what is the actual error? What does nginx say is wrong with the request?

 

Dots Connected

unread,
Mar 20, 2015, 5:01:07 PM3/20/15
to golan...@googlegroups.com, ni...@craig-wood.com
Apologies, the raw error nginx responds with is:

<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
<hr><center>nginx/1.7.10</center>
</body>
</html>

Frustratingly, a simple 400 is logged to nginx's access log, and no further details to the error log.
I'll check to see if I can increase error verbosity.

This response text is identical to the "will not work" curl example I wrote in my first email.
Which leads me to believe that my golang code is not behaving as I would expect relative to curl's -E option.

Piers

unread,
Mar 20, 2015, 5:09:55 PM3/20/15
to golan...@googlegroups.com
>It looks like your client example code does not differ from my example at all, other than using the raw TCP dial, rather than the http package.

Actually there are a couple of differences I spotted comparing what Nick posted with what you linked to in pastebin.

This stands out (the RequireAndVerifyClientCert if highlighting doesn't show up): 

        // Setup HTTPS client
        tlsConfig := &tls.Config{
                Certificates: []tls.Certificate{cert},
                ClientAuth: tls.RequireAndVerifyClientCert,
                InsecureSkipVerify: true,
        }

This is a server-side option saying, the connected Peer must send a client cert. Which doesn't really make sense as an option for a client.

And I don't know what 

    tlsConfig.BuildNameToCertificate()

is for, so can't comment on that. But it's not in Nick's example. No idea if these will help but my "fresh pair of eyes" has spotted them as different.

 

Dots Connected

unread,
Mar 20, 2015, 5:11:24 PM3/20/15
to golan...@googlegroups.com, ni...@craig-wood.com
A brief update: nginx with debug logging will respond to my code with "client sent no required SSL certificate while reading client request headers", which is about what I expected.

Nick's client code (modifying only to change the URL and certificate file location) will instead result in a log entry of "client sent invalid method while reading client request line", but that's not too surprising as Nick's code writes the string "Hello" rather than a valid HTTP request.

Both get the same HTML response body; only the debug log entries differ.

Dots Connected

unread,
Mar 20, 2015, 5:13:29 PM3/20/15
to golan...@googlegroups.com

Actually there are a couple of differences I spotted comparing what Nick posted with what you linked to in pastebin.

Thanks  Piers, you're right. 

Both of those are differences I added attempting to aggregate multiple examples together (largely out of frustration).
My results are the same when commenting those lines out.

Piers

unread,
Mar 20, 2015, 5:18:59 PM3/20/15
to golan...@googlegroups.com
Ahh cool, it's good to rule it out. I just checked my own ssl with client-cert code and it's the same apart from those. 

James Bardin

unread,
Mar 20, 2015, 6:25:35 PM3/20/15
to golan...@googlegroups.com, ni...@craig-wood.com


On Friday, March 20, 2015 at 5:11:24 PM UTC-4, Dots Connected wrote:
A brief update: nginx with debug logging will respond to my code with "client sent no required SSL certificate while reading client request headers", which is about what I expected.


I tried this against nginx, and the go client is definitely sending a cert. Nginx tells me "unable to verify the first certificate", which is the same thing I get with curl.

I messed up the cert/ca creation (hence "unable to verify"), and don't have time at the moment to sort it out; but I would double check for incorrect paths, binaries and typos carefully first. I only get the error you describe by not including the cert at all. 

Another long shot, I notice you have 127.0.0.1 in your curl example, and localhost in the go example. Those could be routing to different paths in nginx via sni, or even connecting to different listeners depending on how localhost resolves.

Dots Connected

unread,
Mar 26, 2015, 2:31:22 PM3/26/15
to golan...@googlegroups.com, ni...@craig-wood.com
I wanted to make sure the issue was reproducible, so I made a gist:

If you already have docker installed, it should be pretty easy to use:

git clone https://gist.github.com/28f59611a744abae94c5.git example && cd example
docker run -it -v $PWD:/example -p 443:443 ubuntu:14.04 /example/run.sh


This will launch a copy of nginx with client ssl cert verification.
After a quick setup, you should see both curl and golang successfully hitting the SSL API.


Example with curl:
<head><title>400 No required SSL certificate was sent</title></head>

Example with curl and client cert:
<title>Welcome to nginx!</title>

Example with golang and client cert:
<title>Welcome to nginx!</title>


That means there must be something in my specific environment that's causing the issue. I'll update if I figure out what that is.
Hopefully this will help anyone else who was confused.

Dots Connected

unread,
Mar 30, 2015, 11:35:29 AM3/30/15
to golan...@googlegroups.com
An additional development: I've discovered the root cause.

It would appear that the golang http stack isn't compatible with nginx's ssl_trusted_certificate directive.
When used to indicate which client certificate to trust, curl will succeed and the go example will fail.
Contrast with the gist I linked above, which uses: ssl_client_certificate /example/client.crt

These two directives vary in subtle ways:

My suspicion is that because trusted_certificate does not send the CA cert to the client, golang does not respond with any client certs and nginx fails the request.
This aligns with the <center>No required SSL certificate was sent</center> message I was receiving, and the debug log: client sent no required SSL certificate while reading client request headers.
Contrast with curl, which will send a client cert unconditionally when the -E flag is used.

A workaround to accept ordinary "real" CA client certs and a whitelist of your own "fake" CAs is to simply combine all CA files.
Ubuntu systems should have a /etc/ssl/certs/ca-certificates.crt file that you can combine for use with ssl_client_certificate.

This command and nginx directive demonstrate what I mean:

cat /etc/ssl/certs/ca-certificates.crt /example/client.crt > /tmp/ca-combined.crt
ssl_client_certificate /tmp/ca-combined.crt;

You would obviously want to put your destination file somewhere more reasonable with correct POSIX controls.

Thanks to everyone who offered their time helping with this issue!

James Bardin

unread,
Mar 30, 2015, 6:07:35 PM3/30/15
to Dots Connected, golan...@googlegroups.com
On Mon, Mar 30, 2015 at 11:35 AM, Dots Connected <5dotsco...@gmail.com> wrote:
An additional development: I've discovered the root cause.

It would appear that the golang http stack isn't compatible with nginx's ssl_trusted_certificate directive.
When used to indicate which client certificate to trust, curl will succeed and the go example will fail.
Contrast with the gist I linked above, which uses: ssl_client_certificate /example/client.crt


As far as I know, ssl_trusted_certificate shouldn't affect it at all, since that is for OCSP Stapling, but you still need to provide an ssl_client_certificate. Maybe you're encountering a bug in a specific version of nginx? What does your nginx config look like for this example?

Interestingly, on the mac I just ran your test on, curl fails to send a certificate, but the Go client is still OK, and the openssl s_client connects fine too. 

Piers

unread,
Apr 1, 2015, 6:18:00 PM4/1/15
to golan...@googlegroups.com
>Contrast with curl, which will send a client cert unconditionally when the -E flag is used.

Oooh.. What version of curl was it? I ruled this whole thing out because in my quick test curl didn't send the client cert I specified with -E if the server didn't mention a CA it was issued by (or indirectly). Glad you worked out what was happening!

Mark Petrovic

unread,
May 19, 2020, 7:27:58 PM5/19/20
to golang-nuts
I know this is an old thread, but I found it, so others might, too.  

I struggled with a similar problem of the Go HTTP client not sending client certs when challenged by the server.  Yet, curl worked fine when given the same client certs, calling the same server endpoint.  The cert chains I was dealing with look like

leaf + intermediate + root


The solution for me was to provide the server (Envoy proxy) with a the root cert AND intermediate cert, the latter of which signed my Go client cert.  When the server challenges the client for its cert, the server will send a TLS Certificate Request with, among other things, the certificate authority that must have signed the client cert.  Without including the intermediate cert with the root cert on the server side, the client cannot not build a complete path from its client cert to the root and therefore cannot determine which cert to send in response.  The client will not use tls.Config.RootCAs in this analysis (RootCAs is for validating the server cert chain).  Moreover, when the client parses its cert with 
cert, err := tls.LoadX509KeyPair(*certFile, *keyFile)

only the leaf is returned even if the certFile contains the concatenated intermediate.

Including the full cert bundle on the server side solves this for me.

Hope that helps.

D.

unread,
Mar 24, 2021, 11:37:13 AM3/24/21
to golang-nuts
Elaborating on the previous answer, which led me to the source of the issue I'm facing but did not quite solve the problem.

In case you have no control over the server, you are using private CA, and the server does not include Certificate Authorities extension in TLS Handshake Certificate Request, in this case golang will not include certificates from private CAs even if they are included in the tls configuration. This can be solved by implementing tls.Config.GetClientCertificate and doing the cert, err := tls.LoadX509KeyPair(certFile, keyFile) part there.

Hope that helps.

Reply all
Reply to author
Forward
0 new messages