SSL_connect:error in SSLv3 flush data - Certificate Verification: Error (20) when setting up replacement server

[Michel, Audrey]

We currently have a Solaris server running apache that serves as a proxy in front of our glassfish server which we use for web services. We are replacing both the proxy and glassfish servers so are setting the new servers up in parallel. The new proxy server already had apache and openssl installed before I started moving files from the old to the new. The issue I am running into is when I run apache with SSLVerifyClient required on the new server I am receiving 'unknown ca' error. If I run with SSLVerifyClient none the xml content for our web service is displayed (confirming that glassfish is ok). The only configuration difference between the proxy server was I moved the location of the certificate directories on the new server, but adjusted all apache config files to reflect the new structure.

My question is does openssl or mod_ssl look anywhere other than the apache ssl configuration to locate the ca?

Thanks in advance for any assistance, server info and log details are below:

-We have a single self signed CA which is used to sign the server and client certs.
-The old and new server are running apache 2.0.63, Openssl 0.9.7d
Apache and openssl were present on the new server before I started bringing files over.
I ran the version commands and it show apache to be the same version and compiled options.
openssl is the same version but includes additional security patches.
OpenSSL 0.9.7d 17 Mar 2004 (+ security fixes for: CVE-2005-2969 CVE-2006-2937 CVE-2006-2940 CVE-2006-3738 CVE-2006-4339 CVE-2006-4343 CVE-2006-7250 CVE-2007-5135 CVE-2007-3108 CVE-2008-5077 CVE-2008-7270 CVE-2009-0590 CVE-2009-2409 CVE-2009-3555 CVE-2010-4180 CVE-2011-4576 CVE-2011-4619 CVE-2012-0884 CVE-2012-1165 CVE-2012-2110 CVE-2012-2131 CVE-2012-2333)
-I rebuilt all hash files and the filenames generated match those on the old server so the CN's are the same.
-I compared permissions and file contents for the apache related directories.

Using s_client with just -connect and -showcerts parameters returns the CA cert, server cert and my client cert. So I can see they are present on the new server.

Below is an abbreviated version of the output when I use the -connect, -cert, -CApath, -showcerts, -state, and -debug parameters. This works without error on the old server (changing only the IP:port).

Loading 'screen' into random state - done
CONNECTED(00000100)
SSL_connect:before/connect initialization
write to 0x1f03cc0 [0x1f085f0] (210 bytes => 210 (0xD2))
..
SSL_connect:SSLv2/v3 write client hello A
read from 0x1f03cc0 [0x1f0db50] (7 bytes => 7 (0x7))
...
SSL_connect:SSLv3 read server hello A
read from 0x1f03cc0 [0x1f0db53] (5 bytes => 5 (0x5))
read from 0x1f03cc0 [0x1f0db58] (1957 bytes => 1957 (0x7A5))
depth=1 CN = ALERTIIS Cert Auth, .. details omitted here by me....
verify error:num=19:self signed certificate in certificate chain
verify return:0
SSL_connect:SSLv3 read server certificate A
read from 0x1f03cc0 [0x1f0db53] (5 bytes => 5 (0x5))
read from 0x1f03cc0 [0x1f0db58] (525 bytes => 525 (0x20D))
..
SSL_connect:SSLv3 read server key exchange A
read from 0x1f03cc0 [0x1f0db53] (5 bytes => 5 (0x5))
read from 0x1f03cc0 [0x1f0db58] (421 bytes => 421 (0x1A5))
..
SSL_connect:SSLv3 read server certificate request A
SSL_connect:SSLv3 read server done A
write to 0x1f03cc0 [0x1f18570] (911 bytes => 911 (0x38F))
..
SSL_connect:SSLv3 write client certificate A
write to 0x1f03cc0 [0x1f18570] (139 bytes => 139 (0x8B))
..
SSL_connect:SSLv3 write client key exchange A
write to 0x1f03cc0 [0x1f18570] (267 bytes => 267 (0x10B))
..
SSL_connect:SSLv3 write certificate verify A
write to 0x1f03cc0 [0x1f18570] (6 bytes => 6 (0x6))
SSL_connect:SSLv3 write change cipher spec A
write to 0x1f03cc0 [0x1f18570] (53 bytes => -1 (0xFFFFFFFF))
SSL_connect:error in SSLv3 flush data
SSL_connect:error in SSLv3 flush data
write:errno=10053
..

At the end of the appache log I see the following error (all other tags are [debug]:
[Thu Jul 18 08:07:07 2013] [error] Certificate Verification: Error (20): unable to get local issuer certificate
[Thu Jul 18 08:07:07 2013] [debug] ssl_engine_kernel.c(1749): OpenSSL: Write: SSLv3 read client certificate B
[Thu Jul 18 08:07:07 2013] [debug] ssl_engine_kernel.c(1768): OpenSSL: Exit: error in SSLv3 read client certificate B
[Thu Jul 18 08:07:07 2013] [debug] ssl_engine_kernel.c(1768): OpenSSL: Exit: error in SSLv3 read client certificate B
[Thu Jul 18 08:07:07 2013] [info] SSL library error 1 in handshake (server xxx.xxx.xxx.xxx:443, client xx.xx.xxx.xxx.xx)
[Thu Jul 18 08:07:07 2013] [info] SSL Library Error: 336105650 error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
[Thu Jul 18 08:07:07 2013] [info] Connection to child 2 closed with abortive shutdown(server xxx.xxx.xxx.xxx:443, client xx.xx.xxx.xxx.xx)
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org

[Dave Thompson]

> From: owner-ope...@openssl.org On Behalf Of Michel, Audrey
> Sent: Thursday, 18 July, 2013 15:24
> Subject: SSL_connect:error in SSLv3 flush data - Certificate
> Verification: Error (20) when setting up replacement server

Aside: the error isn't really in flush-data; it is the server
rejecting the received cert and closing, which occurs immediately
after and as a result of the flush-data.

> We currently have a Solaris server running apache that serves
> as a proxy in front of our glassfish server which we use for
> web services. We are replacing both the proxy and glassfish
> servers so are setting the new servers up in parallel. The
> new proxy server already had apache and openssl installed
> before I started moving files from the old to the new. The
> issue I am running into is when I run apache with
> SSLVerifyClient required on the new server I am receiving
> 'unknown ca' error. If I run with SSLVerifyClient none the
> xml content for our web service is displayed (confirming that
> glassfish is ok). The only configuration difference between
> the proxy server was I moved the location of the certificate
> directories on the new server, but adjusted all apache config
> files to reflect the new structure.
>
> My question is does openssl or mod_ssl look anywhere other
> than the apache ssl configuration to locate the ca?
>

mod_ssl doesn't as far as I know and openssl called from mod_ssl
certainly shouldn't. (openssl in some other situations, especially
but not only commandline, does.)

> Thanks in advance for any assistance, server info and log
> details are below:
>
> -We have a single self signed CA which is used to sign the
> server and client certs.
> -The old and new server are running apache 2.0.63, Openssl 0.9.7d

> <snip> but [openssl] includes additional security patches. <snip>

> -I rebuilt all hash files and the filenames generated match
> those on the old server so the CN's are the same.

Did you generate new certs&keys or copy existing ones? If you
copied then of course everything is the same. If you did new,
then the hash-name for a cert actually proves the subject *DN*
(which can include more than the CN) is the same in a specific
way (equal after canonicalization, not necessarily identical),
and says nothing about the issuer (which also can have CN).

If you did a new CA key&cert (with or without new client and
server certs under it) with the exact same name, that can
cause problems, depending on whether you (consistently) used
SKI/AKI for chaining, and it will certainly cause confusion
to people such as you trying to look at it. Don't do that.
If you want the same CA use the same one or an exact copy,
if want a different CA use a different name.

The only certs you need hashes for are CA certs, and you
say you have only one CA and thus one CA cert. There is
no need for entity certs (server or client) to be in the
truststore (SSLCACertPath or File, -CApath or file) or hashed --
although they do no harm since openssl won't ever find them.

Either way, are you using the same DNS name for the new server
(proxy) as the old, and using resolution trickery to reach
the one you want? If not, if you have the same specific CN
in both server certs it can't match both DNS names; you need
a wildcard in subject CN, or SAN (SubjectAlternativeNames).
(Note that s_client doesn't check "URL" against CN/SAN, but
real clients especially browsers will -- unless they're
configured to be insecure, and then they may be 'owned').

> -I compared permissions and file contents for the apache
> related directories.
>
> Using s_client with just -connect and -showcerts parameters
> returns the CA cert, server cert and my client cert. So I can
> see they are present on the new server.
>

It shouldn't. It should show the (server) chain which here is
the server cert and the CA cert (specifically the server's CA's
cert if that made a difference, which for you it doesn't),
server cert names again, then the requested client-CA *names*.
It doesn't show the client cert even if it sends one, which
it won't if you don't specify -cert and possibly -key. And
s_client will send the specified cert even if it doesn't match
the requested client-CA names (if any), although you say you
have only one CA so they should match.

> Below is an abbreviated version of the output when I use the
> -connect, -cert, -CApath, -showcerts, -state, and -debug
> parameters. This works without error on the old server
> (changing only the IP:port).
>

So apparently you have the client key and cert in one file,
as is a convenient and fairly common choice but not required.

<snip except at read server cert>

> depth=1 CN = ALERTIIS Cert Auth, .. details omitted here by me....
> verify error:num=19:self signed certificate in certificate chain
> verify return:0

That means s_client couldn't find the CA cert the server is using.
Is its -CApath the same directory apache is using (if on same
machine), or a copy with (at least) same CA cert and hashname?

Compare the server CA cert (the second block from -showcerts)
received from old server and from new server to each other
and to the (supposed) CA cert file(s?) in client's -CApath.

> At the end of the appache log I see the following error (all
> other tags are [debug]:
> [Thu Jul 18 08:07:07 2013] [error] Certificate Verification:
> Error (20): unable to get local issuer certificate

<snip>

Assuming this log is for the s_client attempt above, it suggests
that s_client is not sending any CA cert; that means the -cert
you specified for s_client was not issued under the (or any)
CAcert in -CApath. It also doesn't appear to be issued under
a CAcert in the server's truststore. (For other clients, not
sending the CA cert may be normal, but having it not in the
truststore is still a problem.)

[Michel, Audrey]

Thanks for the reply and good information as it helped clear up some misunderstanding I had. Here is some additional information based on your responses.

-The two servers have different DNS names and have two different servernames in their apache config files. Each server is stand alone and has its own copy of everything - just trying to set up mirror images for now.
-I am using an exact copy of the ca cert (verified the second part of the chain using -showcerts and they match).
-I have created a new server cert for the new server using the existing server key but changed the CN to match the ip of the new server. And each -showcerts shows the right server cert in the chain.
-the path used on s_client for the ca is the same one in the ssl.conf file.
-when accessing both servers via s_client I am using the same client cert which is present on each server. Works for the old but not for the new so I agree that the issue is locating the ca cert.

What confuses me is if -showcerts shows the ca in the chain on both servers why is there a problem locating it when I use the same client cert/arguments only on the old server? I even removed the -CApath from both s_client commands and the old server connects but get the same error with the new server.

You mentioned a truststore.. I did a list on the java keystore cacerts on both machines and neither hold my ca. Does apache have its own store? I've never set up apache ssl from scratch and haven't run across anything specific in my searches on that subject.

[Dave Thompson]

> From: owner-ope...@openssl.org On Behalf Of Michel, Audrey

> Sent: Friday, 19 July, 2013 11:41

> Thanks for the reply and good information as it helped clear
> up some misunderstanding I had. Here is some additional
> information based on your responses.
>
> -The two servers have different DNS names and have two
> different servernames in their apache config files. Each
> server is stand alone and has its own copy of everything -
> just trying to set up mirror images for now.

> -I am using an exact copy of the ca cert (verified the second
> part of the chain using -showcerts and they match).

> -I have created a new server cert for the new server using
> the existing server key but changed the CN to match the ip of
> the new server. And each -showcerts shows the right server
> cert in the chain.

Ah, I didn't realize you were using cert for IP-address
not cert for DNS-name like most people do. That means my
questions (and your answers) about DNS are irrelevant.

> -the path used on s_client for the ca is the same one in the
> ssl.conf file.

> -when accessing both servers via s_client I am using the same
> client cert which is present on each server. Works for the
> old but not for the new so I agree that the issue is locating
> the ca cert.
>

The client cert doesn't need to be present on the server
and is ignored if it is. (If you are as I have assumed
using at least one of the servers as your private CA, keeping
any issued cert in the CA is sensible though not required.)
What matters is that the client cert is issued by (aka under)
a CA cert that is on (and in the truststore of) the servers.

> What confuses me is if -showcerts shows the ca in the chain
> on both servers why is there a problem locating it when I use
> the same client cert/arguments only on the old server? I even
> removed the -CApath from both s_client commands and the old
> server connects but get the same error with the new server.
>

One thing occurs to me: are you using SSLCertificateChainFile
to specify the CA cert file? That's not normally needed but if
you're doing so it might be a factor (because then we don't know
if server libssl is filling from truststore, see below).

s_client -CApath only affects s_client verifying the server,
and s_client continues after a verify error anyway. You should
have *verify return* 0 on s_client going to the usable=old server
with correct -CApath, and 19 (or maybe 20 I forget) without,
but still have a successful connection. To the unusable=new
server you don't get nice session display with verify return,
but you can extract it from the callback info in the output.

> You mentioned a truststore.. I did a list on the java
> keystore cacerts on both machines and neither hold my ca.
> Does apache have its own store? I've never set up apache ssl
> from scratch and haven't run across anything specific in my
> searches on that subject.
>

I'm not talking about java. To be sure we're clear, 'apache'
without qualification usually means httpd, a web server in C.
httpd can (or at least could) use either openssl or gnutls for SSL
but mod_ssl means openssl. Apache the organization also has tomcat,
a web server in Java, which can use Java's own SSL implementation
JSSE, or openssl via APR (Apache Portable Runtime) not mod_ssl.

Truststore is a general concept for the set of CA certs something
or somebody trusts. In openssl normally, it is a single PEM file
containing one or more certs (SSLCACertficateFile in mod_ssl,
-CAfile on some commandline utilities) or a directory of one or
more PEM files each containing one cert with hash names or links
(SSLCACertificatePath or -CApath). Or an application can directly load
one or more certs from some other source (a database, an LDAP server,
whatever) but that's more work so people rarely do. In addition to
using the truststore to validate a peer cert/chain, libssl will
also use it to complete its own chain if needed, which is less
obvious and can be a little confusing.

In Java, a standard keystore file (JKS) can contain private keys
(really keypairs) with certs, and/or trusted certs by themselves.
A JKS that contains trusted certs functions as a truststore,
and the cacerts file distributed in each JRE is an important
example; it includes CA certs for the established public CAs
like Verisign and GoDaddy etc. that are widely trusted, so it
is usually suitable as a truststore for applications that want to
access the public Internet and Web, and is the default for JSSE.
You can add to cacerts the cert for any CA you want to generally
trust, including your private one or maybe one for a social group.
There are other kinds of truststores Java can use, somewhat more
easily than openssl because in Java you only have to write
(sub)classes to a fairly simple (checked) interface. But openssl
and mod_ssl can't use a Java JKS. (tomcat can and usually does.)

<snip earlier>

At this point, unless it's chain file confusing us, I'm baffled.

Assuming you have openssl commandline on both servers, I suggest
using s_server as a (much) simpler alternative to httpd and see
if that gives any better info. Run s_server on each server with
-accept available nonprivileged port i.e. >=1024.
-cert mod_ssl's SSLCertificateFile
-key SSLCertificateKeyFile if used
-CApath SSLCACertificatePath if used
-CAfile SSLCACertificateFile if used
-Verify 1 (note uppercase V)
Run s_client with -cert/key (and preferably -CApath or -CAfile)
and -showcerts and maybe -state to each of those addr:port .

HTH, good luck.