nss & curl slow initialization phase

[Barry Beckham]

Hello,

I'm currently trying to migrate from a RHEL5 -> RHEL6 box, which includes a
switch from curl using openssl -> nss by default.

I'm encountering a severe performance hit with curl + nss when attempting
to hit a HTTPS endpoint that requires client cert authentication, to the
tune of an additional ~1s in pre-transfer time.

>From debugging, this appears to be related to the nss initialization phase
and reading the client cert that is sent to the server.

Versions:
RHEL 6.4
libcurl-7.19.7-37.el6_4.x86_64
curl-7.19.7-37.el6_4.x86_64
nss-3.14.3-4.el6_4.x86_64

Example ssldump of problematic connection start:

TCP: <client>(51588) -> <server>(443) Seq 230486183.(0) SYN
TCP: <server>(443) -> <client>(51588) Seq 1803468462.(0) ACK 230486184
SYN
TCP: <client>(51588) -> <server>(443) Seq 230486184.(0) ACK 1803468463
New TCP connection #3: <client>(51588) <-> <server>(443)
TCP: <client>(51588) -> <server>(443) Seq 230486184.(96) ACK 1803468463
PUSH
3 1 1.0777 (1.0777) C>S V3.1(91) Handshake

So 1.0777s between the connection being established and the initial TLS
Handshake message being sent, which points to an issue with curl and/or nss
initialization.

Corresponding curl output:

16:45:30.899345 * About to connect() to <server> port 443 (#0)
16:45:30.899548 * Trying <server ip>... connected
16:45:30.939619 * Connected to <server> (<server ip>) port 443 (#0)
16:45:30.939656 * Initializing NSS with certpath: sql:/etc/pki/nssdb
16:45:30.944097 * CAfile: /var/misc/cacert.crt
CApath: none
16:45:31.987207 * NSS: client certificate from file
...

Note the ~1s incurred during the nss init phase, presumably related to
fetching the client cert file.

I had a hunch the slowdown may be due to using a cert file as opposed to
importing the cert into the nssdb. I've imported the client cert into the
nssdb, but curl can not seem to find it when I specify --cert <nickname>.

$ sudo certutil -A -t "C,," -d sql:/etc/pki/nssdb/ -n clientCert -i
/var/temp/client.crt
Enter Password or Pin for "NSS Certificate DB":

$ sudo certutil -L -d sql:/etc/pki/nssdb/

Certificate Nickname Trust
Attributes

SSL,S/MIME,JAR/XPI

clientCert C,,

$ curl --cert clientCert <everything else...>

...
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS: client certificate not found: clientCert
* NSS error -12227
* Closing connection #0
curl: (35) NSS: client certificate not found: clientCert


Any thoughts?

Thanks!

[thatsaf...@gmail.com]

On Thursday, December 12, 2013 7:01:06 PM UTC, Barry Beckham wrote:
...

> I'm currently trying to migrate from a RHEL5 -> RHEL6 box, which includes a
> switch from curl using openssl -> nss by default.

...

> I'm encountering a severe performance hit with curl + nss when attempting
> to hit a HTTPS endpoint that requires client cert authentication, to the
> tune of an additional ~1s in pre-transfer time.

...

One difference that could be worth exploring is the "access" calls made, see if an strace shows these calls. Also see https://bugzilla.redhat.com/show_bug.cgi?id=1044666

[aaron....@gmail.com]

Hi Barry - I've been having a similar issue and have spent days scouring the internet trying to find a solution to what I think is the same problem. How did you end up solving this?

Any help would be greatly appreciated!

Thanks,

Aaron