Need help diagnosing basic SSL handshake trouble. Possible behavior change between 0.4.12 and 0.6.2

[George Snelling, Seattle]

SSL noob question.   

When trying upgrade node from 0.4.12 to 0.6.2 for our production servers, I ran into trouble making an SSL connection with a signed certificate. In dev mode we use a self-signed SSL cert, and all works fine, both pre- and post- upgrade from 0.4 to 0.6. In production, however, we use a signed cert from godaddy.  Superficially, the production SSL connection code that worked fine using node 0.4.12 now fails using node 0.6.2.  Several aspects of our environment have changed in the upgrade, however, and I suspect our code far more than node's.  My question is this: How do folks diagnose basic SSL handshake problems with a node https server?   

Following the tls docs, (http://nodejs.org/docs/v0.6.3/api/tls.html#tls.Server) I have poked for a few hours into openssl s_client.  Running it locally on the production server yields the following: 

openssl s_client -connect 127.0.0.1:443

CONNECTED(00000003)

140557833025352:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:

---

no peer certificate available

---

No client certificate CA names sent

---

SSL handshake has read 0 bytes and written 113 bytes

---

New, (NONE), Cipher is (NONE)

Secure Renegotiation IS NOT supported

Compression: NONE

Expansion: NONE

---

The same command issued against the same server on a different port using self-singed certs yields much happier output.  Is the answer in front of my nose and I just don't understand it?  Or is there some problem upstream, and if so, is there any way to capture the server's perspective of the request?  

I'm looking not so much for a solution as for tips and tricks for diagnoses. 

Thanks, 

-George 

 

  

[Ben Noordhuis]

George, what options do you pass to https.createServer()? What version
of openssl is node compiled against? (`node -e
'console.error(process.versions)'` will tell you).

Test with `openssl s_client -msg`, that should tell you what parts of
the SSL/TLS handshake succeed (probably just the CLIENT-HELLO but it's
worth testing anyway).

If you get past CLIENT-HELLO, try the -ssl2, -ssl3 and -tls1 options
of s_client and their -no_opt counterparts - maybe the server doesn't
support SSL3/TLS1.

[George Snelling, Seattle]

Hi Ben, 

Thanks for your response.  My limited google-fu leads me to believe that the problem is not with node, but with openssl and my particular cert config.  If you could confirm or deny whether the version of openssl shipped with node changed between 0.4 and 0.6 that might help.  

Both our old and the new production boxes are built up from amazon's linux AMI.

uname -a

Linux domU-12-31-39-0A-19-03 2.6.35.14-103.47.amzn1.x86_64 #1 SMP Fri Nov 18 04:03:11 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

On the new box, the one with the SSL trouble, 

node -e 'console.error(process.versions)' yields

{ node: '0.6.3',

  v8: '3.6.6.8',

  ares: '1.7.5-DEV',

  uv: '0.6',

  openssl: '1.0.0-fips' }

On the new prod box I installed node using yum (yeah!).  On our old prod boxes I built node 0.4.12 from source using defaults.    

Options passed to https.createServer: 

// paraphrase of real code

var options = {

    key: fs.readFileSync(ssl.keyFilePath),

    cert: fs.readFileSync(ssl.certFilePath),

    ca: [ fs.readFileSync(ssl.rootCertFilePath) ]

  }

I did not change any other options from their defaults.  

openssl s_client -msg dies at CLIENT-HELLO, so no luck with the -ssl* options:

=========

root@domU-12-31-39-0A-19-03 lib]# openssl s_client -msg -connect 127.0.0.1:443

CONNECTED(00000003)

>>> TLS 1.0 Handshake [length 006c], ClientHello

    01 00 00 68 03 01 4e d5 c4 9b ca a9 e2 7f e1 2d

    65 a6 4f a4 07 0c 7f 60 e3 86 dd e3 28 2f c9 93

    ef f7 5c 7f 1f 7e 00 00 3a 00 39 00 38 00 88 00

    87 00 35 00 84 00 16 00 13 00 0a 00 33 00 32 00

    9a 00 99 00 45 00 44 00 2f 00 96 00 41 00 05 00

    04 00 15 00 12 00 09 00 14 00 11 00 08 00 06 00

    03 00 ff 02 01 00 00 04 00 23 00 00

139971677091656:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:

---

no peer certificate available

---

No client certificate CA names sent

---

SSL handshake has read 0 bytes and written 113 bytes

---

New, (NONE), Cipher is (NONE)

Secure Renegotiation IS NOT supported

Compression: NONE

Expansion: NONE

---

============

We're using the gd_bundle.crt, which contains three certs in one file.  This worked with node 0.4.12.  Unless you spot an obvious problem with what I'm doing I'm going to take this over to the openSSL list and see if they can help me decipher this little gem: 

139971677091656:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:

Thanks again Ben,

-George 

[Ben Noordhuis]

That message means the server hasn't sent a hello message of his own
(which s_client -msg confirmed). I don't think the openssl people will
be able to help you, this looks like it's either an issue with Node,
your local environment or the distro maintained Node package.

Some more things you can try:

- clone the main repo, `git checkout v0.6.3`, build and test.

- clone the main repo, stay on master. After ./configure, edit
options.gypi and set node_use_system_openssl to false, that makes it
compile against the bundled openssl 0.9.8r).

[George Snelling, Seattle]

The plot thickens.  I uninstalled node using yum, then cloned the repro, checked out v.0.6.3 and buit.  Running the tests, however, did not come up clean.  There were a few complaints about forking a child process.  Don't know if these are already known or not.  I saved the output if you'd like me to post.  However my internal tests on my app all passed, but the same SSL connection problem reproed when using a signed cert.   So ignoring node's test failures for a moment (I know -- probably bad idea), I checked out master and did as you suggested and set node_use_system_openssl to false, then attempted to build again.  This time the build failed: 

...

  CC(target) /usr/local/node/out/Release/obj.target/openssl/deps/openssl/openssl/crypto/dso/dso_openssl.o

  CC(target) /usr/local/node/out/Release/obj.target/openssl/deps/openssl/openssl/crypto/dyn_lck.o

  CC(target) /usr/local/node/out/Release/obj.target/openssl/deps/openssl/openssl/crypto/ebcdic.o

  CC(target) /usr/local/node/out/Release/obj.target/openssl/deps/openssl/openssl/crypto/ec/ec2_mult.o

In file included from ../deps/openssl/openssl/include/openssl/ec.h:1,

                 from ../deps/openssl/openssl/crypto/ec/ec_lcl.h:76,

                 from ../deps/openssl/openssl/crypto/ec/ec2_mult.c:72:

../deps/openssl/openssl/include/openssl/../../crypto/ec/ec.h:78:2: error: #error EC is disabled.

make[1]: *** [/usr/local/node/out/Release/obj.target/openssl/deps/openssl/openssl/crypto/ec/ec2_mult.o] Error 1

make[1]: Leaving directory `/usr/local/node/out'

make: *** [all] Error 2

I'm beginning to think that starting from Amazon's default linux AMI might have been a bad idea...

-George   

[George Snelling, Seattle]

A little more info:  I carefully purged all node / npm / node_modules from the machine.  Then did a fresh clone, ./configure, then set node_use_system_openssl to false, and tried a clean make. Got the same build failure as above.  Just to make sure, I repeated the whole process and ran make as the root user. Same result. I am stumped.  

[George Snelling, Seattle]

Hi Ben, 

Proceeding on the hypothesis that there is some problem between node and my system's openssl lib (output of openssl version: OpenSSL 1.0.0e-fips 6 Sep 2011) I'm trying to build node using its own checked-in and tested version of openssl.  If I sync to master and set node_use_system_openssl to false in options.gypi, the build fails as described above. However, if I checkout any other tag, v0.6.3 or v0.4.12 and run make, the build finishes without complaint, but it doesn't seem to pay attention to node_use_system_openssl: false flag, and I still get my system's openssl lib, not the one checked into node's tree.  e.g: 

Running node -e 'console.error(process.versions)' produces:  

{ node: '0.4.12',  v8: '3.1.8.26',  ares: '1.7.4',  ev: '4.4',  openssl: '1.0.0e-fips' }

This seems bad.  If I can't revert to node version 0.4.12 including node's built-in and tested version of openssl, then I need to revert my OS to an unpatched, unsupported state. 

Put another way, when you said above:  

=== ben ==========

- clone the main repo, stay on master. After ./configure, edit
options.gypi and set node_use_system_openssl to false, that makes it
compile against the bundled openssl 0.9.8r).

=================

why did you say, "stay on master"?

Many thanks, 

-George 

[Ben Noordhuis]

That error you get means something somewhere defined OPENSSL_NO_EC and
it's probably the system's openssl headers in /usr/include or
/usr/local/include. As a workaround you can (temporarily) uninstall
the openssl-devel package.

[Ben Noordhuis]

Building the bundled openssl only works with the master branch right
now (v0.6 and master have different build systems, that's why).

[George Snelling, Seattle]

Got it.  Make sense.  Will try and report back.  Again, many thanks.

-George 

[George Snelling, Seattle]

Hi Ben, 

Ok, finally figured it out.  There's no problem between node and SSL libs, and there's no problem with AWS's linux distribution.  It was all user error on my part.  I installed an incorrect private key on the new machine.   

However, I think there was some change between node 0.4 and node 0.6 that made my mistake harder to diagnose than it should have been.  

In node 4.12, if a client hits an https server that has a mis-matched private key and root cert, node poops the following to stderr, I presume bubbling up from openssl: 

   (node SSL) error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher

In node 6.*, however, given the same problem, no error appears.  The sever simply hangs up on the client, no error is generated at the server, and the client's callback never fires.  Node's swallowing openssl's error in this case, rather than bubbling it up, seems like a bug in node to me. If you agree, I will log it.  

Thanks again for your kind help, 

-George

[Bert Belder]

On Dec 2, 9:10 am, "George Snelling, Seattle" <georg...@gmail.com>
wrote:

> In node 6.*, however, given the same problem, no error appears.  The sever
> simply hangs up on the client, no error is generated at the server, and the
> client's callback never fires.  Node's swallowing openssl's error in this
> case, rather than bubbling it up, seems like a bug in node to me. If you
> agree, I will log it.

It is a bug. Ticket: https://github.com/joyent/node/issues/2246.

- Bert

[Stephen Pair]

I'm not sure it's the same error, but I recently upgraded our pre-production server from 0.4.9 to 0.6.3 and experienced similar sounding issues…the first few requests of a page seem to succeed, then the page loading just hangs and never finishes.  Things seemed to work find in my dev environment (Mac connecting to local host).  The pre-production server is Ubuntu.  We are also using SSL certs signed by godaddy (extended validation).  Everything works fine again after reverting back to 0.4.9.  I haven't had any time yet to further diagnose the problem, but thought it might be helpful to go ahead and report report here what I've observed.  

 

[Stephen Pair]

I should add that issue 2246 seems to imply the only issue here is a failure of node to report an SSL error.  However, from what I've observed, there is a more serious problem (could be some change that my code needs to migrate from 0.4.9 to 0.6.3 that I'm unaware of).  Everything works find on node 0.4.9 with no SSL related errors reported to stdout.  Changing only the version of node to 0.6.3 (or 0.6.2) and https requests start hanging (the first few on a web page seem to work, then 4 or 5 requests into loading a page and the page just seems to hang and never finish loading…hitting refresh in the browser doesn't help (same behavior after hitting refresh…first few requests work, then page loading hangs)).

P.S.  I also tried 0.6.2 (because that's what I had been using in my dev environment and it had the same issues)

[Ben Noordhuis]

Is your problem related to [1]? It's fixed in v0.6.4.

[1] https://github.com/joyent/node/commit/5451ba3

[Stephen Pair]

The description of the problem seems similar, so I imagine it is…I'll give 0.6.4 a try…thanks! 

[George Snelling, Seattle]

Hi Ben, 

I upgraded to 0.6.5 and my original problem still repros:  if I replace the correct private key on the server with an incorrect one, then bounce the server and attempt to reconnect, the connection fails silently and nothing is written to the server's stderr.  I just stuck a comment in the bug that Bert opened above.  (https://github.com/joyent/node/issues/2246)  I'm not blocked any more, but it was hell to figure out.  I do believe this is a regression from 0.4.12.  

Thanks, 

George