[Cherokee] Sporadic SSL Bad signature Error

[Ryan McIntosh]

Hi all,

I am running cherokee 0.99.43 on www.bestbridalprices.com

I am getting sporadic bad SSL cert errors on this domain.  The certificate has been in place since 2008 and this just stated happening about a week after I upgraded to 0.99.43.  The error doesn't happen consistently and there are no errors or related messages in the log files.  The issue appears to correct itself.  What can I look for to determine the cause of this error.

Ryan

[Ryan McIntosh]

Does anyone have information on this error?  The cherokee server is running an ecommerce site and my client is greatly troubled by this issue.  I can restart cherokee with a cron job periodically, but that really isn't a great fix.

I've compiled 0.99.43 and 0.99.42 using the following configuration:
  with options "'--with-wwwgroup=www-data' '--with-wwwuser=www-data' '--with-wwwroot=/var/www/default' '--prefix=/' '--exec-prefix=/usr/local/' '--with-mysql=no'"

It is compiled on Debian Lenny (fully up to date) with openSSL version is 0.9.8g-15+lenny6

When the problem occurs, wget and curl both indicate no connection issues.  Openssl s_client connects properly.  Firefox 3.5 shows a scary looking red screen that states:
    Peer's certificate has an invalid signature.
    (Error code: sec_error_bad_signature)

There is no way to add an exception in firefox or 'work around'


Restarting cherokee resolves the issue (until it happens again at a later random date and time).

It seems to be related to this thread: http://code.google.com/p/cherokee/issues/detail?id=594

I am going to install openssl-0.9.8m (the latest) and recompile cherokee against that version.  I will let you know what happens.  Until then, it would be great if someone acknowledges this issue actually exists.  There are a large number of servers out there running Debian and Ubuntu which are all running openssl 0.9.5g-whatever.  It would be fantastic if there was some minor change that could be made to cherokee to allow these servers to function normally.

Ryan

[sste...@gmail.com]

On Mar 15, 2010, at 12:03 PM, Ryan McIntosh wrote:

> Does anyone have information on this error? The cherokee server is running an ecommerce site and my client is greatly troubled by this issue. I can restart cherokee with a cron job periodically, but that really isn't a great fix.

Have you pulled the "bad certificate" to see whether it's pulling the default cert out sometimes, and serving the real one the rest of the time?

This would show one type of bug in the server whereas, if the certificate's getting corrupted as it's served, it might show something completely different.

S

_______________________________________________
Cherokee mailing list
Cher...@lists.octality.com
http://lists.octality.com/listinfo/cherokee

[Ryan McIntosh]

Testing this with openssl s_client before and after a restart show identical certificates being served - with the exception of the session id.

Ryan

[Ryan McIntosh]

Hi,

When I recompiled cherokee against a newer openssl, I did this:

in /usr/src/openssl-0.9.8m:
./config && make && make test && make install
(all tests passed)

I configured cherokee as follows:
cherokee config.status '0.99.43'
configured by ./configure, generated by GNU Autoconf 2.65,
  with options "'--with-wwwgroup=www-data' '--with-wwwuser=www-data' '--with-wwwroot=/var/www/default' '--prefix=/' '--exec-prefix=/usr/local/' '--with-mysql=no' '--with-libssl=/usr/local/ssl'"

When I did make, I verified -I/usr/local/ssl where applicable

I receive the following error when trying to start the newly compiled cherokee.

[15/03/2010 18:20:43.129] (error) cryptor_libssl.c:340 - OpenSSL: cannot use certificate file '/etc/cherokee/ssl/bestbridalprices.pem':  error:2606A074:engine routines:ENGINE_by_id:no such engine
[15/03/2010 18:20:43.129] (critical) server.c:746 - cannot initialize TLS for 'www.bestbridalprices.com' virtual host

Any ideas?

Ryan

[Stefan de Konink]

On Mon, 15 Mar 2010, Ryan McIntosh wrote:

> [15/03/2010 18:20:43.129] (error) cryptor_libssl.c:340 - OpenSSL: cannot use
> certificate file '/etc/cherokee/ssl/bestbridalprices.pem':
> error:2606A074:engine routines:ENGINE_by_id:no such engine
> [15/03/2010 18:20:43.129] (critical) server.c:746 - cannot initialize TLS
> for 'www.bestbridalprices.com' virtual host
>
> Any ideas?

ldd on the libraries, binary. Might still prefer your old one.

Stefan

[Ryan McIntosh]

Yes, I needed to fiddle with ld.so to get ssh not broken and cherokee using the new openssl since I don't have physical access to that server.

It is running now.  I will keep you posted on whether or not upgrading openssl has resolved this problem. 

Fingers Crossed.

Ryan

[Ryan McIntosh]

Hello,

This just broke again.  Upgrading to openssl 0.9.8M did not resolve this issue.  I really need someone to take a look at this.  It's not just me having the issue and soon, the client will be frustrated enough we will need to redeploy using an alternate front end webserver or loose his business.

What information can I provide to make troubleshooting this feasible?

Ryan

[Alvaro Lopez Ortega]

Hello Ryan,

We'd need to find a way to reproduce the issue consistently. Once we had that, the potential fix shouldn't delay much.

> _______________________________________________
> Cherokee mailing list
> Cher...@lists.octality.com
> http://lists.octality.com/listinfo/cherokee

--
Octality
http://www.octality.com/

[Ryan McIntosh]

OK, I've compiled cherokee with -O0 and -g following Stefan's recommendations.  I've also turned on --enable-trace and --enable-backtrace just in case, although I don't suspect it will show much based on the lack of errors in any log file so far.

I've installed this on the production server and will await the failure - which can take hours or days.  The only time I ever know it's failed is when a customer calls the client and he relays the info to me - by that time it's been broken for some time generally.  Wish I knew how to duplicate this.

Ryan

[Ryan McIntosh]

Unfortunately curl and wget connect just fine - the only https clients I have found that present an error are Firefox 3.0+ and chrome/chromium.  I have thought about writing a selenium test to refresh FF on the home page.  I think I will do so.

Some further information.  The service was up and stable for over a month on a 32-bit debian system (same version of c library and openssl).  Due to memory constraints we migrated the front end web server to a 64-bit Debian at the beginning of February and that's about when this started happening.  Yesterday, I compiled a version of cherokee exactly the same on a 32-bit Debian system and chrooted it on the 64-bit server.  I am going to run this cherokee binary on the 64-bit server and hopefully that alleviates the issue.  If it does, it would indicate an incompatibility with the 64-bit libc shipping with Debian lenny.

Ryan

On Wed, Mar 24, 2010 at 4:12 AM, Stefan de Konink <ste...@konink.de> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Op 22-03-10 23:53, Ryan McIntosh schreef:

> OK, I've compiled cherokee with -O0 and -g following Stefan's
> recommendations.  I've also turned on --enable-trace and
> --enable-backtrace just in case, although I don't suspect it will show
> much based on the lack of errors in any log file so far.
>
> I've installed this on the production server and will await the failure
> - which can take hours or days.  The only time I ever know it's failed
> is when a customer calls the client and he relays the info to me - by
> that time it's been broken for some time generally.  Wish I knew how to
> duplicate this.

You could make something that monitors the connection. For example a
curl script with a timeout that writes a file. If the file is not
present, then your thing is broken.


Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEAREKAAYFAkup13gACgkQYH1+F2Rqwn0rTwCfRKBnc+FyX5eDtdsy6Hjp0gKn
VloAn1y2hjgS7UVN90/zMMdXunU3FMcf
=bnLN
-----END PGP SIGNATURE-----

[Alvaro Lopez Ortega]

On 24/03/2010, at 14:43, Ryan McIntosh wrote:

> Unfortunately curl and wget connect just fine - the only https clients I have found that present an error are Firefox 3.0+ and chrome/chromium.

It may be related to keep-alive then. Neither curl, nor wget can use keep-alive between independent invocations.

> I have thought about writing a selenium test to refresh FF on the home page. I think I will do so.

Please do! That'd be *pretty* useful for tracing and fixing the issue.

> Some further information. The service was up and stable for over a month on a 32-bit debian system (same version of c library and openssl). Due to memory constraints we migrated the front end web server to a 64-bit Debian at the beginning of February and that's about when this started happening. Yesterday, I compiled a version of cherokee exactly the same on a 32-bit Debian system and chrooted it on the 64-bit server. I am going to run this cherokee binary on the 64-bit server and hopefully that alleviates the issue. If it does, it would indicate an incompatibility with the 64-bit libc shipping with Debian lenny.

It could be, although I think it's more likely related to TLS connections being left open (because of a keep-alive request) on situations when they should had been closed.

--
Octality
http://www.octality.com/

[Ryan McIntosh]

I tried disabling keep-alive today.  No dice.  However, I did manage to give the server a good workout.  I also was able to collect the following error using openssl s_client by forcing SSL3 and not allowing SSL2

read from 0x9a12c80 [0x9a18228] (5 bytes => 5 (0x5))
0000 - 16 03 00 01 8d                                    .....
read from 0x9a12c80 [0x9a1822d] (397 bytes => 397 (0x18D))
0000 - 0c 00 01 89 00 80 85 08-ff 6c c1 0c 23 55 c5 f8   .........l..#U..
0010 - 3d 47 6f 23 36 da 98 f3-e4 56 cd a0 f3 02 18 b0   =Go#6....V......
0020 - cb d2 92 4b dc 76 2b 24-2b 20 d9 6d 5b 59 c8 04   ...K.v+$+ .m[Y..
0030 - d1 67 fa 53 5d 46 a9 3a-62 72 1d e6 58 20 d7 60   .g.S]F.:br..X .`
0040 - f0 ad ce 52 d5 7b 21 80-5a 18 78 86 17 d5 e7 ee   ...R.{!.Z.x.....
0050 - 7d c7 35 eb 4d c1 4d 47-2a 89 3d b1 70 41 ce 04   }.5.M.MG*.=.pA..
0060 - 27 db 34 6d a3 0e 76 e6-56 f3 79 0b 95 59 97 4f   '.4m..v.V.y..Y.O
0070 - 24 06 f4 b8 b7 5b ef 7e-06 43 2a 8e 33 69 71 65   $....[.~.C*.3iqe
0080 - 35 bf cb cd b0 5b 00 01-02 00 80 3d 5f fe 0a 26   5....[.....=_..&
0090 - 30 3a f2 25 09 95 50 72-8e 9a 56 dc 69 91 85 6c   0:.%..Pr..V.i..l
00a0 - f0 44 fb 3c dd 45 3a 91-ee 08 1e f8 22 03 e9 9c   .D.<.E:....."...
00b0 - fb 59 ab 97 a0 bd f8 77-50 fb db e3 a1 4c b9 d6   .Y.....wP....L..
00c0 - fa 28 0c 8c 6b 84 d5 2a-be 9e fa 1e b7 dd 96 52   .(..k..*.......R
00d0 - 1f 07 ab 06 a5 12 13 70-9a 83 dd 78 bf a9 45 1c   .......p...x..E.
00e0 - 63 a4 c4 bb 74 9b 29 1c-77 72 88 16 92 82 e4 3e   c...t.).wr.....>
00f0 - 32 9c 5b 2c d5 d1 28 9f-66 d8 c1 70 48 b1 ff 28   2.[,..(.f..pH..(
0100 - 7a 45 98 d3 6f c0 96 fa-75 73 9e 00 80 9a a8 33   zE..o...us.....3
0110 - cc c4 0b 5a 0f 6f e4 b9-7d 22 11 5a 78 c6 aa f3   ...Z.o..}".Zx...
0120 - 54 22 1b 5a 03 8b c7 10-d6 30 e0 38 bc ad df e2   T".Z.....0.8....
0130 - 3a b5 95 aa 76 c7 fc c0-8b 39 cd 7e 9b 25 dc 33   :...v....9.~.%.3
0140 - 00 3e 36 d6 94 53 b7 13-12 f0 dd 14 cc ab 3f 19   .>6..S........?.
0150 - 9d d8 a8 ef 66 63 dc 9a-53 56 72 cd 57 a7 b7 84   ....fc..SVr.W...
0160 - 5c 10 f8 12 9b 14 90 0e-8b 0d dc c4 1c d9 cc 8e   \...............
0170 - 74 d4 d3 b3 23 19 4d 92-33 e6 1a 23 5b bc 32 57   t...#.M.3..#[.2W
0180 - 00 01 97 d0 5e 24 46 5a-27 e6 ac 2f a2            ....^$FZ'../.
write to 0x9a12c80 [0x9a223c0] (7 bytes => 7 (0x7))
0000 - 15 03 00 00 02 02 28                              ......(
10712:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100:
10712:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:699:
10712:error:1408D07B:SSL routines:SSL3_GET_KEY_EXCHANGE:bad signature:s3_clnt.c:1414:

Is that helpful?

There's nothing in the server logs unfortunately.

Ryan

[Ryan McIntosh]

Still having this error about twice a day.  I put a cron job in place to restart the server hourly since selenium testing for the failure proved less stable than the server itself - too many false positives.  Alas, the restart is less than ideal as we're dropping several hundred connections during peak times.

Even with an hourly restart, this error is still occuring sporadically.  Once further piece of information I didn't realize may be significant before is that I have not configured DH parameters.  I'm not sure if they're at all necessary as SSL was still working and I've never had to do this with any other webserver.  Are the DH parameters are used for generating the session keys?  Perhaps creating DH parameter files will do something for me?  I will test and write back.

Ryan

[Alvaro Lopez Ortega]

On 06/04/2010, at 15:53, Ryan McIntosh wrote:

> Even with an hourly restart, this error is still occuring sporadically. Once further piece of information I didn't realize may be significant before is that I have not configured DH parameters. I'm not sure if they're at all necessary as SSL was still working and I've never had to do this with any other webserver. Are the DH parameters are used for generating the session keys? Perhaps creating DH parameter files will do something for me?

The DH parameters file does not have anything to do with the problem, I'm quite sure about that.

I still believe that the problem is somehow related to keep-alive, unfinished connections, bad content-lenght and/or bad content-encoding.

> I will test and write back.

Thanks for all the finding and reports!

[Ryan McIntosh]

Since I turned off gzip encoding in the config this problem hasn't reared it's head yet.  I'll let it run like this for another couple of days for a longer term test, but this seems promising.

I realized something last night though that I am slapping myself for not flagging earlier.  This configuration uses virtual hosts.  The SSL_BAD_SIGNATURE error only occurs on one virtual host when it happens.  It's not always the same one.  Sometimes it's admin.bestbridalprices.com and sometimes it's www.bestbridalprices.com  I've never seen it on any other virtual host.  The configuration is not using a wildcard ssl certificate - the non www virtual hosts are for internal use only.

Since some virtual hosts work fine when the problem is occuring on another, it leads me to believe that whatever error is occuring AFTER the SSL handshake and at least before the http-host header is sent - possibly after.  The problem is arising during the HTTP portion of the communication.  Definately a significant clue.  It is no wonder openssl s_client wasn't producing any meaningful errors but for the one time.  It makes me wonder if the error I did see the once was unrelated.

Ryan

On Tue, Apr 6, 2010 at 12:05 PM, Ryan McIntosh <thebi...@gmail.com> wrote:

You're correct.  Adding the DH Parameter files did not resolve anything.  I just had to restart the server again.

Anything else I can try?

Alvaro, you mention bad content-length and/or bad content-encoding.  I'll try disabling gzip. 

When cherokee calculates content-length, does it consider encodings, or does it just count bytes?  I'm not familiar enough with HTTP to know if that's a dumb question or not.

Ryan

[Alvaro Lopez Ortega]

Hi Ryan,

That is REALLY interesting. Could you please check whether the problem shows up if you access different virtual servers from the same browser?

--

[Ryan McIntosh]

I haven't encountered the problem in nearly a week after turning off gzip encoding.  Any thoughts on why this might be a factor?

Ryan

On Wed, Apr 7, 2010 at 8:20 AM, Ryan McIntosh <thebi...@gmail.com> wrote:

Yes - same firefox 3.6 browser, two tabs - one is https://www.bestbridalprices.com, one is https://staging.bestbridalprices.com

www would be producing the error while staging would be fine.

I woke up at 4am this morning in a cold sweat with that revelation in my head.

Ryan

[Ryan McIntosh]

Just had this problem occur again last night.  With gzip encoding turned off, it seems much better (20 days this time instead of 6 hours!), but alas, it did produce the error until I restarted the webserver.

I will upgrade to the new release and restart tonight.  Time will tell if any of the included fixes may correct this error, too.

Ryan

[Alvaro Lopez Ortega]

On 26/04/2010, at 17:49, Ryan McIntosh wrote:

> Just had this problem occur again last night. With gzip encoding turned off, it seems much better (20 days this time instead of 6 hours!), but alas, it did produce the error until I restarted the webserver.

I couldn't figure the source of the problem yet. However this sort of findings will help a lot to catch the bug. Thanks!

> I will upgrade to the new release and restart tonight. Time will tell if any of the included fixes may correct this error, too.

Unfortunately, I don't think any of the patches on the server side will fix the issue. The vast majority of the changes in the latest Cherokee versions are focused on Cherokee-admin.

--
Octality
http://www.octality.com/

_______________________________________________
Cherokee mailing list
Cher...@lists.octality.com
http://lists.octality.com/listinfo/cherokee

--
Subscription settings: http://groups.google.com/group/cherokee-http/subscribe?hl=en