Segfault when encrypting
Hannes Schüller
I'm trying to create a RSA signature for a message. Here's the rough
code:
-----
hash = (unsigned char *)malloc(RSA_size(rsa) * sizeof(unsigned char));
ciphertext = (char *)malloc(RSA_size(rsa) * sizeof(char));
signature = (char *)malloc(RSA_size(rsa) * sizeof(char));
if (ciphertext != NULL && signature != NULL && hash != NULL) {
memset(ciphertext, 0, RSA_size(rsa));
ok = RSA_private_encrypt(strlen(reply), (unsigned char *)reply,
(unsigned char *)ciphertext, rsa, RSA_PKCS1_PADDING);
if (ok < 0) {
derror1("Message encryption error: %s",
ERR_error_string(ERR_get_error(), (char
*)NULL)); return (true);
} else {
dtrace1("Message encryption successful; return value:
%d", ok); }
len = base64Encode(ciphertext, ok);
memset(hash, 0, RSA_size(rsa));
RIPEMD160((unsigned char *)ciphertext, len, hash);
memset(signature, 0, RSA_size(rsa));
ok = RSA_private_encrypt(RIPEMD160_DIGEST_LENGTH, hash,
(unsigned char *)signature, rsa, RSA_PKCS1_PADDING);
-----
That final line causes a segmentation fault. Here is a backtrace:
-----
#0 0x00007ffff74d0ba6 in ?? () from /lib/libc.so.6
#1 0x00007ffff74d2aa0 in malloc () from /lib/libc.so.6
#2 0x00007ffff7abc962 in CRYPTO_malloc (num=-142946720,
file=0x7ffff7b7f5a0 "bn_ctx.c", line=409) at mem.c:328 #3
0x00007ffff7ae3d17 in BN_POOL_get (ctx=0x62b680) at bn_ctx.c:409
#4 BN_CTX_get (ctx=0x62b680) at bn_ctx.c:293
#5 0x00007ffff7b012c6 in RSA_eay_private_encrypt (flen=20, from=<value
optimized out>, to=<value optimized out>, rsa=0x617010, padding=1) at
rsa_eay.c:358
#6 0x000000000040bd80 in handleCommunication (commSock=6,
confFile=0x7fffffffe010 "/etc/activation.conf", clientAddr=...) at
activation.c:234
#7 0x000000000040c6bf in main (argc=1, argv=0x7fffffffe518) at
activation.c:421
-----
The funny thing is that it *mostly* crashes at this point, but it
occasionally goes well. I've been able to reproduce this problem on
various machines, though, so I think I can rule out a problem with
faulty RAM.
Same behaviour when using the RSA_sign function instead.
After hours of debugging, I'm running out of ideas. What could be
causing this?
Hannes
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org
David Schwartz
Hannes Schuller wrote:
> hash = (unsigned char *)malloc(RSA_size(rsa) * sizeof(unsigned char));
> ciphertext = (char *)malloc(RSA_size(rsa) * sizeof(char));
> signature = (char *)malloc(RSA_size(rsa) * sizeof(char));
> if (ciphertext != NULL && signature != NULL && hash != NULL) {
> memset(ciphertext, 0, RSA_size(rsa));
> ok = RSA_private_encrypt(strlen(reply), (unsigned char *)reply,
> (unsigned char *)ciphertext, rsa, RSA_PKCS1_PADDING);
> if (ok < 0) {
> derror1("Message encryption error: %s",
> ERR_error_string(ERR_get_error(), (char
> *)NULL)); return (true);
> } else {
> dtrace1("Message encryption successful; return value:
> %d", ok); }
> len = base64Encode(ciphertext, ok);
> memset(hash, 0, RSA_size(rsa));
> RIPEMD160((unsigned char *)ciphertext, len, hash);
> memset(signature, 0, RSA_size(rsa));
> ok = RSA_private_encrypt(RIPEMD160_DIGEST_LENGTH, hash,
> (unsigned char *)signature, rsa, RSA_PKCS1_PADDING);
I'm very puzzled here. Why do you sign the reply and then sign a hash of the
signature? You say "Message encryption successful", but that's a signature
you're doing, not an encryption.
> That final line causes a segmentation fault. Here is a backtrace:
>
> -----
> #0 0x00007ffff74d0ba6 in ?? () from /lib/libc.so.6
> #1 0x00007ffff74d2aa0 in malloc () from /lib/libc.so.6
> #2 0x00007ffff7abc962 in CRYPTO_malloc (num=-142946720,
Since the fault is in 'malloc', that indicates something is trashing the
heap. Tools like 'valgrind' can find this for you. My guess would be that
the problem lies in 'base64Encode'. It doesn't seem to have any place to put
its output (and if it operates in place, it may overflow the memory
allocated for 'ciphertext'), but for all I know it calls 'realloc'
internally.
I have to also give you a generic warning -- there are some subtle clues in
your code that suggest that you do not know what you're doing. If you, or
anyone else, is going to rely on this code to meet any security
requirements, I *strongly* urge you to have the code evaluated by a security
expert sooner rather than later. It appears that you have designed this code
such that only the public key is needed to perform an operation that you
think of as decryption, and that's usually a sign of a serious design flaw.
Why are you don't things using these low-level functions anyway? OpenSSL
provides high-level functions with well-defined security properties.
DS
Hannes Schüller
I was under the impression that RSA_private_encrypt and
RSA_public_encrypt do nothing but encrypt the given payload. The
(non-quoted) code before this ensures the reply is shorter than the
regular message digest length.
What would be the prefered way for encryption?
> > That final line causes a segmentation fault. Here is a backtrace:
> >
> > -----
> > #0 0x00007ffff74d0ba6 in ?? () from /lib/libc.so.6
> > #1 0x00007ffff74d2aa0 in malloc () from /lib/libc.so.6
> > #2 0x00007ffff7abc962 in CRYPTO_malloc (num=-142946720,
>
> Since the fault is in 'malloc', that indicates something is trashing
> the heap. Tools like 'valgrind' can find this for you. My guess would
> be that the problem lies in 'base64Encode'. It doesn't seem to have
> any place to put its output (and if it operates in place, it may
> overflow the memory allocated for 'ciphertext'), but for all I know
> it calls 'realloc' internally.
Good point, I'll investigate.
> I have to also give you a generic warning -- there are some subtle
> clues in your code that suggest that you do not know what you're
> doing. If you, or anyone else, is going to rely on this code to meet
> any security requirements, I *strongly* urge you to have the code
> evaluated by a security expert sooner rather than later. It appears
> that you have designed this code such that only the public key is
> needed to perform an operation that you think of as decryption, and
> that's usually a sign of a serious design flaw.
I'm aware this is not the way it is intended and that it will only have
any security effect as far as signing the payload is concerned. Don't
worry. It's a long (boring) story.
Hannes
David Schwartz
Hannes Schuller wrote:
> > I'm very puzzled here. Why do you sign the reply and then sign a hash
> > of the signature? You say "Message encryption successful", but that's
> > a signature you're doing, not an encryption.
> I was under the impression that RSA_private_encrypt and
> RSA_public_encrypt do nothing but encrypt the given payload. The
> (non-quoted) code before this ensures the reply is shorter than the
> regular message digest length.
RSA is a low-level algorithm. It provides primitives called "encryption" and
"decryption". But whether they actually encrypt (in the sense of concealing
contents so that only a desired party can access them) depends on the system
as a whole.
So yes, RSA_private_encrypt performs the RSA primitive operation known as
"encryption". But it doesn't actually encrypt anything in the sense of
concealing contents to that only a desired party can access them. (Because
anyone with the public key can reverse the operation.)
> What would be the prefered way for encryption?
See the various EVP_Encrypt functions that provide encryption in the
conventional sense rather than the RSA_* functions that provide RSA
primitive operations that may or may not meet security objectives.
DS
Hannes Schüller
>
> Hannes Schuller wrote:
>
> > > I'm very puzzled here. Why do you sign the reply and then sign a hash
> > > of the signature? You say "Message encryption successful", but that's
> > > a signature you're doing, not an encryption.
>
> > I was under the impression that RSA_private_encrypt and
> > RSA_public_encrypt do nothing but encrypt the given payload. The
> > (non-quoted) code before this ensures the reply is shorter than the
> > regular message digest length.
>
> "encryption". But it doesn't actually encrypt anything in the sense of
> concealing contents to that only a desired party can access them. (Because
> anyone with the public key can reverse the operation.)
That's right, but in this scenario, the public key isn't supposed to be
public. As I said, I can't say I'm particularly proud of this whole setup,
but you've given me a fair warning which I accept.
Thanks for your help!
Hannes