Evp_Encrypt_Init Segfault

Felix Dorner

unread,

Jan 30, 2006, 5:55:29 PM1/30/06

to

Hi,

the following code executes once, and does fine. Calling the function a
second time gives a segfault during the call marked by "-->"

unsigned char *encrypt_message(unsigned char *message, int inl, int *outl)
{
EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init(&ctx);
-->EVP_EncryptInit(&ctx, EVP_bf_ecb(), NULL, NULL);
EVP_CIPHER_CTX_set_key_length(&ctx, SHA_DIGEST_LENGTH);
EVP_EncryptInit(&ctx, NULL, k, NULL);
char *ret;
int tmp, ol;
ol = 0;
ret = (char *)malloc(inl + EVP_CIPHER_CTX_block_size(&ctx));
EVP_EncryptUpdate(&ctx, &ret[ol], &tmp, message, inl);
ol = tmp;
EVP_EncryptFinal(&ctx, &ret[ol], &tmp);
*outl = ol+tmp;
return ret;
}

Anything obvious that might lead to the segfault?

Thanks,
Felix
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org

Girish Venkatachalam

unread,

Jan 30, 2006, 10:24:29 PM1/30/06

to

Try calling EVP_CIPHER_CTX_cleanup(&ctx) at the end...

--- Felix Dorner <feli...@web.de> wrote:

> Hi,
>=20

> the following code executes once, and does fine.
> Calling the function a
> second time gives a segfault during the call marked
> by "-->"

>=20

> unsigned char *encrypt_message(unsigned char
> *message, int inl, int *outl)
> {
> EVP_CIPHER_CTX ctx;
> EVP_CIPHER_CTX_init(&ctx);
> -->EVP_EncryptInit(&ctx, EVP_bf_ecb(), NULL,
> NULL);
> EVP_CIPHER_CTX_set_key_length(&ctx,
> SHA_DIGEST_LENGTH);
> EVP_EncryptInit(&ctx, NULL, k, NULL);
> char *ret;
> int tmp, ol;

> ol =3D 0;
> ret =3D (char *)malloc(inl +

> EVP_CIPHER_CTX_block_size(&ctx));
> EVP_EncryptUpdate(&ctx, &ret[ol], &tmp,
> message, inl);

> ol =3D tmp;

> EVP_EncryptFinal(&ctx, &ret[ol], &tmp);

> *outl =3D ol+tmp;
> return ret;
> }
>=20
>=20

> Anything obvious that might lead to the segfault?

>=20
> Thanks,
> Felix
>
______________________________________________________________________
> OpenSSL Project =20
> http://www.openssl.org
> User Support Mailing List =20
> openss...@openssl.org
> Automated List Manager =20
> majo...@openssl.org
>=20

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around=20
http://mail.yahoo.com=20

Girish Venkatachalam

unread,

Jan 30, 2006, 10:25:46 PM1/30/06

to

Felix Dorner

unread,

Jan 31, 2006, 3:53:32 AM1/31/06

to

Girish Venkatachalam wrote:

>Try calling EVP_CIPHER_CTX_cleanup(&ctx) at the end...
>
>
>

I have tried this, does not change the situation. gdb output is

200 EVP_EncryptInit(&ctx, EVP_bf_ecb(), NULL, NULL);
(gdb) step

Program received signal SIGSEGV, Segmentation fault.
0xb7df82fb in mallopt () from /lib/libc.so.6

I really dont know whats wrong here,

Felix

Alain Damiral

unread,

Jan 31, 2006, 4:11:47 AM1/31/06

to

Have you tried with the EVP_EncryptInit_ex() family of functions ? I'm
not sure it would help much but it could be worth a try

Felix Dorner wrote:

--
Alain Damiral

clarksom

unread,

Jan 31, 2006, 11:42:28 AM1/31/06

to

I'm not much of an expert with any of this, but you may want to look at
some of the return values of some of the functions to make sure
everything is good, such as on EVP_EncryptFinal. Please take a look at
some code I did up last summer in C++ (but it is almost all C), located at
http://lunir.com/Encryption.cpp. The Function in particular would be int
Encryption::encrypt(std::ifstream &istream, std::ofstream &ostream). It
deals with streams but converts it all into cstrings, so it should
roughtly be the same. Hope this helps.

--
Matthew Clarkson

On Mon, 30 Jan 2006, Felix Dorner wrote:

> Hi,

>
> the following code executes once, and does fine. Calling the function a
> second time gives a segfault during the call marked by "-->"
>

> unsigned char *encrypt_message(unsigned char *message, int inl, int *outl)
> {
> EVP_CIPHER_CTX ctx;
> EVP_CIPHER_CTX_init(&ctx);
> -->EVP_EncryptInit(&ctx, EVP_bf_ecb(), NULL, NULL);
> EVP_CIPHER_CTX_set_key_length(&ctx, SHA_DIGEST_LENGTH);
> EVP_EncryptInit(&ctx, NULL, k, NULL);
> char *ret;
> int tmp, ol;

> ol = 0;
> ret = (char *)malloc(inl + EVP_CIPHER_CTX_block_size(&ctx));

> EVP_EncryptUpdate(&ctx, &ret[ol], &tmp, message, inl);

> ol = tmp;

> EVP_EncryptFinal(&ctx, &ret[ol], &tmp);

> *outl = ol+tmp;
> return ret;

> }
>
>
> Anything obvious that might lead to the segfault?
>

> Thanks,

Felix Dorner

unread,

Jan 31, 2006, 4:35:40 PM1/31/06

to

I somehow corrupted the ctx object by overshooting the malloced area as
Mark had pointed out.
The problem is solved now,

thanks, felix