Transferring hash state
Sands, Daniel N.
We have an application that performs hashing across multiple hosts in a kind of round-robin form. In the past, we could reach into whatever hash we were using and pull its state, and write it into the hash state on the next host to continue processing. With the new opaque hash structures, that ability seems to be lost. Is that the case, or is it still somehow possible?
Viktor Dukhovni
abstractions, then the (deprecated, but still available in OpenSSL 3.x)
functions that take a concrete SHA256_CTX admit serialising the context,
because that structure is not opaque.
<openssl/sha.h>
typedef struct SHA256state_st {
SHA_LONG h[8];
SHA_LONG Nl, Nh;
SHA_LONG data[SHA_LBLOCK];
unsigned int num, md_len;
} SHA256_CTX;
OSSL_DEPRECATEDIN_3_0 int SHA256_Init(SHA256_CTX *c);
OSSL_DEPRECATEDIN_3_0 int SHA256_Update(SHA256_CTX *c,
OSSL_DEPRECATEDIN_3_0 int SHA256_Final(unsigned char *md, SHA256_CTX *c);
If you want "algorithm agility", then unless I'm mistaken, I'm afraid
that indeed the abstract interface currently lacks a way to serialise
and deserialise the internal state, to allow to move "out of process".
If some day implemented, This would surely require new "provider"
mechanisms, and would be available only for provider/algorithm
combinations that support such serialisation. Right now that set of
provider/algorithm combinations is empty.
--
Viktor.
Tomas Mraz
https://github.com/openssl/openssl/issues/14222
Tomas Mraz, OpenSSL
Tomáš Mráz, OpenSSL
Sands, Daniel N.
I have no issues with using provider-specific param names. But as far as the OpenSSL provider (fips=yes or fips=no), I'd say that most of the digest algorithms are pretty stable by now, especially the old fogies. It would be useful to keep providing the OpenSSL internal (currently deprecated) state structures, but only as a method of interpreting the internal state after calling for a pointer to it. So as long as I know which digest I'm using, we can still interpret and manually serialize the state data as we have already been doing, once we get a pointer to it. Just taking a glance at the 3.0.1 source code, I see that MD5, SHAx, and Blake2 seem to still use the deprecated interfaces under the covers.
I suppose MACs can be more complicated, but I'm only asking for digests.
Sent: Friday, October 25, 2024 1:40 AM
To: openss...@openssl.org <openss...@openssl.org>
Subject: [EXTERNAL] Re: Transferring hash state
You received this message because you are subscribed to the Google Groups "openssl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openssl-user...@openssl.org.
To view this discussion visit https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fa%2Fopenssl.org%2Fd%2Fmsgid%2Fopenssl-users%2F83df8edaa3c2bc4fa028083292abb2df08dc86fb.camel%2540openssl.org&data=05%7C02%7Cdnsands%40sandia.gov%7C10dbe824d9b24524be8608dcf4c85bf5%7C7ccb5a20a303498cb0c129007381b574%7C1%7C0%7C638654388578226611%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=dbLxivqEhgLhzBZIwgUOcxBzeCdYw5OY%2FXwjMPP%2FjuI%3D&reserved=0.
Sands, Daniel N.
The docs state this:
EVP_DecryptInit_ex2(), EVP_DecryptInit_ex(), EVP_DecryptUpdate() and EVP_DecryptFinal_ex()
These functions are the corresponding decryption operations. EVP_DecryptFinal() will return an error code if padding is enabled and the final block is not correctly formatted. The parameters and restrictions are identical to the encryption
operations except that if padding is enabled the decrypted data buffer out passed to EVP_DecryptUpdate() should have sufficient room for (inl + cipher_block_size) bytes unless the cipher block size is 1 in which case inl bytes is sufficient.
Why does the decrypted buffer need to be “inl + cipher block size”? The ciphertext will be padded to a multiple of cipher block size, but the amount of decrypted text will be less than the amount of ciphertext given.
In addition, there seems to be a restriction which is NOT documented here (or is it a bug?) I’ll put together a simple example:
uint8_t *buf; <function arg, buffer the size of the expected plaintext>
size_t bufsize; <function arg, expected size>
uint8_t pad[16];
<Read the ciphertext into buf and the final block into pad. Do the usual setup for AES-128 CBC.>
int len;
EVP_DecryptUpdate(ctx, buf, &len, buf, bufsize & ~15);
int padlen;
EVP_DecryptUpdate(ctx, pad, &padlen, pad, 16);
int finallen;
EVP_DecryptFinal_ex(ctx, <any destination buffer>, &finallen) will fail. This seems to be because pad was modified by in-place decryption.
If I instead do the following (very slight tweak here):
int len;
EVP_DecryptUpdate(ctx, buf, &len, buf, bufsize & ~15);
int padlen;
EVP_DecryptUpdate(ctx, buf + len, &padlen, pad, 16);
int finallen;
EVP_DecryptFinal_ex(ctx, <any destination buffer including pad>, &finallen) will succeed. Oddly enough, it will succeed here even if pad was erased just before this call.
<Assert that len + padlen + finallen equals the buffer size>
memcpy(buf + len + padlen, pad, finallen); // Assumes that pad was the destination for EVP_DecryptFinal_ex
Final question: Is the latter sequence considered couth for reading a stream into a buffer that is only the expected decrypted size, or does this depend on functionality that might change?
Matt Caswell
On Wed, 1 Oct 2025 at 22:59, 'Sands, Daniel N.' via openssl-users <openss...@openssl.org> wrote:The docs state this:
EVP_DecryptInit_ex2(), EVP_DecryptInit_ex(), EVP_DecryptUpdate() and EVP_DecryptFinal_ex()
These functions are the corresponding decryption operations. EVP_DecryptFinal() will return an error code if padding is enabled and the final block is not correctly formatted. The parameters and restrictions are identical to the encryption operations except that if padding is enabled the decrypted data buffer out passed to EVP_DecryptUpdate() should have sufficient room for (inl + cipher_block_size) bytes unless the cipher block size is 1 in which case inl bytes is sufficient.
What version of the docs are you using? That isn't what the current version says:"These functions are the corresponding decryption operations. EVP_DecryptFinal() will return an error code if padding is enabled and the final block is not correctly formatted. The parameters and restrictions are identical to the encryption operations. ctx MUST NOT be NULL."Where the relevant encryption restriction is:"For most ciphers and modes, the amount of data written can be anything from zero bytes to (inl + cipher_block_size - 1) bytes"
Why does the decrypted buffer need to be “inl + cipher block size”? The ciphertext will be padded to a multiple of cipher block size, but the amount of decrypted text will be less than the amount of ciphertext given.
You may call `EVP_DecryptUpdate` many times in order to stream data. However, when working with padded data we have no idea when the last block is coming. So if you pass some multiple of the block size bytes of ciphertext into `EVP_DecryptUpdate` then the last block will be buffered internally. If that turns out to be the final block of the whole stream of data then the padding will be stripped in the `EVP_DecryptFinal` call. But if you call `EVP_DecryptUpdate` again then the whole buffered block will be decrypted and passed back.So lets say the block size is 16, padding is in use and you call `EVP_DecryptUpdate` and pass 16 bytes of ciphertext. You will get no output at this point because we don't know if you're going to pass more, or whether that is the final block.Now lets say you pass an additional 1 byte of ciphertext. At this point we can decrypt the previously buffered block and pass all of that back. We can't yet decrypt the 1 byte you just passed because it's not a complete block.So in this case `inl` is 1, but the size of the output is the entire previously buffered block = 16. This is also equal to inl + cipher_block_size - 1 = 1 + 16 - 1 = 16Matt
In addition, there seems to be a restriction which is NOT documented here (or is it a bug?) I’ll put together a simple example:
uint8_t *buf; <function arg, buffer the size of the expected plaintext>size_t bufsize; <function arg, expected size>
uint8_t pad[16];
<Read the ciphertext into buf and the final block into pad. Do the usual setup for AES-128 CBC.>
int len;
EVP_DecryptUpdate(ctx, buf, &len, buf, bufsize & ~15);
int padlen;
EVP_DecryptUpdate(ctx, pad, &padlen, pad, 16);
int finallen;
EVP_DecryptFinal_ex(ctx, <any destination buffer>, &finallen) will fail. This seems to be because pad was modified by in-place decryption.
If I instead do the following (very slight tweak here):
int len;
EVP_DecryptUpdate(ctx, buf, &len, buf, bufsize & ~15);
int padlen;
EVP_DecryptUpdate(ctx, buf + len, &padlen, pad, 16);
int finallen;
EVP_DecryptFinal_ex(ctx, <any destination buffer including pad>, &finallen) will succeed. Oddly enough, it will succeed here even if pad was erased just before this call.
<Assert that len + padlen + finallen equals the buffer size>
memcpy(buf + len + padlen, pad, finallen); // Assumes that pad was the destination for EVP_DecryptFinal_ex
Final question: Is the latter sequence considered couth for reading a stream into a buffer that is only the expected decrypted size, or does this depend on functionality that might change?
--
You received this message because you are subscribed to the Google Groups "openssl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openssl-user...@openssl.org.
Sands, Daniel N.
The docs state this:
EVP_DecryptInit_ex2(), EVP_DecryptInit_ex(), EVP_DecryptUpdate() and EVP_DecryptFinal_ex()
These functions are the corresponding decryption operations. EVP_DecryptFinal() will return an error code if padding is enabled and the final block is not correctly formatted. The parameters and restrictions are identical to the encryption operations except that if padding is enabled the decrypted data buffer out passed to EVP_DecryptUpdate() should have sufficient room for (inl + cipher_block_size) bytes unless the cipher block size is 1 in which case inl bytes is sufficient.
What version of the docs are you using? That isn't what the current version says:
This is current for 3.0 to 3.3, which is what my distro offers.
Why does the decrypted buffer need to be “inl + cipher block size”? The ciphertext will be padded to a multiple of cipher block size, but the amount of decrypted text will be less than the amount of ciphertext given.
You may call `EVP_DecryptUpdate` many times in order to stream data. However, when working with padded data we have no idea when the last block is coming. So if you pass some multiple of the block size bytes of ciphertext into `EVP_DecryptUpdate` then the last block will be buffered internally. If that turns out to be the final block of the whole stream of data then the padding will be stripped in the `EVP_DecryptFinal` call. But if you call `EVP_DecryptUpdate` again then the whole buffered block will be decrypted and passed back.
So lets say the block size is 16, padding is in use and you call `EVP_DecryptUpdate` and pass 16 bytes of ciphertext. You will get no output at this point because we don't know if you're going to pass more, or whether that is the final block.
Now lets say you pass an additional 1 byte of ciphertext. At this point we can decrypt the previously buffered block and pass all of that back. We can't yet decrypt the 1 byte you just passed because it's not a complete block.
So in this case `inl` is 1, but the size of the output is the entire previously buffered block = 16. This is also equal to inl + cipher_block_size - 1 = 1 + 16 - 1 = 16
Okay, so this is for the case of adding less than a block size of cipher-text. That makes sense. The statement in question may be a bit too broad in its wording, which made me wonder, if I added another 16 bytes, why would I need 16+block-size to decrypt into?
Anyway, could you also comment on the weirdness I saw with the first case below, as explained with the pseudo-code, and if the latter case should be trusted as expected behavior?