Re: Please tell me about encryption API of OpenSSL 1.0.1

[MauMau]

Hello,


Thanks a lot for your valuable advice. I'm looking into the CBC with IVs
based on block numbers, CTR, and XTS. I'm refering to the pages below:

Block cipher modes of operation
http://en.wikipedia.org/wiki/Block_cipher_mode

Disk encryption theory
http://en.wikipedia.org/wiki/Disk_encryption_theory

I'll continue the investigation a little more.

Apart from that, let me go back to my original question 4 in my first mail.

----------------------------------------
Q4: Do I have to call EVP_EncryptInit_ex/EVP_DecryptInit_ex for each
block/record? I'm concerned about the overhead of those functions. For
example, I want to make function calls as follows. However, this does not
seem to work.

/* one-time initialization */
ERR_load_crypto_strings();
OpenSSL_add_all_algorithms();
EVP_CIPHER_CTX_init(&enc_ctx);
EVP_CIPHER_CTX_init(&dec_ctx);
EVP_EncryptInit_ex(&enc_ctx, EVP_aes_256_cbc(), NULL, key, iv);
EVP_CIPHER_CTX_set_padding(&enc_ctx, 0);
EVP_DecryptInit_ex(&dec_ctx, EVP_aes_256_cbc(), NULL, key, iv);
EVP_CIPHER_CTX_set_padding(&dec_ctx, 0);

/* encrypt first block */
EVP_EncryptUpdate(&enc_ctx, block1, &outlen, block1, 4096);
/* encrypt second block */
EVP_EncryptUpdate(&enc_ctx, block2, &outlen, block2, 4096);

/* decrypt second block */
EVP_DecryptUpdate(&dec_ctx, block2, &outlen, block2, 4096);
/* decrypt first block */
EVP_DecryptUpdate(&dec_ctx, block1, &outlen, block1, 4096);

The above code produces wrong data for block2. ...
----------------------------------------


I said I'm afraid of the overhead of EVP_EncryptInit_ex/EVP_DecryptInit_ex
calls. Then, I looked through the source code of those functions and knew
that my concern seems correct.

If I call those functions specifying the same cipher, key and IV as the
previous call, those functions call EVP_CIPHER_CTX_cleanup to clean up the
cipher context, re-allocate some memory for the cipher context, and performs
key and IV initialization. Those processing may be too much for each
block/record encryption/decryption.

According to the source code, I think I can pass NULL for cipher, key and IV
to use those specified in the previous call, eliminating most of the
processing overhead. Is this usage correct? The following code worked
successfully, producing the correct data:

----------------------------------------
/* one-time initialization */
ERR_load_crypto_strings();
OpenSSL_add_all_algorithms();
EVP_CIPHER_CTX_init(&enc_ctx);
EVP_CIPHER_CTX_init(&dec_ctx);
EVP_EncryptInit_ex(&enc_ctx, EVP_aes_256_cbc(), NULL, key, iv);
EVP_CIPHER_CTX_set_padding(&enc_ctx, 0);
EVP_DecryptInit_ex(&dec_ctx, EVP_aes_256_cbc(), NULL, key, iv);
EVP_CIPHER_CTX_set_padding(&dec_ctx, 0);

/* encrypt first block */
EVP_EncryptUpdate(&enc_ctx, block1, &outlen, block1, 4096);
/* encrypt second block */
EVP_EncryptInit_ex(&enc_ctx, NULL, NULL, NULL, NULL);
EVP_EncryptUpdate(&enc_ctx, block2, &outlen, block2, 4096);

/* decrypt second block */
EVP_DecryptUpdate(&dec_ctx, block2, &outlen, block2, 4096);
/* decrypt first block */
EVP_DecryptInit_ex(&dec_ctx, NULL, NULL, NULL, NULL);
EVP_DecryptUpdate(&dec_ctx, block1, &outlen, block1, 4096);
----------------------------------------


Regards
MauMau

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org

[Jeffrey Walton]

On Mon, Apr 23, 2012 at 8:16 AM, MauMau <maum...@gmail.com> wrote:
> Hello,
>
>
> Thanks a lot for your valuable advice. I'm looking into the CBC with IVs
> based on block numbers, CTR, and XTS. I'm refering to the pages below:
>
> Block cipher modes of operation
> http://en.wikipedia.org/wiki/Block_cipher_mode
>
> Disk encryption theory
> http://en.wikipedia.org/wiki/Disk_encryption_theory

You should have a look at Microsft's paper by Neils Ferguson on
Bitlocker's design and implementation. Its a very practical and
approachable paper. It seems to me you problem domain has a lot of
overlap with Bitlocker's requirements.

Microsoft did get thrown a bone on authnetication. That is, the CPU is
an arbitrator. If an adversary tampers with a file on disk, the
diffuser layer will perform adequate mixing so that the instructions
executed by the CPU will eventually brick the operating system.

"AES-CBC + Elephant diffuser: A Disk Encryption Algorithm for Windows
Vista," http://download.microsoft.com/download/0/2/3/0238acaf-d3bf-4a6d-b3d6-0a0be4bbb36e/BitLockerCipher200608.pdf.

Jeff

[Matt Caswell (frodo@baggins.org)]

On 23 April 2012 13:16, MauMau <maum...@gmail.com> wrote:

> /* encrypt first block */
> EVP_EncryptUpdate(&enc_ctx, block1, &outlen, block1, 4096);
> /* encrypt second block */
> EVP_EncryptInit_ex(&enc_ctx, NULL, NULL, NULL, NULL);
> EVP_EncryptUpdate(&enc_ctx, block2, &outlen, block2, 4096);

I believe this will reeuse the same IV for block2 that it uses for
block1. It will appear to work but is a really bad idea and will lead
to major security problems.

Matt

[MauMau]

From: <fr...@baggins.org>

> I believe this will reeuse the same IV for block2 that it uses for
> block1. It will appear to work but is a really bad idea and will lead
> to major security problems.

From: "Jeffrey Walton" <nolo...@gmail.com>
> You should have a look at Microsft's paper by Neils Ferguson on
> Bitlocker's design and implementation. Its a very practical and
> approachable paper. It seems to me you problem domain has a lot of
> overlap with Bitlocker's requirements.
> ...
>
> "AES-CBC + Elephant diffuser: A Disk Encryption Algorithm for Windows
> Vista,"
> http://download.microsoft.com/download/0/2/3/0238acaf-d3bf-4a6d-b3d6-0a0be4bbb36e/BitLockerCipher200608.pdf.

Thanks. I've just read the beginning of the Bitlocker paper. It looks
interesting and I'll read through it.

At present, I'm thinking of doing the following:

1.Generate one 128-bit or 256-bit random master key.
2.Generate one random 128-bit or 256-bit file encryption key and one random
128-bit IV for each file. Encrypt those keys and IVs with the master key and
store them on disk. That is, attackers cannot know the file keys and IVs
from the disk.
3.Use the same key and IV for all 4KB blocks in one file.

But folks here gave me suggestions that different IVs should be used for
each 4KB block. I think I should do that, and I'd like to follow those
precious advice.

(However, I'm wondering if it is really dangerous to use the same IV for all
blocks in a file, because the IVs are random and encrypted.)

Regards
MauMau

[Edward Ned Harvey]

> From: owner-ope...@openssl.org [mailto:owner-openssl-
> us...@openssl.org] On Behalf Of MauMau

>
> But folks here gave me suggestions that different IVs should be used for
> each 4KB block. I think I should do that, and I'd like to follow those
> precious advice.
>
> (However, I'm wondering if it is really dangerous to use the same IV for
all
> blocks in a file, because the IVs are random and encrypted.)

Ultimately, everything comes down to one basic point: A block cipher is a
black box function, which takes three inputs (key, IV, plaintext) and
generates one output (ciphertext). If for any reason you repeat your
inputs, then you will get repeat output. You just have to make sure you
don't do that.

Given your plaintext is prone to patterns and repetition... You just have
to make sure you don't repeat a key/IV combination. Just scope your key and
IV accordingly. Suppose you have a 128bit random secret key (dedicated to
some file) that means you've scoped yourself down to the individual file.
Suppose you have an IV which is serializable and guaranteed uniqe for all
the 128-bit blocks in the file... Then you have a solution. Your only leak
happens when the user repeatedly writes the same data back to the same block
where they previously wrote that data. Then an attacker can know the user
repeated their actions on that block of that file.

Instead, suppose you have only one key that you're using for all files. Now
your key is repeated across files, so unless you take some measure to ensure
the IV is unique across all files... Then you would have a problem. So if
you have a single key for all files, then you need some way to ensure the IV
is serializable and unique across all files.

If you are guaranteeing unique key & unique IV for all blocks of all files,
that's certainly sufficent, but it's also overkill.

[Matt Caswell (frodo@baggins.org)]

On 23/04/12 13:16, MauMau wrote:
>
> Apart from that, let me go back to my original question 4 in my first
> mail.
>
> ----------------------------------------
> Q4: Do I have to call EVP_EncryptInit_ex/EVP_DecryptInit_ex for each
> block/record? I'm concerned about the overhead of those functions. For
> example, I want to make function calls as follows. However, this does not
> seem to work.
>
> /* one-time initialization */
> ERR_load_crypto_strings();
> OpenSSL_add_all_algorithms();
> EVP_CIPHER_CTX_init(&enc_ctx);
> EVP_CIPHER_CTX_init(&dec_ctx);
> EVP_EncryptInit_ex(&enc_ctx, EVP_aes_256_cbc(), NULL, key, iv);
> EVP_CIPHER_CTX_set_padding(&enc_ctx, 0);
> EVP_DecryptInit_ex(&dec_ctx, EVP_aes_256_cbc(), NULL, key, iv);
> EVP_CIPHER_CTX_set_padding(&dec_ctx, 0);
>

> /* encrypt first block */
> EVP_EncryptUpdate(&enc_ctx, block1, &outlen, block1, 4096);
> /* encrypt second block */

> EVP_EncryptUpdate(&enc_ctx, block2, &outlen, block2, 4096);
>

> /* decrypt second block */
> EVP_DecryptUpdate(&dec_ctx, block2, &outlen, block2, 4096);
> /* decrypt first block */
> EVP_DecryptUpdate(&dec_ctx, block1, &outlen, block1, 4096);
>
> The above code produces wrong data for block2. ...

I have modified your code to use XTS, which I think will achieve what
you want to do:
- It supports random read and write access to your data
- It is standards based so you don't have to make up your own way of
doing things and potentially open yourself up to security issues
- You do not have to store a separate key and/or IV for each of the
records that you want to encrypt

Some things to note about using XTS:
- XTS is only supported in the latest (1.0.1) version of openssl so
you must be using that for this to work
- The key length is double the size of a standard AES key of the same
strength, i.e. for AES 256 you must supply a 512 bit key
- You do not have to use an IV. Instead you supply a so-called
"tweak". This needs to be unique for each record (what you have called a
"block") that you are working with. However it does not need to be
random - it can be a simple unique identifier for the block.

The modified code is as follows:

char tweak[16] = {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};

/* one-time initialization */
ERR_load_crypto_strings();
OpenSSL_add_all_algorithms();
EVP_CIPHER_CTX_init(&enc_ctx);
EVP_CIPHER_CTX_init(&dec_ctx);

EVP_EncryptInit_ex(&enc_ctx, EVP_aes_256_xts(), NULL, key, NULL);
/* KEY MUST BE 512 BITS!!! */
EVP_CIPHER_CTX_set_padding(&enc_ctx, 0);
EVP_DecryptInit_ex(&dec_ctx, EVP_aes_256_xts(), NULL, key, NULL);
EVP_CIPHER_CTX_set_padding(&dec_ctx, 0);

/* encrypt first block */

tweak[15] = 1; /* Tweak for block1 */
EVP_EncryptInit_ex(&enc_ctx, NULL, NULL, NULL, tweak);

EVP_EncryptUpdate(&enc_ctx, block1, &outlen, block1, 4096);

/* encrypt second block */

tweak[15] = 2; /* Tweak for block2 */
EVP_EncryptInit_ex(&enc_ctx, NULL, NULL, NULL, tweak);

EVP_EncryptUpdate(&enc_ctx, block2, &outlen, block2, 4096);

/* decrypt second block */

tweak[15] = 2; /* Tweak for block2 */
EVP_DecryptInit_ex(&dec_ctx, NULL, NULL, NULL, tweak);

EVP_DecryptUpdate(&dec_ctx, block2, &outlen, block2, 4096);

/* decrypt first block */

tweak[15] = 1; /* Tweak for block1 */
EVP_DecryptInit_ex(&dec_ctx, NULL, NULL, NULL, tweak);

EVP_DecryptUpdate(&dec_ctx, block1, &outlen, block1, 4096);

Hope that helps.

Matt

[MauMau]

From: <fr...@baggins.org>

> I have modified your code to use XTS, which I think will achieve what you
> want to do:
> - It supports random read and write access to your data
> - It is standards based so you don't have to make up your own way of
> doing things and potentially open yourself up to security issues
> - You do not have to store a separate key and/or IV for each of the
> records that you want to encrypt

Thank you for your experiment with XTS mode. I got a feeling that it may fit
me. Could you tell me anything you know about the following regarding XTS?

Q1: Is AES-XTS officially supported by OpenSSL 1.0.1? I'm wondering if XTS
is still an experimental feature in OpenSSL, because the file "Changes" in
the OpenSSL 1.0.1 tarball does not refer to XTS.

Please look at crypt/evp/evp_enc.c. The below code fragment in
EVP_CipherInit_ex() does not appear to have support code for XTS.

--------------------------------------------------
if(!(EVP_CIPHER_CTX_flags(ctx) & EVP_CIPH_CUSTOM_IV)) {
switch(EVP_CIPHER_CTX_mode(ctx)) {

case EVP_CIPH_STREAM_CIPHER:
case EVP_CIPH_ECB_MODE:
break;

case EVP_CIPH_CFB_MODE:
case EVP_CIPH_OFB_MODE:

ctx->num = 0;
/* fall-through */

case EVP_CIPH_CBC_MODE:
...
break;

case EVP_CIPH_CTR_MODE:
...
break;

default:
return 0;
break;
}
}
--------------------------------------------------


Q2: Is AES-XTS slower than AES-CBC? Does AES-NI speed up AES-XTS like
AES-CBC?


Thanks.

Regards
MauMau

[Matt Caswell (frodo@baggins.org)]

On 25/04/12 15:03, MauMau wrote:
>
> Q1: Is AES-XTS officially supported by OpenSSL 1.0.1? I'm wondering if
> XTS is still an experimental feature in OpenSSL, because the file
> "Changes" in the OpenSSL 1.0.1 tarball does not refer to XTS.
>

Well 1.0.1 is the latest stable version, and I have seen nothing to
indicate that the XTS support is anything but supported. I suspect it is
just an oversight in the ChangeLog.

> Please look at crypt/evp/evp_enc.c. The below code fragment in
> EVP_CipherInit_ex() does not appear to have support code for XTS.
>
> --------------------------------------------------
> if(!(EVP_CIPHER_CTX_flags(ctx) & EVP_CIPH_CUSTOM_IV)) {
> switch(EVP_CIPHER_CTX_mode(ctx)) {
>
> case EVP_CIPH_STREAM_CIPHER:
> case EVP_CIPH_ECB_MODE:
> break;
>
> case EVP_CIPH_CFB_MODE:
> case EVP_CIPH_OFB_MODE:
>
> ctx->num = 0;
> /* fall-through */
>
> case EVP_CIPH_CBC_MODE:
> ...
> break;
>
> case EVP_CIPH_CTR_MODE:
> ...
> break;
>
> default:
> return 0;
> break;
> }
> }
> --------------------------------------------------
>

This code is only relevant if the EVP_CIPH_CUSTOM_IV flag is not set. If
it is set it is ignored. XTS sets this flag in e_aes.c:

#define XTS_FLAGS (EVP_CIPH_FLAG_DEFAULT_ASN1 | EVP_CIPH_CUSTOM_IV \
| EVP_CIPH_ALWAYS_CALL_INIT | EVP_CIPH_CTRL_INIT)

That is why it does not appear to handle XTS.

>
> Q2: Is AES-XTS slower than AES-CBC? Does AES-NI speed up AES-XTS like
> AES-CBC?
>

Yes it is slower because there is an additional encryption operation on
the "tweak".
I think AES-NI speeds up the implementation of the underlying AES
cipher, and therefore would be used no matter what the mode (perhaps one
of the openssl developers can confirm??)

To test out the speed implications I knocked together a quick piece of
code to do 1,000,000 AES-256 XTS encryptions of a 4k record followed by
1,000,000 AES-256 CBC encryptions. XTS took approx. 108s to run, whilst
CBC took approx. 41s to run (on my underpowered netbook).

To put this into perspective that means that XTS took approx. 0.1ms to
encrypt a single 4k record, compared to 0.04ms for CBC. In other words a
0.06ms performance penalty. Now I don't know what your application is
attempting to do, but I suggest that in most scenarios that kind of
penalty is not going to be noticed, and will probably be negligible
compared to the file i/o.

I haven't done a test for decryptions but I would expect it to be similar.

Matt

[MauMau]

From: <fr...@baggins.org>

> This code is only relevant if the EVP_CIPH_CUSTOM_IV flag is not set. If
> it is set it is ignored. XTS sets this flag in e_aes.c:
>
> #define XTS_FLAGS (EVP_CIPH_FLAG_DEFAULT_ASN1 | EVP_CIPH_CUSTOM_IV \
> | EVP_CIPH_ALWAYS_CALL_INIT | EVP_CIPH_CTRL_INIT)

Oh, I see. As you say, e_aes.c appears to have some functions exclusively
for AES-XTS.

EVP_CIPH_ALWAYS_CALL_INIT means that even when NULL is passed as a key for
EVP_EncryptInit_ex(), key initialization is executed. This might be one
cause of the slowness of AES-XTS compared to AES-CBC.

>> Q2: Is AES-XTS slower than AES-CBC? Does AES-NI speed up AES-XTS like
>> AES-CBC?
>>
> Yes it is slower because there is an additional encryption operation on
> the "tweak".
> I think AES-NI speeds up the implementation of the underlying AES cipher,
> and therefore would be used no matter what the mode (perhaps one of the
> openssl developers can confirm??)
>
> To test out the speed implications I knocked together a quick piece of
> code to do 1,000,000 AES-256 XTS encryptions of a 4k record followed by
> 1,000,000 AES-256 CBC encryptions. XTS took approx. 108s to run, whilst
> CBC took approx. 41s to run (on my underpowered netbook).
>
> To put this into perspective that means that XTS took approx. 0.1ms to
> encrypt a single 4k record, compared to 0.04ms for CBC. In other words a
> 0.06ms performance penalty. Now I don't know what your application is
> attempting to do, but I suggest that in most scenarios that kind of
> penalty is not going to be noticed, and will probably be negligible
> compared to the file i/o.

I really appreciate your performance measurement. I'll also do some
performance testing on a machine with AES-NI.

I'm thinking of placing files on SSD. So 60us difference per block may be
relatively long compared to tehe I/O. Hmm...

[Matt Caswell (frodo@baggins.org)]

On 29/04/12 05:23, MauMau wrote:
>>> Q2: Is AES-XTS slower than AES-CBC? Does AES-NI speed up AES-XTS
>>> like AES-CBC?
>>>
>> Yes it is slower because there is an additional encryption operation
>> on the "tweak".
>> I think AES-NI speeds up the implementation of the underlying AES
>> cipher, and therefore would be used no matter what the mode (perhaps
>> one of the openssl developers can confirm??)
>>
>> To test out the speed implications I knocked together a quick piece
>> of code to do 1,000,000 AES-256 XTS encryptions of a 4k record
>> followed by 1,000,000 AES-256 CBC encryptions. XTS took approx. 108s
>> to run, whilst CBC took approx. 41s to run (on my underpowered netbook).
>>
>> To put this into perspective that means that XTS took approx. 0.1ms
>> to encrypt a single 4k record, compared to 0.04ms for CBC. In other
>> words a 0.06ms performance penalty. Now I don't know what your
>> application is attempting to do, but I suggest that in most scenarios
>> that kind of penalty is not going to be noticed, and will probably be
>> negligible compared to the file i/o.
>
> I really appreciate your performance measurement. I'll also do some
> performance testing on a machine with AES-NI.
>
> I'm thinking of placing files on SSD. So 60us difference per block may
> be relatively long compared to tehe I/O. Hmm...
>

To transfer 4k data (random access) from an SSD your looking at a
throughput of between 33Mb/s and 11 Mb/s which equates to 0.12ms -
0.36ms (dependent on the disk).
http://www.tomshardware.com/charts/ssd-charts-2011/AS-SSD-4K-Random-Read,2784.html

If your application is multi-threaded with multiple threads trying to
access the disk at the same time the file IO time will go up as threads
are blocked waiting for the disk. So you may well find the overhead of
the XTS operation is not too bad. Don't forget of course that XTS it
typically used for disk encryption....it is designed for exactly this
purpose.

Something else to consider is key length. Obviously you're going to get
a lot better performance out of AES-128 XTS than you will AES-256 XTS.
Its a performance/security trade off. Whether that trade off is
worthwhile is going to depend a lot on the value of the data that you're
encrypting. See:
http://www.keylength.com/en/4/

The table at the bottom gives you an idea of how long an AES-128 bit key
is likely to remain secure for, i.e. sometime after 2030. Now if you're
protecting sensitive government secrets then that's probably not going
to be good enough. However, if you're protecting volatile data that's
only sensitive for a short period of time then 128 bits is more than
enough. So you have to ask yourself what the impact would be of the
disclosure of your data in 2030. If the answer is not much then the
security/performance trade off might be worth it.

Matt