Encrypt and decrypt large files with auto-padding?

[wiired]

I need to encrypt and decrypt large files (up to a few GB) so I need
to do the encryption/decryption one chunk at a time (128KB) to keep
memory utilization low.

Processing in chunks _and_ using padding appears to be more complex
than I thought. Here is the (slightly simplified) technique I'm using.
LSource and LDest are the input and output files read from and written
to directly.

Is there a more simple solution that I'm missing?

Also, is there much benefit in using a random iv? Doesn't that mean
that it would need to be stored with the file and therefore little
more secure than using a fixed value?

const int CBufferSize = Blowfish::BLOCKSIZE * 16384; // 128KB

Encryption:

byte* plainbuffer = NULL;
byte* cypherbuffer = NULL;
try {
plainbuffer = new byte[CBufferSize];
// Allow for padding in encrypted buffer - up to an extra block
size is added.
cypherbuffer = new byte[CBufferSize + Blowfish::BLOCKSIZE];

// Initialise
byte iv[Blowfish::BLOCKSIZE] = { ... }; // 8 bytes
Blowfish::Encryption blowfishEncryption(AKey, AKeyLen);
CBC_Mode_ExternalCipher::Encryption cbcEncryption
(blowfishEncryption, iv);

int LBytesLeft;
int LBytesProcessed;
while ((LBytesLeft = LSource->Size - LSource->Position) > 0) {
LBytesProcessed = LSource->Read(plainbuffer, CBufferSize);

// If at end use a StreamTransformationFilter to handle
padding
if (LBytesLeft <= CBufferSize) {
StreamTransformationFilter cbcEncryptor(cbcEncryption);
cbcEncryptor.Put(plainbuffer, LBytesProcessed);
cbcEncryptor.MessageEnd();
LBytesProcessed = cbcEncryptor.MaxRetrievable();
LBytesProcessed = cbcEncryptor.Get(cypherbuffer,
LBytesProcessed);
} else {
cbcEncryption.ProcessData(cypherbuffer, plainbuffer,
LBytesProcessed);
}

LDest->Write(cypherbuffer, LBytesProcessed);
}
}
__finally {
delete[] cypherbuffer;
delete[] plainbuffer;
}

Decryption:

byte* plainbuffer = NULL;
byte* cypherbuffer = NULL;
try {
plainbuffer = new byte[CBufferSize];
// Allow read of padding in encrypted buffer - up to an extra
block size is added.
cypherbuffer = new byte[CBufferSize + Blowfish::BLOCKSIZE];

// Initialise
byte iv[Blowfish::BLOCKSIZE] = { ... }; // 8 bytes
Blowfish::Decryption blowfishDecryption(AKey, AKeyLen);
CBC_Mode_ExternalCipher::Decryption cbcDecryption
(blowfishDecryption, iv);

int LBytesLeft;
int LBytesProcessed;
while ((LBytesLeft = LSource->Size - LSource->Position) > 0) {
// If at end use a StreamTransformationFilter to handle
padding. Not sure
// if StreamTransformationFilter handles reading just the
padding block at the end
// so group it with the last data blocks.
if (LBytesLeft <= CBufferSize + Blowfish::BLOCKSIZE) {
LBytesProcessed = LSource->Read(cypherbuffer, CBufferSize +
Blowfish::BLOCKSIZE);
StreamTransformationFilter cbcDecryptor(cbcDecryption);
cbcDecryptor.Put(cypherbuffer, LBytesProcessed);
cbcDecryptor.MessageEnd();
LBytesProcessed = cbcDecryptor.MaxRetrievable();
LBytesProcessed = cbcDecryptor.Get(plainbuffer,
LBytesProcessed);
} else {
LBytesProcessed = LSource->Read(cypherbuffer, CBufferSize);
cbcDecryption.ProcessData(plainbuffer, cypherbuffer,
LBytesProcessed);
}

LDest->Write(plainbuffer, LBytesProcessed);
}
}
__finally {
delete[] cypherbuffer;
delete[] plainbuffer;
}

Now this seems to work ok but the conditional code to detect the end
of the input buffer and then use a StreamTransformationFilter seems
like a hack. It would be nice to be able to reuse a single instance of
StreamTransformationFilter and reuse it to process every chunk (Put
followed by a Get) but with padding only at the end of the entire data
(not at the end of each chunk), and to be able to reset its internal
buffers for each chunk so that it doesn't build up the entire data in
its buffers.

Regards,

Jarrod Hollingworth
Complete Time Tracking
http://www.complete-time-tracking.com/

[Jeffrey Walton]

Hi Jarrod,

> I need to encrypt and decrypt large files (up to a few GB) so I need
> to do the encryption/decryption one chunk at a time (128KB) to keep
> memory utilization low.

CTR mode will probably be easier to use for throttling. Otherwise, I
*think* you need to call GetNextIV() from the previous block of 128 KB
so that you can feed it to your next block of 128KB.

> byte* plainbuffer = NULL;
> byte* cypherbuffer = NULL;

> ...

> plainbuffer = new byte[CBufferSize];

A c++ string may simplify this for you. No buffer management, and
Crypto++ works with it natively. Remember to call clear() on the
string between invocations on the 128KB chunk.

Jeff

[Thomas Harning Jr.]

On Thu, May 28, 2009 at 9:20 AM, wiired <jar...@backslash.com.au> wrote:
> I need to encrypt and decrypt large files (up to a few GB) so I need
> to do the encryption/decryption one chunk at a time (128KB) to keep
> memory utilization low.

It looks to me like either writing your own sink, or using an existing
one that writes to a file may be the best bet.

Decryption/Encryption would be a small process of:

Sink( Decryption Filter ( FileSource ) )

The decryption filter would be pulling the minimal chunk to decrypt
from the source and pumping output to the think.
You might also want to put in a buffering source in there (I don't
recall if FileSource helps w/ that) and set
it's buffer level to your 'chunk size' (128KB).

The Decryption/Encryption filters would handle your padding
automatically at the end of the stream and you'd be all set.
--
Thomas Harning Jr.