aes/gcm maximum plaintext size; handling of large files

[Lana Deere]

Using cryptopp 8.9 on debian 12.11 with gcc 14.2 I have an AES/GCM encryption program which I am running on a large file (75GB).  It is failing with the error "AES/GCM: message length exceeds maximum".  I searched around and found information which suggests the maximum file size should be 2**39-256, which is a bit less than 550GB.  So it is not clear to me why I am getting this error message.  Any ideas on what kind of bug I should be looking for?  As a second question, anyone have any advice on ways to handle large files which it is desirable to encrypt?  This is not the largest file I would like to encrypt and it is plausible I would hit the 2**39-256 limit.

The relevant code looks like this:

    CryptoPP::GCM<CryptoPP::AES>::Encryption encryptor;
    encryptor.SetKeyWithIV(reinterpret_cast<const CryptoPP::byte *>(key.data())
                          , KeyBytes
                          , iv.bytes
                          , IVBytes
                          );

    CryptoPP::AuthenticatedEncryptionFilter filter(encryptor);

    CryptoPP::FileSource source(plainfile.c_str(), false);
    CryptoPP::FileSink sink(cipherfile.c_str());

    CryptoPP::ArraySource(iv.bytes, IVBytes, true, new CryptoPP::Redirector(sink));

    source.Attach(new CryptoPP::Redirector(filter));
    filter.Attach(new CryptoPP::Redirector(sink));

    while (!EndOfFile(source) && !source.SourceExhausted())
    {
        source.Pump(TransferBytes);
        filter.Flush(false);
    }
    filter.MessageEnd();

[Jeffrey Walton]

GCM plaintext maximum length is specified in bits, not bytes. See SP800-39D, Section 5.2.1.1 Input Data, p. 8, <https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf>. That leads to:

    2^39 - 256 = 549755813632

    549755813632 / 8 = 68719476704

The limit is declared in gcm.h, <https://github.com/weidai11/cryptopp/blob/master/gcm.h#L61>. The maximum plaintext limit is enforced in authenc.cpp, <https://github.com/weidai11/cryptopp/blob/master/authenc.cpp#L109>.

Jeff

[Lana Deere]

On Tuesday, August 12, 2025 at 8:10:03 AM UTC-4 Jeffrey Walton wrote:

GCM plaintext maximum length is specified in bits, not bytes. See SP800-39D, Section 5.2.1.1 Input Data, p. 8, <https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf>. That leads to:

    2^39 - 256 = 549755813632

    549755813632 / 8 = 68719476704

Is there a standard practice for handling AES encryption of large files?  E.g., create a new IV and resume encryption?  Use something other than GCM which has a higher limit?

Thanks!

 

[Tony Stead]

Hi. I think it kind of depends on your use case. Personally I would look to block the data up into manageable blocks of a certain size, fixed or variable length. There are advantages to this: specifically that you can random access data on a block basis, decrypt and verify as opposed to having to decypt the whole lot in one go just to read a subsection of data. 

Obviously you will need to either create some header on variable length blocks or add padding to the last block if using fixed length blocks 

Regarding the IV. You will need to differentiate for each block. You could generate a ‘base iv’ for the file and add a block counter in the low order bits of the IV relating to the block number (0-n). This way you only need to store one base IV for the file then calculate the counter for the block being read. You also know that each IV for each block is unique (assuming you have enough bits allocated to your counter field in the IV)

[Jeffrey Walton]

The limit you are encountering in GCM mode is due to an internal 32-bit counter. The limit is not directly due to AES.

You could use a block cipher and mode of operation with a 64-bit counter, or a stream cipher with a 64-bit counter. Bernstein's original ChaCha20 has an internal 64-bit counter. So Bernstein's ChaCha20 paired with Poly1305 could be a good replacement for you. See <https://www.cryptopp.com/wiki/ChaCha20>.

The IETF's version of ChaCha20 used in TLS regresses to a 32-bit counter, so don't use it.

Jeff

[Uri Blumenthal]

Filesystems (if memory servers me) use AES in XTS mode. 

On Aug 13, 2025, at 06:46, Jeffrey Walton <nolo...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cryptopp-users/CAH8yC8%3DaTnOxAvMWbOOFto2RnUtqPecxTW_k-%3DmBNaXLt8Mq1A%40mail.gmail.com.

[Lana Deere]

I will look into ChaCha20.  Thank-you everyone who offered suggestions.

.. Lana (lana....@gmail.com)

--

[Jeffrey Walton]

On Wed, Aug 13, 2025 at 12:24 PM Lana Deere <lana....@gmail.com> wrote:

I will look into ChaCha20.  Thank-you everyone who offered suggestions.

AES/CTR might also work for you. It looks like Crypto++ uses the entire initial value as both a nonce and a counter [1,2]. Once you set the initial value with a nonce, the library increments the entire 16-byte block. So you basically have a 128-bit counter.

[1] <https://github.com/weidai11/cryptopp/blob/master/modes.cpp#L137>

[2] <https://github.com/weidai11/cryptopp/blob/master/modes.cpp#L169>

Jeff