Safe file encryption

[Ilya Bizyaev]

I encrypt a file with AES, but I need to store original file size for decryping. Should I decrypt the file size or not?

[Jean-Pierre Münch]

Hey Ilya,

if you want to encrypt a file, I think the goal should be that the complete file is random looking. TrueCrypt does a pretty good job at that.
I propose that you place the file size encrypted at the beginning of the file.
I do even further propose that you might want to take a look at TrueCrypt's container header.
You may wanto to exchange some things (CRC -> SHA-3?) but all in all it's pretty good header.

BR

JPM

[Ilya Bizyaev]

I'm afraid I haven't understood how the TrueCrypt stores the file size:
------------------------------------------------------------------
struct TrueCrypt_Header {

/* Salt is visible (appears as random bytes) */ //I store the IV like this
char salt[64];

/* Format (after decryption), 512 - 64(salt) = 448 bytes */
/** Format is encrypted, isn't it? **/
char magic[4]; /* "TRUE" in ASCII */ //What is it used for?!
uint16_t version; /* Header format version */ //For now, I don't have different versions
uint16_t version_tc; /* Minimal required program version */ //See previous one
uint32_t keys_crc32; /* CRC32 of keys area */ //Hash of keys?
uint64_t _reserved1[2]; /* unused (former modification time) */
uint64_t hidden_volume_size; /* size of hidden volume or zero*/ //That's the file size, isn't it?
uint64_t volume_size; /* size of volume or zero */ //Another file size?
uint64_t mk_offset; /* encrypted payload offset in bytes */ //Haven't understood...
uint64_t mk_size; /* encrypted area size; ignored by cryptsetup */
uint32_t flags; /* system/in-place; ignored by cryptsetup */
uint32_t sector_size; /* sector size in bytes, always 512 */ //What's the sense in writing
//constant value?
uint8_t _reserved2[120]; /* unused */
uint32_t header_crc32; /* CRC32 of header without keys; for version > 3 */ //Hash of full header?

/* Key storage area */
char keys[256]; //And the keys are here as well?!
}
-----------------------------------------------------------------

[Ilya Bizyaev]

It seems to me that it can be compressed like so:
---------------------------------------------------------------
struct TrueCrypt_Header {
char iv[64]; /* Visible IV */
/* Format */
uint32_t keys_crc32; /* CRC32 of keys area */ //Is it safe?
uint64_t hidden_volume_size; /* size of hidden volume */
uint64_t mk_offset; /* encrypted payload offset in bytes */ //?
uint32_t header_crc32; /* CRC32 of header without keys */
char keys[256]; /* Key storage area */
}
---------------------------------------------------------------
I am not good at cryptography as I am a school student, but that's how encrypted files look in my program for now:
1) Name of my program and newline (to be recognised).
2) Random IV.
3) Encrypted size of original file (int, but would be replaced with something bigger) + 12 bytes of random data to make an AES-128 block.
4) Then, the very encrypted data.
But I have some concerns about whether to encrypt orifinal file size or not, as it can be expressed as this inequality: (file_size-buffer_size) < original_size <= file_size (without header, of course). So, it is easier for hacker to find a key.

[Jean-Pierre Münch]

Am Freitag, 20. Februar 2015 14:02:37 UTC+1 schrieb Ilya Bizyaev:

I'm afraid I haven't understood how the TrueCrypt stores the file size:
------------------------------------------------------------------
struct TrueCrypt_Header {

  /* Salt is visible (appears as random bytes) */ //I store the IV like this
  char salt[64];

  /* Format (after decryption), 512 - 64(salt) = 448 bytes */
  /** Format is encrypted, isn't it? **/

     /* before this line, everything's open, after this line everything's encrypted*/

  char magic[4];          /* "TRUE" in ASCII */ //What is it used for?!, fast verification that decryption was successful, as otherwise you get some random noise here
  uint16_t version;       /* Header format version */ //For now, I don't have different versions, you may not have them, but you better make sure that you can distinguish old from new header as processing may vary in the future
  uint16_t version_tc;    /* Minimal required program version */ //See previous one, if you deploy your application, you will delivert patches and security enhancements and hence defining a program version is a good idea
  uint32_t keys_crc32;    /* CRC32 of keys area */ //Hash of keys?, place a hash of the keys here, to check if the keys are correct

  uint64_t _reserved1[2]; /* unused (former modification time) */

  uint64_t hidden_volume_size; /* size of hidden volume or zero*/ //That's the file size, isn't it?, that's the size of the HIDDEN volume, a future TrueCrypt has and you don't need except you want plausible denieability
  uint64_t volume_size;   /* size of volume or zero */ //Another file size?, that's the actual file size
  uint64_t mk_offset;     /* encrypted payload offset in bytes */ //Haven't understood..., where does encrypted data start (I guess you don't need this)
  uint64_t mk_size;       /* encrypted area size; ignored by cryptsetup */ // size of theencrypted are (including header I guess)
  uint32_t flags;         /* system/in-place; ignored by cryptsetup */ // leaving space for flags may be a good idea

  uint32_t sector_size;   /* sector size in bytes, always 512 */ //What's the sense in writing

  //constant value?, it may change in the future as standard volume sector sizes get larger, shouldn't affect you as you don't want to implement virtual drives

  uint8_t  _reserved2[120]; /* unused */

  uint32_t header_crc32;  /* CRC32 of header without keys; for version > 3 */ //Hash of full header?, hash of everything before this value

  /* Key storage area */

  char keys[256]; //And the keys are here as well?!, where else to put the keys?

      // maybe you don't get this: TrueCrypt uses your password to derive a key that decrypts this header, in the header you find the keys actually used for the file, meaning you can change your password without needing to re-encrypt the whole file (GIGABYTES!) - only the header

}
-----------------------------------------------------------------

I propose some modifications for your purpose: You may want to add a field after the "magic" TRUE, of size 64 to 128 chars to place your program's name. (-> you try to decrypt the header, check the magic value and check the file is intended for your program.
The IV should be put (un-encrypted) at the beginning, as the only unencrypted data block.
File size is stored in the header above

I'll quickly outline my proposal:

struct TrueCrypt_Header {

  /* Salt is visible (appears as random bytes) */ //I store the IV like this

  char salt[64]; // looks random -> whole file looks random :)

// from here on, everything is encrypted

     /* before this line, everything's open, after this line everything's encrypted*/

  char magic[4];          /* "TRUE" or something else in ASCII, validate this string's correct */

     char program_name[128] /* your program's name*/

  uint16_t version;       /* Header format version, validate you support this */
  uint16_t version_tc;    /* Minimal required program version, validate you have it */
  char keys_hash[32];    /* SHA-3-256 of keys area, validate this */
  uint64_t file_size;   /* size of the data following this header*/
  uint64_t mk_size;       /* overall file size, as should be reported by OS, validate this*/
  uint32_t flags;         /* zeroed out, until you have ideas for flags, ignore this if you don't have flags yet*/
  char header_hash[32];  /* SHA-3-256 of everything before this, validate this*/

     char _reserved[36]; /* fill up to get 256 bytes, allowing ciphers with up to 2048bits blocksize */

      char keys[256] /* leaves you space to use up to 2048 bits of key material -> even Threefish-1024 is supported :) */

}

[Ilya Bizyaev]

Wow, thank you! Now I see that making an encrypted header at the beginning of the file is a great idea! I won't add any flags, and leave less space for keys (as 128 bits is enough for AES-128 key). However, I would add a field which would make it impossible to decrypt file after certain date (as the user wants, of course). Thanks for such a useful answer!

[Jeffrey Walton]

On Saturday, February 21, 2015 at 7:25:54 AM UTC-5, Ilya Bizyaev wrote:

Wow, thank you! Now I see that making an encrypted header at the beginning of the file is a great idea! I won't add any flags, and leave less space for keys (as 128 bits is enough for AES-128 key). However, I would add a field which would make it impossible to decrypt file after certain date (as the user wants, of course). Thanks for such a useful answer!

A field may not work as expected and your adversary will not honor it. I think there may be other things to consider, too.

I think you will need something like a split key with a trusted escrow agent that "forgets" its share of the encrypted key when it expires. That way, its really not recoverable when it expires.

Also see papers like "File system design with assured delete", https://www.isoc.org/isoc/conferences/ndss/07/papers/file_system_assured_delete.pdf.

[Jeffrey Walton]

      //What is it used for?!, fast verification that decryption was successful,

      //    as otherwise you get some random noise here

      char magic[4];     /* "TRUE" in ASCII */

That is probably broken, and you should not use it. First, its too small at 32-bits. Standards like SP800-38A require 64-bits and above (and recommends 96-bits and above). Second, its easy to defeat. I can tamper with a file in its second or third block so that the first block decrypts OK but the file is junked.

We've seen enough oracles for a lifteime. If you are going to provide integrity assurances, then use a authentication tag designed for for the job. See, for example, http://www.cryptopp.com/wiki/Authenticated_Encryption.

Some interesting stuff should be dropping out of Bernstein's Crypto Competition at http://competitions.cr.yp.to/. They are AEAD ciphers with both public and secret values in the AAD slots. It should prove to be very helpful here, where the header would be a public value treated as AAD and protected like the cipher text (sans encryption).

      /* SHA-3-256 of everything before this, validate this*/

      char header_hash[32];

Same problem here (sans the small size of MAGIC). It can be tampered with all day long and the system would be no wiser.

Also, I'm not aware of how to turn SHA-3 into an HMAC. Apparently its not as easy as one would think because previous HMACs were predicated upon an iterative hash function, while SHA-3 is recursive. Has anyone standardized anything for SHA-3 HMACs? Has the CFRG signed off on anything from the IETF (https://irtf.org/cfrg)?

Jeff

[Mobile Mouse]

On Feb 22, 2015, at 5:09 , Jeffrey Walton <nolo...@gmail.com> wrote:

      //What is it used for?!, fast verification that decryption was successful,
      //    as otherwise you get some random noise here
      char magic[4];     /* "TRUE" in ASCII */

That is probably broken, and you should not use it. First, its too small at 32-bits. Standards like SP800-38A require 64-bits and above (and recommends 96-bits and above). Second, its easy to defeat. I can tamper with a file in its second or third block so that the first block decrypts OK but the file is junked.

As an authentication tag - 32 bits would be fine for low-value online traffic protection (e.g. video stream), and totally inadequate for any long-term security.

We've seen enough oracles for a lifteime. If you are going to provide integrity assurances, then use a authentication tag designed for for the job. See, for example, http://www.cryptopp.com/wiki/Authenticated_Encryption.

Yes, IMHO AEAD is the most reasonable answer here.

Some interesting stuff should be dropping out of Bernstein's Crypto Competition at http://competitions.cr.yp.to/. They are AEAD ciphers with both public and secret values in the AAD slots. It should prove to be very helpful here, where the header would be a public value treated as AAD and protected like the cipher text (sans encryption).

I would not bother with/wait for that competition results, and use AES/GCM instead. It has been accepted in several standards, and analyzed well enough. There is only one possibly better option today - AES/OCB, which comes with some legal use restrictions.

      /* SHA-3-256 of everything before this, validate this*/
      char header_hash[32];

Same problem here (sans the small size of MAGIC). It can be tampered with all day long and the system would be no wiser.

:-)

Also, I'm not aware of how to turn SHA-3 into an HMAC.

You don’t need HMAC with SHA-3. As was published some time ago, Envelope construct is provably secure, and performs better than HMAC. The only thing to watch for is that the key must be in its own block both prepended and appended. (See Preneel, sorry don’t have a reference at hand).

Apparently its not as easy as one would think because previous HMACs were predicated upon an iterative hash function, while SHA-3 is recursive. Has anyone standardized anything for SHA-3 HMACs? Has the CFRG signed off on anything from the IETF (https://irtf.org/cfrg)? 

I’ve seen something on this, but don’t think CFRG did that yet. As I said, with SHA-3 it is unnecessary.

--
--
You received this message because you are subscribed to the "Crypto++ Users" Google Group.
To unsubscribe, send an email to cryptopp-user...@googlegroups.com.
More information about Crypto++ and this group is available at http://www.cryptopp.com.
---
You received this message because you are subscribed to the Google Groups "Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jean-Pierre Münch]

Am Sonntag, 22. Februar 2015 16:10:02 UTC+1 schrieb Mouse:

On Feb 22, 2015, at 5:09 , Jeffrey Walton <nolo...@gmail.com> wrote:

      //What is it used for?!, fast verification that decryption was successful,
      //    as otherwise you get some random noise here
      char magic[4];     /* "TRUE" in ASCII */

That is probably broken, and you should not use it. First, its too small at 32-bits. Standards like SP800-38A require 64-bits and above (and recommends 96-bits and above). Second, its easy to defeat. I can tamper with a file in its second or third block so that the first block decrypts OK but the file is junked.

As an authentication tag - 32 bits would be fine for low-value online traffic protection (e.g. video stream), and totally inadequate for any long-term security.

 

The magic value is by no means intended as authentication tag. For this purpose we do have the hashes / MACs / authenticated cipher.
This is supposed to be a fast way to check if a (valid) user entered the right password.
For validation you'd use the hashes, which in-fact use CRC-32 in TrueCrypt 7.1a (considered secure?), so I proposed SHA-3 as a more adequate solution.
 

We've seen enough oracles for a lifteime. If you are going to provide integrity assurances, then use a authentication tag designed for for the job. See, for example, http://www.cryptopp.com/wiki/Authenticated_Encryption.

Yes, IMHO AEAD is the most reasonable answer here.

That's nice to have but not absolutely neccessary for drive encryption. F.ex. TrueCrypt uses XTS-mode without any authentication (except for the CRCs in the header).
As no one did the work on XTS for Crypto++(yet? - not me!), I guess AEAD is the best way to go.

Some interesting stuff should be dropping out of Bernstein's Crypto Competition at http://competitions.cr.yp.to/. They are AEAD ciphers with both public and secret values in the AAD slots. It should prove to be very helpful here, where the header would be a public value treated as AAD and protected like the cipher text (sans encryption).

I would not bother with/wait for that competition results, and use AES/GCM instead. It has been accepted in several standards, and analyzed well enough. There is only one possibly better option today - AES/OCB, which comes with some legal use restrictions.

 

For now AES/GCM is fine, I guess, but that's why there's a header version :) 2.0 gets the winner?
 

      /* SHA-3-256 of everything before this, validate this*/
      char header_hash[32];

Same problem here (sans the small size of MAGIC). It can be tampered with all day long and the system would be no wiser.

:-)

Of course if one feels more comfortable with it, one can of course use some sort of MAC here, that uses a key that's derived along with the encryption key from the password.

Also, I'm not aware of how to turn SHA-3 into an HMAC.

You don’t need HMAC with SHA-3. As was published some time ago, Envelope construct is provably secure, and performs better than HMAC. The only thing to watch for is that the key must be in its own block both prepended and appended. (See Preneel, sorry don’t have a reference at hand).

Apparently its not as easy as one would think because previous HMACs were predicated upon an iterative hash function, while SHA-3 is recursive. Has anyone standardized anything for SHA-3 HMACs? Has the CFRG signed off on anything from the IETF (https://irtf.org/cfrg)? 

I’ve seen something on this, but don’t think CFRG did that yet. As I said, with SHA-3 it is unnecessary.

BR

JPM
 

[Ilya Bizyaev]

Wow, so much replies!... and so much terms (o_O)

You know, I have even decided to systematize all this material, and got the following: <Google Docs>

So, the best suggested idea is usage of authentificated encryption (AES/GCM).

The questions are:

1) Do I still need the header? And how should it look?

What I now have is:

----------------------------------------------------

struct Entangle_Header { 

char salt[64];

  /* ----- Format ----- */

uint16_t prog_version;       /* Header format version */

uint32_t keys_hash;    /* Should I leave it or not? */

uint64_t file_size;   /* size of original file */

  byte keys[32]; /* AES-256 key storage area */

}

----------------------------------------------------

2) The example in Cryptopp Wiki is as follows:

string plaintext, ciphertext;
...

GCM< AES >::Encryption enc;
enc.SetKeyWithIV( key, sizeof(key), iv, sizeof(iv) );

AuthenticatedEncryptionFilter aef( enc,
    new StringSink( ciphertext )
); // AuthenticatedEncryptionFilter

aef.Put( plaintext.data(), plaintext.size() );
aef.MessageEnd();

Is it possible to use buffers instead of strings (e.g. byte buffer[16384]; aef.Put((byte *) buffer, 16384);)?

3) If decryption fails, how to detect whether the password is incorrect or the file is corrupted?

[Jean-Pierre Münch]

Hey Ilya,

1) keys_hash can be thrown out, if you use authenticated encryption as the algorithm will tell you wether the keys are valid. You can use a hash as second line of defense, but you don't need to.
   I'm not an expert on AEAD (esp. GCM), but 32 bytes may not be enough to hold a key for authentication and encryption.
   You may also need to add a field for an IV (16 bytes) for GCM mode.
2)If I'm reading this rightly (I'm not sure about that either) I do think that string doesn't mean std::string but rather SecByteBlock, as you put some data with some length in there, finalize your message and get some authenticated data out.
   I'd strongly recommend not to use byte Data[100+] buffers for data that will be encrypted, unless you burn the data from the stack. Rather use SecByteBlock and FixedSizeSecBlock<byte,X> for such things as they may also offer you things like non-swapping memory and automatic secure clean up of data.
3)Decrypt the file as you'd do normally. I'm not sure how this works with Filters and stuff, but I guess you'll either have to check return value and know it's been tampered / invalid / wrong key if false is returned. It may also be the case that the operation will just throw and kick you out of your program if you don't expect it. (I'd rather assume second possibilty)

I think Jeff and/or Mouse will clear things up as soon as they answer (may be some days in worst-case).

BR

JPM

[Ilya Bizyaev]

OK, the header now looks like this:

----------------------------------------------------
struct Entangle_Header {
char salt[64];

............. /*More if needed*/

/* ----- Format ----- */
uint16_t prog_version; /* Header format version */

uint64_t file_size; /* size of original file */
byte keys[32]; /* AES-256 key storage area */

............. /*More data for GCM if needed*/
}
----------------------------------------------------
And what about encrypting the header with AES-128/GCM and the very file with AES-256/GCM? That seems to be quite secure for the file and simple for the user (16 char key).

[Jean-Pierre Münch]

I'd prefer AES-256/GCM always, because either you have a hash function with 512 bits output and hence some tolerance for 256-bit keys
or for password derivation you have always something like scrypt/ argon/ PHC winner/ PBKDF2 and hence can chose arbitrary derived key lengths.
I'd see no reason to choose AES-128 over AES-256 (except for maybe speed but that's negligible with AES)

BR

JPM

[Ilya Bizyaev]

The reason was not in speed (the header is lightweight), but in comfort for user. But alright, I would use ASE-256 instead.

[Ilya Bizyaev]

Wow, the AES/GCM implemetation is super-complicated!

1) There are two such strings declared at the beginning:

-------------------------------------------------

    string adata( 16, (char)0x00 );

    string pdata( 16, (char)0x00 );

-------------------------------------------------

What each of them is ment for?

2)

------------------------------------------------------------------------------------

        // AuthenticatedEncryptionFilter::ChannelPut

        //  defines two channels: "" (empty) and "AAD"

        //   channel "" is encrypted and authenticated

        //   channel "AAD" is authenticated

        ef.ChannelPut( "AAD", (const byte*)adata.data(), adata.size() );

        ef.ChannelMessageEnd("AAD");

        // Authenticated data *must* be pushed before

        //  Confidential/Authenticated data. Otherwise

        //  we must catch the BadState exception

        ef.ChannelPut( "", (const byte*)pdata.data(), pdata.size() );

        ef.ChannelMessageEnd("");

------------------------------------------------------------------------------------

The question is: which channel is ment for the data your need to encrypt, and which is for authentification data? They are both called "Authenticated data"... And again, adata and pdata are here without an explanation...

3) Is it possible to do without strings, using SecByteBlocks and arrays?

[Ilya Bizyaev]

Oh, "adata" is for identification, and pdata is encrypted, isn't it?

[Ilya Bizyaev]

StringSink doesn't accept anything except strings! So, in this line:

AuthenticatedEncryptionFilter ef( e, new StringSink( cipher ), false, TAG_SIZE);

... I should somehow replace it, so that my program could encrypt the Entangle_Header structure.

[Jean-Pierre Münch]

Hey Ilya,

concerning your questions:
1) adata seems to be data that is only authenticated but not encrypted. This is mainly data that you must authenticate because of your protocol. In your case you might want to authenticate the salt with the header GCM execution.
pdata is the plain data (=plaintext), that means this is the data that will be encrypted and authenticated. The contents of your target file should be in this category as should be the contents of your header.
2) channel "AAD" is authentication associated data (I hope), that's the data you stored in adata. some goes for empty channel name with pdata.
3) If I'm interpreting the code snippet you showed here using SecByteBlock shouldn't be a problem. F.ex. you can store your adata in a secbyteblock. Normally there's an auto-conersion from SecByteBlock to byte* and const byte* so can simply put "ef.ChannelPut("AAD",data,data.size());" and it should work (again I do hope this).

I do think (but not know for sure) that StringSink does accept SecByteBlocks as they are also "strings" (large blocks of chars). Please test it and see if you get a compiler / testing error.

BR

JPM

[Mobile Mouse]

GCM is one of the AEAD encryption modes (see http://en.m.wikipedia.org/wiki/Authenticated_encryption). AEAD stands for Authenticated Encryption with Associated Data. 

The "main" plaintext is encrypted *and* authenticated ("empty" channel).

The "associated" plaintext (adata) is authenticated *only* ("AAD" channel). It's typical use is integrity-protecting message headers that must be available in the clear to the intermediate nodes to properly route this message. 

Sent from my iPad

--

[Mobile Mouse]

Forgot to add: you do NOT have to have "associated data" at all. Just ignore the "AAD" channel in that case. 

Sent from my iPad

On Feb 25, 2015, at 11:25, Jean-Pierre Münch <jean.pier...@gmail.com> wrote:

--

[Ilya Bizyaev]

Hello! Thanks for replies!

1) Jean-Pierre Münch wrote:

"This is mainly data that you must authenticate because of your protocol. In your case you might want to authenticate the salt with the header GCM execution."

So, you mean that the salt can be stored not in the header, but in the AAD channel before it, don't you?

2) " so can simply put "ef.ChannelPut("AAD",data,data.size());"

The ChannelPut function seems to cause no problems; the problem is in AuthenticatedEncryptionFilter constructor's second argument, which is of BufferedTransformation pointer type; I'll try to provide it with SecByteBlock link, but I don't think this would work.

3) Mouse wrote:

"It's typical use is integrity-protecting message headers that must be available in the clear to the intermediate nodes to properly route this message."

I'm afraid my level of English is not enough to understand this sentence; could you please explain it simplier?

4) And if decryption goes wrong, will AEAD return any value or throw an exception?

[Mobile Mouse]

You don't need to even authenticate salt, because if it gets corrupted or maliciously modified (same thing :), decryption will inevitably fail. 

"Associated Data" is something if you want available in the clear not encrypted. One example would be fields of the IP packet header. So if you protect your IP traffic via IPsec or such, it would make sense to use AEAD, encrypting the packet payload, but putting the packet header (well, some of its fields - because there are fields that must change with each Internet hop) in "adata". Reason - that IP packet will probably go through many routers on its way from its source to destination, and unless it's header data is in the clear, those routers won't be able to pass the packet to its destination.

In other words, consider an example: you're writing a letter to somebody, and want it delivered by postal service. The contents of your letter must be opaque to anybody but you and your recipient, and you also want to protect it from modification by somebody else who might get hold of that letter. So what you write in your letter is the candidate for "pdata".

What you write in the envelope - cannot be encrypted, or post office won't be able to deliver your letter. On the other hand, you don't want the delivery address and/or the return address to be modified - and if it was modified you want it to be detected. So what you write on the envelope is a good candidate for "adata".

Yes, it s a requirement that if AEAD fails to either decrypt or authenticate the pdata, or fails to authenticate adata (if present - but it detects when adata was resent during encryption but was not provided during decryption) - the whole decryption is considered as failed, and error must be returned to the caller. Some implementations may also (in addition to the error code) provide the caller with whatever plaintext they did recover - there's an argument whether such behavior is legal or not. 

BouncyCastle (Java) and Oracle Java throw an exception. I don't remember whether Crypto++ returns an error or throws an exception - you can experiment and report here (or look at the source :).

Sent from my iPad

--

[Ilya Bizyaev]

"Associated Data" is something if you want available in the clear not encrypted. One example would be fields of the IP packet header. So if you protect your IP traffic via IPsec or such, it would make sense to use AEAD, encrypting the packet payload, but putting the packet header (well, some of its fields - because there are fields that must change with each Internet hop) in "adata". Reason - that IP packet will probably go through many routers on its way from its source to destination, and unless it's header data is in the clear, those routers won't be able to pass the packet to its destination.

 

Thank you! Now I see that in my case AEAD is not needed.

BouncyCastle (Java) and Oracle Java throw an exception. I don't remember whether Crypto++ returns an error or throws an exception - you can experiment and report here (or look at the source :).

OK, I'll try to implement this, and then I'll write back.

[Mobile Mouse]

I'm not sure I explained it well. You probably don't need associated data. You definitely would benefit from authenticated encryption (such as AES/GCM) with empty adata. 

Sent from my iPad

--

[Ilya Bizyaev]

No, AuthenticatedEncryptionFilter accepts nothing except BufferedTransformation. So, that should be one of this classes: Crypto++ Wiki.

[Ilya Bizyaev]

No, your explanation is great! I'm afraid that's me who can't explain thoughts correctly... I meant I wouldn't use associated data, but would certainly use AES/GCM!

Excuse me for what I've written ^_^

[Ilya Bizyaev]

I still need your help on this question:
четверг, 26 февраля 2015 г., 15:11:29 UTC+3 пользователь Ilya Bizyaev написал:

[Ilya Bizyaev]

I have found a SecByteBlockSink class, made by Jeffrey Walton; now I try to use it correctly. That's how my code looks for now:

------------------------------------------------------------------------------------

try

{

                //New AES Encryption object

                GCM<AES>::Encryption HeaderEncryption;

                //Setting user key and random IV

                HeaderEncryption.SetKeyWithIV(key, sizeof(key), iv, sizeof(iv));

                //Creating SecByteBlock for the header

                SecByteBlock blockky(0x00, 0);

                //Filter with a SecByteBlockSink

                AuthenticatedEncryptionFilter ef(HeaderEncryption,

                new SecByteBlockSink, false, TAG_SIZE);

                //Encrypting MakeHeader

                ef.ChannelPut("", (const byte*)&MakeHeader, sizeof(MakeHeader));

                ef.ChannelMessageEnd("");

                //Writing SecByteBlock

                Out.write(reinterpret_cast<const char*>(blockky.BytePtr()), blockky.size());

}

------------------------------------------------------------------------------------

[Ilya Bizyaev]

Managed to implement header encryption (the problem was in another part of code, although there is a mistake here, too - new SecByteBlockSink(blockky)).

In decryption they use this:

string enc = cipher.substr(0, cipher.length()-TAG_SIZE);

string mac = cipher.substr(cipher.length()-TAG_SIZE);

Now inventing how to replace it. 

[Ilya Bizyaev]

All this small problems are quite solvable, and I hope I would finish writing and start debugging my program tomorrow.
But there's another feature that I'm not sure about. Before all this modifications, my program could automatically detect whether an encryption or decryption is needed. That was quite simple, if not childish - it wrote the name of my program to the beginning of the file on encryption, and then checked if it's present or not. Is there an adequate way to implement this?

[Ilya Bizyaev]

Program crashes in this line:

-----------------------------------------------------------------------------------------

| enc.Assign(blockky.BytePtr(), blockky.SizeInBytes()-TAG_SIZE); |

-----------------------------------------------------------------------------------------

Debug output:

-----------------------------------------------------------------------------------------

In __cxa_throw () (/usr/lib/x86_64-linux-gnu/libstdc++.so.6)

129 misc.cpp: Нет такого файла или каталога.

#1  0x000000000047a1e3 in CryptoPP::CallNewHandler () at misc.cpp:129

Program received signal SIGSEGV, Segmentation fault.

In SecureWipeBuffer<unsigned char> (n=11028786701735723753, buf=0xf89e3dd3191d48fe <error: Cannot access memory at address 0xf89e3dd3191d48fe>) at misc.h:492 ()

-----------------------------------------------------------------------------------------

[Ilya Bizyaev]

P.S. "Нет такого файла или каталога" means "No such file or directory".

[Ilya Bizyaev]

Fixed this and found one of the most common problems in my code: that's not a good idea to provide ANY functions, working with "char" or "byte" data, with a link to SecByteBlock, as in this case such functions would overwrite not the byte buffer, but all the SecByteBlock. Fro example, standart fstream::read() function does this, and this code:
In.read((char *)&sblock, sblock.size())
leads to bad results!
Why is there no way to get a byte pointer from SecBlock? It can only return const byte pointer!
P.S. Still debugging, hope to finish that in a couple of days.

[Jean-Pierre Münch]

try using the BytePtr() member function of SecByteBlock.

[Ilya Bizyaev]

Finished writing and debugging the code for writing and reading the header! I'm already happy! \_(^_^)_/

By the way, ALL the problems I solved were issues with using SecByteBlocks!!! I now that Jean-Pierre Münch is writing a fork for Cryptopp... hope he would make SecBlocks easier to use and more stable!

Now I should re-implement the very file encryption and optimise what I've already rewritten. I'll keep your informed about the development (^_~)

[Jean-Pierre Münch]

Sorry, that I have to disappoint you, but I won't tocuh this kind of thing in my fork.
I do think that SecByteBlock is easy to use and hence I believe that it'll break more than help if I would change anything there.
I'll concentrate my time on implementing new (requested/needed) algorithms (Threefish/Skein/BLAKE2/Fortuna/...) and not on the core.

BR

JPM

[Ilya Bizyaev]

In all the samples, the plain text consisted only of one block; but what if I read data with a 16K buffer and process it step-by-step? Do I need to put a tag somehow? ChannelPut() throws an exception on the second block if I use MessageEnd() after the first one, so, I had to remove the call of this function.
I've tried processing data like this, and the file size mens it contains no tag; and when I tried using FileSink, the tag was present.
P.S. I'll write more details tomorrow.

[Jean-Pierre Münch]

Hey Ilya,

you can call ChannelPut() as often as you want and if you encounter your version of EOF, you simply call MessageEnd() and you're done.

BR

JPM

[Ilya Bizyaev]

The main algorithm is not finished yet, but I decided to check whether what I've already done works fine on Windows. As usual, there is a problem. My program hits an assert. This line causes the issue:

e.SetKeyWithIV(key.BytePtr(), key.SizeInBytes(), iv, sizeof(iv));

By the way, I have already encountered this problem, caused by this line:

CFB_Mode<AES>::Decryption cfbDecryption(DecryptedHeader.keys, 32, iv);

The solution I've found on this forum looks like this:

CFB_Mode<AES>::Decryption * cfbDecryption = new CFB_Mode<AES>::Decryption(DecryptedHeader.keys, 32, iv);

In my humble opinion, the problem is on deallocation; however, I haven't found a general solution yet.

[Ilya Bizyaev]

Yes, that's really so: 

In my humble opinion, the problem is on deallocation; ...

Again, this approach:

GCM<AES>::Encryption * e = new GCM<AES>::Encryption;

e->SetKeyWithIV(key.BytePtr(), key.SizeInBytes(), iv, sizeof(iv));

fixed an issue.

But this is definitely not a solution, as AuthenticatedEncryptionFilter requires a link to Encryption, and not a pointer.

[Ilya Bizyaev]

And how does it depend on the OS? On Ubuntu everything is fine, and Windows is so naughty!

[Ilya Bizyaev]

Still need your help.

[Ilya Bizyaev]

The assertion is caused by "assert(m_allocated)" (secblock.h, line 197), which means m_allocated is false!

m_allocated can be set to "true" only by this procedures:

pointer allocate(size_type n)

{

assert(IsAlignedOn(m_array, 8));

if (n <= S && !m_allocated)

{

m_allocated = true;

return GetAlignedArray();

}

else

return m_fallbackAllocator.allocate(n);

}

pointer allocate(size_type n, const void *hint)

{

if (n <= S && !m_allocated)

{

m_allocated = true;

return GetAlignedArray();

}

else

return m_fallbackAllocator.allocate(n, hint);

}

- which means none of them is called. But why?

[Ilya Bizyaev]

The copy constructor calls this one:

pointer allocate(size_type n, const void *hint)

{

if (n <= S && !m_allocated)

{

m_allocated = true;

return GetAlignedArray();

}

else

return m_fallbackAllocator.allocate(n, hint);

}

If m_allocated is false, then the reason is that (n <= S && !m_allocated) is false. There are two possible causes: m_allocated is already true (which seems to be impossible, as then it would remain true and memory would be deallcated successfully), or n <= S (but I don't know what is S...).

[Ilya Bizyaev]

But why is FixedSizeAllocatorWithCleanup created?! SecByteBlock is defined as SecBlock<byte>, where A is left default - A = AllocatorWithCleanup<T>, and AllocatorWithCleanup doesn't call FixedSizeAllocatorWithCleanup, but inherited from AllocatorBase!

And the debugger doesn't show all this classes created and destroyed in the call stack!

I've got no idea...

[Jeffrey Walton]

What platform are you working on?

 I've seen similar, but its been a few years. I think I saw it on early iOS SDKs.

As a quick hack, open config.h and add the following:

    #undef CRYPTOPP_ALLOW_UNALIGNED_DATA_ACCESS

Add it around line 345, just after its been set.

Jeff

[Ilya Bizyaev]

My application is cross-platform, so I work on two different platforms - Ubuntu and Windows 7. But this issue occurs only on Windows:

- Windows 7 x64

- MinGW (can't find the version, but it's one of the newest)

- Last Crypto++

- Code::Blocks IDE

[Ilya Bizyaev]

No, it doesn't help (o_O)

[Ilya Bizyaev]

My application is open-source, so, today I decided to make it available on GitHub before it's finished. Here is the link: IlyaBizyaev / Entangle.

[Ilya Bizyaev]

I've got rid of ALL SecBlocks in my program, and everything worked fineon Linux, while Windows caught HashVerification, but that assert didn't occur. Anyway, today I've added a small function which has nothing to do with encryption:

void GoodFinish(fstream & In, fstream & Out, wxString & temp_path, wxString & first)

{

    if(In.is_open()) In.close();

    if(Out.is_open()) Out.close();

    SmartRename(temp_path, first);

    if(FromSink!=NULL) EntangleSink::Clean();

}

And guess what? That assert returned! And I've finally got more information on it! 

That's the GDB output and the call stack.

What do you think about it?

[Ilya Bizyaev]

Some fixes for the last message:

- fine on

- HashVerificationFailed

[Jeffrey Walton]

On Friday, February 20, 2015 at 6:27:42 AM UTC-5, Jean-Pierre Münch wrote:

Hey Ilya,

if you want to encrypt a file, I think the goal should be that the complete file is random looking. TrueCrypt does a pretty good job at that.
...

The Phase II portion of the TrueCrypt audit was released recently. See https://opencryptoaudit.org/.

The header issues we discussed is documented under vulnerability CS-TC-4. 

[Nabil Abdoun]

Hello Ilya,

I am using code::blocks in Windows as my IDE. And, I try to use crypto++ library.

After downloading the library, I unzip the file to a folder. I've been unzipping directly to C:\ (e.g. c:\cryptopp). I think that the next step is to build the library, but this is where I get totally stuck. How is this done? 

Thank you

[Ilya Bizyaev]

Hello, Nabil! :)

Actually, building the library is not complicated at all.
1) Firstly, you need to download the latest MinGW from the official website. I'd better use mingw-get.

Then, install the MinGW somewhere (e.g. C:\MinGW\). While choosing additional packages, don't forget about MSYS.

2) Secondly, copy the mingwvars.bat from the MinGW subdirectory of the Code::Blocks folder. Paste it to your MinGW folder and run it. Then, reboot the PC.

Try compiling something to check whether everything works fine.

3) Now, you are ready to build the library. Run the C:\MinGW\msys\1.0\msys.bat file, and you'll get something like a Linux terminal. Type there the following commands:

cd /c/cryptopp562/  # This depends on the Crypto++ location

./configure

make

4) Wait for all routines to finish, then type:

rm -rf *.o

Now the library is built for you.

Hope everything works properly.

P.S. By the way, what about joining my project?)

[Jeffrey Walton]

On Saturday, April 11, 2015 at 6:48:36 AM UTC-4, Ilya Bizyaev wrote:

Hello, Nabil! :)

Actually, building the library is not complicated at all.
1) Firstly, you need to download the latest MinGW from the official website. I'd better use mingw-get.

Then, install the MinGW somewhere (e.g. C:\MinGW\). While choosing additional packages, don't forget about MSYS.

2) Secondly, copy the mingwvars.bat from the MinGW subdirectory of the Code::Blocks folder. Paste it to your MinGW folder and run it. Then, reboot the PC.

Try compiling something to check whether everything works fine.

3) Now, you are ready to build the library. Run the C:\MinGW\msys\1.0\msys.bat file, and you'll get something like a Linux terminal. Type there the following commands:

cd /c/cryptopp562/  # This depends on the Crypto++ location

./configure

make

4) Wait for all routines to finish, then type:

rm -rf *.o

Crypto++ can (should?) be built with Visual Studio on Windows. If you need Visual Studio for Windows, then you might try Visual Studio Express (https://www.visualstudio.com/en-US/products/visual-studio-express-vs).

Code::Blocks supports the VC++ compiler out of the box (http://wiki.codeblocks.org/index.php?title=Installing_a_supported_compiler). It explicitly states it works with the Express versions of Visual Studio.

There's no need for Cygwin or MinGW. If you like, then that's fine. There's just no need for it in this case.

Jeff

[Ilya Bizyaev]

Yes, Code::Blocks supports MinGW, but it doesn't come with MSYS, which I use for the build. Moreover, the MinGW version there is quite old.
Is it a strict rule to use Visual Studio?

[Jean-Pierre Münch]

As far as I know, using MSVC is the best way to build under Windows as Crypto++ provides the solution files by default.

It isn't a strict rule in that sense, as you can still build it using mingw (and underlying gcc?), but it's guaranteed to build under visual studio.

[Ilya Bizyaev]

Hmm... Maybe that's the reason of problems on Windows?..

Is it possible to build the library in MVS, and then use it with MinGW? 

[Ilya Bizyaev]

Cannot connect the crypopp.lib file to my application. GCC says that references to ~45 objects are not defined.