HMACs of files

[Tom]

I can create HMACs of files using pipelines via filesources but... I can't seem to figure out to verify the HMAC without throwing the file into a string in memory.

like this:

StringSource(plain + mac, true, new HashVerificationFilter(hmac, NULL, flags) ); // StringSource

Is there a way to use a FileSource without loading the file fully into memory?

I think its possible but do I append the hmac if I use a file?

[Jeffrey Walton]

Yeah, that's a problem. We should have some documentation covering it.

I think you need a custom source that takes two sources - the existing
HMAC wrapped in a StringSource and the FileSource. The custom source
then pumps the data to the attached filter.

Another option is a HashVerificationFilter that takes two sources. It
could be tricky since the source is expected to pump its data. I did
not test this option.

Attached is an example. It uses a hash rather than HMAC to simplify the code.

The example has a bug, though. HashVerificationFilter is failing.
Instead of PUT_RESULT (or THROW), I used PUT_HASH to see the hash that
was calculated during verification. When using PUT_HASH, the
calculated digest has 32 0's appending to it. So the calculated digest
in the example program is
AD7FACB2586FC6E966C004D7D1D16B024F5805FF7CB47C7A85DABD8B48892CA7
0000000000000000000000000000000000000000000000000000000000000000.

I'll add the example to the wiki once I get the problem sorted out.

Jeff

[Jeffrey Walton]

On Wednesday, September 8, 2021 at 6:47:45 AM UTC-4 Jeffrey Walton wrote:

On Tue, Sep 7, 2021 at 7:45 AM Tom <thoma...@gmail.com> wrote:
>
> I can create HMACs of files using pipelines via filesources but... I can't seem to figure out to verify the HMAC without throwing the file into a string in memory.
>
> like this:
>
> StringSource(plain + mac, true, new HashVerificationFilter(hmac, NULL, flags) ); // StringSource
>
> Is there a way to use a FileSource without loading the file fully into memory?
>
> I think its possible but do I append the hmac if I use a file?

Yeah, that's a problem. We should have some documentation covering it.

I think you need a custom source that takes two sources - the existing
HMAC wrapped in a StringSource and the FileSource. The custom source
then pumps the data to the attached filter.

Another option is a HashVerificationFilter that takes two sources. It
could be tricky since the source is expected to pump its data. I did
not test this option.

Attached is an example. It uses a hash rather than HMAC to simplify the code.

The example has a bug, though. HashVerificationFilter is failing...

Attached is a corrected example that works as expected. Unfortunately, I was not able to get the CombinedSource class to work as expected. Instead, I had to manually fiddle with both Sources. It is not as elegant, but it should get you through your task.

Jeff

[Jeffrey Walton]

On Wed, Sep 8, 2021 at 6:46 AM Jeffrey Walton <nolo...@gmail.com> wrote:
>
> On Tue, Sep 7, 2021 at 7:45 AM Tom <thoma...@gmail.com> wrote:
> >
> > I can create HMACs of files using pipelines via filesources but... I can't seem to figure out to verify the HMAC without throwing the file into a string in memory.
> >
> > like this:
> >
> > StringSource(plain + mac, true, new HashVerificationFilter(hmac, NULL, flags) ); // StringSource
> >
> > Is there a way to use a FileSource without loading the file fully into memory?
> >
> > I think its possible but do I append the hmac if I use a file?
>
> Yeah, that's a problem. We should have some documentation covering it.

Here is the documentation:
https://www.cryptopp.com/wiki/HashVerificationFilter#String_and_File_Sources

Jeff

[Tom]

calling `FileSource fs("zero.dat", true);` seems to allocate the memory.

[Jeffrey Walton]

On Sat, May 20, 2023 at 3:00 PM Tom <thoma...@gmail.com> wrote:
>
> calling `FileSource fs("zero.dat", true);` seems to allocate the memory.

Try the overload which takes a std::istream reference:
https://www.cryptopp.com/wiki/FileSource

Jeff

[Jeffrey Walton]

On Sat, May 20, 2023 at 4:45 PM Jeffrey Walton <nolo...@gmail.com> wrote:
>
> On Sat, May 20, 2023 at 3:00 PM Tom <thoma...@gmail.com> wrote:
> >
> > calling `FileSource fs("zero.dat", true);` seems to allocate the memory.
>
> Try the overload which takes a std::istream reference:
> https://www.cryptopp.com/wiki/FileSource

I took another look... The issue is you passed 'true' for 'pumpAll'.
That reads the entire file, and puts it in an output buffer for the
next filter.

Use 'false' instead, and then call 'PumpAll()' manually. Something
like the example. Something like this should suffice:

FileSource fs("zero.dat", false, new Filter(..., new Sink(...)));
fs.PumpAll();

It should pump in 4k blocks.

Jeff

[Tom]

Okay.

However, when I looked at the information provided in this link: https://www.cryptopp.com/wiki/HashVerificationFilter#String_and_File_Sources, it appears that the method suggested to avoid memory usage doesn't actually achieve that goal, as the line FileSource fs("zero.dat", true); in the verification step is performing as intended.

Unfortunately, I am unable to use FileSource fs("zero.dat", false, new HashVerificationFilter(sha256, new StringSink(digest))); or FileSource fs("zero.dat", true... either, because it's unclear how to associate the digest with the input file.

[Tom]

When dealing with the `SignatureVerificationFilter`, similar issues arise due to the requirement of having the "tag" at either the beginning or end of the initial part of the pipeline. Adding the "tag" to the filename within a pipeline that starts with `FileSource` does not resolve this problem. Consequently, `FileSource` may attempt to load an incorrect file or, when using an `ifstream`, cause a compile-time type mismatch.

While attaching the "tag" to a file that already has it may seem logically sound, this approach can disrupt the functionality of certain files. For example, executable files cannot have the "tag" prepended, and appending the "tag" can lead to extraction tools generating errors about extra data, especially in the case of archives. Furthermore, this method renders simple hash checks invalid, especially if the hash was generated before the "tag" was added, as the file's content has been altered.

[Tom]

It can work with: https://gist.github.com/thomasb9511/f570ecad82055b34a27ec9e706f596c9

But it sidesteps the pipeline for verification.