Chunking a base64 encoded file

[mahon...@gmail.com]

Would Base64Decoder work gracefully with chunking? Looking at below code which I slapped together from the Wiki example on chunking and with Jeff's help re EndOfFile (which I just did not see above the screen and only tried to search for in the Wiki and source code), I have this question: once the MeterFilter starts slicing and dicing the input, and feeding to the Base64Decoder, would the later be smart enough to process successfully? It seems that from every 3 input bytes Base64Decoder has to recover 2 data bytes. Does it mean that chunk size for this scenario has to be a multiple of 3?

                MeterFilter meter;

                StreamTransformationFilter decryptorFilter((StreamTransformation&)decryptor);

                Base64Decoder base64Dec;

                FileSource source(sourceStream, false);

                FileSink sink(destStream);

                source.Attach(new Redirector(decryptorFilter));

                if (Base64Encode)

                {

                    decryptorFilter.Attach(new Redirector(base64Dec));

                    base64Dec.Attach(new Redirector(meter)); // meter filter feeding base64 decryptor? will this work?

                }

                else

                {

                    decryptorFilter.Attach(new Redirector(meter));

                }

                meter.Attach(new Redirector(sink));

                word64 processed = 0;

                while (!FileUtil::EndOfFile(source) && !source.SourceExhausted())

                {

                    source.Pump(ChunkSize); // so hear we would have to adjust ChunkSize to the nearest greater multiple of 3?

                    decryptorFilter.Flush(false);

                    processed += ChunkSize;

                    if (ProgressCallback != nullptr)

                    {

                        if (processed % (ProgressEvery) == 0)

                        {

                            ProgressCallback(meter.GetTotalBytes());

                        }

                    }

                }

[Jeffrey Walton]

What happens in reality is, you do this at the application layer:

source.Pump(ChunkSize);

You pump data into the attached
transformation/{encoder|encryptor|decoder|decryption|...} filter. The
filter then buffers your requests and flushes them at 4096 bytes or
when it gets a MessageEnd() signal.

I can't find the reference at the moment at the moment. When I do I'll
post it to the wiki page.

Jeff

Jeff

[mahon...@gmail.com]

Are you saying that Base64 decoder knows precisely how many bytes of input to take from the chunk shoved into it by the pipelline, and I would not need to worry about how many bytes to pump at a time?

Thanks,

Mahon

[Jeffrey Walton]

Correct. If all works as expected you get your decoded message when
the destructors run. There may be some data available sooner if you
run a while loop chunking data.

Sometimes you need to call MessageEnd() to tell the machinery to
process all data at this very moment so you can use it before the
destructors run. Otherwise the filter will wait for more data to
stream. See, for example,
https://www.cryptopp.com/wiki/Base64Decoder#Missing_Data .

Jeff