ios locking up during encryption

135 views
Skip to first unread message

jh...@emocha.com

unread,
Dec 16, 2015, 6:18:32 PM12/16/15
to Crypto++ Users
I'm actually using cryptopp on android and ios, and they have dfferent crash behaviors. The android issue is  in another thread, and seems related to stl.

In iOS I get farther in my app, but when it comes time to do a FileSource to FileSync encryption  it locks the device. Has anyone seen this?

Jeffrey Walton

unread,
Dec 16, 2015, 7:17:48 PM12/16/15
to Crypto++ Users


On Wednesday, December 16, 2015 at 6:18:32 PM UTC-5, jh...@emocha.com wrote:
I'm actually using cryptopp on android and ios, and they have dfferent crash behaviors. The android issue is  in another thread, and seems related to stl.

In iOS I get farther in my app, but when it comes time to do a FileSource to FileSync encryption  it locks the device. Has anyone seen this?

Yeah, when you have memory problems :)

I find Android is one of the toughest platforms. I find Linux, OS X and Windows some of the easiest. iOS is somewhere in between due to Xcode's Instruments (https://developer.apple.com/library/tvos/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/index.html). They are somewhat like Visual Studio's Managed Debugging Assistants (https://msdn.microsoft.com/en-us/library/d21c150d%28v=vs.110%29.aspx).

What I do is concentrate my error checking and analysis on the desktop OSes, like OS X and Linux. Once I am satisfied the "core" logic is sound, I port it to another platform, like iOS and Android. The philosophy above means you need a clean separation between "core" and "window/ui" logic or routines. You'll get the hang of it quickly.

If you thoroughly test your code under Linux or OS X, you will likely find a memory error with a tool like Valgrind. That means you have to write the test cases, exercise the core logic routines, and run them under different analysis tools.

Crypto++ eats its own dogfood. We spend a lot of time and effort on error checking, analysis and testing on the desktops OSes. For every 15 minutes we spend writing code, we probably spend 1 or 2 hours testing it on various platforms under different configurations. Also see https://cryptopp.com/wiki/Release_Process

Jeff

Jeffrey Walton

unread,
Dec 16, 2015, 7:37:22 PM12/16/15
to Crypto++ Users

What I do is concentrate my error checking and analysis on the desktop OSes, like OS X and Linux. Once I am satisfied the "core" logic is sound, I port it to another platform, like iOS and Android. The philosophy above means you need a clean separation between "core" and "window/ui" logic or routines. You'll get the hang of it quickly.

BTW, you don't quite have the clean separation. Notice QT objects are cross-pollinating into the core routines. The constData() is part of a QtString.

CFB_Mode<AES>::Encryption e((const byte *) binKey.constData(), binKey.length(), staticIv, 1);

That means you can't test it stand-alone.

Jeff

jh...@emocha.com

unread,
Dec 18, 2015, 10:49:51 AM12/18/15
to Crypto++ Users
I redid the code to use simple static bytes:
	byte iv[16];
	byte bKey[32];
	memcpy(iv, binIv.constData(), sizeof(iv));
	memcpy(bKey, binKey.constData(), sizeof(bKey));

	CFB_Mode<AES>::Encryption e(bKey, 32, iv, 1);

	FileSource src((const char *)inFilename.toUtf8().constData(), true,
		new StreamTransformationFilter(e, //STF
			new Base64Encoder(
				new FileSink((const char *)outFilename.toUtf8().constData())
				, false
			)
		) //STF
	);

Same issue. However, if I comment out the lines with //STF, so that it just removes the STF and just does a base64 encode, the code works. Which makes me think that it's specific to the STF. It is doing something because the phone gets hot. At which point I wonder if the issue is with architecture or packaging. I have created a fat library:

$ xcrun -sdk iphoneos lipo -info libcryptopp-ios.a
Architectures in the fat file: libcryptopp-ios.a are: armv7 x86_64 arm64

It's running on a iPhone 4, which is the arm7 arch, the arm7s is for the iPhone 5 & 5c.

Do you have any idea why the STF would fail in this situation?

Jeffrey Walton

unread,
Dec 19, 2015, 6:50:43 PM12/19/15
to Crypto++ Users

I redid the code to use simple static bytes:
	byte iv[16];
	byte bKey[32];
	memcpy(iv, binIv.constData(), sizeof(iv));
	memcpy(bKey, binKey.constData(), sizeof(bKey));

	CFB_Mode<AES>::Encryption e(bKey, 32, iv, 1);

	FileSource src((const char *)inFilename.toUtf8().constData(), true,
		new StreamTransformationFilter(e, //STF
			new Base64Encoder(
				new FileSink((const char *)outFilename.toUtf8().constData())
				, false
			)
		) //STF
	);

Same issue. However, if I comment out the lines with //STF, so that it just removes the STF and just does a base64 encode, the code works. Which makes me think that it's specific to the STF. It is doing something because the phone gets hot. At which point I wonder if the issue is with architecture or packaging. I have created a fat library:

$ xcrun -sdk iphoneos lipo -info libcryptopp-ios.a
Architectures in the fat file: libcryptopp-ios.a are: armv7 x86_64 arm64

Hmmm... I cannot duplicate it. I just tried again on iOS 6.0 and 8.1.

For iOS, I test using two jail broken devices: (1) iPad2 running 8.1 and (2) iPad3 running iOS 6.0. I cross compile it according to https://www.cryptopp.com/wiki/IOS_%28Command_Line%29 . I SCP cryptest.exe, TestVectors/ and TestData/ to the device. I then SSH into the device, and execute the tests.

Jeff

jh...@emocha.com

unread,
Dec 21, 2015, 5:37:53 PM12/21/15
to Crypto++ Users


Hmmm... I cannot duplicate it. I just tried again on iOS 6.0 and 8.1.

For iOS, I test using two jail broken devices: (1) iPad2 running 8.1 and (2) iPad3 running iOS 6.0. I cross compile it according to https://www.cryptopp.com/wiki/IOS_%28Command_Line%29 . I SCP cryptest.exe, TestVectors/ and TestData/ to the device. I then SSH into the device, and execute the tests.

Jeff


I just pulled from master, it compiled eprecomp.cpp, integer.cpp, vmac.cpp, xtr.cpp, zdeflate.cpp, and it worked... once.  I can't nail it down exactly, but the iPad2 should match my configuration.
I do get a lot of these:
warning: object file (/Users/jason/cryptopp/libcryptopp-ios.a(cryptlib.o)) was built for newer iOS version (9.1) than being linked (8.0)

Would they matter?

jh...@emocha.com

unread,
Dec 21, 2015, 6:01:24 PM12/21/15
to Crypto++ Users


On Monday, December 21, 2015 at 5:37:53 PM UTC-5, jh...@emocha.com wrote:


Hmmm... I cannot duplicate it. I just tried again on iOS 6.0 and 8.1.

For iOS, I test using two jail broken devices: (1) iPad2 running 8.1 and (2) iPad3 running iOS 6.0. I cross compile it according to https://www.cryptopp.com/wiki/IOS_%28Command_Line%29 . I SCP cryptest.exe, TestVectors/ and TestData/ to the device. I then SSH into the device, and execute the tests.

Jeff

 Also this instruction is confusing:

You can also specify different architectures. For example, setenv-ios.sh armv7s will configure for A6 processors and the armv7s instruction set. If you want to build a fat binary (for example, both armv7 and armv7s), then its easier to open GNUMakefile and change it by hand (search for IOS_ARCH after applying the patch).


IOS never appears in the makefile, only makefile-cross.



jh...@emocha.com

unread,
Dec 21, 2015, 6:56:03 PM12/21/15
to Crypto++ Users


On Monday, December 21, 2015 at 6:01:24 PM UTC-5, jh...@emocha.com wrote:


On Monday, December 21, 2015 at 5:37:53 PM UTC-5, jh...@emocha.com wrote:


Hmmm... I cannot duplicate it. I just tried again on iOS 6.0 and 8.1.

For iOS, I test using two jail broken devices: (1) iPad2 running 8.1 and (2) iPad3 running iOS 6.0. I cross compile it according to https://www.cryptopp.com/wiki/IOS_%28Command_Line%29 . I SCP cryptest.exe, TestVectors/ and TestData/ to the device. I then SSH into the device, and execute the tests.

Jeff


Small amounts of data do not seem to crash it. In my app I either get a very small amount of data, or a very large amount if data. A small amount is 7-15kB. A large amount is over a 1MB

Data size (in bytes), result:

1,180,504 locked

2,709,738 locked

8,727,128 locked

7,636 ok

11,429 ok

11,512 ok

11,961 ok

7,672 ok

11,460

975,913 ok (outlier)

11,600 ok



Can you repeat your tests with a lot of data? For comparison on the iPhone 4s, the ~15k files take ~44ms. The big ones, if they work (very rarely) take 17-45 seconds. 




Jeffrey Walton

unread,
Dec 21, 2015, 10:12:22 PM12/21/15
to Crypto++ Users


On Monday, December 21, 2015 at 6:01:24 PM UTC-5, jh...@emocha.com wrote:

 Also this instruction is confusing:

You can also specify different architectures. For example, setenv-ios.sh armv7s will configure for A6 processors and the armv7s instruction set. If you want to build a fat binary (for example, both armv7 and armv7s), then its easier to open GNUMakefile and change it by hand (search for IOS_ARCH after applying the patch).


IOS never appears in the makefile, only makefile-cross.

Ah, you're right. That's text from the pre-5.6.3 days.

I'll get that page updated shortly. Folks who RTFM deserve up-to-date documentation. If you RTFM and you still have questions, then I view that as a gap in the documentation.

Jeff

Jeffrey Walton

unread,
Dec 22, 2015, 2:32:32 AM12/22/15
to Crypto++ Users


On Monday, December 21, 2015 at 6:56:03 PM UTC-5, jh...@emocha.com wrote:

Small amounts of data do not seem to crash it. In my app I either get a very small amount of data, or a very large amount if data. A small amount is 7-15kB. A large amount is over a 1MB

Data size (in bytes), result:

1,180,504 locked

2,709,738 locked

8,727,128 locked

7,636 ok

11,429 ok

11,512 ok

11,961 ok

7,672 ok

11,460

975,913 ok (outlier)

11,600 ok


Yeah, the results look about right. Mobile devices only have 512MB to 1GB of memory, and they *don't* have page files. You should get an out of memory exception or similar if you try to read 2GB or 8GB into memory.

Mobile devices usually require you to decrypt a range (like a 64KB block), and then display the range (in a custom UIViewController). It can be tricky business.

Now, the lock you are seeing could be two things (maybe more). First, it could be CocoaTouch throwing an Objective-C exception. In this case, Crypto++ code would be no wiser because its not a C++ exception. Second, it could be the Crypto++ library spinning on placement operator new. I have never seen how the library performs in this situation.

Related to the second point, there's a brief discussion at AllocatorWithCleanup::allocate (http://www.cryptopp.com/docs/ref/class_allocator_with_cleanup.html#a213d3399b5e89e61f2bf4fc512da289f). You need to look at misc.cpp, line 195 or so (http://www.cryptopp.com/docs/ref/misc_8cpp_source.html#l00195). Pay particular attention to the call to CallNewhandler around line 140 (http://www.cryptopp.com/docs/ref/misc_8cpp_source.html#l00140).

Now, to work around all of this, I think you will need to Pump data in blocks of say, 4096. I *think* the source does this, but I don't believe it performs a Flush. So data accumulates in the Filter or Sink. This should be a non-hard flush. It should not be a MessageEnd() either. See the code below.

Jeff

**********

int main(int argc, char* argv[])
{
  static const unsigned int BIG_SIZE = 2U * 1024U * 1024U * 1024U;
  static const unsigned int BLOCK_SIZE = 4096U;

  try
  {
    SecByteBlock key(32);
    OS_GenerateRandomBlock(false, key.data(), key.size());
 
    cout << "Key: ";
    ArraySource as(key.data(), key.size(), true, new HexEncoder(new FileSink(cout)));
    cout << endl;

    CFB_Mode<AES>::Encryption enc;
    enc.SetKeyWithIV(key.data(), key.size(), key.data());

    MeterFilter* meter;
    StreamTransformationFilter stf(enc, meter = new MeterFilter(new FileSink("null.enc")));
 
    SecByteBlock data(BLOCK_SIZE);
    memset(data, 0x00, data.size());

    unsigned int remaining = BIG_SIZE;
    while(remaining)
    {
      if(remaining % (1024*1024) == 0)
      {
        cout << "Processed: " << meter->GetTotalBytes();
        cout << ", Available: " << stf.MaxRetrievable() << endl;
      }

      const unsigned int req = STDMIN(remaining, BLOCK_SIZE);
      stf.Put(data, req);
      stf.Flush(false);

      remaining -= req;
    }
  }
  catch(const Exception& ex)
  {
    cerr << ex.what() << endl;
  }

  return 0;
}

Jeffrey Walton

unread,
Dec 22, 2015, 4:24:31 AM12/22/15
to Crypto++ Users

This might be a little closer to what you are doing:

    MeterFilter* meter;
    StreamTransformationFilter* stf;

    FileSource fs("/dev/zero", false, stf = new StreamTransformationFilter(enc, meter = new MeterFilter(new FileSink("/dev/null"))));


    unsigned int remaining = BIG_SIZE;
    while(remaining)
    {
      if(remaining % (1024*1024) == 0)
      {
        cout << "Processed: " << meter->GetTotalBytes();
        cout << ", Available: " << stf->MaxRetrievable() << endl;

      }

      const unsigned int req = STDMIN(remaining, BLOCK_SIZE);
      fs.Pump(req);
      remaining -= req;
    }

Jeff

jh...@emocha.com

unread,
Dec 22, 2015, 9:38:40 AM12/22/15
to Crypto++ Users

On Tuesday, December 22, 2015 at 4:24:31 AM UTC-5, Jeffrey Walton wrote:


This might be a little closer to what you are doing:

    MeterFilter* meter;
    StreamTransformationFilter* stf;

    FileSource fs("/dev/zero", false, stf = new StreamTransformationFilter(enc, meter = new MeterFilter(new FileSink("/dev/null"))));

    unsigned int remaining = BIG_SIZE;
    while(remaining)
    {
      if(remaining % (1024*1024) == 0)
      {
        cout << "Processed: " << meter->GetTotalBytes();
        cout << ", Available: " << stf->MaxRetrievable() << endl;
      }

      const unsigned int req = STDMIN(remaining, BLOCK_SIZE);
      fs.Pump(req);
      remaining -= req;
    }

Jeff

Many thanks Jeff! I will give it a shot.

Any idea why this isn't done already internally? I would have figured that this would be how it was implemented. Why doesn't FileSource take a block size?

jh...@emocha.com

unread,
Dec 22, 2015, 4:24:09 PM12/22/15
to Crypto++ Users
The code below is crashing in MeterFilter::PutMaybeModifiable



Stack trace
1 CryptoPP::StreamTransformationFilter::FirstPut(unsigned const char *) filters.cpp 632 0x10008ca83
2 CryptoPP::FilterWithBufferedInput::PutMaybeModifiable(unsigned char *, unsigned long, int, bool, bool) filters.cpp 370 0x10008b968
3 CryptoPP::FilterWithBufferedInput::PutModifiable2(unsigned char *, unsigned long, int, bool) filters.h 344 0x1000131f2
4 CryptoPP::Filter::OutputModifiable(int, unsigned char *, unsigned long, int, bool, std::string const&) filters.cpp 122 0x10008aee4
5 CryptoPP::MeterFilter::PutMaybeModifiable(unsigned char *, unsigned long, int, bool, bool) filters.cpp 213 0x10008aec3
6 CryptoPP::FileStore::TransferTo2(CryptoPP::BufferedTransformation&, unsigned long long&, std::string const&, bool) files.cpp 99 0x100087b99
7 CryptoPP::SourceTemplate<CryptoPP::FileStore>::Pump2(unsigned long long&, bool) filters.h 1214 0x100013dbb
8 CryptoPP::Source::Pump(unsigned long long) filters.h 1146 0x10000f4e4
9 Hash::aesEncryptFilePump() hash.cpp 315 0x10000ecf0

after calling
aesEncryptFileOpen(QString inFile, QString outFile);
aesEncryptFilePump(
is called, to start it then the GUI keeps pumping. I don't know why the MeterFilter should _ever_ crash? readLength is 65536.

 
void Hash::aesEncryptFileOpen(QString inFile, QString outFile) {
	inFilename = inFile;
	outFilename = outFile;

	QString ivHex = hashFile(inFilename, false).mid(0,32);
	QByteArray binIv = QByteArray::fromHex(ivHex.toUtf8());
	char inFilename_[256];
	char outFilename_[256];
	memset(inFilename_, 0, sizeof(inFilename_));
	memset(outFilename_, 0, sizeof(outFilename_));

	strncpy(inFilename_, inFilename.toLocal8Bit().constData(), inFilename.toLocal8Bit().length());
	strncpy(outFilename_, outFilename.toLocal8Bit().constData(), outFilename.toLocal8Bit().length());
	_fileSource = new CryptoPP::FileSource(inFilename_, false);
	_fileSink = new CryptoPP::FileSink(outFilename_);
	_meterFilter= new CryptoPP::MeterFilter();
	CryptoPP::CFB_Mode<CryptoPP::AES>::Encryption e((const byte *)_key.constData(), _key.length(), (const byte *)binIv.constData(), 1);
	_stf = new CryptoPP::StreamTransformationFilter(e);

	if (_base64Output)
		_base64Encoder = new CryptoPP::Base64Encoder();
	else
		_base64Encoder = NULL;

	_fileSource->Attach(_meterFilter);
	_meterFilter->Attach(_stf);
	if (_base64Encoder) {
		_stf->Attach(_base64Encoder);
		_base64Encoder->Attach(_fileSink);
	} else {
		_stf->Attach(_fileSink);
	}
	QFileInfo fi (inFilename);
	totalBytesToEncrypt = fi.size();
	bytesEncrypted = 0;

}

void Hash::aesEncryptFilePump() {
	qlonglong readLength = qMin(_blockSize, totalBytesToEncrypt - bytesEncrypted);
	bytesEncrypted += readLength - _fileSource->Pump(readLength);
	if (_fileSource->SourceExhausted())
	{
		aesEncryptFileClose();
		emit aesEncryptFileComplete(inFilename);
	}
}

Jeffrey Walton

unread,
Dec 22, 2015, 5:10:14 PM12/22/15
to Crypto++ Users


after calling
aesEncryptFileOpen(QString inFile, QString outFile);
aesEncryptFilePump(
is called, to start it then the GUI keeps pumping. I don't know why the MeterFilter should _ever_ crash? readLength is 65536.

Xcode has instruments. You should consider setting up a new Xcode configuration within your project and enable the memory checking tools. Also see https://developer.apple.com/library/mac/recipes/xcode_help-scheme_editor/Articles/SchemeDiagnostics.html
 
void Hash::aesEncryptFileOpen(QString inFile, QString outFile) {
	inFilename = inFile;
	outFilename = outFile;

	QString ivHex = hashFile(inFilename, false).mid(0,32);
	QByteArray binIv = QByteArray::fromHex(ivHex.toUtf8());
	char inFilename_[256];
	char outFilename_[256];
	memset(inFilename_, 0, sizeof(inFilename_));
	memset(outFilename_, 0, sizeof(outFilename_));

	strncpy(inFilename_, inFilename.toLocal8Bit().constData(), inFilename.toLocal8Bit().length());
	strncpy(outFilename_, outFilename.toLocal8Bit().constData(), outFilename.toLocal8Bit().length());

It looks like a potential memory error here. Filenames can get quite long on iOS. They include a ASCII representation of a UUID and the sandbox directories. strncpy may be getting you in trouble because its a breeding ground for buffer overflows and useless return values. Also see http://linux.die.net/man/3/strncpy.

Consider switching to snprintf. snprintf takes a length as a second arg. It also backfills with 0, so the memset is not needed. Finally, it does not NULL terminate if it runs out of space, but it does allow you to test for success/failure. Maybe something like:

    int rc = snprintf(inFilename, COUNTOF(inFilename), ...);
    if( rc < 0 || rc >= COUNTOF(inFilename))
        /* Failure, Truncation or no NULL terminator */

iOS also provides the BSD safer string gear. You might consider using it if you can't use C++. In the case of strncpy, you would use strlcpy. See the Apple Secure Programming Guide for details. (It used be be download-able from Apple's site, but I can't find it at the moment).

 
	_fileSource = new CryptoPP::FileSource(inFilename_, false);
	_fileSink = new CryptoPP::FileSink(outFilename_);
	_meterFilter= new CryptoPP::MeterFilter();
	CryptoPP::CFB_Mode<CryptoPP::AES>::Encryption e((const byte *)_key.constData(), _key.length(), (const byte *)binIv.constData(), 1);
	_stf = new CryptoPP::StreamTransformationFilter(e);

	if (_base64Output)
		_base64Encoder = new CryptoPP::Base64Encoder();
	else
		_base64Encoder = NULL;

	_fileSource->Attach(_meterFilter);
	_meterFilter->Attach(_stf);
	if (_base64Encoder) {
		_stf->Attach(_base64Encoder);
		_base64Encoder->Attach(_fileSink);
	} else {
		_stf->Attach(_fileSink);
	}
	QFileInfo fi (inFilename);
	totalBytesToEncrypt = fi.size();
	bytesEncrypted = 0;

}

It looks like the FileSource is leaked. It needs to be deleted eventually.
 

void Hash::aesEncryptFilePump() {
	qlonglong readLength = qMin(_blockSize, totalBytesToEncrypt - bytesEncrypted);
	bytesEncrypted += readLength - _fileSource->Pump(readLength);
	if (_fileSource->SourceExhausted())
	{
		aesEncryptFileClose();
		emit aesEncryptFileComplete(inFilename);
	}
}

 You should probably call MessageEnd() on the StreamTransformationFilter.

Jeff
 

jh...@emocha.com

unread,
Dec 22, 2015, 5:36:18 PM12/22/15
to Crypto++ Users
Thanks Jeff, but the more I dig into this the more lost I am, and I'm getting frustrated. I've done similar attachment-style cascading IO before.

My issues are:
1. [As discussed before] For some reason FileSource with the pump argument=true attempts to allocate the entire size of the file? This is a terrible idea. Why doesn't it pump() manageable chunks in a loop internally?
2. In order to not lock the GUI, I had to split the calls into a stateful class, where I have Configure, Open and Pump. Then I realized there is no Close(). I was going to cleanup in the Close(). 
3. Also with 2, everything is set to automatically delete the attachments automatically. This is a bad idea and confuses ownership. Then I tried using Ref() but that doesn't work with pointers.
4. I'm targeting many platforms, so XCode specifics should not apply. 


5. In my function
void Hash::aesEncryptFilePump() {
	qlonglong readLength = qMin(_blockSize, totalBytesToEncrypt - bytesEncrypted);
	qlonglong remainder = _fileSource->Pump(readLength);
	bytesEncrypted += readLength - remainder; //_fileSource->Pump(readLength);
	emit bytesEncryptedChanged(bytesEncrypted);
	if (_fileSource->SourceExhausted())
	{
		aesEncryptFileClose();
		emit aesEncryptFileComplete(inFilename);
	}
}

Pump() returns 65535, so nothing got encrypted. Per https://www.cryptopp.com/docs/ref/class_source.html#af8796ef5929f8461e80a5176d5a95a4b, it return the number remaining, not the number read. Correct?

>  You should probably call MessageEnd() on the StreamTransformationFilter.
Where?

How do I Close()? the Source and Sink as to not leak handles and make sure it's all flushed?




jh...@emocha.com

unread,
Dec 22, 2015, 6:15:04 PM12/22/15
to Crypto++ Users
I've refactored my code several times.

You say  " You should probably call MessageEnd() on the StreamTransformationFilter."
I'm not sure how I get to the STF now, as my code now looks like:
	CryptoPP::CFB_Mode<CryptoPP::AES>::Encryption  *e =  new CryptoPP::CFB_Mode<CryptoPP::AES>::Encryption((const byte *)_key.constData(), _key.length(), (const byte *)binIv.constData(), 1);
	_fileSource->Attach(new CryptoPP::MeterFilter);
	_fileSource->Attach(new CryptoPP::StreamTransformationFilter(*e));
	if (_base64Output) {
		_fileSource->Attach(new CryptoPP::Base64Encoder());
	}
	_fileSource->Attach(new CryptoPP::FileSink(outFilename_));

I changed my pump function to be:
void Hash::aesEncryptFilePump() {
	qlonglong readLength = qMin(_blockSize, totalBytesToEncrypt - bytesEncrypted);
	qlonglong remainder = _fileSource->Pump(readLength);
	_fileSource->AttachedTransformation()->MessageEnd();
	bytesEncrypted += readLength - remainder; //_fileSource->Pump(readLength);
	emit bytesEncryptedChanged(bytesEncrypted);
	if (_fileSource->SourceExhausted())
	{
		aesEncryptFileClose();
		emit aesEncryptFileComplete(inFilename);
	}
}

Which seems to be what is needed per https://www.cryptopp.com/wiki/User_Guide:_filters.h However you mentioned calling it on the STF. Shouldn't the MessageEnd() cascade down?


Jeffrey Walton

unread,
Dec 23, 2015, 12:37:27 AM12/23/15
to Crypto++ Users

You should only call MessageEnd() after you have finished encrypting the entire file. MessageEnd() will do things like add padding, if needed. In this respect, I would expect it inside the `_fileSource->SourceExhausted()` conditional.

Jeff

Jeffrey Walton

unread,
Dec 23, 2015, 1:22:27 AM12/23/15
to Crypto++ Users

1. [As discussed before] For some reason FileSource with the pump argument=true attempts to allocate the entire size of the file? This is a terrible idea. Why doesn't it pump() manageable chunks in a loop internally?

I'm not sure. I have not stepped that code in a while. I just manually Pump() when its a concern to me.

You can also avoid it all together, if you like. Just use Put(), MaxRetrievable() and Get(). That's all the pipeline and filters do for you. The upstream source or filter will call Put() on its AttachedTransformation(). There's nothing mystical about it.
 
2. In order to not lock the GUI, I had to split the calls into a stateful class, where I have Configure, Open and Pump. Then I realized there is no Close(). I was going to cleanup in the Close().

Most of the objects you are using follow the Init/Update/Final model. Init() is really just C++ RAII (https://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization). Update() is calls to Put(). Final() is a call to MessageEnd().

When its time to release the resource acquired during construction, delete the object and set its pointer to NULL. The dtor should call MessageEnd() for you.
 
3. Also with 2, everything is set to automatically delete the attachments automatically. This is a bad idea and confuses ownership. Then I tried using Ref() but that doesn't work with pointers.

You should be able to use a Redirector (https://www.cryptopp.com/wiki/Redirector). See the example below.
 
4. I'm targeting many platforms, so XCode specifics should not apply.

You might consider visiting https://wiki.qt.io/Profiling_and_Memory_Checking_Tools , and try to use some of them to locate memory problems. I'm fairly sure they are present.

If you really feel there are no memory problems, then look at configuration issues (like mixing/matching C++ runtime and STL libraries), and platform issues (like running out of memory or C++ ofstream not flushing when its told to do so).

Maybe its a hardware problem... I used to help the OpenSSL Foundation with their iOS FIPS Validations (their office was about 45 minutes from my house). The FIPS validation self tests would run for a day or two. The devices got so hot we had to prop them up and put a fan on them. Otherwise, the tablet would lock up after a few hours.

*****

#include "filters.h"
#include "osrng.h"
#include "modes.h"
#include "files.h"
#include "aes.h"
#include "hex.h"
using namespace CryptoPP;

#include <iostream>
using namespace std;


int main(int argc, char* argv[])
{
  static const unsigned int BIG_SIZE = 2U * 1024U * 1024U;

  static const unsigned int BLOCK_SIZE = 4096U;

  try
    {
      SecByteBlock key(32);
      OS_GenerateRandomBlock(false, key.data(), key.size());
 
      // cout << "Key: ";
      // ArraySource as(key.data(), key.size(), true, new HexEncoder(new FileSink(cout)));
      // cout << endl;


      CFB_Mode<AES>::Encryption enc;
      enc.SetKeyWithIV(key.data(), key.size(), key.data());

      MeterFilter meter;
      StreamTransformationFilter stf(enc);
     
      FileSource source("/dev/zero", false);
      FileSink sink("zero.enc");
 
      source.Attach(new Redirector(stf));
      stf.Attach(new Redirector(meter));
      meter.Attach(new Redirector(sink));


      unsigned int remaining = BIG_SIZE;
      while(remaining && !source.SourceExhausted())
      {
        if(remaining % (1024) == 0)
        {
          cout << "Processed: " << meter.GetTotalBytes() << endl;

        }

      const unsigned int req = STDMIN(remaining, BLOCK_SIZE);
      source.Pump(req);
      source.Flush(false);


      remaining -= req;
    }
  }
  catch(const Exception& ex)
  {
    cerr << ex.what() << endl;
  }

  return 0;
}

*****

Here are the results of running it under a BeagleBone Black. Its resource constrained, too. I can duplicate these results everywhere, from a full fledged desktop to an Android tablet to a JB iOS device.

beaglebone:cryptopp$ rm -f *.enc && ./test.exe
Processed: 0
Processed: 4096
Processed: 8192
Processed: 12288
Processed: 16384
...
Processed: 2088960
Processed: 2093056
beaglebone:cryptopp$ ls -l *.enc
-rw-r--r-- 1 jwalton jwalton 2097152 Dec 23 06:16 zero.enc
beaglebone:cryptopp$

Reply all
Reply to author
Forward
0 new messages