Alternate Alphabet for Base32 Encoding & Decoding

[w8r...@gmail.com]

This question has been asked and answered several times. I've looked at all the old answers and looked at the source code, but I am not able to get base32 to work with an alternative alphabet. I would like to use the RFC4648 alphabet rather than the DUDE alphabet.

* 2011 Question - http://comments.gmane.org/gmane.comp.encryption.cryptopp/5778

    The alt_base32 files that Jeffrey Walton placed on the Cryptopp wiki have been removed from this URLhttp://www.cryptopp.com/wiki/File:RFC4648

* 2010 Question - http://comments.gmane.org/gmane.comp.encryption.cryptopp/5355

    I get the same stack overflow as this poster when I try his source code.

* 2007 Question - http://cryptopp-users.narkive.com/WJrYwoW3/base-32-alphabet

    I can compile and run Wei's example source code without issue. However, it does not decode correctly.

Here's my current test code (based on Wei's 2007 answer). I've tried Initialize and IsolatedInitialize:

75 static const byte RFC4648_ALPHABET[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ234567";
76
77 CryptoPP::Base32Encoder b32encoder;
78 b32encoder.IsolatedInitialize(CryptoPP::MakeParameters(CryptoPP::Name::EncodingLookupArray(),
79 (const byte *)RFC4648_ALPHABET,
80 false));
81
82 static int decoding_array[256];
83 CryptoPP::Base32Decoder::InitializeDecodingLookupArray(decoding_array,
84 RFC4648_ALPHABET,
85 32,
86 true); // false = not case sensitive
87
88 CryptoPP::Base32Decoder b32decoder;
89 b32encoder.IsolatedInitialize(CryptoPP::MakeParameters(CryptoPP::Name::DecodingLookupArray(),
90 (const int*)decoding_array,
91 false));

If anyone on the list could show me how to properly implement an alternate alphabet with base32 in Cryptopp, I'd appreciate it. For now, I'm including Jeffrey Walton's old alt_base32 files with my source code. However, that's becoming too burdensome and I'd like to fix it the right way.

Thanks,

Brad

[Jeffrey Walton]

On Friday, January 8, 2016 at 2:56:36 PM UTC-5, w8r...@gmail.com wrote:

This question has been asked and answered several times. I've looked at all the old answers and looked at the source code, but I am not able to get base32 to work with an alternative alphabet. I would like to use the RFC4648 alphabet rather than the DUDE alphabet.

I'm thinking one of the easiest things to do would be good old copy/paste. That is, copy Base32Encoder, paste it as Base32AltEncoder in the same header file, replace the alphabet, do the same in the source file, and you are ready to go. We did the same at https://github.com/weidai11/cryptopp/commit/ea75b3ae5fabc1c5a01a71ab395ac99a4946c818.

But don't let me dissuade you... If you can provide details on what's going wrong with IsolatedInitialize(), then maybe we can help you work through it.

 

    The alt_base32 files that Jeffrey Walton placed on the Cryptopp wiki have been removed from this URLhttp://www.cryptopp.com/wiki/File:RFC4648

Hmmm.... I wonder if it was deleted, or if the migration over the summer lost it... I'm betting it was migration related.

Jeff

[w8r...@gmail.com]

I'm able to narrow down the failure to decoding base32. Encoding works fine. Decoding fails with a stack overflow just like the report in 2010 here:

    http://comments.gmane.org/gmane.comp.encryption.cryptopp/5355

Here's a very simple test program that demonstrates the base32 decoding error:

    https://github.com/w8rbt/oathgen/blob/master/base32_test.cpp

Here's how I compile the test program:

    g++ -g -g3 -std=c++11 -Wall -Wextra -Werror \

    -Weffc++ -pedantic-errors base32_test.cpp \

    -o b32_test \

    /usr/lib/libcryptopp.so

Here's my g++ version:

    g++ --version

    g++ (Debian 4.7.2-5) 4.7.2

Here's output from valgrind:

==22360== Stack overflow in thread 1: can't grow stack to 0x7fe801ff8

==22360== 

==22360== Process terminating with default action of signal 11 (SIGSEGV)

==22360== Access not within mapped region at address 0x7FE801FF8

==22360== at 0x50F35EE: CryptoPP::Base32Decoder::IsolatedInitialize(CryptoPP::NameValuePairs const&) (in /usr/lib/libcrypto++.so.9.0.0)

Thanks,

Brad

[Jeffrey Walton]

On Monday, January 11, 2016 at 2:10:58 PM UTC-5, w8r...@gmail.com wrote:

I'm able to narrow down the failure to decoding base32. Encoding works fine. Decoding fails with a stack overflow just like the report in 2010 here:

    http://comments.gmane.org/gmane.comp.encryption.cryptopp/5355

Here's a very simple test program that demonstrates the base32 decoding error:

    https://github.com/w8rbt/oathgen/blob/master/base32_test.cpp

Here's how I compile the test program:

    g++ -g -g3 -std=c++11 -Wall -Wextra -Werror \

    -Weffc++ -pedantic-errors base32_test.cpp \

    -o b32_test \

    /usr/lib/libcryptopp.so

It looks like it has something to do with the Decoder's DefaultDecodingLookupArray():

(lldb) n
Process 76482 stopped
* thread #1: tid = 0x5308bf, 0x00007fff8ece5297 libsystem_malloc.dylib`szone_malloc_should_clear + 20, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x7fff5f3ffff4)
    frame #0: 0x00007fff8ece5297 libsystem_malloc.dylib`szone_malloc_should_clear + 20
libsystem_malloc.dylib`szone_malloc_should_clear + 20:
-> 0x7fff8ece5297:  movl   %edx, -0xac(%rbp)
   0x7fff8ece529d:  movq   %rsi, %r14
   0x7fff8ece52a0:  movq   %rdi, %r12
   0x7fff8ece52a3:  movq   %r12, -0x68(%rbp)
(lldb) p Base32Decoder::GetDefaultDecodingLookupArray()
error: Trying to put the stack in unreadable memory at: 0x7fff5f3fff50.

The lookup array actually a parallel data structure, and it needs to be 256 elements in size. It provides indexes into your 32-character alphabet or -1 for 'invalid'. Most entries are marked as invalid. But this looks like a pointer problem....

I've got to finish up  some work in the yard so I can let my dogs out. Give me an hour or two.

Jeff
 

[Jeffrey Walton]

The lookup array actually a parallel data structure, and it needs to be 256 elements in size. It provides indexes into your 32-character alphabet or -1 for 'invalid'. Most entries are marked as invalid. But this looks like a pointer problem....

I've got to finish up  some work in the yard so I can let my dogs out. Give me an hour or two.

It looks like the lookup table is borked after the call to InitializeDecodingLookupArray:

255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,
255,255,26,27,28,29,30,31,
255,255,255,255,255,255,255,255,

There could be other problems afoot.

Jeff

[Jeffrey Walton]

Here's output from valgrind:

==22360== Stack overflow in thread 1: can't grow stack to 0x7fe801ff8

==22360== 

==22360== Process terminating with default action of signal 11 (SIGSEGV)

==22360== Access not within mapped region at address 0x7FE801FF8

==22360== at 0x50F35EE: CryptoPP::Base32Decoder::IsolatedInitialize(CryptoPP::NameValuePairs const&) (in /usr/lib/libcrypto++.so.9.0.0)

We are tracking this at "Crash when trying to set alternate alphabet for Base32 decoder", https://github.com/weidai11/cryptopp/issues/108.

It looks like the stack is being blown away due to recursion, and not  a memory problem.

There's nothing you will be able to do with the shared object you are using. You will probably need to add a Base32AltEncoder and
Base32AltDecoder class so things work as expected for you.

Jeff

[Jeffrey Walton]

Here's output from valgrind:

==22360== Stack overflow in thread 1: can't grow stack to 0x7fe801ff8

==22360== 

==22360== Process terminating with default action of signal 11 (SIGSEGV)

==22360== Access not within mapped region at address 0x7FE801FF8

==22360== at 0x50F35EE: CryptoPP::Base32Decoder::IsolatedInitialize(CryptoPP::NameValuePairs const&) (in /usr/lib/libcrypto++.so.9.0.0)

This was cleared at commit 9a5e359bb3795bf3. It will not be available until the next release, which is probably going to be Crypto++ 5.7.

Jeff