Strange behavior with -static compilation flag

[Dwight Kulkarni]

Hi all,

I have my RSA encrypt function as below.

I compile my program with the following:

g++ -g -c -static -pthread -I../ -I/data/prj/external-libs/include/cryptopp/ ../src/threading/server.cpp

g++ -g ../lbin/*.o -static -pthread -o server -L/data/prj/external-libs/lib/ -l:libcryptopp.a

Here is what is strange.

If I include the -static flag, when I run the encrypt_rsa command below on the remote server it correctly decrypts.

If I remove the -static flag, on the remove server it doesn't get the proper message. However, the message that locally encrypted and decrypted still works.

It doesn't throw any error, encryption completes, but when the server receives it, it doesn't decrypt properly.

During compilation of the .a library from makefile it did not have a -static flag.

std::string encrypt_rsa(std::string message, CryptoPP::RSA::PublicKey key)

{

try{

cout << " In encrypt rsa string " << endl;

message = b64encode(message);

CryptoPP::AutoSeededRandomPool rng;

//CryptoPP::RSAES_OAEP_SHA_Encryptor encryptor(key);

CryptoPP::RSAES_PKCS1v15_Encryptor encryptor(key);

std::string ciphertext;

CryptoPP::StringSource(message, true, new CryptoPP::PK_EncryptorFilter(rng, encryptor, new CryptoPP::StringSink(ciphertext)));

return ciphertext;

}

catch(...)

{

std::cout << "error encrypting RSA";

return "";

}

}

[Dwight Kulkarni]

Further info:

without -static, this is the output from the remote server:

broker got data of size  512
Broker handler thread started... 512
 first two bytes are  128  and  198  and  154
clear text begins with  68 180 -------------->> this should be 00 02
Either no 00 02 beginning block or no 00 separator byte found in proper range
Decryption failed
Exception occurred:  'NoneType' object has no attribute 'decode'

with -static flag, this is the output from the remote server:

broker got data of size  512
Broker handler thread started... 512
 first two bytes are  51  and  252  and  242
clear text begins with  0 2  -------------> this is the correct value
Got base64 bytes  b'eyJtZXNzYWdlIjoiMSIsInRybiI6MzM0NjQ1NjksInNlc3Npb24iOiJzc2Vzc2lvbiIsInBhc3Njb2RlIjoic3Bhc3Njb2RlIiwiZGF0ZXRpbWUiOiIwNC8yNS8yMDIzIDE4OjE1OjA0LjEwOCIsInNlcmlhbCI6InNzZXJpYWwiLCJjYW1lcmFfcG9ydCI6NTAwMCwia2V5IjoiTURObE1XWTBOMlUyTnpFNE1HTXdaak5pWkRZeE1UZ3haVGcyT0dFd05XST0iLCJpdiI6Ik9ETmtZMk0zTVdJek5UUmtaVFUyTUROak1HVTRaakptWlRNME5UQmhZMk09IiwidCI6Mn0='
initializing the lock...

[Dwight Kulkarni]

Hi all:

ldd ./server  compiled without the -static flag shows following .so dependencies.

Since the -static flag will add the same libraries as .a form, it will compile a slightly different version. I am trying to think why the encryption would produce two different results, maybe something like big endian or little endian treatment in a particular function ? Maybe difference in the random generator ? Maybe byte size differences due to typedef ? 

linux-vdso.so.1 (0x00007ffc5e0d7000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8712987000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f87125e9000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f87123d1000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f87121b2000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8711dc1000)
/lib64/ld-linux-x86-64.so.2 (0x00007f87131e2000)

[Dwight Kulkarni]

I got it working by implementing the raw Integer method. See my code below, previous encryption is commented out. I don't know why but it wasn't working otherwise and I think it has something to do with memory allocation maybe it works with -static flag because some references are staying in scope as the entire library is loaded into RAM ? 

I had to implement the PKCS1v15 again but I would like to know what is wrong so I can write the code accordingly.

I have the Integer c.  

1) If I use snippet 1, get the vector and then read it into a string "res" and return it, this works.

2) If I use snippet 2, aka do the exact same thing in the function. It doesn't work. I am trying to think whether the Vector needs to be declared with new ? I am passing back the whole object not a pointer, so it should not go out of scope ?? But maybe the resarr2 is not a deep copy and when it goes out of scope the string data is also destroyed ?? On the other side, the serve complains with Snippet 2 and not with Snippet 1.

Snippet 1:  (works)

std::vector<byte> resarr2 = convert_cryptopp_integer(c);

string res(resarr2.begin(), resarr2.end());

return res;

Snippet  2: (doesn't work)

string res = convert_cryptopp_integer_str(c);

return res;

Functions:

std::vector<byte> convert_cryptopp_integer(Integer n){

const size_t len = n.MinEncodedSize(Integer::UNSIGNED);

std::vector<byte> v;

v.resize(len);

n.Encode((byte*)&v[0], v.size(), Integer::UNSIGNED);

//std::cout << "Iostream: " << std::hex << n << std::endl;

std::cout << "Iostream: " << n << std::endl;

std::cout << " Vector: ";

for(size_t i : v) { std::cout << (i & 0xff); }

std::cout << " Done vector: " << endl;

for(size_t i : v) { std::cout << int(i) << " "; }

std::cout << " Done cout: " << endl;

std::cout << std::endl;

return v;

}

string convert_cryptopp_integer_str(Integer n){

std::vector<byte> resarr2 = convert_cryptopp_integer(n);

string res(resarr2.begin(), resarr2.end());

return res;

}

std::string encrypt_rsa(std::string message, CryptoPP::RSA::PublicKey key)

{

try{

/* This was the previous code I couldn't get working without the -static flag

cout << " In encrypt rsa string " << endl;

std::string str(message.begin(), message.end());

message = b64encode(str);

CryptoPP::AutoSeededRandomPool rng;

//CryptoPP::RSAES_OAEP_SHA_Encryptor encryptor(key);

CryptoPP::RSAES_PKCS1v15_Encryptor encryptor(key);

std::string ciphertext;

CryptoPP::StringSource(message, true, new CryptoPP::PK_EncryptorFilter(rng, encryptor, new CryptoPP::StringSink(ciphertext)));

return ciphertext;

*/

std::string str(message.begin(), message.end());

string message = b64encode(str);

cout << " Got message of length" << message.length() << endl;

//convert the message to b64

string ts="\0"s;

ts += "\2"s;

//Add the 00 02 start header

int target_length = key.GetModulus().ByteCount();

int msglength = message.length();

int padding_length = target_length - msglength - 3;

//calculate the padding length for PKCS1 v1.5

SecByteBlock rand(padding_length);

OS_GenerateRandomBlock(true, rand, padding_length);

string rands;

HexEncoder hex(new StringSink(rands));

hex.Put(rand, rand.size());

hex.MessageEnd();

std::regex reg("\0");

rands = std::regex_replace(rands,reg,"a");

rands.erase(padding_length);

//generate random padding, replace any NULLs with "a"

if(message.size()<key.GetModulus().ByteCount()){

ts += rands;

//add the padding to the message

ts += "\0"s;

//NULL terminate the padding

ts = ts + message;

//add the payload

}

const char* msg_c = ts.c_str();

Integer m((const byte*)ts.data(), ts.size());

//convert it to Integer

Integer c = key.ApplyFunction(m);

//encrypt it

size_t ensize = c.MinEncodedSize();

//get the size

std::vector<byte> resarr2 = convert_cryptopp_integer(c);

string res(resarr2.begin(), resarr2.end());

//This works

//string res = convert_cryptopp_integer_str(c);

//This line above doesn't work is it memory issue of the vector<byte> going out of scope??

return res;

}

catch(Exception e)

{

std::cout << "error encrypting RSA " << e.what();

return "";

}

}

[Jeffrey Walton]

On Tue, Apr 25, 2023 at 7:12 PM Dwight Kulkarni <dwi...@realtime-7.com> wrote:
>
> I got it working by implementing the raw Integer method. See my code below, previous encryption is commented out. I don't know why but it wasn't working otherwise and I think it has something to do with memory allocation maybe it works with -static flag because some references are staying in scope as the entire library is loaded into RAM ?
>
> I had to implement the PKCS1v15 again but I would like to know what is wrong so I can write the code accordingly.
>
> I have the Integer c.
>
> 1) If I use snippet 1, get the vector and then read it into a string "res" and return it, this works.
>
> 2) If I use snippet 2, aka do the exact same thing in the function. It doesn't work. I am trying to think whether the Vector needs to be declared with new ? I am passing back the whole object not a pointer, so it should not go out of scope ?? But maybe the resarr2 is not a deep copy and when it goes out of scope the string data is also destroyed ?? On the other side, the serve complains with Snippet 2 and not with Snippet 1.

If you want someone to look at your programs, you need to provide a
minimal reproducer somewhere it can be cloned like GitHub or GitLab.
I'm not going to try to copy/paste it from the mailing list. I'm also
not going to try to make it compile due to missing functions and
missing main().

Now, onto your memory problems. Use Address Sanitizer or Valgrind to
locate the memory error. I recommend Address Sanitizer (Asan). Asan
produces the best debug and diagnostics. Follow these steps.

# where the Crypto++ sources are
cd cryptopp
make distclean

# make the library
CXXFLAGS="-DNDEBUG -g3 -O1 -fsanitize=address" make -j 5

# test the library. Notice no memory errors
./cryptest.exe vv

# copy your test program. Be sure to name it *.cxx
cp ../../test.cxx .
g++ -o test.exe -DNDEBUG -g3 -O1 -I. -fsanitize=address test.cxx ./libcryptopp.a

# run your program
./test.exe

Attached is my broken test.cxx. I used a vector that was too small:

size_t len = n.MinEncodedSize(Integer::UNSIGNED);

std::vector<byte> v(len-4);
n.Encode((byte*)&v[0], v.size()+4, Integer::UNSIGNED);

Here is the result of my broken test program:

$ g++ -o test.exe -DNDEBUG -g3 -O1 -I. -fsanitize=address test.cxx
./libcryptopp.a
$ ./test.exe
=================================================================
==12167==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x60200000003c at pc 0x7ffb24239e67 bp 0x7ffd10274f30 sp
0x7ffd102746d8
WRITE of size 1 at 0x60200000003c thread T0
#0 0x7ffb24239e66 in __interceptor_memmove
../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810
#1 0x55a0a1c8cac2 in memmove
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:36
#2 0x55a0a1c8cac2 in CryptoPP::ArraySink::Put2(unsigned char
const*, unsigned long, int, bool)
/home/jwalton/cryptopp/filters.cpp:545
#3 0x55a0a1b1f642 in
CryptoPP::BufferedTransformation::Put(unsigned char const*, unsigned
long, bool) /home/jwalton/cryptopp/cryptlib.h:1689
#4 0x55a0a1b1f642 in
CryptoPP::BufferedTransformation::Put(unsigned char, bool)
/home/jwalton/cryptopp/cryptlib.h:1679
#5 0x55a0a1b1f642 in
CryptoPP::Integer::Encode(CryptoPP::BufferedTransformation&, unsigned
long, CryptoPP::Integer::Signedness) const
/home/jwalton/cryptopp/integer.cpp:3439
#6 0x55a0a1b1fa84 in CryptoPP::Integer::Encode(unsigned char*,
unsigned long, CryptoPP::Integer::Signedness) const
/home/jwalton/cryptopp/integer.cpp:3431
#7 0x55a0a1af64ac in main /home/jwalton/cryptopp/test.cxx:18
#8 0x7ffb23a29d8f in __libc_start_call_main
../sysdeps/nptl/libc_start_call_main.h:58
#9 0x7ffb23a29e3f in __libc_start_main_impl ../csu/libc-start.c:392
#10 0x55a0a1af6004 in _start (/home/jwalton/cryptopp/test.exe+0x66004)

0x60200000003c is located 0 bytes to the right of 12-byte region
[0x602000000030,0x60200000003c)
allocated by thread T0 here:
#0 0x7ffb242b61c7 in operator new(unsigned long)
../../../../src/libsanitizer/asan/asan_new_delete.cpp:99
#1 0x55a0a1af632c in __gnu_cxx::new_allocator<unsigned
char>::allocate(unsigned long, void const*)
/usr/include/c++/11/ext/new_allocator.h:127
#2 0x55a0a1af632c in std::allocator_traits<std::allocator<unsigned
char> >::allocate(std::allocator<unsigned char>&, unsigned long)
/usr/include/c++/11/bits/alloc_traits.h:464
#3 0x55a0a1af632c in std::_Vector_base<unsigned char,
std::allocator<unsigned char> >::_M_allocate(unsigned long)
/usr/include/c++/11/bits/stl_vector.h:346
#4 0x55a0a1af632c in std::_Vector_base<unsigned char,
std::allocator<unsigned char> >::_M_create_storage(unsigned long)
/usr/include/c++/11/bits/stl_vector.h:361
#5 0x55a0a1af632c in std::_Vector_base<unsigned char,
std::allocator<unsigned char> >::_Vector_base(unsigned long,
std::allocator<unsigned char> const&)
/usr/include/c++/11/bits/stl_vector.h:305
#6 0x55a0a1af632c in std::vector<unsigned char,
std::allocator<unsigned char> >::vector(unsigned long,
std::allocator<unsigned char> const&)
/usr/include/c++/11/bits/stl_vector.h:511
#7 0x55a0a1af632c in main /home/jwalton/cryptopp/test.cxx:17

SUMMARY: AddressSanitizer: heap-buffer-overflow
../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810
in __interceptor_memmove
Shadow bytes around the buggy address:
0x0c047fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c047fff8000: fa fa 00 fa fa fa 00[04]fa fa fa fa fa fa fa fa
0x0c047fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc

[Dwight Kulkarni]

Hi Jeff,

I ran address sanitizer:  Heap use after free is the problem.  I had to use the NEW operator to allocate the ram.  What is weird is that usually it will error out right away after it goes out of scope, but this was delayed in recovering the memory, so it was working and then poof the memory disappears, but there is no access violation, it just returns bad result but only sometimes. Other times it is working and if there is a delay then the RAM gets reclaimed in that time and the error occurs right in the middle of the function. Really freaky.

char* convert_cryptopp_integer_str(Integer n, size_t& msg_len){

const size_t len = n.MinEncodedSize(Integer::UNSIGNED);

char* v = new char[len];

msg_len = len;

n.Encode((byte*)v, len, Integer::UNSIGNED);

return v;

}

std::vector<byte>* convert_cryptopp_integer(Integer n){

const size_t len = n.MinEncodedSize(Integer::UNSIGNED);

std::vector<byte>* v = new std::vector<byte>(len);

n.Encode((byte*)v, v->size(), Integer::UNSIGNED);

return v;

}

=================================================================
==23926==ERROR: AddressSanitizer: heap-use-after-free on address 0xffff82607880 at pc 0xaaaaca7153cc bp 0xffffdc752330 sp 0xffffdc752350
READ of size 1 at 0xffff82607880 thread T0  

==23926==ABORTING

[Jeffrey Walton]

On Wed, Apr 26, 2023 at 6:13 PM Dwight Kulkarni <dwi...@realtime-7.com> wrote:
>
> I ran address sanitizer: Heap use after free is the problem. I had to use the NEW operator to allocate the ram. What is weird is that usually it will error out right away after it goes out of scope, but this was delayed in recovering the memory, so it was working and then poof the memory disappears, but there is no access violation, it just returns bad result but only sometimes. Other times it is working and if there is a delay then the RAM gets reclaimed in that time and the error occurs right in the middle of the function. Really freaky.
>
> char* convert_cryptopp_integer_str(Integer n, size_t& msg_len){

> const size_t len = n.MinEncodedSize(Integer::UNSIGNED);

> char* v = new char[len];
> msg_len = len;
> n.Encode((byte*)v, len, Integer::UNSIGNED);
> return v;
> }
>
> std::vector<byte>* convert_cryptopp_integer(Integer n){

> const size_t len = n.MinEncodedSize(Integer::UNSIGNED);

> std::vector<byte>* v = new std::vector<byte>(len);
> n.Encode((byte*)v, v->size(), Integer::UNSIGNED);
> return v;
> }

So C-ish... Stop managing allocations with new and free. Let the std
C++ library do the work for you:

std::string convert_cryptopp_integer_str(const CryptoPP::Integer& n)
{
using namespace CryptoPP;

const size_t len = n.MinEncodedSize(Integer::UNSIGNED);
std::string v;

v.resize(len);

n.Encode((byte*)&v[0], v.size(), Integer::UNSIGNED);

return v;

}

std::vector<CryptoPP::byte> convert_cryptopp_integer(const CryptoPP::Integer& n)
{
using namespace CryptoPP;

const size_t len = n.MinEncodedSize(Integer::UNSIGNED);
std::vector<byte> v(len);

n.Encode((byte*)&v[len], v.size(), Integer::UNSIGNED);
return v;
}

Jeff

[Jeffrey Walton]

I probably should have said... This is the _portable_ way to get the
non-const pointer to the first element in the array:

n.Encode((byte*)&v[0], v.size(), Integer::UNSIGNED);

It is portable because it works with C++98 and above. Other methods
exist, but they work with C++11 and above.

This was wrong. It should have used element 0, not element 'len':

n.Encode((byte*)&v[len], v.size(), Integer::UNSIGNED);

Jeff

[Dwight Kulkarni]

Hi Jeff,

Your code is exactly what I had before. But I was getting that heap error. It only went away when I used NEW. I have never seen that before. I also don't like using NEW at all but it wasn't working. 

[Jeffrey Walton]

On Thu, Apr 27, 2023 at 12:19 AM Dwight Kulkarni <dwi...@realtime-7.com> wrote:
>
> Your code is exactly what I had before. But I was getting that heap error. It only went away when I used NEW. I have never seen that before. I also don't like using NEW at all but it wasn't working.
>

When moving an allocation from the stack to the heap (using new), and
behavior changes (like no memory error), it indicates a memory error.
The problem existed before and after. The heap just masked it.

Go back to your original program, and use Address Sanitizer on it. It
will show you where the error is, or give you a toe hold into the
problem.

Jeff

[Jeffrey Walton]

On Thu, Apr 27, 2023 at 12:19 AM Dwight Kulkarni <dwi...@realtime-7.com> wrote:
>

> Your code is exactly what I had before. But I was getting that heap error. It only went away when I used NEW. I have never seen that before. I also don't like using NEW at all but it wasn't working.

convert_cryptopp_intege and convert_cryptopp_integer_str test fine for
the attached program. You have a problem somewhere else in your
program.

$ g++ -o test.exe -g3 -O1 -fsanitize=address test.cxx ./libcryptopp.a
$ ./test.exe
Iostream: e9e930805f30e3ac95845917bef1d708h
Vector: e9e930805f30e3ac95845917bef1d708
Convert1: e9e930805f30e3ac95845917bef1d708
Convert2: e9e930805f30e3ac95845917bef1d708

Jeff

[Arik Agazarian]

We Need wallet address of 7 months for clean transactions, traceable on Blockchain, 

Ratio 45%/45%/10%

Sender/Reciever/Mandate

Telegram sender: @Arik_loader

On Thu, Nov 2, 2023, 7:28 PM Catherine Issabel <issabelc...@gmail.com> wrote:

$500 million dollars available only for good and trusted receiver or mandate
Kindly contact the sender for more information:+1 (336) 345-9681

--
You received this message because you are subscribed to the Google Groups "Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cryptopp-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cryptopp-users/2eb52212-320f-4f5c-92da-6c61f95325c5n%40googlegroups.com.