RE: Gzip & Gunzip code: Correct or rubbish
Scott Barnhart
Based on the code in test.cpp for GzipFile and GunzipFile, I have always
used the following, and it should work for you:
// Gzip contents of plaintext into gzip_text;
StringSource(plaintext, true, new Gzip(new StringSink(gzip_text)));
// Ungzip contents of gzip_text into unzip_text;
StringSource(gzip_text, true, new Gunzip(new StringSink(unzip_text)));
Maybe I am missing something in your requirements? (I have been pretty much
following your whole thread.)
You can also string these together, so if you want to gzip then B64 encode,
you could:
StringSource(plaintext, true, new Gzip(new Base64Encoder(new
StringSink(b64_text))));
And then to undo it back to plaintext, you do:
StringSource(b64_text, true, new Base64Decoder(new Gunzip(new
StringSink(unb64_text))));
Hope this helps,
Scott Barnhart
-----Original Message-----
From: Stephen torri [mailto:sto...@torri.org]
Sent: Tuesday, November 18, 2003 4:38 PM
To: cryptopp
Subject: Gzip & Gunzip code: Correct or rubbish
I decided to work on get my encrypt and decrypt methods in stages. The
first stage is to get the plain text string compressed and decompressed
correctly.
Gcc:3.2.3
Crypto:5.1
Compress:
string gzip_text;
Gzip aes_compressor2 (new StringSink (gzip_text));
StringStore(plaintext).TransferTo(aes_compressor2);
string unzip_text;
Gunzip aes_decompressor2 (new StringSink (unzip_text));
StringStore(gzip_text).TransferTo(aes_decompressor2);
I got this from searching the archive of the mailing list. Shame we do
not have a directory of example programs. Is this right? So far I
believe it does not because what I get in 'unzip_text' is rubbish.
Plaintext char: B o o H o o I a m s o s c
a r e d
Plaintext hex: 0x42 0x6f 0x6f 0x20 0x48 0x6f 0x6f 0x20 0x49
0x20 0x61 0x6d 0x20 0x73 0x6f 0x20 0x73 0x63 0x61 0x72 0x65
0x64
<OUTPUT: Compress plain text>
Compressed text2: 0x1f 0xffffff8b 0x8 0 0 0 0 0 0 0
<OUTPUT: Decompressed plain text>
Decompressed text2:
Stephen
Stephen torri
Walton, Jeffrey
> Is this right? So far I believe it does not because what I get in
> 'unzip_text' is rubbish.
Probably not if you are getting rubbish.
Here's a non-elegant sample (I'm almost embarrassed to post it). Notice how
all the objects basically act the same. You can drop in a base64
(en/de)coder in place of the hex. That's one of the great benefits of using
this library. Sadly, I don't take advantage of chaining...
Jeff
BTW, sample code that we could compile would be nice next time.
#include "gzip.h"
#include "hex.h"
#include <iostream>
#include <string>
#pragma comment( lib, "cryptlibd" )
using std::string;
using std::cout;
using std::endl;
int main(int argc, char* argv[]) {
string original("B o o H o o I a m s o s c a
r e d");
cout << "Original String:" << endl << original << endl << endl;
///////////////////////////
//
// Compress
//
CryptoPP::Gzip compressor;
compressor.PutMessageEnd( (const byte*)original.c_str(), original.size()
);
// Alternately
// compressor.Put( (const byte*)original.c_str(), original.size() );
// compressor.MessageEnd();
byte* zip = NULL;
int size = compressor.MaxRetrievable();
zip = new byte[ size ];
compressor.Get( zip, size );
///////////////////////////
//
// Encode
//
CryptoPP::HexEncoder encoder;
encoder.PutMessageEnd( zip, size );
byte* hex = NULL;
hex = new byte[ encoder.MaxRetrievable() + 1 ];
hex[ encoder.MaxRetrievable() ] = '\0';
encoder.Get( hex, encoder.MaxRetrievable() );
cout << "Compressed String:" << endl << hex << endl << endl;
///////////////////////////
//
// Decode
//
// Would look like the rest of it.
///////////////////////////
//
// Decompress
//
CryptoPP::Gunzip decompressor;
decompressor.PutMessageEnd( zip, size );
string decompressed;
decompressed.resize( decompressor.MaxRetrievable() + 1 );
decompressed[ decompressor.MaxRetrievable() ] = '\0';
decompressor.Get( (byte*)decompressed.c_str(),
decompressor.MaxRetrievable() );
cout << "Decompressed String:" << endl << decompressed << endl << endl;
if( NULL != zip ) {
delete[] zip;
}
if( NULL != zip ) {
delete[] hex;
}
return 0;
}
> -----Original Message-----
> From: cryptopp-l...@eskimo.com
> [mailto:cryptopp-l...@eskimo.com] On Behalf Of Stephen torri
> Sent: Tuesday, November 18, 2003 7:38 PM
> To: cryptopp
> Subject: Gzip & Gunzip code: Correct or rubbish
>
>
Stephen torri
> Stephen,
>
> Based on the code in test.cpp for GzipFile and GunzipFile, I have always
> used the following, and it should work for you:
>
> // Gzip contents of plaintext into gzip_text;
> StringSource(plaintext, true, new Gzip(new StringSink(gzip_text)));
>
> // Ungzip contents of gzip_text into unzip_text;
> StringSource(gzip_text, true, new Gunzip(new StringSink(unzip_text)));
>
> Maybe I am missing something in your requirements? (I have been pretty much
> following your whole thread.)
>
> You can also string these together, so if you want to gzip then B64 encode,
> you could:
>
> StringSource(plaintext, true, new Gzip(new Base64Encoder(new
> StringSink(b64_text))));
>
> And then to undo it back to plaintext, you do:
>
> StringSource(b64_text, true, new Base64Decoder(new Gunzip(new
> StringSink(unb64_text))));
A couple more steps here.
I want to take the plain text and produce an encrypted text using AES.
I want to create a MAC of the encrypted text.
I want to create a single string containing the encrypted text and MAC.
So I can use what you gave to produce a compressed string. So
<encrypt>
plain text -> compress text (Use what you gave me above)
code: StringSource(plaintext, true, new Gzip(new
StringSink(gzip_text)));
compress text -> encrypted text (base64 encoded)
Q: What do you use for encrypting a std::string?
- A StringTransformationFilter?
mac text = mac of encrypted text
Q: What do you use for generating a MAC?
Q: Is a MAC a fixed size for all types? (e.g. SHA512 and SHA256)
cipher text = encrypted text + mac text
code: stringstream str;
str << encrypted text << mac text;
return str.str();
Stephen
--
Stephen Torri
GPG Key: http://www.cs.wustl.edu/~storri/storri.asc
Scott Barnhart
That's where it gets more tricky. I have always used the
DefaultEncryptorWithMAC for most of my encryption needs, and it is already
setup to just plop it in the same way I did with Gzip and Base64Encoder. So
most of your questions are over my head. The class DefaultEncryptorWithMAC
uses DES-EDE2 and HMAC/SHA-1 to do its thing, but it uses Password-Based
encryption, instead of real keys, so I am not exactly sure how that would
affect things.
However, if you just copy the code for DefaultEncryptorWithMAC, and change
its algorithm from DESede to AES, and pass in a real key and IV instead of a
passphrase, you should be able to put something together.
I do sympathize with what you are going thru - everytime I try to do
anything other then what has already been coded in test.cpp by Wei, I end up
killing myself for days to figure it out.
Scott Barnhart
-----Original Message-----
From: Stephen torri [mailto:sto...@torri.org]
Stephen torri
> Stephen,
>
> That's where it gets more tricky. I have always used the
> DefaultEncryptorWithMAC for most of my encryption needs, and it is already
> setup to just plop it in the same way I did with Gzip and Base64Encoder. So
> most of your questions are over my head. The class DefaultEncryptorWithMAC
> uses DES-EDE2 and HMAC/SHA-1 to do its thing, but it uses Password-Based
> encryption, instead of real keys, so I am not exactly sure how that would
> affect things.
>
> However, if you just copy the code for DefaultEncryptorWithMAC, and change
> its algorithm from DESede to AES, and pass in a real key and IV instead of a
> passphrase, you should be able to put something together.
>
> I do sympathize with what you are going thru - everytime I try to do
> anything other then what has already been coded in test.cpp by Wei, I end up
> killing myself for days to figure it out.
>
> Scott Barnhart
Thanks. I will look into it.
Stephen
Stephen torri
> On Tue, 2003-11-18 at 19:14, Scott Barnhart wrote:
> > Stephen,
> >
> > Based on the code in test.cpp for GzipFile and GunzipFile, I have always
> > used the following, and it should work for you:
> >
> > // Gzip contents of plaintext into gzip_text;
> > StringSource(plaintext, true, new Gzip(new StringSink(gzip_text)));
> >
> > // Ungzip contents of gzip_text into unzip_text;
> > StringSource(gzip_text, true, new Gunzip(new StringSink(unzip_text)));
> >
I just decided to run the code as it stands. It core dumps on the Gunzip
line. Here is the backtrace:
#0 0x47355e81 in kill () from /lib/libc.so.6
#1 0x47355c25 in raise () from /lib/libc.so.6
#2 0x4735719b in abort () from /lib/libc.so.6
#3 0x472d3837 in __cxxabiv1::__terminate(void (*)()) ()
from /usr/lib/gcc-lib/i686-pc-linux-gnu/3.2.3/libstdc++.so.5
#4 0x472d3878 in std::terminate() ()
from /usr/lib/gcc-lib/i686-pc-linux-gnu/3.2.3/libstdc++.so.5
#5 0x472d3a1e in __cxa_throw ()
from /usr/lib/gcc-lib/i686-pc-linux-gnu/3.2.3/libstdc++.so.5
#6 0x470e690f in CryptoPP::Inflator::OutputPast(unsigned, unsigned) (
this=0x8065450, length=10, distance=52) at zinflate.cpp:261
#7 0x470e822e in CryptoPP::Inflator::DecodeBody() (this=0x8065450)
at zinflate.cpp:558
#8 0x470e6fb0 in CryptoPP::Inflator::ProcessInput(bool)
(this=0x8065450,
flush=true) at zinflate.cpp:345
#9 0x470e6c39 in CryptoPP::Inflator::Put2(unsigned char const*,
unsigned, int, bool) (this=0x8065450, inString=0x0, length=0,
messageEnd=-1, blocking=true)
at zinflate.cpp:294
#10 0x46f6dd39 in
CryptoPP::BufferedTransformation::ChannelPut2(std::string const&,
unsigned char const*, unsigned, int, bool) (this=0x8065450,
channel=@0x806236c, begin=0x0, length=0, messageEnd=-1,
blocking=true)
at cryptlib.cpp:189
#11 0x46f719ec in
CryptoPP::BufferedTransformation::ChannelMessageEnd(std::string const&,
int, bool) (this=0x8065450, channel=@0x806236c, propagation=-1,
blocking=true) at cryptlib.h:881
#12 0x46f6e88c in
CryptoPP::BufferedTransformation::TransferMessagesTo2(CryptoPP::BufferedTransformation&, unsigned&, std::string const&, bool) (
this=0xb76b3204, target=@0x8065450, messageCount=@0xb76b308c,
channel=@0x806236c, blocking=true) at cryptlib.cpp:354
#13 0x46f6ead0 in
CryptoPP::BufferedTransformation::TransferAllTo2(CryptoPP::BufferedTransformation&, std::string const&, bool) (this=0xb76b3204,
---Type <return> to continue, or q <return> to quit---
target=@0x8065450, channel=@0x806236c, blocking=true) at
cryptlib.cpp:395
#14 0x08058a35 in
CryptoPP::SourceTemplate<CryptoPP::StringStore>::PumpAll2(bool) ()
#15 0x0805948f in CryptoPP::Source::PumpAll() ()
#16 0x0805809a in CryptoPP::Source::SourceInitialize(bool,
CryptoPP::NameValuePairs const&) ()
#17 0x0805314e in
CryptoPP::StringSource::StringSource<std::string>(std::string const&,
bool, CryptoPP::BufferedTransformation*) ()
#18 0x080518e9 in AES_CFB_CipherEngine::encrypt(std::string) ()
#19 0x080527a0 in save(AES_CFB_CipherEngine&, std::string, std::string)
()
#20 0x08052a26 in main ()
#21 0x473427a7 in __libc_start_main () from /lib/libc.so.6
Gcc: 3.2.3
Crypto++: 5.1
-----------------
strace
-----------------
bash-2.05b$ strace ./encryption_test
execve("./encryption_test", ["./encryption_test"], [/* 56 vars */]) = 0
uname({sys="Linux", node="base.torri.org", ...}) = 0
brk(0) = 0x8062660
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x4082b000
open("/etc/ld.so.preload", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
close(3) = 0
open("cryptopp/.libs/i686/mmx/libcryptopp.so.0", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("cryptopp/.libs/i686/libcryptopp.so.0", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("cryptopp/.libs/mmx/libcryptopp.so.0", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("cryptopp/.libs/libcryptopp.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0@\263,\000"...,
1024) = 1024
fstat64(3, {st_mode=S_IFREG|0755, st_size=15678342, ...}) = 0
getcwd("/home/storri/Documents/StephenTorri/cs502_project/src", 128) =
54
mmap2(NULL, 5857216, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) =
0x4082c000
mprotect(0x40d4e000, 475072, PROT_NONE) = 0
mmap2(0x40d4e000, 466944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED,
3, 0x521) = 0x40d4e000
mmap2(0x40dc0000, 8128, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40dc0000
close(3) = 0
open("cryptopp/.libs/i686/mmx/libstdc++.so.5", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("cryptopp/.libs/i686/libstdc++.so.5", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("cryptopp/.libs/mmx/libstdc++.so.5", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("cryptopp/.libs/libstdc++.so.5", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=99875, ...}) = 0
mmap2(NULL, 99875, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40dc2000
close(3) = 0
open("/usr/lib/gcc-lib/i686-pc-linux-gnu/3.2.3/libstdc++.so.5",
O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0p\0\4\000"...,
1024) = 1024
fstat64(3, {st_mode=S_IFREG|0755, st_size=1068068, ...}) = 0
mmap2(NULL, 830140, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40ddb000
mprotect(0x40e9c000, 39612, PROT_NONE) = 0
mmap2(0x40e9c000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3,
0xc1) = 0x40e9c000
mmap2(0x40ea1000, 19132, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40ea1000
close(3) = 0
open("cryptopp/.libs/i686/mmx/libm.so.6", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("cryptopp/.libs/i686/libm.so.6", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("cryptopp/.libs/mmx/libm.so.6", O_RDONLY) = -1 ENOENT (No such file
or directory)
open("cryptopp/.libs/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or
directory)
open("/lib/libm.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\2405\0"...,
1024) = 1024
fstat64(3, {st_mode=S_IFREG|0755, st_size=189625, ...}) = 0
mmap2(NULL, 135616, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40ea6000
mprotect(0x40ec7000, 448, PROT_NONE) = 0
mmap2(0x40ec7000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3,
0x21) = 0x40ec7000
close(3) = 0
open("cryptopp/.libs/i686/mmx/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("cryptopp/.libs/i686/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("cryptopp/.libs/mmx/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("cryptopp/.libs/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file
or directory)
open("/usr/lib/gcc-lib/i686-pc-linux-gnu/3.2.3/libgcc_s.so.1", O_RDONLY)
= 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 \27\0\000"...,
1024) = 1024
fstat64(3, {st_mode=S_IFREG|0644, st_size=153827, ...}) = 0
mmap2(NULL, 35744, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40ec8000
mprotect(0x40ed0000, 2976, PROT_NONE) = 0
mmap2(0x40ed0000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3,
0x7) = 0x40ed0000
close(3) = 0
open("cryptopp/.libs/i686/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("cryptopp/.libs/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such
file or directory)
open("cryptopp/.libs/mmx/libc.so.6", O_RDONLY) = -1 ENOENT (No such file
or directory)
open("cryptopp/.libs/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or
directory)
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0`X\1\000"...,
1024) = 1024
fstat64(3, {st_mode=S_IFREG|0755, st_size=1466302, ...}) = 0
mmap2(NULL, 1231748, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) =
0x40ed1000
mprotect(0x40ff8000, 23428, PROT_NONE) = 0
mmap2(0x40ff8000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3,
0x126) = 0x40ff8000
mmap2(0x40ffc000, 7044, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40ffc000
close(3) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x40ffe000
munmap(0x40dc2000, 99875) = 0
open("/dev/urandom", O_RDONLY) = 3
read(3, "\3\7#\375\32\331\274o\352/\205w\6\222\375\221\267\342\310"...,
32) = 32
close(3) = 0
brk(0) = 0x8062660
brk(0x8063660) = 0x8063660
brk(0x8064000) = 0x8064000
open("Foo.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
brk(0) = 0x8064000
brk(0x8065000) = 0x8065000
brk(0) = 0x8065000
brk(0x8066000) = 0x8066000
fstat64(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 1), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x40dc2000
write(1, "Plaintext char: B o o H "..., 83Plaintext char: B o
o H o o I a m s o s c a r e d
) = 83
write(1, "Plaintext hex: 0x42 0x6f 0x6f"..., 148Plaintext hex: 0x42
0x6f 0x6f 0x20 0x48 0x6f 0x6f 0x20 0x49 0x20 0x61 0x6d 0x20
0x73 0x6f 0x20 0x73 0x63 0x61 0x72 0x65 0x64
) = 148
brk(0) = 0x8066000
brk(0x8067000) = 0x8067000
brk(0) = 0x8067000
brk(0x8077000) = 0x8077000
brk(0) = 0x8077000
brk(0x8087000) = 0x8087000
brk(0) = 0x8087000
brk(0x8097000) = 0x8097000
brk(0) = 0x8097000
brk(0x80a7000) = 0x80a7000
brk(0) = 0x80a7000
brk(0x80a8000) = 0x80a8000
write(1, "Compressed text2: 0x1f 0xfffff"..., 318Compressed text2:
0x1f 0xffffff8b 0x8 0 0 0 0 0 0 0 0x73 0xffffffca
0xffffffcf 0x57 0xfffffff0 0 0x62 0x4f 0xffffff85 0xffffffc4
0x5c 0xffffff85 0xffffffe2 0x7c 0xffffff85 0xffffffe2 0xffffffe4
0xffffffc4 0xffffffa2 0xffffffd4 0x14 0 0x42 0xffffff8f 0x7c
0xffffffd0 0x16 0 0 0
) = 318
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
getpid() = 15300
kill(15300, SIGABRT) = 0
--- SIGABRT (Aborted) @ 0 (0) ---
+++ killed by SIGABRT +++
The code file is attached. Curious.
Stephen
Walton, Jeffrey
> #5 0x472d3a1e in __cxa_throw () from
> /usr/lib/gcc-lib/i686-pc-linux-gnu/3.2.3/libstdc++.so.5
Its probably blowing up on the ASSERT in the header check.
Jeff
> -----Original Message-----
> From: cryptopp-l...@eskimo.com
> [mailto:cryptopp-l...@eskimo.com] On Behalf Of Stephen torri
> Sent: Tuesday, November 18, 2003 11:24 PM
> To: cryptopp
> Cc: sbarn...@earthlink.net
> Subject: RE: Gzip & Gunzip code: Correct or rubbish
>
>
Walton, Jeffrey
Line 50 of gzip.cpp is probably the culprit:
void Gunzip::ProcessPrestreamHeader()
...
if (buf[0] != MAGIC1 || buf[1] != MAGIC2) throw HeaderErr();
...
Can you breakpoint it and verify?
Stephen torri
> Hi Stephen,
>
> Line 50 of gzip.cpp is probably the culprit:
>
> void Gunzip::ProcessPrestreamHeader()
>
> ...
>
> if (buf[0] != MAGIC1 || buf[1] != MAGIC2) throw HeaderErr();
>
> ...
>
> Can you breakpoint it and verify?
>
> Jeff
>
No. Line 50 of gzip.cpp is fine. I am seeing the problem start at the
second time I run at zinflate.cpp at line 558. At the second time
through the m_literal = 10 and m_distance = 52. So the we receive an
BadBlockErr exception at line 261 because we failed the if statement at
line 260. Its at line 26 of zinflate that I receive the SIGABRT.