Compress proto to string

[David Granados]

Hello.
I sucesfully compress/decompress a proto object using files, but i now need to do it same, but using string (no using files).

This code works fine for files:
 google::protobuf::io::FileOutputStream file_stream(filedescriptor);
 GzipOutputStream::Options options;
 options.format = GzipOutputStream::GZIP;
 options.compression_level = NIVEL_COMPRESION;
  google::protobuf::io::GzipOutputStream gzip_stream(&file_stream, options);


I was transformed this code in another to work with string:

const unsigned bufLength = 256;
  unsigned char buffer[bufLength];

  std::cout << buffer << std::endl;

  // Obtenemos un array del proto serializado
  ZeroCopyOutputStream* output = new google::protobuf::io::ArrayOutputStream(buffer,bufLength);

  // Introducimos las opciones de compression
  GzipOutputStream::Options options;
  options.format = GzipOutputStream::GZIP;
  options.compression_level = NIVEL_COMPRESION;

  google::protobuf::io::GzipOutputStream gzip_stream(output,options);
  if (!pProto.SerializeToZeroCopyStream(&gzip_stream))
  {
    std::cerr << "Error comprimiendo el proto a string." << std::endl;
  }

buffer is void. Where can be the error?

Regards.

[David Granados]

this does not work :-(

      212   // Definimos el buffer al cual vamos a comprimir
      213   const unsigned bufLength = 256;
      214   unsigned char buffer[bufLength];
      215
      216   memset(buffer,0,256);
      217
      218
      219   std::cout << buffer << std::endl;
      220
      221   // Obtenemos un array del proto serializado
      222   google::protobuf::io::ArrayOutputStream arrayStream(&buffer,bufLength);
      223
      224   // Introducimos las opciones de compression
      225   GzipOutputStream::Options options;
      226   options.format = GzipOutputStream::GZIP;
      227   options.compression_level = NIVEL_COMPRESION;
      228
      229   google::protobuf::io::GzipOutputStream gzip_stream(&arrayStream);
      230   if (!pProto.SerializeToZeroCopyStream(&gzip_stream))
      231   {
      232     std::cerr << "Error comprimiendo el proto a string." << std::endl;
      233   }

[Oliver Jowett]

On Tue, Jan 29, 2013 at 11:23 AM, David Granados <deiv...@gmail.com> wrote:
> this does not work :-(

That's not really a useful description of the problem.

> 229 google::protobuf::io::GzipOutputStream
> gzip_stream(&arrayStream);
> 230 if (!pProto.SerializeToZeroCopyStream(&gzip_stream))
> 231 {
> 232 std::cerr << "Error comprimiendo el proto a string." <<
> std::endl;
> 233 }

You should call gzip_stream.Flush() or gzip_stream.Close() after this,
before using the results in arrayStream.

Oliver

[Oliver Jowett]

(Readding the list cc:)

On Tue, Jan 29, 2013 at 4:32 PM, David Granados <deiv...@gmail.com> wrote:
> Does not work :-(
>
> Size of proto before: 196, size of proto after gzip: 196

What measures this? Nothing in any code you quoted.

> Now my code are:
> 206 template <class T> int
> 207 Proto<T>::CompressProtoToString(std::string& pOut,T pProto)

You never touch pOut.

> 208 {
> 209 // Resultado
> 210 int l_resultado = -1;
> 211

>
> 212 // Definimos el buffer al cual vamos a comprimir
> 213 const unsigned bufLength = 256;
> 214 unsigned char buffer[bufLength];
> 215
> 216 memset(buffer,0,256);
> 217
> 218
> 219 std::cout << buffer << std::endl;
> 220
> 221 // Obtenemos un array del proto serializado

> 222 ZeroCopyOutputStream* output = new
> google::protobuf::io::ArrayOutputStream(buffer,bufLength);
> 223 // Introducimos las opciones de compression
> 224 GzipOutputStream::Options options;
> 225 options.format = GzipOutputStream::GZIP;
> 226 options.compression_level = NIVEL_COMPRESION;

What is the value of NIVEL_COMPRESION?

> 227
> 228 google::protobuf::io::GzipOutputStream
> gzip_stream(output,options);
> 229 if (!pProto.SerializeToZeroCopyStream(&gzip_stream))
> 230 {
> 231 std::cerr << "Error comprimiendo el proto a string." <<
> std::endl;
> 232 }
> 233 gzip_stream.Close();
> 234 std::cout << ":" << buffer << std::endl;

This will fail if any 0-bytes happen to get serialized (you're
treating buffer as a nul-terminated C-style string)

> 235
> 236 // Devolvemos el resultado
> 237 return l_resultado;
> 238 }
>

I suggest that you post a *complete*, compilable, testcase - it's just
too hard to diagnose what's going wrong by guessing what you are doing
in the surrounding code that you didn't provide.

Oliver

[David Granados]

This is all the code for the example. Thanks in advance.

[Oliver Jowett]

On Wed, Jan 30, 2013 at 8:06 AM, David Granados <deiv...@gmail.com> wrote:
> This is all the code for the example. Thanks in advance.

What is the output that you expect? What do you actually get?

  std::cout << gzip_stream.ByteCount()<<std::endl;

This counts the number of uncompressed bytes written to the stream. Is that what you were expecting to measure?

Oliver

[David Granados]

I need compress a protobuff, and decompress, (i am testing by now).

I have got it using a temporal file but now, i want to get the same, using only string (without temporal file, increase speed).

By now. i'm testing. I know the proto is 192 without compress and 80 (aprox) compressed.

If i write the size and is 192, compress does not work.

But, i need the proto compress.

Regards

[Oliver Jowett]

On Wed, Jan 30, 2013 at 12:45 PM, David Granados <deiv...@gmail.com> wrote:

 

If i write the size and is 192, compress does not work.

In the code you attached, you are writing the *uncompressed* size to cout:

  std::cout << gzip_stream.ByteCount()<<std::endl;

GzipOutputStream::ByteCount() returns the number of uncompressed bytes written *to* the stream, not the number of compressed bytes that were written to the underlying stream. If you want to count the compressed bytes, call ByteCount() on the underlying stream. (i.e. output->ByteCount() in your code)

Also, if you want to eventually end up with a std::string containing the compressed output, why don't you use StringOutputStream? Something like this (untested):

 

  {

    StringOutputStream compressedStream(&outputString);

    GzipOutputStream compressingStream(compressedStream, options);

    pProto.SerializeToZeroCopyStream(&compressingStream); // error checks omitted for brevity

    // let destructors do stream cleanup

  }

Oliver

[David Granados]

This works fine!.
Thanks Oliver!