Serialize with Length Prefix

[Jay Thomas]

Hello

I am looking for a way to serialize/deserialize with length prefix
under c++. The serialized bytes will sent to a TCP socket. I am
aware that C# has a method SerializeWithLengthPrefix() and
DeserializeWithLengthPrefix(). Are there any such analogous methods
for C++? Please point out any documentation that I may have missed
here.

Thanks for any assistance.

Jay

[Peter Keen]

You sort of have to roll your own. In my project I'm doing something like this:

coded_output->WriteVarint32(message->ByteSize());
message->SerializeToCodedStream(coded_output);

And then on the reading side:

uint32_t size;
if (! coded_input->ReadVarint32(&size)) {
return NULL; // just an example. probably don't want to do this;
}

Message * m = new Message;
CodedInputStream::Limit limit = coded_input->PushLimit(size);
message->ParseFromCodedStream(coded_input);
coded_input->PopLimit(limit);

The biggest thing that helped me along was finding the Limit docs. I
couldn't figure out how to parse from a ZeroCopyInputStream without
consuming the entire stream, and then I ran across that and everything
became much more clear.

--Pete

[Jay Thomas]

Some things I forgot to mention:
-working on Slackware Linux platform
-did text search for DeserializeWithLengthPrefix() and haven't found
any header or code files with this method
-was able to find lots of serializing methods in message.h but that
same header file doesn't contain any methods starting with Deserialize
-message.h doesn't contain declaration for SerializeWithLengthPrefix
(). This function doesn't seem to be declared anywhere in any code or
header file.

[Jay Thomas]

This code ideally has to operate with C# code on the other side of the
socket that uses the DeserializeWithLengthPrefix() method call. The
message originates in C++ environment on a linux box and terminates in
C# environment on a windows PC. Why doesn't C++ support the same
methods as C#? The only document that I've seen that has list of
methods available is the message.h header file, installed into my
Linux usr directory.

On Aug 24, 4:05 pm, Peter Keen <peter.k...@gmail.com> wrote:
> You sort of have to roll your own. In my project I'm doing something like this:
>
> coded_output->WriteVarint32(message->ByteSize());
> message->SerializeToCodedStream(coded_output);
>
> And then on the reading side:
>
> uint32_t size;
> if (! coded_input->ReadVarint32(&size)) {
>   return NULL; // just an example. probably don't want to do this;
>
> }
>
> Message * m = new Message;
> CodedInputStream::Limit limit = coded_input->PushLimit(size);
> message->ParseFromCodedStream(coded_input);
> coded_input->PopLimit(limit);
>
> The biggest thing that helped me along was finding the Limit docs. I
> couldn't figure out how to parse from a ZeroCopyInputStream without
> consuming the entire stream, and then I ran across that and everything
> became much more clear.
>
> --Pete
>

[Peter Keen]

On Mon, Aug 24, 2009 at 4:17 PM, Jay Thomas<jaydian...@gmail.com> wrote:
>
> This code ideally has to operate with C# code on the other side of the
> socket that uses the DeserializeWithLengthPrefix() method call.  The
> message originates in C++ environment on a linux box and terminates in
> C# environment on a windows PC.

I'm not familiar with the implementation of
DeserializeWithLengthPrefix(). The same idea should work, but you'll
have to go into the protobuf-net source to see what it uses to write
the length with. If it's not a Varint32 then just adjust it to
whatever it needs to be.

> Why doesn't C++ support the same methods as C#?

They're different projects. protobuf-net has a different maintainer
than the protobuf project.

> The only document that I've seen that has list of
> methods available is the message.h header file, installed into my
> Linux usr directory.

This is the documentation for the generated C++ code, which really
just deals with the fields that are generated by the protoc compliler:

http://code.google.com/apis/protocolbuffers/docs/reference/cpp-generated.html

This is the documentation for the C++ library as a whole:

http://code.google.com/apis/protocolbuffers/docs/reference/cpp/index.html

This is the documentation for the C++ version of Message:

http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.message.html

This is the documentation for the coded_stream stuff I was talking about:

http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.io.coded_stream.html#CodedInputStream.Limit.details

--Pete

[Peter Keen]

I looked at the protobuf-net source a little bit and it looks like if
you write the length with WriteLittleEndian32 instead of WriteVarint32
you should be good to go, as long as you use PrefixStyle.Fixed32
within the C# call to DeserialzeWithLengthPrefix().

--Pete

[Jay Thomas]

cool. thanks Pete. your help is greatly appreciated!

On Aug 24, 4:33 pm, Peter Keen <peter.k...@gmail.com> wrote:
> I looked at the protobuf-net source a little bit and it looks like if
> you write the length with WriteLittleEndian32 instead of WriteVarint32
> you should be good to go, as long as you use PrefixStyle.Fixed32
> within the C# call to DeserialzeWithLengthPrefix().
>
> --Pete
>

[Marc Gravell]

Just to offer my thanks as well, since I was offline at the time and
unable to help.

Regards,

Marc Gravell (protobuf-net)

[Marc Gravell]

this seems to be a resurrection of a protobuf-net specific discussion, so let me jump in...

The SerializeWithLengthPrefix method, by default, aims to represent data in a way that is a valid protobuf stream - in particular, as though it were simply a member of a parent object or list. As such, it encodes a [ field-number and "string" marker ], then the length - as two separate varints.

So assuming the data was stored in this way (the default), consuming 2 defaults should consume the header information - the second of the 2 values being the length of the subsequent payload. If the data was stored with different header-formats, then you may need to adjust - for example the fixed32 obviously requires 4 bytes to be consumed.

Does that make sense? or have I missed the point of the question?

Marc

On 20 February 2013 13:32, Paul Shafer <prab...@gmail.com> wrote:

First time poster and new to protobuf/protobuf-net so forgive me if this has already been addressed.  But I've searched quite a bit and cant find an"official" answer.

If you want to use the fieldNumber argument in your SerializeWithLengthPrefix/TryDeserializeWithLengthPrefix for message type identification (or for whatever reason), all works great between two C#/.NET apps.  However if you have a c++ app using protobuf talking to a C#/.NET app using protobuf-net, the fieldNumber argument will cause grief.  It took wireshark, plus diving into protobuf-net code and protobuf code to find that the fieldNumber is encoded like (I assume) any other field within a message.  Whereas the c++ protobuf has no such notion (that I could find) unless you want to use a CodedOutputSTream - which then has both a WriteTag and WriteVarint32 (which WriteTag calls).  However WriteTag/WriteVarint32 does NOT encode the same as the fieldNumber with SerializeWithLengthPrefix.  So... what do to?

Again, perhaps this has been addressed already but here's my solution (on the c++ side):

#define PROTO_NET_TAG(x) (((x) << 3) | (2 & 7))

...

google::protobuf::io::ZeroCopyOutputStream* raw_output = new google::protobuf::io::StringOutputStream(str);

google::protobuf::io::CodedOutputStream * coded = new google::protobuf::io::CodedOutputStream(raw_output);

coded->WriteTag(PROTO_NET_TAG(msgTag));

coded->WriteVarint32(msg->ByteSize());

msg->SerializeToCodedStream(coded);

...

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Regards,

Marc