buffer sizes when sending messages from c++ to java

[Paul]

Hi,

I am sending messages from a C++ client to a Java server, but I am
wondering about the length of the buffer I am using to send it over.
This is the C++ code:

CLIENT SIDE CODE:
*** the protocol buffers message is called snap1 ***

int numBytesForDelim = sizeof(int);
char *snap_buf2;
snap_buf2 = new char[snap1.ByteSize() + 100];

ZeroCopyOutputStream* raw_output = new ArrayOutputStream(snap_buf2,
snap1.ByteSize()+numBytesForDelim);
CodedOutputStream* coded_output = new
CodedOutputStream(raw_output);

coded_output->WriteVarint32(snap1.ByteSize());
snap1.SerializeToCodedStream(coded_output);

delete coded_output;
delete raw_output;

send(socket, snap_buf2, snap1.ByteSize()+numBytesForDelim/2, 0); //
send over a socket
delete [] snap_buf2;
snap_buf2 = NULL;

When I send this over to the Java server side, the Java code uses
"parseDelimitedFrom" to parse the data. however, I am not sure why
when I call send, the length of the message has to be snap1.ByteSize()
+numBytesForDelim/2 rather than snap1.ByteSize()+numBytesForDelim. I
would think that it would be the latter, but when I try it, I get
exceptions on the Java side saying that the protocol message has not
filled out all the fields. For some reason, it is necessary for me to
cut off the last two bytes when sending over TCP. Any ideas as to why
this is?

Thanks a lot!

Paul

[Mike Dupont]

I dont know this in detail, but in my experience with java and c++ i
needed swap the two bytes before the buffer containing the length.
the length of the buffer is sent before the buffer from java to c++
using the java byte order.
I dont know if this applies, but you should print out the first two
bytes and try swapping them. Look up my old messages or code for
details.
mike

[Kenton Varda]

On Tue, Oct 19, 2010 at 6:28 PM, Paul <mjpa...@gmail.com> wrote:

int numBytesForDelim = sizeof(int);

You seem to be assuming that the delimiter is a simple fixed-width int.

 

coded_output->WriteVarint32(snap1.ByteSize());

But you are actually writing a varint32, which can be anywhere between 1 and 5 bytes depending on the value.

Use CodedOutputStream::Varint32Size() to compute the number of bytes needed to encode a particular value.

[Evan Jones]

On Oct 20, 2010, at 2:13 , Kenton Varda wrote:
> But you are actually writing a varint32, which can be anywhere
> between 1 and 5 bytes depending on the value.
>
> Use CodedOutputStream::Varint32Size() to compute the number of bytes
> needed to encode a particular value.

This has the advantage that you can allocate a buffer of exactly the
right size, rather than adding 100 as an estimate. However, you can
also find the final size after all the writes with
CodedOutputStream::ByteCount()

You should not need to do any byte swapping if you are serializing and
deserializing integers using the protobuf API: it handles any required
byte swapping for you.

Evan

--
Evan Jones
http://evanjones.ca/

[Paul]

ok thanks a lot guys! both ByteCount() and VarintSize32() worked for
me!

Paul

[Anthony Seo]

So how do you change the code:

ZeroCopyOutputStream* raw_output = new ArrayOutputStream(snap_buf2, 

snap1.ByteSize()+numBytesForDelim); //Here?

            CodedOutputStream* coded_output = new 
CodedOutputStream(raw_output); 

coded_output->WriteVarint32(snap1.ByteSize()); //Here?
snap1.SerializeToCodedStream(coded_output); 

Anthony