Java and C++ exchange via socket

[yorick]

Hello,

I installed protobuf 2.4.1 on Ubuntu 10.04 64bits. I executed the
example with C++ and Java and it worked fine.

I'm now trying to write my own program, which is composed of two
parts. I want to exchange a set of data (a message containing two
fields: phi and distance) between a C++ (server) program and a Java
(client) program.

The C++ part which is the server opens a TCP/IP socket and waits for
the Java part to send data. The Java part opens a socket on the same
port/address as the C++ part and sends data.

The socket seems to work, but I get the following error :

C++ part :
Start !
recv size:4
recv size:21
recv size:0
libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse
message of type "gpb.IRobotData" because it is missing required
fields: phi, distance
has not phi
has not distance
phi:0, distance:0
Quit !


Java part:
Start !
phi:0.5, distance:1.6
Quit !


The important parts of the code are the following :
IRobotData.proto :
----------------------------------------------------------------
package gpb;

option java_package = "gpb";
option java_outer_classname = "IRobotProto";

message IRobotData {
required double phi = 1;
required double distance = 2;
}
----------------------------------------------------------------
C++ :
----------------------------------------------------------------
#include <iostream>
#include <ostream>
#include <string>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
#include "iRobot.pb.h"

gpb::IRobotData data;
char buf[1000];
string strbuf;
ssize_t recvSize;

sock = socket(AF_INET, SOCK_STREAM, 0);

if(sock != INVALID_SOCKET)
{
sin.sin_addr.s_addr = htonl(INADDR_ANY);
sin.sin_family = AF_INET;
sin.sin_port = htons(PORT);

sock_err = bind(sock, (SOCKADDR*)&sin, recsize);

if(sock_err != SOCKET_ERROR)
{
sock_err = listen(sock, 5);

if(sock_err != SOCKET_ERROR)
{
csock = accept(sock, (SOCKADDR*)NULL, NULL);

string getData;

while( (recvSize = recv(csock, buf, sizeof(buf), 0)) > 0)
{
cout << "recv size:" << recvSize << endl;
if(recvSize < 0)
cerr << "recv() failed" << endl;
strbuf.append(buf, recvSize);
}
cout << "recv size:" << recvSize << endl;

data.ParseFromString(strbuf);
if(data.has_phi()) {
cout << "has phi" << endl;
} else {
cout << "has not phi" << endl;
}

if(data.has_distance()) {
cout << "has distance" << endl;
} else {
cout << "has not distance" << endl;
}
cout << "phi:" << data.phi() << ", distance:" <<
data.distance() << endl;
}
else
{
cerr << "listen failed" << endl;
}
}
else
{
cerr << "bind failed" << endl;
}
shutdown(csock, 2);
close(csock);
close(sock);
}
else
{
cerr << "socket failed" << endl;
}
----------------------------------------------------------------
Java :
----------------------------------------------------------------
Socket sock;
try {
sock = new Socket("localhost", 10101);
ObjectOutputStream oos = new
ObjectOutputStream(sock.getOutputStream());
CodedOutputStream cos = CodedOutputStream.newInstance(oos);


//for(int i=0 ; i<10 ; i++){
IRobotData data = getRobotData();

System.out.println("phi:"+data.getPhi()+",
distance:"+data.getDistance());
cos.writeRawVarint32(data.getSerializedSize());
data.writeTo(cos);
cos.flush();
oos.flush();

/*
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
} */

sock.close();


} catch (UnknownHostException e1) {
System.err.println("Host inconnu !");
e1.printStackTrace();
} catch (IOException e1) {
System.err.println("I/O erreur !");
e1.printStackTrace();
}
----------------------------------------------------------------

Do you have any idea why I got the error ? I might not manage
correctly the serialisation/deserialisation thing, but I don't know
how to improve it.

Thanks.

[yorick]

I may add a few things. I wrote the data that is send and received
into text files for comparison.

For the Java part, I do the following to write :
---------------------
FileOutputStream output;
try {
output = new FileOutputStream("outputJava.txt", false);
data.writeTo(output);
output.close();
}
---------------------
and get (binary to hex with vim) :
---------------------
0000000: 0900 0000 0000 00e0 3f11 9a99 9999 9999 ........?.......
0000010: f93f 0a .?.
---------------------
For the C++ part, I do the following to write :
---------------------
ofstream myfile;
myfile.open ("outputCpp.txt");
myfile << strbuf;
myfile.close();
---------------------
and get (binary to hex with vim) :
---------------------
0000000: aced 0005 7713 1209 0000 0000 0000 e03f ....w..........?
0000010: 119a 9999 9999 99f9 3f0a ........?.
---------------------

There are 6 bytes in addition when received the data ! I checked with
Wireshark, these extra bytes are sent by the Java part, but are not
part of the IRobotData message.

So I tried to remove these extra bytes and the following happens :
---------------------

Start !
recv size:4
recv size:21
recv size:0

string size:25
is NOT initialized

has not phi
has not distance
phi:0, distance:0
Quit !

---------------------
The error is gone, but the data is not better received. I added a
check for whether the IRobotData variable is initialised, and it is
not. So it appears that the message is still not correctly handled...

[Christopher Head]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

On Wed, 5 Oct 2011 02:48:39 -0700 (PDT)
yorick <yorick...@gmail.com> wrote:

[snip]

> ----------------------------------------------------------------
> Java :
> ----------------------------------------------------------------
> Socket sock;
> try {
> sock = new Socket("localhost", 10101);
> ObjectOutputStream oos = new
> ObjectOutputStream(sock.getOutputStream());
> CodedOutputStream cos =
> CodedOutputStream.newInstance(oos);

Why are you using an ObjectOutputStream? You're not using Java
serialization for anything, so don't do that. It's probably adding
headers itself which your C++ code doesn't expect. For that matter,
there doesn't seem to be much reason to even create a CodedOutputStream
in this case, since AbstractMessageLite.writeTo() accepts an arbitrary
java.io.OutputStream, such as the one returned from
sock.getOutputStream().

>
>
> //for(int i=0 ; i<10 ; i++){
> IRobotData data = getRobotData();
>
> System.out.println("phi:"+data.getPhi()+",
> distance:"+data.getDistance());
> cos.writeRawVarint32(data.getSerializedSize());
> data.writeTo(cos);
> cos.flush();
> oos.flush();

Here, you're writing a Varint32 to the stream before writing the data
object itself, but I don't see anything in your C++ code that reads the
varint before reading the data object.

Chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iEYEAREDAAYFAk6NWSoACgkQXUF6hOTGP7fW+ACffCrOhAiZ42hmX98OSbdYwIQK
1q4AniBG7I9nGOoJRnBMyop4aZ9b32AR
=lmqD
-----END PGP SIGNATURE-----

[yorick]

After reading http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming
that we suggested to me, I added a CodedInputStream in the C++ part.

I have now for the Java (this is unchanged) :
----------------------------------

IRobotData data = getRobotData();
System.out.println("phi:"+data.getPhi()+",
distance:"+data.getDistance());

int serDataSize = data.getSerializedSize();
System.out.println("serDataSize:"+serDataSize);
cos.writeRawVarint32(serDataSize);
data.writeTo(cos);
cos.flush();
oos.flush();
----------------------------------
and for the C++ :
----------------------------------
// Get size
recvSize = recv(csock, buf, sizeof(buf), 0);

if(recvSize < 0)
cerr << "recv() failed" << endl;
cout << "recv size:" << recvSize << endl;

google::protobuf::io::CodedInputStream* cis = new
google::protobuf::io::CodedInputStream(buf, recvSize);
buf[recvSize] = '\0';

unsigned int size;
cis->ReadVarint32(&size);
cout << "size:" << size << endl;
delete cis;

// Get data
recvSize = recv(csock, buf, sizeof(buf), 0);

if(recvSize < 0)
cerr << "recv() failed" << endl;

buf[recvSize] = '\0';

cout << "recv size:" << recvSize << endl;

cis = new google::protobuf::io::CodedInputStream(buf, recvSize);
cis->ReadString(&strbuf, recvSize);
cout << "string size:" << strbuf.size() << endl;
delete cis;
data.ParseFromString(strbuf);
if(data.IsInitialized()) {
cout << "is initialized" << endl;
} else {
cout << "is NOT initialized" << endl;

}

if(data.has_phi()) {
cout << "has phi" << endl;
} else {
cout << "has not phi" << endl;
}

if(data.has_distance()) {
cout << "has distance" << endl;
} else {
cout << "has not distance" << endl;
}
cout << "phi:" << data.phi() << ", distance:" << data.distance() <<
endl;

----------------------------------
The second "cis = new google::protobuf::io::CodedInputStream(buf,
recvSize);" is not totally correct as I should use the size in
variable "size", but as the "size" variable is not correct I kept
"recvSize" for now.

The output is the following for Java :
----------------------------------

Start !
phi:0.5, distance:1.6

serDataSize:18
Quit !
----------------------------------
and for C++ :
----------------------------------

Start !
recv size:4

size:13996
recv size:21
string size:21
is NOT initialized

has not phi
has not distance
phi:0, distance:0
Quit !

----------------------------------
So the size value is not the same... I'm suspecting that the Java
"CodedOutputStream::writeRawVarint32" and the C++
"CodedInputStream::ReadVarint32" do not encode int in the same way...

What should I do to get the same int value on both sides ?

Thanks.

On 6 oct, 09:30, Christopher Head <hea...@gmail.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: RIPEMD160
>
> On Wed, 5 Oct 2011 02:48:39 -0700 (PDT)
>

[Jason Hsueh]

Can you provide a wrapper OutputStream in your Java code so you can intercept the exact bytes that are written to the stream? It wasn't clear if in your last attempt you had removed the ObjectOutputStream issue or not. I would suggest first making sure that the bytes you receive on the C++ side are the same as what is getting sent to the socket in Java; if you also know the size that you are writing you can verify that the bytes written to the stream matches the varint-encoded value.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[Christopher Head]

On Thu, 6 Oct 2011 01:00:37 -0700 (PDT)
yorick <yorick...@gmail.com> wrote:

> I have now for the Java (this is unchanged) :
> ----------------------------------
> IRobotData data = getRobotData();
> System.out.println("phi:"+data.getPhi()+",
> distance:"+data.getDistance());
> int serDataSize = data.getSerializedSize();
> System.out.println("serDataSize:"+serDataSize);
> cos.writeRawVarint32(serDataSize);
> data.writeTo(cos);
> cos.flush();
> oos.flush();

Based on the existence of the line "oos.flush()", you're still using
ObjectOutputStream. Why? Get rid of it!

Also, you seem to be confused about the nature of TCP. In your Java
code, you're sending a varint32 followed by an encoded IRobotData. In
your C++, you're only calling recv() once, then building a
CodedInputStream over it, trying to parse a varint32, and then calling
recv() again and expecting to see the IRobotData message at that point.
TCP is a byte stream. It is NOT a message stream. You will much more
likely see recv() return the varint32 and the IRobotData in a single
call. It's possible, though, depending on the network, that it could do
anything: theoretically, it could return a single byte on each call,
though that's highly unlikely. You can't just expect to send a block of
bytes, e.g., a varint32, and expect the same block to come out the
other side. Notice that your C++ code is reading 4 bytes and then 21
bytes, while your Java code is (except for the OOS bug) writing a varint
plus 18 bytes. The first four bytes are probably from the
ObjectOutputStream, the next 3 might be the varint, or a mixture of the
ObjectOutputStream header and the varint, and the last 18 are probably
the encoded data.

Your previous code was OK because you read everything until EOF into an
std::string, then parsed from there. That works if you're OK waiting
until the connection is closed to parse things. If you want to parse
data as it arrives, you need to loop until enough data is received,
then parse it. It might be easier to use a fixed-size header rather
than a varint to mark the length of an individual message (a uint32
would be suitable). Starting from the beginning, you would then loop,
accumulating data, until you had 4 bytes, then turn those into the
uint32 indicating how big the rest of the message was. You would then
loop again, accumulating data, until you had that many bytes. Then you
could parse the message. If you call recv() with a separate buffer,
like you're doing here with "buf" and "sizeof(buf)", you may find it
returns more data than you actually expect, e.g., while you're
receiving the 4 bytes that make up your uint32, you might actually get,
say, 20 bytes. You need to keep the remaining 16, because they're part
of the following message. The alternative is to only receive as much as
you're willing to deal with in one go, so call recv() with 4 for your
uint32, then if you receive 3 bytes call recv() again with 1, until you
have your uint32, then similarly for receiving the full message.

Finally, there's no reason to NUL-terminate your buffers. You construct
a CodedInputStream and pass it recvSize, so the NUL is meaningless
(even if you didn't it would be meaningless, as there can be NULs in
the middle of encoded data).

Chris

[yorick]

Thanks for your response. I remove the ObjectOutputStream and I
changed the way of reading the socket according what you said and it
works !

I can now exchange data between Java and C++. Thank you ! :)

On 6 oct, 23:21, Christopher Head <hea...@gmail.com> wrote:
> On Thu, 6 Oct 2011 01:00:37 -0700 (PDT)
>