Java -using GPB -which type of stream should I be using over sockets?
248 views
Skip to first unread message
Nigel Pickard
Dec 2, 2009, 10:47:44 AM12/2/09
to Protocol Buffers
OK, so far I like GPB. However, I have a very simple question, but
one which I can't find an answer for:
I have created a test Java application that uses ObjectInputStream and
ObjectOutputStream over sockets. No problems, it works! I use my GPB
class writeTo and parseFrom to send and reconstitute my GPB class
using the ObjectInput and Output Streams....
But: isn't GPB allowing for serialization using bytes? Are Object I/O
Streams the best streams to use? Should I be using something else
like Data I/O streams? E.g. from JavaDocs, "An ObjectOutputStream
writes primitive data types and graphs of Java objects to an
OutputStream" -is there a better, more efficient I/O stream I should
be using?
My concern is that when it comes to attempting to open a socket to a C+
+ app, I assume the correct I/O stream type should be used.
If anyone has a working example of best practice using Data I/O
streams or a recommended stream type, that would be extremely useful
for me (even better if someone has a simple Java to C++ using sockets
example).
Thanks.
one which I can't find an answer for:
I have created a test Java application that uses ObjectInputStream and
ObjectOutputStream over sockets. No problems, it works! I use my GPB
class writeTo and parseFrom to send and reconstitute my GPB class
using the ObjectInput and Output Streams....
But: isn't GPB allowing for serialization using bytes? Are Object I/O
Streams the best streams to use? Should I be using something else
like Data I/O streams? E.g. from JavaDocs, "An ObjectOutputStream
writes primitive data types and graphs of Java objects to an
OutputStream" -is there a better, more efficient I/O stream I should
be using?
My concern is that when it comes to attempting to open a socket to a C+
+ app, I assume the correct I/O stream type should be used.
If anyone has a working example of best practice using Data I/O
streams or a recommended stream type, that would be extremely useful
for me (even better if someone has a simple Java to C++ using sockets
example).
Thanks.
Adam Vartanian
Dec 2, 2009, 1:14:43 PM12/2/09
to Nigel Pickard, Protocol Buffers
> I have created a test Java application that uses ObjectInputStream and
> ObjectOutputStream over sockets. No problems, it works! I use my GPB
> class writeTo and parseFrom to send and reconstitute my GPB class
> using the ObjectInput and Output Streams....
>
> But: isn't GPB allowing for serialization using bytes? Are Object I/O
> Streams the best streams to use? Should I be using something else
> like Data I/O streams? E.g. from JavaDocs, "An ObjectOutputStream
> writes primitive data types and graphs of Java objects to an
> OutputStream" -is there a better, more efficient I/O stream I should
> be using?
Protocol buffer code doesn't use object streams, so when you pass one
> ObjectOutputStream over sockets. No problems, it works! I use my GPB
> class writeTo and parseFrom to send and reconstitute my GPB class
> using the ObjectInput and Output Streams....
>
> But: isn't GPB allowing for serialization using bytes? Are Object I/O
> Streams the best streams to use? Should I be using something else
> like Data I/O streams? E.g. from JavaDocs, "An ObjectOutputStream
> writes primitive data types and graphs of Java objects to an
> OutputStream" -is there a better, more efficient I/O stream I should
> be using?
to its writeTo/parseFrom methods, you're actually passing the object
stream as a regular OutputStream or InputStream, ie, your code is
already serializing to and from bytes. If you'd rather be explicit
about it, you can just switch your stream's type from
ObjectOutputStream to OutputStream (similarly with Input) and it
should continue to work the same. All that should require is instead
of doing something like
ObjectOutputStream output = new ObjectOutputStream(socket.getOutputStream());
message.writeTo(output);
you do something like
OutputStream output = socket.getOutputStream();
message.writeTo(output);
For interacting with C++, you should be able to use the standard C++
API for reading in messages, since the data is already on the wire in
byte format.
- Adam
Nigel Pickard
Dec 2, 2009, 3:13:38 PM12/2/09
to Protocol Buffers
Thanks for the reply Adam -I did a simple sub into my code with your
suggestion, but it doesn't work. However, now it's made me think of
more questions.....
If protocol buffers doesn't use object I/O streams, which streams does
it use? The argument in Java simply asks for an OutputStream
argument, and of course that could mean File, ByteArray, Object, Data,
etc....
And perhaps more importantly, will the choice of Java stream types
impact how GPB objects may be reconstituted in C++? At the simplest
level I assume this can be done:
I can create a GPB class called "GPBCar".
I have an outputstream (of some type, not sure!) over a socket in Java
I want to write to.
After creating my GPBCar object, I use the GPB "writeTo(output)"
method on the Java side.
My C++ app takes some form of this byte stream and reconstitutes my
GPBCar instance by using the equivalent of the "parseFrom" method from
the GPBCar.
My understanding is that I have converted my GPBCar to a series of
bytes that represent that GPBCar instance. Now I have a C++ app at the
other end of the socket connection. But will the series of bytes be
in the correct order (endianness?). Will the outputstream type be
particular to Java, or will I encounter problems by using certain
specific Java outputstreams? Should I only be using the
CodedDataOutputStream provided by GPB? (I note that there is
implementations of this stream in both the C++ and Java GPB
implementations).
I should add I've tried to find answers to these questions on the
board and web, but with no luck. Again, if anyone has a simple
example of how they used GPB to communicate over a socket from a Java
to a C++ app, I'd love to see it (or just point me in the right
direction).
suggestion, but it doesn't work. However, now it's made me think of
more questions.....
If protocol buffers doesn't use object I/O streams, which streams does
it use? The argument in Java simply asks for an OutputStream
argument, and of course that could mean File, ByteArray, Object, Data,
etc....
And perhaps more importantly, will the choice of Java stream types
impact how GPB objects may be reconstituted in C++? At the simplest
level I assume this can be done:
I can create a GPB class called "GPBCar".
I have an outputstream (of some type, not sure!) over a socket in Java
I want to write to.
After creating my GPBCar object, I use the GPB "writeTo(output)"
method on the Java side.
My C++ app takes some form of this byte stream and reconstitutes my
GPBCar instance by using the equivalent of the "parseFrom" method from
the GPBCar.
My understanding is that I have converted my GPBCar to a series of
bytes that represent that GPBCar instance. Now I have a C++ app at the
other end of the socket connection. But will the series of bytes be
in the correct order (endianness?). Will the outputstream type be
particular to Java, or will I encounter problems by using certain
specific Java outputstreams? Should I only be using the
CodedDataOutputStream provided by GPB? (I note that there is
implementations of this stream in both the C++ and Java GPB
implementations).
I should add I've tried to find answers to these questions on the
board and web, but with no luck. Again, if anyone has a simple
example of how they used GPB to communicate over a socket from a Java
to a C++ app, I'd love to see it (or just point me in the right
direction).
Jason Hsueh
Dec 2, 2009, 1:20:34 PM12/2/09
to Nigel Pickard, Protocol Buffers
Protocol buffers implement their own serialization routines: the writeTo(OutputStream output) method will write the binary format to the OutputStream. From a socket, just do:
MyProtoBuf pb = new MyProtoBuf;
...
pb.writeTo(socket.getOutputStream());
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Kenton Varda
Dec 2, 2009, 3:25:34 PM12/2/09
to Nigel Pickard, Protocol Buffers
It doesn't matter what kind of stream you use. Protobufs just read and write bytes. You'll get the exact same result with any kind of stream.
Kenton Varda
Dec 2, 2009, 3:27:18 PM12/2/09
to Nigel Pickard, Protocol Buffers
Well ok, using gzip streams would obviously give you something different (the bytes would come out gzipped). But none of the stream types you listed make a difference.
Jason Hsueh
Dec 2, 2009, 3:29:19 PM12/2/09
to Nigel Pickard, Protocol Buffers
On Wed, Dec 2, 2009 at 12:13 PM, Nigel Pickard <pickar...@gmail.com> wrote:
Thanks for the reply Adam -I did a simple sub into my code with your
suggestion, but it doesn't work. However, now it's made me think of
more questions.....
If protocol buffers doesn't use object I/O streams, which streams does
it use? The argument in Java simply asks for an OutputStream
argument, and of course that could mean File, ByteArray, Object, Data,
etc....
The serialization routine just writes raw bytes in the binary format. You can use a FileOutputStream, a ByteArrayOutputStream, or anything else that allows you to write raw bytes. You don't need an ObjectOutputStream, since that is intended to support Java serialization of classes that implement the Serializable interface - protocol buffers are not such classes. Which OutputStream you want depends on where you want to write the data to. In the case of the socket, Adam's code snippet of socket.getOutputStream() should be sufficient.
And perhaps more importantly, will the choice of Java stream types
impact how GPB objects may be reconstituted in C++? At the simplest
level I assume this can be done:
I can create a GPB class called "GPBCar".
I have an outputstream (of some type, not sure!) over a socket in Java
I want to write to.
After creating my GPBCar object, I use the GPB "writeTo(output)"
method on the Java side.
My C++ app takes some form of this byte stream and reconstitutes my
GPBCar instance by using the equivalent of the "parseFrom" method from
the GPBCar.
My understanding is that I have converted my GPBCar to a series of
bytes that represent that GPBCar instance. Now I have a C++ app at the
other end of the socket connection. But will the series of bytes be
in the correct order (endianness?). Will the outputstream type be
particular to Java, or will I encounter problems by using certain
specific Java outputstreams? Should I only be using the
CodedDataOutputStream provided by GPB? (I note that there is
implementations of this stream in both the C++ and Java GPB
implementations).
Yes, the binary format is designed to work across languages and platforms. In your C++ code, just open up a socket, and use Message::ParseFromFileDescriptor()
I should add I've tried to find answers to these questions on the
board and web, but with no luck. Again, if anyone has a simple
example of how they used GPB to communicate over a socket from a Java
to a C++ app, I'd love to see it (or just point me in the right
direction).
On Dec 2, 1:14 pm, Adam Vartanian <flo...@google.com> wrote:
>
> Protocol buffer code doesn't use object streams, so when you pass one
> to its writeTo/parseFrom methods, you're actually passing the object
> stream as a regular OutputStream or InputStream, ie, your code is
> already serializing to and from bytes. If you'd rather be explicit
> about it, you can just switch your stream's type from
> ObjectOutputStream to OutputStream (similarly with Input) and it
> should continue to work the same. All that should require is instead
> of doing something like
>
> ObjectOutputStream output = new ObjectOutputStream(socket.getOutputStream());
> message.writeTo(output);
>
> you do something like
>
> OutputStream output = socket.getOutputStream();
> message.writeTo(output);
>
> For interacting with C++, you should be able to use the standard C++
> API for reading in messages, since the data is already on the wire in
> byte format.
>
> - Adam
Adam Vartanian
Dec 2, 2009, 3:30:13 PM12/2/09
to Nigel Pickard, Protocol Buffers
> If protocol buffers doesn't use object I/O streams, which streams does
> it use? The argument in Java simply asks for an OutputStream
> argument, and of course that could mean File, ByteArray, Object, Data,
> etc....
I mean that it only depends on the methods in OutputStream. You can
> it use? The argument in Java simply asks for an OutputStream
> argument, and of course that could mean File, ByteArray, Object, Data,
> etc....
provide anything that implements OutputStream (including
ObjectOutputStream, if you wish, but it won't encode things using
ObjectOutputStream's additional methods for encoding Java objects, it
will only write bytes to the stream).
> And perhaps more importantly, will the choice of Java stream types
> impact how GPB objects may be reconstituted in C++? At the simplest
> level I assume this can be done:
>
> I can create a GPB class called "GPBCar".
> I have an outputstream (of some type, not sure!) over a socket in Java
> I want to write to.
> After creating my GPBCar object, I use the GPB "writeTo(output)"
> method on the Java side.
> My C++ app takes some form of this byte stream and reconstitutes my
> GPBCar instance by using the equivalent of the "parseFrom" method from
> the GPBCar.
> My understanding is that I have converted my GPBCar to a series of
> bytes that represent that GPBCar instance. Now I have a C++ app at the
> other end of the socket connection. But will the series of bytes be
> in the correct order (endianness?). Will the outputstream type be
> particular to Java, or will I encounter problems by using certain
> specific Java outputstreams? Should I only be using the
> CodedDataOutputStream provided by GPB? (I note that there is
> implementations of this stream in both the C++ and Java GPB
> implementations).
specifies the byte order of everything (and the protocol buffer code
properly implements that). It could get screwed up if some other
system on either end is attempting to modify the byte order, but in
general that shouldn't be the case.
What kind of OutputStream you use should have no effect, unless it's
an OutputStream that transforms the output in some way. You don't
have to use CodedOutputStream.
- Adam
Nigel Pickard
Dec 2, 2009, 3:31:03 PM12/2/09
to Protocol Buffers
Jason -do you mean any Java output stream type will work (as all Java
IO stream types ultimately use bytes), and that the byte stream
produced will have the bytes put in an accepted standard order by the
writeTo method? In other words, using the writeTo method for *any*
Java output stream type will produce a series of bytes that will
*always* be able to be read using the C++ equivalent of parseFrom on
the C++ side?
On Dec 2, 1:20 pm, Jason Hsueh <jas...@google.com> wrote:
> Protocol buffers implement their own serialization routines: the
> writeTo(OutputStream output) method will write the binary format to the
> OutputStream. From a socket, just do:
>
> MyProtoBuf pb = new MyProtoBuf;
> ...
> pb.writeTo(socket.getOutputStream());
>
> > protobuf+u...@googlegroups.com<protobuf%2Bunsu...@googlegroups.com>
> > .
IO stream types ultimately use bytes), and that the byte stream
produced will have the bytes put in an accepted standard order by the
writeTo method? In other words, using the writeTo method for *any*
Java output stream type will produce a series of bytes that will
*always* be able to be read using the C++ equivalent of parseFrom on
the C++ side?
On Dec 2, 1:20 pm, Jason Hsueh <jas...@google.com> wrote:
> Protocol buffers implement their own serialization routines: the
> writeTo(OutputStream output) method will write the binary format to the
> OutputStream. From a socket, just do:
>
> MyProtoBuf pb = new MyProtoBuf;
> ...
> pb.writeTo(socket.getOutputStream());
>
> > .
Nigel Pickard
Dec 2, 2009, 3:35:01 PM12/2/09
to Protocol Buffers
Aaaah I think the fog is lifting..... thanks!
So am I correct in thinking that byte order is taken in to account by
the GPB writeTo and parseFrom methods, no matter whether it's Java, C+
+, etc?
So am I correct in thinking that byte order is taken in to account by
the GPB writeTo and parseFrom methods, no matter whether it's Java, C+
+, etc?
Henner Zeller
Dec 2, 2009, 3:36:45 PM12/2/09
to Nigel Pickard, Protocol Buffers
On Wed, Dec 2, 2009 at 12:31, Nigel Pickard <pickar...@gmail.com> wrote:
> Jason -do you mean any Java output stream type will work (as all Java
> IO stream types ultimately use bytes), and that the byte stream
> produced will have the bytes put in an accepted standard order by the
> writeTo method? In other words, using the writeTo method for *any*
> Java output stream type will produce a series of bytes that will
> *always* be able to be read using the C++ equivalent of parseFrom on
> the C++ side?
Yes. Protocol Buffers are designed to be platform and language
> Jason -do you mean any Java output stream type will work (as all Java
> IO stream types ultimately use bytes), and that the byte stream
> produced will have the bytes put in an accepted standard order by the
> writeTo method? In other words, using the writeTo method for *any*
> Java output stream type will produce a series of bytes that will
> *always* be able to be read using the C++ equivalent of parseFrom on
> the C++ side?
independent so given a transparent way to transport raw bytes (any
output stream), serialization and deserialization between any
combination works as you expect. Since you asked: raw types such as
integers are serialized in a proto buffer specific way with a
consistent byte order independent of the platform.
-h
Nigel Pickard
Dec 2, 2009, 3:41:43 PM12/2/09
to Protocol Buffers
Adam, Jason, Kenton and Henner, thanks and much appreciated!
Nigel Pickard
Dec 2, 2009, 4:11:36 PM12/2/09
to Protocol Buffers
Oh, wait.... So I have a test Java app where one thread is running as
a server, one as a client.
I've been sending a GPB defined class instance between them, no
problem when I use Object I/O streams (e.g. ObjectOutputStream on the
server, ObjectInputStream on the client. I've been calling writeTo
and parseFrom on the GBP object successfully.
If I understand the previous posts correctly it should not make a
difference which outputstream or inputstream type I use, right? The
stream of whatever type it is should just be a series of bytes, able
to be parsed by my GBP object.... But if I change my
ObjectOutputStream to DataOutputStream on my client (so I'm leaving my
server still using ObjectInputStream) I get the following error:
"com.google.protobuf.InvalidProtocolBufferException: Protocol message
end-group tag did not match expected tag."
Wouldn't this suggest there is a difference (at least in Java)
concerning which inputstream you use?
a server, one as a client.
I've been sending a GPB defined class instance between them, no
problem when I use Object I/O streams (e.g. ObjectOutputStream on the
server, ObjectInputStream on the client. I've been calling writeTo
and parseFrom on the GBP object successfully.
If I understand the previous posts correctly it should not make a
difference which outputstream or inputstream type I use, right? The
stream of whatever type it is should just be a series of bytes, able
to be parsed by my GBP object.... But if I change my
ObjectOutputStream to DataOutputStream on my client (so I'm leaving my
server still using ObjectInputStream) I get the following error:
"com.google.protobuf.InvalidProtocolBufferException: Protocol message
end-group tag did not match expected tag."
Wouldn't this suggest there is a difference (at least in Java)
concerning which inputstream you use?
Henner Zeller
Dec 2, 2009, 4:17:28 PM12/2/09
to Nigel Pickard, Protocol Buffers
Are your ObjectOutputStreams transparent ? Or do they prepend/append
things to the data ?
things to the data ?
Nigel Pickard
Dec 2, 2009, 6:21:55 PM12/2/09
to Protocol Buffers
Henner:
I'm not sure what you mean by the datastream being transparent, but
basically I'm sending a GPB instance from the server to the client.
Initially I used ObjectOutputStream on the server, ObjectInputStream
on the client. I then changed the client to use DataInputStream
(apologies, my previous post stated output when I meant input).
I'm just sending one instance each run for simplicity (my object is a
simple one) -the exact same GPB instance that worked when I used
Object IO streams on the client and server that worked -and then
stopping. The only thing that has changed has been the input stream
type on the client.
Basically, on the client side I called the GPB object's "parseFrom"
method with the DataInputStream.as an argument and expected it to work
but got that error message.
On Dec 2, 4:17 pm, Henner Zeller <henner.zel...@googlemail.com> wrote:
> Are your ObjectOutputStreams transparent ? Or do they prepend/append
> things to the data ?
>
I'm not sure what you mean by the datastream being transparent, but
basically I'm sending a GPB instance from the server to the client.
Initially I used ObjectOutputStream on the server, ObjectInputStream
on the client. I then changed the client to use DataInputStream
(apologies, my previous post stated output when I meant input).
I'm just sending one instance each run for simplicity (my object is a
simple one) -the exact same GPB instance that worked when I used
Object IO streams on the client and server that worked -and then
stopping. The only thing that has changed has been the input stream
type on the client.
Basically, on the client side I called the GPB object's "parseFrom"
method with the DataInputStream.as an argument and expected it to work
but got that error message.
On Dec 2, 4:17 pm, Henner Zeller <henner.zel...@googlemail.com> wrote:
> Are your ObjectOutputStreams transparent ? Or do they prepend/append
> things to the data ?
>
Henner Zeller
Dec 2, 2009, 6:31:48 PM12/2/09
to Nigel Pickard, Protocol Buffers
Hi,
you see a difference then this might be because the ObjectOutputStream
might add something to the data you provide, such as writing
delimiters or something. So I guess the simple write(byte[]) will not
actually write the content of the arrays but as well the length or
something (this is a wild guess - haven't looked at the sources, but
it would explain what you see).
Why do you use the ObjectOutputStream anyway instead of the simpler
OutputStream ?
On Wed, Dec 2, 2009 at 15:21, Nigel Pickard <pickar...@gmail.com> wrote:
> Henner:
>
> I'm not sure what you mean by the datastream being transparent, but
> basically I'm sending a GPB instance from the server to the client.
> Initially I used ObjectOutputStream on the server, ObjectInputStream
> on the client. I then changed the client to use DataInputStream
> (apologies, my previous post stated output when I meant input).
transparent as in: the same data is written out that you put in. If
> Henner:
>
> I'm not sure what you mean by the datastream being transparent, but
> basically I'm sending a GPB instance from the server to the client.
> Initially I used ObjectOutputStream on the server, ObjectInputStream
> on the client. I then changed the client to use DataInputStream
> (apologies, my previous post stated output when I meant input).
you see a difference then this might be because the ObjectOutputStream
might add something to the data you provide, such as writing
delimiters or something. So I guess the simple write(byte[]) will not
actually write the content of the arrays but as well the length or
something (this is a wild guess - haven't looked at the sources, but
it would explain what you see).
Why do you use the ObjectOutputStream anyway instead of the simpler
OutputStream ?
Nigel Pickard
Dec 2, 2009, 7:15:02 PM12/2/09
to Protocol Buffers
When the Java test app works, I'd say the IO stream is transparent. I
use the same simple object instance each time (have a hard coded
method that simply returns the same object so it's identical), and I
simply call the toString of the instance to see what it is (before I
call writeTo on the server side, and after I call parseFrom on the
client side). When it doesn't work, I don't even get to parsing the
input stream.....
Concerning the IO stream types -well, I was just experimenting with
various IO stream types, and seeing if there was a difference. I
think I tried the simpler InputStream and OutputStream but couldn't
get it to work. I know for sure I could get File, Data and Object IO
streams to work without trouble (well, assuming I used the same IO
types on both client and server).
I keep coming around to the fact that it looks like a specific data
stream type in Java does do something to the byte stream (perhaps
adding extra bytes for a header or some sort of meta information like
you mentioned? And in Java you might need the same input stream to
read it properly? I dunno -but again from my original post, I want to
make sure if I do get that series of bytes over to a C++ app, it's in
a form that can be handled.).
Anyway, I'm going to experiment further to see what I get. If I get a
simple example working, I'll post it as I'm sure I can't be the first
person with this dilemma.
On Dec 2, 6:31 pm, Henner Zeller <henner.zel...@googlemail.com> wrote:
> Hi,
use the same simple object instance each time (have a hard coded
method that simply returns the same object so it's identical), and I
simply call the toString of the instance to see what it is (before I
call writeTo on the server side, and after I call parseFrom on the
client side). When it doesn't work, I don't even get to parsing the
input stream.....
Concerning the IO stream types -well, I was just experimenting with
various IO stream types, and seeing if there was a difference. I
think I tried the simpler InputStream and OutputStream but couldn't
get it to work. I know for sure I could get File, Data and Object IO
streams to work without trouble (well, assuming I used the same IO
types on both client and server).
I keep coming around to the fact that it looks like a specific data
stream type in Java does do something to the byte stream (perhaps
adding extra bytes for a header or some sort of meta information like
you mentioned? And in Java you might need the same input stream to
read it properly? I dunno -but again from my original post, I want to
make sure if I do get that series of bytes over to a C++ app, it's in
a form that can be handled.).
Anyway, I'm going to experiment further to see what I get. If I get a
simple example working, I'll post it as I'm sure I can't be the first
person with this dilemma.
On Dec 2, 6:31 pm, Henner Zeller <henner.zel...@googlemail.com> wrote:
> Hi,
Kenton Varda
Dec 2, 2009, 7:28:44 PM12/2/09
to Nigel Pickard, Protocol Buffers
There is no reason to use ObjectInputStream/ObjectOutputStream with protocol buffers.
Jason Hsueh
Dec 2, 2009, 10:46:38 PM12/2/09
to Nigel Pickard, Protocol Buffers
On Wed, Dec 2, 2009 at 4:15 PM, Nigel Pickard <pickar...@gmail.com> wrote:
When the Java test app works, I'd say the IO stream is transparent. I
use the same simple object instance each time (have a hard coded
method that simply returns the same object so it's identical), and I
simply call the toString of the instance to see what it is (before I
call writeTo on the server side, and after I call parseFrom on the
client side). When it doesn't work, I don't even get to parsing the
input stream.....
Concerning the IO stream types -well, I was just experimenting with
various IO stream types, and seeing if there was a difference. I
think I tried the simpler InputStream and OutputStream but couldn't
get it to work. I know for sure I could get File, Data and Object IO
streams to work without trouble (well, assuming I used the same IO
types on both client and server).
Well, InputStream and OutputStream are abstract classes. I'm not sure how you instantiated them in your tests to determine that they do not work (unless the problem you got was that you could not create an abstract OutputStream object, which has nothing to do with protobuf code). You need to provide a concrete implementation of OutputStream. writeTo() only cares about writing raw bytes to the OutputStream, so the additional methods like writeInt() and writeObject() provided by DataOutputStream and ObjectOutputStream, respectively, do not matter. The only thing that would matter is if the bytes that were passed to the write() method were not the actual bytes written. As Kenton pointed out this is the case for, say, a gzipped output stream. You would obviously need to have your reader be aware that it is reading gzipped data, rather than raw bytes. Otherwise, it doesn't matter what output stream implementation you choose, so long as it writes the bytes that are passed to the write() method.
If message.writeTo(socket.getOutputStream()); does not work for you, please post a small reproducible example.
I keep coming around to the fact that it looks like a specific data
stream type in Java does do something to the byte stream (perhaps
adding extra bytes for a header or some sort of meta information like
you mentioned? And in Java you might need the same input stream to
read it properly? I dunno -but again from my original post, I want to
make sure if I do get that series of bytes over to a C++ app, it's in
a form that can be handled.).
Anyway, I'm going to experiment further to see what I get. If I get a
simple example working, I'll post it as I'm sure I can't be the first
person with this dilemma.
On Dec 2, 6:31 pm, Henner Zeller <henner.zel...@googlemail.com> wrote:
> Hi,
>
> transparent as in: the same data is written out that you put in. If
> you see a difference then this might be because the ObjectOutputStream
> might add something to the data you provide, such as writing
> delimiters or something. So I guess the simple write(byte[]) will not
> actually write the content of the arrays but as well the length or
> something (this is a wild guess - haven't looked at the sources, but
> it would explain what you see).
>
> Why do you use the ObjectOutputStream anyway instead of the simpler
> OutputStream ?
>
Nigel Pickard
Dec 3, 2009, 8:42:33 AM12/3/09
to Protocol Buffers
Jason:
You know that's what I thought too about the InputStream/OutputStream
after reading the Javadoc. I'm using Eclipse Galileo with Java
1.6.0.17 -for some reason it let me compile it and run it, creating an
instance -I figured that was probably one of the problems.
I'll post some code later.
> > protobuf+u...@googlegroups.com<protobuf%2Bunsu...@googlegroups.com>
> > .
You know that's what I thought too about the InputStream/OutputStream
after reading the Javadoc. I'm using Eclipse Galileo with Java
1.6.0.17 -for some reason it let me compile it and run it, creating an
instance -I figured that was probably one of the problems.
I'll post some code later.
> > .
Nigel Pickard
Dec 3, 2009, 10:11:14 AM12/3/09
to Protocol Buffers
Here's my sample code. In this code, everything works for me -however
when I change the GPBClient to use Data IO streams (just comment out
the Object IO references and uncomment the Data IO references in
GPBClient), that's when I get the
"com.google.protobuf.InvalidProtocolBufferException: Protocol message
end-group tag did not match expected tag." error.
I call the 2 threads to use it:
Thread server = new Thread(new GPBServer());
server.start();
Thread client = new Thread(new GPBClient());
client.start();
2 classes -GPBServer and GPBClient:
public class GPBServer implements Runnable{
ServerSocket serverSocket;
Socket socket = null;
ObjectOutputStream objectOutputStream;
ObjectInputStream objectInputStream;
public GPBServer(){}
private void runThis(){
try{
//create server socket
serverSocket = new ServerSocket(2004, 10);
//get connection
System.out.println("server: getting connection");
socket = serverSocket.accept();
System.out.println("server: Connection received from " +
socket.getInetAddress().getHostName());
//create IO streams
objectOutputStream = new ObjectOutputStream(socket.getOutputStream
());
objectOutputStream.flush();
objectInputStream = new ObjectInputStream(socket.getInputStream());
//send my GPB object
sendMyGPBObject();
}catch(IOException ioException){
ioException.printStackTrace();
}finally{
//Close connections and socket
try{
objectInputStream.close();
objectOutputStream.close();
serverSocket.close();
System.out.println("server: Closed all connection and streams.");
}catch(IOException ioException){
ioException.printStackTrace();
}
}
}
public void sendMyGPBObject(){
CarObject.Car car = null;
try{
//used a factory to create my car instance
using GPB builder stuff (did all GPB compilation successfully)
car = CarFactory.instance.createCar();
//print it out to see what its like (I used a wrapper to get useful
stuff to tostring but
//for sake of brevityhaven't included it here)
System.out.println("server: The original server side object is \n"
+ car.toString());
car.writeTo(objectOutputStream);
objectOutputStream.flush();
}catch(IOException e){
System.out.println("serverside: Error is " + e);
}
}
@Override
public void run() {
while(true){
runThis();
}
}
}
public class GPBClient implements Runnable{
Socket socket;
ObjectInputStream objectInputStream;
ObjectOutputStream objectOutputStream;
//DataInputStream dataInputStream;
//DataOutputStream dataOutputStream;
public GPBClient(){
}
private void runThis(){
try{
//creating socket to server
socket = new Socket("localhost", 2004);
System.out.println("Connected to localhost in port 2004");
//IO streams
objectInputStream = new ObjectInputStream(socket.getInputStream());
objectOutputStream = new ObjectOutputStream(socket.getOutputStream
());
objectOutputStream.flush();
//dataInputStream = new DataInputStream(socket.getInputStream());
//dataOutputStream = new DataOutputStream(socket.getOutputStream
());
//dataOutputStream.flush();
//talk with server
try{
CarObject.Car car = CarObject.Car.parseFrom(objectInputStream);
//CarObject.Car car = CarObject.Car.parseFrom(dataInputStream);
//again, I used a wrapper in my actual
code to get good stuff in the tostring, left this out for
//brevity
System.out.println("client: the transmitted object from the server
is... \n" + car.toString());
}catch(ClassCastException classCastException){
System.err.println("object parsed is not a car.");
}catch(Exception exception){
System.err.println(exception);
}
System.exit(0);
}catch(UnknownHostException unknownHost){
System.err.println("You are trying to connect to an unknown
host!");
}catch(IOException ioException){
ioException.printStackTrace();
}finally{
//close connections and socket
try{
objectInputStream.close();
objectOutputStream.close();
//dataInputStream.close();
//dataOutputStream.close();
socket.close();
System.out.println("client: Closed all connection and streams.");
}catch(IOException ioException){
ioException.printStackTrace();
}
}
}
@Override
public void run() {
runThis();
}
}
when I change the GPBClient to use Data IO streams (just comment out
the Object IO references and uncomment the Data IO references in
GPBClient), that's when I get the
"com.google.protobuf.InvalidProtocolBufferException: Protocol message
end-group tag did not match expected tag." error.
I call the 2 threads to use it:
Thread server = new Thread(new GPBServer());
server.start();
Thread client = new Thread(new GPBClient());
client.start();
2 classes -GPBServer and GPBClient:
public class GPBServer implements Runnable{
ServerSocket serverSocket;
Socket socket = null;
ObjectOutputStream objectOutputStream;
ObjectInputStream objectInputStream;
public GPBServer(){}
private void runThis(){
try{
//create server socket
serverSocket = new ServerSocket(2004, 10);
//get connection
System.out.println("server: getting connection");
socket = serverSocket.accept();
System.out.println("server: Connection received from " +
socket.getInetAddress().getHostName());
//create IO streams
objectOutputStream = new ObjectOutputStream(socket.getOutputStream
());
objectOutputStream.flush();
objectInputStream = new ObjectInputStream(socket.getInputStream());
//send my GPB object
sendMyGPBObject();
}catch(IOException ioException){
ioException.printStackTrace();
}finally{
//Close connections and socket
try{
objectInputStream.close();
objectOutputStream.close();
serverSocket.close();
System.out.println("server: Closed all connection and streams.");
}catch(IOException ioException){
ioException.printStackTrace();
}
}
}
public void sendMyGPBObject(){
CarObject.Car car = null;
try{
//used a factory to create my car instance
using GPB builder stuff (did all GPB compilation successfully)
car = CarFactory.instance.createCar();
//print it out to see what its like (I used a wrapper to get useful
stuff to tostring but
//for sake of brevityhaven't included it here)
System.out.println("server: The original server side object is \n"
+ car.toString());
car.writeTo(objectOutputStream);
objectOutputStream.flush();
}catch(IOException e){
System.out.println("serverside: Error is " + e);
}
}
@Override
public void run() {
while(true){
runThis();
}
}
}
public class GPBClient implements Runnable{
Socket socket;
ObjectInputStream objectInputStream;
ObjectOutputStream objectOutputStream;
//DataInputStream dataInputStream;
//DataOutputStream dataOutputStream;
public GPBClient(){
}
private void runThis(){
try{
//creating socket to server
socket = new Socket("localhost", 2004);
System.out.println("Connected to localhost in port 2004");
//IO streams
objectInputStream = new ObjectInputStream(socket.getInputStream());
objectOutputStream = new ObjectOutputStream(socket.getOutputStream
());
objectOutputStream.flush();
//dataInputStream = new DataInputStream(socket.getInputStream());
//dataOutputStream = new DataOutputStream(socket.getOutputStream
());
//dataOutputStream.flush();
//talk with server
try{
CarObject.Car car = CarObject.Car.parseFrom(objectInputStream);
//CarObject.Car car = CarObject.Car.parseFrom(dataInputStream);
//again, I used a wrapper in my actual
code to get good stuff in the tostring, left this out for
//brevity
System.out.println("client: the transmitted object from the server
is... \n" + car.toString());
}catch(ClassCastException classCastException){
System.err.println("object parsed is not a car.");
}catch(Exception exception){
System.err.println(exception);
}
System.exit(0);
}catch(UnknownHostException unknownHost){
System.err.println("You are trying to connect to an unknown
host!");
}catch(IOException ioException){
ioException.printStackTrace();
}finally{
//close connections and socket
try{
objectInputStream.close();
objectOutputStream.close();
//dataInputStream.close();
//dataOutputStream.close();
socket.close();
System.out.println("client: Closed all connection and streams.");
}catch(IOException ioException){
ioException.printStackTrace();
}
}
}
@Override
public void run() {
runThis();
}
}
Oliver Jowett
Dec 7, 2009, 4:35:12 AM12/7/09
to Nigel Pickard, Protocol Buffers
Nigel Pickard wrote:
> Here's my sample code. In this code, everything works for me -however
> when I change the GPBClient to use Data IO streams (just comment out
> the Object IO references and uncomment the Data IO references in
> GPBClient), that's when I get the
> "com.google.protobuf.InvalidProtocolBufferException: Protocol message
> end-group tag did not match expected tag." error.
ObjectOutputStream doesn't just write raw bytes directly to the
> Here's my sample code. In this code, everything works for me -however
> when I change the GPBClient to use Data IO streams (just comment out
> the Object IO references and uncomment the Data IO references in
> GPBClient), that's when I get the
> "com.google.protobuf.InvalidProtocolBufferException: Protocol message
> end-group tag did not match expected tag." error.
underlying stream, it also adds some additional header and framing data.
So in general you can't write data with an ObjectOutputStream and read
it back with a plain DataInputStream and expect things to work.
There's really no reason to use ObjectOutputStream unless you are using
Java serialization.
-O
Nigel Pickard
Dec 7, 2009, 9:33:20 AM12/7/09
to Protocol Buffers
Oliver:
thanks for the reply. I'm not going to use Object IO streams, rather
I used it as an example of how the IO stream type seems to make a
difference at each end. Further, I now realize that once I use an IO
stream type, I must use the same type of stream at each ends of a Java
app.... I was under the mistaken impression that a stream produced by
a Java IO type is the same -i.e. just a stream of bytes with no meta
stuff -and that the type of Stream encapsulated the functionality
required for that type of stream (e.g. object, file, etc).
For communicating with a C++ app, I'm going to use a
bytearrayoutputstream. I assume that the bytearrayoutputstream just
produces a simple stream of bytes -with no meta stuff and is able to
be used with C++. Once I get a simple example up and running I'll
post as unless I missed an example in the tutorial or missed that
someone has posted one already, I haven't seen a real example of
sending an object over a socket from Java to C++ and vice versa. I'm
guessing it might be a nice example to put up.
Anyway, for anyone else wanting to use ByteArrayStream in Java for
writing a Google , here is some code I've used:
ServerSocket serverSocket = new ServerSocket(2004, 10);
Socket socket = serverSocket.accept();
OutputStream outputStream = socket.getOutputStream();
outputStream.flush();
MyGPBObjects.MyGPBObject myGPBObject =
MyGPBObjectsFactory.instance.createMyGPBObject();
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream
();
myGPBObject.writeTo(byteArrayOutputStream);
byteArrayOutputStream.flush();
byteArrayOutputStream.writeTo(outputStream);
Obviously without all the try/catches here and where myGPBObject is
the Google Protocol Buffers object I created (in my example I create
one from a factory called MyGPBObjectsFactory.)
thanks for the reply. I'm not going to use Object IO streams, rather
I used it as an example of how the IO stream type seems to make a
difference at each end. Further, I now realize that once I use an IO
stream type, I must use the same type of stream at each ends of a Java
app.... I was under the mistaken impression that a stream produced by
a Java IO type is the same -i.e. just a stream of bytes with no meta
stuff -and that the type of Stream encapsulated the functionality
required for that type of stream (e.g. object, file, etc).
For communicating with a C++ app, I'm going to use a
bytearrayoutputstream. I assume that the bytearrayoutputstream just
produces a simple stream of bytes -with no meta stuff and is able to
be used with C++. Once I get a simple example up and running I'll
post as unless I missed an example in the tutorial or missed that
someone has posted one already, I haven't seen a real example of
sending an object over a socket from Java to C++ and vice versa. I'm
guessing it might be a nice example to put up.
Anyway, for anyone else wanting to use ByteArrayStream in Java for
writing a Google , here is some code I've used:
ServerSocket serverSocket = new ServerSocket(2004, 10);
Socket socket = serverSocket.accept();
OutputStream outputStream = socket.getOutputStream();
outputStream.flush();
MyGPBObjects.MyGPBObject myGPBObject =
MyGPBObjectsFactory.instance.createMyGPBObject();
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream
();
myGPBObject.writeTo(byteArrayOutputStream);
byteArrayOutputStream.flush();
byteArrayOutputStream.writeTo(outputStream);
Obviously without all the try/catches here and where myGPBObject is
the Google Protocol Buffers object I created (in my example I create
one from a factory called MyGPBObjectsFactory.)
Adam Vartanian
Dec 7, 2009, 10:59:42 AM12/7/09
to Nigel Pickard, Protocol Buffers
> Anyway, for anyone else wanting to use ByteArrayStream in Java for
> writing a Google , here is some code I've used:
>
> ServerSocket serverSocket = new ServerSocket(2004, 10);
> Socket socket = serverSocket.accept();
> OutputStream outputStream = socket.getOutputStream();
> outputStream.flush();
>
> MyGPBObjects.MyGPBObject myGPBObject =
> MyGPBObjectsFactory.instance.createMyGPBObject();
> ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream
> ();
> myGPBObject.writeTo(byteArrayOutputStream);
> byteArrayOutputStream.flush();
> byteArrayOutputStream.writeTo(outputStream);
What you're doing with the ByteArrayOutputStream will do the same thing as:
> writing a Google , here is some code I've used:
>
> ServerSocket serverSocket = new ServerSocket(2004, 10);
> Socket socket = serverSocket.accept();
> OutputStream outputStream = socket.getOutputStream();
> outputStream.flush();
>
> MyGPBObjects.MyGPBObject myGPBObject =
> MyGPBObjectsFactory.instance.createMyGPBObject();
> ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream
> ();
> myGPBObject.writeTo(byteArrayOutputStream);
> byteArrayOutputStream.flush();
> byteArrayOutputStream.writeTo(outputStream);
OutputStream outputStream = socket.getOutputStream();
myGPBObject.writeTo(outputStream);
except that it takes more time and uses more memory. You don't need
an intermediate output stream, you already have the socket's output
stream. You can just write directly to that.
- Adam
Reply all
Reply to author
Forward
0 new messages