java parse with class known at runtime (and compiled proto)

[Koert Kuipers]

Hey all,

I found a bunch of threads on this topic already, but i cannot figure it out.

This is my situation:
* proto files have been compiled and java classes for them are available
* i need to deserialize messages with different types
* the class of the message type is available at the time of deserialization as a generic type

so i have a class with one function that looks like this:


import com.google.protobuf.Message

public class ProtobufDeserializer<T extends Message> {
    public T fromByteBuffer(ByteBuffer byteBuffer) {
       *** do something ***
    }
}

Any ideas? Is Message  the wrong class/interface to extend from? I have not been able to find any other class/interface where i can get it to work.

Thanks Koert

[Evan Jones]

On Dec 3, 2010, at 14:21 , Koert Kuipers wrote:
> public class ProtobufDeserializer<T extends Message> {
> public T fromByteBuffer(ByteBuffer byteBuffer) {

I don't *think* the generic type is going to be enough due to erasure,
but I'm not a generics expert. I know something like the following
works (I may be messing up the generics syntax since I'm not super
familiar with it):

<T extends MessageLite> public T fromByteBuffer(ByteBuffer byteBuffer,
T defaultInstance) {
Builder b = defaultInstance.newBuilderForType();
b.mergeFrom(ByteString.copyFrom(byteBuffer));
return b.build();
}

You can get defaultInstance from
ConcreteMessageType.getDefaultInstance();

You may want to create a tiny InputStream wrapper around ByteBuffer to
avoid an extra copy, or if you know it is a heap byte buffer, use the
array mergeFrom().

Hope that helps,

Evan

--
Evan Jones
http://evanjones.ca/

[Siju Varghese]

Take a look at

https://github.com/sijuv/protobuf-codec/blob/master/src/main/java/com/google/protobuf/codec/Codec.java

 https://github.com/sijuv/protobuf-codec/blob/master/src/main/java/com/google/protobuf/codec/AbstractCodec.java

https://github.com/sijuv/protobuf-codec is a library which I plan to extend to support other serialization schemes, currently supports JSON.    

Regards,

Siju

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[users ....]

What u are doing will work. You extend message
An then u can call a get builder function. With
That you can merge from bytes. I don't have a computer
right now but if u post again if you don't find the
function and I'll post a snippet on Monday.

[users ....]

Here is the snippet ...

public class ProtocolBufferSerializationProvider<T extends Message> {
private final Class<T> type;
private final T instance;
//......

@SuppressWarnings("unchecked")
public T deserialize(byte[] data) throws
InvalidProtocolBufferException {
final Message.Builder builder = instance.newBuilderForType();
builder.mergeFrom(data);
final Message message = builder.build();
if (type.isInstance(message)) {
return (T) message;
} else {
throw new ClassCastException("Message type "
+ message.getClass().getName()
+ " cannot be cast to the configured type "
+ type.getName());
}
}
}

instance needs to be created with something like:
instance = (T) type.getMethod("getDefaultInstance").invoke(null);

> > Thanks Koert- Hide quoted text -
>
> - Show quoted text -

[Koert Kuipers]

hey

thanks for replies. this works indeed. unfortunately i saw the reply below to late, and ended up writing something very similar over weekend... (took me a while to figure the klass.getMethod("getDefaultInstance").invoke(null) trick out).

i ended up caching the default instances for multiple protobuf message subclasses to avoid repeating the relative slow reflection-based code to get from Class to a Message instance. so now i basically have a collection of singleton deserializers (one for each Message subclass).

also i added an builder.isInitialized() check before doing the builder.build().

i am still confused why protobuffers does not have a parseFrom() method in the message Interface. that would have been a lot cleaner i think. or am i missing something?

thanks again,
koert

[Evan Jones]

On Dec 6, 2010, at 9:05 , Koert Kuipers wrote:
> i am still confused why protobuffers does not have a parseFrom()
> method in the message Interface. that would have been a lot cleaner
> i think. or am i missing something?

Because the serialized representation does not include anything to
describe the type. Thus, based on just the raw bytes, you can't tell
what type of message it is. If you need this functionality, you need
to build it yourself, either using union types:

http://code.google.com/apis/protocolbuffers/docs/techniques.html#union

Or by doing some custom thing (including some unique identifier, or
the fully qualified message name, or something in some header first).

[Koert Kuipers]

But that doesn't make a parseFrom() in message interface invalid, does it?
Indeed some other information outside the raw bytes will be needed to pick to right Message subclass. But that's fine.

One could then:
1) pick the right subclass of Message based upon some information outside the raw bytes (in my case something stored in a protobuf wrapper around the raw bytes)
2) call subclass.parseFrom(bytes)

now we have to jump through more hoops for step 2 (create instance of Message subclass, newBuilderForType, mergeFrom, isInitialized, build)

Or am i missing something more fundament

[Evan Jones]

On Dec 6, 2010, at 10:31 , Koert Kuipers wrote:
> But that doesn't make a parseFrom() in message interface invalid,
> does it?
> Indeed some other information outside the raw bytes will be needed
> to pick to right Message subclass. But that's fine.

Oh, sorry, I misunderstood your question, so my answer is somewhat
invalid.

> One could then:
> 1) pick the right subclass of Message based upon some information
> outside the raw bytes (in my case something stored in a protobuf
> wrapper around the raw bytes)
> 2) call subclass.parseFrom(bytes)
>
> now we have to jump through more hoops for step 2 (create instance
> of Message subclass, newBuilderForType, mergeFrom, isInitialized,
> build)

The MessageLite.Builder interface has a mergeFrom method that does
what you want. What you should do is something like:

* Get a MessageLite instance for the message type you want to parse
(eg. something like MyMessageType.getDefaultInstance(), or
MessageLite.getDefaultInstanceForType())
* Hold on to that MessageLite instance in some sort of registry.
(HashMap<Integer, MessageLite>?)
* When you get a message, look at the protobuf wrapper to determine
the type.
* Look up the "prototype" MessageLite instance in your registry.
* Call prototypeInstance.newBuilderForType().mergeFrom(bytes).build()

This only creates a single instance of the message each time.
The .build() method will automatically check that the message is
initialized, so you don't need to call isInitialized (although you may
want to catch the exception it could throw?).

This Builder pattern is used so that the Message objects are
immutable. This means they can be passed between threads without
requiring any synchronization. See:

http://code.google.com/apis/protocolbuffers/docs/javatutorial.html#builders

Hope this helps,

[Kenton Varda]

Evan is correct.  The best way to write code which deals with a generic protobuf type is to have it take a default instance as a parameter.  From that you can do everything else.

This is actually better than passing around Class objects, because it allows users to use DynamicMessages with your code.  Using Class objects forces users to use only generated types.  Also, Java reflection may be slow or even unavailable on some platforms.