A bit more protobuf help please

[Dale Durham]

Hi All,

Again I am new to this, so please continue to bear with me and thanks
to Benjamin Wright for all the help so far!! I think that I am
thinking myself into a circle between the FileDescriptors,
FileDescriptorSets, FileDescriptorProtos, and so on. It is not yet
really clear what is what between them all.

So, with help from Benjamin, I can get all the base proto and
dependency file names and field names in one way or another. Also, I
am creating the descriptors files "on the fly" if they do not already
exist for each message type (I have not control over the messsages
being passed and cannot add the deccriptors to the messages
themselves). I am still fuzzy on if I should put all the fields into
one Desriptor proto and pass that or all the files in one
DescriptorSet and pass that, or something else, but I'll figure that
out eventually.

Then I need to (or at least I THINK that I need to):

1) Create a new protobuf message based on the definition defined for
the protobuf files and fields based on all the Descriptor and
dependency stuff above (I think this should be a DynamicMessage based
on what I have seen and read)
2) Do something similar to the C++ equivalent of protobuf-
>ParseFromString(message.getData()); to deserialize and get the byte
array I have containing the actual message data into it

I suspect that since there is protobuf no parsing in Java, I'll have
to call out to protoc again, but if so am not sure what to pass it or
even if my steps above are where I really need to go next. It may be
that I need to iterate through each field, get it's type, and then
parse from the byte stream based on that iteration.

I also wonder if it would be faster and easier to just create a Java
version of the message based on the C++ version that I am being passed
in the byte array and use that instead?

The overall base steps I need to do are:
1) Get a byte array containing protobuf message that was was compiled
and serialized in C++
2) Get the structure of that message (base type and dependency files
and fields in order and I can get this now, but not sure what I am
doing is optimal)
3) Glue the message data into #2 above and display the it (again I
think this may require a protoc call out, but maybe not)

Getting the files and fields is great and cool considering that I just
started playing with these - not because I wanted to but because I
have to! Now I just want to be sure am doing it the best way and need
to figure out how best to glue it all together. Most of the examples I
can find assume the message was compiled for Java and that I know the
type (the infamous Person Class in most of them) in advance and I do
not. Any additional help or thoughts are appreciated.

Thanks again,
Dale

[Benjamin Wright]

Dale:

I'm glad to be of help. You're definitely over thinking the problem.

File descriptor set- a protobuf describing one put more proto files.
Contains file descriptor proto (s)

File descriptor proto- a protobuf describing a Single proto file

File descriptor-a java class containing the descriptor information for
a proto file in a format usable by the dynamic message class and for
other "reflection" over a proto message. These can be created by
getting them from a compiled proto "root"class or built at run time
from a file descriptor proto (how you're getting them).

Descriptor- a Java class that provides descriptor information for a
particular message type that can be serialized or parsed. You get
these from a file descriptor.

Dynamic message is a class provided to build and parse messages for
which you have no compiled Java proto you must provide a descriptor
to parse messages but not for serialization.

You should not need to parse from text unless the messages are passed
as proto strings which would be an awful waste of bandwidth and
performance.

i'm not in front of a computer (on my way home from a vacation) and I
can't recall the exact method signatures but look at the dynamic
message class closer. There should be a parse our build method that
takes bytes or input stream, a descriptor (for a message type), and
optionally an extension registry. If you jabber extensions to deal
with let me know and I'll explain that better as It works differently
in Java than in cpp.


:) good luck
Sorry about grammar in places i'm on my phone.... That was a lot to
type using Swype.

[Benjamin Wright]

PS.. Just in case it want clear you should not need protoc for
anything after you have the descriptors. The Java protobuf library
can do all the dynamic parsing logic for you.

[Dale Durham]

Hi Benjamin,

Thanks again, I think that am very close now. I have the hash map with
all the descriptors in it and I see that DynamicMessage has a
parseFrom method that takes (Descriptor type, Byte[] data). So I can
pass my messages byte stream in as the data, but cannot quite figure
out what to pass for the 1st parameter since each message has 3 or
more descriptors (all but one of which are dependency descriptors). I
also do not see any "merge" capability in the class.

Can you (or anyone else) shed any light on how to do this when you get
a chance? How do you give all of the descriptors to the
DynamicMessage?

TIA,
Dale

[Benjamin Wright]

You just need the descriptor for the top level message. It has the
links back to all the Contained types. That is why you needed the
descriptors of all of all the dependent files in order to build the
file descriptor.

[Dale Durham]

OK, I think I may have missed a step here. Right now, I have:

1) A file stream with the top level message descriptor - fds
2) A hash map with all the dependency descriptors - fdl

final FileDescriptorSet fds = FileDescriptorSet.parseFrom(new
FileInputStream(myFile)); // load the file however you feel
appropriate - but close your streams : )
// special note - if you have custom options, you will
need to create an ExtensionRegistry with them and use it while
parsing
the FileDescriptorSet
try {
Map<String, FileDescriptor> fdl = new HashMap<String,
FileDescriptor>();
FileDescriptor current = null;
for (FileDescriptorProto fdp : fds.getFileList()) {
final List<String> dependencyList =
fdp.getDependencyList();
final FileDescriptor[] fda = new
FileDescriptor[dependencyList
.size()];
for (int i = 0; i < fda.length; i++) {
FileDescriptor fddd =
fdl.get(dependencyList.get(i));
if (fddd == null) {
// missing imports! - this should not happen
unless you left off the --include_imports directive
} else {
fda[i] = fddd;
}
}
current = FileDescriptor.buildFrom(fdp, fda);
fdl.put(current.getName(), current);
}
// the "fdl" object now has all the descriptors - grab
the
one you need.
} catch (DescriptorValidationException e) {
// panic ?
}

How do I link of glue them all together? Do I loop through fdl and
somehow add links in fds?

Thanks again,
Dale

[Benjamin Wright]

Let's say you have message Card in file deck.proto ....

Descriptor card = fdl.get("deck.proto").findMessage("Card");

DynamicMessage.parse (bytes, card)

[Dale Durham]

I get that (kind of), but in the sample you posted, when the call to
"FileDescriptor fddd = fdl.get(dependencyList.get(i));" is made the is
nothing in fdl because nothing was ever "put" onto it at that point.
So like said either I am missing something or there is a step missing
here...

final FileDescriptorSet fds = FileDescriptorSet.parseFrom(new
FileInputStream(myFile)); // load the file however you feel
appropriate - but close your streams : )
// special note - if you have custom options, you will
need to create an ExtensionRegistry with them and use it while
parsing
the FileDescriptorSet
try {
Map<String, FileDescriptor> fdl = new HashMap<String,
FileDescriptor>();
FileDescriptor current = null;
for (FileDescriptorProto fdp : fds.getFileList()) {
final List<String> dependencyList =
fdp.getDependencyList();
final FileDescriptor[] fda = new
FileDescriptor[dependencyList
.size()];
for (int i = 0; i < fda.length; i++) {
FileDescriptor fddd =

fdl.get(dependencyList.get(i)); <- can't get something that is not
there

if (fddd == null) {
// missing imports! - this should not happen
unless you left off the --include_imports directive
} else {
fda[i] = fddd;
}
}
current = FileDescriptor.buildFrom(fdp, fda);
fdl.put(current.getName(), current);
}
// the "fdl" object now has all the descriptors - grab
the
one you need.
} catch (DescriptorValidationException e) {
// panic ?
}

[Dale Durham]

Never mind, I figured it out. My descriptor files are named
differently. So when the dependency list is passed, I get xxx.proto
and the file that I really want is xxx.desc. I guess I'll have to
parse the extension out, change it then put the correct ones into fdl.

Dale

> ...
>
> read more »

[Benjamin Wright]

Dale - sounds like you figured it out, but the quick answer anyway
(for anyone else listening...)

The code I provided expects that the Descriptor files you have are
created with the "--include_imports" directive provided to protoc...
which means that all the imported files will be included in the same
file before they are used. In this case the entries needed have
already been added to the list by the time they are looked for. The
first file has no imports, after that files may have imported one or
more of the previous entries in the FileDescriptorSet.

If you did not use the "--include_imports" directive, you will need to
provide these descriptors from elsewhere and modify the "lookup" line
of code you pointed out accordingly. Either add them to the map ahead
of time or grab them directly from your ".desc" files (I usually name
them "*.fds" for FileDescriptorSet).

> ...
>
> read more »

[Dale Durham]

Thanks again for all your help Benjamin!!! I just got it all working
and would still probably be tinkering without your help.

Dale

> ...
>
> read more »