Partial Decoding of message

[itt...@gmail.com]

Hi Group,

I am using protobuf in a multi-threaded software. Here manager thread decodes the protobuf encoded message and then assign the message to a particular worker thread based on key. I want to minimize per message processing at manager thread. Is it possible to encode the key at the head of message and decode only this key at manager thread. Complete decoding of message will be moved to the actual worker thread.

thanks

Ittium

[Ilia Mirkin]

Unfortunately it's not guaranteed that earlier fields appear earlier
in the message. Although that is often the case, I wouldn't recommend
writing your code s.t. it assumes this. The usual way that I handle
this is by splitting the message into a header and data message, and
then send something like

<header length varint>
<header data>
<data length varint>
<data data>

That way your manager thread just decodes the header figures out what
to do, and sends it on. Then the actual worker thread decodes the
data.

If this is not an option, you can write a custom decoder that just
skips over fields you don't need to read. This is a little tricky, but
if you're not trying to be too generic it shouldn't be that much code.

> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+u...@googlegroups.com.
> To post to this group, send email to prot...@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Kevin Regan]

This is how I handle the same issue.  This would be similar to most multi-threaded daemons taking client input.  The manager reads the message type and passes the socket/stream to a handling thread.

[phreed]

On Monday, July 8, 2013 12:59:12 PM UTC-5, Ilia Mirkin wrote:

If this is not an option, you can write a custom decoder that just
skips over fields you don't need to read. This is a little tricky, but
if you're not trying to be too generic it shouldn't be that much code.

Do you have an example decoder that can be used as a model 

for such a custom decoder? 

My situation is a bit different than the original poster.

I have a set of content sensitive network bridges.

Each bridge is interested in (potentially) different content and

as such I would rather not incur parsing costs that are not needed.

[Ilia Mirkin]

On Tue, Jul 9, 2013 at 11:36 AM, phreed <phr...@gmail.com> wrote:
>
>
> On Monday, July 8, 2013 12:59:12 PM UTC-5, Ilia Mirkin wrote:
>>
>>
>>
>> If this is not an option, you can write a custom decoder that just
>> skips over fields you don't need to read. This is a little tricky, but
>> if you're not trying to be too generic it shouldn't be that much code.
>
>
> Do you have an example decoder that can be used as a model
> for such a custom decoder?

Sadly, no. Perhaps I can poke the responsible parties for open-sourcing it.

> My situation is a bit different than the original poster.
> I have a set of content sensitive network bridges.
> Each bridge is interested in (potentially) different content and
> as such I would rather not incur parsing costs that are not needed.
>

Well, if your proto looks something like

message {
optional Foo a = 1;
optional Bar b = 2;
...
}

And each thing just needs either a or b or c etc, then it's fairly
easy. If it's more involved, then it's more difficult :)

The basic idea is that you read the tag, which includes the field
type. If it's a tag you want, you decode it, e.g. by doing
Foo.ParseFromString(). If it's not, you skip it (which you can do
based on the field type, included in the tag). There are helpers in
WireFormatLite for reading tags/skipping things. (At least for C++,
not sure how the Java parsing code is organized.)

-ilia

[phreed]

On Tuesday, July 9, 2013 1:51:51 PM UTC-5, Ilia Mirkin wrote:

On Tue, Jul 9, 2013 at 11:36 AM, phreed <phr...@gmail.com> wrote:
>
>
> On Monday, July 8, 2013 12:59:12 PM UTC-5, Ilia Mirkin wrote:
>>
>>
>>
>> If this is not an option, you can write a custom decoder that just
>> skips over fields you don't need to read. This is a little tricky, but
>> if you're not trying to be too generic it shouldn't be that much code.
>
>
> Do you have an example decoder that can be used as a model
> for such a custom decoder?

Sadly, no. Perhaps I can poke the responsible parties for open-sourcing it.

I would appreciate that.

 

> My situation is a bit different than the original poster.
> I have a set of content sensitive network bridges.
> Each bridge is interested in (potentially) different content and
> as such I would rather not incur parsing costs that are not needed.
>

Well, if your proto looks something like

message {
  optional Foo a = 1;
  optional Bar b = 2;
   ...
}

And each thing just needs either a or b or c etc, then it's fairly
easy. If it's more involved, then it's more difficult :)

The basic idea is that you read the tag, which includes the field
type. If it's a tag you want, you decode it, e.g. by doing
Foo.ParseFromString(). If it's not, you skip it (which you can do
based on the field type, included in the tag). There are helpers in
WireFormatLite for reading tags/skipping things. (At least for C++,
not sure how the Java parsing code is organized.)

https://code.google.com/p/protobuf/source/browse/trunk/java/src/main/java/com/google/protobuf/WireFormat.java?r=349

Looks like the thing I need for java.

Thanks 

  -ilia

[Ilia Mirkin]

You may also want to glance at AbstractMessage.mergeFrom and
mergeFieldFrom. Basically you'd only call the mergeFieldFrom logic for
fields you care about, and just skip over the correct number of bytes
for the ones you don't care about (for which you still need
mergeFieldFrom-style logic, of course).

-ilia