Detecting end of messages
39 views
Skip to first unread message
Jan Borsodi
Jul 14, 2008, 5:04:10 AM7/14/08
to Protocol Buffers
I'm considering using PB for a network protocol, but I have one issue
which I could not find a solution for in the docs.
PB doesn't define a way to detect when one message ends and the next
starts, this is a problem since I will send message after message over
the network (over a socket).
One way is to send the length of the message before the message (a bit
like an embedded message) encoded as a varint. This is quite OK but
means the message stream is not 100% PB.
I was also thinking about the tag field, since a tag value of 0 is not
allowed it would be possible to end the message with a byte containing
0. However this is not part of the PB standard and might cause
problems in the future if 0 is used for something else.
Any recommandations?
which I could not find a solution for in the docs.
PB doesn't define a way to detect when one message ends and the next
starts, this is a problem since I will send message after message over
the network (over a socket).
One way is to send the length of the message before the message (a bit
like an embedded message) encoded as a varint. This is quite OK but
means the message stream is not 100% PB.
I was also thinking about the tag field, since a tag value of 0 is not
allowed it would be possible to end the message with a byte containing
0. However this is not part of the PB standard and might cause
problems in the future if 0 is used for something else.
Any recommandations?
Torbjörn Gyllebring
Jul 14, 2008, 5:32:17 AM7/14/08
to Protocol Buffers
I haven't tried this myself but after browsing the docs and thinking
about it this idea appeared.
Can't you define your stream as a message with only optional embedded
fields for everything you want to send (Im guessing extensions are
mighty usefull here).
The second piece of the puzzle would be to implement a Message.Builder
that instead of building a new message simply dispatches all fields as
they are read.
Conceptually that would mean that you have an infinite message that
you're processing one field at a time instead of reading it to memory
for for later processing.
The only drawback I can see with this approach is that network
failiures could be a hassle but that's life with any streaming
solution like this :)
about it this idea appeared.
Can't you define your stream as a message with only optional embedded
fields for everything you want to send (Im guessing extensions are
mighty usefull here).
The second piece of the puzzle would be to implement a Message.Builder
that instead of building a new message simply dispatches all fields as
they are read.
Conceptually that would mean that you have an infinite message that
you're processing one field at a time instead of reading it to memory
for for later processing.
The only drawback I can see with this approach is that network
failiures could be a hassle but that's life with any streaming
solution like this :)
Alek Storm
Jul 14, 2008, 4:12:45 PM7/14/08
to Protocol Buffers
On Jul 14, 4:04 am, Jan Borsodi <jbors...@gmail.com> wrote:
> One way is to send the length of the message before the message (a bit
> like an embedded message) encoded as a varint. This is quite OK but
> means the message stream is not 100% PB.
Why not add a required field to the message, specifying the length? It
> One way is to send the length of the message before the message (a bit
> like an embedded message) encoded as a varint. This is quite OK but
> means the message stream is not 100% PB.
doesn't even have to specify the total number of bytes - just the
total number of fields, which I believe is easy to get from the API.
100% PB.
Torbjörn Gyllebring
Jul 14, 2008, 4:32:31 PM7/14/08
to Protocol Buffers
Doesn't this lead to a recursion?
To know when to stop parsing this message I need to parse this message
so I can get the field saying how long it is.
Whe not make a message like this
message meta {
required int32 next_message_length = 1
}
And simply send a stream like {meta, message, meta, message}
That way you know that you can always read the next message by
1: read meta message
2: read meta.next_message_length bytes
3: parse message from known bytes
repeat.
To know when to stop parsing this message I need to parse this message
so I can get the field saying how long it is.
Whe not make a message like this
message meta {
required int32 next_message_length = 1
}
And simply send a stream like {meta, message, meta, message}
That way you know that you can always read the next message by
1: read meta message
2: read meta.next_message_length bytes
3: parse message from known bytes
repeat.
Alek Storm
Jul 14, 2008, 4:52:12 PM7/14/08
to Protocol Buffers
On Jul 14, 3:32 pm, Torbjörn Gyllebring
continue to parse the message until you get the requisite number of
fields. Although if a length field never comes (in a malformed or
malicious message), you're screwed, unless you impose a hard cap on
message length. However, you get the same problem here:
> And simply send a stream like {meta, message, meta, message}
>
> That way you know that you can always read the next message by
> 1: read meta message
> 2: read meta.next_message_length bytes
> 3: parse message from known bytes
> repeat.
The meta.length field could just be really really big, making you read
in up to 4 gigs of data. Also, in this solution, you have to store
two separate messages on your server, unless you just store the
message, then regenerate meta every time you send it out.
(It's actually a really good idea, I'm just pointing out the relative
merits of both solutions.)
<torbjorn.gyllebr...@gmail.com> wrote:
> Doesn't this lead to a recursion?
> To know when to stop parsing this message I need to parse this message
> so I can get the field saying how long it is.
No, you just parse the message until you find the length field, then
> Doesn't this lead to a recursion?
> To know when to stop parsing this message I need to parse this message
> so I can get the field saying how long it is.
continue to parse the message until you get the requisite number of
fields. Although if a length field never comes (in a malformed or
malicious message), you're screwed, unless you impose a hard cap on
message length. However, you get the same problem here:
> And simply send a stream like {meta, message, meta, message}
>
> That way you know that you can always read the next message by
> 1: read meta message
> 2: read meta.next_message_length bytes
> 3: parse message from known bytes
> repeat.
in up to 4 gigs of data. Also, in this solution, you have to store
two separate messages on your server, unless you just store the
message, then regenerate meta every time you send it out.
(It's actually a really good idea, I'm just pointing out the relative
merits of both solutions.)
Torbjörn Gyllebring
Jul 14, 2008, 6:02:47 PM7/14/08
to Protocol Buffers
> No, you just parse the message until you find the length field, then
> continue to parse the message until you get the requisite number of
> fields. Although if a length field never comes (in a malformed or
> malicious message), you're screwed, unless you impose a hard cap on
> message length. However, you get the same problem here:
Im way to unfamilar with the API to realize this was an "out of the
> continue to parse the message until you get the requisite number of
> fields. Although if a length field never comes (in a malformed or
> malicious message), you're screwed, unless you impose a hard cap on
> message length. However, you get the same problem here:
box" option I was
living by the assumption that nessages we're parsed and built
completly in memory.
If incremental building / streaming is easy then this route is
absolutly a good one albeit maybe
a tad hard to explain to your buddies. (Quite a funky FSM needed to
keep track of thing I would suppose.. but, intresting).
>
> > And simply send a stream like {meta, message, meta, message}
>
> > That way you know that you can always read the next message by
> > 1: read meta message
> > 2: read meta.next_message_length bytes
> > 3: parse message from known bytes
> > repeat.
>
> The meta.length field could just be really really big, making you read
> in up to 4 gigs of data. Also, in this solution, you have to store
> two separate messages on your server, unless you just store the
> message, then regenerate meta every time you send it out.
True. Need to take into consideration how large messages usually are.
> > And simply send a stream like {meta, message, meta, message}
>
> > That way you know that you can always read the next message by
> > 1: read meta message
> > 2: read meta.next_message_length bytes
> > 3: parse message from known bytes
> > repeat.
>
> The meta.length field could just be really really big, making you read
> in up to 4 gigs of data. Also, in this solution, you have to store
> two separate messages on your server, unless you just store the
> message, then regenerate meta every time you send it out.
Actually I think the "meta" message would be more of an implementation
detail of the
communication channel than anything else depending on my consumer it
might, or might not be
needed, for a stream as talked about here it serves the role of a
delimiter, otoh if we're sending it
to some consumer that doesn't need it no need to keep it around.
Alek Storm
Jul 14, 2008, 6:36:53 PM7/14/08
to Protocol Buffers
On Jul 14, 5:02 pm, Torbjörn Gyllebring
message must be read before it's parsed. So in order for this to
work, the parser must be rewritten to support streaming. I'm
surprised it doesn't already, but hey, it gives the community
something to do. So I guess Jan will need your solution until that
happens.
> > > And simply send a stream like {meta, message, meta, message}
>
> > > That way you know that you can always read the next message by
> > > 1: read meta message
> > > 2: read meta.next_message_length bytes
> > > 3: parse message from known bytes
> > > repeat.
>
> > The meta.length field could just be really really big, making you read
> > in up to 4 gigs of data. Also, in this solution, you have to store
> > two separate messages on your server, unless you just store the
> > message, then regenerate meta every time you send it out.
>
> True. Need to take into consideration how large messages usually are.
> Actually I think the "meta" message would be more of an implementation
> detail of the
> communication channel than anything else depending on my consumer it
> might, or might not be
> needed, for a stream as talked about here it serves the role of a
> delimiter, otoh if we're sending it
> to some consumer that doesn't need it no need to keep it around.
Yup. You can do it either way - if the messages are immutable on your
end, you can store them alongside their meta, at the cost of
complexity and a bit more size, which ain't good if you have a ton of
messages. Or, you can just store the messages, and generate the meta
only when you send them out, at the cost of speed.
Also, I think I read that PB messages are designed to be small and
plentiful, but I doubt people will stick to that recommendation.
<torbjorn.gyllebr...@gmail.com> wrote:
> > No, you just parse the message until you find the length field, then
> > continue to parse the message until you get the requisite number of
> > fields. Although if a length field never comes (in a malformed or
> > malicious message), you're screwed, unless you impose a hard cap on
> > message length. However, you get the same problem here:
>
> Im way to unfamilar with the API to realize this was an "out of the
> box" option I was
> living by the assumption that nessages we're parsed and built
> completly in memory.
> If incremental building / streaming is easy then this route is
> absolutly a good one albeit maybe
> a tad hard to explain to your buddies. (Quite a funky FSM needed to
> keep track of thing I would suppose.. but, intresting).
You've actually pointed out a fatal flaw in my plan - the whole
> > No, you just parse the message until you find the length field, then
> > continue to parse the message until you get the requisite number of
> > fields. Although if a length field never comes (in a malformed or
> > malicious message), you're screwed, unless you impose a hard cap on
> > message length. However, you get the same problem here:
>
> Im way to unfamilar with the API to realize this was an "out of the
> box" option I was
> living by the assumption that nessages we're parsed and built
> completly in memory.
> If incremental building / streaming is easy then this route is
> absolutly a good one albeit maybe
> a tad hard to explain to your buddies. (Quite a funky FSM needed to
> keep track of thing I would suppose.. but, intresting).
message must be read before it's parsed. So in order for this to
work, the parser must be rewritten to support streaming. I'm
surprised it doesn't already, but hey, it gives the community
something to do. So I guess Jan will need your solution until that
happens.
> > > And simply send a stream like {meta, message, meta, message}
>
> > > That way you know that you can always read the next message by
> > > 1: read meta message
> > > 2: read meta.next_message_length bytes
> > > 3: parse message from known bytes
> > > repeat.
>
> > The meta.length field could just be really really big, making you read
> > in up to 4 gigs of data. Also, in this solution, you have to store
> > two separate messages on your server, unless you just store the
> > message, then regenerate meta every time you send it out.
>
> True. Need to take into consideration how large messages usually are.
> Actually I think the "meta" message would be more of an implementation
> detail of the
> communication channel than anything else depending on my consumer it
> might, or might not be
> needed, for a stream as talked about here it serves the role of a
> delimiter, otoh if we're sending it
> to some consumer that doesn't need it no need to keep it around.
end, you can store them alongside their meta, at the cost of
complexity and a bit more size, which ain't good if you have a ton of
messages. Or, you can just store the messages, and generate the meta
only when you send them out, at the cost of speed.
Also, I think I read that PB messages are designed to be small and
plentiful, but I doubt people will stick to that recommendation.
Kenton Varda
Jul 15, 2008, 12:22:14 AM7/15/08
to Jan Borsodi, Protocol Buffers
I think the best solution here is to simply write a size before each message.
If you wanted your protocol to look like "pure protocol buffers", you could do something like Torbjorn suggested. That is, have your protocol be defined as:
message Stream {
repeated Event event = 1;
}
Then, write your own parser for Stream which does something with each Event immediately after reading it, rather than waiting until EOF. This should be fairly trivial to write.
However, I'm not sure if there's any real advantage to making the top-level protocol be "pure protocol buffers". Protobufs are intended to represent chunks of data which may be contained using some other container protocol.
Reply all
Reply to author
Forward
0 new messages