Maximizing fixed size buffer without losing the whole message

37 views
Skip to first unread message

edan

unread,
Jul 8, 2009, 7:25:23 AM7/8/09
to prot...@googlegroups.com
Hi-
We have a fixed-size buffer into which we are serializing a message (a pretty large and convoluted message, with many optional sub-messages and repeated fields, etc).
We have found that when the message size gets bigger than the buffer size, SerializeToArray just fails to write anything - we lose the whole message.
We'd like to be able to write as much message as we can, and just give up on some of the optional fields, or, more likely, to stop adding elements to repeated fields, etc.
What is the best way to do this efficiently?  It seems that calling ByteSize() all the time will cost us in performance.  Is there a way to efficiently compute the space needing during our message-building phase?  Or is there a way to serialize as we go using some of the append methods, and then just stop appending when we reach the limit?  What I don't understand is how to avoid re-appending the same parts of the message that we already serialized?  Let's also assume that most of the places we are likely to "run out of space" is while we are adding more and more elements to repeated fields.
Thanks for any help.
--edan

Kenton Varda

unread,
Jul 8, 2009, 2:38:44 PM7/8/09
to edan, prot...@googlegroups.com
Sorry, there is no magic solution here.  The best suggestions I have are either:

1) Use SerializeToString() instead, which will always allocate a string large enough to hold the message.

2) Build your message, then call ByteSize(), and if it is too large, remove some fields and try again.  Once it is big enough, call SerializeWithCachedSizesToArray() -- since you already called ByteSize(), the sizes are already cached.  This way, if the message fits on the first attempt, you perform just as well as you did before, because SerializeToArray() calls ByteSize() followed by SerializeWithCachedSizesToArray().
Reply all
Reply to author
Forward
0 new messages