Deserialization of large message (> 64MB) slient-ly fails randomly.

33 views
Skip to first unread message

yangyi...@gmail.com

unread,
Aug 7, 2021, 2:45:55 AMAug 7
to Protocol Buffers
Hi folks.

We are using c++ protocol buffer v3.0.2 for some structures' serialization and deserialization.
We are able to serialize a large PB message (~ 600MB) to a buffer and verfied the serialized size equals to PB.ByteSize(). 
Then when we try to deserialize the buffer back to the PB message. We are expecting the deserialization fails. But randomly, sometimes we see the deserialization succeeds, but the deserialized PB message only has a partial content with a ByteSize() == 64MB.

We wonder if there is anyone encountered this issue before. We would like to have a consistent behavior that all deserialization should fail (not succeed but return some wrong result).

serialization function we use: SerializeWithCachedSizesToArray(uint8 * target) const
deserialization function we use: ParseFromArray(const void * data, int size).

PB structure:
message response
{
    message A
    {
         required uint64 a = 1;
         repeated uint64 b = 2;
    }
    repeated A c = 1;
}

Thanks,
Yihao

Adam Cozzette

unread,
Aug 9, 2021, 11:57:07 AMAug 9
to yangyi...@gmail.com, Protocol Buffers
I would guess that there is a problem on the serialization side. Any of the WithCachedSizes serialization methods rely on the cached sizes, which means that you need to have called ByteSizeLong() or ByteSize() right before serializing. If the cached sizes are stale then things will most likely not work correctly.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/ed13812d-d115-4c10-a3a6-d07066698050n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages