Error: Byte size calculation and serialization were inconsistent

[Robby Zinchak]

Hello,

I have been using protobuf successfully for my project for a number of
months now when I compile my project in gcc. Recently I went to
compile my project in gcc with the -O optimization flag, but when
running the project, I soon get an error when attempting to call
SerializeAsString():

libprotobuf FATAL google/protobuf/message_lite.cc:65] CHECK failed:
(bytes_produced_by_serialization) == (byte_size_before_serialization):
Byte size calculation and serialization were inconsistent. This may
indicate a bug in protocol buffers or it may be caused by concurrent
modification of the message.
terminate called after throwing an instance of
'google::protobuf::FatalException'
what(): CHECK failed: (bytes_produced_by_serialization) ==
(byte_size_before_serialization): Byte size calculation and
serialization were inconsistent. This may indicate a bug in protocol
buffers or it may be caused by concurrent modification of the message.

Please note the project is single-threaded, so I am unsure how
concurrent modification would be possible.

Does anyone know how I might resolve this? I'm kinda lost on this
error, so any help would be much appreciated.

Thank you,
Robby

-------------------------------------------------------------------------------
Details:

Protobuf version: getting this in both 2.4.0 and 2.4.1

GCC version:
htmud@archive:~/testsvn$ gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian
4.4.5-8' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --
enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-
suffix=-4.4 --enable-shared --enable-multiarch --enable-linker-build-
id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --
libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-
debug --enable-objc-gc --with-arch-32=i586 --with-tune=generic --
enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-
gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.4.5 (Debian 4.4.5-8)

Core dump:
Core was generated by `./HTMudServ'.
Program terminated with signal 6, Aborted.
#0 0x00002b5698bfb1b5 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x00002b5698bfb1b5 in raise () from /lib/libc.so.6
#1 0x00002b5698bfdfc0 in abort () from /lib/libc.so.6
#2 0x00002b56984e8dc5 in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/libstdc++.so.6
#3 0x00002b56984e7166 in ?? () from /usr/lib/libstdc++.so.6
#4 0x00002b56984e7193 in std::terminate() () from /usr/lib/libstdc+
+.so.6
#5 0x00002b56984e728e in __cxa_throw () from /usr/lib/libstdc++.so.6
#6 0x00002b569817d9db in
google::protobuf::internal::LogMessage::Finish (this=0x7fff7434ae10)
at google/protobuf/stubs/common.cc:195
#7 0x00002b5698185d73 in ByteSizeConsistencyError
(byte_size_before_serialization=<value optimized out>,
byte_size_after_serialization=<value optimized out>,
bytes_produced_by_serialization=46)
at google/protobuf/message_lite.cc:68
#8 0x00002b5698186098 in
google::protobuf::MessageLite::AppendPartialToString
(this=0x7fff7434aee0, output=<value optimized out>) at google/protobuf/
message_lite.cc:284
#9 0x00002b56981865f1 in
google::protobuf::MessageLite::SerializeAsString (this=0x430f) at
google/protobuf/message_lite.cc:321
#10 0x0000000000426ce4 in CObject::sendItemToPlayer(int, CPlayer*) ()
#11 0x000000000042145b in CPlayer::sendItemToPlayer(int, CPlayer*) ()
#12 0x0000000000421a90 in CPlayer::teleportToLocation(int, WorldPos)
()
#13 0x000000000041002f in HandleIncomingMessage(char*, int, int, int)
()
#14 0x000000000040bfbf in main ()

And here is my code from frame #10
HTMud::EnvAdd item;
item.set_id(ID);
item.set_idtype(typeID);
item.set_x(X);
item.set_y(Y);
item.set_z(Z);
item.set_lockdown(lockdown);
item.set_mapid(map);
item.set_tilesetno(tilesetNo);
item.set_tilesetx(tilesetX);
item.set_regionx(regionX);
item.set_regiony(regionZ);

if (volume == VOL_SELF)
objects->SendPlrPacket(toWhom, volume, item.SerializeAsString(), 10);

[Evan Jones]

This is weird. I don't see any clear potential cause, so I have a few questions:

> HTMud::EnvAdd item;
> item.set_id(ID);
> item.set_idtype(typeID);
> item.set_x(X);
> item.set_y(Y);
> item.set_z(Z);
> item.set_lockdown(lockdown);
> item.set_mapid(map);
> item.set_tilesetno(tilesetNo);
> item.set_tilesetx(tilesetX);
> item.set_regionx(regionX);
> item.set_regiony(regionZ);

Are all these values primitives? Are any of them protocol buffers?

Have you tried dumping the values that are being set when it dies, and trying a standalone program that sets the values and calls SerializeToString to see if it has the same problem?

Have you made any changes to the protocol buffers library? I'm assuming you are using the released version of 2.4.1?

Have you tried running this under valgrind? I'm wondering if there could be other weird memory corruption that is happening? That seems to be a frequent cause of "this shouldn't be happening" type errors, particularly things that appear/disappear occur with optimization enabled/disabled.

Evan

--
http://evanjones.ca/

[Robby Zinchak]

Ah, thank you very much Evan.

It turned out to be an uninitialized boolean.  Properly setting the value in question seems to allow things to proceed normally.

Thanks again,
Robby

[Evan Jones]

On Feb 6, 2012, at 21:54 , Robby Zinchak wrote:
> It turned out to be an uninitialized boolean. Properly setting the value in question seems to allow things to proceed normally.

Ah! Interesting. So one of your .set_* properties is a boolean, and one of them was uninitialized? That would do it. This was discussed previously and dismissed as a "wont fix" problem, because it is hard/impossible to make portable code that will test for this:

http://code.google.com/p/protobuf/issues/detail?id=234

Although its somewhat confusing since WireFormatLite::WriteBoolNoTag contains code to try to avoid this problem, which GCC helpfully optimizes away.

I am not able to get the exact crash as the one you reported, but I can get it to crash in MessageLite::SerializeWithCachedSizesToArray by creating a boolean with a value of 0x80 (serializing to two bytes instead of one, causing it to create a message larger than it expects). I can't figure out how it could crash at the point you report the crash, but that doesn't really matter.

Glad you got it working,

Evan

--
http://evanjones.ca/

[Robert Graham]

I am having problems where my message fails to serialize. A fatal exception is thrown for the ' ResponseMessage.SerializeToCodedStream(coded_output);'. ResponseMessage contains all the required fields, and is 104 bytes without the varint....

I am compiling this for 64 bit windows, using the 32bit protoc.exe.

// Send the response message

         // Get the byte size of the response message

         uint32_t response_message_size = ResponseMessage.ByteSize();

         // Add the varint size to the total message size

         uint32_t response_message_total_size = response_message_size + google::protobuf::io::CodedOutputStream::VarintSize32(response_message_size);

         boost::asio::streambuf buffer_stream;

         std::ostream output_stream(&buffer_stream);

         google::protobuf::io::ZeroCopyOutputStream* raw_output =

            new google::protobuf::io::OstreamOutputStream(&output_stream);

         google::protobuf::io::CodedOutputStream* coded_output =

            new google::protobuf::io::CodedOutputStream(raw_output);

         try

         {

            coded_output->WriteVarint32(response_message_size);

            ResponseMessage.SerializeToCodedStream(coded_output);

            delete coded_output;

            delete raw_output;

            bytes_sent = m_Socket.send(buffer_stream.data());

         }

         catch( google::protobuf::FatalException )

         {

            assert(false);

            delete raw_output;

            delete coded_output;

            bytes_sent = 0;

         }

[snail thum]

I have same problem now, and there is no uninitialized boolean attr

[Niklas Hambüchen]

I found another reason this can happen: If you compile with an `-march` newer than the target CPU.

For example, if I compile a minimal protobuf example using SerializeToArray() using `-march=haswell` and execute it on a Sandy Bridge CPU, I get:

[libprotobuf FATAL google/protobuf/message_lite.cc:71] CHECK failed: (bytes_produced_by_serialization) == (byte_size_before_serialization): Byte size calculation and serialization were inconsistent.  This may indicate a bug in protocol buffers or it may be caused by concurrent modification of myproject.proto.MyMessage.

This is very surprising, because usually using a too new march only results at most in an obvious `illegal instruction` error, not an incorrect result at runtime.

I have investigated why this is the case:

I bisected the feature flags enabled by `-march=haswell` and found that `-mno-lzcnt` fixes the issue (so `-march=haswell -mno-lzcnt` works fine on Sandy Bridge for my example code).

Here is why:

* https://nextmovesoftware.com/blog/2017/06/06/the-perils-of-using-__lzcnt-with-msvc/

* https://stackoverflow.com/questions/25683690/confusion-about-bsr-and-lzcnt/43443701#43443701

> instead of getting an ‘illegal instruction’ result when you run it, Intel [...] decided to reuse or repurpose existing opcodes so that CPUs without ‘lzcnt’ instead did a ‘bsr’ (bit scan reverse). This was why (a) the results were different/wrong, and (b) why a value of 0 gave gibberish (the docs for ‘bsr’ say the results are undefined in that case).

I have confirmed that the code generated by `-march=haswell` contains `lzcnt` instructions.

Hope this helps anybody!

Niklas

[Andrey O]

Wow, same issue over here, 7 years later practically to the day! Your comment was a major hint at where to look (uninitialized/experimental/unused vars are definitely a known weakness of my team).

[Seb]

Same issue here, setting with an uninitialized boolean. Pretty much took my whole week to figure out this was the problem. It was different surfacing though, it instead was crashing with CHECK failed !coded_out.HadError() .
I figured out the buffer was too small for the info we were trying to store, and my boss pointed out "Huh, that's weird, because it should be showing this error (the one you got) instead:

CHECK failed:
(bytes_produced_by_serialization) == (byte_size_before_serialization):
Byte size calculation and serialization were inconsistent. This may
indicate a bug in protocol buffers or it may be caused by concurrent
modification of the message.

I don't know why we didn't get to that point, but thank goodness his insight led me here. Posting my version in the hopes of helping someone else in the future.