C++ JSON Serialization Performance

[Benjamin Mahler]

Hi,

I've searched the list but haven't found an adequate treatment of this subject from the existing threads.

In the system I work on, we weren't aware of the provided protobuf to json facilities (or they didn't exist so many years ago) and so we built our own protobuf to json conversion that is based on protobuf reflection + rapidjson for the json encoding/serialization. It appears that the provided protobuf to json conversion facility is much slower than our approach.

Here are numbers in seconds for 10 runs [1] of serializing a large message:

SerializeAsString: 2.07,2.01,2.97,1.76,1.70,2.23,2.02,1.66,1.97,1.90 (median 1.99 seconds)

MessageToJsonString: 23.70,24.22,24.22,24.13,24.00,24.73,24.75,24.82,24.98,24.25 (median 24.24 seconds)

To json via our reflection-based approach: 4.81,4.07,3.79,4.28,4.90,6.38,5.53,4.37,4.12,4.20 (median 4.33 seconds)

So the built-in facility for protobuf message to serialized json takes 460% longer than our approach, which you can find here:

https://github.com/apache/mesos/blob/1.9.0/3rdparty/stout/include/stout/protobuf.hpp#L810-L1042

Which leverages https://github.com/apache/mesos/blob/1.9.0/3rdparty/stout/include/stout/jsonify.hpp

Is this expected? It seems the protobuf built in facility, at least for going from message to serialized json, is extremely slow and can be much faster.

I suppose another technique that would be even faster than our approach would be for the json serialization logic to be generated by the protobuf compiler on a per message basis (much like protobuf serialization is generated), allowing reflection to be avoided entirely.

Would love to hear any thoughts on comments on this topic.

Ben

[Benjamin Mahler]

For posterity, filed an issue here: https://github.com/protocolbuffers/protobuf/issues/6907