Python Protobuf Serialization Performance In 2023

[Daniel Koohmarey]

I stumbled upon a post from 2010 mentioning using CExtensions to improve python protobuf serialization times:
https://groups.google.com/g/protobuf/c/z7E80KYJscc/m/ysCjHHmoraUJ
where the author states "~13x speedups" as a result of "Python code with PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp actually
also *searches for the symbols for any pre-generated C++ code in the
current process*, and uses them if available instead of
DynamicMessage." when using a CExtension. I can find reference code on github that uses this technique, and references a corresponding speedup, however I do notice that all code using this approach is ~10 years old. Is this approach still beneficial in any way to improve serialization performance in python? Or would protobuf 3.19+ with PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp result in equally performant serialization when serializing in Python3? Thanks

-  Daniel

[Daniel Koohmarey]

Note for anyone stumbling upon this thread in the future, leveraging cpp generated protobuf serialization that was pybound to python yielded a 5x+ improvement in serialization times over just PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp .

[Adam Cozzette]

Starting with 4.21.0, the default Python implementation is now based on upb (see here), and this should be faster than the older Python C++ implementation. Python-C++ is actually deprecated, and we are thinking about deleting it at some point.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/2cc6947d-6b59-4b3e-b3df-767418a866f4n%40googlegroups.com.