Regarding Protobuf packages and compiled Python packages

[Jens Troeger]

Hello,

I’m a bit confused about Protobuf package declarations and how they’re compiled to a Python package.

From the Python docs:

The Python code generated by the protocol buffer compiler is completely unaffected by the package name defined in the .proto file. Instead, Python packages are identified by directory structure.

Take Google’s Timestamp for example: the declared package is

package google.protobuf

When I install the googleapis-common-protos Python package, then the Timestamp class lives in google/protobuf/timestamp_pb2.py — so that’s the google.protobuf package, and the timestamp_pb2 module according to Python.

Looking at the information built into the compiled Python class, I see only the original Protobuf package information:

>>> from google.protobuf.timestamp_pb2 import Timestamp
>>>
>>> Timestamp.DESCRIPTOR.file.name
'google/protobuf/timestamp.proto'
>>> Timestamp.DESCRIPTOR.file.package
'google.protobuf'
>>> Timestamp.DESCRIPTOR.full_name
'google.protobuf.Timestamp'

Note that google.protobuf.Timestamp is actually not a valid Python package/module, and therefore it’s not possible (by default) to import classes based on their declared package names:

>>> importlib.import_module(Timestamp.DESCRIPTOR.full_name)
ModuleNotFoundError: No module named 'google.protobuf.Timestamp'

or the incorrect package is being imported:

>>> importlib.import_module(Timestamp.DESCRIPTOR.file.package)
<module 'google.protobuf' from '/path/to/.venv/lib/python3.10/site-packages/google/protobuf/__init__.py'>

which doesn’t actually contain the Timestamp class. That, in turn, makes it difficult to recursively walk and load nested Protobuf messages in Python!

So… what’s the recommendation here for my own Protobuf packages? Should I name them such that they match the Python package names (and thereby confuse other languages) or should I stick with Google’s example and skip the file name in the package declaration (i.e. the Python module)? Does it matter?

How do I programmatically walk the compiled Python classes for nested Protobuf messages?

Much thanks!

Jens

[Xuan Wang]

Hi, this is the group for gRPC specific discussions, since this question is related only to Protobuf, I'd suggest ask this in the Protobuf group.

Best,

Xuan

[Jens Troeger]

Thank you, Xuan, posted here.

Jens