How to use/distribute proto files in Python packages

[Shareef Jalloq]

Hi all,

this is more of a request for best practice advice.  

I'm trying to work out how to use proto files across projects.  I've seen examples of people suggesting to submodule in the proto files to all projects that use them.  That works nicely for most use cases where you have a pure Git repo but what do people do when they're trying to distribute a Python package that uses those proto files?

If I want to be able to set up CI jobs to push packages to an index, and have the Python *_pb2.py files generated as part of the flow, what do people do here?  Or should I be looking at running protoc separately and committing the resulting Python files before running the release flow?

Shareef.

[Shareef Jalloq]

At the moment I've started with the following directory structure:

repo/

    service/

        cpp/

        lib/

        python/

            service_pb2.py

            service_pb2_grpc.py

        service.proto

        CMakeLists.txt

and I've just tried adding an __init__.py to that python/ directory and pointing setuptools at service/python as the package_dir.  The issue then is that I get a module named service and I can import service.service_pb2_grpc.  However, service_pb2_grpc.py has an import service_pb2 instead of import service.service_pb2.

Not really sure how I can create a package using the auto-generated Python.

[Shareef Jalloq]

I can hack this to work by prepending sys.path but that feels horrible:

# __init__.py

import os, sys

sys.path.insert(0, os.path.dirname(__file__))

[Shareef Jalloq]

Looks like this has been an issue since 2014 with various work arounds and hacks.

https://github.com/protocolbuffers/protobuf/issues/1491