Cannot generate Python modules from containerd's API .proto files

[TheDiveO]

The containerd project doesn't publish a Python package for its containerd gRPC-based API, so I want to generate Python modules for it myself. For API version 1.3 of containerd, the .proto files can be found in https://github.com/containerd/containerd/tree/release/1.3/api. Unfortunately, I'm hitting two road blocks when trying to generate Python modules from the API .proto files and when to run them.

For example, the some of the containerd API .proto files reference protobuf plugin .proto files using an (for lack of a better term on my side) absolute import path: api/events/container.proto does an:

import weak "github.com/containerd/containerd/protobuf/plugin/fieldpath.proto";

However, the Python grpc compiler always wants to resolve such references in the local file system and containerd's source have ./protobuf/plugin/fieldpath.proto -- so this won't ever resolve correctly (using -I ...), because it lacks the github.lcom/containerd/containerd path elements. Please note that the Python grpc protocol compiler resolves such import paths in form of chopping down the full import path into parts and then searching its -I ... include directories.

Trying the lazy route by simply copying over these sources into inside vendor/github.com/... inside the containerd source tree will later cause runtime errors when trying to use the generated Python modules: this is because the grpc compiler considers the same containerd API .proto file in two locations (paths) to be separate instances. In consequence we get duplicate modules which unfortunately now try to register with grpc for the same protocol element names. Consequently, the gRPC Python runtime thus throws an error and terminates.

How can I correctly get this resolved when using python3 -m grpc.tools.protoc ...? What am I missing here or getting wrong?

[Richard Belleville]

Can you please include the runtime errors you're encountering?

[TheDiveO]

This is the error happening when trying to use the created packages on Ubuntu 20.10 on ARM64; the lsctr.py script is from an example script from the Github project https://github.com/siemens/pycontainerd, importing and using the generated Python3 modules:

File "lsctr.py", line 28, in <module>

    from containerd.services.namespaces.v1 import namespace_pb2_grpc, namespace_pb2

  File "/home/x/.local/lib/python3.8/site-packages/containerd/services/namespaces/v1/namespace_pb2_grpc.py", line 4, in <module>

    from containerd.services.namespaces.v1 import namespace_pb2 as containerd_dot_services_dot_namespaces_dot_v1_dot_namespace__pb2

  File "/home/x/.local/lib/python3.8/site-packages/containerd/services/namespaces/v1/namespace_pb2.py", line 16, in <module>

    from containerd.vendor.gogoproto import gogo_pb2 as containerd_dot_vendor_dot_gogoproto_dot_gogo__pb2

  File "/home/x/.local/lib/python3.8/site-packages/containerd/vendor/gogoproto/gogo_pb2.py", line 16, in <module>

    from containerd.vendor.google.protobuf import descriptor_pb2 as containerd_dot_vendor_dot_google_dot_protobuf_dot_descriptor__pb2

  File "/home/x/.local/lib/python3.8/site-packages/containerd/vendor/google/protobuf/descriptor_pb2.py", line 2005, in <module>

    'Location' : _reflection.GeneratedProtocolMessageType('Location', (_message.Message,), {

  File "/home/x/.local/lib/python3.8/site-packages/google/protobuf/internal/python_message.py", line 196, in __init__

    _AttachFieldHelpers(cls, field)

  File "/home/x/.local/lib/python3.8/site-packages/google/protobuf/internal/python_message.py", line 303, in _AttachFieldHelpers

    field_descriptor.GetOptions().packed)

  File "/home/x/.local/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 167, in GetOptions

    from google.protobuf import descriptor_pb2

  File "/home/x/.local/lib/python3.8/site-packages/google/protobuf/descriptor_pb2.py", line 130, in <module>

    _sym_db.RegisterEnumDescriptor(_FIELDDESCRIPTORPROTO_TYPE)

  File "/home/x/.local/lib/python3.8/site-packages/google/protobuf/symbol_database.py", line 108, in RegisterEnumDescriptor

    self.pool._AddEnumDescriptor(enum_descriptor)

  File "/home/x/.local/lib/python3.8/site-packages/google/protobuf/descriptor_pool.py", line 263, in _AddEnumDescriptor

    self._CheckConflictRegister(enum_desc, enum_desc.full_name, file_name)

  File "/home/x/.local/lib/python3.8/site-packages/google/protobuf/descriptor_pool.py", line 191, in _CheckConflictRegister

    raise TypeError(error_msg)

TypeError: Conflict register for file "google/protobuf/descriptor.proto": google.protobuf.FieldDescriptorProto.Type is already defined in file "containerd/vendor/google/protobuf/descriptor.proto". Please fix the conflict by adding package name on the proto file, or use different name for the duplication.

[TheDiveO]

After a deeper analysis and some trial+error I have now a working containerd Python API package. Bottom line was to comb through the process of generating a proper .proto files file structure of only those including vendored dependencies needed and then fixing the import paths to avoid any import path aliasing. The details can be found here: https://github.com/TheDiveO/pycontainerd/blob/master/script/genpb2.sh ... heavens, that containerd API and vendor tree really is ... challenging ... when generating Python modules.

As I can work with the unmodified existing containerd sources, I can now send a PR upstream to the Python containerd package project.