protoc and python imports

[Alan Kligman]

I'm having a problem with protoc where python imports are not done
correctly. Here's the situation:

I have a directory structure like this:

proto/a.proto
proto/a/b.proto
proto/a/c.proto

a.proto provides some common definitions for both b.proto and c.proto.
I build the output like this:

protoc --proto_path=. --python_out=../dist *.proto
protoc --proto_path=. --python_out=../dist a/*.proto

assuming that proto is the current directory. Because a.proto is
included in both b.proto and c.proto, they both import it like this:

import "a.proto"; # relative to the current directory

After building the protobuf files with protoc, the resulting python
output has import statements for a_pb2.py that look like:

import a_pb2.py

which is wrong, because a_pb2.py is actually in the directory one
above b_pb2.py and c_pb2.py. Is there a way to get protoc to do this
properly? Is it a bug? Python2.5 handles relative imports, but there
is no nice way to do it in python2.4.

Thoughts?

[Kenton Varda]

I'm not sure I understand.  What would you expect the import line importing a_pb2 to look like?  My understanding is that Python imports are absolute, not relative to the importing file.

[Alan Kligman]

I need the line to look like:

from .. import a_pb2.py

The reason this is a problem is because I'm building the protocol
buffers into the middle of an existing project. The problem is that
protoc assumes that the output is either at the top of the package, or
that the related files are all in the same sub-package (which is
rarely true). Python2.5 supports relative intra-package imports (like
the one above). More details here: http://docs.python.org/tut/node8.html#SECTION008420000000000000000.

I think this is probably worth fixing. The workaround is to do some
post-processing on the output from protoc, which could get nasty.

On Oct 27, 5:44 pm, "Kenton Varda" <ken...@google.com> wrote:
> I'm not sure I understand. What would you expect the import line importing
> a_pb2 to look like? My understanding is that Python imports are absolute,
> not relative to the importing file.
>

[Kenton Varda]

The model used by the protocol compiler is to assume that the .proto files are located in a tree that parallels the Python package tree.  We don't want to get into relative imports because they can get complicated and error-prone.

If you don't want to put your .proto files into a tree matching your Python package tree, you could alternatively map them into such a tree virtually like so:

protoc --proto_path=mypkg=proto

This maps the virtual directory "mypkg" to the physical directory "proto".  You would then have to write your imports like:

  import "mypkg/a.proto"

You can also map individual files.

If this is insufficient then I guess we need a way to specify the python package explicitly in the .proto file, similar to the java_package option, rather than just inferring it from the location of the .proto file.

[Alan Kligman]

That works for me.

Cheers,
alan

On Oct 28, 2:40 pm, "Kenton Varda" <ken...@google.com> wrote:
> The model used by the protocol compiler is to assume that the .proto files
> are located in a tree that parallels the Python package tree. We don't want
> to get into relative imports because they can get complicated and
> error-prone.
> If you don't want to put your .proto files into a tree matching your Python
> package tree, you could alternatively map them into such a tree virtually
> like so:
>
> protoc --proto_path=mypkg=proto
>
> This maps the virtual directory "mypkg" to the physical directory "proto".
> You would then have to write your imports like:
>
> import "mypkg/a.proto"
>
> You can also map individual files.
>
> If this is insufficient then I guess we need a way to specify the python
> package explicitly in the .proto file, similar to the java_package option,
> rather than just inferring it from the location of the .proto file.
>