Python installation does not build plugin_pb2

[Louis-Marie]

Hi all,

It looks like python installation of protocol buffers does not
generate the google.protobuf.compiler.plugin_pb2 python file, while
google.protobuf.descriptor_pb2 is explicitly generated by
protobuf/python/setup.py

generate_proto("../src/google/protobuf/descriptor.proto")

Shouldn't the plugin.proto file be compiled and installed the same
way? Maybe I am missing something there, be I feel very uncomfortable
recompiling it when I need to write a plugin.

Thanks,

Louis-Marie

[Kenton Varda]

It's not generated because none of the python implementation actually uses it.  So, generating it and including it in the egg would just increase the library size for everyone, when most people don't need it.

What makes you feel uncomfortable about generating it yourself?

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[Louis-Marie]

Thanks for your answer.

To give you a little bit more information, here's what I'm trying to
do. I want to deliver a tool using a custom protoc plugin implemented
in python, so that end user can generate code from its own proto
files.

There would be nothing special to do before using this plugin, but
since it depends on plugin_pb2, I need to find the plugin.proto file
(using pkg-config), compile it, and put it in some appropriate
location.

Also, I can't put it in its "natural" parent python package
(google.protobuf) which is already provided by protobuf installation.
On the other side, the c++ code is generated (and I guess, compiled
into the shared library), which makes it a little bit more
straightforward to write c++ plugins.

What would be the best way to achieve what I'm trying to do? If the
python library size increase is not acceptable there, couldn't the
plugin_pb2 file be generated in an independent location, so that one
could still rely on it being present on any protobuf install?

Thanks for your advices,

Louis-Marie

2010/10/6 Kenton Varda <ken...@google.com>:

[Jason Hsueh]

As Kenton said, including plugin.proto would bloat the core library. Only people implementing proto compilers such as yourself need to use it.

On the C++ side, you would typically just build a statically linked binary that has all of the plugin generated code linked in. It's not included in the C++ core library either. There are various mechanisms to package python programs, e.g. http://www.pyinstaller.org/, that would allow you to give users a functioning plugin program without requiring that they install the generated modules for plugin.proto.

[Gregory Szorc]

I agree with Louis-Marie: I would like the compiler file(s) included in the Python package.

 

Every protoc plugin written in Python will need to import the google.protobuf.compiler.plugin_pb2 module. This means that any user wishing to use that plugin (it isn't limited to plugin developers themselves) will be out of luck with a default Python protobuf install, which omits this file.

 

There are a couple of workarounds (non-exhaustive list):

 

1) compiler plugin packages can copy generated code from plugin_pb2.py to their project/module and distribute it themselves.

2) Python plugins can tell end-users to manually compile and install plugin_pb2.py.

3) Don't use Python for the compiler plugin.

 

So, these workarounds require me to feel dirty that I'm copying code instead of using a library verbatim (1), take a risk that the produced plugin_pb2.py file won't change with a future release (1), force all users of my plugin to jump through extra steps (2), and/or write my compiler plugin in a language in which I'm not as proficient (3).

 

While it is true that the majority of installations will likely never utilize the compiler plugin module, for those that do, the barrier to entry can be increased and resolving the issue can be extremely frustrating, especially if compiler plugins don't have the documentation telling users how to resolve the import failure. Since it doesn't "just work," the experience tarnishes the image of the plugin and of protocol buffers itself.

 

The only argument against inclusion so far is package bloat. On my system, plugin_pb2.py and plugin_pb2.pyc are 6247 and 4750 bytes, respectively. The current uncompressed egg weighs in around 500K. Are we really objecting over a 11k/2% size increase? That seems trivial to me, especially since the added file(s) won't conflict with any existing ones and thus there is little risk to adding it.

 

While I would prefer that the Python installer be updated to include compiler.plugin_pb2, at the least could the documentation be updated to reflect a preferred workaround? (And I don't think forcing the use of a program packager like PyInstaller is the correct route either.)

 

Greg

[Kenton Varda]

I guess it would not be a big deal to include this code, since Python won't load it until it is explicitly imported anyway.  Can someone file a bug to have plugin_pb2.py included in the base installation for the next release?