[protobuf] Issue 187 in protobuf: Command-line argument to override the "optimize_for" option

[prot...@googlecode.com]

Status: New
Owner: ken...@google.com
Labels: Type-Defect Priority-Medium

New issue 187 by dima: Command-line argument to override the "optimize_for"
option
http://code.google.com/p/protobuf/issues/detail?id=187

The project I'm working has C++ and Java code that need to share a few
.proto files. However, in C++, we only build the lite version of the
protobuf library, while the Java code needs to have the full version. Since
the "optimize_for" option can only be specified in the .proto files
themselves, that makes sharing of .proto files very difficult.

This is not specific to C++ or Java: in general, it may be useful to
codegen multiple source files with different optimization settings from the
same .proto file.

I suggest that protoc have an "--optimize_for" command-line argument that
overrides any such options in the .proto files. I'm attaching a patch that
solves this in a very hacky way; hopefully, someone will know the proper
way to fix this.


Attachments:
optimize_for.patch 2.3 KB

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[prot...@googlecode.com]

Comment #1 on issue 187 by dima: Command-line argument to override

the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

Actually, the patch is wrong... It overrides optimize_for if it's set
in .proto
files, but does not override the default value.

[prot...@googlecode.com]

Comment #2 on issue 187 by ni...@chromium.org: Command-line argument to

override the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

Would it be much more difficult to support arbitrary option overrides on
the command
line?

[prot...@googlecode.com]

Comment #3 on issue 187 by mike.weller: Command-line argument to override

the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

This is something I've always thought was needed. It makes much more sense
to me to
define these kinds of options on the command line so you can easily switch
them out
as part of a build process. This is especially useful when you are
targeting multiple
languages, as mentioned in the original post.

[prot...@googlecode.com]

Comment #4 on issue 187 by ken...@google.com: Command-line argument to

override the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

I agree, we should be able to override options on the command-line. The
only problem
is that it's unclear how far this support needs to go. Should you be able
to override
message-level and field-level options, or just file-level? Should an
override apply
to an individual file, or should it apply to all the files it imports,
too? In the
case of optimize_for, we'd probably want the override to apply to imports,
but for
something like java_package we probably don't.

[prot...@googlecode.com]

Comment #5 on issue 187 by esho...@fusionio.com: Command-line argument to

override the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

We also need this feature. We're sharing .proto files between applications.
One of the apps has a requirement to be optimized for code size and the
other apps need to be optimized for speed. I don't care if arbitrary option
overrides are supported but if they are it seems to make sense that they
only be for file-level options.

[prot...@googlecode.com]

Comment #6 on issue 187 by esh...@fusionio.com: Command-line argument to

override the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

What about an even more generic implementation where you can optionally poke
arbitrary strings into the file from the commandline? So, for example,
instead of
overriding options like optimize_for we could simply take a string from the
command
line and prepend it at the top of the file something like this:

$ protoc --prepend-string "option optimize_for LITE_RUNTIME;"

Obviously, this only helps when there is not already an optimize_for option
in the
file but this simple feature could handle lots of build issues like
optimization.
The user would have to be responsible for using it judiciously.

Comments?

[prot...@googlecode.com]

Comment #7 on issue 187 by ken...@google.com: Command-line argument to

override the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

If you just want to append a string to the file, you can easily do that
like:

cp foo.proto foo2.proto
echo 'option optimize_for LITE_RUNTIME;' >> foo2.proto
protoc foo.proto

Any reasonable build system should allow you to do this without changing
protoc.

[prot...@googlecode.com]

Comment #8 on issue 187 by dima: Command-line argument to override

the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

Now suppose foo.proto is imported by a few other .proto files... You'll
have to modify each one of them, too.

Every other compiler - gcc, javac, etc. - lets the user specify the
optimization level on the command-line. Why not protoc?

[prot...@googlecode.com]

Comment #9 on issue 187 by ken...@google.com: Command-line argument to

override the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

I agree that protoc should provide something. I was merely pointing out
that eshober's suggestion can trivially be implemented without protoc's
help.

Note that the optimization option for gcc and javac is very different from
lite vs. non-lite. When you compile a C++ file with -O2, the compiler
doesn't need to know what optimization level was used for all the other
files you intend to link against -- it doesn't make a difference. But
protoc needs to know not just whether the file it is compiling is using
LITE_RUNTIME, but also whether all the files imported by that file are
using it.

This makes the problem significantly more complicated, because if you
specified the optimization options for each of those imports on the command
line when you compiled them, then you have to somehow express the same
information on the command line for the dependency. This could get really
awkward:

protoc foo.proto --option=foo.proto=optimize_for:LITE_RUNTIME
--option=some_other_package/bar.proto=optimize_for:CODE_SIZE
--option=yet_another/baz.proto=optimize_for:SPEED

To make matters more complicated, imagine this situation: You are linking
against a library which includes foo.proto. The library compiles in this
proto in lite mode. You, however, want to import the proto into a file
that is compiled in regular mode (and you actually need reflection, so you
can't just switch to lite). This means you need to compile a copy of
foo.proto into your binary, compiled with different options. Because you
are linking against a library that already has the lite version compiled
in, you need to make sure that your copy doesn't have conflicting symbol
names, and so you probably want to alter the package name too. But, you
don't want to maintain your own parallel foo.proto with these changes --
you really want to take the one from the other project and apply some
command-line options to alter it.

What sort of command-line syntax would support this?

My concern is that a lot of people think this problem is trivial, but don't
realize that the trivial solutions they are thinking of don't actually
work, or only work for a smallish subset of cases. I don't want to
introduce something that we'll just have to immediately deprecate and
replace with something more general.

[prot...@googlecode.com]

Comment #10 on issue 187 by esh...@fusionio.com: Command-line argument to

override the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

Admittedly, my suggestion above does not handle imports and that makes it
mostly unusable whether done in or outside of protoc. The assumption on my
part was that the line inserted would affect all imported files like a C
header include directive which simply inserts the text from the included
file at that spot.

The only thing that makes sense to me and that doesn't get too complicated
is that we add a command-line optimization option override that applies to
the specified file and all imported files.

protoc foo.proto --optimize_for=LITE_RUNTIME

I would need to use this to create a lite and a non-lite library so that my
applications can choose which to link in without duplicating .proto files.

[prot...@googlecode.com]

Comment #11 on issue 187 by ken...@google.com: Command-line argument to

override the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

Hi all,

Attached is a little protoc plugin written in Python which converts all the
inputs to LITE_RUNTIME -- including renaming so that they do not conflict
-- and then passes them on to some other plugin. See readme.txt in the
archive for details. Basically with this you can do:

protoc --litify_out=cpp:. foo.proto

And this will produce foo_lite.pb.h and foo_lite.pb.cc, which contain foo's
types compiled with optimize_for=LITE_RUNTIME. You can do Java too:

protoc --litify_out=java:. foo.proto

You can easily modify this plugin's code to produce any arbitrary
transformation you want. It's just a Python script that processes the
FileDescriptorProtos representing the inputs.

I like this approach because it is completely general. There is nothing
that you can't express in a Turing-complete language. In contrast, *none*
of the other proposals made so far can express even what this simple plugin
does (in particular no one has proposed a solution for the renaming part).

I also like that this requires no modification whatsoever to protoc. Not
just because I want to avoid work, but because keeping the code code as
simple as possible ensures that it retains agile.

What do you think?

Attachments:
litify.tar.gz 4.1 KB

[prot...@googlecode.com]

Comment #12 on issue 187 by ken...@google.com: Command-line argument to

override the "optimize_for" option
http://code.google.com/p/protobuf/issues/detail?id=187

s/code code/core code/

s/retains/remains/

*sigh*

[prot...@googlecode.com]

Comment #14 on issue 187 by austin.l...@gmail.com: Command-line argument to
override the "optimize_for" option
https://code.google.com/p/protobuf/issues/detail?id=187

I was able to use Keaton's plugin as a starting point and am using it now.
I didn't know that the protobuf compiler was that extensible. Way cool.

FYI, it is less lines of code to write the entire plugin in C++ and
directly call the C++ generator from there, though it is a bit trickier to
get it right.

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings