"Lite" mode is in SVN -- mostly

[Kenton Varda]

I just committed the new "lite mode" refactoring for C++ and Java to SVN.

To use the new feature, add this line to your .proto file:

  option optimize_for = LITE_RUNTIME;

This will cause protoc to generate code which is just as fast as with optimize_for = SPEED, but lacks descriptors, reflection, UnknownFieldSet, and features which are tied to them.  This means the code can be linked against a much smaller subset of the protocol buffers runtime library and will not spend time at start-up building descriptors.

Currently, the C++ Makefile will compile libprotobuf and libprotobuf-lite as independent, stand-alone libraries, even though the latter is a subset of the former.  Is this what we want?  I'm not sure.  If we make libprotobuf depend on libprotobuf-lite, then users who want the full library will have to specify *both* when linking their own apps, unless we provide a "protobuf-config" script or something like that.  Complicating things further, if you app does not use extensions, then extension_set.cc can be removed from the lite library to make it even smaller, but having three separate libraries just seems excessive!

For Java, I have not yet updated the Maven POM to separate the libraries.  I'm not sure how -- Greg, can you work on this?  "Lite" messages need only the following classes:

  AbstractMessageLite.java

  ByteString.java

  CodedInputStream.java

  CodedOutputStream.java

  ExtensionRegistryLite.java

  FieldSet.java

  GeneratedMessageLite.java

  InvalidProtocolBufferException.java

  Internal.java

  MessageLite.java

  UninitializedMessageException.java

  WireFormat.java

Numbers

C++, as measured by the "size" command on Linux:

  libprotobuf.so: 948k

  libprotobuf-lite.so: 148k

  libprotobuf-lite.so with extension_set.cc removed: 91k

Java, measured by totalling the sizes of the .class files:

  full library:  904k

  lite library:  120k

I suspect we can further reduce some of these numbers with some more work.  For instance, an extension-less Java lite library might be as much as 50k smaller, though will require a bit more work than simply yanking some classes.

[ijuma]

Hi Kenton,

On Jul 29, 3:50 am, Kenton Varda <ken...@google.com> wrote:
> I just committed the new "lite mode" refactoring for C++ and Java to SVN.

Thanks, that sounds useful. I noticed another change in the
Changes.txt:

"Put Builder objects on a freelist after build() is called, so they
may be reused later."

Since the diff is quite huge, I was not able to find where this change
was done. If you could point out where I should look, that would be
appreciated.

Best,
Ismael

[Gregory Kick]

I'll try to get to this by the end of the week.

--
Greg Kick
Data Liberation & Feedburner
Sent from Chicago, IL, United States

[Kenton Varda]

I think that touched java_message.cc in the compiler -- grep for "QuickQueue".  It's an entirely transparent change -- you do not have to change your code at all to take advantage of it.  I'm told it improved performance on Dalvik (Android's JVM), which has a relatively weak garbage collector.  I doubt it helps any on the official JVM.

[Kenton Varda]

On Tue, Jul 28, 2009 at 7:50 PM, Kenton Varda <ken...@google.com> wrote:

C++, as measured by the "size" command on Linux:

  libprotobuf.so: 948k

  libprotobuf-lite.so: 148k

  libprotobuf-lite.so with extension_set.cc removed: 91k

I just shaved another 23k off the lite library by eliminating use of strutil.  So now it's 125k or 68k without extensions.

[ijuma]

On Jul 29, 7:15 pm, Kenton Varda <ken...@google.com> wrote:
> I think that touched java_message.cc in the compiler -- grep for
> "QuickQueue".

Thanks Kenton.

>  It's an entirely transparent change -- you do not have to
> change your code at all to take advantage of it.

Yes, I understand. I was just curious about the implementation and the
motivation for the change. The answer for the first is that it uses a
thread-local queue and you gave the answer for the second question.

> I'm told it improved
> performance on Dalvik (Android's JVM), which has a relatively weak garbage
> collector.  I doubt it helps any on the official JVM.

Is there a way to disable this behaviour? As you say, it doesn't seem
particularly useful for sophisticated JVMs and unless I am missing
something, the queue is currently unbounded so it could end up
retaining quite a bit of memory for some usage patterns.

Best,
Ismael

[Kenton Varda]

On Wed, Jul 29, 2009 at 3:41 PM, ijuma <ism...@juma.me.uk> wrote:

Is there a way to disable this behaviour? As you say, it doesn't seem

particularly useful for sophisticated JVMs and unless I am missing
something, the queue is currently unbounded so it could end up
retaining quite a bit of memory for some usage patterns.

There's no way to disable it at present, but I would be fine with adding a way.  In fact I think this should probably be off by default.

 

Best,
Ismael

[Jon Skeet]

On Jul 29, 3:50 am, Kenton Varda <ken...@google.com> wrote:

> I just committed the new "lite mode" refactoring for C++ and Java to SVN.
> To use the new feature, add this line to your .proto file:
>   option optimize_for = LITE_RUNTIME;
> This will cause protoc to generate code which is just as fast as with
> optimize_for = SPEED, but lacks descriptors, reflection, UnknownFieldSet,
> and features which are tied to them.  This means the code can be linked
> against a much smaller subset of the protocol buffers runtime library and
> will not spend time at start-up building descriptors.

Just in case anyone's wondering, I'm hoping to at least *start* work
on porting this to C# early next week. (I have two days dedicated to
20% time next week, but I also want to do some Wave work.)

Out of interest Kenton, does this make the bootstrapping code simpler?
I'd imagine that can be built with just a "lite" version. It would be
nice to get rid of some of the nastiness that's involved in C# just to
get the PB-specific types to work :)

Jon

[Kenton Varda]

On Sat, Aug 1, 2009 at 4:42 AM, Jon Skeet <sk...@pobox.com> wrote:

Out of interest Kenton, does this make the bootstrapping code simpler?

I'd imagine that can be built with just a "lite" version. It would be
nice to get rid of some of the nastiness that's involved in C# just to
get the PB-specific types to work :)

In theory, descriptor.proto could become "lite", and this might simplify bootstrapping. In practice, though, we actually use "heavy" features on the types in descriptor.proto, which is part of the reason why bootstrapping is so complicated.  So, no, I don't think that would work.

 

Jon