Debug output for protobuf's MessageLite

[Mathieu Perreault]

Hi, 

We are moving to use protocol buffers instead of XML for one of our client/server interactions in Chrome (yes!). While certainly more efficient, it seems like the optimize_for = LITE_RUNTIME; option we use in Chrome's proto files gives us a MessageLite subclass which has no way of outputting a human-readable format (DebugString and using TextFormat only seem to work for Message).

The team is using human-readable debug output quite a bit so I'd like to keep supporting it, any ideas?

Thanks

[Peter Kasting]

Is life any better with an up-to-date version of protobufs and/or proto3?  I know some changes have been made to the lite runtime there.

If so, then the right answer is probably to upgrade our protobufs library, which we want to do anyway.  I can help with the expertise if someone can help with the putting-the-time-in.

PK 

[Ilya Sherman]

I'm not sure about proto3, but I don't think there's any way to preserve debug output using MessageLite with the current code.  If the goal is to have debug output available for local debugging, it's possible that we could generate a full Message in Debug builds, and MessageLite in Release builds. 

[Peter Kasting]

On Thu, Jan 14, 2016 at 5:37 PM, Ilya Sherman <ishe...@chromium.org> wrote:

On Thu, Jan 14, 2016 at 4:27 PM, Peter Kasting <pkas...@chromium.org> wrote:

On Thu, Jan 14, 2016 at 4:25 PM, Mathieu Perreault <ma...@chromium.org> wrote:

Hi, 

We are moving to use protocol buffers instead of XML for one of our client/server interactions in Chrome (yes!). While certainly more efficient, it seems like the optimize_for = LITE_RUNTIME; option we use in Chrome's proto files gives us a MessageLite subclass which has no way of outputting a human-readable format (DebugString and using TextFormat only seem to work for Message).

The team is using human-readable debug output quite a bit so I'd like to keep supporting it, any ideas?

Is life any better with an up-to-date version of protobufs and/or proto3?  I know some changes have been made to the lite runtime there.

If so, then the right answer is probably to upgrade our protobufs library, which we want to do anyway.  I can help with the expertise if someone can help with the putting-the-time-in.

I'm not sure about proto3, but I don't think there's any way to preserve debug output using MessageLite with the current code.

Does "current code" mean "the current trunk version of proto2" distinct from what's in the Chromium tree?  That's what I was asking about, since the current Chromium import is years out of date.

 

  If the goal is to have debug output available for local debugging, it's possible that we could generate a full Message in Debug builds, and MessageLite in Release builds. 

That might need significant changes to how our protobuf buildfiles are set up, as we go to some length to not use the full version of protobufs in them.

PK

[Ilya Sherman]

On Thu, Jan 14, 2016 at 5:42 PM, Peter Kasting <pkas...@chromium.org> wrote:

On Thu, Jan 14, 2016 at 5:37 PM, Ilya Sherman <ishe...@chromium.org> wrote:

On Thu, Jan 14, 2016 at 4:27 PM, Peter Kasting <pkas...@chromium.org> wrote:

On Thu, Jan 14, 2016 at 4:25 PM, Mathieu Perreault <ma...@chromium.org> wrote:

Hi, 

We are moving to use protocol buffers instead of XML for one of our client/server interactions in Chrome (yes!). While certainly more efficient, it seems like the optimize_for = LITE_RUNTIME; option we use in Chrome's proto files gives us a MessageLite subclass which has no way of outputting a human-readable format (DebugString and using TextFormat only seem to work for Message).

The team is using human-readable debug output quite a bit so I'd like to keep supporting it, any ideas?

Is life any better with an up-to-date version of protobufs and/or proto3?  I know some changes have been made to the lite runtime there.

If so, then the right answer is probably to upgrade our protobufs library, which we want to do anyway.  I can help with the expertise if someone can help with the putting-the-time-in.

I'm not sure about proto3, but I don't think there's any way to preserve debug output using MessageLite with the current code.

Does "current code" mean "the current trunk version of proto2" distinct from what's in the Chromium tree?  That's what I was asking about, since the current Chromium import is years out of date.

I think so?  I glanced through the developer.google.com docs and didn't find anything that seemed promising, though I'm not 100% sure what version of protobufs I was looking at.

  

  If the goal is to have debug output available for local debugging, it's possible that we could generate a full Message in Debug builds, and MessageLite in Release builds. 

That might need significant changes to how our protobuf buildfiles are set up, as we go to some length to not use the full version of protobufs in them.

I seem to recall locally hacking this in a few years back, and having not too much trouble with it.  It was long enough ago to where I don't remember the details, though.

 

PK

[Peter Kasting]

Could be.  I haven't tried to do it.

We do have a "protobufs full do not use" build target that we build in order to link against protoc, so the files at least exist in our tree.

PK 

[Mathieu Perreault]

Thanks, 

I do like the idea of generating Message protos for Debug, and MessageLite for Release. 

I found a discussion on the protobuf project on a command-line flag to override the lite annotation, but sadly it seems like it was never implemented (the feature request came back recently). 

In the current state, it looks like once the "optimize_for = LITE_RUNTIME" annotation is there, it's hard to tell the compiler to generate anything else than MessageLite.

One thing I see, would be a pre-build step which would strip the "optimize_for = LITE_RUNTIME" annotation in Debug builds. Perhaps in tools/protoc_wrapper.py which is called from protoc.gypi/proto_library.gni?

[Mathieu Perreault]

https://codereview.chromium.org/1592203002/ is a draft of what removing LITE optimizations in Debug builds* could look like (only implemented for gyp, not cleaned up).

Callers could use something like  DVLOG(1) << "Debug output: " << my_proto.DebugString(); which would not cause compile-errors in non-Debug builds. 

*Note: It's probably too heavy-handed to remove LITE optimizations on all protos, so perhaps it could be a variable of some kind that individual targets could set.

[Alexei Svitkine]

What's the cost of removing LITE optimizations for all protos? Is it compile speed? Binary size? Runtime performance?

[Mathieu Perreault]

I seem to remember that it was a concern on Android that the full "Message" would generate too many methods (there's a method limit), hence these Lite protos (and even mention of nano protos at one point). +nyquist since I vaguely remember him knowing about this. I could also see binary size being a component of this. 

[Scott Hess]

Also, I think the full protobuf library allows a disturbing amount of abstract introspection.  We could have guidelines to prevent such things from creeping in, but Lite enforces that.

-scott

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Tommy Nyquist]

Yeah, for protobuf messages used only in Java, we use nano. It's from this repository: https://android.googlesource.com/platform/external/protobuf.git

There is a special version of protoc and runtime from that repo. For details, see: proto_java_library GN template

For details on the switch from lite to nano see this bug: crbug.com/377891 (for Googlers only, sorry).

[Nathan Parker]

Bringing back this thread...

I'm running into a similar problem with the fact that we can't parse text protos with the MessageLite build.  Q: Why not allow different optimize_for settings on a per-proto basis? Would that not strike a balance between code size and functionality?

Use case: I'd like to create a data resource by putting a human-readable proto in the chrome/browser/resources/ dir, to be included in release builds.  I can't currently parse it at run-time with MessageLite.  I could convert it to a binary file at build time, but I'm not clear how to insert that into the build infrastructure.   Ideas?

 -- Nathan

[Alexei Svitkine]

If this will be parsed by production Chrome, it would be more efficient to have the resource be binary.

So then, as you said, you just need to set up the build to do that. Here's an example of how we produce a generated source file somewhere else:

https://code.google.com/p/chromium/codesearch#chromium/src/chrome/chrome_resources.gyp&l=297

Basically, that runs a Python program with some parameters and input files to generate the files listed in 'outputs'.

[林Gg]

You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.