Issue 579 in protobuf: Java code fails to compile with error "code too large"

[prot...@googlecode.com]

Status: New
Owner: liu...@google.com
Labels: Type-Defect Priority-Medium

New issue 579 by daniel.m...@crowdstrike.com: Java code fails to compile
with error "code too large"
http://code.google.com/p/protobuf/issues/detail?id=579

What steps will reproduce the problem?
1. Have a single .proto file that through additions over time grows to have
800 or 900 different "message"s in it.
2. Use protoc to turn this into java without any special options beyond
--java_out
3. Try to compile the resulting monster java file.

What is the expected output? What do you see instead?
I expected the file to compile as had always happened in the past. Instead,
yesterday morning I was greeted by this error from our continuous build:

02-Dec-2013 15:01:58 javac -sourcepath generated_code -d
generated_code/target -J-Xmx512M -source 1.7 -target 1.7 -cp
/usr/share/java/protobuf-java.jar:/opt/local/share/java/protobuf.jar
@generated_code/sources.list
02-Dec-2013 15:02:27 generated_code/our/company/name/OurEvents.java:591040:
error: code too large
02-Dec-2013 15:02:27 public com.google.protobuf.ExtensionRegistry
assignDescriptors(
02-Dec-2013 15:02:27 ^
02-Dec-2013 15:02:27 1 error
02-Dec-2013 15:02:27 make: *** [javac-protobufs] Error 1

The issue is that the java generated by the proto compiler does all of its
static initialization in one method. (either a single static block, or a
single "assignDescriptors" method). This means that it is quite easy for
the proto compiler to generate a static initialization method so large that
it is larger than 65535 bytes of bytecode when compiled; this hits a
hard-coded limit in the jvm, and we get this error. Note that at this
point, nothing about the generated java class is even close to any other
jvm limits except the size of this one static initialization method.

What version of the product are you using? On what operating system?
This happened with protoc 2.4.1; indeed, with 2.5.0 we don't hit the "code
too large" error yet, but we probably will by next summer - with 2.5.0 the
static initialization method is already over 61K bytes of bytecode.

Please provide any additional information below.

I have attached a patch to the latest version of the proto compiler which
avoids this issue by splitting excessively large static initialization
blocks into multiple methods. It does not change any of the public
interface of the generated java code at all; all changes are implemented as
new private methods.

In our environment, we are currently using a similar change, but have
implemented it as a python post-process hack on the java code generated by
protoc. Obviously, it would be more robust if this change were integrated
into protoc instead of operating as a bit of regular expression-based
guesswork.

Note that the java code protoc currently generates already splits embedded
strings to avoid a similar "code too large" compilation error.

Attachments:
proto-code-too-large-fix.patch 9.1 KB

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[prot...@googlecode.com]

Comment #1 on issue 579 by cbsm...@gmail.com: Java code fails to compile

with error "code too large"
http://code.google.com/p/protobuf/issues/detail?id=579

At 25% your message count, this seems like a great future proofing patch.
Thanks!

[prot...@googlecode.com]

Comment #2 on issue 579 by daniel.m...@crowdstrike.com: Java code fails to

compile with error "code too large"
http://code.google.com/p/protobuf/issues/detail?id=579

I have improved the patch, and it is attached. This patch splits static
initialization code more precisely, by splitting based on an estimate of
bytecode size rather than a simple count of the number of statements.

Attachments:
proto-code-too-large-fix-v2.patch 9.7 KB

[techv...@gmail.com]

I am facing the same issue. Can you please share the proto-code-too-large-fix.patch file?