no difference between clean and incremental buildings of java_library() ?

[Leo Yin]

Hi,

On a machine of 4 CPUs/4G MEM, I am using bazel of version 0.22 to build a Java project. One of its main compiling job is to compile about 2000 java files of 500,000 LOCs to one jar file.

java_library() is used to do the job and it takes about 20s for a clean build ('bazel clean' and 'bazel build //:compile_target').

However, I found for a incremental compiling (modify one java file and then execute 'bazel build //:compile_target') it also takes about 20s.  I tried many times and always got the similar results.

It seems there is no difference between clean and incremental buildings of java_library(). Is that true and everything ok?

Thank you very much.

-Leo

[Kristina Chodorow]

That is very surprising.  You could try using --explain (http://bazel.io/docs/bazel-user-manual.html#flag--explain) or profiling (https://www.kchodorow.com/blog/2015/09/18/build-y-u-go-slow/) to figure out why it's taking so long for an incremental build.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/fcc6c2aa-e259-48bb-bb50-67658924d8d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Leo Yin]

Hi Kristina,

The explain and profiling did not give much information. They just showed the incremental compiling took about 16s.

explain result：

Build options: --explain=ex --verbose_explanations --profile=pf.incr

Executing action 'BazelWorkspaceStatusAction stable-status.txt': unconditional execution is requested.

Executing action 'Building libreporter.jar (2088 source files)': One of the files has changed.

profile result：

Critical path (16.742 s):

    Id        Time Percentage   Description

    76    16.742 s  100.00%   Building libreporter.jar (2088 source files)

-Leo

在 2016年5月5日星期四 UTC+8下午10:57:29，Kristina Chodorow写道：

[Damien Martin-guillerez]

Hi,

You have only one, big library, so it calls javac with the 2000 java files and cannot do incremental build. You want to split into several java_library so Bazel rebuild only the library that needs it.

Also, you might want to use the java workers to speed things up: http://bazel.io/blog/2015/12/10/java-workers.html

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/006d6117-e537-4ab4-b0ea-d750c8f23292%40googlegroups.com.

[Leo Yin]

For the incremental compiling, I used '--subcommands' option to see the which commmands are execute:

____Loading...

____Found 64 targets...

____[0 / 1] BazelWorkspaceStatusAction stable-status.txt

____[1 / 2] Building libreporter.jar (2088 source files)

>>>>>>>>> # //:reporter [action 'Building libreporter.jar (2088 source files)']

(cd /root/.cache/bazel/_bazel_root/172c3c523723d1ac9fd40665cdc272a1/SPA_V3R5_BUILD && \

  exec env - \

  external/local_jdk/bin/java -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac.jar -client -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/local_linux-fastbuild/bin/libreporter.jar-2.params)

____[1 / 2] Still waiting for 1 job to complete:

      Running (sandbox):

        Building libreporter.jar (2088 source files), 10 s

____From Building libreporter.jar (2088 source files):

Note: Some input files use or override a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

Note: Some input files use unchecked or unsafe operations.

Note: Recompile with -Xlint:unchecked for details.

____Building complete.

____Elapsed time: 17.563s, Critical Path: 17.00s

The content of file bazel-out/local_linux-fastbuild/bin/libreporter.jar-2.params is as follows.

   1 --classdir

   2 bazel-out/local_linux-fastbuild/bin/_javac/reporter/libreporter_classes

   3 --tempdir

   4 bazel-out/local_linux-fastbuild/bin/_javac/reporter/libreporter_temp

   5 --output

   6 bazel-out/local_linux-fastbuild/bin/libreporter.jar

   7 --sourcegendir

   8 bazel-out/local_linux-fastbuild/bin/_javac/reporter/libreporter_sourcegenfiles

   9 --output_manifest_proto

  10 bazel-out/local_linux-fastbuild/bin/libreporter.jar_manifest_proto

  11 --compress_jar

  12 --output_deps_proto

  13 bazel-out/local_linux-fastbuild/bin/libreporter.jdeps

  ...

  18 --sources

  19 systemAnalyzer/reporterSrc/src/main/FE/FE-BE-adapter/java/com/huawei/ism/adapter/AdapterUtils.java

  20 systemAnalyzer/reporterSrc/src/main/FE/FE-BE-adapter/java/com/huawei/ism/adapter/ApplicationContextHolder.java

  21 systemAnalyzer/reporterSrc/src/main/FE/FE-BE-adapter/java/com/huawei/ism/adapter/GenerateDefaultMoPreProcessor.java

  22 systemAnalyzer/reporterSrc/src/main/FE/FE-BE-adapter/java/com/huawei/ism/adapter/InitServlet.java

  23 systemAnalyzer/reporterSrc/src/main/FE/FE-BE-adapter/java/com/huawei/ism/adapter/LegoSystemBuilder.java

  24 systemAnalyzer/reporterSrc/src/main/FE/FE-BE-adapter/java/com/huawei/ism/adapter/ListenerInitializeCallback.java

  25 systemAnalyzer/reporterSrc/src/main/BE/Lego-Core-Log-Bundle/com/huawei/lego/core/log/DateFormatUtil.java

  ...

2104 systemAnalyzer/reporterSrc/src/main/dynamictablepro/com/huawei/ism/report/tablemanager/ResourceTypeUtil.java

2105 systemAnalyzer/reporterSrc/src/main/dynamictablepro/com/huawei/ism/report/tablemanager/TableChecker.java

2106 systemAnalyzer/reporterSrc/src/main/dynamictablepro/com/huawei/ism/report/tablemanager/UserAclUtil.java

2107 --javacopts

2108 -source

2109 8
  ...

All the java files (2088) are listed in the file bazel-out/local_linux-fastbuild/bin/libreporter.jar-2.params. 

Is that the reason that the incremental compiling took the similar time as a clean compiling?

I also performed some incremental compilings for other java_library() rules of small numbers of java files. The experiments showed "*.jar-2.params" files always contained the whole source files.

-Leo

在 2016年5月6日星期五 UTC+8下午5:34:31，Leo Yin写道：

[Leo Yin]

Hi Damien,

Yes, the java workder can improve the compiling speed about 60%~80% for the project. Thank you very much!

For small libraries (only containing dozens of java files), it seems bazel cannot do real incremental build yet -- bazel-out/local_linux-fastbuild/bin/libXXX.jar-2.params file always contains ALL the source files of the library for the option '--sources'.

-Leo

在 2016年5月6日星期五 UTC+8下午5:41:06，Damien Martin-guillerez写道：

[Damien Martin-guillerez]

Yes. Bazel does not use an incremental java compiler for various reason (the most important is that none of those are really reproducible). Instead the solution is to split the java compilation in many small libraries so that bazel only rebuild the one that have changed.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/1a97a13f-5e93-4de2-8f48-0e544affd534%40googlegroups.com.

[Leo Yin]

Understood. Thank you very much~

在 2016年5月6日星期五 UTC+8下午7:03:27，Damien Martin-guillerez写道：

[pauld...@gmail.com]

> Bazel does not use an incremental java compiler for various reason

https://groups.google.com/d/msg/bazel-discuss/3iUy5jxS3S0/smqhBOa0AQAJ seems to suggest otherwise.

"we've done (hundreds of?) thousands of builds with the incremental Java compiler, and we haven't encountered any correctness issues recently. However, back when I experimented with it, Javac was using second-precision timestamps to detect file changes, even on file systems that actually support sub-second timestamps, and this was causing issues in some scenarios."

What's that about?