On Sun, Apr 28, 2013 at 11:05 AM, Dariusz Luksza
>> available in the build. This allows us to list our dependencies in the
>> BUCK build files, and Buck gets what is needed for the target(s) you
>> asked it to build.
> Checking out jars into Git feels like step backward, not mentioning
> that Gerrit's git repo will explode in size.
Both of which are why we won't check JARs into Git. Instead they are
obtained automatically during the build.
> Side question: Did you consider other build systems? eg. Leiningen,
> the build system for Clojure? AFAIK it is highly expendable and can
> work with maven repositories, maybe it will not be as fast as Buck,
> but it already have support for Eclipse. Another pros for Lainingen is
> maturity and community build around it. The down side could be Clojure
> it self with is 'moder lisp for JVM'
>  http://leiningen.org/
Never heard of Leiningen before. Maybe because its build language is
based on Lisp. I don't want to write my build specifications in Lisp.
Its bad enough we use Java, GWT and Prolog in Gerrit. Adding Lisp for
the build process is, well, one language too many in a relatively
This of course is also a counter-argument for Buck, Gradle and buildr.
Each one uses a language for the build files. At least Buck is such a
tiny subset of Python that you never need to really learn or worry
about Python. buildr seems to force you to learn some Ruby to grok the
> For me it should be easy and work 'out of the box' without multi pages
> long howto's and additional configuration/software.
I'm not sure git clone is hard. But sure if you want it to be, you can
make git clone hard to use.
buildr's setup is more difficult. It actually scared me away from
trying a buildr prototype build.
>> Buck allows us to define very fine-grained targets around specific
>> sections of the code. E.g. gerrit-server can be broken down into
>> smaller subsets with just a few lines of BUCK file changes. We can use
>> this to further reduce the surface of APIs we expose to plugins, which
>> in turn makes the plugin API faster because the final JAR it creates
>> is smaller. Making this segmentation costs almost nothing to the
>> overall build time.
> Modularity is good, but to certain extent. Small modules tend to have
> circular dependencies and adds confusion where given interface/class
> should belong to.
True. Within Google we use an approach similar to what I described.
Modules are typically along Java package boundaries. Almost everything
below some package name is part of the same module in the build
system. There is no issue with circular dependencies because you put
related code into the same package, its the same directory in the
source tree, its easy to reason about.
The small modules let you break the module down into smaller segments
if you need to. E.g. you could make a target from the package that
consists only of the public interface files, and another that also has
the implementation. Client code links to the interface target while
the server binary can point to the implementation target, which
includes the interface target by transitive dependency:
INTERFACES = ["Fooer.java", "BarMaker.java"]
name = "interface",
src = INTERFACES,
name = "impl",
src = glob(["*.java"], excludes = INTERFACES),
deps = [":interface"],
Just try doing that in Maven, or most other build systems actually.
Its hard to be that fine-grained when you want to do it. :-)
>> Doing the same thing in Maven requires us to setup a new top level
>> directory with a bunch of new configuration files (.gitignore,
>> .settings/, pom.xml) and adds several seconds to the build while Maven
>> figures out how to parse that XML, start the necessary plugins, build
>> the JAR. It also further clutters the source tree. We can't just
>> partition on the Java package names.
> Good point, I agree with this.
I want to increase the modularity of the server, but doing the source
code tree refactorings to please Maven scares me. So I just keep not
doing it. Maven is actively keeping me from making changes that are
probably valuable to the long-term health of the source tree. Yay--.
>> With Buck I could eventually have a very clean source tree with
>> everything under "src/com/google/gerrit/..." and place BUCK files down
>> within that Java package hierarchy next to each Java package we want
>> to build as its own unit. This gives you a very natural mapping of
>> Java class name to location on the filesystem (no more "is that in
>> -common? -reviewdb? why?" questions). Try that with Maven sometime.
>> You will go screaming for the hills, I just don't think its possible.
> How this 'BUCK file' will figure out dependencies for given
> package-module. Or maybe more general question, does Buck support
> transient dependencies?
Buck automatically handles transitive dependencies within the BUCK
files you declared. So it knows gerrit-server needs gerrit-reviewdb,
which needs gwtorm.
> Personally I don't want to bother with
> dependency of my dependency, I know that maven stores pom files in
> .m2/repository to figure out on their own, does Buck how similar
It doesn't follow m2 data, because it doesn't know anything about
Maven POM files. Buck needs everything declared in the BUCK files in
our Git repository.
Which is actually OK because we dive into every Maven POM and examine
the dependency tree to check licenses before we include something. In
the case of openid4java we have to explicitly drop some dependencies
because we can't use them under the license they offer.
As it turns out this exercise looking through our deps for Buck has
taught me we have a number of very risky/broken dependency chains in
Gerrit. Did you know that Guice seems to be linking to asm3 while
Gerrit itself uses asm4? Did you know that Gerrit could break if these
wind up in the wrong order? No Maven is not helping with this. It
happens 3 levels deep in transitive dependencies from POM files.
>> I say eventually here because step 1 is to replace Maven. Step 2 is to
>> refactor the source tree into a more clean format. At least with Buck
>> there is no need to do these two steps as one big bang. Its not quite
>> like Maven where it ties your hands and (roughly) demands you follow
>> its rules... or else.
> I'm with you here... would even add third step 'convert modules to
> OSGi bundles' ;)
Yes, based on the recent plugin work.. I am reconsidering the idea of
bundling an OSGi container and having Gerrit plugins managed by an
>>> In case of build performance, as far as I see we spent most of time
>>> compiling GWT code and replacing build system will not solve this problem.
>> Sure. But we run two copies of the GWT compiler now. And Maven doesn't
>> do anything in parallel. At least Buck will run steps in parallel
>> where it can.
> AFAIK GWT compiler since version 1.6 have option 'localWorkers' which
> allows you utilize all cpu's/core's so running two compilers in
> parallel doesn't really make sense.
For GWT yes. But it also depends on how may cores you have. If you
have say 16 CPUs, we only use 6 during the GWT step, because we only
build 6 permutations of the web UI. On a 16-way system you could build
both GWT steps in parallel, if your build system knew what threads
>> Also I think the GWT build takes ~1 minute on my system. Gerrit takes
>> ~2 minutes to build. So the other 50% of the build time is
>> attributable to a "normal Maven build". No build system will reduce
>> this to 0, so we are always going to be >1 minute... but I think there
>> is some room for this time to be reduced if multiple cores can be
>> taken advantage of.
> Personally I would invest in better cpu and/or additional ram sticks
> then investing developer time to cut down 5 or 10s of build
> initialization step. I'm running full gerrit compilation up to 4 times
> per week (mvn -P all clean install) to verify compatibility of our
> plugins with current Gerrit version and few seconds won't make me a
Maybe I wasn't entirely clear; I am not really focused on making the
build faster. I am trying to make the build system do its job...
create a reliable build all of the time. Its not doing that. If
replacing it also makes it faster, that's a nice result.
>>> Another thing is that usage of custom/not popular build system can raise
>>> entry level for new contributors. Maven is a standard nowadays most of java
>>> developers already have it configured on theirs dev environments, forcing
>>> them to install new software and configure theirs system in some more or
>>> less 'strange' way may be blocker for some users.
>> OK, but building Gerrit is not trivial these days. Hell, I botched the
>> rc1 release by not having documentation in it... and I founded the
>> project, wrote most of the pom.xml we have today, and wrote the build
>> code that embeds the documentation into the release WAR files. So the
>> build we have is crap and needs to be fixed.
> I don't know what was the reasons for using ./tools/release.sh script
> instead of maven release goal.
Because `maven release` wants to shit all over your `git log` history
with meaningless wanking as it updates the version field in your
pom.xml files. It wants to push all of this shit to your central
repository immediately, as though running `mvn release` is going to
produce a perfect build that you can share with the world immediately,
without testing the resulting binary in real-world situations that are
hard to run under Maven.
I have a version control system. It knows what version the software
is. I don't need my build system wanking itself in my version control
The new acceptance-tests package makes it easier to run some sanity
checking under Maven, but its still not sufficient in my mind before
pushing a release.
> One thing that I learned during few
> years of using maven is that it works really great when you do
> everything 'the maven way' any customization in the process will hit
> you sooner or later...
See above about making the server more modular. The Maven way to do
this is to move my source files all over the tree and make lots of
more pom.xml files. I guess I could start doing this rather than
complaining about it, but its always been painful, e.g. the move to
create -openid or -cache-h2.
>> For me, I am only willing to keep Maven if someone can fix the release
>> build process so that I can't botch another rc. I'm not convinced this
>> is doable in Maven, and have no desire to spend a week of my own time
>> poking pom.xml trying to make it work.
> I'm sure that we cannot achieve all yours goals for build system with
> maven, but I'm also not sure if Buck is the right direction to go. I
> have heard that people in huge and complicated projects are really
> happy with Gradle there days.
I'm not really happy with Gradle's build specification format, or the
fact that its basically the same braindamaged view of the source tree
as Maven. But I wouldn't mind reviewing an RFC change that converted
the build to Gradle, as a comparison data point.