replacing Gerrit's build system

[Shawn Pearce]

https://gerrit-review.googlesource.com/45110 is still incomplete but
shows what a Buck based build format would look like and manages to
compile some sections of Gerrit's tree.

The biggest con for buck is the lack of Windows support. Buck like
Google's build system is heavily dependent upon symlinks and some
short shell commands. The handful of shell might work OK on Windows
through MSYS, but symlinks are absolutely required. Worse you need not
just buck to understand symlinks but also "native" tools like javac,
jar, etc.

--

Implement a new build system using Buck[1], Facebook's
open source clone of Google's internal build system.

Pros:
- Fast
- Concise build language
- Extensible with simple blocks of Python

Cons:
- No Windows support
- Bootstrap of buck requires Ant

Getting started:

git clone https://github.com/facebook/buck.git
cd buck
ant

Mac OS X:
alias buck='PATH=/System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands:$PATH
ANDROID_HOME=/tmp $HOME/git/buck/bin'

Linux:
alias buck='ANDROID_HOME=/tmp $HOME/git/buck/bin'

# partial list of targets that are (mostly) ported
buck build //gerrit-pgm:gerrit //gerrit-common:server //gerrit-common:clie

[1] http://facebook.github.io/buck/

--

FWIW I like where this is going. We may have to fix a few things in
Buck itself, but that should be doable. The BUCK files alongside of
the pom.xml files give a good comparison with Maven. Buck is much more
flexible than Maven is, and its build descriptions are a good deal
shorter than Gradle.

I hope the lack of Windows support isn't an issue for our existing contributors.

[Dariusz Luksza]

On 04/28/2013 03:18 AM, Shawn Pearce wrote:
> The biggest con for buck is the lack of Windows support. Buck like
> Google's build system is heavily dependent upon symlinks and some
> short shell commands. The handful of shell might work OK on Windows
> through MSYS, but symlinks are absolutely required. Worse you need not
> just buck to understand symlinks but also "native" tools like javac,
> jar, etc.

How Buck handles dependency management? Is it depending on path to 'lib/' folder with jars somewhere in the files system?

For me more important than Windows support is support in IDE (eclipse in my case). In case of maven we can say that there is decent a support in all popular Java IDE's like eclipse, intelij and
netbeanse. How it is with Buck?

I understood that maven configuration files are huge and you limited by plugin configuration options, buy is there any legitimate use case that can't be achieved with maven but is very simple to
achieve in Buck?

In case of build performance, as far as I see we spent most of time compiling GWT code and replacing build system will not solve this problem.

Another thing is that usage of custom/not popular build system can raise entry level for new contributors. Maven is a standard nowadays most of java developers already have it configured on theirs dev
environments, forcing them to install new software and configure theirs system in some more or less 'strange' way may be blocker for some users.

--
Best regards

GSM: +49 017 445 41235
Blog: http://luksza.org
LinkedIn: http://www.linkedin.com/in/dariuszluksza

[Alex Blewitt]

On 28 Apr 2013, at 02:18, Shawn Pearce wrote:

> https://gerrit-review.googlesource.com/45110 is still incomplete but
> shows what a Buck based build format would look like and manages to
> compile some sections of Gerrit's tree.
>
> The biggest con for buck is the lack of Windows support. Buck like
> Google's build system is heavily dependent upon symlinks and some
> short shell commands. The handful of shell might work OK on Windows
> through MSYS, but symlinks are absolutely required. Worse you need not
> just buck to understand symlinks but also "native" tools like javac,
> jar, etc.

As -1'd on the review:

Building on Windows is essential for developers that still use Windows as their primary development platform.

Being able to use a standard IDE's tool support for performing compilation and project dependencies is far more important than the build tool itself. All major IDEs can deal with Maven builds and allow cross-platform builds.

Even if a build system added an some advantages (build speed, size of build file) it would not offset the disadvantages of the lack of IDE support and Windows use cases.

After all, developers spend most of the time in their IDE, not on the command line invoking builds or editing build scripts. Optimising for the latter at the expense of the former is not a valid trade-off.

Alex

[Thomas Broyer]

+1 to this and what Dariusz Łuksza said.

FYI, we chose to move GWT to Maven not because Maven is the best tool for the job but because it's the most used and with the best tooling/IDE integration.

Granted, our goals are different: the move to Maven for GWT is to lower the barrier to entry and make it easier for people to jump in and contribute; this is not necessarily a goal for Gerrit.

If another build system (Gradle or anything else) becomes more mainstream and has technical benefits over Maven, then maybe we'll move to that, but that time hasn't yet come.

Managed dependencies (Ivy or Maven) is also a must-have if you ask me; I like how they're downloaded in this Buck setup but the transitive dependencies have to be re-written as Buck dependencies, rather than being generated/inferred from the POMs.

BTW, some comments on Buck from a Xoogler:

https://twitter.com/JakeWharton/status/325002919034363907

https://twitter.com/JakeWharton/status/325342363427954689 (about dependency management)

[Shawn Pearce]

On Sat, Apr 27, 2013 at 8:59 PM, Dariusz Luksza
<dariusz...@gmail.com> wrote:
> On 04/28/2013 03:18 AM, Shawn Pearce wrote:
>>
>> The biggest con for buck is the lack of Windows support. Buck like
>> Google's build system is heavily dependent upon symlinks and some
>> short shell commands. The handful of shell might work OK on Windows
>> through MSYS, but symlinks are absolutely required. Worse you need not
>> just buck to understand symlinks but also "native" tools like javac,
>> jar, etc.
>
>
> How Buck handles dependency management? Is it depending on path to 'lib/'
> folder with jars somewhere in the files system?

Buck's native dependency system is to use a set of JARs within the
project's source, e.g. checked into Git. I hand-wrote ~47 lines of
code to make Buck download JARs from Maven central and make them
available in the build. This allows us to list our dependencies in the
BUCK build files, and Buck gets what is needed for the target(s) you
asked it to build.

What I don't have is a separate local cache of the JARs. If you `buck
clean` the JARs are discarded and the build has to download them
again.

> For me more important than Windows support is support in IDE (eclipse in my
> case). In case of maven we can say that there is decent a support in all
> popular Java IDE's like eclipse, intelij and netbeanse. How it is with Buck?

Buck is missing Eclipse support, but has native support to create
InteliJ configuration files so the IDE has a basic clue about the
project layout. I am also heavily dependent on Eclipse for my
day-to-day development so in order to carry through with change like
this I would need to modify Buck to support creating Eclipse IDE
files.

> I understood that maven configuration files are huge and you limited by
> plugin configuration options, buy is there any legitimate use case that
> can't be achieved with maven but is very simple to achieve in Buck?

I think it depends on how much you care about the build. :-)

Buck allows us to define very fine-grained targets around specific
sections of the code. E.g. gerrit-server can be broken down into
smaller subsets with just a few lines of BUCK file changes. We can use
this to further reduce the surface of APIs we expose to plugins, which
in turn makes the plugin API faster because the final JAR it creates
is smaller. Making this segmentation costs almost nothing to the
overall build time.

Doing the same thing in Maven requires us to setup a new top level
directory with a bunch of new configuration files (.gitignore,
.settings/, pom.xml) and adds several seconds to the build while Maven
figures out how to parse that XML, start the necessary plugins, build
the JAR. It also further clutters the source tree. We can't just
partition on the Java package names.

With Buck I could eventually have a very clean source tree with
everything under "src/com/google/gerrit/..." and place BUCK files down
within that Java package hierarchy next to each Java package we want
to build as its own unit. This gives you a very natural mapping of
Java class name to location on the filesystem (no more "is that in
-common? -reviewdb? why?" questions). Try that with Maven sometime.
You will go screaming for the hills, I just don't think its possible.

I say eventually here because step 1 is to replace Maven. Step 2 is to
refactor the source tree into a more clean format. At least with Buck
there is no need to do these two steps as one big bang. Its not quite
like Maven where it ties your hands and (roughly) demands you follow
its rules... or else.

> In case of build performance, as far as I see we spent most of time
> compiling GWT code and replacing build system will not solve this problem.

Sure. But we run two copies of the GWT compiler now. And Maven doesn't
do anything in parallel. At least Buck will run steps in parallel
where it can.

Also I think the GWT build takes ~1 minute on my system. Gerrit takes
~2 minutes to build. So the other 50% of the build time is
attributable to a "normal Maven build". No build system will reduce
this to 0, so we are always going to be >1 minute... but I think there
is some room for this time to be reduced if multiple cores can be
taken advantage of.

> Another thing is that usage of custom/not popular build system can raise
> entry level for new contributors. Maven is a standard nowadays most of java
> developers already have it configured on theirs dev environments, forcing
> them to install new software and configure theirs system in some more or
> less 'strange' way may be blocker for some users.

OK, but building Gerrit is not trivial these days. Hell, I botched the
rc1 release by not having documentation in it... and I founded the
project, wrote most of the pom.xml we have today, and wrote the build
code that embeds the documentation into the release WAR files. So the
build we have is crap and needs to be fixed.

You are right that Maven is commonly available. Most developers are
held hostage by it. Given my experiences with Gerrit's pom.xmls I am
ready to throw it out and move to anything else. Ant and Make are
equally bad for Java development these days, but I can probably get a
working build in one of them.


For me, I am only willing to keep Maven if someone can fix the release
build process so that I can't botch another rc. I'm not convinced this
is doable in Maven, and have no desire to spend a week of my own time
poking pom.xml trying to make it work.

[Shawn Pearce]

On Sun, Apr 28, 2013 at 2:19 AM, Alex Blewitt <alex.b...@gmail.com> wrote:
> On 28 Apr 2013, at 02:18, Shawn Pearce wrote:
>
>> https://gerrit-review.googlesource.com/45110 is still incomplete but
>> shows what a Buck based build format would look like and manages to
>> compile some sections of Gerrit's tree.
>>
>> The biggest con for buck is the lack of Windows support. Buck like
>> Google's build system is heavily dependent upon symlinks and some
>> short shell commands. The handful of shell might work OK on Windows
>> through MSYS, but symlinks are absolutely required. Worse you need not
>> just buck to understand symlinks but also "native" tools like javac,
>> jar, etc.
>
>
> As -1'd on the review:
>
> Building on Windows is essential for developers that still use Windows as their primary development platform.

Obviously building on your development machine is important. So here
is a sad face about that platform being Windows... :-(

> Being able to use a standard IDE's tool support for performing compilation and project dependencies is far more important than the build tool itself. All major IDEs can deal with Maven builds and allow cross-platform builds.

Our Maven IDE support isn't perfect. Its often confused in Eclipse
about the ANTLR generated source files we have. It eventually figures
out what the build path is and compiles it correctly, but the Maven
Eclipse integration still doesn't get it.

Our Maven IDE support is still somewhat broken. The PrologCafe build
of gerrit_common.pl doesn't always get the classes picked up in the
IDE when you launch a server from Eclipse's Java debugger. When this
happens you get a weird failure about a predicate missing, shutdown
the server, `mvn clean package` in the command line, go raid the local
micro-kitchen while that is executing, come back and refresh Eclipse,
then restart the debugger. This happens to me several times per day.

So sure its nice that the IDE can import the project from a `git
clone` and have the classpath setup. But its still broken and fails
multiple times per day for me. Our classpath changes infrequently. I
can hand-edit the classpath when needed. I haven't yet figured out why
the predicate disappears from Eclipse.

> Even if a build system added an some advantages (build speed, size of build file) it would not offset the disadvantages of the lack of IDE support

IDE support isn't everything. I can probably put together a short
script to query Buck and get the list of JARs to go into a .classpath
file for Eclipse. Eclipse IDE support: done.

> and Windows use cases.

This is harder to fix. Maybe its fixable by using a copy of a file
instead of a symlink, and having MSYS tools in PATH. If you are on
Windows and using Git and using a command line, you probably are using
msysgit.

> After all, developers spend most of the time in their IDE, not on the command line invoking builds or editing build scripts. Optimising for the latter at the expense of the former is not a valid trade-off.

I know. But its also no good that the current build is crap.

[Shawn Pearce]

On Sun, Apr 28, 2013 at 3:09 AM, Thomas Broyer <t.br...@gmail.com> wrote:
> On Sunday, April 28, 2013 11:19:28 AM UTC+2, AlBlue wrote:
>>
>> On 28 Apr 2013, at 02:18, Shawn Pearce wrote:
>>
>> > https://gerrit-review.googlesource.com/45110 is still incomplete but
>> > shows what a Buck based build format would look like and manages to
>> > compile some sections of Gerrit's tree.
>>

>> After all, developers spend most of the time in their IDE, not on the
>> command line invoking builds or editing build scripts. Optimising for the
>> latter at the expense of the former is not a valid trade-off.
>
>
> +1 to this and what Dariusz Łuksza said.
>
> FYI, we chose to move GWT to Maven not because Maven is the best tool for
> the job but because it's the most used and with the best tooling/IDE
> integration.

The more I work with Maven the more I consider Maven to be actually
harmful to the community. But I guess its OK since everyone else uses
it. :-)

> Granted, our goals are different: the move to Maven for GWT is to lower the
> barrier to entry and make it easier for people to jump in and contribute;
> this is not necessarily a goal for Gerrit.

Actually lowering the barrier to entry is still a goal for Gerrit. One
of the things we need to do in that direction is switch from a GWT
based UI to a native HTML5 based UI... because a number of potential
UI contributors have told us they can't work on GWT code. This has
scary ramifications thanks to innerHTML being available from JS. Yay.
:-(

However. The primary purpose of a build system is to go from source
code to release artifact(s) in a reliable process that can't be
screwed up. This isn't working anymore for Gerrit. If the build system
isn't doing its job, it doesn't matter that it might already be
installed on someone's workstation, or that it runs on a minority
development platform (*cough* Windows *cough*), or that it is popular.

And actually Maven isn't really installed on most people's
workstations. It has to first download all of its plugins before it
can build Gerrit. I have no idea how much data that is but I have seen
people complain about the volume of Maven plugin installation that
happens on their first build of Gerrit.

> If another build system (Gradle or anything else) becomes more mainstream
> and has technical benefits over Maven, then maybe we'll move to that, but
> that time hasn't yet come.

Gradle looks like another mistake, buildr might be OK. This of course
means buildr will die and Gradle will win. :-)

> Managed dependencies (Ivy or Maven) is also a must-have if you ask me; I
> like how they're downloaded in this Buck setup

Thanks. I'm fairly happy with how short that is too. :-)

> but the transitive
> dependencies have to be re-written as Buck dependencies, rather than being
> generated/inferred from the POMs.

No transitive dependencies is actually a desired feature. We have a
ton of Maven configuration to exclude transitive dependencies where an
upstream project has listed them as compile-time scope but in reality
we don't need that JAR to run a Gerrit server. Worse in at least one
case the license of the transitive dependency does not permit us to
ship the transitive dependency with Gerrit.

Every time we touch a dependency in Gerrit we manually chase the
transitive dependency chain to verify license, and see if the
dependencies are even necessary. It only took me a short while to
chase everything down in search.maven.org.

One thing this process has taught me is our current Maven dependencies
are actually very bad. We have several different copies of
javax.servlet.* in our classpath for example. I actually don't know
which one is being used by javac at compile time. Buck has been
helpful at identifying some of this, but I haven't exactly solved the
problem in the Buck based build either. (gwt-user is definitely not
helping here.)

> BTW, some comments on Buck from a Xoogler:
> https://twitter.com/JakeWharton/status/325002919034363907
> https://twitter.com/JakeWharton/status/325342363427954689 (about dependency
> management)

:-)

[Shawn Pearce]

On Sun, Apr 28, 2013 at 2:19 AM, Alex Blewitt <alex.b...@gmail.com> wrote:

> On 28 Apr 2013, at 02:18, Shawn Pearce wrote:
>
>> https://gerrit-review.googlesource.com/45110 is still incomplete but
>> shows what a Buck based build format would look like and manages to
>> compile some sections of Gerrit's tree.
>
>

> As -1'd on the review:
>
> Building on Windows is essential for developers that still use Windows as their primary development platform.
>
> Being able to use a standard IDE's tool support for performing compilation and project dependencies is far more important than the build tool itself. All major IDEs can deal with Maven builds and allow cross-platform builds.

If you want to keep Maven, step up and fix the Maven build:

Issue 1) Maven needs to build a correct release WAR with one command.

The fact that it can't do this is why I have dragged my feet on 2.6,
and why the rc1 was not built correctly. Its no longer fun to make a
release WAR of Gerrit. But at least make it work right so I can't do
bad things.

Issue 2) Prolog predicates are broken in Eclipse.

I expect to be able to launch a debug server from Eclipse anytime.
Periodically Eclipse loses predicates from gerrit_common.pl and a
server startup crashes. Fix this.

Issue 3) Publishing API JARs is a disaster.

Everyone is complaining about this from their own plugin builds.
The API JARs aren't easy to make, aren't easy to get picked up,
etc. What we have isn't working.

As for me, I am done messing around with Maven pom.xml files. I might
look at buildr and try to prototype a build in that to compare against
the Buck prototype I have now, but Maven is dead to me.

[Dariusz Luksza]

On 04/28/2013 06:15 PM, Shawn Pearce wrote:
> On Sat, Apr 27, 2013 at 8:59 PM, Dariusz Luksza
> <dariusz...@gmail.com> wrote:
>> How Buck handles dependency management? Is it depending on path to 'lib/'
>> folder with jars somewhere in the files system?
>
> Buck's native dependency system is to use a set of JARs within the
> project's source, e.g. checked into Git. I hand-wrote ~47 lines of
> code to make Buck download JARs from Maven central and make them
> available in the build. This allows us to list our dependencies in the
> BUCK build files, and Buck gets what is needed for the target(s) you
> asked it to build.

Checking out jars into Git feels like step backward, not mentioning
that Gerrit's git repo will explode in size. This remembers me my
first experiences with commercial software development where we had
separate 'lib/' directory checkout into CVS for our dependencies ;)

> What I don't have is a separate local cache of the JARs. If you `buck
> clean` the JARs are discarded and the build has to download them
> again.

Side question: Did you consider other build systems? eg. Leiningen[1],
the build system for Clojure? AFAIK it is highly expendable and can
work with maven repositories, maybe it will not be as fast as Buck,
but it already have support for Eclipse. Another pros for Lainingen is
maturity and community build around it. The down side could be Clojure
it self with is 'moder lisp for JVM'

>> I understood that maven configuration files are huge and you limited by
>> plugin configuration options, buy is there any legitimate use case that
>> can't be achieved with maven but is very simple to achieve in Buck?
>
> I think it depends on how much you care about the build. :-)

For me it should be easy and work 'out of the box' without multi pages
long howto's and additional configuration/software.

> Buck allows us to define very fine-grained targets around specific
> sections of the code. E.g. gerrit-server can be broken down into
> smaller subsets with just a few lines of BUCK file changes. We can use
> this to further reduce the surface of APIs we expose to plugins, which
> in turn makes the plugin API faster because the final JAR it creates
> is smaller. Making this segmentation costs almost nothing to the
> overall build time.

Modularity is good, but to certain extent. Small modules tend to have
circular dependencies and adds confusion where given interface/class
should belong to.

> Doing the same thing in Maven requires us to setup a new top level
> directory with a bunch of new configuration files (.gitignore,
> .settings/, pom.xml) and adds several seconds to the build while Maven
> figures out how to parse that XML, start the necessary plugins, build
> the JAR. It also further clutters the source tree. We can't just
> partition on the Java package names.

Good point, I agree with this.

> With Buck I could eventually have a very clean source tree with
> everything under "src/com/google/gerrit/..." and place BUCK files down
> within that Java package hierarchy next to each Java package we want
> to build as its own unit. This gives you a very natural mapping of
> Java class name to location on the filesystem (no more "is that in
> -common? -reviewdb? why?" questions). Try that with Maven sometime.
> You will go screaming for the hills, I just don't think its possible.

How this 'BUCK file' will figure out dependencies for given
package-module. Or maybe more general question, does Buck support
transient dependencies? Personally I don't want to bother with
dependency of my dependency, I know that maven stores pom files in
.m2/repository to figure out on their own, does Buck how similar
mechanism?

> I say eventually here because step 1 is to replace Maven. Step 2 is to
> refactor the source tree into a more clean format. At least with Buck
> there is no need to do these two steps as one big bang. Its not quite
> like Maven where it ties your hands and (roughly) demands you follow
> its rules... or else.

I'm with you here... would even add third step 'convert modules to
OSGi bundles' ;)

>> In case of build performance, as far as I see we spent most of time
>> compiling GWT code and replacing build system will not solve this problem.
>
> Sure. But we run two copies of the GWT compiler now. And Maven doesn't
> do anything in parallel. At least Buck will run steps in parallel
> where it can.

AFAIK GWT compiler since version 1.6 have option 'localWorkers' which
allows you utilize all cpu's/core's so running two compilers in
parallel doesn't really make sense.

> Also I think the GWT build takes ~1 minute on my system. Gerrit takes
> ~2 minutes to build. So the other 50% of the build time is
> attributable to a "normal Maven build". No build system will reduce
> this to 0, so we are always going to be >1 minute... but I think there
> is some room for this time to be reduced if multiple cores can be
> taken advantage of.

Personally I would invest in better cpu and/or additional ram sticks
then investing developer time to cut down 5 or 10s of build
initialization step. I'm running full gerrit compilation up to 4 times
per week (mvn -P all clean install) to verify compatibility of our
plugins with current Gerrit version and few seconds won't make me a
difference.

>> Another thing is that usage of custom/not popular build system can raise
>> entry level for new contributors. Maven is a standard nowadays most of java
>> developers already have it configured on theirs dev environments, forcing
>> them to install new software and configure theirs system in some more or
>> less 'strange' way may be blocker for some users.
>
> OK, but building Gerrit is not trivial these days. Hell, I botched the
> rc1 release by not having documentation in it... and I founded the
> project, wrote most of the pom.xml we have today, and wrote the build
> code that embeds the documentation into the release WAR files. So the
> build we have is crap and needs to be fixed.

I don't know what was the reasons for using ./tools/release.sh script
instead of maven release goal. One thing that I learned during few
years of using maven is that it works really great when you do
everything 'the maven way' any customization in the process will hit
you sooner or later...

> You are right that Maven is commonly available. Most developers are
> held hostage by it. Given my experiences with Gerrit's pom.xmls I am
> ready to throw it out and move to anything else. Ant and Make are
> equally bad for Java development these days, but I can probably get a
> working build in one of them.

Give a try to Leiningen :D

> For me, I am only willing to keep Maven if someone can fix the release
> build process so that I can't botch another rc. I'm not convinced this
> is doable in Maven, and have no desire to spend a week of my own time
> poking pom.xml trying to make it work.

I'm sure that we cannot achieve all yours goals for build system with
maven, but I'm also not sure if Buck is the right direction to go. I
have heard that people in huge and complicated projects are really
happy with Gradle there days.

[1] http://leiningen.org/

[Shawn Pearce]

On Sun, Apr 28, 2013 at 11:05 AM, Dariusz Luksza
<dariusz...@gmail.com> wrote:
>> available in the build. This allows us to list our dependencies in the
>> BUCK build files, and Buck gets what is needed for the target(s) you
>> asked it to build.
>
> Checking out jars into Git feels like step backward, not mentioning
> that Gerrit's git repo will explode in size.

Both of which are why we won't check JARs into Git. Instead they are
obtained automatically during the build.

> Side question: Did you consider other build systems? eg. Leiningen[1],
> the build system for Clojure? AFAIK it is highly expendable and can
> work with maven repositories, maybe it will not be as fast as Buck,
> but it already have support for Eclipse. Another pros for Lainingen is
> maturity and community build around it. The down side could be Clojure
> it self with is 'moder lisp for JVM'
>

> [1] http://leiningen.org/

Never heard of Leiningen before. Maybe because its build language is
based on Lisp. I don't want to write my build specifications in Lisp.
Its bad enough we use Java, GWT and Prolog in Gerrit. Adding Lisp for
the build process is, well, one language too many in a relatively
small project.

This of course is also a counter-argument for Buck, Gradle and buildr.
Each one uses a language for the build files. At least Buck is such a
tiny subset of Python that you never need to really learn or worry
about Python. buildr seems to force you to learn some Ruby to grok the
files.

> For me it should be easy and work 'out of the box' without multi pages
> long howto's and additional configuration/software.

I'm not sure git clone is hard. But sure if you want it to be, you can
make git clone hard to use.

buildr's setup is more difficult. It actually scared me away from
trying a buildr prototype build.

>> Buck allows us to define very fine-grained targets around specific
>> sections of the code. E.g. gerrit-server can be broken down into
>> smaller subsets with just a few lines of BUCK file changes. We can use
>> this to further reduce the surface of APIs we expose to plugins, which
>> in turn makes the plugin API faster because the final JAR it creates
>> is smaller. Making this segmentation costs almost nothing to the
>> overall build time.
>
> Modularity is good, but to certain extent. Small modules tend to have
> circular dependencies and adds confusion where given interface/class
> should belong to.

True. Within Google we use an approach similar to what I described.
Modules are typically along Java package boundaries. Almost everything
below some package name is part of the same module in the build
system. There is no issue with circular dependencies because you put
related code into the same package, its the same directory in the
source tree, its easy to reason about.

The small modules let you break the module down into smaller segments
if you need to. E.g. you could make a target from the package that
consists only of the public interface files, and another that also has
the implementation. Client code links to the interface target while
the server binary can point to the implementation target, which
includes the interface target by transitive dependency:

INTERFACES = ["Fooer.java", "BarMaker.java"]

java_library(
name = "interface",
src = INTERFACES,
)

java_library(
name = "impl",
src = glob(["*.java"], excludes = INTERFACES),
deps = [":interface"],
)

Just try doing that in Maven, or most other build systems actually.
Its hard to be that fine-grained when you want to do it. :-)

>> Doing the same thing in Maven requires us to setup a new top level
>> directory with a bunch of new configuration files (.gitignore,
>> .settings/, pom.xml) and adds several seconds to the build while Maven
>> figures out how to parse that XML, start the necessary plugins, build
>> the JAR. It also further clutters the source tree. We can't just
>> partition on the Java package names.
>
> Good point, I agree with this.

I want to increase the modularity of the server, but doing the source
code tree refactorings to please Maven scares me. So I just keep not
doing it. Maven is actively keeping me from making changes that are
probably valuable to the long-term health of the source tree. Yay--.

>> With Buck I could eventually have a very clean source tree with
>> everything under "src/com/google/gerrit/..." and place BUCK files down
>> within that Java package hierarchy next to each Java package we want
>> to build as its own unit. This gives you a very natural mapping of
>> Java class name to location on the filesystem (no more "is that in
>> -common? -reviewdb? why?" questions). Try that with Maven sometime.
>> You will go screaming for the hills, I just don't think its possible.
>
> How this 'BUCK file' will figure out dependencies for given
> package-module. Or maybe more general question, does Buck support
> transient dependencies?

Buck automatically handles transitive dependencies within the BUCK
files you declared. So it knows gerrit-server needs gerrit-reviewdb,
which needs gwtorm.

> Personally I don't want to bother with
> dependency of my dependency, I know that maven stores pom files in
> .m2/repository to figure out on their own, does Buck how similar
> mechanism?

It doesn't follow m2 data, because it doesn't know anything about
Maven POM files. Buck needs everything declared in the BUCK files in
our Git repository.

Which is actually OK because we dive into every Maven POM and examine
the dependency tree to check licenses before we include something. In
the case of openid4java we have to explicitly drop some dependencies
because we can't use them under the license they offer.

As it turns out this exercise looking through our deps for Buck has
taught me we have a number of very risky/broken dependency chains in
Gerrit. Did you know that Guice seems to be linking to asm3 while
Gerrit itself uses asm4? Did you know that Gerrit could break if these
wind up in the wrong order? No Maven is not helping with this. It
happens 3 levels deep in transitive dependencies from POM files.

>> I say eventually here because step 1 is to replace Maven. Step 2 is to
>> refactor the source tree into a more clean format. At least with Buck
>> there is no need to do these two steps as one big bang. Its not quite
>> like Maven where it ties your hands and (roughly) demands you follow
>> its rules... or else.
>
> I'm with you here... would even add third step 'convert modules to
> OSGi bundles' ;)

Yes, based on the recent plugin work.. I am reconsidering the idea of
bundling an OSGi container and having Gerrit plugins managed by an
OSGi runtime.

>>> In case of build performance, as far as I see we spent most of time
>>> compiling GWT code and replacing build system will not solve this problem.
>>
>> Sure. But we run two copies of the GWT compiler now. And Maven doesn't
>> do anything in parallel. At least Buck will run steps in parallel
>> where it can.
>
> AFAIK GWT compiler since version 1.6 have option 'localWorkers' which
> allows you utilize all cpu's/core's so running two compilers in
> parallel doesn't really make sense.

For GWT yes. But it also depends on how may cores you have. If you
have say 16 CPUs, we only use 6 during the GWT step, because we only
build 6 permutations of the web UI. On a 16-way system you could build
both GWT steps in parallel, if your build system knew what threads
was. :-)

>> Also I think the GWT build takes ~1 minute on my system. Gerrit takes
>> ~2 minutes to build. So the other 50% of the build time is
>> attributable to a "normal Maven build". No build system will reduce
>> this to 0, so we are always going to be >1 minute... but I think there
>> is some room for this time to be reduced if multiple cores can be
>> taken advantage of.
>
> Personally I would invest in better cpu and/or additional ram sticks
> then investing developer time to cut down 5 or 10s of build
> initialization step. I'm running full gerrit compilation up to 4 times
> per week (mvn -P all clean install) to verify compatibility of our
> plugins with current Gerrit version and few seconds won't make me a
> difference.

Maybe I wasn't entirely clear; I am not really focused on making the
build faster. I am trying to make the build system do its job...
create a reliable build all of the time. Its not doing that. If
replacing it also makes it faster, that's a nice result.

>>> Another thing is that usage of custom/not popular build system can raise
>>> entry level for new contributors. Maven is a standard nowadays most of java
>>> developers already have it configured on theirs dev environments, forcing
>>> them to install new software and configure theirs system in some more or
>>> less 'strange' way may be blocker for some users.
>>
>> OK, but building Gerrit is not trivial these days. Hell, I botched the
>> rc1 release by not having documentation in it... and I founded the
>> project, wrote most of the pom.xml we have today, and wrote the build
>> code that embeds the documentation into the release WAR files. So the
>> build we have is crap and needs to be fixed.
>
> I don't know what was the reasons for using ./tools/release.sh script
> instead of maven release goal.

<rant>
Because `maven release` wants to shit all over your `git log` history
with meaningless wanking as it updates the version field in your
pom.xml files. It wants to push all of this shit to your central
repository immediately, as though running `mvn release` is going to
produce a perfect build that you can share with the world immediately,
without testing the resulting binary in real-world situations that are
hard to run under Maven.

I have a version control system. It knows what version the software
is. I don't need my build system wanking itself in my version control
history.
</rant>

The new acceptance-tests package makes it easier to run some sanity
checking under Maven, but its still not sufficient in my mind before
pushing a release.

> One thing that I learned during few
> years of using maven is that it works really great when you do
> everything 'the maven way' any customization in the process will hit
> you sooner or later...

See above about making the server more modular. The Maven way to do
this is to move my source files all over the tree and make lots of
more pom.xml files. I guess I could start doing this rather than
complaining about it, but its always been painful, e.g. the move to
create -openid or -cache-h2.

>> For me, I am only willing to keep Maven if someone can fix the release
>> build process so that I can't botch another rc. I'm not convinced this
>> is doable in Maven, and have no desire to spend a week of my own time
>> poking pom.xml trying to make it work.
>
> I'm sure that we cannot achieve all yours goals for build system with
> maven, but I'm also not sure if Buck is the right direction to go. I
> have heard that people in huge and complicated projects are really
> happy with Gradle there days.

I'm not really happy with Gradle's build specification format, or the
fact that its basically the same braindamaged view of the source tree
as Maven. But I wouldn't mind reviewing an RFC change that converted
the build to Gradle, as a comparison data point.

[Dariusz Luksza]

On Sun, Apr 28, 2013 at 8:42 PM, Shawn Pearce <s...@google.com> wrote:
> On Sun, Apr 28, 2013 at 11:05 AM, Dariusz Luksza

>> Modularity is good, but to certain extent. Small modules tend to have
>> circular dependencies and adds confusion where given interface/class
>> should belong to.
>
> True. Within Google we use an approach similar to what I described.
> Modules are typically along Java package boundaries. Almost everything
> below some package name is part of the same module in the build
> system. There is no issue with circular dependencies because you put
> related code into the same package, its the same directory in the
> source tree, its easy to reason about.

That's true for 'main app', but what if you want to reuse some part of
this application in eg. plugin or another application?
Different question is if usage of BUCK will be forced on plugin developers?

> The small modules let you break the module down into smaller segments
> if you need to. E.g. you could make a target from the package that
> consists only of the public interface files, and another that also has
> the implementation. Client code links to the interface target while
> the server binary can point to the implementation target, which
> includes the interface target by transitive dependency:
>
> INTERFACES = ["Fooer.java", "BarMaker.java"]
>
> java_library(
> name = "interface",
> src = INTERFACES,
> )
>
> java_library(
> name = "impl",
> src = glob(["*.java"], excludes = INTERFACES),
> deps = [":interface"],
> )
>
> Just try doing that in Maven, or most other build systems actually.
> Its hard to be that fine-grained when you want to do it. :-)

I'm not sure if it is so hard to do that in Maven. I use to work with
project that had '-api' module and it was easier to work with that
then working with separate '-api' project. But in this case decision
for having '-api' module was one of the first one, therefore we didn't
had any problems connected with splitting implementation from api.

> As it turns out this exercise looking through our deps for Buck has
> taught me we have a number of very risky/broken dependency chains in
> Gerrit. Did you know that Guice seems to be linking to asm3 while
> Gerrit itself uses asm4? Did you know that Gerrit could break if these
> wind up in the wrong order? No Maven is not helping with this. It
> happens 3 levels deep in transitive dependencies from POM files.

Yes I had problems with asm3 and asm4 and also have exclusion for one
of those in one of my pom.xml files. I'm wondering how my plugin pom's
files would look like after gerrit migrates to eg. BUCK. Do I need to
include all transitive dependencies in my poms? Would I be forced to
manage versions of transitive dependencies? Could I still use maven
for my 'trivial' Gerrit plugin project?

>> I'm with you here... would even add third step 'convert modules to
>> OSGi bundles' ;)
>
> Yes, based on the recent plugin work.. I am reconsidering the idea of
> bundling an OSGi container and having Gerrit plugins managed by an
> OSGi runtime.

YEAH! This is already GREAT news from this thread ;)

> <rant>
> Because `maven release` wants to shit all over your `git log` history
> with meaningless wanking as it updates the version field in your
> pom.xml files. It wants to push all of this shit to your central
> repository immediately, as though running `mvn release` is going to
> produce a perfect build that you can share with the world immediately,
> without testing the resulting binary in real-world situations that are
> hard to run under Maven.
>
> I have a version control system. It knows what version the software
> is. I don't need my build system wanking itself in my version control
> history.
> </rant>

Thank you for giving this background. Yes maven release process is
very verbose, this is the cost of common approach to release process
and version control system.

[Matthias Sohn]

On Sun, Apr 28, 2013 at 8:42 PM, Shawn Pearce <s...@google.com> wrote:

that's cool

 

>> Doing the same thing in Maven requires us to setup a new top level
>> directory with a bunch of new configuration files (.gitignore,
>> .settings/, pom.xml) and adds several seconds to the build while Maven
>> figures out how to parse that XML, start the necessary plugins, build
>> the JAR. It also further clutters the source tree. We can't just
>> partition on the Java package names.
>
> Good point, I agree with this.

I want to increase the modularity of the server, but doing the source
code tree refactorings to please Maven scares me. So I just keep not
doing it. Maven is actively keeping me from making changes that are
probably valuable to the long-term health of the source tree. Yay--.

yeah, Maven is too intrusive in that respect

I agree automatic transitive dependencies look nice on the first glance

but easily become a nightmare in more complex projects with more than

a handful of dependencies

 

>> I say eventually here because step 1 is to replace Maven. Step 2 is to
>> refactor the source tree into a more clean format. At least with Buck
>> there is no need to do these two steps as one big bang. Its not quite
>> like Maven where it ties your hands and (roughly) demands you follow
>> its rules... or else.
>
> I'm with you here... would even add third step 'convert modules to
> OSGi bundles' ;)

Yes, based on the recent plugin work.. I am reconsidering the idea of
bundling an OSGi container and having Gerrit plugins managed by an
OSGi runtime.

+1

 

>>> In case of build performance, as far as I see we spent most of time
>>> compiling GWT code and replacing build system will not solve this problem.
>>
>> Sure. But we run two copies of the GWT compiler now. And Maven doesn't
>> do anything in parallel. At least Buck will run steps in parallel
>> where it can.
>
> AFAIK GWT compiler since version 1.6 have option 'localWorkers' which
> allows you utilize all cpu's/core's so running two compilers in
> parallel doesn't really make sense.

For GWT yes. But it also depends on how may cores you have. If you
have say 16 CPUs, we only use 6 during the GWT step, because we only
build 6 permutations of the web UI. On a 16-way system you could build
both GWT steps in parallel, if your build system knew what threads
was. :-)

looks like I should upgrade my hardware :)

+1 maven's way of forcing it's own idea of project versions into the

projects history is a pain

 

The new acceptance-tests package makes it easier to run some sanity
checking under Maven, but its still not sufficient in my mind before
pushing a release.

> One thing that I learned during few
> years of using maven is that it works really great when you do
> everything 'the maven way' any customization in the process will hit
> you sooner or later...

See above about making the server more modular. The Maven way to do
this is to move my source files all over the tree and make lots of
more pom.xml files. I guess I could start doing this rather than
complaining about it, but its always been painful, e.g. the move to
create -openid or -cache-h2.

yes, refactoring a Maven build usually shuffles your source tree which

is painful each time it happens

 

>> For me, I am only willing to keep Maven if someone can fix the release
>> build process so that I can't botch another rc. I'm not convinced this
>> is doable in Maven, and have no desire to spend a week of my own time
>> poking pom.xml trying to make it work.

I feel similar pain when creating a jgit/egit release, buck sounds really

promising to fix this. I also like the clean and dense syntax compared

to lengthy unreadable pom.xmls configuring tons of Maven plugins.

Looks like I should spend some time to learn more about buck.

Though I think contributors working on Windows shouldn't be ignored,

maybe some of them could step up and try to teach buck to work on

Windows ;-)

--

Matthias

[Shawn Pearce]

On Sun, Apr 28, 2013 at 1:10 PM, Dariusz Luksza
<dariusz...@gmail.com> wrote:
> That's true for 'main app', but what if you want to reuse some part of
> this application in eg. plugin or another application?

You, uh, use the JAR?

> Different question is if usage of BUCK will be forced on plugin developers?

No, but we may encourage it as the recommended way to build a plugin
if the template is easier. Core plugins like replication, etc. that
come bundled with Gerrit will probably move if Gerrit moves.

>> The small modules let you break the module down into smaller segments
>> if you need to. E.g. you could make a target from the package that
>> consists only of the public interface files, and another that also has
>> the implementation. Client code links to the interface target while
>> the server binary can point to the implementation target, which
>> includes the interface target by transitive dependency:
>>
>> INTERFACES = ["Fooer.java", "BarMaker.java"]
>>
>> java_library(
>> name = "interface",
>> src = INTERFACES,
>> )
>>
>> java_library(
>> name = "impl",
>> src = glob(["*.java"], excludes = INTERFACES),
>> deps = [":interface"],
>> )
>>
>> Just try doing that in Maven, or most other build systems actually.
>> Its hard to be that fine-grained when you want to do it. :-)
>
> I'm not sure if it is so hard to do that in Maven. I use to work with
> project that had '-api' module and it was easier to work with that
> then working with separate '-api' project. But in this case decision
> for having '-api' module was one of the first one, therefore we didn't
> had any problems connected with splitting implementation from api.

Its hard to do this after-the-fact, you need to move the source files
to their own directory tree. You need to bury them down inside of an
api/src/main/java separate from impl/src/main/java. Its annoying as
hell to separate a package across so many directory levels to appease
a build system's single world view.

At least with Buck we can perform this sort of split incrementally, in
stages. You can't do that in Maven... because its Maven.

>> As it turns out this exercise looking through our deps for Buck has
>> taught me we have a number of very risky/broken dependency chains in
>> Gerrit. Did you know that Guice seems to be linking to asm3 while
>> Gerrit itself uses asm4? Did you know that Gerrit could break if these
>> wind up in the wrong order? No Maven is not helping with this. It
>> happens 3 levels deep in transitive dependencies from POM files.
>
> Yes I had problems with asm3 and asm4 and also have exclusion for one
> of those in one of my pom.xml files.

Right. So we have problems with our build and Maven isn't even telling
us. Its just praying for the best. I am at least finding the problems
in Buck. It does complicate re-implementing the build in Buck, these
problems aren't necessarily easy to fix and have to be resolved no
matter what build system we use. :-)

> I'm wondering how my plugin pom's
> files would look like after gerrit migrates to eg. BUCK. Do I need to
> include all transitive dependencies in my poms? Would I be forced to
> manage versions of transitive dependencies?

The extension-api and plugin-api JARs currently built by Maven are
shaded JARs, they contain the transitive closure of the dependencies
as part of the JAR you consume in Maven. We can build this same format
of JAR with Buck for these two JARs. Which means we just need a small
trivial pom.xml to register the artifacts in a Maven repository. mvn
has a tool to make such a tiny pom.xml from the command line, but we
can also just template it and have a genrule() in Buck make the
pom.xml file as part of our build.

We could in the future switch these to being non-shaded JARs and also
have Buck emit a pom.xml from a genrule() that includes the transitive
dependency information. Buck's classpath data for a target can be
queried with `buck audit`. It shouldn't be difficult to reformat that
as pom data.

Either way the plugin developer doesn't need to know the transitive
dependencies directly, but they do need to realize they are stuck with
our version of Guice in either extension or plugin, and in a plugin
are stuck with our version of JGit (hey gitblit!), and other
dependencies the server has.

> Could I still use maven
> for my 'trivial' Gerrit plugin project?

Yes, see above. We should still make the API JARs available to Maven
consumers. Just because we are using Buck to build the server doesn't
mean a plugin developer can't use Maven. Same goes for any other build
system.

>>> I'm with you here... would even add third step 'convert modules to
>>> OSGi bundles' ;)
>>
>> Yes, based on the recent plugin work.. I am reconsidering the idea of
>> bundling an OSGi container and having Gerrit plugins managed by an
>> OSGi runtime.
>
> YEAH! This is already GREAT news from this thread ;)

This is not an easy project. But I kind of want to ditch Maven before
attempting this because we need to make a number of changes to the
build process in order to get targets an OSGi container can run. Of
course some OSGi folks really like using Maven 3 with e.g. the Tycho
P2 resolver, or just the Apache Felix Maven plugin. So its not like
you can't do OSGi builds with Maven. _I_ just don't want to.

I actually sort of want to mash the source tree all together as one
huge java/com/google/gerrit tree before I start to break it up again
as a series of OSGi bundles. Along the way we can fix Java packages to
make more sense, isolate things into packages that are related, split
packages up where the contents aren't related, and start assigning
package trees to specific JARs, and then those JARs turn into OSGi
bundles.

[Shawn Pearce]

On Sun, Apr 28, 2013 at 2:10 PM, Matthias Sohn <matthi...@gmail.com> wrote:
> On Sun, Apr 28, 2013 at 8:42 PM, Shawn Pearce <s...@google.com> wrote:
>>
>> As it turns out this exercise looking through our deps for Buck has
>> taught me we have a number of very risky/broken dependency chains in
>> Gerrit. Did you know that Guice seems to be linking to asm3 while
>> Gerrit itself uses asm4? Did you know that Gerrit could break if these
>> wind up in the wrong order? No Maven is not helping with this. It
>> happens 3 levels deep in transitive dependencies from POM files.
>
>
> I agree automatic transitive dependencies look nice on the first glance
> but easily become a nightmare in more complex projects with more than
> a handful of dependencies

Automatic transitive dependencies are nice when all of your
dependencies are within your company's own technology stack, e.g. you
own 100% of the code and there are no license concerns, version skew
concerns, etc.

They also work well when you are building a startup and don't care
about open source license compliance, or are running the binary only
on your internal systems and never redistribute. Assuming you don't
run into version skew problems like we have with asm3 and asm4 in
Gerrit.

Unfortunately Gerrit is in neither of these positions. I have always
traced through the dependency chains to check licenses, but somehow I
did bungle the asm3/4 thing along the way, and javax.servlet.* coming
from multiple places is another disaster.

>> For GWT yes. But it also depends on how may cores you have. If you
>> have say 16 CPUs, we only use 6 during the GWT step, because we only
>> build 6 permutations of the web UI. On a 16-way system you could build
>> both GWT steps in parallel, if your build system knew what threads
>> was. :-)
>
> looks like I should upgrade my hardware :)

Heh. My laptop is not 16-way. Neither is my desktop. But I know some
people have 16-way desktops now, they are somewhat affordable. For
some definition of "somewhat" and "affordable". Everything is
relative. :-)

>> >> For me, I am only willing to keep Maven if someone can fix the release
>> >> build process so that I can't botch another rc. I'm not convinced this
>> >> is doable in Maven, and have no desire to spend a week of my own time
>> >> poking pom.xml trying to make it work.
>
>
> I feel similar pain when creating a jgit/egit release,

FWIW this is also why I passed the JGit release process onto you and
haven't attempted to make a release since JGit joined the Eclipse
Foundation. I fixed my JGit release woes by having you handle them.
:-)

> buck sounds really
> promising to fix this. I also like the clean and dense syntax compared
> to lengthy unreadable pom.xmls configuring tons of Maven plugins.
> Looks like I should spend some time to learn more about buck.

Coming from Google's build system, there really wasn't much for me to
learn. Mostly I was learning how Buck differs from our build system,
which is more ways than its similar. Might actually be easier to learn
Buck when you know nothing about Google's build system. :-)

> Though I think contributors working on Windows shouldn't be ignored,

Unfortunately I agree. I would like to just ignore the Windows problem
since I haven't touched the OS myself in 5 years, but there are still
many developers who are issued Windows desktops at work and have no
other option for a general purpose computing device at $DAY_JOB.

> maybe some of them could step up and try to teach buck to work on
> Windows ;-)

Good point. I honestly have no idea of the effort required here. I do
know Buck likes to use symlinks. These might be faked by doing copies
instead, but this breaks some nice behaviors where changes to a file
in the source tree are immediately seen because the output tree is
just a symlink to the source file. I am especially thinking of
resource files for JARs that are accessed from a server running under
the debugger.

[David Ostrovsky]

Inspired by your work, i am trying to implement Gerrit build with buildr[1]

[1] https://gerrit-review.googlesource.com/#/c/45120/

David

[Dariusz Luksza]

On Mon, Apr 29, 2013 at 2:48 AM, Shawn Pearce <s...@google.com> wrote:
> On Sun, Apr 28, 2013 at 1:10 PM, Dariusz Luksza
> <dariusz...@gmail.com> wrote:
>> That's true for 'main app', but what if you want to reuse some part of
>> this application in eg. plugin or another application?
>
> You, uh, use the JAR?

Yes, will use JAR, but if it isn't shaded and don't have definition of
transitive dependency 'somewhere' then I'll need to figure them out
the hard way.

>> Different question is if usage of BUCK will be forced on plugin developers?
>
> No, but we may encourage it as the recommended way to build a plugin
> if the template is easier. Core plugins like replication, etc. that
> come bundled with Gerrit will probably move if Gerrit moves.

Sounds OK for me.

> The extension-api and plugin-api JARs currently built by Maven are
> shaded JARs, they contain the transitive closure of the dependencies
> as part of the JAR you consume in Maven. We can build this same format
> of JAR with Buck for these two JARs. Which means we just need a small
> trivial pom.xml to register the artifacts in a Maven repository. mvn
> has a tool to make such a tiny pom.xml from the command line, but we
> can also just template it and have a genrule() in Buck make the
> pom.xml file as part of our build.

AFAIK maven can only generate simple pom files without any dependency
definitions. Strange thing with mvn install:install-file is that it
don't even look into META-INF directory for pom.xml, but it
automatically generate 'tiny' xml file only with artifactId, groupId
and version... I had strange problems with this behavior of mvn.

FWIK Leiningen generates pom.xml file based on project.clj this way
one could easily import project to any maven-aware system. This should
be also the direction where Gerrit with BUCK should go, and as you
mentioned this shouldn't be difficult to achieve this.

>>> Yes, based on the recent plugin work.. I am reconsidering the idea of
>>> bundling an OSGi container and having Gerrit plugins managed by an
>>> OSGi runtime.
>>
>> YEAH! This is already GREAT news from this thread ;)
>
> This is not an easy project. But I kind of want to ditch Maven before
> attempting this because we need to make a number of changes to the
> build process in order to get targets an OSGi container can run. Of
> course some OSGi folks really like using Maven 3 with e.g. the Tycho
> P2 resolver, or just the Apache Felix Maven plugin. So its not like
> you can't do OSGi builds with Maven. _I_ just don't want to.
>
> I actually sort of want to mash the source tree all together as one
> huge java/com/google/gerrit tree before I start to break it up again
> as a series of OSGi bundles. Along the way we can fix Java packages to
> make more sense, isolate things into packages that are related, split
> packages up where the contents aren't related, and start assigning
> package trees to specific JARs, and then those JARs turn into OSGi
> bundles.

Marrying Buck Gerrit with P2 OSGi could be challenging (based on what
I heard about mvn and tycho integration), but about this problem we'll
bother in the future ;)

To be honest at first moment when I've read yours mail about replacing
maven with 'something-different-and-less-know' I was really for 'NO'
but I'm starting to be convinced that this is the right way to go.

[Alex Blewitt]

On 28 Apr 2013, at 19:42, Shawn Pearce wrote:

<rant>
Because `maven release` wants to shit all over your `git log` history
with meaningless wanking as it updates the version field in your
pom.xml files. It wants to push all of this shit to your central
repository immediately, as though running `mvn release` is going to
produce a perfect build that you can share with the world immediately,
without testing the resulting binary in real-world situations that are
hard to run under Maven.

I have a version control system. It knows what version the software
is. I don't need my build system wanking itself in my version control
history.
</rant>

If you ust want to change the version numbers in the POMs, then use:

  mvn versions:set -DnewVersion=2.6-rc2

This doesn't involve any of the repository management that the mvn release does.

Alex

[Alex Blewitt]

On 28 Apr 2013, at 18:02, Shawn Pearce wrote:

> On Sun, Apr 28, 2013 at 2:19 AM, Alex Blewitt <alex.b...@gmail.com> wrote:
>> On 28 Apr 2013, at 02:18, Shawn Pearce wrote:
>>
>>> https://gerrit-review.googlesource.com/45110 is still incomplete but
>>> shows what a Buck based build format would look like and manages to
>>> compile some sections of Gerrit's tree.
>>
>>
>> As -1'd on the review:
>>
>> Building on Windows is essential for developers that still use Windows as their primary development platform.
>>
>> Being able to use a standard IDE's tool support for performing compilation and project dependencies is far more important than the build tool itself. All major IDEs can deal with Maven builds and allow cross-platform builds.
>
> If you want to keep Maven, step up and fix the Maven build:
>
> Issue 1) Maven needs to build a correct release WAR with one command.
>
> The fact that it can't do this is why I have dragged my feet on 2.6,
> and why the rc1 was not built correctly. Its no longer fun to make a
> release WAR of Gerrit. But at least make it work right so I can't do
> bad things.

OK. I think that the main issue is relating to the documentation being built, correct? The documentation format is built on some specific external tools that don't work in the Maven build (in fact, I don't have asciidocs as part of my OSX system either) but as part of a different process. It looks like the tools/release.sh does the necessary work, and will include the documentation in the Maven build when -Dgerrit.incldue-documentation=1 is specified.

I would say the right way to fix this would be to figure out how to do the asciidoc generation in the Maven build instead of using a separate tool, but I don't know how much work that is. Alternatively, Maven has some site-generated doc mechanisms instead.

> Issue 2) Prolog predicates are broken in Eclipse.
>
> I expect to be able to launch a debug server from Eclipse anytime.
> Periodically Eclipse loses predicates from gerrit_common.pl and a
> server startup crashes. Fix this.

This is likely an issue where Eclipse has cleaned/recompiled the folder, and it's not running the generate sources step. I haven't seen this myself, but will look into it.

> Issue 3) Publishing API JARs is a disaster.
>
> Everyone is complaining about this from their own plugin builds.
> The API JARs aren't easy to make, aren't easy to get picked up,
> etc. What we have isn't working.

I think that's because the API Jar isn't published into Maven central. Once 2.6 is built, it will be easy for others to be able to build on top of that API. The issue at the moment is that it's not possible to build 'just the API jar' without building everything. Is that really an issue?

If this is still seen as a big problem, then it should be easy enough to define a profile for the build that just includes the API modules.

> As for me, I am done messing around with Maven pom.xml files. I might
> look at buildr and try to prototype a build in that to compare against
> the Buck prototype I have now, but Maven is dead to me.

Understood. Will try and make it continue working.

Alex

[Thomas Broyer]

On Sun, Apr 28, 2013 at 6:44 PM, Shawn Pearce <s...@google.com> wrote:
> On Sun, Apr 28, 2013 at 3:09 AM, Thomas Broyer <t.br...@gmail.com> wrote:
>> On Sunday, April 28, 2013 11:19:28 AM UTC+2, AlBlue wrote:
>>>
>>> On 28 Apr 2013, at 02:18, Shawn Pearce wrote:
>>>
>>> > https://gerrit-review.googlesource.com/45110 is still incomplete but
>>> > shows what a Buck based build format would look like and manages to
>>> > compile some sections of Gerrit's tree.
>>>
>>> After all, developers spend most of the time in their IDE, not on the
>>> command line invoking builds or editing build scripts. Optimising for the
>>> latter at the expense of the former is not a valid trade-off.
>>
>>
>> +1 to this and what Dariusz Łuksza said.
>>
>> FYI, we chose to move GWT to Maven not because Maven is the best tool for
>> the job but because it's the most used and with the best tooling/IDE
>> integration.
>
> The more I work with Maven the more I consider Maven to be actually
> harmful to the community. But I guess its OK since everyone else uses
> it. :-)

You're not alone:
http://blog.lexspoon.org/2012/12/recursive-maven-considered-harmful.html
(another Xoogler BTW)
And I won't dispute this view as Maven is far from perfect.
I find Buck appealing (I haven't yet read all its docs, or even tried
it); if only it worked with Maven repositories (have a maven_jar dep
that automatically resolves its transitive dependencies from the
downloaded POM for instance, and store a single copy in a shared
folder)

Most people having gripes with Maven are actually trying to bend Maven
to their way of thinking, rather than embracing The Maven Way™
http://developer-blog.cloudbees.com/2013/04/the-maven-way.html
IIRC that's not the problem in Gerrit's case.

>> Granted, our goals are different: the move to Maven for GWT is to lower the
>> barrier to entry and make it easier for people to jump in and contribute;
>> this is not necessarily a goal for Gerrit.
>
> Actually lowering the barrier to entry is still a goal for Gerrit. One
> of the things we need to do in that direction is switch from a GWT
> based UI to a native HTML5 based UI... because a number of potential
> UI contributors have told us they can't work on GWT code. This has
> scary ramifications thanks to innerHTML being available from JS. Yay.
> :-(
>
> However. The primary purpose of a build system is to go from source
> code to release artifact(s) in a reliable process that can't be
> screwed up. This isn't working anymore for Gerrit. If the build system
> isn't doing its job, it doesn't matter that it might already be
> installed on someone's workstation, or that it runs on a minority
> development platform (*cough* Windows *cough*), or that it is popular.

How could one disagree with that? ;-)

Would you mind explaining how "this isn't working anymore for Gerrit"?

(BTW, I didn't say anything about it but I actually don't care about Windows)

> And actually Maven isn't really installed on most people's
> workstations.

Not sure what you mean here.
According to ZeroTurnaround's 2012 survey report, 67% of Java
developers are using Maven:
http://zeroturnaround.com/rebellabs/devs/developer-productivity-report-2012/
But it's true that Maven will possibly need to "download itself" (the
core plugins: compiler, resources, etc.) for each project if they use
different versions of the plugins.

> It has to first download all of its plugins before it
> can build Gerrit. I have no idea how much data that is but I have seen
> people complain about the volume of Maven plugin installation that
> happens on their first build of Gerrit.

Yes, it's scary!

>> If another build system (Gradle or anything else) becomes more mainstream
>> and has technical benefits over Maven, then maybe we'll move to that, but
>> that time hasn't yet come.
>
> Gradle looks like another mistake, buildr might be OK. This of course
> means buildr will die and Gradle will win. :-)

:-)

(I, for one, don't like DSLs that are executable code; buildr==ruby,
gradle==groovy, sbt==scala, etc. AIUI Buck's DSL is not Python, only
extensible through Python functions, just like Maven is extensible
through plugins but in a much simpler way; on the other hand, Buck's
way of sharing extensions is via duplication…)

>> Managed dependencies (Ivy or Maven) is also a must-have if you ask me; I
>> like how they're downloaded in this Buck setup
>
> Thanks. I'm fairly happy with how short that is too. :-)
>
>> but the transitive
>> dependencies have to be re-written as Buck dependencies, rather than being
>> generated/inferred from the POMs.
>
> No transitive dependencies is actually a desired feature. We have a
> ton of Maven configuration to exclude transitive dependencies where an
> upstream project has listed them as compile-time scope but in reality
> we don't need that JAR to run a Gerrit server. Worse in at least one
> case the license of the transitive dependency does not permit us to
> ship the transitive dependency with Gerrit.

I agree there are way too many badly declared dependencies; because
many people use Maven "because it's what others use" without
understanding it or understanding what they're doing (this is a
problem with every tool; like people (*cough* Buck *cough*) storing
big binaries in Git)
…and among the worst such projects are ones from the Apache
Foundation, where Maven is developped (ActiveMQ and UIMA come to mind)
:'-(

> Every time we touch a dependency in Gerrit we manually chase the
> transitive dependency chain to verify license, and see if the
> dependencies are even necessary. It only took me a short while to
> chase everything down in search.maven.org.

There are tools that can help you; e.g. Sonatype Insight:
http://www.sonatype.com/Products/Application-Health-Check
Which has a Jenkins/Hudson plugin:
https://wiki.jenkins-ci.org/display/JENKINS/Sonatype+CI+Plugins
and Nexus integration: http://books.sonatype.com/nexus-book/reference/rhc.html
for your everyday use.

> One thing this process has taught me is our current Maven dependencies
> are actually very bad. We have several different copies of
> javax.servlet.* in our classpath for example.

This particular case is probably because Sun/Oracle didn't initially
want to play well with the Maven community: either they didn't publish
JSR APIs to a Maven repo or the published JARs were "stripped" in a
way that they could only be used to compile against them but not *run*
code, not even unit-tests (I don't know Java bytecode enough to
understand how that can be, but that's what I've read).
So Tomcat, Jetty, JBoss, etc. deployed their own versions of the APIs
and there's no way of saying/telling they can conflict with each
other.

"mvn dependency:tree" can help here.

(managed dependencies doesn't mean you don't have to analyze your
dependencies and their transitive dependencies, it just makes it
easier; particularly when trying new versions or new dependencies)

> I actually don't know
> which one is being used by javac at compile time. Buck has been
> helpful at identifying some of this, but I haven't exactly solved the
> problem in the Buck based build either. (gwt-user is definitely not
> helping here.)

I'm working on this on the GWT side (modularization, no fat-jars).

>> BTW, some comments on Buck from a Xoogler:
>> https://twitter.com/JakeWharton/status/325002919034363907
>> https://twitter.com/JakeWharton/status/325342363427954689 (about dependency
>> management)
>
> :-)

--
Thomas Broyer
/tɔ.ma.bʁwa.je/

[Magnus Bäck]

On Sunday, April 28, 2013 at 21:01 EDT,

Shawn Pearce <s...@google.com> wrote:

> On Sun, Apr 28, 2013 at 2:10 PM, Matthias Sohn
> <matthi...@gmail.com> wrote:

[...]

> > maybe some of them could step up and try to teach buck to work on
> > Windows ;-)
>
> Good point. I honestly have no idea of the effort required here. I do
> know Buck likes to use symlinks. These might be faked by doing copies
> instead, but this breaks some nice behaviors where changes to a file
> in the source tree are immediately seen because the output tree is
> just a symlink to the source file. I am especially thinking of
> resource files for JARs that are accessed from a server running under
> the debugger.

Windows does support symlinks since Vista, although for some reason the
default security policy restricts the use of them to administrators.
If requiring users to be granted this privilege isn't a dealbreaker it
might be worthwhile to look into what other things are stopping Buck
from working on Windows. Of course, the symlinks issue could still be
problematic if Java or whatever layer that tries to create the symlinks
doesn't support Windows.

--
Magnus Bäck
ba...@google.com

[Thomas Broyer]

It could be the Stockholm Syndrome (I'm not a fan of Maven, but it has always worked for me) but I like having each "component" in its own source tree. One net advantage is that you cannot include a class in more than one JAR (unless you explicitly shade JARs).

The question "what part of the internal API you expose to plugins", for instance, becomes "what *is* the plugin API?" (that something will then have to implement).

I'm not saying other approaches are wrong, just that having Maven forcing you to think that ways is not necessarily a bad thing; it forces you to *think* (design) more though, which could have a negative impact on productivity (from some definition of productivity when you have to track down a intersection between your includes/excludes rules for JARs produced from the same Java package)

Try that with Maven sometime.
You will go screaming for the hills, I just don't think its possible.

It is, but then your IDEs (Eclipse at least) won't like it.

(that was my first approach when modularizing GWT, not moving files but simply add pom.xml files)

I say eventually here because step 1 is to replace Maven. Step 2 is to
refactor the source tree into a more clean format. At least with Buck
there is no need to do these two steps as one big bang. Its not quite
like Maven where it ties your hands and (roughly) demands you follow
its rules... or else.

> In case of build performance, as far as I see we spent most of time
> compiling GWT code and replacing build system will not solve this problem.

Sure. But we run two copies of the GWT compiler now. And Maven doesn't
do anything in parallel. At least Buck will run steps in parallel
where it can.

I could agree if it were true; but it's not: https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3

[Alex Blewitt]

On 29 Apr 2013, at 09:54, Alex Blewitt wrote:

OK. I think that the main issue is relating to the documentation being built, correct? The documentation format is built on some specific external tools that don't work in the Maven build (in fact, I don't have asciidocs as part of my OSX system either) but as part of a different process. It looks like the tools/release.sh does the necessary work, and will include the documentation in the Maven build when -Dgerrit.incldue-documentation=1 is specified.

I would say the right way to fix this would be to figure out how to do the asciidoc generation in the Maven build instead of using a separate tool, but I don't know how much work that is. Alternatively, Maven has some site-generated doc mechanisms instead.

I have pushed a draft proposal for the documentation to be generated with the Asciidoc maven plugin:

https://gerrit-review.googlesource.com/#/c/45170/

The styles are not the same, and the images are not copied over to the generated-sources directory. I suspect a post-processing of the images and the generated HTML files would be required in order to replicate what the current command line/shell script does. However, I'd like to get feedback on this so far.

Alex

[Saša Živkov]

On Sun, Apr 28, 2013 at 7:02 PM, Shawn Pearce <s...@google.com> wrote:

On Sun, Apr 28, 2013 at 2:19 AM, Alex Blewitt <alex.b...@gmail.com> wrote:

> On 28 Apr 2013, at 02:18, Shawn Pearce wrote:
>
>> https://gerrit-review.googlesource.com/45110 is still incomplete but
>> shows what a Buck based build format would look like and manages to
>> compile some sections of Gerrit's tree.
>
>

> As -1'd on the review:
>
> Building on Windows is essential for developers that still use Windows as their primary development platform.
>
> Being able to use a standard IDE's tool support for performing compilation and project dependencies is far more important than the build tool itself. All major IDEs can deal with Maven builds and allow cross-platform builds.

If you want to keep Maven, step up and fix the Maven build:

Issue 1)  Maven needs to build a correct release WAR with one command.

  The fact that it can't do this is why I have dragged my feet on 2.6,
  and why the rc1 was not built correctly. Its no longer fun to make a
  release WAR of Gerrit. But at least make it work right so I can't do
  bad things.

I can confirm that building a full Gerrit release (core + documentation + core plugins)

is a pain and error prone. One week ago, when I created a full release of our internal

Gerrit fork, it took me the whole afternoon to get all things right.

 

Issue 2)  Prolog predicates are broken in Eclipse.

  I expect to be able to launch a debug server from Eclipse anytime.
  Periodically Eclipse loses predicates from gerrit_common.pl and a
  server startup crashes. Fix this.

This also happens to me quite often.

 

Issue 3)  Publishing API JARs is a disaster.

  Everyone is complaining about this from their own plugin builds.
  The API JARs aren't easy to make, aren't easy to get picked up,
  etc. What we have isn't working.

This was also one of the reasons why the full release took me so long.

From a first look, BUCK files seem much more readable than pom.xml files.

Can one also add python code to BUCK file(s) if a specific control logic needs

to be programmed? This is usually quite difficult and ugly to do in maven.

Take the following example from the gerrit-plugin-archetype/src/main/resources/archetype-resources/pom.xml

  <repositories>

    <repository>

      <id>gerrit-api-repository</id>

#if ($gerritApiVersion.endsWith("SNAPSHOT"))

      <url>https://gerrit-api.commondatastorage.googleapis.com/snapshot/</url>

#else

      <url>https://gerrit-api.commondatastorage.googleapis.com/release/</url>

#end

    </repository>

  </repositories>

Here we use velocity macros in order to generate proper url depending on the $gerritApiVersion.

But this helps only for the initial plugin generation.

Once a plugin skeleton is generated, we must make sure that the URL is correct

every time the gerritApiVersion changes.

Is buck better in solving this kind of problems?

Just for curiosity: is there any support for archetypes (generation of projects from a template) in buck?

A main advantage of buck would be if we could in one step build:

- gerrit core

- plugin/extension api

- documentation

- core plugins

- full Gerrit release (all of the above packaged in one .war file)

Obviously, I would like to experiment with an alternative to maven.

Saša

As for me, I am done messing around with Maven pom.xml files. I might
look at buildr and try to prototype a build in that to compare against
the Buck prototype I have now, but Maven is dead to me.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Magnus Bäck]

On Tuesday, April 30, 2013 at 08:25 EDT,
Saša Živkov <ziv...@gmail.com> wrote:

[...]

> From a first look, BUCK files seem much more readable than pom.xml
> files. Can one also add python code to BUCK file(s) if a specific
> control logic needs to be programmed?

Yes. Arbitrary Python code can be used, but the documentation explicitly
discourages it. That said, I'd guess that if this is ever changed Buck
would still allow a limited subset of Python, so if one's not doing too
funky stuff (that one probably shouldn't be doing anyway) it'll be okay.

http://facebook.github.io/buck/extending/macros.html

[...]

--
Magnus Bäck
ba...@google.com

[Thomas Broyer]

On Tuesday, April 30, 2013 2:25:20 PM UTC+2, zivkov wrote:

From a first look, BUCK files seem much more readable than pom.xml files.

Can one also add python code to BUCK file(s) if a specific control logic needs

to be programmed? This is usually quite difficult and ugly to do in maven.

Take the following example from the gerrit-plugin-archetype/src/main/resources/archetype-resources/pom.xml

  <repositories>

    <repository>

      <id>gerrit-api-repository</id>

#if ($gerritApiVersion.endsWith("SNAPSHOT"))

      <url>https://gerrit-api.commondatastorage.googleapis.com/snapshot/</url>

#else

      <url>https://gerrit-api.commondatastorage.googleapis.com/release/</url>

#end

    </repository>

  </repositories>

Here we use velocity macros in order to generate proper url depending on the $gerritApiVersion.

But this helps only for the initial plugin generation.

Once a plugin skeleton is generated, we must make sure that the URL is correct

every time the gerritApiVersion changes.

Why aren't both repos added?

<repository>

  <id>gerrit-api-repository</id>

  <url>https://gerrit-api.commondatastorage.googleapis.com/release/</url>

  <releases><enabled>true</enabled></releases>

  <snapshots><enabled>false</enabled></snapshots>

</repository>

<repository>

  <id>gerrit-api-repository-snapshots</id>

  <url>https://gerrit-api.commondatastorage.googleapis.com/snapshot/</url>

  <releases><enabled>false</enabled></releases>

  <snapshots><enabled>true</enabled></snapshots>

</repository>

[Saša Živkov]

Because neither myself nor the reviewer(s) of that change knew about this maven feature :-|

I should fix that.

Still, I find this way of specifying conditional logic much harder to figure out how to write

and understand that something like:

  if gerritApiVersion.endswith("SNAPSHOT"):

    GERRIT_API_REPO = "https://gerrit-api.commondatastorage.googleapis.com/snapshot/"

  else:

    GERRIT_API_REPO = "https://gerrit-api.commondatastorage.googleapis.com/release/"

which is, I guess, how it would look in a BUCK file :-)

[Thomas Swindells (tswindel)]

[Thomas Swindells] In both cases you should probably make both repositories available – this becomes much more important if things like interface definition or plugin jars gain their own version lifecycle and you start to have snapshot versions referring to released versions of interfaces etc.

 

[Saša Živkov]

In my example, the "gerritApiVersion" represents the used version of the gerrit-plugin-api.

My plugin version may be 1.0-SNAPSHOT and it may refer to gerritApiVersion=2.5 (the plugin and

the gerrit-plugin-api have different lifecycles).

In that case the conditional statement wants to say that, since the referenced gerrit-plugin-api version is a released

version, only the repository containing released gerrit-plugin-api versions should be available.

I don't see why, in this case, the /snapshot/ repository should also be available?

 

 

[Thomas Broyer]

If you use Buck for your plugin, assuming you copy Shawn's script and maven_jar macro, you'll likely specify the repo for your dependency, switching the constant when you change the dep version (or possibly using a new specific macro that does the if/else branching)

[Thomas Swindells (tswindel)]

Still, I find this way of specifying conditional logic much harder to figure out how to write

and understand that something like:

 

  if gerritApiVersion.endswith("SNAPSHOT"):

    GERRIT_API_REPO = "https://gerrit-api.commondatastorage.googleapis.com/snapshot/"

  else:

    GERRIT_API_REPO = "https://gerrit-api.commondatastorage.googleapis.com/release/"

 

which is, I guess, how it would look in a BUCK file :-)

[Thomas Swindells] In both cases you should probably make both repositories available – this becomes much more important if things like interface definition or plugin jars gain their own version lifecycle and you start to have snapshot versions referring to released versions of interfaces etc.

 

In my example, the "gerritApiVersion" represents the used version of the gerrit-plugin-api.

My plugin version may be 1.0-SNAPSHOT and it may refer to gerritApiVersion=2.5 (the plugin and

the gerrit-plugin-api have different lifecycles).

In that case the conditional statement wants to say that, since the referenced gerrit-plugin-api version is a released

version, only the repository containing released gerrit-plugin-api versions should be available.

I don't see why, in this case, the /snapshot/ repository should also be available?

 

[Thomas Swindells] From a Maven world at least, you aren’t saying that the repo SHOULD be available, you are saying the artifact may be in that repo, if the repo isn’t available or doesn’t contain the artifact it will try looking in one of the other repos until it resolves the artifact. This means you can just point at them all and let the tools worry about the logic of working out where to get what type of artifact.

 

Thomas

[Thomas Broyer]

…with the unfortunate limitation that Maven only distinguishes
snapshots and releases and will otherwise look for an artifact in
*all* known repos in sequence (until it finds it).
This can severely impact the build performance when artifacts are to
be resolved (new dependencies, updated versions, or snapshot update
check), and is why it is strongly recommended to:
* deploy releases on Central rather than your own repo
* use a repository manager at your organization to act as a global
proxy for all repositories

--
Thomas Broyer
/tɔ.ma.bʁwa.je/