Buck based build is ready

[Shawn Pearce]

https://gerrit-review.googlesource.com/45110 is fully functional.

Getting started:

git clone https://github.com/facebook/buck.git
cd buck
ant

Mac OS X:
PATH="`pwd`/bin/buck:/System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands:$PATH"

Linux:
PATH="`pwd`/bin/buck:$PATH"

Building a normal WAR:

buck build :gerrit ; # or :withdocs
java -jar buck-out/gen/gerrit.war

Building the API:

buck build :api
ls -lh buck-out/gen/{extension,plugin}-api.jar

Building a release WAR:

# note plugins are pulled in by submodule:
git submodule init ; git submodule update

buck build :release
java -jar buck-out/gen/release.war


There are some other fun targets like :firefox and :chrome to perform
draft compiles.

:-)

[Dave Borowitz]

$ buck build :api

BUILDING //:api

Created JAR at buck-out/gen/lib/antlr/antlr-tool.jar

Created JAR at buck-out/gen//extension-api.jar

java.util.zip.ZipException: invalid entry compressed size (expected 4271 but got 4275 bytes)

        at java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:264)

        at java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:359)

        at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:238)

        at java.util.zip.ZipOutputStream.close(ZipOutputStream.java:376)

        at com.google.common.io.Closer.close(Closer.java:206)

        at com.facebook.buck.shell.JarDirectoryCommand.createJarFile(JarDirectoryCommand.java:174)

        at com.facebook.buck.shell.JarDirectoryCommand.execute(JarDirectoryCommand.java:119)

        at com.facebook.buck.shell.DefaultCommandRunner.runCommand(DefaultCommandRunner.java:67)

        at com.facebook.buck.rules.AbstractCachingBuildRule$2.call(AbstractCachingBuildRule.java:343)

        at com.facebook.buck.rules.AbstractCachingBuildRule$2.call(AbstractCachingBuildRule.java:334)

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

        at java.util.concurrent.FutureTask.run(FutureTask.java:166)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:722)

BUILD FAILED: com.facebook.buck.shell.CommandFailedException: Failed with 1: jar cf buck-out/gen//plugin-api.jar  buck-out/gen/gerrit-httpd/lib__httpd__output/httpd.jar buck-out/gen/gerrit-gwtexpui/lib__linker_server__output/linker_server.jar buck-out/gen/gerrit-gwtexpui/lib__server__output/server.jar buck-out/gen/lib/jgit/org.eclipse.jgit-2.3.1.201302201838-r.175-g1b4320f.jar buck-out/gen/lib/jgit/javaewah-0.5.6.jar buck-out/gen/lib/gson-2.1.jar buck-out/gen/gerrit-patch-jgit/lib__server__output/server.jar buck-out/gen/lib/gwtorm-1.6.jar buck-out/gen/gerrit-reviewdb/lib__server__output/server.jar buck-out/gen/lib/guava-14.0.jar buck-out/gen/lib/gwtjsonrpc-1.3.jar buck-out/gen/gerrit-prettify/lib__server__output/server.jar buck-out/gen/gerrit-server/lib__server__output/server.jar buck-out/gen/lib/antlr/antlr-runtime-3.2.jar buck-out/gen/gerrit-antlr/lib__antlr__output/antlr.jar buck-out/gen/lib/jsr305-1.3.9.jar buck-out/gen/gerrit-common/lib__server__output/server.jar buck-out/gen/gerrit-extension-api/lib__api__output/api.jar buck-out/gen/gerrit-util-ssl/lib__ssl__output/ssl.jar buck-out/gen/lib/log/slf4j-api-1.6.1.jar buck-out/gen/lib/commons-codec-1.4.jar buck-out/gen/lib/commons-net-2.2.jar buck-out/gen/gerrit-patch-commonsnet/lib__commons-net__output/commons-net.jar buck-out/gen/lib/guice/guice-3.0.jar buck-out/gen/lib/guice/aopalliance-1.0.jar buck-out/gen/lib/guice/javax.inject-1.jar buck-out/gen/lib/guice/guice-assistedinject-3.0.jar buck-out/gen/lib/args4j-2.0.16.jar buck-out/gen/gerrit-util-cli/lib__cli__output/cli.jar buck-out/gen/lib/guice/guice-servlet-3.0.jar buck-out/gen/lib/prolog/prologcafe-1.3.jar buck-out/gen/lib/automaton-1.11-8.jar buck-out/gen/lib/commons-dbcp-1.4.jar buck-out/gen/lib/commons-pool-1.5.5.jar buck-out/gen/lib/commons-lang-2.5.jar buck-out/gen/lib/jsch-0.1.44-1.jar buck-out/gen/lib/juniversalchardet-1.0.3.jar buck-out/gen/lib/mime-util-2.1.3.jar buck-out/gen/lib/asm-4.0.jar buck-out/gen/lib/asm-tree-4.0.jar buck-out/gen/lib/asm-util-4.0.jar buck-out/gen/lib/pegdown-1.1.0.jar buck-out/gen/lib/parboiled-java-1.1.3.jar buck-out/gen/lib/asm-analysis-4.0.jar buck-out/gen/lib/parboiled-core-1.1.3.jar buck-out/gen/lib/velocity-1.6.4.jar buck-out/gen/lib/commons-collections-3.2.1.jar buck-out/gen/lib/oro-2.0.8.jar buck-out/gen/lib/jgit/org.eclipse.jgit.http.server-2.3.1.201302201838-r.175-g1b4320f.jar buck-out/gen/gerrit-sshd/lib__sshd__output/sshd.jar buck-out/gen/lib/h2-1.3.168.jar buck-out/gen/gerrit-cache-h2/lib__cache-h2__output/cache-h2.jar buck-out/gen/lib/log/log4j-1.2.16.jar buck-out/gen/lib/mina/mina-core-2.0.5.jar buck-out/gen/lib/mina/sshd-core-0.6.0.jar

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dariusz Luksza]

I've same error 'invalid entry compressed size'

--
Best regards

GSM: +49 017 445 41235
Blog: http://luksza.org
LinkedIn: http://www.linkedin.com/in/dariuszluksza

[Shawn Pearce]

On Wed, May 1, 2013 at 8:23 AM, Dariusz Luksza <dariusz...@gmail.com> wrote:
> I've same error 'invalid entry compressed size'
>
> On Wed, May 1, 2013 at 5:03 PM, Dave Borowitz <dbor...@google.com> wrote:
>> $ buck build :api
>> BUILDING //:api
>> Created JAR at buck-out/gen/lib/antlr/antlr-tool.jar
>> Created JAR at buck-out/gen//extension-api.jar
>> java.util.zip.ZipException: invalid entry compressed size (expected 4271 but
>> got 4275 bytes)

*sigh*

I forgot the :api targets use java_binary(). There is a bug fix you
need in Buck:

https://github.com/spearce/buck/commit/5fd88c68bd4ba4f0ca65336b5a150ad8f30db45b

I started a pull request with Facebook but am trying to work through
the complexities of getting one large company to sign another large
company's CLA. I'm sure you guys understand... :-)

[Dariusz Luksza]

Before we had just a problem with building system... now we have problem with building system and corporate politics ;)

[Shawn Pearce]

On Wed, May 1, 2013 at 10:01 AM, Dariusz Luksza
<dariusz...@gmail.com> wrote:
> On 05/01/2013 06:55 PM, Shawn Pearce wrote:
>>
>> I started a pull request with Facebook but am trying to work through
>> the complexities of getting one large company to sign another large
>> company's CLA. I'm sure you guys understand... :-)
>>
>
> Before we had just a problem with building system... now we have problem
> with building system and corporate politics ;)

I don't think there is an issue with corporate politics here. It just
takes time to get contributor agreements signed. I know some of the
Gerrit contributors know what I mean; they have also had multi-week
delays getting the AOSP CLA signed on their side, and verified on
Google's side.

[Nasser Grainawi]

multi-week would be nice. I think my wait was more like a year. :-/

>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--

The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

[David Ostrovsky]

Am Mittwoch, 1. Mai 2013 18:55:45 UTC+2 schrieb Shawn Pearce:

On Wed, May 1, 2013 at 8:23 AM, Dariusz Luksza <dariusz...@gmail.com> wrote:
> I've same error 'invalid entry compressed size'
>
> On Wed, May 1, 2013 at 5:03 PM, Dave Borowitz <dbor...@google.com> wrote:
>> $ buck build :api
>> BUILDING //:api
>> Created JAR at buck-out/gen/lib/antlr/antlr-tool.jar
>> Created JAR at buck-out/gen//extension-api.jar
>> java.util.zip.ZipException: invalid entry compressed size (expected 4271 but
>> got 4275 bytes)

*sigh*

I forgot the :api targets use java_binary(). There is a bug fix you
need in Buck:

  https://github.com/spearce/buck/commit/5fd88c68bd4ba4f0ca65336b5a150ad8f30db45b

with this patch applied it worked for me.

With opened gerrit-review change and parallel discussion on ML i am not sure,
what is the expected/desired workflow: to comment on change in Gerrit or
to discuss it on ML?

Anyhow, shouldn't we in post maven world still reflect the version/release in artifact name?

gerrit.war => gerrit-2.7-SNAPSHOT.war
plugin-api.jar => plugin-api-2.7-SNAPSHOT.jar
[...]

For working on the same tree for different branches it would be advantageous not to override
the artifacts, not to mention the tracking of version/release if the artifacts are copied or moved around.

Just to make it clear: are you going to release 2.6 final with buck?

[Shawn Pearce]

On Wed, May 1, 2013 at 12:35 PM, David Ostrovsky
<david.o...@gmail.com> wrote:
>>
>> https://github.com/spearce/buck/commit/5fd88c68bd4ba4f0ca65336b5a150ad8f30db45b
>>
> with this patch applied it worked for me.
>
> With opened gerrit-review change and parallel discussion on ML i am not
> sure,
> what is the expected/desired workflow: to comment on change in Gerrit or
> to discuss it on ML?

Thus far we have been using the mailing list for general discussion
about the concept, and the review change for specific details on
specific lines of the build. I think that is a reasonable trade-off.
Lets face it, Gerrit is OK for reviewing source files line-by-line but
not so good for a threaded discussion replying to specific paragraphs
of text. :-)

> Anyhow, shouldn't we in post maven world still reflect the version/release
> in artifact name?
>
> gerrit.war => gerrit-2.7-SNAPSHOT.war
> plugin-api.jar => plugin-api-2.7-SNAPSHOT.jar
> [...]

In the case of gerrit.war I chose not to do that. I considered it, but
rejected it because I find it annoying when I switch between branches
e.g. stable-2.6 and master that the name of the output file I now have
to run with `java -jar` changes.

I am more open to the idea for the :api and :release targets. It is
maybe worth doing.

> For working on the same tree for different branches it would be advantageous
> not to override
> the artifacts,

Actually, no. There is a lot of delta between two branches in Gerrit.
Buck is going to rebuild all of the intermediate dependencies due to
differences, so there isn't much value in trying to cache the
top-level result using different names. If you really want to hang
onto that artifact there is a handy command called "cp".

> not to mention the tracking of version/release if the
> artifacts are copied or moved around.

For the API, sure. For the release WAR, sure. For gerrit.war? You are
probably going to either just put it in a test site, or put it into
your production server. Either way you name it "gerrit.war" in its
final resting place. Uhm, you threw out the version number. If you
need the version number you can run `java -jar gerrit.war version` and
it spits it back out.

> Just to make it clear: are you going to release 2.6 final with buck?

I considered this because the build actually works. But I have to
rebase a number of things over, and the gwtexpui package doesn't exist
in the stable-2.6 branch. So I have also been considering abandoning
the 2.6 rc series and starting 2.7... :-(

[Dariusz Luksza]

On 05/01/2013 11:47 PM, Shawn Pearce wrote:
> On Wed, May 1, 2013 at 12:35 PM, David Ostrovsky
> <david.o...@gmail.com> wrote:
>>>
>>> https://github.com/spearce/buck/commit/5fd88c68bd4ba4f0ca65336b5a150ad8f30db45b
>>>
>> with this patch applied it worked for me.
>>
>> With opened gerrit-review change and parallel discussion on ML i am not
>> sure,
>> what is the expected/desired workflow: to comment on change in Gerrit or
>> to discuss it on ML?
>
> Thus far we have been using the mailing list for general discussion
> about the concept, and the review change for specific details on
> specific lines of the build. I think that is a reasonable trade-off.
> Lets face it, Gerrit is OK for reviewing source files line-by-line but
> not so good for a threaded discussion replying to specific paragraphs
> of text. :-)

Sounds like a future request ;)

In huge teams threaded code comments (and review comments also) could be useful. Especially when more then three persons participate in code review, then threading comments would be helpful to keep
track of discussion.

[Alex Blewitt]

On 1 May 2013, at 22:47, Shawn Pearce wrote:

Just to make it clear: are you going to release 2.6 final with buck?

I considered this because the build actually works. But I have to
rebase a number of things over, and the gwtexpui package doesn't exist
in the stable-2.6 branch. So I have also been considering abandoning
the 2.6 rc series and starting 2.7... :-(

I would suggest just cutting 2.6, and then switching builds immediately after that point. That way, those who are unable to move to the new build system can continue to do so from a 'last known good' point, and the Buck/Gerrit evolution can happen as part of future releases. If 2.6-rc1 happens to be that last known good point, then just relabel that.

Unfortunately the use of a build tool which is still in a compile-it-yourself model, doesn't work on Windows, and doesn't work when running behind firewalls means that it will be necessary to continue the evolution of Maven/Gerrit separately from Buck/Gerrit.

Alex

[Dariusz Luksza]

On 05/01/2013 11:47 PM, Shawn Pearce wrote:

> On Wed, May 1, 2013 at 12:35 PM, David Ostrovsky

>> Just to make it clear: are you going to release 2.6 final with buck?
>
> I considered this because the build actually works. But I have to
> rebase a number of things over, and the gwtexpui package doesn't exist
> in the stable-2.6 branch. So I have also been considering abandoning
> the 2.6 rc series and starting 2.7... :-(
>

Are you really serious about dropping 2.6?

If yes, then this is '-1' for me. IMO dropping release is not an option, even when preparing it is really painful. I understand that from you point of view 'releases' are not important and you prefer
to run latest version of Gerrit master (I also love to be on blading edge with features and versions), but you must also consider all other Gerrit adopters that are forced by their's organizations
policies to stick to stable version. Many people out there are waiting for 2.6, forcing them to wait another half a year for 2.7 would be really cruel.

[Shawn Pearce]

On Wed, May 1, 2013 at 3:16 PM, Dariusz Luksza <dariusz...@gmail.com> wrote:
> On 05/01/2013 11:47 PM, Shawn Pearce wrote:
>>
>> On Wed, May 1, 2013 at 12:35 PM, David Ostrovsky
>>>
>>> Just to make it clear: are you going to release 2.6 final with buck?
>>
>>
>> I considered this because the build actually works. But I have to
>> rebase a number of things over, and the gwtexpui package doesn't exist
>> in the stable-2.6 branch. So I have also been considering abandoning
>> the 2.6 rc series and starting 2.7... :-(
>>
>
> Are you really serious about dropping 2.6?
>
> If yes, then this is '-1' for me. IMO dropping release is not an option,

No, I am not serious about that. Its what I would like to do, but its
unrealistic as you point out.

[Shawn Pearce]

On Wed, May 1, 2013 at 3:16 PM, Alex Blewitt <alex.b...@gmail.com> wrote:
> > Just to make it clear: are you going to release 2.6 final with buck?

No. 2.6 will be built from the existing Maven process.

> I would suggest just cutting 2.6, and then switching builds immediately
> after that point.

Yes, stable-2.6 will be produced using Maven, master going forward
soon-ish will be produced with something else. Buck is a strong
contender. I am interested in David Ostrovsky's buildr proposal too
but have not had time to try running it myself, and its still
incomplete. The hard parts about Gerrit's build are still not done in
the buildr based process.

> That way, those who are unable to move to the new build
> system can continue to do so from a 'last known good' point, and the
> Buck/Gerrit evolution can happen as part of future releases. If 2.6-rc1
> happens to be that last known good point, then just relabel that.

rc1 isn't suitable for a final release. We need at least an rc2. There
is work in stable-2.6 that has valuable bug fixes.

> Unfortunately the use of a build tool which is still in a
> compile-it-yourself model,

I guess I didn't realize that building software from source was a
problem. Good thing we are talking about users who are trying to
compile software?

> doesn't work on Windows,

This is an issue, but I am not in a position to really do anything
about it. As I said in another thread I haven't touched Windows in 5
years. I don't have a Windows system to even try running the existing
Maven build process on, let alone something else.

It would be good to get an idea of what is broken on Windows so we can
at least size the effort involved here.

> and doesn't work when
> running behind firewalls means that it will be necessary to continue the
> evolution of Maven/Gerrit separately from Buck/Gerrit.

How do you run Maven? Is your firewall magical and allows Maven
Central requests from "mvn" but rejects everything else? Or do you
have an internal mirror of all of Maven Central that mvn knows how to
use? Or do you set proxy options in the environment so that mvn can
authenticate through a proxy?

The downloader in the Buck based build uses curl. curl can in some
cases authenticate through a proxy.

[Alex Blewitt]

On 2 May 2013, at 01:01, Shawn Pearce wrote:

> On Wed, May 1, 2013 at 3:16 PM, Alex Blewitt <alex.b...@gmail.com> wrote:
>> Unfortunately the use of a build tool which is still in a
>> compile-it-yourself model,
>
> I guess I didn't realize that building software from source was a
> problem. Good thing we are talking about users who are trying to
> compile software?

That's not how it works in large organisations. The build tools and IDEs in large organisations tend to be centrally managed and distributed via software channels rather than individuals doing what they want. Partially this is to avoid fragmentation (i.e. anyone can pick up any project and get used to it) and partially it's to make lives easier so that developers don't have to bootstrap their own environment. The point is not every organisation is as open as Google when it comes to letting developers do their own thing.

For hobbyist contributions or those wanting to submit small fixes, it's probably not going to be much of an issue. And if it's really small fixes (typos etc.) then there's still the possibility of submitting fixes without bothering to build it.

>> doesn't work on Windows,
>
> This is an issue, but I am not in a position to really do anything
> about it. As I said in another thread I haven't touched Windows in 5
> years. I don't have a Windows system to even try running the existing
> Maven build process on, let alone something else.
>
> It would be good to get an idea of what is broken on Windows so we can
> at least size the effort involved here.

Well, if the source tree is sliced and diced using symlinks then that's not going to work, for a start. The other aspect in the review code is that many references to PATHs and executables (e.g. /bin/curl) won't be in the same locations, or even assumed to exist, on Windows platforms. But it doesn't really matter, in the sense that people who have to use Windows probably have other constraints that prevent the use of Buck, so it's an academic question.

>> and doesn't work when
>> running behind firewalls means that it will be necessary to continue the
>> evolution of Maven/Gerrit separately from Buck/Gerrit.
>
> How do you run Maven? Is your firewall magical and allows Maven
> Central requests from "mvn" but rejects everything else? Or do you
> have an internal mirror of all of Maven Central that mvn knows how to
> use? Or do you set proxy options in the environment so that mvn can
> authenticate through a proxy?
>
> The downloader in the Buck based build uses curl. curl can in some
> cases authenticate through a proxy.

No, the firewall in fact prevents any connections from going outside. Instead JARs are served from repository managers like Artifactory or Nexus. One of the reasons for the series of changes submitted previously has been to get Gerrit working by just building against Maven Central, which is typically proxied by such repository managers. In fact, the only non-Maven Central dependencies that exist in Gerrit now are the gwt* binaries; if those were in Maven Central then the Gerrit repository would not be required in the download of content.

The way that the Maven (or Ivy) settings are configured are externally to the build, so that the build knows of the GAV co-ordinates and the local settings says which repositories to resolve those JARs from. Tools that expect a full-on access to the internet for downloading what they want will break, and it's not just a case of specifying an HTTP proxy to fix it.

I appreciate the set of problems faced by these choices aren't the same set of problems you faced (which, incidentally, included the 'Eclipse sometimes loses the Prolog compiled code' which isn't solved by any of these other tools …) but the fact of the matter is assumptions about an open internet for all doesn't necessarily exist in large organisations.

Alex

[Thomas Swindells (tswindel)]

> > Before we had just a problem with building system... now we have
> > problem with building system and corporate politics ;)
>
> I don't think there is an issue with corporate politics here. It just takes time to
> get contributor agreements signed. I know some of the Gerrit contributors
> know what I mean; they have also had multi-week delays getting the AOSP
> CLA signed on their side, and verified on Google's side.

Which reminded me that I had to send yet another email to chase getting it signed.
At least I've got to the stage where my companies systems have recorded it as officially allowed.

Hopefully want it gets submitted Google's side will be quicker.

Thomas

[Pursehouse, David (Sony Mobile)]

I've been playing with the buck build this afternoon.

Overall it seems to work as expected; it's faster than the Maven build and it was even able to download all the dependencies through my corporate proxy without me having to twiddle with anything. Yay!

There are a few (minor) issues though:

- The gerrit build fails unless using buck built with Shawn's patch [1] applied.

- The python scripts require 2.7 or above. I've had to make a few workarounds to make it work on my 2.6.5 installation; I've uploaded the changes in case anyone else is stuck on 2.6.x [2].

- The plugin submodules don't work due to the relevant changes still being under review; they need to be manually cherry-picked and submodules reinitialized.

- Plugins cannot be built from within the plugins folder.


[1] https://github.com/facebook/buck/pull/9
[2] https://gerrit-review.googlesource.com/#/c/45231/

[Pursehouse, David (Sony Mobile)]

Update: the build runs OK, but the resulting .war file does not work properly :(

Browsing to "All open" or "my changes" results in a 500 internal server error. The error log shows:

[2013-05-02 18:14:08,438] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/?n=25&O=1
{EXISTENCE ERROR:procedure :(gerrit,/(locate_submit_rule,1)) does not exist}

[Saša Živkov]

Just tried the buck build. After a short time nothing moves, no CPU activity [1]

There are quite some Compiler processes [2].

And each of them is dead-locked. The thread dump shows the following dead-lock for all of them [3].

Anybody else seeing the same issue?

Any idea?

[1]

[~/dev/gerrit ((8e7b5e1...) *)]$ buck build :gerrit

BUILDING //:gerrit

[2]

[~/dev/gerrit ((8e7b5e1...) *)]$ jps

73815 Compiler

86921 Compiler

86920 Compiler

77310 Compiler

77309 Compiler

73817 Compiler

74614 Compiler

87988 Jps

74612 Compiler

87441 Compiler

87378 Main

71920 Compiler

71921 Compiler

[3]

...

Found one Java-level deadlock:

=============================

"Thread-4":

  waiting to lock monitor 7ff6530037f0 (object 7dc0bc000, a sun.misc.Launcher$AppClassLoader),

  which is held by "main"

"main":

  waiting to lock monitor 7ff6530035f8 (object 7dc0b8de8, a java.lang.Runtime),

  which is held by "Thread-5"

"Thread-5":

  waiting to lock monitor 7ff6530037f0 (object 7dc0bc000, a sun.misc.Launcher$AppClassLoader),

  which is held by "main"

Java stack information for the threads listed above:

===================================================

"Thread-4":

at com.google.gwt.dev.javac.PersistentUnitCache$5.run(PersistentUnitCache.java:273)

"main":

at java.lang.Runtime.loadLibrary0(Runtime.java:815)

- waiting to lock <7dc0b8de8> (a java.lang.Runtime)

at java.lang.System.loadLibrary(System.java:1045)

at sun.security.pkcs11.wrapper.PKCS11$1.run(PKCS11.java:88)

at java.security.AccessController.doPrivileged(Native Method)

at sun.security.pkcs11.wrapper.PKCS11.<clinit>(PKCS11.java:86)

at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:281)

at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:86)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

at sun.security.jca.ProviderConfig$4.run(ProviderConfig.java:262)

at java.security.AccessController.doPrivileged(Native Method)

at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:244)

at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:224)

- locked <7dc0bc000> (a sun.misc.Launcher$AppClassLoader)

at sun.security.jca.ProviderList.getProvider(ProviderList.java:215)

at sun.security.jca.ProviderList.getService(ProviderList.java:313)

at sun.security.jca.GetInstance.getInstance(GetInstance.java:140)

at java.security.Security.getImpl(Security.java:659)

at java.security.MessageDigest.getInstance(MessageDigest.java:129)

at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1754)

at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:50)

at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:203)

at java.security.AccessController.doPrivileged(Native Method)

at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:200)

at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:556)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1599)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1494)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1748)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)

at com.google.gwt.dev.javac.CompilationUnitArchive.createFromStream(CompilationUnitArchive.java:58)

at com.google.gwt.dev.javac.CompilationUnitArchive.createFromURL(CompilationUnitArchive.java:65)

at com.google.gwt.dev.ArchivePreloader.preloadArchives(ArchivePreloader.java:63)

at com.google.gwt.dev.Precompile.precompile(Precompile.java:243)

at com.google.gwt.dev.Precompile.precompile(Precompile.java:229)

at com.google.gwt.dev.Precompile.precompile(Precompile.java:141)

at com.google.gwt.dev.Compiler.run(Compiler.java:232)

at com.google.gwt.dev.Compiler.run(Compiler.java:198)

at com.google.gwt.dev.Compiler$1.run(Compiler.java:170)

at com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:88)

at com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:82)

at com.google.gwt.dev.Compiler.main(Compiler.java:177)

"Thread-5":

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:293)

- waiting to lock <7dc0bc000> (a sun.misc.Launcher$AppClassLoader)

at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

at java.util.ResourceBundle$RBClassLoader.loadClass(ResourceBundle.java:435)

at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2289)

at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1364)

at java.util.ResourceBundle.findBundle(ResourceBundle.java:1328)

at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1224)

at java.util.ResourceBundle.getBundle(ResourceBundle.java:747)

at java.awt.Toolkit$3.run(Toolkit.java:1616)

at java.security.AccessController.doPrivileged(Native Method)

at java.awt.Toolkit.<clinit>(Toolkit.java:1612)

at java.lang.ClassLoader$NativeLibrary.load(Native Method)

at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1827)

- locked <7dc0b6268> (a java.util.Vector)

- locked <7dc0b6288> (a java.util.Vector)

at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1724)

at java.lang.Runtime.loadLibrary0(Runtime.java:823)

- locked <7dc0b8de8> (a java.lang.Runtime)

at java.lang.System.loadLibrary(System.java:1045)

at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50)

at java.security.AccessController.doPrivileged(Native Method)

at apple.awt.CGraphicsEnvironment.<clinit>(CGraphicsEnvironment.java:23)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:171)

at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:68)

- locked <7fb8a6de8> (a java.lang.Class for java.awt.GraphicsEnvironment)

at com.google.gwt.dev.GraphicsInitThread.run(GraphicsInitThread.java:39)

Found 1 deadlock.

[Saša Živkov]

On Thu, May 2, 2013 at 2:11 PM, Saša Živkov <ziv...@gmail.com> wrote:

Just tried the buck build. After a short time nothing moves, no CPU activity [1]

There are quite some Compiler processes [2].

And each of them is dead-locked. The thread dump shows the following dead-lock for all of them [3].

Anybody else seeing the same issue?

Any idea?

Update: after stopping the buck build :gerrit with CTRL-C all the compiler processes

had to be killed manually.

Running the buck build :gerrit again resulted in a successful build:

[d038025 ~/dev/gerrit ((8e7b5e1...) *)]$ buck build :gerrit

BUILDING //:gerrit

BUILD SUCCESSFUL

[Saša Živkov]

On Thu, May 2, 2013 at 2:18 PM, Saša Živkov <ziv...@gmail.com> wrote:

On Thu, May 2, 2013 at 2:11 PM, Saša Živkov <ziv...@gmail.com> wrote:

Just tried the buck build. After a short time nothing moves, no CPU activity [1]

There are quite some Compiler processes [2].

And each of them is dead-locked. The thread dump shows the following dead-lock for all of them [3].

Anybody else seeing the same issue?

Any idea?

Update: after stopping the buck build :gerrit with CTRL-C all the compiler processes

had to be killed manually.

Running the buck build :gerrit again resulted in a successful build:

[d038025 ~/dev/gerrit ((8e7b5e1...) *)]$ buck build :gerrit

BUILDING //:gerrit

BUILD SUCCESSFUL

Update 2: the dead-lock problem is there again. It happens more often than the successful build :-(

The only new thing I discovered is that when the dead-lock happens there is only one Compiler process.

The 10 dead-locked Compiler processes I initially reported are probably remains of the previous buck builds.

[Thomas Broyer]

On Thursday, May 2, 2013 2:51:04 PM UTC+2, zivkov wrote:

On Thu, May 2, 2013 at 2:18 PM, Saša Živkov <ziv...@gmail.com> wrote:

On Thu, May 2, 2013 at 2:11 PM, Saša Živkov <ziv...@gmail.com> wrote:

Just tried the buck build. After a short time nothing moves, no CPU activity [1]

There are quite some Compiler processes [2].

And each of them is dead-locked. The thread dump shows the following dead-lock for all of them [3].

Anybody else seeing the same issue?

Any idea?

Update: after stopping the buck build :gerrit with CTRL-C all the compiler processes

had to be killed manually.

Running the buck build :gerrit again resulted in a successful build:

[d038025 ~/dev/gerrit ((8e7b5e1...) *)]$ buck build :gerrit

BUILDING //:gerrit

BUILD SUCCESSFUL

Update 2: the dead-lock problem is there again. It happens more often than the successful build :-(

The only new thing I discovered is that when the dead-lock happens there is only one Compiler process.

The 10 dead-locked Compiler processes I initially reported are probably remains of the previous buck builds.

Given that the com.google.gwt.dev.Compiler is a forked JVM, I doubt the deadlock is attributable to the Buck build (and thus should also happen in the Maven build, which also forks a JVM; if it doesn't, it might be environmental: different JDK being used, different JVM options, different Compiler options).

Could you please report it in the GWT issue tracker? https://code.google.com/p/google-web-toolkit/issues/entry

[Shawn Pearce]

On Thu, May 2, 2013 at 2:09 AM, Pursehouse, David (Sony Mobile)
<David.Pu...@sonymobile.com> wrote:
> I've been playing with the buck build this afternoon.
>
> Overall it seems to work as expected; it's faster than the Maven build and it was even able to download all the dependencies through my corporate proxy without me having to twiddle with anything. Yay!
>
> There are a few (minor) issues though:
>
> - The gerrit build fails unless using buck built with Shawn's patch [1] applied.

Hopefully we can get this upstream in buck soon enough.

> - The python scripts require 2.7 or above. I've had to make a few workarounds to make it work on my 2.6.5 installation; I've uploaded the changes in case anyone else is stuck on 2.6.x [2].

I plan on integrating all of your suggestions to make this work out of
the box on 2.6.5. Would you mind if I squashed all of
https://gerrit-review.googlesource.com/#/c/45231/ into my change? I'll
add an attribution in the commit message footer.

> - The plugin submodules don't work due to the relevant changes still being under review; they need to be manually cherry-picked and submodules reinitialized.

Obviously this is just caused by git submodule not being able to pick
up these pending changes. Its a workflow problem no matter what build
system is used. :-(

> - Plugins cannot be built from within the plugins folder.

They can be built from plugins, but not e.g. plugins/replication. I'm
not yet sure how we will solve that. Hey, at least you can build a
single subpackage anytime. I never could do that with Maven.

[Shawn Pearce]

Dammit. I missed including the rules. I think this is how I will fix it:

diff --git a/gerrit-acceptance-tests/BUCK b/gerrit-acceptance-tests/BUCK
index 9180f46..c3e82ac 100644
--- a/gerrit-acceptance-tests/BUCK
+++ b/gerrit-acceptance-tests/BUCK
@@ -8,7 +8,6 @@ java_test(
name = 'acceptance_tests',
srcs = glob(['src/test/java/**/*.java']),
deps = TEST + [
- '//gerrit-server:common_rules',
'//gerrit-launcher:launcher',
'//gerrit-pgm:pgm',
'//lib:servlet-api-3_0',
diff --git a/gerrit-pgm/BUCK b/gerrit-pgm/BUCK
index 3d70242..77a66b3 100644
--- a/gerrit-pgm/BUCK
+++ b/gerrit-pgm/BUCK
@@ -3,6 +3,7 @@ java_library2(
srcs = glob(['src/main/java/**/*.java']),
resources = glob(['src/main/resources/**/*']),
deps = [
+ '//gerrit-server:common_rules',
'//gerrit-server:server',
'//gerrit-httpd:httpd',
'//gerrit-sshd:sshd',
diff --git a/gerrit-war/BUCK b/gerrit-war/BUCK
index 4b96cee..722c0b3 100644
--- a/gerrit-war/BUCK
+++ b/gerrit-war/BUCK
@@ -2,6 +2,7 @@ java_library(
name = 'init',
srcs = glob(['src/main/java/**/*.java']),
deps = [
+ '//gerrit-server:common_rules',
'//gerrit-server:server',
'//gerrit-httpd:httpd',
'//gerrit-sshd:sshd',

[Shawn Pearce]

Is this happening because lib/gwt/compiler.py passes
-Dgwt.persistentunitcachedir=$TMP/unit_cache but e.g. the Maven build
doesn't?

[Thomas Broyer]

No. As the Maven build outputs to a folder, it'll use ${war}/.. by default (which should be target/):

https://code.google.com/p/google-web-toolkit/source/browse/tags/2.5.0/dev/core/src/com/google/gwt/dev/Compiler.java#213 (I don't understand why it only checks for ".jar" suffix, looks like a bug to me; but is irrelevant here unless I misread something)

https://code.google.com/p/google-web-toolkit/source/browse/tags/2.5.0/dev/core/src/com/google/gwt/dev/javac/UnitCacheFactory.java

When using Maven, you should see a gerrit-gwtui/target/gwt-unitCache folder with gwt-unitCache-xxx files in it.

[Saša Živkov]

On Thu, May 2, 2013 at 2:51 PM, Saša Živkov <ziv...@gmail.com> wrote:

On Thu, May 2, 2013 at 2:18 PM, Saša Živkov <ziv...@gmail.com> wrote:

On Thu, May 2, 2013 at 2:11 PM, Saša Živkov <ziv...@gmail.com> wrote:

Just tried the buck build. After a short time nothing moves, no CPU activity [1]

There are quite some Compiler processes [2].

And each of them is dead-locked. The thread dump shows the following dead-lock for all of them [3].

Anybody else seeing the same issue?

Any idea?

Update: after stopping the buck build :gerrit with CTRL-C all the compiler processes

had to be killed manually.

Running the buck build :gerrit again resulted in a successful build:

[d038025 ~/dev/gerrit ((8e7b5e1...) *)]$ buck build :gerrit

BUILDING //:gerrit

BUILD SUCCESSFUL

Update 2: the dead-lock problem is there again. It happens more often than the successful build :-(

The only new thing I discovered is that when the dead-lock happens there is only one Compiler process.

The 10 dead-locked Compiler processes I initially reported are probably remains of the previous buck builds.

Update 3: after installing JDK-7 and uninstalling JDK-6 buck build works well!

Platform: OSX Lion.

[Shawn Pearce]

On Fri, May 3, 2013 at 1:46 AM, Saša Živkov <ziv...@gmail.com> wrote:
>>>> Just tried the buck build. After a short time nothing moves, no CPU
>>>> activity [1]
>>>> There are quite some Compiler processes [2].
>>>> And each of them is dead-locked. The thread dump shows the following
>>>> dead-lock for all of them [3].

...

> Update 3: after installing JDK-7 and uninstalling JDK-6 buck build works
> well!
> Platform: OSX Lion.

http://facebook.github.io/buck/setup/install.html indicates the
requirements are:

* Oracle JDK 7
* Ant
* Python 2.7
* Git

It seems to be OK with Python 2.6.5, and I find it odd that it was the
GWT compiler that deadlocked under JDK-6. I haven't studied all of the
BUCK implementation code yet but I think their requirement for Ant is
to bootstrap BUCK. The Facebook team does want to start delivering
binary "releases" of BUCK so you don't need to bootstrap it yourself,
which would drop the Ant requirement.

[Thomas Broyer]

On Friday, May 3, 2013 7:03:09 PM UTC+2, Shawn Pearce wrote:

On Fri, May 3, 2013 at 1:46 AM, Saša Živkov <ziv...@gmail.com> wrote:
>>>> Just tried the buck build. After a short time nothing moves, no CPU
>>>> activity [1]
>>>> There are quite some Compiler processes [2].
>>>> And each of them is dead-locked. The thread dump shows the following
>>>> dead-lock for all of them [3].
...
> Update 3: after installing JDK-7 and uninstalling JDK-6 buck build works
> well!
> Platform: OSX Lion.

http://facebook.github.io/buck/setup/install.html indicates the
requirements are:

* Oracle JDK 7
* Ant
* Python 2.7
* Git

It seems to be OK with Python 2.6.5,

Buck forks a python process [1] with the com/facebook/buck/parser/buck.py script for parsing BUCK files, and that script uses argparse [2], which I read in the review [3] isn't in Python 2.6.5 (at least by default; David had to install python-argparse).

[1] https://github.com/facebook/buck/blob/3ffe61a2cbc549805d73716154614fde426c8f35/src/com/facebook/buck/json/BuildFileToJsonParser.java#L160

[2] https://github.com/facebook/buck/blob/3ffe61a2cbc549805d73716154614fde426c8f35/src/com/facebook/buck/parser/buck.py

[3] https://gerrit-review.googlesource.com/#/c/45110/13/lib/download_jar.py

[Chad Horohoe]

While I appreciate all the hard work that's gone into trying to make this work,

I think there's a couple of fatal flaws with Buck that make it untenable as a

build system:

- In a couple of days of fighting this, I've just now managed to get a working

build using OpenJDK7. This may be operator error, but one should never spend

this long using their *build tool*. By comparison, I can do a full release build of

Gerrit (`mvn clean install -e -Dgerrit.include-documentation=true -P all`) in less

than 5 minutes (still <10m if I delete my entire .m2/repository too).

- Introducing a new build tool breaks existing integration setups that build Gerrit.

Depending on a company's policies, this may or may not be easy to deal with.

For me, it means I'll have to adjust a bunch of Jenkins jobs (can't use the maven

plugin), as well as package Buck for Ubuntu.

- We're implicitly raising the requirements for building to (Oracle|Open) JDK 7.

Currently, everything Just Works with the JDK 6 too. This is not a problem for

me, and I don't know of anyone who's said it'd be a problem, but I don't know if

it's been sufficiently highlighted.

I realize we're having some issues with Maven, but for me the comfort of sticking

with Maven (coupled with ways to improve its result or speed it up) just outweigh

the trouble of dealing with Buck. It's certainly an interesting tool (and the build

times are impressive, once you get them), but I'm just not sure if it's the tool for

Gerrit.

-Chad

[Thomas Broyer]

For those interested, I blogged about Maven, Buck and build tools in general: http://blog.ltgt.net/in-quest-of-the-ultimate-build-tool/

[Saša Živkov]

On Mon, May 6, 2013 at 9:05 PM, Chad Horohoe <chor...@wikimedia.org> wrote:

While I appreciate all the hard work that's gone into trying to make this work,

I think there's a couple of fatal flaws with Buck that make it untenable as a

build system:

- In a couple of days of fighting this, I've just now managed to get a working

build using OpenJDK7. This may be operator error, but one should never spend

this long using their *build tool*. By comparison, I can do a full release build of

 

We were just making the first steps with a new build tool. Some issues are to

be expected at this stage. 

 

Gerrit (`mvn clean install -e -Dgerrit.include-documentation=true -P all`) in less

than 5 minutes (still <10m if I delete my entire .m2/repository too).

This is without the core plugins included.

Try building it with the core plugins.

- Introducing a new build tool breaks existing integration setups that build Gerrit.

Depending on a company's policies, this may or may not be easy to deal with.

For me, it means I'll have to adjust a bunch of Jenkins jobs (can't use the maven

plugin), as well as package Buck for Ubuntu.

- We're implicitly raising the requirements for building to (Oracle|Open) JDK 7.

Currently, everything Just Works with the JDK 6 too. This is not a problem for

me, and I don't know of anyone who's said it'd be a problem, but I don't know if

it's been sufficiently highlighted.

I realize we're having some issues with Maven, but for me the comfort of sticking

with Maven (coupled with ways to improve its result or speed it up) just outweigh

the trouble of dealing with Buck. It's certainly an interesting tool (and the build

times are impressive, once you get them), but I'm just not sure if it's the tool for

Gerrit.

 

We cannot be sure if we don't try it :-) 

[Edwin Kempin]

I'm surprised to find the changes about the Buck build beeing abandoned.
I was keeping silence about the Buck build so far, basically because being on Windows
I couldn't try it out.
I do see the issues with Maven that Shawn described very well and I find it painful to work
with Maven especially if we talk about doing release builds.
It's painful, but not yet as painful that I would invest a lot of my own time to switch to
another build tool.
Still any work which would improve the current situation would be very welcome to me,
even if it would mean that I have to abandon Windows ;-)
So I find it a pity to see that the Buck build is given up now.

2013/5/7 Saša Živkov <ziv...@gmail.com>

[Brad Larson]

On Tuesday, May 7, 2013 4:41:05 AM UTC-5, Edwin Kempin wrote:

I'm surprised to find the changes about the Buck build beeing abandoned.
I was keeping silence about the Buck build so far, basically because being on Windows
I couldn't try it out.
I do see the issues with Maven that Shawn described very well and I find it painful to work
with Maven especially if we talk about doing release builds.
It's painful, but not yet as painful that I would invest a lot of my own time to switch to
another build tool.
Still any work which would improve the current situation would be very welcome to me,
even if it would mean that I have to abandon Windows ;-)
So I find it a pity to see that the Buck build is given up now.

I'm with Edwin.  I haven't played around with Buck because I haven't had time to make any Gerrit contributions in way too long, but the research sounds promising.  I am frustrated with Maven on a near-constant basis and am more than willing to put up with some growing pains if it allows us to switch to something better.

One example of my Maven frustrations is the uncertainty when creating a new plugin (for Gerrit, Jenkins/Hudson, etc).  It is near impossible to figure out what attributes need to be set in the pom and which can be ignored.  I try looking at a few examples from the more popular plugins, and they never seem to agree.  With hundreds of plugins, Jenkins is far worse about this than Gerrit, but I worry it is only a matter of time...

I'm hopeful Buck can be revisited in future 3.0 discussion.
 

[Chad Horohoe]

I'm also fine with iterating this further, and I don't want my original e-mail to

be read as blind devotion to Maven. I too have had my share of wtf moments

with Maven, and the pain points raised are valid. My main point was that Buck,

as it stands now, just probably wasn't the answer (and I was thus a little leery

of moving master over to it at this time). Experimenting is good, and I'm not

opposed to trying out other solutions.

-Chad

[Dave Borowitz]

I think many of those in favor of Buck have perhaps not been giving the idea the support we think it deserves. Consensus among contributors at the hackathon today was that it is worth keeping Buck around, at least in the short term.

For a short while, we are willing to maintain parallel build systems while contributors get to know Buck and some of its kinks get ironed out. We will have a set of exit criteria (e.g. Windows support), and if at the end of some short time horizon (e.g. 3 months) Buck is not suiting our needs, then we will back it out. If it is suiting our needs, we will remove Maven support, presumably coinciding with a new release. We will not make any non-essential changes to module layout etc. while in this parallel build system state.

Expect an email in the next day from someone who was taking notes (Fredrik?) with more detailed exit criteria, and let's move the discussion to that thread. 

[Edwin Kempin]

Thanks for summarizing this! I think this is really the right way forward. Buck looks exciting and we should really give it a serious try.
Shawn, your efforts with setting this up are really appreciated! Getting the build right is always an ungrateful task, but in the end everybody benefits from it!

[Shawn Pearce]

So I refreshed the Buck change
https://gerrit-review.googlesource.com/45110 with my latest revision.
I have an Eclipse project generator working and producing an
environment in Eclipse I can run a debug init or daemon out of. This
gives the pgm_debug equivalent using precompiled JS and not the GWT
debugger. See the commit message for setup instructions.

Next up is to work on the GWT debug case from Eclipse, but I think
that is solvable so I don't think its going to derail anything.

There are 26 compile errors in WrappedContext. This is a particularly
painful class for me. It highlights the strength of BUCK at being able
to compile the class with servlet API 2.5 while using it in a 3.0
container, but the way this is setup in Eclipse as a single project
prevents Eclipse from having the correct compile time classpath here.
One would argue Gerrit should only use a single servlet API version
consistently throughout our source code, which at this point in time
would be servlet 3.0. Except my internal build for gerrit-review links
against a 2.5 container and can't compile with a 3.0 API. Except I
don't use plugins there, and this class exists only for plugins, so I
shouldn't have to build it. Except its somewhat tangled with some
other code I do compile so *sigh*. This single damn class surfaces
almost anything that could go wrong for me.

[Matthias Sohn]

On Wed, May 8, 2013 at 2:36 AM, Shawn Pearce <s...@google.com> wrote:

So I refreshed the Buck change
https://gerrit-review.googlesource.com/45110 with my latest revision.
I have an Eclipse project generator working and producing an
environment in Eclipse I can run a debug init or daemon out of. This
gives the pgm_debug equivalent using precompiled JS and not the GWT
debugger. See the commit message for setup instructions.

Next up is to work on the GWT debug case from Eclipse, but I think
that is solvable so I don't think its going to derail anything.

There are 26 compile errors in WrappedContext. This is a particularly
painful class for me. It highlights the strength of BUCK at being able
to compile the class with servlet API 2.5 while using it in a 3.0
container, but the way this is setup in Eclipse as a single project
prevents Eclipse from having the correct compile time classpath here.
One would argue Gerrit should only use a single servlet API version
consistently throughout our source code, which at this point in time
would be servlet 3.0. Except my internal build for gerrit-review links
against a 2.5 container and can't compile with a 3.0 API. Except I
don't use plugins there, and this class exists only for plugins, so I
shouldn't have to build it. Except its somewhat tangled with some
other code I do compile so *sigh*. This single damn class surfaces
almost anything that could go wrong for me.

could the servlet API version become a parameter for the build so that you

can compile it without plugins against 2.5 and others build against 3.0 ?

Will give this a try tomorrow morning, need some sleep now

--

Matthias

[Shawn Pearce]

On Tue, May 7, 2013 at 5:51 PM, Matthias Sohn <matthi...@gmail.com> wrote:
> On Wed, May 8, 2013 at 2:36 AM, Shawn Pearce <s...@google.com> wrote:
>>
>> So I refreshed the Buck change
>> https://gerrit-review.googlesource.com/45110 with my latest revision.
>> I have an Eclipse project generator working and producing an
>> environment in Eclipse I can run a debug init or daemon out of. This
>> gives the pgm_debug equivalent using precompiled JS and not the GWT
>> debugger. See the commit message for setup instructions.
>>
>> Next up is to work on the GWT debug case from Eclipse, but I think
>> that is solvable so I don't think its going to derail anything.

I just pushed d5a3e75 to update the change with GWT debug support.

>> There are 26 compile errors in WrappedContext. This is a particularly
>> painful class for me. It highlights the strength of BUCK at being able
>> to compile the class with servlet API 2.5 while using it in a 3.0
>> container, but the way this is setup in Eclipse as a single project
>> prevents Eclipse from having the correct compile time classpath here.
>> One would argue Gerrit should only use a single servlet API version
>> consistently throughout our source code, which at this point in time
>> would be servlet 3.0. Except my internal build for gerrit-review links
>> against a 2.5 container and can't compile with a 3.0 API. Except I
>> don't use plugins there, and this class exists only for plugins, so I
>> shouldn't have to build it. Except its somewhat tangled with some
>> other code I do compile so *sigh*. This single damn class surfaces
>> almost anything that could go wrong for me.

d5a3e75 also fixes this error.

> could the servlet API version become a parameter for the build so that you
> can compile it without plugins against 2.5 and others build against 3.0 ?

Its not that simple unfortunately. Servlet API 3.0 defines new methods
and types that our implementation has to implement that causes the
same implementation to fail to compile on 2.5. I created a new change
tonight to reimplement this glue using java.lang.reflect.Proxy.

[Shawn Pearce]

On Tue, May 7, 2013 at 11:32 PM, Shawn Pearce <s...@google.com> wrote:
> On Tue, May 7, 2013 at 5:51 PM, Matthias Sohn <matthi...@gmail.com> wrote:
>> On Wed, May 8, 2013 at 2:36 AM, Shawn Pearce <s...@google.com> wrote:
>>>
>>> So I refreshed the Buck change
>>> https://gerrit-review.googlesource.com/45110 with my latest revision.
>>> I have an Eclipse project generator working and producing an
>>> environment in Eclipse I can run a debug init or daemon out of. This
>>> gives the pgm_debug equivalent using precompiled JS and not the GWT
>>> debugger. See the commit message for setup instructions.
>>>
>>> Next up is to work on the GWT debug case from Eclipse, but I think
>>> that is solvable so I don't think its going to derail anything.
>
> I just pushed d5a3e75 to update the change with GWT debug support.

I need to recheck unit and acceptance tests, but I think d5a3e75 is
now at a position where both my Mac and Linux development machines are
giving me a working development environment, and I'm going to start
using it as my daily tooling. We'll see what happens. :-)

[Shawn Pearce]

On Tue, May 7, 2013 at 11:52 PM, Shawn Pearce <s...@google.com> wrote:
>
> I need to recheck unit and acceptance tests

The tests don't work correctly under Buck if you have more than zero
CPUs. Buck runs multiple java_test() rules in parallel. Many of our
tests rely on JGit's test framework code to create (and clean up)
temporary Git repositories. Unfortunately JGit is hard-coded to use
"$(pwd)/target/trash" as the location for the test to create,
populate, and recursively delete. Which causes consistent random
failures as one test in e.g. gerrit-server deletes the files used by
gerrit-httpd. Worse JGit has a bad unique name generator, so some
tests also manage to use the same "unique" test repository. *sigh*

[Shawn Pearce]

Tests are fixed.

I like how fast Eclipse now is.

I like how launching buck_daemon from Eclipse can quickly check the
GWT code is up-to-date, and if not, recompiles it. This saves me from
having stale JS, I can edit UI code and just launch and *boom* its
doing the new code using the native JS version, which sometimes
differs in behavior from the GWT hosted mode debugger thing.

Every plugin under //plugins is now loaded automatically into Eclipse.
Just drop a plugin there with a suitable BUCK file using the
gerrit_plugin() or gerrit_extension() rule, update with `buck build
:eclipse`, and refresh the project in Eclipse. Plugin code loaded this
way winds up in the system class loader during debugging sessions, so
Eclipse can trivially hot-swap code.

[Dariusz Luksza]

WOW, that sounds really AWESOME! Looking forward for buck based build in master

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--

Best regards

GSM: +49 017 445 41235
Blog: http://luksza.org
LinkedIn: http://www.linkedin.com/in/dariuszluksza

[Dariusz Luksza]

Shawn, if we decide to use some JS libraries like CodeMirror or
AngularJs, how hard it would be to teach BUCK to fetch them form
internet and put in proper place so that they can be loaded during
development and be included in WAR file. The idea behind that is to
not include external JS libs in Gerrit git repository, just fetch them
during initialization/build time.

[Dave Borowitz]

The Maven dependency bit is just a script to download some jar from some URL, it could be easily adapted to fetching any kind of dependency.

[David Pursehouse]

On Thursday, May 9, 2013 5:53:49 AM UTC+1, Shawn Pearce wrote:

Tests are fixed.

Is this now in a state good enough to be merged?  I'm sure there are plenty of people who will be happy to contribute further fixes and tweaks; right now it's not easy to do that with the change still being open.  See my python 2.6.x fixes for example.

[Dave Borowitz]

I will go one step further and say that if this is merged, it will noticeably increase productivity at the hackathon. Since Shawn is probably asleep I might just consider merging it without his approval :)

[Shawn Pearce]

On Thu, May 9, 2013 at 3:23 AM, Dave Borowitz <dbor...@google.com> wrote:
> I will go one step further and say that if this is merged, it will
> noticeably increase productivity at the hackathon. Since Shawn is probably
> asleep I might just consider merging it without his approval :)

I see this was merged. I was going to suggest merging the latest patch
set and continue iterations in-tree, so I'm happy to see we have done
this.

[Shawn Pearce]

On Thu, May 9, 2013 at 12:21 AM, Dave Borowitz <dbor...@google.com> wrote:
> The Maven dependency bit is just a script to download some jar from some
> URL, it could be easily adapted to fetching any kind of dependency.
>
> On May 9, 2013 8:13 AM, "Dariusz Luksza" <dariusz...@gmail.com> wrote:
>>
>> Shawn, if we decide to use some JS libraries like CodeMirror or
>> AngularJs, how hard it would be to teach BUCK to fetch them form
>> internet and put in proper place so that they can be loaded during
>> development and be included in WAR file. The idea behind that is to
>> not include external JS libs in Gerrit git repository, just fetch them
>> during initialization/build time.

Yes, this should be fairly simple to download JS and other assets.
Most of the "download" code doesn't care what it is fetching, or
where.

[Bruce Zu]

$ buck --version
buck version c4df74bef4e101a7e5d0176831825b7946ea64a3
$ python --version
Python 2.7
$ git log -1 --format="%H"
7eef8edcba77058e4112c87b2d220517265b412d
$ echo $http_proxy
http://proxy.global.sonyericsson.net:8080
$ buck build eclipse
BUILDING //tools/eclipse:eclipse
http://repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk16/1.44/bcprov-jdk16-1.44.jar:
expected 6327a5f7a3dc45e0fd735adb5d08c5a74c05c20c
received de6b5225831762a91db5ec6b5d577a57ca90b450
         ov-jdk16-1.44.jar-6327a5f7a3dc45e0fd735adb5d08c5a74c05c20c

http://repo1.maven.org/maven2/org/apache/velocity/velocity/1.6.4/velocity-1.6.4.jar:
expected fcc58693dd8fc83d714fba149789be37cc19b66d
received 4af239e975010e59e704aadc61874e8249ebd436
         city-1.6.4.jar-fcc58693dd8fc83d714fba149789be37cc19b66d

BUILD FAILED: com.facebook.buck.step.StepFailedException: Failed with 1: SRCS= OUT=/home/CORPUSERS/28850272/gerrit-code/google-gerrit/gerrit/buck-out/gen/lib/bouncycastle/bcprov/bcprov-jdk16-1.44.jar DEPS= SRCDIR=/home/CORPUSERS/28850272/gerrit-code/google-gerrit/gerrit/buck-out/gen/lib/bouncycastle/bcprov__download_bin__srcs TMP=/home/CORPUSERS/28850272/gerrit-code/google-gerrit/gerrit/buck-out/gen/lib/bouncycastle/bcprov__download_bin__tmp /bin/bash -ec 'PYTHONPATH= python /home/CORPUSERS/28850272/gerrit-code/google-gerrit/gerrit/tools/download_jar.py -o $OUT -u MAVEN_CENTRAL:/org/bouncycastle/bcprov-jdk16/1.44/bcprov-jdk16-1.44.jar -v 6327a5f7a3dc45e0fd735adb5d08c5a74c05c20c'
 


any hint?

Thanks!
/Bruce

[Saša Živkov]

On Tue, May 21, 2013 at 10:32 AM, Bruce Zu <bruc...@sonymobile.com> wrote:

$ buck --version
buck version c4df74bef4e101a7e5d0176831825b7946ea64a3
$ python --version
Python 2.7
$ git log -1 --format="%H"
7eef8edcba77058e4112c87b2d220517265b412d
$ echo $http_proxy
http://proxy.global.sonyericsson.net:8080
$ buck build eclipse
BUILDING //tools/eclipse:eclipse
http://repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk16/1.44/bcprov-jdk16-1.44.jar:
expected 6327a5f7a3dc45e0fd735adb5d08c5a74c05c20c
received de6b5225831762a91db5ec6b5d577a57ca90b450
         ov-jdk16-1.44.jar-6327a5f7a3dc45e0fd735adb5d08c5a74c05c20c

http://repo1.maven.org/maven2/org/apache/velocity/velocity/1.6.4/velocity-1.6.4.jar:
expected fcc58693dd8fc83d714fba149789be37cc19b66d
received 4af239e975010e59e704aadc61874e8249ebd436
         city-1.6.4.jar-fcc58693dd8fc83d714fba149789be37cc19b66d

BUILD FAILED: com.facebook.buck.step.StepFailedException: Failed with 1: SRCS= OUT=/home/CORPUSERS/28850272/gerrit-code/google-gerrit/gerrit/buck-out/gen/lib/bouncycastle/bcprov/bcprov-jdk16-1.44.jar DEPS= SRCDIR=/home/CORPUSERS/28850272/gerrit-code/google-gerrit/gerrit/buck-out/gen/lib/bouncycastle/bcprov__download_bin__srcs TMP=/home/CORPUSERS/28850272/gerrit-code/google-gerrit/gerrit/buck-out/gen/lib/bouncycastle/bcprov__download_bin__tmp /bin/bash -ec 'PYTHONPATH= python /home/CORPUSERS/28850272/gerrit-code/google-gerrit/gerrit/tools/download_jar.py -o $OUT -u MAVEN_CENTRAL:/org/bouncycastle/bcprov-jdk16/1.44/bcprov-jdk16-1.44.jar -v 6327a5f7a3dc45e0fd735adb5d08c5a74c05c20c'

Looks like /bin/bash exited with 1.

Are you on the latest master?

The download_jar.py was renamed to download_file.py in 8713eb859eb4d6680f7e0da55725141aebd907e2.

However, your build is trying to run the download_jar.py.

Otherwise, in your case, I would try executing just the download_jar.py (or download_file.py for the master branch)

as given above (with the variables replaced with their values) and check where it exits.

 

 


any hint?

Thanks!
/Bruce

--

[Richard Bywater]

I'm seeing the similar issue on master. Looks to me like the SHA1 sums its checking against are wrong? (I manually checked a couple of Jar files and the sum was right against the maven repo so not sure why they aren't matching)

Richard.

[Shawn Pearce]

On Tue, May 21, 2013 at 12:01 PM, Richard Bywater <ric...@byh2o.com> wrote:
> I'm seeing the similar issue on master. Looks to me like the SHA1 sums its
> checking against are wrong? (I manually checked a couple of Jar files and
> the sum was right against the maven repo so not sure why they aren't
> matching)
>

> On Tue, May 21, 2013 at 10:32 AM, Bruce Zu <bruc...@sonymobile.com> wrote:
>>
>> $ buck --version
>> buck version c4df74bef4e101a7e5d0176831825b7946ea64a3
>> $ python --version
>> Python 2.7
>> $ git log -1 --format="%H"
>> 7eef8edcba77058e4112c87b2d220517265b412d
>> $ echo $http_proxy
>> http://proxy.global.sonyericsson.net:8080
>> $ buck build eclipse
>> BUILDING //tools/eclipse:eclipse
>>
>> http://repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk16/1.44/bcprov-jdk16-1.44.jar:
>> expected 6327a5f7a3dc45e0fd735adb5d08c5a74c05c20c
>> received de6b5225831762a91db5ec6b5d577a57ca90b450
>> ov-jdk16-1.44.jar-6327a5f7a3dc45e0fd735adb5d08c5a74c05c20c

I'm not seeing this at all. Looks like you are talking to a Maven
mirror that disagrees with Maven Central.

[Richard Bywater]

That's what I was afraid someone would say :)

I'll have another go when I get a chance and hope the Maven gods have
seen fit for the mirrors to be in alignment...

Richard.

[Bruce Zu]

'rm -rf ~/.gerritcodereview/'
and make sure python is 2.7
'git clone'  the latest Gerrit code

'buck build download'
'buck build eclipse'
going successfully now.

Thanks a lot!
/Bruce

[Richard Bywater]

Thanks - seems to have fixed the problem for me too. I though there'd be a cache to try clearing somewhere but couldn't work out where it was :)

Richard.

--

[Pursehouse, David (Sony Mobile)]

> 'rm -rf ~/.gerritcodereview/'

This will also cause Gerrit's tmp folder to be wiped out.

It would be better to specifically wipe the buck-cache directory:

$ rm -rf ~/.gerritcodereview/buck-cache

[Shawn Pearce]

On Wed, May 22, 2013 at 1:39 AM, Pursehouse, David (Sony Mobile)
<David.Pu...@sonymobile.com> wrote:
>> 'rm -rf ~/.gerritcodereview/'
>
> This will also cause Gerrit's tmp folder to be wiped out.

Which is only temporary and can be removed anytime... so long as there
is no server running. I don't run servers as myself, except to debug,
so its usually safe to clear this directory. :-)

> It would be better to specifically wipe the buck-cache directory:
>
> $ rm -rf ~/.gerritcodereview/buck-cache

Yes. Hiding the cache like this makes it harder to clear. Not sure how
corruption entered, maybe we had a bad download and kept the cache
entry around.

[Bruce Zu]

I find some error when run on  v2.7-rc1-348-gace391b
$ buck build download_sources
BUILDING //:download_sources
BUILD FAILED: No rule found when resolving target //lib/codemirror:codemirror__download_src in build file //lib/codemirror/BUCK
BUILD FAILED: //:download_sources failed on step "genrule: PYTHONPATH= python /home/CORPUSERS/28850272/gerrit-code/google-gerrit/gerrit/tools/download_all.py --src" with exit code 1

update lib/codemirror/BUCK by adding below genrule can fix it, but I think it is not a right patch here
 
+
+
+genrule(
+  name = 'codemirror__download_src',
+  cmd = '${//tools:download_file}' +
+    ' -o $OUT' +
+    ' -u ' + URL +
+    ' -v ' + SHA1,
+  srcs = [],
+  deps = ['//tools:download_file'],
+  out = 'codemirror-' + VERSION + '-src.zip',

[Thomas Broyer]

On Wednesday, May 29, 2013 12:05:45 PM UTC+2, Bruce Zu wrote:

I find some error when run on  v2.7-rc1-348-gace391b
$ buck build download_sources
BUILDING //:download_sources
BUILD FAILED: No rule found when resolving target //lib/codemirror:codemirror__download_src in build file //lib/codemirror/BUCK
BUILD FAILED: //:download_sources failed on step "genrule: PYTHONPATH= python /home/CORPUSERS/28850272/gerrit-code/google-gerrit/gerrit/tools/download_all.py --src" with exit code 1

update lib/codemirror/BUCK by adding below genrule can fix it, but I think it is not a right patch here

Actually, the fix is probably to rename codemirror__download_bin to codemirror__download_src.

See https://gerrit-review.googlesource.com/46300

[Pursehouse, David (Sony Mobile)]

There are a couple of things to be aware of with the buck build on latest master.

 

1. Buck now requires Java 7 [1] and the build will fail if not installed.

 

2. Buck will try to update itself to the required revision, but this will fail if the default remote is not Shawn’s fork at googlesource.com [2], i.e. if you have originally cloned buck from github and then added googlesource as an additional remote.  Get around this by removing and re-cloning the repository from googlesource, or applying this patch that I have submitted [3] that changes `git fetch` to `git fetch --all`.

 

[1] https://github.com/facebook/buck/commit/952ceec8018f35ef0f47c130ea9c8c0308eca215

[2] https://gerrit.googlesource.com/buck.git

[3] https://github.com/facebook/buck/pull/32

 

 

[Shawn Pearce]

On Thu, May 30, 2013 at 8:23 PM, Pursehouse, David (Sony Mobile)
<David.Pu...@sonymobile.com> wrote:
> There are a couple of things to be aware of with the buck build on latest
> master.
>
>
>
> 1. Buck now requires Java 7 [1] and the build will fail if not installed.
>
>
>
> 2. Buck will try to update itself to the required revision, but this will
> fail if the default remote is not Shawn’s fork at googlesource.com [2], i.e.
> if you have originally cloned buck from github and then added googlesource
> as an additional remote. Get around this by removing and re-cloning the
> repository from googlesource, or applying this patch that I have submitted
> [3] that changes `git fetch` to `git fetch --all`.

Or just change your origin:

git config remote.origin.url https://gerrit.googlesource.com/buck

That is what I did on my computers. :-)

>
>
> [1]
> https://github.com/facebook/buck/commit/952ceec8018f35ef0f47c130ea9c8c0308eca215
>
> [2] https://gerrit.googlesource.com/buck.git
>
> [3] https://github.com/facebook/buck/pull/32
>
>
>
>
>