ANN: Preliminary Clojure Support in Buildr

33 views
Skip to first unread message

Daniel Spiewak

unread,
Feb 21, 2009, 4:33:30 PM2/21/09
to Clojure
I'm pleased to announce preliminary (and very experimental) support
for the Clojure AOT compiler and REPL within Apache Buildr (http://
buildr.apache.org). At present, this support is only available within
my Git fork available here: git://github.com/djspiewak/buildr.git
More specifically, Clojure support is available within the "clojure"
and "master" branches ("master" branch alone contains REPL support).
It should be possible to install this particular version of Buildr by
using the following commands, though I'm honestly not sure how up to
date GitHub's gem repository is:

gem sources -a http://gems.github.com
sudo gem install djspiewak-buildr

Once installed, Clojure support is activated in a project simply by
storing your .clj scripts within the src/main/clojure directory. Note
that the (ns) directive will need to match the subdirectory, otherwise
compilation will fail. By default, every script is compiled to the
target/classes directory. Namespaces are auto-detected from the
directory structure. Only updated files are re-compiled (based on
mtime of .clj file and its corresponding *__init.class). If you wish
to override the auto-detection and specify a reduced set of
namespaces, it can be done using the `compile.using` directive within
your project definition in your buildfile. Thusly:

define 'clojure-contrib' do
compile.using :libs => ['clojure.contrib.command-line',
'clojure.contrib.mmap']
end

Any scripts which are *not* pre-compiled will be copied verbatim to
the target/classes directory w.r.t. their position in the directory
structure. Note that you will need to have set CLOJURE_HOME for this
to work.

You will have to be using the "master" branch from my git repository
in order to use the Clojure REPL through Buildr (or install via the
gem command given above). To invoke, simply run the following command
somewhere in your project hierarchy:

buildr shell

This will launch the Clojure REPL pointing at your project's
dependencies and the updated target/classes directory (compilation is
re-run if necessary). Additionally, if you have a valid license for
JavaRebel, you can make use of it with the REPL by setting the
REBEL_HOME environment variable.

Note that you cannot mix Java and Clojure sources within the same
project. However, this is fairly easy to overcome by splitting the
languages into separate sub-projects. Thus, your top-level project
might contain all of your Clojure sources, while the sub-project might
contain Java. There are more details regarding this process on the
Buildr project page.

One thing to keep in mind is that Buildr was designed to serve as a
build system for more static languages (specifically: Java, Scala,
Groovy). Thus, it is pre-biased toward things like a separate
compilation phase (the REPL points to target/classes rather than src/
main/clojure).

Fair warning: this language support is *extremely* experimental and
probably not too reliable at this point. Also note that while it is
possible that Clojure support will be merged into the Buildr trunk in
future, it has not yet been decided one way or another (see
https://issues.apache.org/jira/browse/BUILDR-259). Use at your own
risk!

Christian Vest Hansen

unread,
Feb 26, 2009, 11:58:31 AM2/26/09
to clo...@googlegroups.com
Nice initiative!

However, it the net-ssh dependency has problems:

[cvh: ~]$ sudo gem install djspiewak-buildr
ERROR: While executing gem ... (Gem::RemoteFetcher::FetchError)
timed out (http://gems.rubyforge.org/gems/net-ssh-2.0.4.gem)
Aww... :(

>  However, this is fairly easy to overcome by splitting the
> languages into separate sub-projects.  Thus, your top-level project
> might contain all of your Clojure sources, while the sub-project might
> contain Java.  There are more details regarding this process on the
> Buildr project page.
>
> One thing to keep in mind is that Buildr was designed to serve as a
> build system for more static languages (specifically: Java, Scala,
> Groovy).  Thus, it is pre-biased toward things like a separate
> compilation phase (the REPL points to target/classes rather than src/
> main/clojure).
>
> Fair warning: this language support is *extremely* experimental and
> probably not too reliable at this point.  Also note that while it is
> possible that Clojure support will be merged into the Buildr trunk in
> future, it has not yet been decided one way or another (see
> https://issues.apache.org/jira/browse/BUILDR-259).  Use at your own
> risk!
> >
>



--
Venlig hilsen / Kind regards,
Christian Vest Hansen.

Daniel Spiewak

unread,
Feb 26, 2009, 12:17:12 PM2/26/09
to Clojure
Odd. Must be a problem with RubyForge. If you try again, does it
work?

Daniel

On Feb 26, 10:58 am, Christian Vest Hansen <karmazi...@gmail.com>
wrote:
> Nice initiative!
>
> However, it the net-ssh dependency has problems:
>
> [cvh: ~]$ sudo gem install djspiewak-buildr
> ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
>     timed out (http://gems.rubyforge.org/gems/net-ssh-2.0.4.gem)
>
>
>
> On Sat, Feb 21, 2009 at 10:33 PM, Daniel Spiewak <djspie...@gmail.com> wrote:
>
> > I'm pleased to announce preliminary (and very experimental) support
> > for the Clojure AOT compiler and REPL within Apache Buildr (http://
> > buildr.apache.org).  At present, this support is only available within
> > my Git fork available here: git://github.com/djspiewak/buildr.git
> > More specifically, Clojure support is available within the "clojure"
> > and "master" branches ("master" branch alone contains REPL support).
> > It should be possible to install this particular version of Buildr by
> > using the following commands, though I'm honestly not sure how up to
> > date GitHub's gem repository is:
>
> >  gem sources -ahttp://gems.github.com

Laurent PETIT

unread,
Feb 26, 2009, 12:21:23 PM2/26/09
to clo...@googlegroups.com
Wow, great !

Some notes I took while reading your e-mail : (those are some pitfalls I came through while implementing clojuredev's eclipse auto-build feature)

 * does it support namespaces separated in several files (handling files that begin with 'in-ns, or just not trying to compile them ?)
 * if so, will it support the scenario of multiple files per ns, where just another file (and not the file defining the ns) is modified ?
 * there is also another problem : if some macros are defined in a namespace A, used in a namespace B, then if you just change the macro and recompile namespace A, I don't think code using the macros in namespace B will see the changes ?

Those considerations, among other things, have made me consider just recompiling the entire project every time a resource or set of resources change, so that it is reliable, if not totally optimized (but so far, for the size of the projects, I guess it's still not a performance bottleneck).

My 0,02 €,

--
Laurent


2009/2/21 Daniel Spiewak <djsp...@gmail.com>

Daniel Spiewak

unread,
Feb 26, 2009, 12:23:32 PM2/26/09
to Clojure

> > Note that you cannot mix Java and Clojure sources within the same
> > project.
>
> Aww... :(

Joint-compilation is actually a hard problem normally. However, since
Clojure is late-bound, I should be able to do it without too much
horror. Actually, I should be able to do joint compilation with Java,
Scala or Groovy (Scala and Groovy would be mutually exclusive). I
just have to figure out how to trick Buildr into letting me register a
compiler in this fashion. So, the feature is coming, it's just not
immediately present.

Daniel

Daniel Spiewak

unread,
Feb 26, 2009, 1:29:04 PM2/26/09
to Clojure

>  * does it support namespaces separated in several files (handling files
> that begin with 'in-ns, or just not trying to compile them ?)
>  * if so, will it support the scenario of multiple files per ns, where just
> another file (and not the file defining the ns) is modified ?

I didn't even know Clojure's AOT compiler supported this. I thought
that the name of the file was forced to match the name of the
namespace it represents? If that is not the case, then the solution
would probably be to just not compile the in-ns files, though that
seems annoyingly-ugly.

Right now, compilation detection is exclusively based on filenames.
So if you have a file src/main/clojure/clojure/contrib/command-
line.clj, then Buildr will dispatch the compilation of namespace
"clojure.contrib.command-line". I thought about doing fancy parsing
to checkout the ns declarations, but given Clojure's dynamic nature, I
would never be able to *guarantee* any sort of accuracy in that
approach.

>  * there is also another problem : if some macros are defined in a namespace
> A, used in a namespace B, then if you just change the macro and recompile
> namespace A, I don't think code using the macros in namespace B will see the
> changes ?

This case didn't occur to me, but you're right that it would be
problematic. I should probably change the compilation to just
recompile everything when any file has changed. Darn those macros...

Daniel

Daniel Spiewak

unread,
Feb 26, 2009, 2:36:52 PM2/26/09
to Clojure
> > > Note that you cannot mix Java and Clojure sources within the same
> > > project.

It is supported in the latest changeset. The following configurations
are possible:

---------------------
Just Clojure:
* src/main/clojure

Clojure and Scala:
* src/main/clojure
* src/main/scala

Clojure, Scala and Java:
* src/main/clojure
* src/main/scala
* src/main/java

Clojure and Groovy:
* src/main/clojure
* src/main/groovy

Clojure, Groovy and Java:
* src/main/clojure
* src/main/groovy
* src/main/java

Clojure and Java:
* src/main/clojure
* src/main/java
---------------------

You'll notice that "Clojure, Scala and Groovy" is not among the
possibilities. This is because the Groovy joint compiler and the
Scala joint compiler both expect to be able to interleave Java
dependencies within their type resolution process. They do not have
support for the same functionality with each other, meaning that
circular dependencies cannot be compiled between the two languages.

Note that this is all predicated on a (possibly faulty) assumption
about the Clojure compiler: everything must be late-bound. If
Clojure's compiler needs to resolve types at compile-time, then this
could potentially fail in unexpected ways. Specifically, circular
dependencies between Clojure and Java/Scala/Groovy would not be
possible. In such a case, you would need to somehow factor the
circularity out into a separate .clj file and then exclude that file
from compilation (forcing late binding). I honestly don't know if
this is the compiler's behavior or not, but I thought it was worth the
warning.

One small change: in order to use this functionality now, you will
need to include the following line at the top of your buildfile:

require 'buildr/clojure'

Joint compilation with Scala/Java or Groovy/Java does not necessitate
an extra "require", but this first one is necessary to get the process
started and ensure that Clojure's compiler becomes the dominant
selection.

Daniel

Christian Vest Hansen

unread,
Feb 26, 2009, 4:16:45 PM2/26/09
to clo...@googlegroups.com
On Thu, Feb 26, 2009 at 6:17 PM, Daniel Spiewak <djsp...@gmail.com> wrote:
>
> Odd.  Must be a problem with RubyForge.  If you try again, does it
> work?

I tried again at home and got quite a bit further. Maybe it was just a
hiccup at rubyforge.

However, I still see some questionable things in the output, and I
have yet to actually try it out (this is the first time I'm trying out
buildr). Here in verbatim:

rowe:~$ sudo gem install djspiewak-buildr
Building native extensions. This could take a while...
Successfully installed builder-2.1.2
Successfully installed net-ssh-2.0.4
Successfully installed net-sftp-2.0.1
Successfully installed rubyzip-0.9.1
Successfully installed highline-1.5.0
Successfully installed rubyforge-1.0.1
Successfully installed hoe-1.7.0
Successfully installed rjb-1.1.6
Successfully installed Antwrap-0.7.0
Successfully installed rspec-1.1.4
Successfully installed xml-simple-1.0.11
Successfully installed archive-tar-minitar-0.5.2
Successfully installed djspiewak-buildr-1.3.4
13 gems installed
Installing ri documentation for builder-2.1.2...
ERROR: While generating documentation for builder-2.1.2
... MESSAGE: Unhandled special: Special: type=17, text="<!-- HI -->"
... RDOC args: --ri --op
/opt/local/lib/ruby/gems/1.8/doc/builder-2.1.2/ri --title Builder --
Easy XML Building --main README --line-numbers --quiet lib CHANGES
Rakefile README doc/releases/builder-1.2.4.rdoc
doc/releases/builder-2.0.0.rdoc doc/releases/builder-2.1.1.rdoc
(continuing with the rest of the installation)
Installing ri documentation for net-ssh-2.0.4...
Installing ri documentation for net-sftp-2.0.1...
Installing ri documentation for highline-1.5.0...
Installing ri documentation for rubyforge-1.0.1...
Installing ri documentation for hoe-1.7.0...
Installing ri documentation for Antwrap-0.7.0...
Installing ri documentation for rspec-1.1.4...
Installing ri documentation for archive-tar-minitar-0.5.2...
Installing ri documentation for djspiewak-buildr-1.3.4...
File not found: lib
rowe:~$ ruby --version
ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin8]
rowe:~$


A failure to generate the docs I can live with, but that "File not
found" line looks pretty suspect.

Daniel Spiewak

unread,
Feb 26, 2009, 4:30:00 PM2/26/09
to Clojure
I'm not sure what the File not found thing is all about, but you
should still be ok (crazy gems). Try the following:

buildr --version

Daniel

On Feb 26, 3:16 pm, Christian Vest Hansen <karmazi...@gmail.com>
wrote:

Christian Vest Hansen

unread,
Feb 26, 2009, 4:59:07 PM2/26/09
to clo...@googlegroups.com
rowe:~$ buildr --version
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:36:in
`gem_original_require': no such file to load -- buildr (LoadError)
from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:36:in
`require'
from /opt/local/lib/ruby/gems/1.8/gems/djspiewak-buildr-1.3.4/bin/buildr:18
from /opt/local/bin/buildr:19:in `load'
from /opt/local/bin/buildr:19
rowe:~$ gem --version
1.3.1
rowe:~$

Daniel Spiewak

unread,
Feb 26, 2009, 8:59:08 PM2/26/09
to Clojure
Crud. I suspect this is something weird with the way that the GitHib
gem server works. I'll try to repeat the problem on Ubuntu as soon as
I get back to my computer. In the meantime, you could try these
commands:

sudo gem uninstall djspiewak-buildr
sudo gem install djspiewak-buildr

Daniel

On Feb 26, 3:59 pm, Christian Vest Hansen <karmazi...@gmail.com>
wrote:
> >> ruby1.8.7(2008-08-11 patchlevel 72) [i686-darwin8]

Daniel Spiewak

unread,
Feb 27, 2009, 12:44:06 PM2/27/09
to Clojure
I was able to repeat the problem. I suspect that the issue is the way
in which GitHub is building its gems. I think Assaf (the lead dev for
Buildr) has implemented a workaround in some of the more recent
commits, but I haven't been able to merge with his master so I really
couldn't say.

For the moment, the solution is to just clone the git repository and
build using rake. If you were able to get djspiewak-buildr to install
(as you did), then you should be able to just run the following:

sudo gem uninstall djspiewak-buildr
git clone g...@github.com:djspiewak/buildr.git
cd buildr
rake install

Note that there is no need to run rake under sudo, the install task
will handle that for you.

Daniel

Daniel Spiewak

unread,
Feb 27, 2009, 12:47:41 PM2/27/09
to Clojure
Correction:

git clone git://github.com:djspiewak/buildr.git

Daniel

Christian Vest Hansen

unread,
Feb 27, 2009, 3:39:18 PM2/27/09
to clo...@googlegroups.com
On Fri, Feb 27, 2009 at 6:47 PM, Daniel Spiewak <djsp...@gmail.com> wrote:
>
> Correction:
>
>  git clone git://github.com:djspiewak/buildr.git
git clone git://github.com/djspiewak/buildr.git

Excellent. Got it installed.

Daniel Spiewak

unread,
Feb 27, 2009, 5:21:45 PM2/27/09
to Clojure
As an aside, "gem install djspiewak-buildr" probably works again.
I've merged Assaf's fixes, and GitHub *claims* to be building things
correctly. So, I haven't tried it, but I can't think of a reason why
it wouldn't work.

Daniel

On Feb 27, 2:39 pm, Christian Vest Hansen <karmazi...@gmail.com>
wrote:
> ...
>
> read more »

Daniel Spiewak

unread,
Mar 3, 2009, 12:38:29 PM3/3/09
to Clojure
Warning: if you're using a clone of my Git repository, you will
probably need to reclone from scratch, otherwise things will no longer
work. Because of some changes in the upstream Buildr SVN repository,
everybody on the project has been doing a ton of rebasing lately.
This means that the latest stuff on GitHub is *unrelated* (at the Git
level) to what came before. The repository and commit history is
internally consistent, but it has no shared root with the old. This
means that if you try to do a git pull, you'll run into merge
conflicts out the wazoo. The good news is that this is probably the
last time we will need to do this sort of rebasing, meaning that you
should be able to trust the remote refs from this point forward.

As mentioned previously, the preferred way to install my fork is to
use the following commands:

gem sources -a http://gems.github.com
gem install djspiewak-buildr

This will keep you up to date with the latest stuff in my fork without
the need to clone it yourself or go through the conniptions of
installing all of the build dependencies (which include some other
weird installs like assaf-docter).

Daniel
> ...
>
> read more »

Christian Vest Hansen

unread,
Mar 23, 2009, 7:18:34 PM3/23/09
to clo...@googlegroups.com
Who needs those other languages anyway ;)

>
> Note that this is all predicated on a (possibly faulty) assumption
> about the Clojure compiler: everything must be late-bound.  If
> Clojure's compiler needs to resolve types at compile-time, then this
> could potentially fail in unexpected ways.  Specifically, circular
> dependencies between Clojure and Java/Scala/Groovy would not be
> possible.  In such a case, you would need to somehow factor the
> circularity out into a separate .clj file and then exclude that file
> from compilation (forcing late binding).  I honestly don't know if
> this is the compiler's behavior or not, but I thought it was worth the
> warning.

To bump this one up again....

I only got around to playing with this again recently, and it seems
that I can't access ie. static methods on my Java types. If I
understand correctly, the Clojure code is compiled before the Java
code. I don't think that makes much sense, since Java would call into
Clojure through Vars which are always resolved at runtime, whereas
Clojure would like to know about concrete types when calling into Java
so it can avoid expensive reflection.


>
> One small change: in order to use this functionality now, you will
> need to include the following line at the top of your buildfile:
>
> require 'buildr/clojure'
>
> Joint compilation with Scala/Java or Groovy/Java does not necessitate
> an extra "require", but this first one is necessary to get the process
> started and ensure that Clojure's compiler becomes the dominant
> selection.
>
> Daniel
> >
>



Reply all
Reply to author
Forward
0 new messages