File organization & bootstrapping

[Greg Harman]

Hi all,

I'm struggling a bit with how best to organize and bootstrap a
multiple-source file project so that it can be run either command-line
(java -cp clojure.jar:myapp.jar ...) or by running a load method from
the REPL without having to explicitly change the classpath in my
user.clj before running.

Currently my file structure looks like this:

/bootstrap.clj
/package1/foo.clj
/package2/bar.clj

And bootstrap.clj looks something like:

(def *namespaces*
'("package1/foo" "package2/bar"))

(defn load []
(for [namespace *namespaces*]
(load namespace)))

This seems to work sometimes, if my REPL's "working directory" is /
(relative to my project), but not if it's anything else (get a
java.io.FileNotFoundException).

So, I guess my question is: is this whole approach silly and is there
a better way to bootstrap a multiple file/package project? If not, how
best to tweak this?

-Greg

[Greg Harman]

In partial answer to my own question, I found these threads helpful:

http://groups.google.com/group/clojure/browse_thread/thread/bf9672989524a1bf/1f35a50643bd6325?lnk=raot
http://groups.google.com/group/clojure/msg/58e3f8e5dfb876c9

Everything seems to work well, except compilation. I'll post any
modifications when I solve this issue (which I may start in another
thread if I remain stuck).

[Stuart Sierra]

Hi Greg,
I think you may be working at too low a level with "load". You should
be able to accomplish the same thing with:

(ns my-app
(:require package1 package2))

(package2/start-app)

my_app, package1, and package2 all need to be immediately under a
directory (or JAR file) on the Java classpath. Also check out the Ant
build.xml files from Clojure core and clojure-contrib.

-Stuart Sierra

[Greg Harman]

Thanks Stuart,

That's a good simplification, and everything seem kosher in terms of
loading from the REPL. I would like to structure my libs (package1,
package2 in the example) such that they span multiple files. So the
main file for the namespace/lib still needs to have "load" operations
to get the other files for that namespace (clojure/core.clj does this,
for example).

It works for "require" and for calling functions in the package(s),
but the problem now is that it doesn't work for AOT from the REPL.

(compile 'package1) crashes with:

java.io.IOException: No such file or directory (package.clj:4)

Where line 4 is the first "load" operation (and the load works fine if
I execute it directly). If I remove the load operation altogether as a
test (making a single-file package), now I get:

java.io.IOException: No such file or directory (NO_SOURCE_FILE:0)

As far as I can tell, this is now identical to the example that Rich
gave in one of the two threads that I referenced in my earlier post.
I'm not really sure how go about tracing "NO_SOURCE_FILE:0"...

-Greg

[Greg Harman]

Nevermind, with a fresh start today (and perhaps more importantly,
perhaps, a fresh environment) compiling seems to work fine.

[Phil Hagelberg]

> I think you may be working at too low a level with "load". You should
> be able to accomplish the same thing with:
>
> (ns my-app
> (:require package1 package2))
>
> (package2/start-app)
>
> my_app, package1, and package2 all need to be immediately under a
> directory (or JAR file) on the Java classpath. Also check out the Ant
> build.xml files from Clojure core and clojure-contrib.

The problem with this is that now you have to repeat your classpath in
two places: your SLIME config and the shell script that you use to
launch your app. This is a DRY[1] violation.

One solution would be to load a single file and then have that file use
add-classpath to set the classpath, but add-classpath is unreliable;
I've had problems getting it to work consistently and have been told in
#clojure that I shouldn't be using it.

My solution for the time being is to always launch via SLIME and
consider that method of launching it canonical, essentially deprecating
the method of launching it from the shell. But obviously this is not
ideal because for some reason not everyone uses Emacs. =) Clearly the
classpath belongs in the codebase so it can be used independently of how
the application was launched.

There's got to be a way around this problem, but I haven't found it yet.

-Phil

[1] - http://en.wikipedia.org/wiki/Don%27t_repeat_yourself

[Greg Harman]

This is frustrating - with a fresh REPL I'm back to the compile
problem. I can't think of anything I changed that would cause it to
work intermittently, and I don't know what file it's looking for...
the source files are on the classpath (it "require"s just fine) and
the directory in *compile-path* is also on the cp (via add-
classpath).

It's not the "load" operation that caused it - I removed that entirely
and just put a placeholder (defn foo [] true), and the compilation
still crashes on this line.

[Greg Harman]

> One solution would be to load a single file and then have that file use
> add-classpath to set the classpath, but add-classpath is unreliable;
> I've had problems getting it to work consistently and have been told in
> #clojure that I shouldn't be using it.

Possibly the cause of the compile problem I had in my last message(s)?
I'll dig around and see if I can find that thread & understand what
the issue with add-classpath is.

> My solution for the time being is to always launch via SLIME and
> consider that method of launching it canonical, essentially deprecating
> the method of launching it from the shell. But obviously this is not
> ideal because for some reason not everyone uses Emacs. =) Clearly the
> classpath belongs in the codebase so it can be used independently of how
> the application was launched.

Beyond not using emacs, if you want something that's deployable in a
server environment and not just on a development box it has to run
with no human intervention ala emacs bootstrapping. I realize that
emacs could be configured to automatically launch the clojure app, but
that seems too hokey for a real production deployment.

-Greg

[Shawn Hoover]

On Wed, Jan 7, 2009 at 12:34 PM, Greg Harman <gha...@gmail.com> wrote:

This is frustrating - with a fresh REPL I'm back to the compile
problem. I can't think of anything I changed that would cause it to
work intermittently, and I don't know what file it's looking for...
the source files are on the classpath (it "require"s just fine) and
the directory in *compile-path* is also on the cp (via add-
classpath).

It's not the "load" operation that caused it - I removed that entirely
and just put a placeholder (defn foo [] true), and the compilation
still crashes on this line.

Are your files still called package1/foo.clj and package2/bar.clj? If so you would need to (compile 'package1.foo) and (compile 'package2.bar).

Also, make sure the directory named by *compile-path* exists on the file system. Compilation creates subdirs, but not *compile-path* itself.

Shawn

[Greg Harman]

Bingo - *compile-path* was a relative dir. Defining it as the full
path did the trick.

Thanks!

[Shawn Hoover]

You're welcome!

Clojure developers, would it be a good idea for Compiler.java or clojure.core/compile to check if *compile-path* exists, else throw an exception specific to that problem?

Shawn

[Stephen C. Gilardi]

On Jan 7, 2009, at 2:04 PM, Shawn Hoover wrote:

> Clojure developers, would it be a good idea for Compiler.java or
> clojure.core/compile to check if *compile-path* exists, else throw
> an exception specific to that problem?

I like the idea. There is a subtle problem I don't think it will catch
though. The JRE appears to scan CLASSPATH only once at startup
checking whether each item in it is actually a viable jar file or
directory. If a path appears in CLASSPATH, but does not exist at JRE
startup, but does exist at compile time, compilation will still fail
and it will still be very hard to figure out why.

Perhaps this is a case for speculation in the error message:

- Exception: Compilation target directory doesn't exist
- Exception: Compilation target directory exists, but load-after-
compile failed, did it exist when the JVM was launched?

(assuming the latter is detectable)

--Steve

[Craig McDaniel]

Getting back to Phil Hagelberg's comment that maintaining a project's
classpath in both a SLIME config and shell script for each project/
application is a "Don't Repeat Yourself" violation, there is a way to
avoid that:

Don't bother with swank-clojure-extra-classpaths. Instead, include /
path/to/swank-clojure in the classpath of each of your project's
startup shell scripts. Each project starts up a swank server using
something like the following in its initialization code, using a
unique port for each project:

(require 'swank.swank)
(swank.swank/ignore-protocol-version "2009-01-01")
(swank.swank/start-server "/dev/null" :port 7777 :encoding "iso-
latin-1-unix")

Connect to your running app from emacs by using "M-x slime-connect"
rather than "M-x slime". When you're done, use "M-x slime-
disconnect" (or from the REPL, "," then "disconnect") to leave your
process running. In addtion, including swank in your running
production app is a real benefit since you can connect to it from
emacs at any later time for diagnosis, debugging, etc...

-Craig

[Phil Hagelberg]

Craig McDaniel <crai...@gmail.com> writes:

> Connect to your running app from emacs by using "M-x slime-connect"
> rather than "M-x slime". When you're done, use "M-x slime-
> disconnect" (or from the REPL, "," then "disconnect") to leave your
> process running. In addtion, including swank in your running
> production app is a real benefit since you can connect to it from
> emacs at any later time for diagnosis, debugging, etc...

That's good to know; thanks.

I'm still a little uneasy about introducing swank as an external
dependency. It's not a big deal for stuff you'd use internally, but for
open-source projects I like to keep the installation process to a couple
lines. I guess I could bundle a copy of swank-clojure to streamline
things.

Or I could wait until I actually have something to release before I
obsess about making installation and distribution easy. =)

-Phil

[Stuart Sierra]

On Jan 7, 12:19 am, "Phil Hagelberg" <p...@hagelb.org> wrote:
> The problem with this is that now you have to repeat your classpath in
> two places: your SLIME config and the shell script that you use to
> launch your app. This is a DRY[1] violation.

I guess I just take that for granted with Java. The classpath is a
nuisance, but it works. If you want to launch your app at the command
line, use Ant to build a JAR that includes Clojure and all the
necessary libraries.

-Stuart Sierra