"require" function to find and load source files

[Stuart Sierra]

Hi all,
Just uploaded my idea of a "require" function for Clojure:
http://clojure.googlegroups.com/web/require.clj

(require "foo/bar") will search the Java classpath -- including JAR
files -- for a file named "foo/bar.clj" and load it. It uses the
system ClassLoader to find the files. It remembers which files have
been loaded and won't load the same file twice. Use (require! "foo")
to force a file to be reloaded.

You can require multiple files with a single call: (require "foo"
"baz" "quux")

It's pretty simple; it doesn't assume any relationship between file
names and Clojure namespaces. You still have to use "refer" to import
symbols.

This was partially inspired by the "require" function defined in
Webjure.

-Stuart

[Arto Bendiken]

On Mar 30, 8:05 pm, Stuart Sierra <the.stuart.sie...@gmail.com> wrote:
>
> (require "foo/bar") will search the Java classpath -- including JAR
> files -- for a file named "foo/bar.clj" and load it.  It uses the
> system ClassLoader to find the files.  It remembers which files have
> been loaded and won't load the same file twice.  Use (require! "foo")
> to force a file to be reloaded.

Excellent - I was just thinking of implementing something like this,
you saved me the trouble :-)

One nitpick: I would much prefer to use symbols/keywords over strings
as the arguments to (require). Would it be possible to allow "duck
typing" so that any input is always coerced to the form you store
internally in the *required* list?

What I mean is that currently, it does:

require=> (require "test")
("test")
require=> (require 'test)
(test "test")

...whereas I think it would be preferable to consider both of those to
be equivalent forms and store e.g. only the symbol in the list.

Arto

[christop...@gmail.com]

Wow... It just clicked: thanks to Vars there's no need for tracking
dependencies between requirers and "requirees" to support dynamic
reloading.

Christophe

On Mar 30, 8:05 pm, Stuart Sierra <the.stuart.sie...@gmail.com> wrote:

[Adam Jones]

On the surface it looks like it does this already. In fact anything
that the 'str' function can translate into a string can be used. A
really simple test shows that this works as expected.

-Adam

[Arto Bendiken]

On Mar 30, 9:47 pm, Adam Jones <ajon...@gmail.com> wrote:
>
> On the surface it looks like it does this already. In fact anything
> that the 'str' function can translate into a string can be used. A
> really simple test shows that this works as expected.

It indeed does work with symbols, kind of; the problem demonstrated by
the example in my original reply is that the coercion happens after
the passed-in input has already been stored in the '*required*' var.
This means that 'require!' will get unnecessarily invoked to reload
the name with any other data type than a string.

Should be a minor change to do the coercion earlier. In general, I
think duck typing has lots of potential in Clojure, as long as we come
up with some conventions for the just-in-time type coercions. Rubyists
rejoice :-)

Arto

[Stuart Sierra]

On Mar 30, 4:09 pm, Arto Bendiken <arto.bendi...@gmail.com> wrote:
> It indeed does work with symbols, kind of; the problem demonstrated by
> the example in my original reply is that the coercion happens after
> the passed-in input has already been stored in the '*required*' var.
> This means that 'require!' will get unnecessarily invoked to reload
> the name with any other data type than a string.
>
> Should be a minor change to do the coercion earlier. In general, I
> think duck typing has lots of potential in Clojure, as long as we come
> up with some conventions for the just-in-time type coercions. Rubyists
> rejoice :-)

Fixed, now: symbols get coerced to strings, so (require 'foo) works
correctly:
http://clojure.googlegroups.com/web/require%20%282%29.clj

Google Groups won't let me rename the file to remove the (2) from the
name.

-Stuart

[Stuart Sierra]

On Mar 30, 3:46 pm, "christo...@cgrand.net"

<christophe.gr...@gmail.com> wrote:
> Wow... It just clicked: thanks to Vars there's no need for tracking
> dependencies between requirers and "requirees" to support dynamic
> reloading.

Well, not quite. (require "foo" "bar") will dynamically reload
foo.clj and bar.clj, but not the files that are 'require'd by those
files. But you could force reloading of dependencies like this:

(binding [require/*required* (ref (list))] (require "foo" "bar"))

Maybe "require!" should work that way.

-Stuart

[christop...@gmail.com]

I was thinking of automatic (or manual) reloading when the file is
changed.

If there's two file a.clj and b.clj, with a depending on b.clj, b is
modified you may not need to reload a.clj in many cases. It depends on
whether a.clj use the values of vars of b.clj at compile time. (eg if
there's something like (def fooa (comp foob bara)) (where fooa and
bara are defined in a.clj and foob is defined in b.clj) you'll have to
reload a.clj.)
So I was naive and wrong: vars don't buy you much in terms of
dependencies management.

Christophe

[Stephen C. Gilardi]

On Mar 30, 2008, at 11:28 PM, Stuart Sierra wrote:

Well, not quite.  (require "foo" "bar") will dynamically reload
foo.clj and bar.clj, but not the files that are 'require'd by those
files.  But you could force reloading of dependencies like this:

   (binding [require/*required* (ref (list))] (require "foo" "bar"))

Maybe "require!" should work that way.

I've been working on a "require/provide" system modeled after emacs.  Based on what you wrote in your message (and inspired by the code you posted), here's what I now have for "require!":

(def
 #^{:doc "The set of symbols representing provided packages."
    :private true}
    *packages* (ref #{}))
[...]

(defn require!
  "Like 'require' but will reload packages (and their dependencies)
  that have already been loaded."
  [package & filters]
  (let [global-packages *packages*]

    (binding [*packages* (ref #{})]
      (apply require package filters)

        (dosync
         (commute global-packages set/union @*packages*)))
   nil))

It seems to work.  The idea is that the set of packages after a require! is the union of the global set and the set accumulated during this require!.

I'd welcome critiques.

--Steve

My "require" for reference:

(defn require
  "Loads package if it's not previously provided.  If the
  implementation file called provide and defined a namespace of the
  same name, refer to that namespace with filters."
  [package & filters]
  (when-not (provided? package)
    (load-package package))
  (when (and (provided? package) (find-ns package))
    (apply refer package filters)))

[Rich Hickey]

Did I miss where you define provide and provided?

Rich

[Stephen C. Gilardi]

> Did I miss where you define provide and provided?

I hadn't posted it all. Here's what I have currently. Thanks for
taking a look.

--Steve

; pkg.clj
;
; Clojure source file implementation loading and dependency via
; require/provide. Packages are represented by symbols.
;
; (load-package copied from Stuart Sierra's public domain require.clj)
;
; scgilardi (gmail)
; 30 March 2008
;
; Copyright (c) Stephen C. Gilardi. All rights reserved.
; The use and distribution terms for this software are covered by the
; Common Public License 1.0 (http://opensource.org/licenses/cpl.php)
; which can be found in the file CPL.TXT at the root of the Clojure
; distribution.
; By using this software in any fashion, you are agreeing to be
bound by
; the terms of this license.
; You must not remove this notice, or any other, from this software.

(clojure/in-ns 'pkg)
(clojure/refer 'clojure)

;; Private

(def
#^{:doc "The set of symbols representing provided packages."
:private true}
*packages* (ref #{}))

(defn- load-package
"Load a package implementation file from within CLASSPATH"
[package]
(let [full-name (str package ".clj")
url (. ClassLoader (getSystemResource full-name))]
(when-not url
(throw (new Exception (str "require: Cannot find \"" full-name
"\" in CLASSPATH."))))
(if (= "file" (. url (getProtocol)))
(load-file (. url (getFile)))
(with-open reader (new java.io.BufferedReader
(new java.io.InputStreamReader
(. url (openStream))))
(load reader)))
(println "loaded" package)))

;; Public

(defn packages
"Returns the set of symbols representing provided packages."
[]
@*packages*)

(defn provided?
"Returns true if package has been provided, else false."
[package]
(contains? @*packages* package))

(defn require
"Load packages if they have not already been marked as provided."
[& packages]
(doseq package packages

(when-not (provided? package)
(load-package package))

(when-not (provided? package)
(throw (new Exception
(str "\"" package ".clj\" did not provide "
package))))))

(defn require!
"Like 'require' but will reload packages (and their dependencies)
that have already been loaded."

[& packages]
(dosync
(commute *packages* set/union

(binding [*packages* (ref #{})]

(apply require packages)
@*packages*)))
nil)

(defn require-ns
"Requires a package and then refers to the namespace of the same
name with filters"
[package & filters]
(require package)
(apply refer package filters))

(defn require-ns!
"Like 'require-ns' but will reload packages (and their dependencies)

that have already been loaded."
[package & filters]

(dosync
(commute *packages* set/union

(binding [*packages* (ref #{})]

(apply require-ns package filters)
@*packages*)))
nil)

(defn provide
"Marks a package as provided."
[package]
(dosync
(commute *packages* conj package)
nil))

(provide 'pkg)