Packaging and naming

[Stuart Sierra]

Hi Stephen G.,

I noticed pkg.clj in the clojure-contrib svn. I like it, and I'm
happy that you used my resource-loading code. The one thing I'm not
sure about is (provide ...). I like the idea of being able load
packages from sources other than files (e.g. direct from svn/git/
http), but does every package need (provide ...)?

I can think of situations where a specialized variant of (require
'foo) would not actually be loading a file named foo.clj, but I can't
think of a situation in which (require 'foo) succeeds but 'foo is not
added to *packages*.

It's also possible that someone will mistakenly write (provide ...) at
the TOP of a source file, so that even if loading the entire file
fails, it still ends up on *packages* (Emacs undoes (provide ...) in
this case).

So my thought is: if (require 'foo) succeeds it should automatically
(provide 'foo). That way package writers don't have to do anything
special other than use the same names for files and namespaces. A
package may still (provide ..) additional packages if it wants to.

Let me know what you think.

-Stuart

[Rich Hickey]

I thought the same thing - I don't really understand provide.

Rich

[Stephen C. Gilardi]

I noticed pkg.clj in the clojure-contrib svn.  I like it, and I'm
happy that you used my resource-loading code.

Thanks very much!  An earlier version introduced a "Clojure Path" concept, but I think using CLASSPATH is a much better solution: more useful, more flexible, and more in keeping with embracing the JVM.

The one thing I'm not
sure about is (provide ...).  I like the idea of being able load
packages from sources other than files (e.g. direct from svn/git/
http), but does every package need (provide ...)?

I can think of situations where a specialized variant of (require
'foo) would not actually be loading a file named foo.clj, but I can't
think of a situation in which (require 'foo) succeeds but 'foo is not
added to *packages*.

I had been thinking of it as a kind of handshake so one could be sure that it was the correct file that was loaded.  However, any benefit from avoiding "unanticipated files in CLASSPATH" confusion is swamped by the utility of being able to load arbitrary source files using "require" and know you'll only be loading them once.

It's also possible that someone will mistakenly write (provide ...) at
the TOP of a source file, so that even if loading the entire file
fails, it still ends up on *packages* (Emacs undoes (provide ...) in
this case).

So my thought is: if (require 'foo) succeeds it should automatically
(provide 'foo).  That way package writers don't have to do anything
special other than use the same names for files and namespaces.  A
package may still (provide ..) additional packages if it wants to.

Let me know what you think.

I think it's a fine idea and I've made that change to pkg.clj.  Thanks very much for your thoughts and explanation.

--Steve

[Stephen C. Gilardi]

I thought the same thing - I don't really understand provide.

Rich

I made the change Stuart suggested, so "provide" is no longer required in an implementation file.  I think it's a big improvement.

I do think 'provide' (as a separate function) is still useful.

"require" says "I need this capability, ensure it's present"

"provide" says "Mark this capability as present"

By default a capability comes to be present when its implementation file (whose name is derived from the package name) is loaded.

Because 'provide' is available separately, it's possible to override that default.

One use of that is to use customized versions of packages without changing CLASSPATH to shadow the original implementation file with a different file of the same name (which can be confusing).

For example:

Package A requires package B.

My code requires package A and I'm working on a custom version of package B called "MyB".

My code can look like this:

(require 'MyB)

(provide 'B)

(require 'A)

The call to "(provide 'B)" could either be inline as shown or could be inside "MyB.clj".  It asserts that A's requirement for B is satisfied and suppresses loading "B.clj".

A separately available provide also allows the implementation of a package to be done inline rather than in its own implementation file; one could define several packages in one file.  One could do something like the following in a "clojure init" file:

(defn sin ...)

(defn cos ...)

(defn tan ...)

[...]

(provide 'scg-trig)

(defn even? ...)

(defn odd? ...)

[...]

(provide 'scg-pred)

It's also possible that someone will mistakenly write (provide ...) at

the TOP of a source file, so that even if loading the entire file

fails, it still ends up on *packages* (Emacs undoes (provide ...) in

this case).

Rich, I think I read that Emacs also makes the guarantee that unless a file loads completely, any definitions that were made while loading it are rolled back.  A transactional load-file seems kinda cool.  I know "kinda cool" isn't a great reason to add something, but I wondered if the data structures used in Clojure's loading mechanism would support this kind of thing easily or not.

--Steve

[Rich Hickey]

On Apr 3, 12:37 pm, "Stephen C. Gilardi" <scgila...@gmail.com> wrote:
> > I thought the same thing - I don't really understand provide.
>
> > Rich
>
> I made the change Stuart suggested, so "provide" is no longer required
> in an implementation file. I think it's a big improvement.
>
> I do think 'provide' (as a separate function) is still useful.
>
> "require" says "I need this capability, ensure it's present"
> "provide" says "Mark this capability as present"
>
> By default a capability comes to be present when its implementation
> file (whose name is derived from the package name) is loaded.
>
> Because 'provide' is available separately, it's possible to override
> that default.

> ...

Ok, thanks, that makes sense.

> Rich, I think I read that Emacs also makes the guarantee that unless a
> file loads completely, any definitions that were made while loading it
> are rolled back. A transactional load-file seems kinda cool. I know
> "kinda cool" isn't a great reason to add something, but I wondered if
> the data structures used in Clojure's loading mechanism would support
> this kind of thing easily or not.
>

Heh - There was a point when Vars and Refs were unified, e.g. refs
could be thread-bound and were used to store what are now var root
values, and one of the ideas then was to have rollback on failed
compilation.

It ends up that conflating the two is too confusing and has perf
issues, so I split them up, and haven't revisited the rollback
scenario since. I do think it is interesting, cool even, but not a
priority.

Rich

[Stuart Sierra]

On Apr 3, 11:51 am, "Stephen C. Gilardi" <scgila...@gmail.com> wrote:
> > I noticed pkg.clj in the clojure-contrib svn. I like it, and I'm
> > happy that you used my resource-loading code.
>
> Thanks very much! An earlier version introduced a "Clojure Path"
> concept, but I think using CLASSPATH is a much better solution: more
> useful, more flexible, and more in keeping with embracing the JVM.

I agree; I did the same thing in require.clj before deciding to use
the ClassLoader. It may also open up possibilities for using
'require' with exotic custom ClassLoaders.

> > So my thought is: if (require 'foo) succeeds it should automatically
> > (provide 'foo). That way package writers don't have to do anything
> > special other than use the same names for files and namespaces. A
> > package may still (provide ..) additional packages if it wants to.
>
> > Let me know what you think.
>
> I think it's a fine idea and I've made that change to pkg.clj. Thanks
> very much for your thoughts and explanation.

Thanks, Steve, glad we seem to be thinking alike here. So what do you
say we move on to consider the next challenge, which, I think, is
hierarchical package names. That is, if I want to load a file at foo/
bar/baz.clj, what's the package name?

(require 'foo.bar.baz) mimics Java, which might be good or might be
confusing. According to the docs[1], symbols containing '.' represent
qualified Java class names. Rich, you discouraged the use of '.' in
namespaces[2]; do you have an alternative in mind?

foo:bar:baz and foo/bar/baz don't parse as symbols. foo-bar-baz and
foo_bar_baz are ambiguous w.r.t. file names. Allowable characters in
symbols[1] give us foo!bar!baz, foo*bar*baz, foo+bar+baz, and foo?bar?
baz, none of which I'm very happy with.

Another possibility is (require '(foo bar baz)), but I don't care for
that since it looks like you're requiring 3 separate packages.

In require.clj I punted on the issue (and imitated Ruby) by using
strings as package names, but I think using symbols is more
appropriate to Clojure.

Thoughts, all?

-Stuart

[1]: http://clojure.sourceforge.net/reference/reader.html
[2]: http://groups.google.com/group/clojure/msg/44d65d39b8b3a471

[Rich Hickey]

Are you sure you want to create a fixed relationship between a
multipart name and filesystem structure? I'm not sure that's one of
the more popular parts of Java.

And do you want the entire hierarchy in the namespace name?

Does the hierarchy have to support n levels?

You can utilize the ns part of the symbols for a different purpose in
require, since the symbols are unevaluated:

(require 'whizbang/wb-utils)
(require 'whizbang/wb-sql)

could mean wb-utils and wb-sql libs/namespaces from the whizbang
directory.

I'm not sure it needs to get more complex than that. Also require
could ensure that no 2 requires for the same lib had different dirs,
and thus could be used for versions:

whizbangV2/wb-utils
whizbangV2/wb-sql

Just riffing here, not a real plan...

N.B. We shouldn't be using the term "package" for Clojure stuff, I
avoided using it so we could always distinguish Java packages from
Clojure namespaces.

Rich

[Stuart Sierra]

On Apr 3, 5:27 pm, Rich Hickey <richhic...@gmail.com> wrote:
> Are you sure you want to create a fixed relationship between a
> multipart name and filesystem structure? I'm not sure that's one of
> the more popular parts of Java.

No, I'm not sure. It's just the most common convention I know: Java,
Perl, Python. Ruby goes even farther: package names are just file
paths with the extension chopped off.

> And do you want the entire hierarchy in the namespace name?

Not necessarily, but I would like to be able to distinguish between,
say, text/utils/sql and text/utils/html.

> Does the hierarchy have to support n levels?

Good question, I don't know. All I know is most package systems do.
I sometimes organize code more than 2 directories deep, but I could
probably get by without it.

> You can utilize the ns part of the symbols for a different purpose in
> require, since the symbols are unevaluated:
>
> (require 'whizbang/wb-utils)
> (require 'whizbang/wb-sql)
>
> could mean wb-utils and wb-sql libs/namespaces from the whizbang
> directory.

I think I would find that really confusing, because it looks like
whizbang is a namespace, when in fact wb-utils and wb-sql are
namespaces.

> I'm not sure it needs to get more complex than that. Also require
> could ensure that no 2 requires for the same lib had different dirs,
> and thus could be used for versions:
>
> whizbangV2/wb-utils
> whizbangV2/wb-sql
>
> Just riffing here, not a real plan...

To continue riffing, here's one more idea:
(require-ns 'baz :in "foo/bar")
loads the file foo/bar/baz.clj and refers the 'baz namespace.

Also permits
(require-ns 'baz :in "/usr/local/lib/my-stuff")
for files that are not on the classpath.

> N.B. We shouldn't be using the term "package" for Clojure stuff, I
> avoided using it so we could always distinguish Java packages from
> Clojure namespaces.

Got it, that makes sense. Maybe we should just call them "libraries,"
avoiding the loaded term "module."

-Stuart

[Stephen C. Gilardi]

I'm not sure it needs to get more complex than that. Also require

could ensure that no 2 requires for the same lib had different dirs,

and thus could be used for versions:

whizbangV2/wb-utils

whizbangV2/wb-sql

Just riffing here, not a real plan...

To continue riffing, here's one more idea:
(require-ns 'baz :in "foo/bar")
loads the file foo/bar/baz.clj and refers the 'baz namespace.

Also permits
(require-ns 'baz :in "/usr/local/lib/my-stuff")
for files that are not on the classpath.

I like this.  The argument for ":in" might have rules like these:

 - Absolute paths are considered to be in the filesystem

 - relative paths search classpath (it looks like getSystemResource supports that by default)

 - (maybe) if a relative path isn't found in classpath, also search with "current working directory" as the root

The syntax could also grow in the future to support URLs of types other than the implicit "file://" (or "classpath://")  That makes a string a good choice for the location qualifier while leaving a symbol as the namespace and implementation filename specifier.

Allowing ":in" gives flexibility for any given namespace implementation to either use an ":in" argument or extend classpath so that it is unnecessary.

N.B. We shouldn't be using the term "package" for Clojure stuff, I

avoided using it so we could always distinguish Java packages from

Clojure namespaces.

Got it, that makes sense.  Maybe we should just call them "libraries,"
avoiding the loaded term "module."

OK, here's some more riffing...

It's unfortunate that module is a loaded term as "mojule" seems like it would be a natural--compare pronunciations of "Clojure" vs. "closure" and "mojule" vs. "module".  A namespace mechanism does give some of the good properties that people associate with "modules".  Another seems to be separate compilation which is either missing or moot in Clojure depending on how you look at it.  Could we reasonably call these "mojules"?

I started thinking about alternatives.  The "require/provide" stuff has lots of precedent in other lisps as tracking "features".  We could use that.  It has the benefit of being very generic and familiar to many.

Alternatively, what do "closures" do--capture environments.  Thinking about "capture", I considered "capjure", but seems a little awkward.. but... "capsule"... that seems to have some potential.  An implementation of a namespace... a capsule of capability.  I like the "cap" notion there.

Python and Chicken Scheme have eggs

Snow has snowballs

Ruby has gems

What for Clojure... "libraries", "mojules", "features", "capjures", "jems", "capsules", "capjules", "joules", "jewels"?

Of those, my favorite is "jewels", it's relatively short, conveys a sense of value, has precedent in engineering and programming as "a useful thing", is a nod to "gems", is a nod to "mojule", and would be something that could be easily associated with Clojure because it has the "j" thing going for it.

And, like Clojure, jewels are beautiful.

I look forward to changing my "pkg.clj" to stop using "package" and start using a new term when we come up with one that folks like and that Rich blesses.

Any ideas?

--Steve

[Stuart Sierra]

On Apr 4, 3:17 pm, "Stephen C. Gilardi" <scgila...@gmail.com> wrote:
> > To continue riffing, here's one more idea:
> > (require-ns 'baz :in "foo/bar")
> > loads the file foo/bar/baz.clj and refers the 'baz namespace.
> > Also permits
> > (require-ns 'baz :in "/usr/local/lib/my-stuff")
> > for files that are not on the classpath.
>
> I like this. The argument for ":in" might have rules like these:
>
> - Absolute paths are considered to be in the filesystem
> - relative paths search classpath (it looks like getSystemResource
> supports that by default)
> - (maybe) if a relative path isn't found in classpath, also search
> with "current working directory" as the root
>
> The syntax could also grow in the future to support URLs of types
> other than the implicit "file://" (or "classpath://") That makes a
> string a good choice for the location qualifier while leaving a symbol
> as the namespace and implementation filename specifier.

Yes! Now that I think about it, maybe :from would be a better name.
This would be cool someday:
(require 'best-library :from "svn://example.com/my-libraries/trunk")
with appropriate caching, etc.

If a namespace names is always a simple, unadorned symbol, it makes
sense that (require...) forms should use the same unadorned symbol.

> Python and Chicken Scheme have eggs
> Snow has snowballs
> Ruby has gems
> What for Clojure... "libraries", "mojules", "features", "capjures",
> "jems", "capsules", "capjules", "joules", "jewels"?

How about "cloj"? Rhymes with "kludge." Just kidding.

Anything short is good. "jewels" is ok, it's easy to remember.
mojos, jacks, joints, junks, jolts? This is fun. "lattes" (Lispy
Java)

-Stuart

[jon]

On the subject of libraries etc, wouldn't it be more robust if it
everybody always stuck to the convention of fully qualifying
(in-ns 'wibble) as (clojure/in-ns 'wibble) instead?
It would avoid this possibility:

user=> (in-ns 'wibble)
#<Namespace: wibble>
wibble=> (clojure/load-file "/home/user/clojure/src/inspector.clj")
java.lang.Exception: Unable to resolve symbol: in-ns in this context
clojure.lang.Compiler$CompilerException: inspector.clj:9: Unable to
resolve symbol: in-ns in this context
at clojure.lang.Compiler.analyzeSeq(Compiler.java:3366)
at clojure.lang.Compiler.analyze(Compiler.java:3298)
at clojure.lang.Compiler.analyze(Compiler.java:3273)
.... etc
oops!

Stuart and Stephen G. seem to be doing it that way.. but generally
other code doesn't (eg. in files attached to the group, clojure-
contrib and clojure.jar)
Jon

[Christophe Grand]

On Apr 5, 12:07 am, jon <superuser...@googlemail.com> wrote:
> On the subject of libraries etc, wouldn't it be more robust if it
> everybody always stuck to the convention of fully qualifying
> (in-ns 'wibble) as (clojure/in-ns 'wibble) instead?

You can also redefine (or locally bind) load-file to locally-bind *ns*
so that *ns* can't leak.
(let [leaking-ns-load-file load-file]
(defn load-file[s] (binding [*ns* *ns*] (leaking-ns-load-file s))))

Christophe

[Stephen C. Gilardi]

On Apr 4, 2008, at 4:55 PM, Stuart Sierra wrote:

Yes!  Now that I think about it, maybe :from would be a better name.

This would be cool someday:

(require 'best-library :from "svn://example.com/my-libraries/trunk")

with appropriate caching, etc.

If a namespace names is always a simple, unadorned symbol, it makes

sense that (require...) forms should use the same unadorned symbol.

I've added "loader.clj" to clojure-contrib.  We can use "namespace" as our unit of loadable capability.  "loader.clj" provides namespace "loader" which includes these two functions:

user=> (doc ensure-ns)

-------------------------

loader/ensure-ns

([ns-sym & options])

  Ensures that a namespace is loaded. If it is not yet loaded, searches

  for an implementation file whose name is the namespace name followed by

  '.clj' and loads it. If no :from option is present, the search considers

  only the classpath roots. Options may include one each of:

  :from string

  :reload boolean

  :reload-all boolean

  :verbose boolean

  An argument to :from specifies a path to the implementation file's parent

  directory. An absolute path (beginning with '/') specifies a directory in

  the filesystem. A relative path specifies directories relative to each of

  the classpath roots.

  When :reload is true, the namespace will be reloaded if already loaded.

  When :reload-all is true, all directly and indirectly required namespaces

  are also reloaded.

  When :verbose is true, a 'loaded' message is printed after each load.

nil

user=> (doc require)

-------------------------

loader/require

([ns-sym & options])

  Ensures that a namespace is loaded and then refers to it.  Options may

  include options for ensure-ns and/or filters for refer.

nil

user=> 

Comments are welcome.

--Steve

[Stephen C. Gilardi]

Reposting to make descriptions readable on the website (word wrap):

[Stuart Sierra]

On Apr 5, 3:48 pm, Stephen C. Gilardi <scgila...@gmail.com> wrote:
> I've added "loader.clj" to clojure-contrib. We can use "namespace"
> as our unit of loadable capability. "loader.clj" provides namespace
> "loader" which includes these two functions:

> loader/ensure-ns
> loader/require

Just want to clarify -- is this intended to replace "pkg.clj" and
(provide ...)?

> An argument to :from specifies a path to the implementation file's
> parent directory. An absolute path (beginning with '/') specifies a
> directory in the filesystem. A relative path specifies directories
> relative to each of the classpath roots.

Our non-Unix users (anyone?) might prefer File.isAbsolute() and
File.pathSeparatorChar.

> When :reload is true, the namespace will be reloaded if already
> loaded.
> When :reload-all is true, all directly and indirectly required
> namespaces are also reloaded.

This is a useful distinction.

I'll look some more at this later, now I need to sleep. :)
-Stuart

[Stephen C. Gilardi]

On Apr 5, 2008, at 11:51 PM, Stuart Sierra wrote:
>
> On Apr 5, 3:48 pm, Stephen C. Gilardi <scgila...@gmail.com> wrote:
>> I've added "loader.clj" to clojure-contrib. We can use "namespace"
>> as our unit of loadable capability. "loader.clj" provides namespace
>> "loader" which includes these two functions:
>> loader/ensure-ns
>> loader/require
>
> Just want to clarify -- is this intended to replace "pkg.clj" and
> (provide ...)?

They're alternative solutions to a problem. Right now they exist side
by side. Pkg violates the "avoid the word package when working with
Clojure" guideline from Rich so I'll likely withdraw it soon.

I've spent some time with "loader/require" today and I like it.

>> An argument to :from specifies a path to the implementation file's
>> parent directory. An absolute path (beginning with '/') specifies a
>> directory in the filesystem. A relative path specifies directories
>> relative to each of the classpath roots.
>
> Our non-Unix users (anyone?) might prefer File.isAbsolute() and
> File.pathSeparatorChar.

Good point, I had seen that "/" is a platform independent separator
for paths used by getSystemResource, but here I'm checking for an
absolute pathname. Is it even worth supporting an absolute ":from"
when "load-file" is available for that?

>> When :reload is true, the namespace will be reloaded if already
>> loaded.
>> When :reload-all is true, all directly and indirectly required
>> namespaces are also reloaded.
>
> This is a useful distinction.
>
> I'll look some more at this later, now I need to sleep. :)

Great! Thanks as always for your feedback and help.

--Steve