Question about namespaces

[Mark Engelberg]

I've seen people say here that it's relatively easy to break up a
namespace into smaller components, so I'm wondering if I'm missing
something. I'd appreciate some guidance about how to keep namespaces
well-organized.

In Python, let's say I have a library "mylibrary.py", and from various
files I say "from mylibrary import *".
If I want to split mylibrary.py into two parts, say, library1.py and
library2.py, then I can just change mylibrary.py to say:
from library1 import *
from library2 import *

All the functions from library1 and library2 become available to
mylibrary, which in turn becomes available to all files that import
from mylibrary. In other words, nothing breaks by splitting into two
files.

Now in Clojure, let's say I have a file mylibrary.clj and from various
files I say:
(ns a-random-file-that-consumes-my-library
(:use mylibrary))

So now, if I want to split mylibrary.clj into library1.clj and
library2.clj, I change mylibrary.clj to say:
(ns mylibrary
(:use library1 library2))

Unfortunately, this seems to break my consumer code. Although
mylibrary can see the functions from library1 and library2, consumers
of mylibrary cannot. So I end up manually having to go to all my
consumer files and changing them to:
(ns a-random-file-that-consumes-my-library
(:use library1)
(:use library2))

Blech. This makes it very difficult to refactor. What am I missing?

On a related note, another thing I'm missing from Python is that in
mylibrary.py, when I say "from library1 import *", Python
automatically knows to look in the same directory. This keeps the
import declarations nice and short for a group of related files, and
more importantly, they don't break if I rename the directory they are
in or move them around.

On the other hand, I'm finding in Clojure that I need to explicitly
name the directory as part of the namespace, and I'm finding this to
be a nuisance. Is there any way to make this a bit more pleasant?

Thanks.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Laurent PETIT]

Maybe this could help:

http://richhickey.github.com/clojure-contrib/ns-utils-api.html#clojure.contrib.ns-utils/immigrate

HTH,

--
Laurent

2010/5/9 Mark Engelberg <mark.en...@gmail.com>:

[David Nolen]

If you just need to break up your code into smaller files another technique is:

; me/lib.clj

(ns me.lib)

(load "me/foo")

(load "me/bar")

; me/foo.clj

(in-ns 'me.lib)

; me/bar.clj

(in-ns 'me.lib)

[Stuart Sierra]

> http://www.google.com/url?sa=D&q=http://richhickey.github.com/clojure-contrib/ns-utils-api.html%23clojure.contrib.ns-utils/immigrate

'immigrate' is bad, it creates new Vars instead of new mappings to
existing Vars, leading to very subtle and difficult-to-fix bugs.

-S

[Laurent PETIT]

2010/5/9 Stuart Sierra <the.stua...@gmail.com>:

>> http://www.google.com/url?sa=D&q=http://richhickey.github.com/clojure-contrib/ns-utils-api.html%23clojure.contrib.ns-utils/immigrate
>
> 'immigrate' is bad, it creates new Vars instead of new mappings to
> existing Vars, leading to very subtle and difficult-to-fix bugs.

Oh, indeed. Let's deprecate it then ?

[Mark Engelberg]

On Sun, May 9, 2010 at 8:01 AM, David Nolen <dnolen...@gmail.com> wrote:

If you just need to break up your code into smaller files another technique is:

; me/lib.clj

(ns me.lib)

(load "me/foo")

(load "me/bar")

; me/foo.clj

(in-ns 'me.lib)

; me/bar.clj

(in-ns 'me.lib)

I think this solves part of my problem.

But as another example, consider that I've developed a library of set functions to augment what comes with clojure.

; me/setlib.clj
(ns me.setlib
  (:use clojure.set clojure.contrib.set))

Right now, if I want to use my own augmented set library, I have to also remember to use clojure.set and clojure.contrib.set to get the "full set library".

; me/setconsumer.clj
(ns me.setconsumer
  (:use clojure.set clojure.contrib.set me.setlib))

What can I do in setlib.clj so that in my consumer file I can just say:
(ns me.setconsumer
  (:use me.setlib))
and get all the set functions?

In PLT Scheme, the module system allows you to specify both what you "require" from other libraries and what you want to "provide" to other libraries.  There is an easy mechanism to say you want to provide (export) all the things you've required (imported) from another module.  Is there any way to do that in Clojure -- to essentially say, "I'm using things from this other namespace, and I want to turn around and make them available as if they were part of my own namespace."? 

It's also worth noting that PLT Scheme's module system used to require you to begin each file with a (module name) command where the name needed to match the file you were working in, much like the way Clojure currently does things.  They eventually reworked things so that this became unnecessary -- the old syntax is still available if you need to specify a module directly at the REPL, or for some reason set up a module that differs from the filename, but in general, the module name is now automatically determined from the filename.  It's so much easier this way.

So I find myself wondering why Clojure's namespace/file system is the way it is.  I'm assuming that some of it has to do with Java cultural and/or classpath baggage that I don't really know much about it.  Possibly some of it has to do with limitations of how this code will be interactively loaded in bits and pieces by emacs and other IDEs.  But it still seems like it would be possible to make it so each Clojure file would be automatically loaded in it's own namespace, and that require statements could find other Clojure files relative to their own location.  So for example:
(ns me.setconsumer (:use me.setlib)) would just become
(ns (:use setlib))

Is there a benefit to the current system that I'm not fully appreciating?

Thanks.

References:

PLT Module information: http://docs.plt-scheme.org/guide/modules.html

(PLT Scheme also has another concept called units, which supposedly can handle even more sophisticated types of polymorphic/dynamic linkages, but I've never fully appreciated what extra benefits you gain from the extra complexity.  Modules have been sufficient for my purposes.  But for those who are interested: http://docs.plt-scheme.org/guide/units.html)

[Armando Blancas]

> Unfortunately, this seems to break my consumer code.  Although
> mylibrary can see the functions from library1 and library2, consumers
> of mylibrary cannot.  So I end up manually having to go to all my
> consumer files and changing them to:
> (ns a-random-file-that-consumes-my-library
>   (:use library1)
>   (:use library2))
>
> Blech.  This makes it very difficult to refactor.  What am I missing?

Clojure doesn't have a sys.modules dictionary. Use does a refer, which
maps names to the current namespace, not to a global map. This will
give your random-file the ability to control what's in its namespace.
Considering that mylibrary.clj is a now a good candidate for
refactoring, seems like some more work is in order in any case.

[James Reeves]

On 9 May 2010 19:02, Laurent PETIT <lauren...@gmail.com> wrote:
> 2010/5/9 Stuart Sierra <the.stua...@gmail.com>:
>>> http://www.google.com/url?sa=D&q=http://richhickey.github.com/clojure-contrib/ns-utils-api.html%23clojure.contrib.ns-utils/immigrate
>>
>> 'immigrate' is bad, it creates new Vars instead of new mappings to
>> existing Vars, leading to very subtle and difficult-to-fix bugs.
>
> Oh, indeed. Let's deprecate it then ?

It would be nice if there were a proper "immigrate" function. There
are edge cases where it's a useful function.

- James

[Mark Engelberg]

The more I think about this, the more I feel like I don't have a good mental model of what's going on with namespaces/loading/reading/compiling.  What is happening when you "load" a file without reading/compiling it (e.g., load-reader or load-file)?  How does referring/using match up with those low-level concepts?

I'd really appreciate a clear explanation of how these mechanisms fit together.

[Meikel Brandmeyer]

Hi,

On 10 Mai, 00:15, Mark Engelberg <mark.engelb...@gmail.com> wrote:

> The more I think about this, the more I feel like I don't have a good mental
> model of what's going on with namespaces/loading/reading/compiling.  What is
> happening when you "load" a file without reading/compiling it (e.g.,
> load-reader or load-file)?  How does referring/using match up with those
> low-level concepts?

Clojure's view of the world is actually quite easy: it read forms from
the input, compiles and evaluates them. That's it. Really!

Consider the following file:

(ns foo.bar)

(require '[foo.baz :as baz])

(defn frobnicator
[flogiston-reactor]
(baz/do-frobnication flogiston-reactor))

What does happen here? Easy! Clojure reads the file and first gets the
ns declaration back from the Reader. It creates the foo.bar namespace
if
necessary and does any require/use/import things. Then it reads the
next
form: require. It gets compiled and evaluated. That is, it tries to
load
the foo.baz namespace and - if successful - sets up an namespace alias
by virtue of the alias function. Next it reads frobnicator and sets up
the Var and the function it contains. That's all. Nothing more. You
can
see this by moving the frobnicator call above the require and see the
load fail. (In a fresh Repl of course)

The same happens whether you load it from the classpath or via
load-file. Put things in a file-named-independent-from-foo.bar.clj and
load it via load-file: it will still work. Put things in a String and
use a StringReader and things will still work. Namespace vs. name of
file convention just helps finding things on the classpath. And this
is
same in Python, Ruby, OCaml, etc.

Require and friends just set up what *this* namespace needs. I think,
95% of the use cases for immigrate should probably read load.

Hope that helps.

Sincerely
Meikel

[Adrian Cuthbertson]

Some time ago I experimented with migrating definitions between
namespaces. Here's a post from then that covers some lower-level
detail regarding namespaces.

http://groups.google.com/group/clojure/msg/f00a82b747c1636f

-Hth, Adrian.

[Mark Engelberg]

I spent the past half-hour experimenting with what happens in Clojure if the ns declaration doesn't match the file name, and what happens if the ns declaration is not at the top of the file.

Here's what I've found so far:
1.  When you require/use/load a file, Clojure finds it by the filename.
2.  If your ns declaration does not match the filename, Clojure happily goes ahead and puts any definitions that follow the ns declaration inside the namespace named by the ns.
3.  If you have no ns declaration, then the code in the file gets loaded into the namespace of the file which is requiring it.
4.  If the ns declaration is not at the top of the file, then the code above the ns declaration gets loaded into the namespace of the file which is requiring it, and the code after it goes into whatever namespace is specified by the ns declaration.
5.  I am unable to find any difference (other than syntax) between requiring a file and loading it.  Is there any difference? (Perhaps if you load vs. require the same file from two other files, there's a difference?)

I'm still not having much luck loading another file using a path relative to the location of the file that is doing the requiring, and don't know if that's supposed to be possible.  Everything seems to be relative to the root of the part of the classpath containing the requiring file.

So what are some good examples of why you'd want to do any of these funky things, e.g., using namespaces that don't match the filenames, putting namespace declarations in the middle or omitting them entirely?

[Konrad Hinsen]

On 9 May 2010, at 04:44, Mark Engelberg wrote:

> In Python, let's say I have a library "mylibrary.py", and from various
> files I say "from mylibrary import *".
> If I want to split mylibrary.py into two parts, say, library1.py and
> library2.py, then I can just change mylibrary.py to say:
> from library1 import *
> from library2 import *

Since you mention Python, there is one very important difference to
keep in mind: Clojure has a two-step symbol resolution, whereas
Python's is one-step. In other words, Clojure has one more level of
indirection.

In Python, a module (the closest equivalent to a Clojure namespace) is
a mapping from symbols to values. Defining a function or assigning to
a variable inside a module assigns a new value to a symbol, creating
it if necessary. Importing names from another modules has the same
result: values are assigned to symbols in the importing namespace.

In Clojure, there are symbols and vars where Python has just symbols.
A namespace has interned vars, which in a certain sense "belong" to
the namespace. def and its variants assign new values to interned
vars, creating them if necessary. But a namespace also has a mapping
from symbols to vars. A symbol can be mapped to a var in the same
namespace (the case of interned vars), or to a var in another
namespace. The latter kind of reference is created by :use in the ns
form.

The difference matters when the values of vars change, either by
redefinition or inside a binding form. In Python, an imported symbol
retains its value even when the symbol's definition in the original
namespace changes. In Clojure, a reference to another namespace's var
always reflects the current value of that var.

A while ago I wrote a blog post that goes into more detail on this
topic:
http://onclojure.com/2010/02/17/managing-namespaces/


I agree with your observations about difficulties with namespace
refactoring in Clojure, and I would like to see additional namespace
management options for the cases you mention. I am rather sure that
all of this can be done without any change to the fundamental
namespace mechanism, just by providing additional options/functions
for namespace management. I have started a small library for
exploring possibilities:

http://code.google.com/p/clj-nstools/

Comments and enhancements are always welcome.

Konrad.