Why does require always try to reach the disk ?

[Laurent PETIT]

hello,

When doing interactive development with the REPL, it's not uncommon
that I start creating namespaces, with dependencies between them,
live.

But this fails and is easy to reproduce in the simplest possible case, look:

=> (ns a)
nil
=> (ns b (:require a))
FileNotFoundException Could not locate a__init.class or a.clj on
classpath: clojure.lang.RT.load (RT.java:430)


I would expect that require not try to reach the disk if the required
namespace is already loaded. And that only :reload / :reload-all
options would try to reach the disk bindly.


The current behavior is getting in the way of at least 2 use cases I'm
trying to make smooth:

- several namespaces concatenated into the same file, and loaded at
once via "load-file" - think when you keep things in gists / snippets.
- when developing with complex setup where namespaces live in separate
projects which are handled by separate classloaders at runtime, such
as OSGi where you split things into separate bundles (this one can be
worked-around, tho, but the first issue is annoying me most).


Is there a reason I'm missing that suggests the current behaviour is
desired over what I'm suggesting?

May I file an issue in JIRA ?

-- Laurent

[Stuart Sierra]

require/use don't always go to disk. But they don't look at namespaces currently in existence, just a set of symbols that were previously loaded with require/use, in clojure.core/*loaded-libs*.

I'm not sure what the implications of this change would be.

-S

[Laurent PETIT]

2012/11/23 Stuart Sierra <the.stua...@gmail.com>:

First line of (doc require):

"Loads libs, skipping any that are already loaded."

Isn't a call to (find-ns 'the-ns) sufficent to determine that a lib
has already been loaded ?

[Laurent PETIT]

2012/11/23 Laurent PETIT <lauren...@gmail.com>:

As I understand the source code, it seems that 'require has been
opened to the possibility that a "lib" may not have an associated
namespace. It only ensures it has an associated namespace if :use or
:as options are used.

Do you know of any usage of a call to (require) not resulting in the
creation of at least one namespace named along the lib name ? Really ?
I don't think so, and think it would be seen as daaark magic, only
understandable by a few, wouldn't it ?

As I understand it, clojure.core/*loaded-libs* may help in preventing
infinite loops and returning errors when facing libs that
cross-reference themselves. Not sure if this *loaded-libs* can be
totally removed, because it may help when used in combination with
reload-all.

Anyway, if I can wrap my head around the current source code, and find
a fix, would there be any reason not to do it ?

Cheers,

--
Laurent

[Stuart Sierra]

> Isn't a call to (find-ns 'the-ns) sufficent to determine that a lib
> has already been loaded ?

`find-ns` could easily be true in cases where `the-ns` has *not* been loaded. For example:

    user=> (in-ns 'foo)
    foo=>
     ;; darn, forgot to load it
    foo=> (clojure.core/require 'foo)

Furthermore, even if you could do this, how would you implement `(require ... :reload-all)`?

As an alternative: It's not documented, but you can use `:refer` in `ns`:

    user=> (ns a)
    nil
    a=> (defn foo [] "foo")
    #'a/foo
    a=> (ns b (:refer a))
    nil
    b=> (foo)
    "foo"

-S

[Laurent PETIT]

2012/11/24 Stuart Sierra <the.stua...@gmail.com>:

>> Isn't a call to (find-ns 'the-ns) sufficent to determine that a lib
>> has already been loaded ?
>
> `find-ns` could easily be true in cases where `the-ns` has *not* been
> loaded. For example:
>
> user=> (in-ns 'foo)
> foo=>
> ;; darn, forgot to load it
> foo=> (clojure.core/require 'foo)

Isn't this a case for (clojure.core/require 'foo :reload) ? And
besides, in-ns creates an empty ns instead of eg throwing an
exception, so I see no reason why this would be problematic? It's
expected!

Said otherwise, what is different between not specifying :reload and
specifying it, then?

>
> Furthermore, even if you could do this, how would you implement `(require
> ... :reload-all)`?
>
> As an alternative: It's not documented, but you can use `:refer` in `ns`:
>
> user=> (ns a)
> nil
> a=> (defn foo [] "foo")
> #'a/foo
> a=> (ns b (:refer a))
> nil
> b=> (foo)
> "foo"

I can try it. But then, it won't do what's expected when not working
from the REPL, meaning I'll have to replace :require with :refer
depending on how I organize my code. Don't like that.

[Stuart Sierra]

I don't want to discourage you. Give the implementation a try, let's see how it works out in practice. :)

-S

[Chas Emerick]

FWIW, I'd +10 this if I could.  It's one of those things that I became inured to long ago, and now silently and automatically work around.

- Chas

[Frank Siebenlist]

Another +10 from me…

-FrankS

> --
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/clojure-dev/-/f4YXFOti7cYJ.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.

[Herwig Hochleitner]

+1 

This was really surprising for me, the first time I tried to feed a bunch of new code into a remote process.

2012/11/26 Frank Siebenlist <frank.si...@gmail.com>

[Jim Crossley]

+1

This complicates my dream of one day being able to build an app from scratch via a REPL running inside Immutant on Openshift.

[Laurent PETIT]

tl;dr: I have a working patch. Review/try it from here:

https://github.com/laurentpetit/clojure/commit/0fe0c5238c00ef871ff6523d4b50d07a1bd86aa8

See below for a discussion on what's been done, and why it's been done this way.


OK, I've made some progress on this. Thanks to Christophe Grand also
for helping me decipher some dark corners of clojure.core loading
(more on this later).

Basically, the problem is that a namespace is considered as loaded if
it has been required, but not when (ns ....) has been called.

So a namespace enters the *loaded-libs* set only once it's been
(required) at least once.

A simple one-liner fix is to add, at the end of the macro-expanded
code for 'ns, the following line:

(dosync (commute *loaded-libs* conj '~name))

This has the advantage of addressing Stuart's issue with premature
calls to 'in-ns, since 'in-ns is not touched and would not add the ns
to *loaded-libs*

Except it will work in every case but when clojure.core itself is
loaded *if it has been AOT compiled*.

And that's an interesting story of itself :

The first line of clojure.core is (ns ^{:some "totally" :ignored
"metadata"} clojure.core)

- When clojure is not AOT compiled:
- the ns used is the "bootstrap" ns, defined in RT.java. It's doing
nothing more than the "bootstrap" in-ns: create the namespace, set it
as the current namespace
- note that in this case the metadata is ignored. The boostrap ns
discards it. That's why a short docstring is added later via an
alter-meta call

- When clojure is AOT compiled:
- clojure.core initializing is compiled in clojure/core__init.class,
and the 'ns code that will be called is not the bootstrap code, but
the compiled ns defined in clojure.core during the AOT compilation !
- this means that clojure.core is not loaded the same way depending
on whether clojure has been AOT compiled or not. In one case it's via
the bootstrap ns, in the other case via the result of the
macro-expansion of ns defined in itself (still following ? :-) )
- as a result, clojure/core__init.class evaluates code which can
only use interop calls to clojure.lang java clojure, because it's the
only thing available !!


This complicate things for me, when I want to add, at the end of 'ns
macro, code to access clojure.core/*loaded-libs*.
In either case, when applied to clojure.core itself, it's trying to
access a var yet to be defined.


Fortunately, there's an easy solution: a special case in the 'ns macro
for clojure.core.
Given all the specifics which apply to clojure.core already, I take it
for granted that is should not be a problem ?


Alternative solutions exist for solving, this:

- change the start of clojure/core.clj : replace the (ns ....
clojure.core) with (in-ns 'clojure.core)
- this would allow removing bootstrap 'ns code from RT.java (raw
in-ns would be sufficient)
- would remove the duplication of metadata for clojure.core (moved
in one place)
- solving the problem, since then the macro-expansion of ns would
never be called on clojure.core before it is initialized

- do things differently to solve the 'require problem:
- do not add, at the end of 'ns macro-expansion, code to add the ns
to *loaded-libs*, but rather just add a metadata to the ns as a sign
that 'ns has been called on it (and not just 'in-ns)
- and then, in 'load-lib, complement the test for loaded-libs with
the presence of this metadata on the ns


All in all, the above mentioned solutions seem to provide easy paths
towards solving the problem. There's just a choice to be made.

Thoughts, gentlemen?


2012/11/27 Jim Crossley <jcros...@gmail.com>:

[Laurent PETIT]

Hello there,

In the mean time, I've managed to talk a bit to Rich about it, and now
it is created in JIRA, and awaits for screeners to approve it to be
included in 1.5.

Please, all who did +1 (and even +10 !) it, take the time to test it
and screen it !

http://dev.clojure.org/jira/browse/CLJ-1116

Cheers,

--
Laurent

2012/11/29 Laurent PETIT <lauren...@gmail.com>:

[Laurent PETIT]

The ticket solving this issue has already been screened twice successfully.

It would be great if it could get more votes (I think it deserves more
than what it already has, since at least 3-4 people gave him +1 to +10
on this ml, but it only does have one vote at the moment).

http://dev.clojure.org/jira/browse/CLJ-1116

Thanks,

[Rich Hickey]

On Dec 9, 2012, at 9:09 AM, Laurent PETIT wrote:

> The ticket solving this issue has already been screened twice successfully.
>
> It would be great if it could get more votes (I think it deserves more
> than what it already has, since at least 3-4 people gave him +1 to +10
> on this ml, but it only does have one vote at the moment).
>
> http://dev.clojure.org/jira/browse/CLJ-1116
>

Please, no.

Votes are for issues not yet screened.

This is not only screened, it is already ok'ed. It will be in the next build.

[Laurent PETIT]

On Dec 9, 2012 3:39 PM, "Rich Hickey" <richh...@gmail.com> wrote:
>
>
> On Dec 9, 2012, at 9:09 AM, Laurent PETIT wrote:
>
> > The ticket solving this issue has already been screened twice successfully.
> >
> > It would be great if it could get more votes (I think it deserves more
> > than what it already has, since at least 3-4 people gave him +1 to +10
> > on this ml, but it only does have one vote at the moment).
> >
> > http://dev.clojure.org/jira/browse/CLJ-1116
> >
>
> Please, no.
>
> Votes are for issues not yet screened.

Oh sorry, I hadn't understood it that way, thanks for the clarification.

>
> This is not only screened, it is already ok'ed. It will be in the next build.
>
>