Why does gen-class executes with *ns* bound to clojure.core?

[Didier]

Take this code:

(ns dda.main

  (:gen-class))

(defn -main [] (prn (ns-name *ns*)))

If you bootstrap this through clojure.main, such as what lein does, it will print the namespace as "user". But if you bootsrap it through the generated java main class, it will return "clojure.core".

My intuition would have assumed it would be running under "dda.main" as the *ns*, or at least under "user", since that's a common convention.

I'm curious to know why "clojure.core"?

[Didier]

Especially given this:

(ns dda.main

  (:gen-class))

(def should-exist "Do I exist?")

(defn -main [] (prn (str should-exist \space (ns-name *ns*))))

The (def) is evaluated, since should-exist is found when -main is called from the generated class, but why isn't (ns) evaluated? Which should have changed *ns* to "dda.main".

[Justin Smith]

*ns* is a dynamic var, so it points to the current namespace when your function is running. Most code doesn't switch into a target ns in order to execute functions from it.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sean Corfield]

The “trick” I’ve adopted to get around this:

 

(def my-ns *ns*)

 

I stick that near the top of any file in which I want to refer to _that_ namespace (as opposed to whatever the _current_ value of *ns* is during evaluation, since it’s dynamic).

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

--

[Didier]

*ns* is a dynamic var, so it points to the current namespace when your function is running. Most code doesn't switch into a target ns in order to execute functions from it.

I understand that, and I understand most code does not switch into the target ns. But this is not most code, this is the code generated by the gen-class directive to bootstrap your -main method from java. Both lein and clojure.main bootstrap your main method from inside "user", but gen-class does so from inside "clojure.core".

So I want to know:

1) Why does the generated java class from gen-class has "clojure.core" as its current namespace?

2) Why doesn't the (ns) function inside my namespace change the current namespace?

Thanks

[Alex Miller]

On Tuesday, June 20, 2017 at 7:43:53 PM UTC-5, Didier wrote:

*ns* is a dynamic var, so it points to the current namespace when your function is running. Most code doesn't switch into a target ns in order to execute functions from it.

I understand that, and I understand most code does not switch into the target ns. But this is not most code, this is the code generated by the gen-class directive to bootstrap your -main method from java. Both lein and clojure.main bootstrap your main method from inside "user", but gen-class does so from inside "clojure.core".

So I want to know:

1) Why does the generated java class from gen-class has "clojure.core" as its current namespace?

gen-class causes a namespace to be AOT compiled to a class. Leiningen runs the AOT compilation by invoking the compiler. The initial value of *ns* is clojure.core (see RT.java). During compilation, nothing ever changes this, so that's what gets compiled into the Java class on disk. When you invoke it, this value has been compiled into the class already.

When you invoke -main directly, you are running in the repl. The repl sets the value of *ns* to "user" when it starts, so that's what you see.

 

2) Why doesn't the (ns) function inside my namespace change the current namespace?

It does, but only when you run it. The AOT compiled class also creates an init class that will cause the same effects as loading a non-compiled clj file. That file has already expanded `ns` but it will include a call to `in-ns` that will change the current namespace. However, the compiled class for the -main function already has the value at the time of compilation, so that doesn't matter.

 

[Alex Miller]

Actually, scratch some of what I said (it's late). It's not that the value gets compiled into the class - the -main function will actually look up the value of *ns*. The difference is only in the ordering of what happens in the two scenarios.

If you do "lein run", that will invoke clojure.main with a main class. Going through clojure.main will set the *ns* to `user`.

If you do "java -jar target/whatever-standalone.jar", then you will just load and invoke the main class set in the manifest (dda.main). Loading that class will (as a side effect) load RT where *ns* is set to clojure.core, but will not run through the code in clojure.main.

On Tuesday, June 20, 2017 at 11:03:35 PM UTC-5, Alex Miller wrote:

On Tuesday, June 20, 2017 at 7:43:53 PM UTC-5, Didier wrote:

*ns* is a dynamic var, so it points to the current namespace when your function is running. Most code doesn't switch into a target ns in order to execute functions from it.

I understand that, and I understand most code does not switch into the target ns. But this is not most code, this is the code generated by the gen-class directive to bootstrap your -main method from java. Both lein and clojure.main bootstrap your main method from inside "user", but gen-class does so from inside "clojure.core".

So I want to know:

1) Why does the generated java class from gen-class has "clojure.core" as its current namespace?

gen-class causes a namespace to be AOT compiled to a class. Leiningen runs the AOT compilation by invoking the compiler. The initial value of *ns* is clojure.core (see RT.java).

 

During compilation, nothing ever changes this, so that's what gets compiled into the Java class on disk. When you invoke it, this value has been compiled into the class already.

^^ wrong

 

When you invoke -main directly, you are running in the repl. The repl sets the value of *ns* to "user" when it starts, so that's what you see.

 

2) Why doesn't the (ns) function inside my namespace change the current namespace?

It does, but only when you run it. The AOT compiled class also creates an init class that will cause the same effects as loading a non-compiled clj file. That file has already expanded `ns` but it will include a call to `in-ns` that will change the current namespace.

 

However, the compiled class for the -main function already has the value at the time of compilation, so that doesn't matter.

^^ wrong

 

 

[Didier]

That makes sense. Actually if you use Clojure's Java API and you require a namespace and invoke a Var, it will do the same, *ns* will just be clojure.core.

So I get it is simply because the last thing to set ns was RT.

One more question:

So I assume "load" would eval ns in my namespace, setting *ns* to dda.main, then it would run through the file, eval all forms, and finally it would reset *ns* to the previous value. Is that correct, or is there something about loading that's different?

Thanks

[Alex Miller]

Pretty much. You could be a little more precise to say that load pushes a local *ns* dynamic binding onto the stack and that context is popped when you leave the load, so really it's not resetting *ns*, it's just popping the nested context off (and that value was there all along). You can see some of that in RT.load().

 

Thanks

[Didier]

I think I answered myself, looks like it does here:

try {
	Var.pushThreadBindings(
	RT.mapUniqueKeys(CURRENT_NS, CURRENT_NS.deref(),
	WARN_ON_REFLECTION, WARN_ON_REFLECTION.deref()
	,RT.UNCHECKED_MATH, RT.UNCHECKED_MATH.deref()));
	loaded = (loadClassForName(scriptbase.replace('/', '.') + LOADER_SUFFIX) != null);
	}
	finally {
	Var.popThreadBindings();
	}

So now I understand, but I wonder if it makes sense. 99% of the time, it won't be an issue, but if you're doing anything with *ns*, you'll run into a situation where if you compile and bootstrap yourself through gen-class or clojure's java API, your code will not behave similar to when you're loaded at runtime or bootsraped through clojure.main or lein.

(ns dda.main

  (:gen-class))

(def should-exist "Do I exist?")

(defn -main []

  (in-ns 'other)

  (ns-name *ns*))

A bad example, but this code will work through lein, clojure.main, and when loaded from a REPL, but not when started from gen-class or Clojure's java API. Is there a good reason for the discrepancy? Shouldn't all methods to bootsrap your code start out with an equal initialization scheme? I would suggest that Clojure's java API (which I assume gen-class uses under the hood) should also initialize the common bindings and set the namespace to user. That way, all entry point always behave similarly.

[Alex Miller]

On Wednesday, June 21, 2017 at 9:42:00 AM UTC-5, Didier wrote:

I think I answered myself, looks like it does here:

try {
	Var.pushThreadBindings(
	RT.mapUniqueKeys(CURRENT_NS, CURRENT_NS.deref(),
	WARN_ON_REFLECTION, WARN_ON_REFLECTION.deref()
	,RT.UNCHECKED_MATH, RT.UNCHECKED_MATH.deref()));
	loaded = (loadClassForName(scriptbase.replace('/', '.') + LOADER_SUFFIX) != null);
	}
	finally {
	Var.popThreadBindings();
	}

So now I understand, but I wonder if it makes sense. 99% of the time, it won't be an issue, but if you're doing anything with *ns*, you'll run into a situation where if you compile and bootstrap yourself through gen-class or clojure's java API, your code will not behave similar to when you're loaded at runtime or bootsraped through clojure.main or lein.

It would be helpful to be clearer about what you mean with these different cases as it's ambiguous to me at least. I think you mean:

1. "compile and bootstrap yourself through gen-class" means: you generate a class and then invoke the main() method of that class

2. "loaded at runtime" means: you (presumably) start the REPL, load a namespace (either from a source file or a class file), then invoke the -main function

3. "bootstrapped through clojure.main" means: you invoke clojure.main (a compiled program provided with Clojure) with the -m arg specifying a namespace. This will start the Clojure runtime, load the namespace, then invoke the -main function

4. "bootstrapped through lein" means: I assume this is "lein run" with a :main set or "lein run -m" but I think lein supports specifying a namespace or a function in a namespace or a class with a main() method. Depending which of those you're doing, this is like similar to either #1 or #2 and here lein is effectively doing similar work as #3.

5. There is a Clojure Java API (http://clojure.github.io/clojure/javadoc/), but I'm not sure if you are actually referring to this or something else. Doing so would basically mean going through that API to do the same thing as #2. 

 

(ns dda.main

  (:gen-class))

(def should-exist "Do I exist?")

(defn -main []

  (in-ns 'other)

  (ns-name *ns*))

A bad example, but this code will work through lein, clojure.main, and when loaded from a REPL, but not when started from gen-class or Clojure's java API. Is there a good reason for the discrepancy?

You are doing different things, and thus experiencing different results, so in some sense I would expect them not to be identical. In particular, both #3 clojure.main and #4 Leiningen are effectively programs wrapping your program - since you're using a wrapper, you should expect that wrapper to do more stuff and have an effect on your runtime environment. What I think maybe is interesting here is the difference between invoking a main() method first vs loading and invoking the function from the Clojure runtime (#1 vs #2). There you see that some things are bound in #2 that have not yet been bound in #1. More below at the end.

 

Shouldn't all methods to bootstrap your code start out with an equal initialization scheme?

Maybe. We are also interacting with the host initialization here and that complicates things. The only way to start a Java program is ultimately to specify a class with a main() method, so that's a constraint we have to deal with.

 

I would suggest that Clojure's java API (which I assume gen-class uses under the hood)

Again, I'm struggling to connect "Clojure's Java API" here to what that means to me. gen-class is not involved at runtime - it's compile-time functionality that produces a class file. That generated class file uses the *Clojure runtime* to load and invoke Clojure functions. Most of that is exactly the same as what happens when running typical Clojure functions. The part that's different is that there is some additional scaffolding created to handle the namespace-level effects when loading AOT classes.

 

should also initialize the common bindings and set the namespace to user. That way, all entry point always behave similarly.

I think if I were to restate your suggestion, I would say that a genclass-compiled main entry point should initialize the Clojure runtime and invoke the code in a binding as if it were being invoked from other Clojure code. I can see some logic in that (although that then also affects #3 and #4 as they go through this code too).

[Didier]

Great breakdown.

1. "compile and bootstrap yourself through gen-class" means: you generate a class and then invoke the main() method of that class

I'd go further then just calling main. So I mean you generate a class, and every function call from Java using the generated class should be executed as if it were executed through clojure.main, so with the bindings set and inside the user namespace.

 

2. "loaded at runtime" means: you (presumably) start the REPL, load a namespace (either from a source file or a class file), then invoke the -main function

Correct. It's the case of a non AOT compiled namespace being loaded from one of the many (load) functions of Clojure. Such as:

(load "dda/main")
(dda.main/-main)

 

3. "bootstrapped through clojure.main" means: you invoke clojure.main (a compiled program provided with Clojure) with the -m arg specifying a namespace. This will start the Clojure runtime, load the namespace, then invoke the -main function

Correct.

 

4. "bootstrapped through lein" means: I assume this is "lein run" with a :main set or "lein run -m" but I think lein supports specifying a namespace or a function in a namespace or a class with a main() method. Depending which of those you're doing, this is like similar to either #1 or #2 and here lein is effectively doing similar work as #3.

Correct, yes internally lein delegates to clojure.main I believe.

 

5. There is a Clojure Java API (http://clojure.github.io/clojure/javadoc/), but I'm not sure if you are actually referring to this or something else. Doing so would basically mean going through that API to do the same thing as #2. 

I am talking about that API. I believe it uses RT under the hood, and ends up doing the same thing that #1 does. So in my test, if you did:

IFn require = Clojure.var("dda.main", "-main");
require.invoke();

This will run inside "clojure.core", and will not have any of the default bindings set available.

I think if I were to restate your suggestion, I would say that a genclass-compiled main entry point should initialize the Clojure runtime and invoke the code in a binding as if it were being invoked from other Clojure code. I can see some logic in that (although that then also affects #3 and #4 as they go through this code too).

Ya, arguably, I wonder why the clojure runtime doesn't initialize the bindings and sets the namespace to 'user before delegating execution back to user code. If it did, then every invocation of Clojure from Java would always be bootstrapped in a similar way, and run under an equal context. Clojure.main wouldn't need to do it anymore.

But there might be a good reason for not doing that inside the clojure runtime. In which case, it still leaves open the question of should clojure invocations through a generated class, and invocations from the Clojure API mimic the bootstrapping of clojure.main?

P.S.: I also know that in practice, this has not caused that many issues, due to the fact that Clojure is now many years old, and I've never heard of other people complain about the discrepancies between generated classes and the clojure API versus clojure.main. So I don't think its a pressing issue, but I still feel it could be worth a thought.

[Alex Miller]

On Wednesday, June 21, 2017 at 6:09:43 PM UTC-5, Didier wrote:

Great breakdown.

1. "compile and bootstrap yourself through gen-class" means: you generate a class and then invoke the main() method of that class

I'd go further then just calling main. So I mean you generate a class, and every function call from Java using the generated class should be executed as if it were executed through clojure.main, so with the bindings set and inside the user namespace.

I do not think you should treat clojure.main as the gold standard. clojure.main is a helper program and it makes some choices that may or may not be what every program wants. There is nothing special about the "user" namespace for example. Other repls choose other things (pretty sure the boot repl uses boot.user example). Likewise, there's nothing special about the set of things clojure.main chooses to bind on startup - lein, boot, and other tools make different choices. If anything, I'd say #2 is what you should consider the most canonical case.

 

 

2. "loaded at runtime" means: you (presumably) start the REPL, load a namespace (either from a source file or a class file), then invoke the -main function

Correct. It's the case of a non AOT compiled namespace being loaded from one of the many (load) functions of Clojure. Such as:

(load "dda/main")
(dda.main/-main)

 

3. "bootstrapped through clojure.main" means: you invoke clojure.main (a compiled program provided with Clojure) with the -m arg specifying a namespace. This will start the Clojure runtime, load the namespace, then invoke the -main function

Correct.

 

4. "bootstrapped through lein" means: I assume this is "lein run" with a :main set or "lein run -m" but I think lein supports specifying a namespace or a function in a namespace or a class with a main() method. Depending which of those you're doing, this is like similar to either #1 or #2 and here lein is effectively doing similar work as #3.

Correct, yes internally lein delegates to clojure.main I believe.

No, I don't think lein uses clojure.main at all.

 

 

5. There is a Clojure Java API (http://clojure.github.io/clojure/javadoc/), but I'm not sure if you are actually referring to this or something else. Doing so would basically mean going through that API to do the same thing as #2. 

I am talking about that API. I believe it uses RT under the hood, and ends up doing the same thing that #1 does. So in my test, if you did:

This is actually the same as #2, not #1. It's not making an interop call, it's invoking a var function, just like any other Clojure call. 

IFn require = Clojure.var("dda.main", "-main");
require.invoke();

This will run inside "clojure.core", and will not have any of the default bindings set available.

I think if I were to restate your suggestion, I would say that a genclass-compiled main entry point should initialize the Clojure runtime and invoke the code in a binding as if it were being invoked from other Clojure code. I can see some logic in that (although that then also affects #3 and #4 as they go through this code too).

Ya, arguably, I wonder why the clojure runtime doesn't initialize the bindings and sets the namespace to 'user before delegating execution back to user code.

Again, there is nothing special about user and I wouldn't get hung up on that. It does seem like it would be reasonable to set up a binding context for *ns* though so it's set!-able - that's the thing that's missing.

 

If it did, then every invocation of Clojure from Java would always be bootstrapped in a similar way, and run under an equal context. Clojure.main wouldn't need to do it anymore.

But there might be a good reason for not doing that inside the clojure runtime. In which case, it still leaves open the question of should clojure invocations through a generated class, and invocations from the Clojure API mimic the bootstrapping of clojure.main?

I would say, no. :) But it might be reasonable to mimic a Clojure function invocation that has a binding for *ns*. If you want to file a ticket, I'm happy to think about it more.

[Alex Miller]

Was just musing some more - the Clojure Java API was added pretty late (~1.6 or 1.7). I wonder if

1) changing the Java Clojure API to ensure a certain binding context and

2) changing the main() method to invoke the Clojure function through the Java Clojure API

would be a good end game. Would need a lot more analysis.

[Didier]

I opened a Jira enhancement: https://dev.clojure.org/jira/browse/CLJ-2185

Hope I did a good job at it.

[Jeff Rabinowitz]

If I may be so bold to interject -- 

I'm a colleague of Didier's, and I originally brought this to his attention when I hit this issue at work. I just want to add that part of the confusion arises from some miscommunication/incomplete information on the official docs for Namespaces;

At the Repl it’s best to use in-ns, in which case the new namespace will contain mappings only for the classnames in java.lang. In order to access the names from the clojure.core namespace you must execute (clojure.core/refer 'clojure.core). The user namespace at the Repl has already done this.

 

The current namespace, *ns* can and should be set only with a call to in-ns or the ns macro, both of which create the namespace if it doesn’t exist.

I believe this is a major source of the confusion (at least it was for me). The official documentation does mention that in-ns is recommended in the REPL; and in a subsequent paragraph, the official documentation does state that in-ns (or ns but it macroexpands to in-ns as well so is just as bad in context) is the preferred way of creating a namespace. But nowhere on the official documentation page does it qualify that *ns*, as a dynamic Var, will require curation if bootstrapping an application from a static context! Instead, users are thrown with the ugly IllegalStateException("Can't change/establish root binding of: *ns* with set"), which is fairly inscrutable in context. (Nowhere else that I'm aware of in the runtime can a user not have ever created a dynamic var themselves, have manipulated it without realizing they're using the set! API, and end up trying to set! it at runtime before calling binding.) It's not a lie or intentionally misleading, but allows an application developer to walk away with an incomplete and inconsistent understanding of the operation of those functions. Somehow the logical decision was taken at every step, but yet there's room for users to trip up here.

People who tinker with macros and evals are probably accustomed to battling issues with dynamic vars, or at the very least are remembering that dynamic vars have different semantics, and so can easily search online to discover that they ran afoul of the Var stack frame issue and need to add a binding. But in-ns is a real head-scratcher! If implemented in Clojure, it would look like (set! clojure.core/*ns* (create-ns 'some-namespace)), but because it's defined in RT.java, there's no source available; what's more, the documentation attached to the Var for in-ns doesn't allude to the mechanics it employs. 

Hands down I can guarantee all Clojure developers who aren't involved in the language maintenance will respond to the exception above by thinking, "Let me check the source using the source macro and see what just happened." They'll call clojure.repl/source and get -- nothing. Then they'll check the docs which say -- "Sets *ns* to the namespace named by the symbol, creating it if needed." While technically correct, this is hands down more likely to be read in the colloquial English meaning of "casually changes", rather than the Clojure-technical meaning of "Does something akin to calling `set!`". And they'll either give up after an hour of fruitless searching, or if they have a lot of free time, will spend several hours poring over compiler source code until they realize that nobody documented this case; and that the mechanics at runtime, while internally consistent, are completely counter-intuitive given the information available on the public documentation.

I'm glad that you're considering "plugging" the other Clojure entry points to eliminate this stumbling block altogether. In the meantime, perhaps some clarification of the documentation (either those attached to the Var itself, or on the site, or both) are in order?

Thanks,

- Jeff

[Alex Miller]

Hey Jeff, I appreciate the comments. The fact that the Clojure REPL sets up a dynamic context (not just with *ns*, but with many other dynamic bindings as well) that we take  for granted is indeed confusing in the gen-class and other scenarios. If you want to file an issue for the reference docs on the site, the place to do that is here:

https://github.com/clojure/clojure-site/issues

Alex