Cyclic namespace dependencies!

[Mars0i]

I love Clojure.  But:  My number one complaint about it is that the compiler won't allow cyclic namespace dependencies.  I am not a compiler-writer, but as far as I can tell the only reason for this restriction is ... philosophical.  It forces bad code organization, and it wastes time.  Those who think that cyclic code is an evil are free to encourage its avoidance.  I'm happy to avoid it!  But sometimes I can't.  I absolutely hate the cyclic namespace restriction.  Drives me crazy when I am forced to encounter it.

My latest case is that I'm trying to debug a weird Java interop problem--something that works in one app I've written, in which I ended up putting most of the application into one file, but doesn't work in the one I'm working on now.   And I had finally come up with a clever way of avoiding sticking most of the code into one namespace.  I'd worked around the conflict between Clojure's requirements and the requirements of the Java library I'm using.  But now there's this one problem, and I want to try an experiment to see whether it fixes the problem.  It looks like I'm going to have to pack all of the code into one file and rewrite everything that that necessitates just for this purpose. Aghh.

In another post a while ago, I also noted that whether there's a cyclic dependency sometimes depends on seemingly arbitrary changes to code--e.g. add a type hint to avoid reflection, and all of a sudden you've got cyclic code that you didn't have before.  This isn't the current case--I haven't even gotten to that stage.

I have sometimes ranted a bit in this group, and in the end some of my rants were unreasonable.  I don't think this is unreasonable.  The cyclic dependency restriction is absurd.  I'm not trying to start a flame ware.  I feel strongly about this, and I'm frustrated.

What I want: I'm not trying to make everyone sit around tapping their fingers while code to goes through multi-pass compliations.  All I'm asking for is a compiler option that will allow cyclic dependencies when requested.

I'm pretty sure that the way I've put things above illustrates some ignorance on my part, but still ... how hard can this be?  It's a reasonable thing to ask of a sophisticated compiler.

Finally, is there a JIRA ticket for this issue?  I looked but didn't find it.  I will add a ticket if not, even if it's not likely to be acted upon any time soon.

Again: I sincerely love Clojure!
But.

[Nathan Davis]

On Friday, December 30, 2016 at 10:13:49 AM UTC-6, Mars0i wrote:

I love Clojure.  But:  My number one complaint about it is that the compiler won't allow cyclic namespace dependencies.  I am not a compiler-writer, but as far as I can tell the only reason for this restriction is ... philosophical.  It forces bad code organization, and it wastes time.  Those who think that cyclic code is an evil are free to encourage its avoidance.  I'm happy to avoid it!  But sometimes I can't.  I absolutely hate the cyclic namespace restriction.  Drives me crazy when I am forced to encounter it.

First of all, there is no such restriction.  So there is no reason for it, philosophical or otherwise.  It is true that if you declare all your dependencies in ns forms, you will get an exception with cyclic namespaces.  That's a good thing -- the result probably wouldn't be what you want anyway.  However, there's no requirement that all dependencies be declared using ns -- you can use require at any time.

What is required is that all the vars a form uses exist when it is compiled.  The reason for this is purely technical -- resolving symbols to vars at compile time means less work (more efficiency) at run time.
 

My latest case is that I'm trying to debug a weird Java interop problem--something that works in one app I've written, in which I ended up putting most of the application into one file, but doesn't work in the one I'm working on now.   And I had finally come up with a clever way of avoiding sticking most of the code into one namespace.  I'd worked around the conflict between Clojure's requirements and the requirements of the Java library I'm using.  But now there's this one problem, and I want to try an experiment to see whether it fixes the problem.  It looks like I'm going to have to pack all of the code into one file and rewrite everything that that necessitates just for this purpose. Aghh.

Perhaps if you provided a coherent description the specifics of your problem, someone would be able to suggest a satisfactory solution.
 

I have sometimes ranted a bit in this group, and in the end some of my rants were unreasonable.  I don't think this is unreasonable.  The cyclic dependency restriction is absurd.  I'm not trying to start a flame ware.  I feel strongly about this, and I'm frustrated.

What would be reasonable is asking questions about things until you understand them well enough to be in a position to make an informed opinion.  In this case:

1. Clojure does allow namespaces to refer to each other (just maybe not how you've tried).
2. There are plenty of technical considerations when it comes to mixing dynamic languages and cyclic references.  Cyclic namespaces are a subset of this problem.  So describing restrictions on cyclic namespaces (whether real or imagined) as "absurd" is absurd.

A rant is just a rant, regardless of whether a problem actually exists.  If you feel strongly about something and only rant about it, don't be surprised when you feel frustrated.  Frustration can be relieved by solutions.  If you don't understand the problem enough to present a solution, ask questions until you do.  Don't expect others to do all the work -- especially after ranting about it.

 

What I want: I'm not trying to make everyone sit around tapping their fingers while code to goes through multi-pass compliations.  All I'm asking for is a compiler option that will allow cyclic dependencies when requested.

I don't think multi-pass compilation would play well with things like macros or REPL-based development, so I doubt we'll have to worry about that.
 

I'm pretty sure that the way I've put things above illustrates some ignorance on my part, but still ... how hard can this be?  It's a reasonable thing to ask of a sophisticated compiler.

Questions like "How hard can this be?" are not only unconstructive, but they also belie your sincerity in resolving your self-claimed ignorance.

[Mars0i]

Thanks Nathan!  (See what I mean about my ignorance?)

On Friday, December 30, 2016 at 3:53:27 PM UTC-6, Nathan Davis wrote:

On Friday, December 30, 2016 at 10:13:49 AM UTC-6, Mars0i wrote:

I love Clojure.  But:  My number one complaint about it is that the compiler won't allow cyclic namespace dependencies.  I am not a compiler-writer, but as far as I can tell the only reason for this restriction is ... philosophical.  It forces bad code organization, and it wastes time.  Those who think that cyclic code is an evil are free to encourage its avoidance.  I'm happy to avoid it!  But sometimes I can't.  I absolutely hate the cyclic namespace restriction.  Drives me crazy when I am forced to encounter it.

First of all, there is no such restriction.  So there is no reason for it, philosophical or otherwise. 

What I wrote was based on my understanding from past conversations in this group and things I'd read, and on a fair amount of experience hitting my head against walls in different ways.  I'm happy to be disabused of my misunderstandings, even if they weren't completely baseless.

It is true that if you declare all your dependencies in ns forms, you will get an exception with cyclic namespaces.  That's a good thing -- the result probably wouldn't be what you want anyway.  However, there's no requirement that all dependencies be declared using ns -- you can use require at any time.

Well that's interesting.  I'll have to explore that option.  Thanks.
 

What is required is that all the vars a form uses exist when it is compiled.  The reason for this is purely technical -- resolving symbols to vars at compile time means less work (more efficiency) at run time.

There's more depth to that statement than I have at that this moment, I believe.  There's a lot that's resolved at run time in Clojure.

 

 

My latest case is that I'm trying to debug a weird Java interop problem--something that works in one app I've written, in which I ended up putting most of the application into one file, but doesn't work in the one I'm working on now.   And I had finally come up with a clever way of avoiding sticking most of the code into one namespace.  I'd worked around the conflict between Clojure's requirements and the requirements of the Java library I'm using.  But now there's this one problem, and I want to try an experiment to see whether it fixes the problem.  It looks like I'm going to have to pack all of the code into one file and rewrite everything that that necessitates just for this purpose. Aghh.

Perhaps if you provided a coherent description the specifics of your problem, someone would be able to suggest a satisfactory solution.

I understand the point, and would have provided a description if I thought it was worth the trouble for me or for readers of this group for a one-off experiment, and I didn't think there was a good solution, obviously.  I wasn't looking for that kind of help this time.

Given what you've said, though, I'll think about asking a more specific question about a cyclic dependency when it comes up again.  In the past when I encountered similar problems, I didn't find help other than "It's bad in general to organize programs non-hierarchically", or "Yeah, my solution is to have one slop namespace where I throw all of the things that have to refer to each other."  :-)

 

I have sometimes ranted a bit in this group, and in the end some of my rants were unreasonable.  I don't think this is unreasonable.  The cyclic dependency restriction is absurd.  I'm not trying to start a flame ware.  I feel strongly about this, and I'm frustrated.

What would be reasonable is asking questions about things until you understand them well enough to be in a position to make an informed opinion. 

I very much appreciate you clarifying some points that I thought I understood.  I did think I understood well enough to have the general opinion I expressed.  If you're suggesting that I didn't have a right to express an opinion given what I thought I understood, we can disagree, though what you wrote doesn't clearly imply that.
 

In this case:

1. Clojure does allow namespaces to refer to each other (just maybe not how you've tried).
2. There are plenty of technical considerations when it comes to mixing dynamic languages and cyclic references.  Cyclic namespaces are a subset of this problem.  So describing restrictions on cyclic namespaces (whether real or imagined) as "absurd" is absurd.

If you're right, then the point is well put.
 

A rant is just a rant, regardless of whether a problem actually exists.  If you feel strongly about something and only rant about it, don't be surprised when you feel frustrated. 

Not every problem is best solved only by accepting existing designs and working around them.  Sometimes it's worth suggesting, complaining, or criticizing.  It's not necessarily a bad thing for those able to change design to be aware of pain that design choices are causing.  Rants, when justified, are not necessarily useless.  Maybe "rant" isn't the right term here.  I was being glib in calling it that.

You're arguing that I'm mistaken to think that I'm suffering because of language design choices--that my "rant" was not justified--and I accept that you might be right and appreciate your clarifications.  I don't accept that it's always illegitimate to object with an emotional valence--but you didn't say that it was.
 

Frustration can be relieved by solutions. 

We agree.
 

If you don't understand the problem enough to present a solution, ask questions until you do.  Don't expect others to do all the work -- especially after ranting about it.

For the record, I think you misunderstood where I was coming from, who I am as a programmer, and what my level of experience is.  I don't need to be told that, as it happens.  But there's no reason that you would know, so I have no reason to be offended.

 

What I want: I'm not trying to make everyone sit around tapping their fingers while code to goes through multi-pass compliations.  All I'm asking for is a compiler option that will allow cyclic dependencies when requested.

I don't think multi-pass compilation would play well with things like macros or REPL-based development, so I doubt we'll have to worry about that.

Another interesting point.  Thanks. Have to think about it, or learn more, but if you're right, then perhaps that's a drawback of REPL-based dev, of which I am a big fan.

 

I'm pretty sure that the way I've put things above illustrates some ignorance on my part, but still ... how hard can this be?  It's a reasonable thing to ask of a sophisticated compiler.

Questions like "How hard can this be?" are not only unconstructive, but they also belie your sincerity in resolving your self-claimed ignorance.

 

I accept that the tone of that statement in the form of a question wasn't so helpful.  Apart from tone, the point I intended to make seemed reasonable: I figured that if Java can do it, why can't Clojure?  Your point about the REPL and macros seems relevant to that question.

Anyway, thanks again.

[Aaron Cohen]

You already know this has been discussed a lot over the years.

I just wanted to cite the best reasoning that I've seen from Rich about why Clojure does it the way that it does, which I believe is the argument that surrounded an blog post fro Steve Yegge that Rich responded to and some excellent discussion surrounding it on hackernews and the mailing list: https://news.ycombinator.com/item?id=2466731

On hackernews, Rich made a post that I believe is the best "last word" explaining his reasoning, which I'll quote here: 

The issue is not single-pass vs multi-pass. It is instead, what constitutes a compilation unit, i.e., a pass over what?

Clojure, like many Lisps before it, does not have a strong notion of a compilation unit. Lisps were designed to receive a set of interactions/forms via a REPL, not to compile files/modules/programs etc. This means you can build up a Lisp program interactively in very small pieces, switching between namespaces as you go, etc. It is a very valuable part of the Lisp programming experience. It implies that you can stream fragments of Lisp programs as small as a single form over sockets, and have them be compiled and evaluated as they arrive. It implies that you can define a macro and immediately have the compiler incorporate it in the compilation of the next form, or evaluate some small section of an otherwise broken file. Etc, etc. That "joke from the 1980's" still has legs, and can enable things large-unit/multi-unit compilers cannot. FWIW, Clojure's compiler is two-pass, but the units are tiny (top-level forms).

What Yegge is really asking for is multi-unit (and larger unit) compilation for circular reference, whereby one unit can refer to another, and vice versa, and the compilation of both units will leave hanging some references that can only be resolved after consideration of the other, and tying things together in a subsequent 'pass'. What would constitute such a unit in Clojure? Should Clojure start requiring files and defining semantics for them? (it does not now)

Forward reference need not require multi-pass nor compilation units. Common Lisp allows references to undeclared and undefined things, and generates runtime errors should they not be defined by then. Clojure could have taken the same approach. The tradeoffs with that are as follows:

1) less help at compilation time 2) interning clashes

While #1 is arguably the fundamental dynamic language tradeoff, there is no doubt that this checking is convenient and useful. Clojure supports 'declare' so you are not forced to define your functions in any particular order.

#2 is the devil in the details. Clojure, like Common Lisp, is designed to be compiled, and does not in general look things up by name at runtime. (You can of course design fast languages that look things up, as do good Smalltalk implementations, but remember these languages focus on dealing with dictionary-carrying objects, Lisps do not). So, both Clojure and CL reify names into things whose addresses can be bound in the compiled code (symbols for CL, vars for Clojure). These reified things are 'interned', such that any reference to the same name refers to the same object, and thus compilation can proceed referring to things whose values are not yet defined.

But, what should happen here, when the compiler has never before seen bar?

    (defn foo [] (bar))

or in CL:

    (defun foo () (bar))

CL happily compiles it, and if bar is never defined, a runtime error will occur. Ok, but, what reified thing (symbol) did it use for bar during compilation? The symbol it interned when the form was read. So, what happens when you get the runtime error and realize that bar is defined in another package you forgot to import. You try to import other-package and, BAM!, another error - conflict, other-package:bar conflicts with read-in-package:bar. Then you go learn about uninterning.

In Clojure, the form doesn't compile, you get a message, and no var is interned for bar. You require other-namespace and continue.

I vastly prefer this experience, and so made these tradeoffs. Many other benefits came about from using a non-interning reader, and interning only on definition/declaration. I'm not inclined to give them up, nor the benefits mentioned earlier, in order to support circular reference.

Rich

[Timothy Baldridge]

I've been programming Clojure for something like 6 years now, and yes, I've hit the cyclic dependency error a few times. How did I solve it? Each time via abstraction, and parameterization of functions. Most of the time this means writing a "interfaces.clj" file that contains all my defprotocols and defmulti statements, then I go and write namespaces that use these abstract interfaces. So instead of B requiring A directly it requires InterfaceA from the interfaces.clj file. 

Now for global data, the simple answer is to don't use global data. Instead of NamespaceA requiring def'd data from NamespaceB, just have all the functions in NamespaceA require a hash map of dependency data. Now everything is pure, and you can write NamespaceC that wires everything up. 

So the layout looks like this:

Interfaces.clj

             |

------------------------------------------------

|                              |

ImplementationA   Implementation B

|                              |

------------------------------------------------

         |

Orchestration (Setup) Namespace

I haven't encountered a problem yet that can't be solved with these methods. Maybe they will help you as well. 

Timothy

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Mark Engelberg]

I feel your pain.  I also run up against this time and time again and view it as a significant limitation -- one which often forces me to contort the structure of my Clojure programs into something less natural.  And as the Clojure language grows, the problem becomes even more acute.

For example, consider specs.

One common recommendation is to put your specs in a separate, parallel namespace.

However, implementing specs often requires domain-specific logic, i.e., functions from your main namespace.

And your main namespace often requires specs because the logic of your functions may need the specs in order to conform some data as part of its processing.

So what to do?  These kinds of things happen a lot in my experience, and a solution would be incredibly valuable.

Best solution I've found is to hoist mutually dependent things into a common namespace and use potemkin to clone them into a more logical partitioning.  But that's never been very satisfactory to me.

[Mark Engelberg]

On Fri, Dec 30, 2016 at 4:38 PM, Timothy Baldridge <tbald...@gmail.com> wrote:

So the layout looks like this:

Interfaces.clj

             |

------------------------------------------------

|                              |

ImplementationA   Implementation B

|                              |

------------------------------------------------

         |

Orchestration (Setup) Namespace

The problem I've had with this is that Implementations A and B are generally going to be records.  For best performance, you want to write the implementation of a record's protocols directly inside of the record.  If the implementation of A requires constructing a B, and implementing a B requires constructing an A, you've got a problem because a record's constructors aren't part of Interfaces.clj, but are part of the namespaces in which the records are defined.

[Timothy Baldridge]

I can see that, and even spec has this use case. In Spec it's solved by having both A and B in one namespace and using declare to forward-declare the constructors (or defns in this case).

So I guess the way I see it the tradeoff is a declare and all-in-one-namespace vs a massive complexity addition to the compiler and the redefinition of compilation units. The declare method seems like the cleaner route. 

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Engelberg]

On Fri, Dec 30, 2016 at 4:55 PM, Timothy Baldridge <tbald...@gmail.com> wrote:

I can see that, and even spec has this use case. In Spec it's solved by having both A and B in one namespace and using declare to forward-declare the constructors (or defns in this case).

So I guess the way I see it the tradeoff is a declare and all-in-one-namespace vs a massive complexity addition to the compiler and the redefinition of compilation units. The declare method seems like the cleaner route. 

I wonder whether there could be something like an `external-declare` that would satisfy Rich's concerns about knowing how to intern unencountered vars, while allowing cyclical references when needed.

 

[Mars0i]

Aaron, thanks for the RH quote.  I had looked at parts of that Yegge discussion at one point, I think, but I don't recall reading this section of Rich's remarks.  I never succeeded in understanding Common Lisp packages well enough to use the successfully.  I tried and gave up.  Rich may be pointing the reasons why.  Clojure namespaces, by contrast, are trivial to use in most situations, once you learn a few easy conventions.

tbc++ and puzzler, thanks for those useful strategies.  I'll keep them in mind.  potemkin looks potentially very useful.  I didn't know about it (or maybe encountered it when I had no use for it and so didn't remember it when I did).  In my case, I've only had to struggle with cyclic dependencies when using Java library that (almost) forces them (http://cs.gmu.edu/~eclab/projects/mason).  It's a great library for its purpose, but it's not entirely Clojure-friendly.  Or Clojure's not entirely friendly to this kind of library--take your pick.  I think I may have avoided the obvious cycle problem with this library, if everything works as I expect, however.

[Mars0i]

On Friday, December 30, 2016 at 7:59:46 PM UTC-6, puzzler wrote:

On Fri, Dec 30, 2016 at 4:55 PM, Timothy Baldridge <tbald...@gmail.com> wrote:

I wonder whether there could be something like an `external-declare` that would satisfy Rich's concerns about knowing how to intern unencountered vars, while allowing cyclical references when needed.

I was thinking that even if a compile option allowing cyclic dependencies is bad for pulling arbitrary code into the REPL, but you just would know the drawbacks if you used the option.  In my case I'm aot-compiling some of the primary namespaces anyway because of the Java lib's needs, so maybe that would be OK.

Maybe extern-declare would be better solution, though.

[squeegee]

The macros defined in ‘scarlett.core’ here: https://github.com/scgilardi/scarlett are an experiment along those lines.

scarlett

Provides macros to declare vars in namespaces other than *ns*

To be used sparingly to overcome otherwise cyclic namespace dependencies

Usage

  (ns-declare my-module startup shutdown)

declares my-module/startup and my-module/shutdown

  (declare+ module-a/startup module-b/startup)

declares module-a/startup and module-b/startup

They work by using “declare” after switching to the target namespace using “in-ns” (and switching back at the end). They have to be issued at the “top level” (as is highly recommended for ns, declare, def, etc.) to be effective and they contain code to notify with an exception if they are used at other than the top level. (The exception is preferable to silent failure or a confusing error message.)

The macros rely on the compiler behavior of handling top-level “do” forms in a special way: evaluating each of the contained forms sequentially at the top level, effectively removing the nesting and allowing each contained form to be compiled and evaluated before compiling the next one. Macro expansion facilities other than the compiler (e.g., in an editor or debugger) may not duplicate that subtle behavior so the expansions they produce may not accurately reflect the expansion that will be produced and seen by the compiler. There may be contexts where that causes trouble at development time.

I’m not aware of these being used anywhere in other than experimental/POC code.

—Steve 

[Timothy Baldridge]

Be really careful with Potemkin. I've had a lot of build issues (especially around AOT) with that library. I'll try to put this as kindly as I can...it's a bit of a hack that leverages some undocumented aspects of the Clojure API. As such it's been the source of some really weird compile errors that have taken me many hours to debug. So much so that some libraries have go so far as to copy-paste certain files from potemkin (macros around map-like type creation), in order not to get the more fragile parts of the library (var reorganization). 

In short...the language wasn't designed to do what Potemkin tries to make it do when it comes to vars. I really recommend against using the library. 

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Colin Fleming]

As a counterpoint to this, note that ClojureScript does the same thing (for different reasons, but the same mechanism):

https://github.com/clojure/clojurescript/blob/master/src/main/clojure/cljs/core.cljc#L91-L101

[William la Forge]

I think Timothy Baldridge had a great answer. For example, I often have records which use each other's constructors. But you can put your factoy methods in a map of dependencies.

I generally have a build function which takes a hash map as an argument, associates the various data and functions that other modules depend on, and returns the updated hashmap. Each module has such a function and at the top level I build the composite map with the modules I'll be using.

Not only does this eliminate those nasty cyclic dependencies, it allows me to easily swap different implementations by just changing the choice of build functions to call at the top level.