Feature expressions - just because?

[Alan Dipert]

Hi all,

I was a little surprised and unenthused to learn recently from Alex Miller [1] that Feature Expressions [2] are expected in 1.7.

I was surprised because I have yet to understand why they're needed, after using Clojure and ClojureScript together for a long time, and after reading the design document.  I was unenthused because the only thing clear to me about them is that they introduce syntactic complexity, which is probably the last thing I'd want for my favorite Lisp, and language.

My hope for any change to the syntax of Clojure would be that behind it are strong advocates with measured arguments, and I suspect feature expressions have both, I just don't know them yet.  With or without feature expressions Clojure is so awesome I will of course defer to Alex and Rich/Cognitect as steward and benefactor, but I like to think that by registering my disagreement I help deliver on the reason Clojure was open-sourced in the first place :-)

What I don't understand is: what do feature expressions give us that cljx-like tools don't, when combined with the things - like system properties and macros - that platforms and languages already do?  The design page enumerates how various problems can be solved with both cljx and feature expressions, but speaks little to things only solved by the presented design.  This leads to my feeling of "they're happening just because, here's exactly how."

The complexity I see them introducing is very large, compared to the benefits I can understand them bringing, which seem small.  First, it's another way to mix read and run time.  Because feature expressions (like data readers) use syntax instead of S-expr to convey sequence, they also introduce new questions of precedence, like the one Herwig Hochleitner raises [3].  Precedence conundrums - in Lisp.  Seems like a bad sign, no?

Colin Fleming explores a additional implications in the next comment [4].  In essence, read-time conditionals make Clojure code more difficult for tools to understand without running.  One advantage of cljx-style preprocessing is that tooling doesn't need to care about understanding how to recognize features for other platforms, because none of the code being read is necessarily being run at pre-processing time.  Transformations are "by a 3rd party".  With feature expressions, tools need to at least recognize the feature/platform boundaries, and it's not clear this is possible in all cases.

As for the runtime *features* component that cljx doesn't offer - are there examples of this being used to solve problems that a combination of system properties and macros can't?  While Clojure is now written for multiple platforms, there is little variation per platform - certainly not as much variation as once existed between competing, proprietary Common Lisp implementations in the 80s, which I think was the original use case.  Clojure and its family of open, noncompeting implementations, divided between platforms for platform affordances, not competing language feature/implementations, seems like a totally different use case.

I really like that Clojure's relative syntactic and evaluation simplicity give tools and editors a lot of room to innovate.  As co-author and maintainer of various build tools and multi-platform projects, I am afraid that feature expressions will ultimately remove flexibility and complicate my life instead of empowering me to go further and faster.  I could also be totally wrong and my fear completely unfounded.

Alan

1. https://github.com/cognitect/clojure-conj/wiki/Clojure-conj-2014-Unsessions#feature-expression-overview

2. http://dev.clojure.org/display/design/Feature+Expressions

3. http://dev.clojure.org/display/design/Feature+Expressions?focusedCommentId=8192107#comment-8192107

4. http://dev.clojure.org/display/design/Feature+Expressions?focusedCommentId=9928713#comment-9928713

[Timothy Baldridge]

" I am afraid that feature expressions will ultimately remove flexibility and complicate my life instead of empowering me to go further and faster.  I could also be totally wrong and my fear completely unfounded." 

Which implementation (of the four) are referring too? I have many of the same concerns as you, except the last proposal (cross platform file extensions) seems to deal with most of these issue. From my understanding this approach would define a new file extension that would designate that file as being cross-platform. The only change needed by any tools at that point would be to say "on the JVM load .clj and .cljp" (for portable files). On CLJS "load .cljs or .cljp files". 

Seems pretty simple to me, but then again I'm biased, as I've been advocating that approach for years. 

As to "why not cljx", I'd say that it'd be nice to not have that feature tied to a 3rd party tool, seems like such a simple change would fit into Clojure's core nicely. 

Timothy

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Alan Dipert]

I'm invested most in Clojure (JVM) and ClojureScript, so it's mostly applications involving these two that color my sensibilities.

I see the fact that the feature is currently provided by a 3rd party tool, to the satisfaction of most, (all?) as a credit to the current, relative simplicity of Clojure's file conventions, syntax, and evaluation model.

What tools like cljx will emerge in the future, I can't say.  What I can say is that adding syntax to Clojure today will definitely make them harder, and maybe even impossible, to build.

This is why I'm not sure it's necessarily nice - without strong advocacy, or my own enlightenment - for this feature to find itself in the core.

It seems like something that most people are comfortable enough with (or unfamiliar enough with the problem) to not object to, but that could have a longer-term, undoable, highly detrimental impact on our emerging ecosystem of tools.  This is as strongly as I can put my argument, because my burden - to produce concrete examples of tools that are impossible to make if we do this - is also impossible.  I just have a feeling.  That, and I don't see anything close to simple about the proposal, but that is an even purer matter of taste.

Alan

[Alex Miller]

On Friday, November 14, 2014 11:02:54 AM UTC-6, Alan Dipert wrote:

Hi all,

Hi Alan!

 

I was a little surprised and unenthused to learn recently from Alex Miller [1] that Feature Expressions [2] are expected in 1.7.

I was surprised because I have yet to understand why they're needed, after using Clojure and ClojureScript together for a long time, and after reading the design document.  I was unenthused because the only thing clear to me about them is that they introduce syntactic complexity, which is probably the last thing I'd want for my favorite Lisp, and language.

My hope for any change to the syntax of Clojure would be that behind it are strong advocates with measured arguments, and I suspect feature expressions have both, I just don't know them yet.  With or without feature expressions Clojure is so awesome I will of course defer to Alex and Rich/Cognitect as steward and benefactor, but I like to think that by registering my disagreement I help deliver on the reason Clojure was open-sourced in the first place :-)

What I don't understand is: what do feature expressions give us that cljx-like tools don't, when combined with the things - like system properties and macros - that platforms and languages already do?  The design page enumerates how various problems can be solved with both cljx and feature expressions, but speaks little to things only solved by the presented design.  This leads to my feeling of "they're happening just because, here's exactly how."

The best statement I have of the problem to be solved is what's on the page: "There are Clojure dialects (Clojure, ClojureScript, ClojureCLR) hosted on several different platforms. We wish to write libraries that can share as much portable code as possible while leaving flexibility to provide platform-specific bits as needed, then have this code run on all of them."

The need is backed up by relatively common use of cljx and is reiterated as one of the most common themes in what ClojureScript users are looking for in the last state of clojurescript survey. With the recent developments around Arcadia and .NET, we might be poised for increasing usage of ClojureCLR too, which is currently 
"porting" much of the JVM Clojure world via copying.

The current feature expressions proposal/patch set has the following advantages over cljx:

- Feature expressions will be in the reader, not a preprocessor. This simplifies some aspects of the tool chain and dev tooling.

- Cross-platform jars can be created that contain a single portable file, not per-platform versions of each file.

- The new common extension (.cljc) means that it becomes possible to ship both platform-specific and cross-platform fallback versions of a namespace.

- A "standard" way to do portability

I think the major tradeoffs in feature expressions vs cljx are:

- More complicated reader

- The tooling-related issues like those mentioned by Colin

- Ad-hoc portability 

- Unforeseen consequences :) 

 

The complexity I see them introducing is very large, compared to the benefits I can understand them bringing, which seem small.  First, it's another way to mix read and run time.  Because feature expressions (like data readers) use syntax instead of S-expr to convey sequence, they also introduce new questions of precedence, like the one Herwig Hochleitner raises [3].  Precedence conundrums - in Lisp.  Seems like a bad sign, no?

I don't think the example given in [3] is supported or intended to be supported by the current design. Specifically, I expect that any expression in a feature expression is readable by the reader. Reader literals are a bit of a special case and while a reader literal may not be *understood* on a platform it should be syntactically readable. I'm probably not using exactly the right words for what I'm trying to say.

 

Colin Fleming explores a additional implications in the next comment [4].  In essence, read-time conditionals make Clojure code more difficult for tools to understand without running.  One advantage of cljx-style preprocessing is that tooling doesn't need to care about understanding how to recognize features for other platforms, because none of the code being read is necessarily being run at pre-processing time.  Transformations are "by a 3rd party".  With feature expressions, tools need to at least recognize the feature/platform boundaries, and it's not clear this is possible in all cases.

I agree that there are tooling tradeoffs here. I'm hoping to have a release and some early feedback re tools before we get to final release.

 

As for the runtime *features* component that cljx doesn't offer - are there examples of this being used to solve problems that a combination of system properties and macros can't?  While Clojure is now written for multiple platforms, there is little variation per platform - certainly not as much variation as once existed between competing, proprietary Common Lisp implementations in the 80s, which I think was the original use case.  Clojure and its family of open, noncompeting implementations, divided between platforms for platform affordances, not competing language feature/implementations, seems like a totally different use case.

I think this is a question mark with respect to usefulness myself. I would like to hear more, particularly if this solves a problem you can't solve otherwise. I can envision many theoretical solutions but perhaps in reality it would never be used and it's not worth doing.

 

I really like that Clojure's relative syntactic and evaluation simplicity give tools and editors a lot of room to innovate.  As co-author and maintainer of various build tools and multi-platform projects, I am afraid that feature expressions will ultimately remove flexibility and complicate my life instead of empowering me to go further and faster.  I could also be totally wrong and my fear completely unfounded.

I really appreciate the feedback. I don't think you'll be there unfortunately, but there will be an unsession at the conj where I would like to hear more feedback and ensure we're understanding the tradeoffs we're making and creating the best solution.

Alex

[Alex Miller]

On Friday, November 14, 2014 3:05:13 PM UTC-6, Alan Dipert wrote:

It seems like something that most people are comfortable enough with (or unfamiliar enough with the problem) to not object to, but that could have a longer-term, undoable, highly detrimental impact on our emerging ecosystem of tools.  This is as strongly as I can put my argument, because my burden - to produce concrete examples of tools that are impossible to make if we do this - is also impossible.  I just have a feeling.  That, and I don't see anything close to simple about the proposal, but that is an even purer matter of taste.

I would turn this on its head and ask whether not having a standard way to create portable Clojure libraries across dialects is having a detrimental impact on our ability to nurture a consistent emerging ecosystem of libraries now?  

I don't feel like I have strong answers to either question really, just proposing another way to look at it.

Alex

[Alan Dipert]

Hi Alex, thanks for the detailed response.

Your inversion is apt, but I would say: because code is data, and because Clojure code can be read (except data literals) without any special context or preparation, the bar couldn't be lower for preprocessors and macros that serve portability needs.  cljx emerged and has become popular because the problem it solves is a popular one, and because Clojure is data it could solve the problem in a robust way.

Yes, Henry Ford gave them a car when they asked for a faster horse.  But for a lot of the problems the feature expression work seems to address, I haven't seen anyone even ask for a faster horse yet.  So, to answer your question, I would say: no, from what I've heard, nobody is hurting for this or anything like it.  If they were, we would have seen tools.  I could also be living in a bubble, and this is actually something we desperately need to move to the next level of libraries.

Alan

[Timothy Baldridge]

See anything interesting here? https://medium.com/@facorro/clojure-survey-2014-6896bc2b28e1

Timothy

--

You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

[Francis Avila]

I am a little surprised that people want feature expressions instead of either platform-agnostic libraries or more core-blessed abstractions to make code cross platform.

Long before cljx pedestal-app had a "platform" namespace of functions which was simply reimplemented some common functions (e.g., for string manipulation, dates, etc) in both clojure and clojurescript. I think Peter Taoussanis's "encore" library has a similar purpose (but of course it uses cljx and has a lot more extra stuff in it).

Libraries like these and the abstractions they provide are absolutely necessary to make any non-trivial clojure namespace truly cross-platform because so much very basic functionality is not in Clojure core. Do we really expect people to write a different platform-specific string-to-date parsing expression for every platform? What about platforms they are not familiar with? (E.g., I don't know how to parse a string to an int in ClojureCLR.) The only reason this is necessary is because of Clojure's philosophy of leaving a lot of basic types and operations completely up to the platform and providing no abstraction on top. I really do mean basic, like doing stuff with strings and dates. And sometimes when Clojure does provide an abstraction (e.g. regexes), it differs significantly across platforms and limits the usefulness of the abstraction.

Feature expressions in core doesn't particularly upset me, but I don't understand why people clamor for them because they would be better served by building the cross-platform abstractions missing from Clojure. Perhaps it's because this task is much more difficult?

On Friday, November 14, 2014 11:02:54 AM UTC-6, Alan Dipert wrote:

[Alex Miller]

On Fri, Nov 14, 2014 at 6:02 PM, Francis Avila <fav...@breezeehr.com> wrote:

I am a little surprised that people want feature expressions instead of either platform-agnostic libraries or more core-blessed abstractions to make code cross platform.

Maybe "feature expressions" is a phrase that has come to mean "write portable code across Clojure dialects" in place of whatever the actual feature is. I can't really say. It would be great to hear from ClojureScript users that want "feature expressions" whether they really expect "essentially what cljx has but in the reader" or something else. 

Perhaps on the survey next year, a better question would be: "what problem is the most important to solve?".

However, I don't think that feature expressions are mutually exclusive of language abstractions or library assistance to make code portable. 

 

Long before cljx pedestal-app had a "platform" namespace of functions which was simply reimplemented some common functions (e.g., for string manipulation, dates, etc) in both clojure and clojurescript. I think Peter Taoussanis's "encore" library has a similar purpose (but of course it uses cljx and has a lot more extra stuff in it).

This idea is definitely of interest and I tried to outline this path a bit under "Case 3: Library designed for open extension" in the Feature Expressions page. I don't think there is any reason this can't be done now (as in the libraries you mention). The new portable file extension allows the existence of a common stub or implementation library for compilation, which I think is useful.

We have had some discussions around this idea but I don't know whether it would be worth creating abstract namespaces around things like io/strings/math/random/datetime/etc. That's a very different kind of effort and while the easy parts are easy, the hard parts are really hard. 

[Colin Fleming]

This is a change I feel pretty strongly about, but unfortunately I don't have a lot of time to do the discussion justice right now. It's clearly something that's required, although I would also be very interested to know why CLJS users feel that CLJX isn't sufficient for them and this has to be in core - I haven't seen much about this.

I'd also be very interested to know more about the use cases leading to some of the proposed features that I think make this change much more complicated, namely allowing user-defined features and allowing boolean expressions with combinations of features. Realistically, I'll never be able to do anything useful with #+(clj or clj.net or (cljs and node)) for some server-side functionality. As a side note, the design page states that the proposed syntax is the same as CLJX, but AFAIK CLJX doesn't support these boolean combinations, and I'd be interested to hear what this is holding people back from doing.

If we're going down this road, I'm a huge fan of the explicit cljc extension - having the clj extension suddenly mean that it might actually contain CLJS or CLR code is really going to make things difficult, for me at least.

However, I don't think that feature expressions are mutually exclusive of language abstractions or library assistance to make code portable.

I agree with this, but the problem is that if feature expressions come first then they'll immediately be used in place of these abstractions or libraries, and I think it's considerably less likely that these would happen. And even if they do eventually happen we'll still be stuck supporting feature expressions indefinitely.

I'm looking forward to the unsession :-)

[Colin Fleming]

BTW I also attempted to solicit some feedback from the CLJS list here: https://groups.google.com/d/topic/clojurescript/DVSjWNs4V84/discussion

[Alex Miller]

Thanks....

[Julien]

Hi,

I am personally very much in favor of having something like Feature Expressions being standardized and as integrated as possible. CLJX is just generating files blindly and this has all sorts of nasty consequences in a number of tools.

A very concrete example of this is a recent release of leiningen which broke all libraries using CLJX with the default configuration [1]. Another one is the various hacks needed to have a repl running. The fact that CLJX is in a dormant stage doesn't help either.

[1] https://github.com/lynaghk/cljx/issues/51

Julien

[Micha Niskin]

Hi friends,

What follows are my own (admittedly naive) opinions, formed over the last few years. I have a great deal of time invested in Clojure tooling projects, projects with ambitious goals (ambitious in the age of OOP, but laughable in comparison to what the old Lisps had). Why is tooling so hard in Clojure? I believe it's because of the complexities introduced in the reader with each new release.

The powerful tooling of the beforetimes leveraged code-is-data and minimized the importance of files. In Clojure we have the opposite: yes, we're stuck with files as the canonical representation but Clojure source files can't even be read by the reader without an evaluation context. For example `::foo/bar` requires filthy hacks to sniff namespace declarations and the evaluation semantic of collection literals like `#{0 0}` vs `(hash-set 0 0)` and `{:x}` vs `(hash-map :x)` complect reading and evaluation. And don't even get me started on tagged literals :) Combine these complexities with mutable namespaces and expression-at-a-time evaluation and it's a catch-22 situation for tooling, because it's very difficult to safely evaluate so you can read.

I can't say concretely what feature expressions will make impossible, and no matter what I say there are more sophisticated minds who can suggest techniques for coping with complexity that are beyond my expertise and for which I can't form a rebuttal without actually coding a counter example, so I won't speculate. I do, however, have a great deal of experience with boots-on-the-ground Clojure tooling development, and I can emphatically assert that a simpler reader would result in vastly better tooling. Feature expressions, from my naive workmanlike perspective, are yet another step toward "easy" and away from "simple", and I have not yet seen a compelling argument in favor of the monolithic, everything-in-the-reader approach over layered tooling.

Now let's have a group hug. I love Clojure and I'm grateful to Alex for all the work you do that makes Clojure viable in the real world. I can say without exaggeration that I would have left this industry long ago if I hadn't found Clojure. So big ups, thanks for improving the quality of the last 3-4 years of my life.  

[Reid McKenzie]

I'll chime in here briefly.

In short, agree with Micha and Tim. I think that trying to enshrine CLJX
or something within epsilon of it in core is more or less asking for
trouble due to the unreadable symbols problem discussed at length on the
design wiki. I think that Tim's proposal to divide host-dependent code
among multiple conditionally loaded files on the source path solves this
issue elegantly by sweeping unreadable forms under the rug and silently
providing the sort of open dispatch host extension that Francis was
talking about.

Consider the case of a library providing an "API" namespace, and then a
collection of platform postfixed files providing alternate
platform-dependent implementations of the API. This requires no
additional support in the reader, the addition of no "otherwise
unreadable symbol" syntax, is trivially extensible by new "Common
Clojure" implementers and requires no effort from existing platform
maintainers to support targeting new platforms. Due to the unreadable
forms problem on the other hand, this would not be the case for host
expressions as all "Common Clojure" implementations wishing to be able
to share libraries must be able to read and ignore each-other's host forms.

I think that having an official vehicle for allowing single libraries to
go cross-platform is critical especially as ClojureScript takes off
forget other efforts like Pixie, Kiss, Oxlang or
$YET_UNBORN_CLOJURE_DIALECT. The more leverage we can offer programmers
in terms of being able to learn Clojure and be able to plug it in where
needed as a thin wrapper around $HOST bringing in familiar libraries the
better.

If we can promote this sort of flexibility with minimal complexity ala
host files, that's just gravy.

Reid

[Alex Miller]

Can you be specific what you are referring to by "Tim's proposal to divide host-dependent code among multiple conditionally loaded files"? I either missed that or haven't read it. What does "platform postfixed files" mean exactly? Who is responsible for choosing those and when? When you say "Common Clojure" implementers, do you mean implementors of Clojure/ClojureScript/ClojureCLR or implementers of libraries or something else?

We walked a bit down the design path for a library aliasing mechanism but Rich ultimately decided that it made loading more complicated without giving you anything you didn't already have if you just set up your classpath properly. 

It sounds like this path ignores the (fairly common) case of namespaces that have *almost* but not exactly, the same implementation on different platforms. It seems to me that there are three levels here:

1) Language abstractions that can be implemented in common across dialects (high bar for implementation but high benefit once you get the abstraction right - an example gap right now is an abstraction for "catch all" exceptions)

2) Library abstractions that can buffer portable code from platform specific differences in things like math, strings, date/time, random, urls, etc. Ideally any "standard" library abstractions would need to be defined as standard to foster this approach (implementations might not).

3) Everything else - whatever is not covered above.

Ideally we want these problems to be solved first at #1, then at #2, and only last resort in #3. 

Your proposal seems to be focusing in the area of #2. Feature expressions mostly focuses in #3. These are not necessarily mutually exclusive. Could some combination of #1/2 cover some or all of needs today? Possibly, but I doubt it.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.

[Nicola Mometto]

I agree with Alex that the big value of an approach like feature
expression is in the common case where code is almost the same between
the clojure/clojureCLR impl and the clojurescript one but differs
only slightly.

In this regard I want to point out that in my experience a big chunk the
"mostly the same but slightly different" code differs only for type
names, and this could be avoided without feature expressions at all, by
simply:
- providing a complete set of predicate functions for all the literal
types and clojure/cljs interfaces/protocols (e.g. `obj?`,
`reference?`, `regex?`, `ex-info?`)
- automatically importing the clojure.lang interfaces so that extending
a type to "IPersistentMap" will pick clojure.lang.IPersistentMap for
clojure and clojure.core/IPersistentMap for cljs, or catching
ExceptionInfo will use clojure.lang.ExceptionInfo for clojure and
clojure.core/ExceptionInfo for clj (extending :refer-clojure :exclude
to unmap automatically imported classes would probably be helpful)

Some existing tickets that are also necessary in addressing this:
http://dev.clojure.org/jira/browse/CLJ-1293
http://dev.clojure.org/jira/browse/CLJ-1232

Nicola

>> email to clojure-dev...@googlegroups.com.

>> To post to this group, send email to cloju...@googlegroups.com.
>> Visit this group at http://groups.google.com/group/clojure-dev.
>> For more options, visit https://groups.google.com/d/optout.
>>

--

[Brandon Bloom]

The powerful tooling of the beforetimes leveraged code-is-data and minimized the importance of files. In Clojure we have the opposite: yes, we're stuck with files as the canonical representation but Clojure source files can't even be read by the reader without an evaluation context.

It has never been the case that Clojure code files were "data" in the sense you want. Clojure source has always been a superset of Edn. If you want to provide *Clojure* tooling, as opposed to *Edn* tooling, you have to consider contexts and evaluation rules. It's just the superficial similarity that gives you that "if only..." feeling.

 

For example `::foo/bar` requires filthy hacks to sniff namespace declarations and the evaluation semantic of collection literals like `#{0 0}` vs `(hash-set 0 0)` and `{:x}` vs `(hash-map :x)` complect reading and evaluation. And don't even get me started on tagged literals :)

Data structure literals, including tagged literals, provide this important distinction due to phase separation. Once you have macros (as opposed to something more like fexprs), you've got the problem of getting data of the right types to those macros as arguments. Without reader-knowledge of this literals, every macro would have to be capable of interpreting `(hash-set 0 0) on their own!

 

Combine these complexities with mutable namespaces and expression-at-a-time evaluation and it's a catch-22 situation for tooling, because it's very difficult to safely evaluate so you can read.

I think that, given what you've rediscovered, there's an opportunity with tools.reader to make tooling easier: tools.reader can (and I believe should) grow and standardize extra data types for representing ::style/keywords, uninterpreted tagged literals, and yes, unresolved feature expressions.

In summary, we can't put the genie back in the bottle. Instead of forcing Clojure to be less-different-than Edn, we should admit that they are in fact different. Let's standardize the runtime interpretation of Clojure's syntax as data, so tools can interoperate on that superset of Edn's syntax and types.

[kovas boguta]

Many good points made by people in this thread. I largely agree with Micha, including gratitude for the people improving Clojure. Let me add my 3 cents.

1. IMHO a solution to the problem feature expressions is targeting should only happen if there is a solid design supported by the community. The downsides of getting it wrong are big. The cost of waiting, not that big. 

2. Source cool tooling is a bit of a PITA as others have mentioned. While I agree we shouldn't be making the problem worse, a conservative approach to evolving Clojure syntax doesn't address the root problems.

The root problems are 

A) Safely parsing the source

B) Getting a canonical representation for it 

I believe these are solvable problems. As an example of a possible approach, see my codn library (https://github.com/kovasb/codn) which separates parsing from "reading". The continuation of this approach would be things like a symbolic interpreter or partial evaluator.

Immutable namespaces would also be a very powerful tool, for this and many other applications. We don't need it to be default or even ship with Clojure. If there was a reasonable a la carte solution here, tooling people could just use it without impact on the average app. 

3. The proposed design.

Its hard to look at cljx and see something other than a hack. Fine for a library filling a void, glad it exists, but not something I'm happy with in core. 

Its putting the construct, and a particular form of programmer convenience, above the artifact. Its certainly not making anything simpler.

Even wrt programmer convenience, it has major problems. It sucks to read. The more it is used in a project, the more unreadable it gets. I challenge people look at some cljx heavy code and tell me it looks clean. 

The cljx "code smell" comes from the fact that it is difficult to abstract. It would take a heroic (and ill-advised) macro effort to remove the idioms you are forced to repeat. 

So what's a better idea? Instead of finding ways to complect maybe we should be finding what to make simple. Namespaces. Reading. Separating platform-specific turns of phrase from logic. Theres a lot of work and ideas in this space, I don't think we need to settle for something we are likely to regret. 

[Alex Miller]

On Sun, Nov 16, 2014 at 5:20 PM, kovas boguta <kovas....@gmail.com> wrote:

Many good points made by people in this thread. I largely agree with Micha, including gratitude for the people improving Clojure. Let me add my 3 cents.

1. IMHO a solution to the problem feature expressions is targeting should only happen if there is a solid design supported by the community. The downsides of getting it wrong are big. The cost of waiting, not that big. 

I will note that there have been serious pushes in this direction prior to my involvement in this role and it was ultimately put off. It's a visible issue in every feature/pain poll. It slows down dev on things like core.async. I do not want to wait; I want to actively solve the problem.

 

2. Source cool tooling is a bit of a PITA as others have mentioned. While I agree we shouldn't be making the problem worse, a conservative approach to evolving Clojure syntax doesn't address the root problems.

I think the points here are good, but a tangent.

3. The proposed design.

Its hard to look at cljx and see something other than a hack. Fine for a library filling a void, glad it exists, but not something I'm happy with in core. 

I'm not going to judge that. I don't know that I get the same sense from it's long history in Common Lisp (but I am admittedly not in or from that community).

 

Its putting the construct, and a particular form of programmer convenience, above the artifact. Its certainly not making anything simpler.

Even wrt programmer convenience, it has major problems. It sucks to read. The more it is used in a project, the more unreadable it gets. I challenge people look at some cljx heavy code and tell me it looks clean. 

I think this is an argument for more portable abstractions and constructs, which is useful regardless, but it's unclear to me whether that's enough to actually achieve the goal.

[Alex Miller]

I think these are constructive concrete suggestions and I'm very interested in investigating it.

Perhaps in tandem with some library abstractions for common areas, we could deliver enough of these to forestall needing feature expressions at all or scale it back significantly. I will do some more analysis.

Alex

[Brandon Bloom]

Perhaps in tandem with some library abstractions for common areas, we could deliver enough of these to forestall needing feature expressions at all or scale it back significantly. I will do some more analysis.

Even if it doesn't forestall feature expressions, it's probably a good idea to tackle the most common cross-platform discrepancies either before or simultaneously with feature expressions. If feature expressions come out first, they'll see a whole bunch of use that will immediately become technical debt when the things Nicola mentioned are addressed.

[Reid McKenzie]

On 11/15/14 12:39, Alex Miller wrote:
> Can you be specific what you are referring to by "Tim's proposal to
> divide host-dependent code among multiple conditionally loaded files"?

After rereading the design wiki you're right this proposal doesn't occur
cleanly/clearly on the wiki. I'll fix that in a bit.

The idea is that "we" introduce a ".cljc" or some other "common" clojure
file extension. These "common" files may contain no host interop forms,
and as such are expected to be portable across all Clojure platforms
with zero change. This covers the common case of Clojure code that I've
seen where application & library code is built in Clojure for Clojure.

Code with host interop is then segregated into platform specific
namespaces. So for CLJS we have ".cljs", for CLJCLR perhaps ".cljn" and
for CLJ we just keep ".clj".

On a given platform, load first attempts to read and compile _only_ the
target path as a platform file for the current platform. If that file
does not exist, then load falls back to the "common" file extension and
finally fails to load.

> What does "platform postfixed files" mean exactly?

src/foo/core.cljc <- common
src/foo/core.cljs <- CLJS specific
src/foo/core.clj <- CLJ specific
src/foo/core.cljn <- CLJCLR specific

So (load "src/foo/core") X not in #{"clj" "cljs" "cljclr"} would hit
src/foo/core.clj. Otherwise the platform file is hit.

Note that this provides the implicit expectation that all existing
Clojure code was written targeting CLJ which is a correct if portability
wise minimal default.

> Who is responsible for choosing those and when?

The dialect implementer in load at load time.

> When you say "Common Clojure" implementers, do you mean implementors of
> Clojure/ClojureScript/ClojureCLR or implementers of libraries or
> something else?

The language implementers.

> We walked a bit down the design path for a library aliasing mechanism
> but Rich ultimately decided that it made loading more complicated
> without giving you anything you didn't already have if you just set up
> your classpath properly.

That presumes you have a merge based class or resource path rather than
a shared source tree.

Strictly you're right, if you had

src/clj/...
src/cljs/...
src/cljc/...
src/cljn/...

and then used the .clj file extension everwhere that would indeed buy
you the same kind of code partition I'm talking about. Now this becomes
a usability and tooling issue rather than a question of language extension.

> It sounds like this path ignores the (fairly common) case of namespaces
> that have *almost* but not exactly, the same implementation on different
> platforms.

Yes and no. If you are within epsilon of having identical code across
multiple platforms, it is my opinion (and I will stress opinion as I
have no use study to back this up) that it should be trivial to factor
such code so as to isolate platform interop code into platform specific
functions which become candidates for a host file providing a shared
host interface across platforms.

In this structure your shared and platform independent code is now
decoupled from the code that interacts with the host platform. This is a
very traditional structure by which C and other languages achieve cross
platform code sharing by having a common interface wrapping the target
platform.

> It seems to me that there are three levels here:
> 1) Language abstractions that can be implemented in common across
> dialects (high bar for implementation but high benefit once you get the
> abstraction right - an example gap right now is an abstraction for

> 2) Library abstractions that can buffer portable code from platform
> specific differences in things like math, strings, date/time, random,
> urls, etc. Ideally any "standard" library abstractions would need to be
> defined as standard to foster this approach (implementations might not).
> 3) Everything else - whatever is not covered above.
>
> Ideally we want these problems to be solved first at #1, then at #2, and
> only last resort in #3.

Agreed.

> Your proposal seems to be focusing in the area of #2. Feature
> expressions mostly focuses in #3. These are not necessarily mutually
> exclusive. Could some combination of #1/2 cover some or all of needs
> today? Possibly, but I doubt it.

Perhaps. But I think that involves a serious push to hide a _lot_ of the
implementation details of Clojure on its various platforms by
introducing say a common exception abstraction which may need special
implementation support rather than just punting to the OS. Having a
Clojure call stack abstraction would be nice as well in this vein. These
sort of changes are something that will not happen overnight or even for
1.7. Feature expressions are on the table now and we should consider the
extent to which the enable and encourage such the use of host portable
abstractions.

The answer is they don't. Host expressions are simply a preprocessor
hack to work around the absence of 1 and 2 by enabling library
implementers or 3 to do without. As such I think it's fine for user land
tooling but are tool which will not serve us well in the long run should
it be admitted to core.

In the short term, CLJX works. It's not nice, clearly it's not optimal
as the user surveys show but it gets the job done. If users want to use
CLJX, then I propose they can use CLJX to generate platform files if
they so choose. Or they can just write platform files, since being
platform files there should be near zero shared code if it is well
factored. That becomes a tooling choice. We as a language punt on the
CLJX question, encourage/force some better software engineering via host
files and separating host independent code from host dependent code and
then in future releases strive to resolve any friction points in this
approach.

To date, Clojure has always superset the previous release's reader
syntax. If we add host expressions to the reader we'll be stuck with
them effectively forever as well as all the complexity that they bring
thanks to the unreadable forms problem. Host files are literally simple.
They serve to (forcibly) decomplect host dependent code from host
independent code. Implementing them is also trivial and requires no
reader changes. They should enable people to get the job done. If they
do not, then that is a failing of the various Clojure platforms in
regards to your 1 and 2 which we should strive to resolve.

Reid

[Alex Miller]

On Mon, Nov 17, 2014 at 10:47 AM, Reid McKenzie <rmcke...@gmail.com> wrote:

On 11/15/14 12:39, Alex Miller wrote:

Can you be specific what you are referring to by "Tim's proposal to
divide host-dependent code among multiple conditionally loaded files"?

After rereading the design wiki you're right this proposal doesn't occur cleanly/clearly on the wiki. I'll fix that in a bit.

At the moment, I'd prefer to be the one making any changes to the feature expression page, just because of where we are in the process.

 

The idea is that "we" introduce a ".cljc" or some other "common" clojure file extension. These "common" files may contain no host interop forms, and as such are expected to be portable across all Clojure platforms with zero change. This covers the common case of Clojure code that I've seen where application & library code is built in Clojure for Clojure.

This is already part of the proposal and included in the patches available.

Code with host interop is then segregated into platform specific namespaces. So for CLJS we have ".cljs", for CLJCLR perhaps ".cljn" and for CLJ we just keep ".clj".

On a given platform, load first attempts to read and compile _only_ the target path as a platform file for the current platform. If that file does not exist, then load falls back to the "common" file extension and finally fails to load.

This is already the implementation in the patches for CLJ and CLJS.

 

> What does "platform postfixed files" mean exactly?

src/foo/core.cljc <- common
src/foo/core.cljs <- CLJS specific
src/foo/core.clj  <- CLJ specific
src/foo/core.cljn <- CLJCLR specific

So (load "src/foo/core") X not in #{"clj" "cljs" "cljclr"} would hit src/foo/core.clj. Otherwise the platform file is hit.

Note that this provides the implicit expectation that all existing Clojure code was written targeting CLJ which is a correct if portability wise minimal default.

 

We walked a bit down the design path for a library aliasing mechanism
but Rich ultimately decided that it made loading more complicated
without giving you anything you didn't already have if you just set up
your classpath properly.

That presumes you have a merge based class or resource path rather than a shared source tree.

Clojure and ClojureScript both use the JVM classpath which is a merged classpath.

 

Strictly you're right, if you had

src/clj/...
src/cljs/...
src/cljc/...
src/cljn/...

It is not feasible to maintain such a split along with jar delivery (which requires a merged classpath as well). This has potential use in ClojureScript and was considered in a few places on the page as an option for finding common source files, but it also has significant limits.

 

and then used the .clj file extension everwhere that would indeed buy you the same kind of code partition I'm talking about. Now this becomes a usability and tooling issue rather than a question of language extension.

It sounds like this path ignores the (fairly common) case of namespaces
that have *almost* but not exactly, the same implementation on different
platforms.

Yes and no. If you are within epsilon of having identical code across multiple platforms, it is my opinion (and I will stress opinion as I have no use study to back this up) that it should be trivial to factor such code so as to isolate platform interop code into platform specific functions which become candidates for a host file providing a shared host interface across platforms.

From having looked at many cljx example files, I do not think it is trival to refactor code in this way in enough cases that we can avoid considering them. 

I had not previously considered the notion of *just* a cljc extension without feature expressions. I think that's a new point to consider.

[kovas boguta]

On Sun, Nov 16, 2014 at 11:15 PM, Alex Miller <al...@puredanger.com> wrote:

involvement in this role and it was ultimately put off. It's a visible issue in every feature/pain poll. It slows down dev on things like core.async. I do not want to wait; I want to actively solve the problem.

I think this is a good point that has been missing in the discussion.

Things like core.async can't use a 3rd party tool for this problem. And that is a problem not experienced by people like me. Now I understand _something_ should go into core.  

What muddies the waters somewhat is that there a list of rationales, many of which don't have precedent justifying this kind of change, hence the title of this thread.

Reading the design document, its not clear to me what about achieving the goal (rather than the particular mechanism) requires changing the reader, which is the crux of the issue. 

If the benefits of this particular mechanism are large, it would be great to see that spelled out at the unsession.  

[Thomas Heller]

Hello everyone,

sorry for jumping over everyone's comments but I thought I'd jump in with my 2 cents.

I cannot say whether feature expressions are a good idea or not but I can say that I do not really like looking at the code they produce. Fortunately I only use them in very few places in my app but they do get ugly quite fast, mostly because it is usually not only the form they annotate but also the extra require/import statements in the (ns ...) that come with them. Maybe we can define a cleaner solution from the code perspective that solves the host interop issues.

I tried to sketch such an attempt based on the suggestions of others in this thread, basically we create a clojure common source file (.cljc) that includes a new "definterop". That just defines a interface for a var that must be implemented in a platform specific file (eg. .cljs). That file has its own (ns ...) and can include vars from the .cljc file as well as create its own vars which are private and cannot be called from outside the interop file. I cannot say how hard the implementation (reader, platform) part is for this I just can say that I like the code a bit more since I can basically ignore the platform specific files until I really need to look at them. I created a small example that I currently solve with .cljx in my app.

;; .cljc
(ns my-app.util)
 
(defn some-helper [it]
  it)
 
(definterop format-date [date])
 
;; .clj
(ns my-app.util
  (:import [java.time.format DateTimeFormatter])
  (:include [some-helper]))
 
(defn format-date [date]
  (some-helper (.format DateTimeFormatter/ISO date)))
 
;; .cljs
(ns my-app.util
  (:import [goog.i18n.DateTimeFormat])
  (:include [some-helper]))
 
(def date-format-iso
  (goog.i18n.DateTimeFormat. "yyyy-MM-dd"))
 
(def format-date [date]
  (some-helper (.format date-format-iso date)))

I cannot say whether this solves all interop issues or is any better than feature expressions but I like the idea of defining an interface (definterop) in one place and have the implementation somewhere else (even another library if needed), this way someone could implement a CLJR version without me over looking at CLJR. Sometimes there are other host issues that are not covered by "implementing" an interface function but maybe the default can just be that every platform implementation must also load platform specific files when loading a .cljc file, I'm thinking of the different implementations of CLJ (defmethod print-method ...) and CLJS IPrintWithWriter for example.

Hope the idea is clear, sorry for jumping off track a bit.

Regards,

/thomas

[Chas Emerick]

Since I'm mostly to blame for the current implementation of cljx and much of the noise around it that contributed to whatever degree of usage it sees, I thought I should say a few things. Technically replying to Alan here as the OP, but I'll refer and reply obliquely to various things mentioned in the thread so far. All of the below is basically in no particular order:

So as to not bury the lede, I'll say up front that I don't have a particularly strong opinion either way re: whether Clojure[Script|CLR|etc] itself should include facilities for supporting multiple platform targets or not.

I think I probably did the community a disservice by saying repeatedly that cljx is an implementation of "feature expressions". cljx is actually much, much simpler than feature expressions. It only offers two annotations, #+clj and #+cljs, does not support arbitrary expressions related to "features" (e.g. conjunction, disjunction, etc), and has no runtime component at all. By muddying the waters, I may have made e.g. survey requests referring to "feature expressions" specifically far more common and casual than they would have been otherwise.

(I'll happily acknowledge that the implementation of cljx is a hack, especially in how it makes itself available within the REPL. I won't apologize for that. It gets the job done, and makes it possible/feasible to write and maintain code that wouldn't otherwise exist.)

That cljx has remained as it is for as long as it has (over the course of ~2 years since the current implementation's conception) is a useful signal, I think. It's safe to say that I've used cljx more than most, and I've never once wanted more than it offers (maybe a tautology given my relationship to the project). Even in the I-want-a-pony world of Github issue feature requests, it's not been suggested once that cljx's scope should be expanded. That makes me think that its feature set is sufficient for the problem it's aiming to solve. One big thing I think the current wiki page lacks is a discussion of the rationale for the more complicated aspects of the proposal, i.e. arbitrary expressions and runtime *features*. Which use cases do they address beyond those that cljx is focused on?

It's been noted that a motivation for bringing multiplatform support into the language itself is to support contrib libraries like core.async (test.check could easily be in the same camp[1]). As much as I appreciate the need, I dislike the fact that language changes are contemplated/scheduled to accommodate contrib's policy of abstention from useful tools and libraries available in the broader community. It's not clear that that constraint informs language-level changes that are best for everyone that uses and depends upon Clojure, etc.

The concerns that have been raised about the additional read-time complexity associated with the current proposal are valid IMO. Brandon is correct that it's always been thus, but this is small consolation to Clojure tooling authors and their users. Clojure and ClojureScript have never made things particularly easy for tool authors, but what's under consideration now will exacerbate that condition more than any other significant feature addition so far AFAICT. This is one area where I'm very happy with cljx: it leaves no trace of its existence outside of the projects it is used within. Library authors only ever touch one set of files, but their users only ever see the code corresponding to their intended platform target, and they can navigate to that source easily because of the different *generated* files. From a tooling perspective, that the resulting jars contain multiple files per namespace (foo/bar.clj and foo/bar.cljs, etc) seems like a benefit, not a drawback.

I've rambled enough. Thanks to everyone that has put work into this effort. It's a problem that I think deserves serious consideration, thus my efforts to date elsewhere.

Cheers,

- Chas

[1] https://github.com/cemerick/double-check

On 11/14/2014 12:02 PM, Alan Dipert wrote:

Hi all,

I was a little surprised and unenthused to learn recently from Alex Miller [1] that Feature Expressions [2] are expected in 1.7.

I was surprised because I have yet to understand why they're needed, after using Clojure and ClojureScript together for a long time, and after reading the design document.  I was unenthused because the only thing clear to me about them is that they introduce syntactic complexity, which is probably the last thing I'd want for my favorite Lisp, and language.

My hope for any change to the syntax of Clojure would be that behind it are strong advocates with measured arguments, and I suspect feature expressions have both, I just don't know them yet.  With or without feature expressions Clojure is so awesome I will of course defer to Alex and Rich/Cognitect as steward and benefactor, but I like to think that by registering my disagreement I help deliver on the reason Clojure was open-sourced in the first place :-)

What I don't understand is: what do feature expressions give us that cljx-like tools don't, when combined with the things - like system properties and macros - that platforms and languages already do?  The design page enumerates how various problems can be solved with both cljx and feature expressions, but speaks little to things only solved by the presented design.  This leads to my feeling of "they're happening just because, here's exactly how."

The complexity I see them introducing is very large, compared to the benefits I can understand them bringing, which seem small.  First, it's another way to mix read and run time.  Because feature expressions (like data readers) use syntax instead of S-expr to convey sequence, they also introduce new questions of precedence, like the one Herwig Hochleitner raises [3].  Precedence conundrums - in Lisp.  Seems like a bad sign, no?

Colin Fleming explores a additional implications in the next comment [4].  In essence, read-time conditionals make Clojure code more difficult for tools to understand without running.  One advantage of cljx-style preprocessing is that tooling doesn't need to care about understanding how to recognize features for other platforms, because none of the code being read is necessarily being run at pre-processing time.  Transformations are "by a 3rd party".  With feature expressions, tools need to at least recognize the feature/platform boundaries, and it's not clear this is possible in all cases.

As for the runtime *features* component that cljx doesn't offer - are there examples of this being used to solve problems that a combination of system properties and macros can't?  While Clojure is now written for multiple platforms, there is little variation per platform - certainly not as much variation as once existed between competing, proprietary Common Lisp implementations in the 80s, which I think was the original use case.  Clojure and its family of open, noncompeting implementations, divided between platforms for platform affordances, not competing language feature/implementations, seems like a totally different use case.

I really like that Clojure's relative syntactic and evaluation simplicity give tools and editors a lot of room to innovate.  As co-author and maintainer of various build tools and multi-platform projects, I am afraid that feature expressions will ultimately remove flexibility and complicate my life instead of empowering me to go further and faster.  I could also be totally wrong and my fear completely unfounded.

Alan

1. https://github.com/cognitect/clojure-conj/wiki/Clojure-conj-2014-Unsessions#feature-expression-overview

2. http://dev.clojure.org/display/design/Feature+Expressions

3. http://dev.clojure.org/display/design/Feature+Expressions?focusedCommentId=8192107#comment-8192107

4. http://dev.clojure.org/display/design/Feature+Expressions?focusedCommentId=9928713#comment-9928713

[David Pollak]

Some disjointed thoughts from an "outsider".

Clojure is my first Lisp. I generally adopt languages before there is good tooling (the only except has been C#/Visual Studio), so I'm mostly a printf debuging kinda person.

As a side note, I am very, very impressed with the sustained quality of both community interaction and code in Clojure-land. I have a very deep level of respect for what you all have accomplished.

I used cljx extensively for Dragonmark [1] and it was an excellent experience.

Would I want cljx-style conditional code in Clojure? Meh. Cljx works fine for me and it works fine in the REPL and as long as that is true, I don't care if it's part of the Clojure reader or not.

Do I want more unified libraries across Clojure and ClojureScript? *YES*. I cannot be sufficiently emphatic about this. Having libraries where the require is identical across Clojure & ClojureScript (and where Clojure ignores :require-macros) and the API is identical is very important. Having the same core, string-handling API, etc. across Clojure and ClojureScript is critical to writing algorithms that work in both. If I were to wave my magic wand, I'd ask that the Clojure community unify the Protocols and APIs across Clojure and ClojureScript so most of the code we write is target agnostic.

Please take the above comments as data points. I am sure whatever the results, they will be excellent.

Thanks,

David

[1] https://github.com/dragonmark

--

Brick Alloy http://brickalloy.com

Lift, the simply functional web framework http://liftweb.net

Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im

[Brent Millare]

Since I was unable to attend the unsession at the conj, I was just curious, was there a summary of the points made from that discussion about the current state of feature expressions? From this post https://groups.google.com/d/msg/clojure-dev/6pnIeXFRwnI/5zq4rVkO_mEJ it seemed there was interest to address some issue there.

[Daniel Solano Gómez]

Hello, all,

I realise I am a bit late in coming to this thread, but I thought I
might share a slightly different point of view. Many of the points
expressed thus far have been in relation to handling platform
differences between Clojure, ClojureCLR, and ClojureScript. I'd like to
just mention a several possible uses of feature expressions that I am
not sure have gotten much attention.

First of all, I'd like to talk about the Clojure/Android platform. In
many ways, it works just like the standard Clojure platform. However,
there are few differences. For example, Android lacks some of the APIs
that are part of the Java platform. As a result, some of standard
Clojure functions are unavailable at runtime. Currently, if you were
trying to use one of these functions, I don't think it'd show up until
runtime. It would be convenient if those functions were completely
unavailable at compile time if the 'android' feature were present.
Additionally, there are number of API differences between versions of
Android. Feature expressions could potentially help in handling these
things in a library like Neko.

Speaking of things that could go missing, there has been talk of having
'knobs' available to make Clojure software either more
developer-friendly (additional compile-time checks, debugger support,
etc.) or more production-friendly (omitting the compiler/evaluator,
pruning metadata, etc.). Making these features might ensure that
certain code won't compile, e.g. you can't use 'eval' without the
compiler.

Of course, there is the possibility of program- or library-specific
feature expressions. What if a logging library supported an expression
that completely omits certain logging levels from generating any code
when enabled?

Lastly, I wonder to what extent existing Clojure features such as
*unchecked-math* might also be better implemented as features.


Finally, just because some of the above may be reasons to use feature
expressions, they do come at a cost. I can certainly understand the
fear of combinatorial explosion resulting from feature expression being
used in all of the above different ways. I see feature expressions as
being very analogous to C macros, which certainly make reasoning about a
C or C++ codebase more difficult. Nonetheless, C macros have been very
successful in being able to support portable code. I have also
witnessed some pretty horrifying uses of macros.

Anyway, that's about it for now.

Sincerely,

Daniel

[Herwig Hochleitner]

2014-11-14 18:02 GMT+01:00 Alan Dipert <al...@dipert.org>:

What I don't understand is: what do feature expressions give us that cljx-like tools don't, when combined with the things - like system properties and macros - that platforms and languages already do?  The design page enumerates how various problems can be solved with both cljx and feature expressions, but speaks little to things only solved by the presented design.  This leads to my feeling of "they're happening just because, here's exactly how."

The way I understand it, Feature Expressions are about letting you customize expressions that can't be customized with either macros or reader tags, that is: Expressions containing Foreign reader tags and Parts of Macro bodies (think (ns (:require ..))). There is also the effect of #_ which can't be emulated by either reader tags or macros.

I share your concern, that enshrining #+ and #- into clojure's syntax might complicate matters further, especially in an area, that is already a bit murky from reader tags.

 

The complexity I see them introducing is very large, compared to the benefits I can understand them bringing, which seem small.  First, it's another way to mix read and run time.  Because feature expressions (like data readers) use syntax instead of S-expr to convey sequence, they also introduce new questions of precedence, like the one Herwig Hochleitner raises [3].  Precedence conundrums - in Lisp.  Seems like a bad sign, no?

Thanks for mentioning that comment. As Alex said, I think the current design addresses that issue.

Please get me right: I'm fine with #+clj et al working as described, and also with a *features* var. What I'd like to reconsider, is their specification: Instead of complicating clojure's reader syntax, why not start with the question: What will the poor dsl author have to do, who wants to override #+clj? How can we save her from requiring users to pass strings. Also: Can EDN read it?

Here is an idea on how to do it, by slightly extending reader tags in a compatible way:

1. Allow reader-tag vars to be annotated with ^:read-generic or ^:clojure.generic/reader-tag. Such annotated reader-tag-functions will get a generic representation of the source code, where each child with a reader-tag is a generic `tagged` instance.

2. Allow reader tag fns to return the special value ##_ or :#/_ in order to cut out the form, similar to #_

3. Have #+clj #-clj, ... in the default reader tag environment, implement as per confluence

I'd argue that transformations 1. and 2. are worthwhile changes in their own right and they should be sufficient to do everything, feature expressions require. Building on those, 3. could provide feature expressions in a way that's easy to understand and easy to customize for build tools, etc

This doesn't address feature expressions like #+(foo or bar), or was it #+(or foo bar)? Either way, don't think I'd want those.

To summarize:

I'd like to see an approach where feature expression syntax is provided as reader tags (extending their API to make this possible). This would also channel the attention of build-tool authors to data_readers.clj, which is currently the single point where clojure's read-time rubber currently hits the road.

[Herwig Hochleitner]

I just realized, that having access to a generified representation of the source code (1. in previous message) is actually kind of a big deal, for homoiconicity.

Consider the expression:

#remote/eval (println {:platform #+clj "Clojure" #+cljs "Clojurescript"})

where the implementing tag var for #remote/eval is a feature-aware tag with ^:read-generic

Taking a page out of fressian's book, the generified representation passed to the #remote/eval tag-fn might be:

(println #map [:platform #+clj "Clojure" #+cljs "Clojurescript"]) ;; note that there are 3 tagged instances in the expr tree: #map #+clj and #+cljs

This way, feature reader tags can even manipulate uneven-armed maps on the syntax level. Since they operate on a generic representation of their contained tagged instances, they can even override feature expressions in their lexical extent.

Does this make sense?