Enhanced host call syntax

[Rich Hickey]

I've added some syntactic sugar for host calls (as of SVN rev 793).
It's syntax I developed for my first Lisp (DotLisp) several years ago.
At the time, I liked the user experience, but didn't like the
difficulty of producing the syntax (e.g. in macros) or
programmatically consuming it (e.g. in code walkers), and the
transformations happened during reading, which posed other problems.
While the current Clojure host syntax is very programmable, it sits
between Lispy and host-like.

The first thing that has been bothering me has been the extra parens
on method calls (vs Java):

(. s (substring 2 5))

So I've made them optional:

(. s substring 2 5)

Note that this creates the need to disambiguate between field access
and no-args method calls. The only clash is when there is both a
public field and a public no-arg method with the same name. These are
resolved in favor of the method. Please let me know if you encounter
one of these in practice.

I've always wanted the DotLisp operator-position dot syntax back, and
finally figured it out - it works out well when the transformation
happens at macroexpansion time. I've enhanced macroexpansion to
perform the following transformations:

//lispy
(.substring s 2 5) => (. s substring 2 5)

//hosty
(s.substring 2 5) => (. s substring 2 5)

//easier new (note dot after classname)
(StringBuilder. s) => (new StringBuilder s)

Since it is a macroexpansion to the canonic form, macros and code
generators can continue to emit the canonic forms, and code walkers
can call macroexpand and see only the canonic forms.

Please let me know what you think and if you encounter any problems,

Rich

[Arto Bendiken]

On Apr 6, 12:48 am, Rich Hickey <richhic...@gmail.com> wrote:
> I've added some syntactic sugar for host calls (as of SVN rev 793).

> ...

This is great. The extra parentheses on '.' had been bothering me,
too. As for the new dot syntax transformation, I imagine I'll be using
the "lispy" variant, as "hosty" may be going too far in the "syntaxy"
department ;-) ...but we'll see, certainly good to have both options.

> //easier new (note dot after classname)
> (StringBuilder. s) => (new StringBuilder s)

This is perhaps a slight convenience, but I've found "new" readable
enough as it is (except that it being a special form, I still would
like a way to pass a variable list of arguments to it).

However, if syntactic sugar is to be added here, the question follows
why not eliminate the trailing dot altogether? Using a class name in
the operator position is currently an error:

user=> (StringBuilder)
java.lang.Exception: Expecting var, but StringBuilder is mapped to
class java.lang.StringBuilder

...but perhaps this could be made equivalent to "new"? It seems like a
logical enough construct.

In general, might it perhaps be useful to have support for an apply-
hook mechanism such as found e.g. in many Scheme systems, allowing the
user to create evaluation rules for various data types in operator
position without having to hack the runtime?

I don't know how well that would fit into your vision for Clojure, but
from experience in Scheme it's something that would come handy -
indeed even so for the present topic at hand, where operator-position
classes could mean instantiation by expansion into a 'new' form, the
latter which then could be redefined as an ordinary procedure,
allowing it to also be easily called with a variable-length list of
arguments when needed.

Arto

[Stephen C. Gilardi]

On Apr 5, 2008, at 6:48 PM, Rich Hickey wrote:

I've added some syntactic sugar for host calls (as of SVN rev 793).

The changes look very nice.

//lispy

(.substring s 2 5) => (. s substring 2 5)

//hosty

(s.substring 2 5) => (. s substring 2 5)

The hosty syntax doesn't seem to work when "s" is unquoted in a macro body:

user=> (defmacro with-conn[con init & body]`(with-open ~con ~init~@body(. ~con commit)))nil

user=> (defmacro with-conn[con init & body]`(with-open ~con ~init~@body(~con.commit)))java.lang.ClassNotFoundException: con.commitclojure.lang.Compiler$CompilerException: NO_SOURCE_FILE:15: con.commit at clojure.lang.Compiler.analyzeSeq(Compiler.java:3422) at clojure.lang.Compiler.analyze(Compiler.java:3310)

--Steve

[Stephen C. Gilardi]

Same problem with auto-gensym symbols:

(defmacro ...

(stmt#.executeQuery)

...)

gives an error.

--Steve

[Stuart Sierra]

On Apr 5, 6:48 pm, Rich Hickey <richhic...@gmail.com> wrote:
> I've added some syntactic sugar for host calls (as of SVN rev 793).

I like it! Especially with static methods:
(WordUtils.wrap my-string 70)

Static methods don't work with the full class name:
(org.apache.commons.lang.WordUtils.wrap my-string 70)
=> java.lang.ClassNotFoundException:
org.apache.commons.lang.WordUtils.wrap
But I can't think that I'd ever want to write something like that.

> (. s substring 2 5)
>
> Note that this creates the need to disambiguate between field access
> and no-args method calls. The only clash is when there is both a
> public field and a public no-arg method with the same name. These are
> resolved in favor of the method. Please let me know if you encounter
> one of these in practice.

I assume having a method and public field with the same name is
extremely rare (I wasn't even sure it would compile until I tried
it). I suppose you could avoid the problem by resolving in favor of
the field, and require the extra parens for the method in that case.

-Stuart

[Stephen C. Gilardi]

On Apr 5, 2008, at 6:48 PM, Rich Hickey wrote:

> The first thing that has been bothering me has been the extra parens
> on method calls (vs Java):
>
> (. s (substring 2 5))
>
> So I've made them optional:
>
> (. s substring 2 5)
>
> Note that this creates the need to disambiguate between field access
> and no-args method calls. The only clash is when there is both a
> public field and a public no-arg method with the same name. These are
> resolved in favor of the method. Please let me know if you encounter
> one of these in practice.

In my "sql.clj" at clojure-contrib, one call doesn't work after
removing the parens. As driven by the "db-write" example code:

This works:

(defn execute-prepared-statement
"Executes a prepared statement with a sequence of parameter sets"
[con sql sets]
(with-open stmt (. con prepareStatement sql)
(doseq set sets
(doseq arg (map vector (iterate inc 1) set)
(. stmt setObject (arg 0) (arg 1)))
(. stmt (addBatch))) ; <<<<< with parens
(. stmt executeBatch)))

This compiles but fails at runtime:

(defn execute-prepared-statement
"Executes a prepared statement with a sequence of parameter sets"
[con sql sets]
(with-open stmt (. con prepareStatement sql)
(doseq set sets
(doseq arg (map vector (iterate inc 1) set)
(. stmt setObject (arg 0) (arg 1)))
(. stmt addBatch)) ; <<<<< without parens
(. stmt executeBatch)))

user=> (db-write)
java.sql.SQLException: count (0) < 1
java.lang.Exception: transaction rolled back
at clojure.fns.sql2_test.db_write__1281.invoke(sql2-test.clj:36)
at clojure.lang.AFn.applyToHelper(AFn.java:181)
at clojure.lang.AFn.applyTo(AFn.java:174)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:2388)
at clojure.lang.Compiler.eval(Compiler.java:3443)
at clojure.lang.Repl.main(Repl.java:75)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:585)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:71)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:207)
at clojure.fns.sql2.execute_prepared_statement__1273.invoke(sql2.clj:
57)
at clojure.fns.sql2_test.db_write__1281.invoke(sql2-test.clj:17)
... 5 more
Caused by: java.sql.SQLException: count (0) < 1
at org.sqlite.DB.executeBatch(DB.java:220)
at org.sqlite.PrepStmt.executeBatch(PrepStmt.java:98)
... 12 more
user=>

It may be significant that PreparedStatement has an addBatch member
that takes no arguments (which I'm intending to call here):

http://java.sun.com/javase/6/docs/api/java/sql/PreparedStatement.html#addBatch()

while its superclass Statement has a member of the same name, but with
an argument:

http://java.sun.com/javase/6/docs/api/java/sql/Statement.html#addBatch(java.lang.String)

--Steve

[jon]

Hi, 2 problems..
----------
1) At the moment this error case isn't handled well..

user=> (.substring)
java.lang.NullPointerException
<SNIP>
Caused by: java.lang.NullPointerException
at clojure.lang.Compiler.macroexpand1(Compiler.java:3375)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:3406)
... 4 more
----------
2) I realize '.' is reserved, but with more special meaning for the
'.' character, shouldn't we now formally enforce the policy of
disallowing macros and functions with '.' in their names - to
eliminate any possible confusion?
The current changes introduce an asymmetry that mean you can still
define macros and functions called ".substring", "s.substring" or
"substring.", but you can successfully call the macros, whereas you
cannot call the functions.
(ps.I just also tried straight 'def's of vars including a '.', and
they behave in another slightly different way)
This leaves a question of what to do about the '..' macro? If it's
still going to be called '..', by side-stepping the restriction,
shouldn't the user theoretically have a way to redefine it if desired?
Thanks, Jon

user=> (defmacro .substring [f] `(do ~f ~f))
nil
user=> (.substring (prn "hi"))
"hi"
"hi"
nil

user=> (defn .substring [f] (prn f) (prn f))
#<Var: user/.substring>
user=> (.substring "hi")
java.lang.NoSuchFieldException: substring
<SNIP>
Caused by: java.lang.NoSuchFieldException: substring
at java.lang.Class.getField(Class.java:1507)
at clojure.lang.Compiler$InstanceFieldExpr.<init>(Compiler.java:786)
at clojure.lang.Compiler$HostExpr$Parser.parse(Compiler.java:697)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:3415)
... 7 more

user=> (def .substring 77)
#<Var: user/.substring>
user=> .substring
77

user=> (def s.substring 77)
#<Var: user/s.substring>
user=> s.substring
java.lang.ClassNotFoundException: s.substring
java.lang.ClassNotFoundException: s.substring
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
<SNIP>

user=> (def substring. 77)
#<Var: user/substring.>
user=> substring.
java.lang.ClassNotFoundException: substring.
java.lang.ClassNotFoundException: substring.
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
<SNIP>
----------

[Stephen C. Gilardi]

> On Apr 5, 2008, at 6:48 PM, Rich Hickey wrote:
>
>> The first thing that has been bothering me has been the extra parens
>> on method calls (vs Java):
>>
>> (. s (substring 2 5))
>>
>> So I've made them optional:
>>
>> (. s substring 2 5)
>>
>> Note that this creates the need to disambiguate between field access
>> and no-args method calls. The only clash is when there is both a
>> public field and a public no-arg method with the same name. These are
>> resolved in favor of the method. Please let me know if you encounter
>> one of these in practice.

In case it helps narrow down the problem for you, I found another
piece of code that doesn't work right when the parentheses are removed:

From ants.clj:

this works:

(defn animation [x]
(when running
(send-off *agent* #'animation))
(. panel (repaint)) <<<<< with parens, works
(Thread.sleep animation-sleep-ms)
nil)

this doesn't:

(defn animation [x]
(when running
(send-off *agent* #'animation))
(. panel repaint) <<<<< without parens, doesn't work
(Thread.sleep animation-sleep-ms)
nil)

--Steve

[Rich Hickey]

On Apr 5, 7:13 pm, Arto Bendiken <arto.bendi...@gmail.com> wrote:
> On Apr 6, 12:48 am, Rich Hickey <richhic...@gmail.com> wrote:
>
> > I've added some syntactic sugar for host calls (as of SVN rev 793).
> > ...
>

> However, if syntactic sugar is to be added here, the question follows
> why not eliminate the trailing dot altogether? Using a class name in
> the operator position is currently an error:
>
> user=> (StringBuilder)
> java.lang.Exception: Expecting var, but StringBuilder is mapped to
> class java.lang.StringBuilder
>
> ...but perhaps this could be made equivalent to "new"? It seems like a
> logical enough construct.
>

Heh. It actually has to go the other way (dot always). I probably
shouldn't have let people use otherwise unadorned classnames to refer
to classes outside of the member-access special op (.) Doing so has
prevented me from adding the last bit of sugar - allowing x.y (no
parens) anywhere to become (. x y) - sweet. In DotLisp, classnames had
to be followed by the trailing dot always, and allowed them to be
distinguished. As far as the classnames in the operator position, the
problem is given:

(x.y ...)

I can't distinguish object.member from package.class, and
unfortunately there is no way to tell if a string _might_ name a class
without incurring an exception, which I simply refuse to do in normal-
path logic. And I'd like to avoid any situations calling for a
classname where you can't fully qualify it.

> In general, might it perhaps be useful to have support for an apply-
> hook mechanism such as found e.g. in many Scheme systems, allowing the
> user to create evaluation rules for various data types in operator
> position without having to hack the runtime?
>
> I don't know how well that would fit into your vision for Clojure, but
> from experience in Scheme it's something that would come handy -
> indeed even so for the present topic at hand, where operator-position
> classes could mean instantiation by expansion into a 'new' form, the
> latter which then could be redefined as an ordinary procedure,
> allowing it to also be easily called with a variable-length list of
> arguments when needed.
>

I am definitely becoming satisfied with Clojure being somewhat less
programmable than CL or Scheme, leaving such programmability at macros
and avoiding user-level reader macros or hooks such as you suggest.
There is a tradeoff, as usual. Full programmability makes each
programmer the king of their own island, but also leaves them on an
island, as in some respect each programmer is developing in a
different language. That is because reader macros and other hooks are
not composable, whereas macros+namespaces are. Certainly when I survey
the Lisp landscape I see lots of islands. Clojure users may be stuck
on the same island, but might have some friends :)

Rich

[Rich Hickey]

On Apr 5, 9:39 pm, "Stephen C. Gilardi" <scgila...@gmail.com> wrote:
> > On Apr 5, 2008, at 6:48 PM, Rich Hickey wrote:
>
> > I've added some syntactic sugar for host calls (as of SVN rev 793).
>
> The changes look very nice.
>
> > //lispy
> > (.substring s 2 5) => (. s substring 2 5)
>
> > //hosty
> > (s.substring 2 5) => (. s substring 2 5)
>
> The hosty syntax doesn't seem to work when "s" is unquoted in a macro
> body:

Yes, I neglected to specify that this syntax should be avoided in
syntax quote (for now at least).

Rich

[Rich Hickey]

On Apr 5, 10:03 pm, Stuart Sierra <the.stuart.sie...@gmail.com> wrote:
> On Apr 5, 6:48 pm, Rich Hickey <richhic...@gmail.com> wrote:
>
> > I've added some syntactic sugar for host calls (as of SVN rev 793).
>
> I like it! Especially with static methods:
> (WordUtils.wrap my-string 70)
>
> Static methods don't work with the full class name:
> (org.apache.commons.lang.WordUtils.wrap my-string 70)
> => java.lang.ClassNotFoundException:
> org.apache.commons.lang.WordUtils.wrap
> But I can't think that I'd ever want to write something like that.
>

Well, it should work and I've fixed it so it does.

> I assume having a method and public field with the same name is
> extremely rare (I wasn't even sure it would compile until I tried
> it). I suppose you could avoid the problem by resolving in favor of
> the field, and require the extra parens for the method in that case.
>

You can't resolve in favor of a field without always incurring a
reflective call. Consider a name x known to refer (via type hint or
inference) to an interface X with a method foo(). The compiler
couldn't compile (. x foo) to a call to X.foo() since x might actually
refer to some object with a public foo field, but that can't be known
until runtime. As you say, I expect it to be extremely rare, I think
it is in violation of the style guidelines, and if someone does that
they get what they deserve - some dynamic language choosing for
them :) If anything, if you had both for some reason, the behavior
ought to be identical, IMO.

Rich

[Rich Hickey]

On Apr 6, 12:44 am, jon <superuser...@googlemail.com> wrote:
> Hi, 2 problems..
> ----------
> 1) At the moment this error case isn't handled well..
>
> user=> (.substring)
> java.lang.NullPointerException

Fixed - now throws a more informative exception.

> 2) I realize '.' is reserved, but with more special meaning for the
> '.' character, shouldn't we now formally enforce the policy of
> disallowing macros and functions with '.' in their names - to
> eliminate any possible confusion?

No. I appreciate the sentiment, but as I said before, it is
disallowed, it is documented as such.

> The current changes introduce an asymmetry that mean you can still
> define macros and functions called ".substring", "s.substring" or
> "substring.", but you can successfully call the macros, whereas you
> cannot call the functions.

The asymmetry is that I can define such macros and you shouldn't. But
the compiler can't tell the difference between me and you, so it's
just an unenforced rule.

Rich

[Rich Hickey]

On Apr 6, 2:43 am, Stephen C. Gilardi <scgila...@gmail.com> wrote:
> > On Apr 5, 2008, at 6:48 PM, Rich Hickey wrote:
>
> >> The first thing that has been bothering me has been the extra parens
> >> on method calls (vs Java):
>
> >> (. s (substring 2 5))
>
> >> So I've made them optional:
>
> >> (. s substring 2 5)
>
> >> Note that this creates the need to disambiguate between field access
> >> and no-args method calls. The only clash is when there is both a
> >> public field and a public no-arg method with the same name. These are
> >> resolved in favor of the method. Please let me know if you encounter
> >> one of these in practice.
>
> In case it helps narrow down the problem for you, I found another
> piece of code that doesn't work right when the parentheses are removed:
>

Fixed - thanks for the reports,

Rich

[Stuart Sierra]

On Apr 6, 8:13 am, Rich Hickey <richhic...@gmail.com> wrote:
> Heh. It actually has to go the other way (dot always). I probably
> shouldn't have let people use otherwise unadorned classnames to refer
> to classes outside of the member-access special op (.) Doing so has
> prevented me from adding the last bit of sugar - allowing x.y (no
> parens) anywhere to become (. x y) - sweet.

That would be very nice for static constants like "Level.FINE" (Java
logging). But "(Level.FINE)" isn't so bad.

-Stuart

[Toralf Wittner]

On Sat, 2008-04-05 at 15:48 -0700, Rich Hickey wrote:
> //lispy
> (.substring s 2 5) => (. s substring 2 5)
>
> //hosty
> (s.substring 2 5) => (. s substring 2 5)

There seems to be a slight asymmetry between these two with regards to
literals. For example the lispy (.charAt "123" 0) works fine, but the
hosty ("123".charAt 0) produces:

java.lang.Exception: Unable to resolve symbol: .charAt in this context
clojure.lang.Compiler$CompilerException: NO_SOURCE_FILE:18: Unable to
resolve symbol: .charAt in this context
at clojure.lang.Compiler.analyzeSeq(Compiler.java:3455)
at clojure.lang.Compiler.analyze(Compiler.java:3339)
at clojure.lang.Compiler.analyze(Compiler.java:3314)
at clojure.lang.Compiler.eval(Compiler.java:3475)
at clojure.lang.Repl.main(Repl.java:75)

I can see that Compiler.macroexpand1 checks if the operator is a symbol
and thus does not recognize the second form. Maybe in case the operator
is not a symbol the first operand - if a symbol - should be checked as
well for the dot? Or maybe there is a deeper reason why this it not
allowed. The asymmetry just catched my eye.

-Toralf

[Rich Hickey]

On Apr 6, 4:19 pm, Toralf Wittner <toralf.witt...@gmail.com> wrote:
> On Sat, 2008-04-05 at 15:48 -0700, Rich Hickey wrote:
> > //lispy
> > (.substring s 2 5) => (. s substring 2 5)
>
> > //hosty
> > (s.substring 2 5) => (. s substring 2 5)
>
> There seems to be a slight asymmetry between these two with regards to
> literals. For example the lispy (.charAt "123" 0) works fine, but the
> hosty ("123".charAt 0) produces:
>
> java.lang.Exception: Unable to resolve symbol: .charAt in this context
> clojure.lang.Compiler$CompilerException: NO_SOURCE_FILE:18: Unable to
> resolve symbol: .charAt in this context

Well, there is an asymmetry in power, but not in the rules. In both
cases the first element must be a symbol, i.e. foo or foo.bar, but
"foo".bar is still a string followed by a symbol.

> I can see that Compiler.macroexpand1 checks if the operator is a symbol
> and thus does not recognize the second form. Maybe in case the operator
> is not a symbol the first operand - if a symbol - should be checked as
> well for the dot? Or maybe there is a deeper reason why this it not
> allowed.

It's not really syntax, or at least is not syntax that extends beyond
the interpretation of a symbol/var as a macroexpansion trigger. I
don't intend to extend it to support any-arbitrary-expression.member.

Rich

[Arto Bendiken]

On Apr 6, 2:13 pm, Rich Hickey <richhic...@gmail.com> wrote:
> On Apr 5, 7:13 pm, Arto Bendiken <arto.bendi...@gmail.com> wrote:
>
> Heh. It actually has to go the other way (dot always). I probably
> shouldn't have let people use otherwise unadorned classnames to refer
> to classes outside of the member-access special op (.) Doing so has
> prevented me from adding the last bit of sugar - allowing x.y (no
> parens) anywhere to become (. x y) - sweet. In DotLisp, classnames had
> to be followed by the trailing dot always, and allowed them to be
> distinguished. As far as the classnames in the operator position, the
> problem is given:
>
> (x.y ...)
>
> I can't distinguish object.member from package.class, and
> unfortunately there is no way to tell if a string _might_ name a class
> without incurring an exception, which I simply refuse to do in normal-
> path logic. And I'd like to avoid any situations calling for a
> classname where you can't fully qualify it.

OK, makes sense. As I said, I'm happy enough with the previous 'new'
instantiation syntax, anyhow.

> > In general, might it perhaps be useful to have support for an apply-
> > hook mechanism such as found e.g. in many Scheme systems, allowing the
> > user to create evaluation rules for various data types in operator
> > position without having to hack the runtime?
>
> > I don't know how well that would fit into your vision for Clojure, but
> > from experience in Scheme it's something that would come handy -
> > indeed even so for the present topic at hand, where operator-position
> > classes could mean instantiation by expansion into a 'new' form, the
> > latter which then could be redefined as an ordinary procedure,
> > allowing it to also be easily called with a variable-length list of
> > arguments when needed.
>
> I am definitely becoming satisfied with Clojure being somewhat less
> programmable than CL or Scheme, leaving such programmability at macros
> and avoiding user-level reader macros or hooks such as you suggest.
> There is a tradeoff, as usual. Full programmability makes each
> programmer the king of their own island, but also leaves them on an
> island, as in some respect each programmer is developing in a
> different language. That is because reader macros and other hooks are
> not composable, whereas macros+namespaces are. Certainly when I survey
> the Lisp landscape I see lots of islands. Clojure users may be stuck
> on the same island, but might have some friends :)

Well, you would be satisfied, wouldn't you - you designed this
particular island to your own specs, after all ;-)

This is deviating from the topic at hand a bit, but I'm not a big fan
of the Principle of Least Power in programming languages, so I have to
say that so far my biggest gripe with Clojure is the uncertainty how
far I will ultimately be allowed to "push it" without permission from
the language designer.

For instance, Clojure's constrained reader has thrown two roadblocks
at me already, one where I would have wanted to process namespace-
qualified XML and RDF in S-expression form (but the character ':'
isn't allowed in symbols), and the other where I would have really
needed hexadecimal number literals, but couldn't add them myself (in a
Scheme that was missing them, I could have just added a readtable
dispatcher on #x and got right back to work).

With a Scheme background, I'm used to being able to deal with any
incidental restrictions the language may throw at me, because the
underlying engine is as extensible as possible.

Clojure at present makes many hard things simple, but it doesn't cater
for making impossible things possible. Certainly one can always appeal
to Turing completeness, and indeed for my reader problem I suppose I
could just code up a custom, extensible S-expression parser in Clojure
to get done what I need done. It's not a deal-breaker, but I can't
help wondering how many roadblocks like this are awaiting down the
road once one starts locking down the meta-extensibility of a
language?

Of course, you have every right to design the language to your own
vision. All I'm trying to say is that restricting e.g. reader syntax
does create real problems. I think it's fine that there's a somewhat
restricted, locked-down subset of the language that is defined for
interoperability purposes, but locking it down and throwing away the
key isn't a prerequisite to ensuring that - after all, taking that to
its logical conclusion, thither lie languages like Java.

(Sorry for the off-topic rant.)

[Toralf Wittner]

On Sun, 2008-04-06 at 13:32 -0700, Rich Hickey wrote:
> Well, there is an asymmetry in power, but not in the rules. In both
> cases the first element must be a symbol, i.e. foo or foo.bar, but
> "foo".bar is still a string followed by a symbol.

Yeah. I didn't get that from your original email and just played a
little with the new candy not really thinking that it might be
restricted to symbols.

> It's not really syntax, or at least is not syntax that extends beyond
> the interpretation of a symbol/var as a macroexpansion trigger. I
> don't intend to extend it to support any-arbitrary-expression.member.

Fair enough.

-Toralf

[Rich Hickey]

On Apr 6, 4:37 pm, Arto Bendiken <arto.bendi...@gmail.com> wrote:
> On Apr 6, 2:13 pm, Rich Hickey <richhic...@gmail.com> wrote:

> > I am definitely becoming satisfied with Clojure being somewhat less
> > programmable than CL or Scheme, leaving such programmability at macros
> > and avoiding user-level reader macros or hooks such as you suggest.
> > There is a tradeoff, as usual. Full programmability makes each
> > programmer the king of their own island, but also leaves them on an
> > island, as in some respect each programmer is developing in a
> > different language. That is because reader macros and other hooks are
> > not composable, whereas macros+namespaces are. Certainly when I survey
> > the Lisp landscape I see lots of islands. Clojure users may be stuck
> > on the same island, but might have some friends :)
>
> Well, you would be satisfied, wouldn't you - you designed this
> particular island to your own specs, after all ;-)
>
> This is deviating from the topic at hand a bit, but I'm not a big fan
> of the Principle of Least Power in programming languages, so I have to
> say that so far my biggest gripe with Clojure is the uncertainty how
> far I will ultimately be allowed to "push it" without permission from
> the language designer.
>

I am sympathetic to this argument, of course. I wrote a Lisp because I
value extensibility, and coming from Common Lisp, am aware of the
kinds of extension possible. But it is important to recognize a
continuum of tradeoffs. Were you to morph CL using its extensibility
features into something unrecognizable as CL, would it still be CL,
the language? Or would you just be leveraging CL as a runtime, parser
and library? Clojure's LispReader class is sitting right there.

Leaving room for extension trades off with features, for instance CL
leaves [] and {} for users, but fails to provide vector and map
literals. Is that more powerful? Can you replace car/cdr in CL/Scheme
with an abstraction like Clojure's seq? No. Can you define your own
data structures and have the CL/Scheme's standard library work with
them? No. Etc. There are many kinds of power, and Clojure's is
certainly not 'least'.

> For instance, Clojure's constrained reader has thrown two roadblocks
> at me already, one where I would have wanted to process namespace-
> qualified XML and RDF in S-expression form (but the character ':'
> isn't allowed in symbols), and the other where I would have really
> needed hexadecimal number literals, but couldn't add them myself

I think it is important to distinguish extension of the reader to
construct user-specific syntax and doing so for data IO purposes -
I'm amenable to the latter.

> (in a
> Scheme that was missing them, I could have just added a readtable
> dispatcher on #x and got right back to work).
>

It's funny that you mention Scheme, as Scheme the language has no
provisions for doing what you say, only specific Schemes do, and with
limited portability. Common Lisp is the real deal for standardized
extensibility. But it doesn't solve the 'how do you simultaneously use
2 libraries that have commandeered [] (or some other characters) for
different purposes'?

> Clojure at present makes many hard things simple, but it doesn't cater
> for making impossible things possible.

You are overstating this.

> Certainly one can always appeal
> to Turing completeness, and indeed for my reader problem I suppose I
> could just code up a custom, extensible S-expression parser in Clojure
> to get done what I need done. It's not a deal-breaker, but I can't
> help wondering how many roadblocks like this are awaiting down the
> road once one starts locking down the meta-extensibility of a
> language?
>

I am interested in sustainable, interoperable extensibility. My
problem with reader macros for syntax extension is that they're not
interoperable/composable. I'm open to suggestions.

> Of course, you have every right to design the language to your own
> vision. All I'm trying to say is that restricting e.g. reader syntax
> does create real problems. I think it's fine that there's a somewhat
> restricted, locked-down subset of the language that is defined for
> interoperability purposes, but locking it down and throwing away the
> key isn't a prerequisite to ensuring that - after all, taking that to
> its logical conclusion, thither lie languages like Java.
>

Again, overstated, it's not binary. Nor have I said 'never' about
anything. If you want to know what you'll have - macros in namespaces,
dynamic var rebinding, including fns, and thus context-based
programming. And again, Clojure is built on a large set of
abstractions with public interfaces, and is therefore extensible in a
much deeper and, I would argue, more important way than any other
Lisp.

>(Sorry for the off-topic rant.)

On the contrary, I think this kind of discussion is important. I
recognize that people coming from CL/Scheme may feel some sense of
loss vs. where they were, but I hope they realize that where they were
was like anyplace else, a place where some things were fixed and some
variable, some decisions made and some left open, with tradeoffs at
every turn.

Rich

[Arto Bendiken]

On Apr 6, 11:41 pm, Rich Hickey <richhic...@gmail.com> wrote:
> On Apr 6, 4:37 pm, Arto Bendiken <arto.bendi...@gmail.com> wrote:
>
> > Well, you would be satisfied, wouldn't you - you designed this
> > particular island to your own specs, after all ;-)
>
> > This is deviating from the topic at hand a bit, but I'm not a big fan
> > of the Principle of Least Power in programming languages, so I have to
> > say that so far my biggest gripe with Clojure is the uncertainty how
> > far I will ultimately be allowed to "push it" without permission from
> > the language designer.
>
> I am sympathetic to this argument, of course. I wrote a Lisp because I
> value extensibility, and coming from Common Lisp, am aware of the
> kinds of extension possible. But it is important to recognize a
> continuum of tradeoffs. Were you to morph CL using its extensibility
> features into something unrecognizable as CL, would it still be CL,
> the language? Or would you just be leveraging CL as a runtime, parser
> and library? Clojure's LispReader class is sitting right there.

That's a good point. Arguably, though, syntactic extensibility in the
reader is a useful and very powerful part of the domain-specific
language building features that Lisps give their users - and with good
reason given that S-expressions at their "purest" are not necessarily
all that user-friendly. (I'm certainly enjoying the sugared vector and
hashtable syntax in Clojure.)

With that in mind, I think the distinction you draw is somewhat
blurry. If I'm leveraging Clojure's runtime, parser and library, I
would still consider myself to be building on top of Clojure and
within the Clojure ecosystem, regardless of the incidental specifics
of the DSL I've built.

> Leaving room for extension trades off with features, for instance CL
> leaves [] and {} for users, but fails to provide vector and map
> literals. Is that more powerful? Can you replace car/cdr in CL/Scheme
> with an abstraction like Clojure's seq? No. Can you define your own
> data structures and have the CL/Scheme's standard library work with
> them? No. Etc. There are many kinds of power, and Clojure's is
> certainly not 'least'.

No, certainly Clojure isn't 'least', not by a long shot. I'm merely
talking about the general principle (let's forget its name) of locking
down or strictly enough defining a language to enforce constraints
such as interoperability.

(Not to get sidetracked even further ;-) but as for replacing car/cdr
and providing a seq-like infinite stream interface in Scheme, I do
believe it's been done even as a de-facto standard in an SRFI document
- unless you mean something like a "pure" R5RS Scheme, of course,
where even the existence of a "standard library" to begin with would
be dubious.)

> > For instance, Clojure's constrained reader has thrown two roadblocks
> > at me already, one where I would have wanted to process namespace-
> > qualified XML and RDF in S-expression form (but the character ':'
> > isn't allowed in symbols), and the other where I would have really
> > needed hexadecimal number literals, but couldn't add them myself
>
> I think it is important to distinguish extension of the reader to
> construct user-specific syntax and doing so for data IO purposes -
> I'm amenable to the latter.

OK, great. But where to draw such a line, though? Code is data is
code ;-)

> > (in a
> > Scheme that was missing them, I could have just added a readtable
> > dispatcher on #x and got right back to work).
>
> It's funny that you mention Scheme, as Scheme the language has no
> provisions for doing what you say, only specific Schemes do, and with
> limited portability. Common Lisp is the real deal for standardized
> extensibility. But it doesn't solve the 'how do you simultaneously use
> 2 libraries that have commandeered [] (or some other characters) for
> different purposes'?

Fair enough. I should qualify that when I say "Scheme", I certainly
don't mean just the R5RS level of functionality, but rather the actual
functionality one would expect to be available on any of the major
Scheme systems today, including the relevant SRFI extensions. Any of
the top five Scheme systems easily rival Common Lisp. So, regardless
of the narrow scope of the until-recently-venerated RnRS documents, in
practice one has been guaranteed an extensible readtable on any "real"
Scheme system.

> > Clojure at present makes many hard things simple, but it doesn't cater
> > for making impossible things possible.
>
> You are overstating this.

Yes, I am - with dramatic license. The point being that there
shouldn't be anything that is impossible in a Lisp ;-)

> > Certainly one can always appeal
> > to Turing completeness, and indeed for my reader problem I suppose I
> > could just code up a custom, extensible S-expression parser in Clojure
> > to get done what I need done. It's not a deal-breaker, but I can't
> > help wondering how many roadblocks like this are awaiting down the
> > road once one starts locking down the meta-extensibility of a
> > language?
>
> I am interested in sustainable, interoperable extensibility. My
> problem with reader macros for syntax extension is that they're not
> interoperable/composable. I'm open to suggestions.

I believe the question comes down to this: is it necessary to
*enforce* constraints in language extensibility, on the implementation
level, in order to *ensure* desirable characteristics (such as
interoperability)?

I for one don't think it is. There are any number of possible
undesirable characteristics in software. Take static typing: enforcing
draconian type safety does not guarantee bug-free programs. Or at the
more frivolous end, enforcing well-defined, uniform code conventions
(thinking of Python here) does not guarantee readily understandable
programs. Perhaps these enforced means may contribute something
towards the mentioned ends, but at what cost?

I really don't believe language extensibility should be artificially
limited; lacking omniscience it will always come back to bite somebody
- if not the designer(s) of the language, certainly its users. The
language designer(s) cannot possibly foresee every use case for the
language.

For Clojure, I believe that instead of imposing restrictions by fiat
on the implementation level, we might do better by attempting to
design/evolve a set of "Clojure best practices", thus actively
encouraging certain constraints on a social level as well as,
possibly, enforcing them on a library/repository level.

For instance, one might want to enforce that code committed to clojure-
contrib's SVN, or whatever the source might be for an eventual library
packaging/distribution system, is always written explicitly with
interoperability constraints in mind - such as the absence of custom
syntax. (Additionally, one might imagine that it would be desirable to
enforce or at least encourage documentation and unit tests.)

But if someone (such as myself) wants to (occasionally) write non-
interoperable Clojure code (whether for "frivolous" cool hacks or
"really important" boring proprietary work-related solutions), why not
let them? Where's the harm?

For many of the use cases where one would need extensibility, it's not
a case of a zero-sum game, writing without extensions or not. For
instance, on those rare occasions when you need to extend the
readtable, you *really* need it. Any artificial lack in extensibility
will simply drive users to seek out (or create) another tool that will
solve their problems (best case, creating an interpreter on top of
Clojure; worst case, going somewhere else). So by definition,
extensibility widens the range of problems that Clojure can be used to
solve, which just can't be bad for Clojure.

From the cool hacks department, experimentation could even lead to
something genuinely useful for later consideration for inclusion into
Clojure - why set any aspect of the language in stone (even if it is a
beautifully consistent stone, given the consistency of design
resulting from a single designer), and why narrow the set of
directions whence exploration could lead to innovation?

> > Of course, you have every right to design the language to your own
> > vision. All I'm trying to say is that restricting e.g. reader syntax
> > does create real problems. I think it's fine that there's a somewhat
> > restricted, locked-down subset of the language that is defined for
> > interoperability purposes, but locking it down and throwing away the
> > key isn't a prerequisite to ensuring that - after all, taking that to
> > its logical conclusion, thither lie languages like Java.
>
> Again, overstated, it's not binary. Nor have I said 'never' about
> anything. If you want to know what you'll have - macros in namespaces,
> dynamic var rebinding, including fns, and thus context-based
> programming. And again, Clojure is built on a large set of
> abstractions with public interfaces, and is therefore extensible in a
> much deeper and, I would argue, more important way than any other
> Lisp.

Yes, I like all the aspects of Clojure that you enumerated. But I'm
sure you can appreciate that the extensibility argument does not sit
that well with me if, in fact, e.g. any customization of the reader
syntax would require extending LispReader in Java, or, alternatively,
would mean writing my own S-expression parser in Clojure ;-)

Also, having "metaextensibility" of the sort that e.g. reader
extensions provide would mean that people wouldn't keep bugging you
(so much) with requests to add some syntax (regex notation, further
numeric bases, or whatever) when they could do it themselves. If you
were to then "bless" some syntax for inclusion into Clojure core, it
could be implemented as Clojure code in boot.clj instead of as Java
code in LispReader.java. One small step closer to a potentially self-
hosting system.

I would think those could be pretty good benefits from your point of
view. Looking at the archives, something like hexadecimal number
literals has been requested several times on this list, and with
extensible reader syntax a standard reply could be "implement it as a
reader extension and submit a patch, and I might consider it if it
fits my vision and is good enough".

So this might alleviate bottlenecks from the user's point of view (the
bottleneck being you ;-)) and some irritation from your point of view
(from constant nagging about some feature that isn't on your roadmap
in the immediate future or at all).

> >(Sorry for the off-topic rant.)
>
> On the contrary, I think this kind of discussion is important. I
> recognize that people coming from CL/Scheme may feel some sense of
> loss vs. where they were, but I hope they realize that where they were
> was like anyplace else, a place where some things were fixed and some
> variable, some decisions made and some left open, with tradeoffs at
> every turn.

Certainly. I'm not terribly excited about where Scheme is going with
R6RS, so I'm looking around. Clojure is the most promising Lisp I've
coded in lately.

I don't have a problem with "opinionated software" per se. To go
entirely apples to oranges on you: I took up Ruby on Rails back in the
day, and learned to eventually understand and enjoy many of the
constraints they imposed (or maybe I just became institutionalized).
The key thing was, though, that the fully dynamic, extensible nature
of Ruby's runtime meant that nothing was ever forbidden me if I wanted
to disagree. (Within the incidental constraints of a lesser language
like Ruby, of course - no decisions needed as for syntactic
extensibility, there.)

The Rails folks led by example, not by restriction. Unlike Perl's
TIMTOWTDI or the converse in Python, the philosophy offered was "this
is really the best way to do it, and we will very much discourage you
from doing it another way - but if you really, really want to do it
another way, go right ahead, it's your choice". Not easily captured as
an acronym, but the nuances make all the difference.

I'm hoping Clojure can eventually be described with a tagline not too
unlike that one :-)

Arto

[jon]

Hi,
Just a wild thought, maybe solving a non-problem..
I really don't know anything about this stuff, and this idea might be
exploited in exactly the way Rich doesn't want..

But could there possibly be a middle-ground solution? - some kind of /
very simplistic/ "plugin" ability in the standard, off-the-peg clojure
runtime.
- Most of the time clojure would be standard interoperable clojure.
- If someone had sufficient need / motivation for say, extending the
reader, they can copy and alter (or inherit from?) LispReader.java,
compile it to DslLispReader.class, and let's say rename it to
DslLispReader.plugin
- Then there could be a standard clojure function, something like
(with-plugin "DslLispReader" blah blah) -- dynamically bound(?), which
temporarily overrides some part of clojure, the reader in this case.

So then, when really needed, domain-specific plugins could be
distributed with the domain-specific code requiring them and could run
on any pre-installed clojure runtime.
In other words, the plugin ability is just a slight convenience, to
save you from having to bundle an incompatible Repl and/or clojure.jar
along with your domain-specific LispReader and domain-specific
*.clj's.

However, it would hopefully be quite uncommon, as most coders wouldn't
feel enough need for the extra work involved, and people who were
interested in interoperability-at-all-costs, such as clojure-contrib,
could either avoid any distributed code containing *.plugin and/or
make (with-plugin) throw an Exception.

Well, just brainstorming,
Cheers, Jon

[Rich Hickey]

On Apr 6, 8:04 pm, Arto Bendiken <arto.bendi...@gmail.com> wrote:
> ...much...

Sorry, this has become too long-winded to respond in place.

You've defined an equivalence between extensibility and reader macros
which is false. Reader macros are neither sufficient nor necessary for
extensibility. I've enumerated several ways in which Clojure is
extensible, some unique to Clojure (vs other Lisps). No language has
everything - that's life.

You've used Scheme as an example, a language whose many authors, over
many revisions, have not considered reader macros important enough to
include in the language, in spite of their obvious familiarity with
the technique and any value it may have.

You say "code is data is code", another logical over-simplification,
since while all code is data, not all data is code. The distinction
between the use of the reader for data I/O and the use of the reader
by the compiler is an important one, since the former can be supported
without engendering the dialect-splintering of the latter.

You keep asking 'what's the harm' without acknowledging any harms:

It destroys interoperability
Creates dialects
Hinders the core language which has to leave room for extensions
Breaks editor support
When used by shared programs, is just a recipe for a mess

You also mentioned the extensibility of Ruby's runtime. And yet I keep
reading articles about the problems of 'monkey-patching'. It's not all
roses.

Your 2 use cases - embedded ':' in symbols, and hex numbers, are best
handled by other means. Multiple number formats are going to be
supported (patch already submitted), but will follow moving to
standard boxed numbers (in-progress). Embedded ':'s are currently
disallowed in a conservative restriction put in place when I wasn't
sure I wouldn't be using them for namespaces, a restriction that might
be eased if you ask nicely :)

You perhaps haven't been here long enough to see that language
features advocated by users, and not that important to me personally,
e.g. regex, have been added. So it's not just about what I want, but
it is of a single vision.

Bottom line(s):

- Clojure is very extensible - I'll not have it labeled otherwise
because of a lack of reader macros.

- Clojure does not take the 'By default, allow' approach. For an
experiment with that paradigm - try Arc.

- Clojure may never have user-extensible reader macros that impact
the input to the compiler - if that is a must have for people, I
recommend CL, which has a standard, portable way to do that.

IMO, language design is equally about leaving things out as putting
them in. Where's the design in a free-for-all? I feel strongly that a
programming language is as much about communicating with other users
as it is communicating with the computer, and thus interoperability is
not a "constraint", it's an objective. If we each had our own language
we could only talk to ourselves.

Rich

[Arto Bendiken]

Rich,

I'm not trying to purposefully annoy you, you know. You invited
suggestions, and I responded at length the best I could. It's clear
this topic is something we'll have to agree to disagree on. But to at
least quickly address your reply:

On Mon, Apr 7, 2008 at 4:52 PM, Rich Hickey <richh...@gmail.com> wrote:
>
> You've defined an equivalence between extensibility and reader macros
> which is false. Reader macros are neither sufficient nor necessary for
> extensibility. I've enumerated several ways in which Clojure is
> extensible, some unique to Clojure (vs other Lisps). No language has
> everything - that's life.

I constrained myself to complaining about reader syntax as that's
where I've been running into walls. I can't say as of yet whether
there may be any other areas of Clojure that would turn out to not be
easily extensible. You've made the argument that there aren't (at
least if extending it in Java.)

> You've used Scheme as an example, a language whose many authors, over
> many revisions, have not considered reader macros important enough to
> include in the language, in spite of their obvious familiarity with
> the technique and any value it may have.

There are any number of useful things that the Scheme RnRS authors
haven't considered important enough to include in their historically
minimalist reports, including anything and everything that makes
Clojure a practical Lisp (say, a standard library - before R6RS,
anyway.) I don't intend to belabor the point, but when I talk about
Scheme systems I do *not* mean any abstract RnRS standard, nor would
you find many Schemers who would unduly constrain themselves so.
Please don't mischaracterize this.

> You say "code is data is code", another logical over-simplification,
> since while all code is data, not all data is code. The distinction
> between the use of the reader for data I/O and the use of the reader
> by the compiler is an important one, since the former can be supported
> without engendering the dialect-splintering of the latter.
>
> You keep asking 'what's the harm' without acknowledging any harms:
>
> It destroys interoperability
> Creates dialects
> Hinders the core language which has to leave room for extensions
> Breaks editor support
> When used by shared programs, is just a recipe for a mess

You did not address my essential point and key suggestion of
preventing these admitted harms on a level other than the
implementation level. Therefore I take it you disagree with that
suggestion. That's fine; I can't claim to know for sure that social
best practices would be sufficient to ensure interoperability in the
context of a Lisp dialect, even if such measures have proven
themselves on many other more conventional projects. I would elaborate
on this point at length, but let's not go there. Consider the question
buried.

> You also mentioned the extensibility of Ruby's runtime. And yet I keep
> reading articles about the problems of 'monkey-patching'. It's not all
> roses.

I will not get dragged into a Ruby discussion here except to say that
this is no more of an inherent "problem" in Ruby than it is in Lisp.
After all, "monkey-patching" is nothing more than the runtime dynamism
that provides for the possibility of redefining any existing function
(or method, in Ruby's case). It's not as if you *have* to always, day
in and day out, make use of the capability to, say, define 1 +1 = 3
just because you *can*. Experience brings restraint.

However, not finding that capability available hurts when you do need
it. As I've understood it from other threads on the list, Clojure
intends to (and does) explicitly support this capability - which just
happens to be known as "monkey-patching" among the less dynamic
languages - so any criticism of the facility will be equally
applicable to Clojure. Indeed if Clojure ever gets to be used on the
scale that Ruby presently is (and I do think it is the Lisp system
with at least the best shot at it, at present), I consider it
inevitable that we'll be seeing such articles from the folks who will
consider this and other advanced Lispy capabilities "dangerous"
features to be prevented by fiat instead of as a matter of
case-specific good judgment.

> Your 2 use cases - embedded ':' in symbols, and hex numbers, are best
> handled by other means. Multiple number formats are going to be
> supported (patch already submitted), but will follow moving to
> standard boxed numbers (in-progress). Embedded ':'s are currently
> disallowed in a conservative restriction put in place when I wasn't
> sure I wouldn't be using them for namespaces, a restriction that might
> be eased if you ask nicely :)

OK, makes sense. Hopefully this could indeed be implemented, as it
would remove the immediate need to go and write a reader of my own or
revert to Scheme for the use cases that I need this for - so pretty
please, consider this a feature request.

> You perhaps haven't been here long enough to see that language
> features advocated by users, and not that important to me personally,
> e.g. regex, have been added. So it's not just about what I want, but
> it is of a single vision.
>
> Bottom line(s):
>
> - Clojure is very extensible - I'll not have it labeled otherwise
> because of a lack of reader macros.
>
> - Clojure does not take the 'By default, allow' approach. For an
> experiment with that paradigm - try Arc.
>
> - Clojure may never have user-extensible reader macros that impact
> the input to the compiler - if that is a must have for people, I
> recommend CL, which has a standard, portable way to do that.
>
> IMO, language design is equally about leaving things out as putting
> them in. Where's the design in a free-for-all? I feel strongly that a
> programming language is as much about communicating with other users
> as it is communicating with the computer, and thus interoperability is
> not a "constraint", it's an objective. If we each had our own language
> we could only talk to ourselves.

That's fine. I couldn't agree more regarding the importance of
consistently saying 'no', though I, of course, don't enjoy being at
the butt end of such a decree. I do feel the present matter at hand is
a false dichotomy - as I've outlined. But it's your language, your
vision, and I'm not going to tell you how to run your show. I've made
my case to such an extent that it should be made, here, and further
irritating you regarding my own take on "extensibility" etc. is
unlikely to yield any positive results.

If I do need some capability that you're not willing to include in the
language, it's not the end of the world - being a Lisp, Clojure does
make it comparatively easy to Greenspun my own reader, or anything
else for that matter. We can consider the matter closed - though maybe
it could be revisited in a friendly way over beer sometime, if I
should happen to find myself on your side of the big pond.

Arto

--
Arto Bendiken | http://bendiken.net/

[Rich Hickey]

On Apr 8, 12:41 am, "Arto Bendiken" <arto.bendi...@gmail.com> wrote:
> Rich,
>
> I'm not trying to purposefully annoy you, you know. You invited
> suggestions, and I responded at length the best I could.

It seems I had my tea late and was a bit grumpy - sorry.

In the interest of remaining a productive person, I'll let all the
arguments stand. Except for this bit about 'monkey-patching':

> I will not get dragged into a Ruby discussion here except to say that
> this is no more of an inherent "problem" in Ruby than it is in Lisp.
> After all, "monkey-patching" is nothing more than the runtime dynamism
> that provides for the possibility of redefining any existing function
> (or method, in Ruby's case).

I'd like to correct this, as there are many people coming to Clojure
from OO languages without any knowledge of generic functions/
multimethods in packages/namespaces and are confused as to how to
organize their programs, and attain polymorphism and extensibility.

I'll not single out Ruby, as this argument applies to all similar
languages. Polymorphism in this context means being able to say (foo
x) or x.foo() and have what happens be different depending on some
characteristic of x. In traditional OO languages (single-dispatch,
methods in classes) the only characteristic of x that can be leveraged
is its type/class, say X. There is a second, more subtle aspect, which
is, which foo are we talking about? In traditional OO languages the
call usually takes the second form, as the question is answered by
looking up foo in the scope of the class of x. That scope may include
superclasses etc, but what is essential is that it constitutes a
namespace. So, traditional OO languages unify namespaces and
polymorphism.

In static OO languages (C++/Java et al), the scope is closed on
definition of the class, the original author having the final say. No
more methods can be added, and no more names introduced. (Although C#
is trying to allow the feeling of extension in a composable manner by
offering pseudo-extensions that live in scopes). In dynamic OO
languages (Smalltalk, Python, Ruby, et al), there is usually some
means whereby the set of methods in a class scope can be changed or
extended without changing the class definition (monkey-patching).
While the first case, changing some base functionality, is inherently
fraught with danger, the second, extending, seems desirable and
reasonable, and the rest of this discussion will focus on extension.

So, Fred wants to add bar() to X, and uses the monkey-patching
facilities of the language to do so. He calls x.bar() and it works.
Ethel, working independently, also wants to add a bar method to X,
with her own semantics. She can and does, and it works. Ricky uses
both Fred's and Ethel's libraries, creates an X, and calls x.bar() -
what happens? Nothing good. Could this have been avoided? Perhaps Fred
and Ethel could have written independent functions, without injecting
them into X, e.g. fredlib.bar(X) and ethellib.bar(X)? Presumably they
didn't because they wanted bar to be polymorphic, i.e. maybe they
added bar to classes X, Y, and Z, so they could call xyorz.bar() and
have the right thing happen depending on the type of xyorz. So, the
problem with monkey-patching is that it is non-composable, because it
forces all extensions to live in a single (class) namespace.

Is there another way? Yes, the designers of CLOS, in their great
wisdom, and with a desire to support multiple dispatch, realized the
limitations of having polymorphic functions be in or of a class. They
invented generic functions - stand-alone functions that allowed for
extensible polymorphic dispatch through the definition of one or more
generic methods of the function, such methods being selected based
upon the type or value of one or more arguments to the function. And
they did it in a language, Common Lisp, that had packages to separate
namespaces. Setting aside the very powerful multiple-dispatch
capability, this scheme has the advantage of separating namespaces,
class definitions and polymorphism. The result is strictly more
powerful and composable. Clojure follows CLOS in having namespaces
(for packages) and multimethods (for generic functions).

So, using Clojure, Fred, who wants a function bar that is polymorphic
on the type of its argument, working in his own namespace, defines a
multimethod:

(in-ns 'fred)
(clojure/refer 'clojure)
(defmulti bar class)
(defmethod bar String [s] :fred-bar-string)
(defmethod bar Integer [i] :fred-bar-int)
(bar "foo")
-> :fred-bar-string
(bar 2)
-> :fred-bar-int

and Ethel does similarly:

(in-ns 'ethel)
(clojure/refer 'clojure)
(defmulti bar class)
(defmethod bar String [s] :ethel-bar-string)
(defmethod bar Symbol [s] :ethel-bar-sym)
(bar "foo")
-> :ethel-bar-string
(bar 'foo)
-> :ethel-bar-sym

Ricky, wanting to use the libraries of both Fred and Ethel, has many
choices re: bar. He may only care about Fred's bar, in which case he
will :exclude bar when he refers to ethel, or, he may use both
equally, and not refer to either, preferring to fully qualify each
call as fred/bar and ethel/bar, or he may find those names tedious
and :rename them barf and bare. The important thing is that Ricky can
be aware of the semantic differences between fred/bar and ethel/bar,
can make choices about which is in use when, and is never precluded by
the existence of one from accessing the other. And all of these
decisions are completely independent of the polymorphism (or lack
thereof) of bar.

So, with generic functions/multimethods, you don't need to modify
someone else's scope in order to provide polymorphic extensions. Thus
'monkey-patching' is not needed to provide the kind of extension in CL
or Clojure for which there is no alternative to monkey-patching in
some other dynamic OO languages (short of building extra-lingual CLOS-
like dispatching). Generic functions/multimethods in packages/
namespaces are composable, because they allow for independent non-
conflicting extension.

Rich

[Feng]

defmulti/defmethod does not seem to be composable across namespaces.

First try,

(in-ns 'ricky)
(clojure/refer 'clojure)
(refer 'fred)
(refer 'ethel)
(defmulti bar class)
(defmethod bar String [s] [(fred/bar s) (ethel/bar s)])

ricky=> java.lang.IllegalStateException: bar already refers to: #<Var:
fred/bar> in namespace: ricky
...
ricky=> java.lang.Exception: Name conflict, can't def bar because
namespace: ricky refers to:#<Var: fred/bar>
...

Then, I tried

(in-ns 'ricky)

(clojure/refer 'clojure)
(defmulti bar class)

(defmethod bar String [s] [(fred/bar s) (ethel/bar s)])

ethel=> #<Namespace: ricky>
ricky=> nil
ricky=> #<Var: ricky/bar>
ricky=> nil
ricky=> (bar 1)
java.lang.IllegalArgumentException: No method for dispatch value:
class java.lang.Integer
...
ricky=> (bar 'foo)
java.lang.IllegalArgumentException: No method for dispatch value:
class clojure.lang.Symbol
...
ricky=>

Manual forwarding works, but is it optimal way to do what I'm trying
to do?

(in-ns 'ricky)
(clojure/refer 'clojure)
(import '(clojure.lang Symbol))
(defmulti bar class)
(defmethod bar String [s] [(ethel/bar s) (fred/bar s)])
(defmethod bar Integer [s] (fred/bar s))
(defmethod bar Symbol [s] (ethel/bar s))

regards,
Feng

[Rich Hickey]

I am presuming you are using the fred and ethel code from the message
you quoted, in which case I don't know why you didn't get:

java.lang.IllegalStateException: bar already refers to: #<Var: fred/
bar> in namespace: ricky

as soon as you called (refer 'ethel)? Since both fred and ethel define
a public bar you can't refer to them both without excluding/renaming
bar from one, the other or both.

Rich

[Feng]

Yes, I did get this as well, but somehow deleted it when trying to
remove long stack trace in between. Sorry about confusion.

> as soon as you called (refer 'ethel)? Since both fred and ethel define
> a public bar you can't refer to them both without excluding/renaming
> bar from one, the other or both.
>

After looking at defmulti/defmethod and MultiFn code, this makes total
sense now. Each (defmulti bar) in fred, ethel and ricky binds a new
MultiFn to separate Var.

Because I was *imagining* that (refer 'fred) could create a new bar
Var binding in ricky namespace, and initialize it to a copy of MultiFn
from fred/foo, then all subsequence (defmethod bar ...) just adds new
method to the same MultiFn.

Something like,

(in-ns 'ricky)
(refer 'fred)
(refer 'ethel)
(defmethod bar String [s] [(fred/foo s) (ethel/foo s)])
(bar 1)
--> :fred-bar-int
(bar 'foo)
---> :ethel-bar-sym
(bar "foo")
---> [:fred-bar-string :ethel-bar-string]

> Rich

Thank you very much for the response.

regards,
Feng

[Rich Hickey]

You _can_ add methods to a multimethod defined in another namespace:

(in-ns 'fred)
(clojure/refer 'clojure)
(defmulti foo class)
(defmethod foo Integer [i] :foo-int)
(foo 5)
fred=> :foo-int
(in-ns 'user)
(refer 'fred)
(defmethod foo String [s] :foo-string)
(foo "blah")
user=> :foo-string
(foo 5)
user=> :foo-int

Rich

[Feng]

Yes, I understand this can be done. But this is to mutate *other*
people's multimethod in place, which can cause problems to existing
clients of fred namespace (same as what monkeypatch did for open
class). What I prefer (meant to ask in previous post) is to compose a
*new* multimethod in my own namespace while leaving referred
multimetod intact (IMHP, more modular what you showed above). Clients
of mine can dispatch on 3 methods, all reuse fred/ethel's methods in
some degree, but leaving them alone. I did that in my last part of
previous post, and fine with the result.

> Rich

[Rich Hickey]

When someone defines a multimethod, they are acknowledging that the
set of possibilities is open, otherwise they would handle them all in
one place. It is an important way to offer extensibility to consumers
of a library. So, it is not inherently bad to add methods to a
multimethod from another namespace. The key is to handle your own
cases, e.g. if the designer of the multimethod made it dispatch on
attribute :x, and handled known cases 'a 'b 'c in the library itself,
and you later create objects whose value of :x might be 'my-ns/foo, it
is completely proper and safe to add a method for dispatch value 'my-
ns/foo. This is not monkeypatching.

> What I prefer (meant to ask in previous post) is to compose a
> *new* multimethod in my own namespace while leaving referred
> multimetod intact (IMHP, more modular what you showed above). Clients
> of mine can dispatch on 3 methods, all reuse fred/ethel's methods in
> some degree, but leaving them alone. I did that in my last part of
> previous post, and fine with the result.
>

This is a maintenance problem as fred and ethel add new cases that you
fail to forward.

Rich

[Feng]

Both are good points. Dispatching on namespaced key is a good
approach to ensure modularity.

Thanks,
Feng
> Rich

[a r]

On Wed, Apr 16, 2008 at 2:31 AM, Rich Hickey <richh...@gmail.com> wrote:
>
> You _can_ add methods to a multimethod defined in another namespace:
>
>
> (in-ns 'fred)
> (clojure/refer 'clojure)
> (defmulti foo class)
> (defmethod foo Integer [i] :foo-int)
> (foo 5)
> fred=> :foo-int
> (in-ns 'user)
> (refer 'fred)
> (defmethod foo String [s] :foo-string)
> (foo "blah")
> user=> :foo-string
> (foo 5)
> user=> :foo-int

Is it possible to extend dispatch function as well? Say, I'm using a
function that dispatches methods based on the contents of a data
structure. Now, if the data structure evolves, the dispatch function
must be extended to cover new additions.

The only solution that comes to my mind is to tag the data structure
with a ":type" field and dispatch methods using this field. However,
in this case, I need to implement the dispatch code twice - once in
"defmulti" and once in the constructor of the data structure. The
final object is also not very convenient to use. If the original data
structure was a list, it must be wrapped with a map before use etc.

Cheers,
-r.

[Christophe Grand]

a r a écrit :

> Is it possible to extend dispatch function as well?

If you define your multimethod with (defmulti name dispatch-fn), instead
you should (defmulti name #'dispatch-fn) and then redef or bind
dispatch-fn at will: your multimethod will be "up-to-date" with the
current binding of dispatch-fn.
(A var can be used as a function, in this case it delegates the call to
its value.)
... or you can use a multimethod as your dispatch function (half-kidding).

> The only solution that comes to my mind is to tag the data structure
> with a ":type" field and dispatch methods using this field. However,
> in this case, I need to implement the dispatch code twice - once in
> "defmulti" and once in the constructor of the data structure. The
> final object is also not very convenient to use. If the original data
> structure was a list, it must be wrapped with a map before use etc.
>

You can put your :type field in metadata instead of wrapping everything
in a struct.

Christophe

[Christophe Grand]

Christophe Grand a écrit :

> ... or you can use a multimethod as your dispatch function (half-kidding).
>

Self-replying: no you can't without resorting to #' (I forgot that
defmethod changes the root binding, MultiFns are immutable...)

[Rich Hickey]

Thanks for chiming in with good suggestions. I would add that I don't
think swapping out the dispatch function is generally supportable -
it's part of the contract of the multimethod. Since the idea behind
multimethods is potential extension by those other than the author of
defmulti, changing the dispatch function after the fact could
undermine the presumptions of extenders.

Using a multimethod as a dispatch function is a fine idea. While it
still doesn't support swapping out, it does allow for some
extensibility in how one finds the dispatch value.

Rich

[a r]

On Thu, Apr 17, 2008 at 1:27 PM, Rich Hickey <richh...@gmail.com> wrote:
> On Apr 17, 3:22 am, Christophe Grand <christo...@cgrand.net> wrote:
>
> > ... or you can use a multimethod as your dispatch function (half-kidding).

[...]

> > You can put your :type field in metadata instead of wrapping everything
> > in a struct.
> >
>
> Thanks for chiming in with good suggestions. I would add that I don't
> think swapping out the dispatch function is generally supportable -
> it's part of the contract of the multimethod. Since the idea behind
> multimethods is potential extension by those other than the author of
> defmulti, changing the dispatch function after the fact could
> undermine the presumptions of extenders.

Thank you guys for all suggestions. I particularly like the idea of
putting type tags in metadata. I haven't thought about this - perhaps
because I considered metadata, like exceptions, a tool that should not
be used for the flow control (in general).

> Using a multimethod as a dispatch function is a fine idea. While it
> still doesn't support swapping out, it does allow for some
> extensibility in how one finds the dispatch value.

I think the "dispatch multimethods" will have same limitations as the
original multimethod we wanted to extend. Besides, I would rather
avoid piling code in the dispatch function for performance reasons.

Another question:

How does "recur" behaves inside a multimethod? Is there any way of
looping through the multimethod (together with its dispatch function)
iteratively?

Cheers!
-r.

[a r]

On Thu, Apr 17, 2008 at 9:52 PM, a r <nbs.p...@gmail.com> wrote:
>
> Another question:
>
> How does "recur" behaves inside a multimethod? Is there any way of
> looping through the multimethod (together with its dispatch function)
> iteratively?

Similarly, I would nice to have "recur" aware of argument matching in
regular functions:

user=> (defn a
([]
(a 1))
([n]
(recur)))
java.lang.IllegalArgumentException: Mismatched argument count to
recur, expected: 1 args, got: 0

-r.

[Rich Hickey]

On Apr 17, 4:52 pm, "a r" <nbs.pub...@gmail.com> wrote:

The methods are ordinary functions and recur returns to the top of the
method. There is no connection back to the dispatch function, though.

Rich

[Rich Hickey]

On Apr 17, 8:17 pm, "a r" <nbs.pub...@gmail.com> wrote:

recur is a local looping construct. Transferring control to a
different arity would be a non-local jump. You'll have to make a named
call.

Rich

[Toralf Wittner]

On Sat, 2008-04-05 at 15:48 -0700, Rich Hickey wrote:

> Please let me know what you think and if you encounter any problems,

This example produces a NPE in InstanceFieldExpr.emit():

(defn f []
(let [u (new java.net.URL "http://www.google.com")
c (.openConnection u)]
(.getResponseCode c)))

The peculiarity of this code is that the getResponseCode method is part
of a subclass of URLConnection (i.e. HttpURLConnection) which is
returned by openConnection. Thus the HostExpr.Parser.parse() method
considers this to be an instance field expression because it can not
find the method in URLConnection. This makes me feel a bit concerned
about the ambiguity introduced by the new forms regarding field vs.
method access.

-Toralf

[Rich Hickey]

On May 3, 6:55 pm, Toralf Wittner <toralf.witt...@gmail.com> wrote:
> On Sat, 2008-04-05 at 15:48 -0700, Rich Hickey wrote:
> > Please let me know what you think and if you encounter any problems,
>
> This example produces a NPE in InstanceFieldExpr.emit():
>
> (defn f []
> (let [u (new java.net.URL "http://www.google.com")
> c (.openConnection u)]
> (.getResponseCode c)))
>

Fixed - thanks for the report!

Rich

[Rich Hickey]

On Apr 5, 6:48 pm, Rich Hickey <richhic...@gmail.com> wrote:
> I've added some syntactic sugar for host calls (as of SVN rev 793).
> It's syntax I developed for my first Lisp (DotLisp) several years ago.
> At the time, I liked the user experience, but didn't like the
> difficulty of producing the syntax (e.g. in macros) or
> programmatically consuming it (e.g. in code walkers), and the
> transformations happened during reading, which posed other problems.
> While the current Clojure host syntax is very programmable, it sits
> between Lispy and host-like.
>
> The first thing that has been bothering me has been the extra parens
> on method calls (vs Java):
>
> (. s (substring 2 5))
>
> So I've made them optional:
>
> (. s substring 2 5)
>
> Note that this creates the need to disambiguate between field access
> and no-args method calls. The only clash is when there is both a
> public field and a public no-arg method with the same name. These are
> resolved in favor of the method. Please let me know if you encounter
> one of these in practice.
>
> I've always wanted the DotLisp operator-position dot syntax back, and
> finally figured it out - it works out well when the transformation
> happens at macroexpansion time. I've enhanced macroexpansion to
> perform the following transformations:
>
> //lispy
> (.substring s 2 5) => (. s substring 2 5)
>
> //hosty
> (s.substring 2 5) => (. s substring 2 5)
>
> //easier new (note dot after classname)
> (StringBuilder. s) => (new StringBuilder s)
>
> Since it is a macroexpansion to the canonic form, macros and code
> generators can continue to emit the canonic forms, and code walkers
> can call macroexpand and see only the canonic forms.

>
> Please let me know what you think and if you encounter any problems,
>

After some experience and feedback I think I'm happy with these
enhancements with the exception of the 'hosty' variant (obj.member
args). Many people didn't get that obj.member is read as a single
symbol, and as such the entire symbol was treated as a macro. So, many
people presumed it was syntax for juxtaposed expressions and tried
things like concatenation (obj.member.member ...), adding a .member
onto an arbitrary expression ((foo).member ...) etc. This leads me to
believe it is too tricky, and so I am inclined to remove it,
especially as it doesn't represent a significant reduction over (. obj
member) and has the same operand order.

The one case it would be nice to retain would be
(ClassName.staticMember), as (Color.red) seems more readable than (.
Color red).

But, before I do anything, I'd like some feedback. Are you using the
hosty variant? Love it, hate it, been confused by it?

Thanks,

Rich

[Phil Jordan]

Rich Hickey wrote:
> But, before I do anything, I'd like some feedback. Are you using the
> hosty variant? Love it, hate it, been confused by it?

I briefly tried the "hosty" variant, but I'm now back to using the .
macro. I find it clearer as it highlights the non-lispy parts of the
code - '(. ' sticks out against other macros/functions - which is
particularly relevant in Clojure as you have to pay attention to lazy
evaluation whenever anything might have side effects, and Java functions
often fall into this category... When I end up using a simple Java
function more than once I typically wrap it in a Clojure function anyway.

~phil

[Duane]

FWIW, I prefer syntax that maintains consistency with the rest of the
language's "lispish" feel. So I am not in favor of the "hosty" or
"easier new". I think that on the whole they make the language less
understandable because they are less visibly consistent.

Also, Phil makes an excellent point about recognizing Java integration
points.

-- Duane

On May 17, 11:31 am, Rich Hickey <richhic...@gmail.com> wrote:
> On Apr 5, 6:48 pm, Rich Hickey <richhic...@gmail.com> wrote:
>
>
>

> > I've added some syntactic sugar forhostcalls (as of SVN rev 793).
> > It'ssyntaxI developed for my first Lisp (DotLisp) several years ago.

> > At the time, I liked the user experience, but didn't like the

> > difficulty of producing thesyntax(e.g. in macros) or

> > programmatically consuming it (e.g. in code walkers), and the
> > transformations happened during reading, which posed other problems.

> > While the current Clojurehostsyntaxis very programmable, it sits

> > between Lispy andhost-like.
>
> > The first thing that has been bothering me has been the extra parens
> > on method calls (vs Java):
>
> > (. s (substring 2 5))
>
> > So I've made them optional:
>
> > (. s substring 2 5)
>
> > Note that this creates the need to disambiguate between field access
> > and no-args method calls. The only clash is when there is both a
> > public field and a public no-arg method with the same name. These are
> > resolved in favor of the method. Please let me know if you encounter
> > one of these in practice.
>

> > I've always wanted the DotLisp operator-position dotsyntaxback, and

> > finally figured it out - it works out well when the transformation

> > happens at macroexpansion time. I'veenhancedmacroexpansion to

> > perform the following transformations:
>
> > //lispy
> > (.substring s 2 5) => (. s substring 2 5)
>
> > //hosty
> > (s.substring 2 5) => (. s substring 2 5)
>
> > //easier new (note dot after classname)
> > (StringBuilder. s) => (new StringBuilder s)
>
> > Since it is a macroexpansion to the canonic form, macros and code
> > generators can continue to emit the canonic forms, and code walkers

> > cancallmacroexpand and see only the canonic forms.

>
> > Please let me know what you think and if you encounter any problems,
>
> After some experience and feedback I think I'm happy with these
> enhancements with the exception of the 'hosty' variant (obj.member
> args). Many people didn't get that obj.member is read as a single
> symbol, and as such the entire symbol was treated as a macro. So, many

> people presumed it wassyntaxfor juxtaposed expressions and tried

[Chouser]

I used the hosty syntax some for a while, but now almost always use
the lispy variety. As mentioned, (Color.red) is nice, as (.red Color)
just seems weird. But I think (. Color red) isn't bad enough that the
hosty form is really needed.

--Chouser

[Christophe Grand]

Chouser a écrit :

> But I think (. Color red) isn't bad enough that the
> hosty form is really needed.
>

I agree on all points.

Christophe

[Ozzi Lee]

Pro lispy, anti hosty, not real crazy about the easier new.

Not sure I like the optional parens on method calls using dot:

(. s substring 2 5)

as long as we have the lispy syntax that doesn't need the parens:

(.substring s 2 5)

The lack of parens on the first version seems more confusing than anything.

Ozzi

[Ozzi Lee]

The idea just popped into my head, but what about using : for field access?

(Color:red)

You could also have : figure out getters, so that if foo had a getBar
method, (foo:bar) would work.

You could get real crazy and eliminate the parens, too.

(print foo:bar)

instead of

(print (.getBar foo))

Perhaps it's just useless (and confusing) sugar after all, though.

Ozzi

[Dudley Flanders]

On May 17, 2008, at 6:27 PM, Ozzi Lee wrote:
> The idea just popped into my head, but what about using : for field
> access?

FWIW, Kawa uses a similar colon notation[0] for field and static
method access.

:dudley

[0] http://www.gnu.org/software/kawa/Method-operations.html

[Stephen C. Gilardi]

I've been sticking with the lispy variant. I like the fewer
parentheses in the new syntax.

I like the new syntax for "new" a lot. I gather you've considered an
unadorned Classname as the first item in a list as a trigger for an
implicit "new" and rejected it. If that could work it would be a
natural.

--Steve

[Chouser]

I hadn't tried the new "new" syntax yet. Not sure if I like it or
not, but this is sure convenient:

(-> urlstr URL. .openConnection .getContent InputStreamReader. BufferedReader.)

--Chouser

[Dimitry Gashinsky]

I liked the lispy syntax a lot and did not like the hosty and the new
new as much. The new new was especially confusing and felt like
something is not finished about the symbol and my eyes would try to
jump to the next line and then back.

It is probably my habits from reading a lot of Java code.
I found the lispy was different enough from Java that it stand out and
did not turn on my Java pattern matching in my brain.

The hosty one kept turning on little alarms, I could not get used to
it.

Regards,
--
,`,`,`,`,`,`,`,`,`,`,`,`,
,`,`,`,` @ `,`,`,` .com ,
, Dimitry Gashinsky ,`,`,
,`,`,`,`,`,`,`,`,`,`,`,`,