Lisp/Clojure doesn't have syntax?

30 views
Skip to first unread message

Paul Barry

unread,
Dec 11, 2008, 4:37:18 PM12/11/08
to Clojure
I've been reading the latest chapter from Stuart's book, Chapter 7:
Macros, and he makes this statement:

"Clojure has no special syntax for code. Code is simply Clojure data.
This is true for normal functions, but also for special forms and
macros. Consider a language with syntax, such as Java. ..."

It seems to me that just like all lisps, Clojure has syntax. The
first and most obvious piece of syntax is the parenthesises. Lists
start with an open paren and end with a closing paren. This is syntax
and you can't change it with a macro.

Next is the single quote, which is just an alias for quote. Somewhere
along the line, someone decided that (quote foo) was too verbose and
they wanted 'foo to be syntactic sugar for (quote foo). That wasn't
and can't be done as a macro. For example, if I wanted to define my
own single quote, say $foo, I can't without modifying the parser.

Clojure goes on to add a lot of syntax. The literal syntax for
vectors [], maps {}, sets #{}, functions #(), keywords :, etc. are all
syntax, not possible with macros, and then there are all the "reader
macros" that are listed in Section 2.2, Comment ;, Deref @, Meta ^,
Metadata #^, regex #"", syntax-quote `, unquote ~, unquote-splicing
~@, and var-quote #'.

So is it really valid to claim Clojure has no syntax?

Randall R Schulz

unread,
Dec 11, 2008, 4:44:10 PM12/11/08
to clo...@googlegroups.com
On Thursday 11 December 2008 13:37, Paul Barry wrote:
> I've been reading the latest chapter from Stuart's book, Chapter 7:
> Macros, and he makes this statement:
>
> "Clojure has no special syntax for code. Code is simply Clojure data.
> This is true for normal functions, but also for special forms and
> macros. Consider a language with syntax, such as Java. ..."
>
> It seems to me that just like all lisps, Clojure has syntax. The
> first and most obvious piece of syntax is the parenthesises. Lists
> start with an open paren and end with a closing paren. This is
> syntax and you can't change it with a macro.

Punctuation is not syntax. Lisp can be said to have a syntax, that of
the S-Expression. Beyond that, as someone else pointed out in another
thread here recently, when we write Lisp code, we're basically writing
directly the AST that other languages would need a parser implementing
a complicated grammar to produce.


> Next is the single quote, which is just an alias for quote.
> Somewhere along the line, someone decided that (quote foo) was too
> verbose and they wanted 'foo to be syntactic sugar for (quote foo).
> That wasn't and can't be done as a macro. For example, if I wanted
> to define my own single quote, say $foo, I can't without modifying
> the parser.
>
> Clojure goes on to add a lot of syntax. The literal syntax for
> vectors [], maps {}, sets #{}, functions #(), keywords :, etc. are
> all syntax, not possible with macros, and then there are all the
> "reader macros" that are listed in Section 2.2, Comment ;, Deref @,
> Meta ^, Metadata #^, regex #"", syntax-quote `, unquote ~,
> unquote-splicing ~@, and var-quote #'.

All these things are syntactic sugar. Shorthand ways to write things
that have vanilla S-Expression counterparts. Again, I would not call
them syntax.


> So is it really valid to claim Clojure has no syntax?

Very nearly so.


Randall Schulz

Sean Spencer

unread,
Dec 11, 2008, 4:47:13 PM12/11/08
to clo...@googlegroups.com
That was one of the best explanations of code as data I've ever read.  Kudos!

Paul Barry

unread,
Dec 11, 2008, 4:50:06 PM12/11/08
to Clojure

On Dec 11, 4:44 pm, Randall R Schulz <rsch...@sonic.net> wrote:
> All these things are syntactic sugar. Shorthand ways to write things
> that have vanilla S-Expression counterparts. Again, I would not call
> them syntax.

syntactic sugar is not syntax?

Randall R Schulz

unread,
Dec 11, 2008, 4:58:10 PM12/11/08
to clo...@googlegroups.com
On Thursday 11 December 2008 13:47, Sean Spencer wrote:
> That was one of the best explanations of code as data I've ever read.
> Kudos!

Thanks. You forced me to look up the reference to which I alluded:

On Thursday 11 December 2008 06:32, evins...@gmail.com wrote:
> Subject: Re: In core structure editor, anyone?
>
> Structure editors are not in common use, maybe because, while they're
> a valid and maybe cool alternative to text editors for lisp code,
> it's not clear how useful they are for other kinds of code. In most
> programming languages a lexer and parser construct an abstract syntax
> tree from the source code. In lisp, the abstract syntax tree *is* the
> source code. That being the case, it makes a lot of sense with lisp
> to operate directly on the parse tree. In other languages, not so
> much.

So the kudos go to Mikel Evins (or is it Evins Mikel?).


Randall Schulz

Stuart Halloway

unread,
Dec 11, 2008, 5:03:44 PM12/11/08
to clo...@googlegroups.com
Randall,

Well said, and I need to find a place to make this explanation in the
book prior to chapter 7. :-)

Stuart

Daniel Eklund

unread,
Dec 11, 2008, 5:03:57 PM12/11/08
to Clojure


> I've been reading the latest chapter from Stuart's book, Chapter 7:
> Macros, and he makes this statement:
>
> "Clojure has no special syntax for code. Code is simply Clojure data.
> This is true for normal functions, but also for special forms and
> macros. Consider a language with syntax, such as Java. ..."
>
> It seems to me that just like all lisps, Clojure has syntax.  The
> first and most obvious piece of syntax is the parenthesises.  Lists
> start with an open paren and end with a closing paren.  This is syntax
> and you can't change it with a macro.

I would agree with you, but I do not feel too strongly about the
matter. The statement "clojure has _no_ syntax" is probably not
defensible with an eye towards formal rigor.. I have no idea what a
syntax-less language would look like. I feel fairly certain that the
book's aims are not to provide a hard stance on no-syntax but to open
the eyes of those people who have been dealing with the Java's and the
C's for their entire careers, where syntax is locked in by convention
and grammar specs.

Despite quote and {} and [] being syntax-tic sugar, they certainly
have their sexp counterparts, and all of this is driven from the
reader which could possibly be extended or reduced for particular
needs -- although Clojure does not allow for programmer-defined
reader macros (unlike other lisps).

The existence of the reader makes the "no syntax" claim a lot easier
to understand, but in the end, this could be a long discussion about
what each person feels "syntax" is.

my $0.02

Randall R Schulz

unread,
Dec 11, 2008, 5:13:31 PM12/11/08
to clo...@googlegroups.com

I have to say "no."

Syntax is about complex ordering rules. Take a look at the grammar for
Java or C or Pascal or Ada or even for YACC or JavaCC! They're
horrendously complicated. There's nothing like that in Lisps.

Syntactic sugar of the sort present in Lisps is very simple. One or two
characters trigger a small modification or adornment of what
immediately follows.

A more interesting contrast, perhaps, is CLIF, the Common Logic
Interchange Format (successor to KIF). If I flashed a page of it in
front of you, you'd think it was Lisp. It uses prefix, fully
parenthesized notation. But it requires a grammar specified in BNF.

So you _can_ have syntax in such notational forms, but Lisp doesn't


Randall Schulz

Dave Newton

unread,
Dec 11, 2008, 6:09:32 PM12/11/08
to clo...@googlegroups.com
--- On Thu, 12/11/08, Paul Barry wrote:
> syntactic sugar is not syntax?

I think that depends on which particular nits are being picked.

Is it strictly true that Clojure has "no syntax"? Meh--probably not.

(defun foo [bar] ...) has more unique characters than (defun foo (bar) ...) or (define (foo bar) ...) but is it a *meaningful* difference?

Is '(foo) meaningfully different than (quote (foo))? Again, more unique characters... but is it in the language, or is it in the reader? They're called reader macros for a reason.

All in all, not something I'll be terribly concerned with either way.

Dave

Rich Hickey

unread,
Dec 11, 2008, 6:19:13 PM12/11/08
to Clojure
Yes, the critical point is that the text-based representation of code
is completely secondary:

(def x (list (list (symbol "fn") (vector) "Hello World")))

(class (second (first x)))
-> clojure.lang.PersistentVector

(class (first (first x)))
-> clojure.lang.Symbol

x
-> ((fn [] "Hello World"))

(eval x)
-> "Hello World"

You could build code via Java calls to the Clojure lib:

(import '(clojure.lang RT Symbol PersistentVector))

(def y (RT/list (RT/list (Symbol/create "fn")
(PersistentVector/EMPTY)
(Symbol/create "Hello World")))))

Parens, square brackets, #, "", etc are reader syntax for data
structures. You could define a different reader and completely
different text-based syntax for the language.

The language syntax is defined in terms of data structures, not text,
and in that respect doesn't have (text-based) syntax. The reader has
text-based syntax for data.

Rich

Dave Griffith

unread,
Dec 11, 2008, 6:24:58 PM12/11/08
to Clojure

My view is that Lisps have very a simple syntax, achieved at the cost
of moving a fair amount of error checking until runtime. If you
ignore reader macros, you can tell if a Clojure expression is well-
formed by just keeping a count of open parentheses, which is about the
least amount of state that a parser could theoretically have. On the
downside, you can't tell until runtime whether a given function call
has an acceptable arity, which pretty much any other popular language
can check at edit-time.

--Dave Griffith

Randall R Schulz

unread,
Dec 11, 2008, 6:37:15 PM12/11/08
to clo...@googlegroups.com
On Thursday 11 December 2008 15:24, Dave Griffith wrote:
> ...
> On the downside, you can't tell until runtime whether a given function
> call has an acceptable arity, which pretty much any other popular
> language can check at edit-time.

Given that function definitions are closed, it's eminently feasible for
an editor to detect whether a calling form is valid or not. Keep in
mind that the only kind of function overloading in Clojure is that
based on arity.

More complex forms of overloading require the use of multimethods, which
do their dispatching dynamically and arbitrarily based on the supplied
arguments (dynamically). In that case, not all calling forms will be
statically validateable, though some still will.


> --Dave Griffith


Randall Schulz

Mon Key

unread,
Dec 12, 2008, 1:35:01 AM12/12/08
to Clojure
> Clojure does not allow for programmer-defined
> reader macros (unlike other lisps).

I know this has been touched upon last Spring - and Stu Halloway refs
at least one discussion of this in his book.

From a practical standpoint I am beginning to understand more why the
choice was made to not support reader macros.

But can someone (RH?) clarify the following?

Is it possible to introduce reader macros (albeit unsupported frowned
upon ones).

> You could define a different reader and completely different text-based syntax for the language.

Given the suggestion that Clojure could potentially be reimplemented
with a different text-based syntax utilizing an alternate reader does
this imply that reader macros are possible under a modified REPL?.

From a purely speculative standpoint, if this is possible - the
assertion that "Clojure does not allow for..." may be misplaced.

I'm wondering to the extent that it *may* be possible to 'Rig the
REPL' where would/does the Clojure license stand on this?

I ask, because I remember reading in one of P. Graham's books '.*
LISP' that some commercial LISPs were/are distributed with exceptions
for calling `eval' at the top level - presumably to prevent one from
modifying the REPL with some combination of macros/reader macros.

Has Clojure taken an alternate approach whereby code distributed under
license simply doesn't permit user defined reader macros?

And if so - would it not be more appropriate to communicate (as a
matter of policy - and stakeholder prudence) - "The Clojure License
agreement doesn't *permit* user introduction of reader macros which
when expanded at compile time attempt to modify the designer(s)
intended specification of Clojure's REPL"?

I tend to recoil at the use of the phrase, "not allowed", and while
I'm not personally even remotely capable of even beginning an attempt
to endeavor to introduce any sort of modification as envisioned above
I can certainly imagine situations/environments where such an attempt
might be made - advocate(s) of competitive open source license models
being one of the more benign examples...

If I have mis-characterized something or have otherwise misunderstood
the situation/possibilities I apologize.

s_P

On Dec 11, 5:03 pm, Daniel Eklund <doekl...@gmail.com> wrote:

Mon Key

unread,
Dec 12, 2008, 1:35:41 AM12/12/08
to Clojure
> Clojure does not allow for programmer-defined
> reader macros (unlike other lisps).

evins...@gmail.com

unread,
Dec 12, 2008, 3:50:56 AM12/12/08
to Clojure


On Dec 11, 3:58 pm, Randall R Schulz <rsch...@sonic.net> wrote:
> On Thursday 11 December 2008 13:47, Sean Spencer wrote:
>
> > That was one of the best explanations of code as data I've ever read.
> > Kudos!
>
> Thanks. You forced me to look up the reference to which I alluded:
>
> On Thursday 11 December 2008 06:32, evins.mi...@gmail.com wrote:
>
> > Subject: Re: In core structure editor, anyone?
>
> > Structure editors are not in common use, maybe because, while they're
> > a valid and maybe cool alternative to text editors for lisp code,
> > it's not clear how useful they are for other kinds of code. In most
> > programming languages a lexer and parser construct an abstract syntax
> > tree from the source code. In lisp, the abstract syntax tree *is* the
> > source code. That being the case, it makes a lot of sense with lisp
> > to operate directly on the parse tree. In other languages, not so
> > much.
>
> So the kudos go to Mikel Evins (or is it Evins Mikel?).

Thanks.

It's mikel evins. GMail somehow arranged a few years ago to make my
preferred account name ("mikel") and my second choice ("mikelevins")
and my third choice ("mikel.evins") inaccessible to me. I haven't
bothered to understand what the problem is.

Rich Hickey

unread,
Dec 12, 2008, 7:55:17 AM12/12/08
to Clojure
I'm not going to rehash reader macros or licensing again - they've
been discussed here at length, except to say the two are unrelated.
The license doesn't say anything about the reader or repls - it's the
standard CPL.

People are going to put their Clojure programs in text files they
expect to be read by the standard reader and evaluated, and it is a
design goal of Clojure that programs by different people be able to be
combined. User-defined reader macros don't compose, they clash.

If you did so under Clojure's license, you could make a version of
Clojure with a modified reader - the reader source is there. But the
programs you wrote to your version of the reader would be incompatible
with those written by someone else who did the same thing. So, it has
less to do with technical or licensing issues than it has to do with
it being a bad idea, if you don't want to encourage programmers to
create islands. CL has this feature and it is very rarely used for
that exact reason - programs that require custom reader macros are a
pain.

People have mentioned wanting to use the reader to read programs in
other languages or data formats, and while I'm not generally opposed
to the idea of a data reader whose output is not destined for the
compiler, I'm also skeptical of the generality of such a thing. A lisp-
style reader is a page or two of Clojure code. People are making
parser combinator libs etc. There are ways to solve this problem
without making the standard reader fungible, with the added complexity
of providing an API and semantics for reader macros, and permanently
leaving characters available for customization that that entails.

If you want to modify the reader to create syntax for yourself,
convince me and the community that the syntax has wide utility and
I'll consider putting it in the standard reader so everyone can use it
portably.

Rich

Mark Volkmann

unread,
Dec 12, 2008, 8:38:46 AM12/12/08
to clo...@googlegroups.com
On Thu, Dec 11, 2008 at 3:44 PM, Randall R Schulz <rsc...@sonic.net> wrote:
>
> Punctuation is not syntax.

>> Clojure goes on to add a lot of syntax. The literal syntax for
>> vectors [], maps {}, sets #{}, functions #(), keywords :, etc. are
>> all syntax, not possible with macros, and then there are all the
>> "reader macros" that are listed in Section 2.2, Comment ;, Deref @,
>> Meta ^, Metadata #^, regex #"", syntax-quote `, unquote ~,
>> unquote-splicing ~@, and var-quote #'.
>
> All these things are syntactic sugar. Shorthand ways to write things
> that have vanilla S-Expression counterparts. Again, I would not call
> them syntax.

I really have trouble understanding the idea that punctuation and
syntactic sugar shouldn't be considered syntax. To me, syntax is
anything that a human reader has to comprehend in order to understand
the structure of the code. I say "structure" because that excludes
having to understand what every function and macro does.

For example, it is very important for human readers of Clojure code to
understand that ^# adds the metadata that follows to the form that
follows that.

--
R. Mark Volkmann
Object Computing, Inc.

r

unread,
Dec 12, 2008, 12:55:16 PM12/12/08
to clo...@googlegroups.com
On Fri, Dec 12, 2008 at 1:38 PM, Mark Volkmann
<r.mark....@gmail.com> wrote:
>
> I really have trouble understanding the idea that punctuation and
> syntactic sugar shouldn't be considered syntax.

S-Expressions and its counterparts are technically a syntax. I guess
the distinction was made to stress that the Clojure's "syntax" is only
a small subset of what is called syntax in other languages - namely
data literals.

But then, I agree with you that any text format has associated syntax.
Just like XML or HTTP protocol do.

> To me, syntax is
> anything that a human reader has to comprehend in order to understand
> the structure of the code. I say "structure" because that excludes
> having to understand what every function and macro does.

The trick is that the source code is not really represents "code" at
all. It just a text description (literal) of ordinary Clojure data
that happen to describe the "code".

Regards,
-r.

Paul Barry

unread,
Dec 12, 2008, 2:11:36 PM12/12/08
to Clojure


On Dec 11, 6:19 pm, Rich Hickey <richhic...@gmail.com> wrote:
> Yes, the critical point is that the text-based representation of code
> is completely secondary:
> (def x (list (list (symbol "fn") (vector) "Hello World")))
>
> (class (second (first x)))
> -> clojure.lang.PersistentVector
>
> (class (first (first x)))
> -> clojure.lang.Symbol
>
> x
> -> ((fn [] "Hello World"))
>
> (eval x)
> -> "Hello World"
>
> You could build code via Java calls to the Clojure lib:
>
> (import '(clojure.lang RT Symbol PersistentVector))
>
> (def y (RT/list (RT/list (Symbol/create "fn")
> (PersistentVector/EMPTY)
> (Symbol/create "Hello World")))))
>
> Parens, square brackets, #, "", etc are reader syntax for data
> structures. You could define a different reader and completely
> different text-based syntax for the language.
>
> The language syntax is defined in terms of data structures, not text,
> and in that respect doesn't have (text-based) syntax. The reader has
> text-based syntax for data.
>
> Rich

Ok, so it's fair to say the Clojure Reader has syntax. I don't see
how this is fundamentally different than how Ruby works, for example:

http://www.igvita.com/2008/12/11/ruby-ast-for-fun-and-profit/

The clojure.lang.LispReader parses an input stream of text into a Java
Object, the same way that the Ruby parser parses code into an AST.

One nice thing I will say about the Clojure syntax, as in all lisps,
is that is there are no reserved words. Most languages, Java, Ruby,
Python, JavaScript, etc. have whole list of words you can't use for
variables. This is cool:

user=> (def def 4)
#'user/def
user=> def
4
user=> (def x 5)
#'user/x
user=> x
5

Stuart, this might be a good point to mention in the book, in that you
will never see a page like these for clojure:

http://java.sun.com/docs/books/tutorial/java/nutsandbolts/_keywords.html
http://www.zenspider.com/Languages/Ruby/QuickRef.html#3
http://www.python.org/doc/2.5.2/ref/keywords.html

Joost

unread,
Dec 12, 2008, 4:13:46 PM12/12/08
to Clojure
Paul Barry schreef:
> Ok, so it's fair to say the Clojure Reader has syntax. I don't see
> how this is fundamentally different than how Ruby works, for example:
>
> http://www.igvita.com/2008/12/11/ruby-ast-for-fun-and-profit/
>
> The clojure.lang.LispReader parses an input stream of text into a Java
> Object, the same way that the Ruby parser parses code into an AST.

Note that there is no standard whatsoever to guarantee that different
ruby implementations generate the same AST from the same source code.

The difference is that Lisps in general are defined in terms of their
AST, not in terms of syntax/source file structure. All reader macros/
syntactic sugar translates directly into defined data structures, and
the language is defined in terms of those data structures. Since
clojure doesn't allow for user defined reader macros, and has more
built-in reader constructs, the "no syntax" mantra is a bit harder to
maintain, at least superficially, but Common Lisp programs for
example, can completely do away with S-expressions and still be the
same language.

Rich Hickey

unread,
Dec 12, 2008, 5:29:30 PM12/12/08
to Clojure
I'm not sure Clojure has more built-in reader constructs than Common
Lisp, which has reader syntax for vectors, arrays, pathnames, dotted
pairs, bitvectors, read-time evaluation, labels, label references,
read-time conditionals, many kinds of numbers, functions, balanced
comments and uninterned symbols, in addition to the usual lists,
symbols, strings, numbers and characters.

Rich

Joost

unread,
Dec 12, 2008, 8:28:44 PM12/12/08
to Clojure
Rich Hickey schreef:
> I'm not sure Clojure has more built-in reader constructs than Common
> Lisp, which has reader syntax for vectors, arrays, pathnames, dotted
> pairs, bitvectors, read-time evaluation, labels, label references,
> read-time conditionals, many kinds of numbers, functions, balanced
> comments and uninterned symbols, in addition to the usual lists,
> symbols, strings, numbers and characters.

Let's just say that clojuire has more *useful* (as in actually used)
built-in reader constructs than CL.

I'm not complaining (much) - I think most of the clojure constructs
are sound.
Reply all
Reply to author
Forward
0 new messages