I've been watching the recent threads on special variables, top-level
setq-s and the like, and I've come to the conclusion that the current
situation is badly broken.  I have a proposal on how to fix it that I'd
like some feedback on.

The intractable problem is that beginners are chronically confused by
the fact that Lisp does not make an adequate syntactic distinction
between two completely different ways of binding variables.  Imagine
you are reading some code and you encounter the following two functions:

(defun foo (x) (blarg x))

(defun baz (y) (blarg y))

Do FOO and BAZ do the same thing?  Well, they might or they might not.
It depends.  It depends on whether X or Y have been DEFVARed.  It depends
on whether there are any free references to X or Y in any function called
by BLARG.  It depends on whether you are trying to write thread-safe code.
It depends on whether performance differences matter.  It depends on whether
your code is interpreted or compiled.  And if you want to explain the
reasons it depends on all these things you have to start talking about
the differences between symbols and variables, what bindings are, and
that there's nothing at all special about "special" variables, it's just
an ancient piece of Lisp terminological baggage that means nothing more
than "dynamic scope."

A beginner's initiation into this state of affairs invariably comes when
he or she one day as an expedient types (defvar x 1) and then spends the
rest of the day (or week) trying to figure out why her code is suddenly
exhibiting mysterious bugs.  The admonition to always use *...* notation
for DAFVARed variables is a temporary fix, but it generally takes a very
long time for a typical programmer to wrap their brains around what is
really going on.

The problem is pervasive and subtle.  Consider:

(funcall (lambda (x y) (lambda () ... x ... y ...)) 1 2)

Does this return a lexical closure over X and Y?  Well, it might, or it
might not.  It depends.

Now, this has not been much of an issue because artificially separating
the name spaces of lexical and dynamic variables using the *...* convention
generally works to keep the problem at bay.  But the trouble is that this
solution only works if you use it, and you can only use it if you are
aware of it.  And even if you are aware of it, your collaborator might
not be.  *You* might not have DEFVARed X, but how do you know that Joe
didn't?

The solution to this problem is to change the language so that the
distinction between dynamic and lexical bindings is locally manifested.
This is technically simple, but politically complex because it means
either adding some kind of lexical declaration or getting rid of pervasive
SPECIAL declarations (e.g. getting rid of DEFVAR, (PROCLAIM (SPECIAL ...))
etc.)  My personal preference is the latter.  Top-level declarations that
radically change the semantics of code in ways that are not lexically
apparent are EVIL!

Even local special declarations are IMO evil.  The semantics of a piece
of code ought not to depend on forward references.  It's also very much
against the spirit of a declaration.  Declarations are for providing extra
information for the compiler.  They are not for changing the semantics of
the program.  It should be possible to remove all declarations without
changing anything about the program except how fast it runs.

The solution is obvious.  Special variables are nothing more than the
symbol-value slot of some symbol.  Why not make that simple fact manifest
itself in the syntax?  For example:

(let ( (x 1)                   ; A lexical binding
       ((symbol-value x) 2) )  ; A dynamic binding

While we're at it we could fold in the binding of multiple values and
destructing into one uniform syntax:

(superlet ( (x 1)                   ; A lexical binding (guaranteed)
            ((symbol-value x) 2)    ; == (let ((x 2)) (declare (special x))
            ((values a b c) (foo))  ; == (multiple-value-bind (a b c) (foo)
            ((values a (symbol-value b) c) (baz))
                                    ; As above, but with (declare (special b))
            ((a b c) (foo)) )       ; A DESTRUCTING-BIND

A simple reader macro helps make things a lot less wordy:

$x == (symbol-value 'x)

So now we can write:

(let ( (x 1) ($y 2) ) ; X is a lexical binding, Y is a dynamic binding

This would work at top-level too:

(setf $x 1) ; Set the global X with no warnings and no semantic ambiguity
(setf x 1) --> Warning! No lexical variable named X visible here.  Setting
$X instead.

To implement this proposal completely requires the cooperation of Lisp
implementors because you have to change the way lambda lists are
processed.  But once that's done you can write (lambda (x $y) ...)
instead of (lambda (x y) (declare (special y)) ...), or
(multiple-value-bind (x $y) (foo) ...) instead of (multiple-value-bind
(x y) (foo) (declare (special y)) ...) or (defun foo (x $y) ...) instead
of (defun foo (x y) (declare (special y))  ...)

As a prototype I have implemented a macro called BIND that works as
described above.  BIND binds lexical and dynamic variables, and handles
multiple values and destructuring.  It also uses a paren-less LOOP-like
syntax.  I present it as both an illustration of how simple my proposal
would be to implement properly, and as an illustration of how easy it is
to change Lisp completely within the standard.

The syntax for BIND is:

(BIND var [=] value [and] ... in &rest body)

var : symbol | (symbol-value symbol) | (values var var ...) | list

Keywords in square brackets are optional.

Example: (BIND x = 1 and (z ($y) q) = (foo) and (values $a b c) = (baz) in ...)

expands to:

(let ( (x 1) )
  (destructuring-bind (z (y) q) (foo)
    (declare (special y))
    (multiple-value-bind (a b c) (baz)
       (declare (special a))
       ...

I conjecture that beginners at least will find BIND easier to deal with
than the things it replaces.

Erann Gat
g...@jpl.nasa.gov
---

(set-macro-character #\$
                     (lambda (s c)
                       (declare (ignore c))
                       (if (whitespacep (peek-char nil s))
                         '|$|
                         `(%symval ,(read s))))
                     t)

(defmacro %symval (x) `(symbol-value ',x))

(defun process-vartree (vl)
  (let ( (specials '()) )
    (setf vl
          (iterate loop1 ( (v vl) )
            (cond ( (atom v) v )
                  ( (eq (car v) '%symval)
                    (push (second v) specials)
                    (second v) )
                  (t (cons (loop1 (car v)) (loop1 (cdr v)))))))
    (values vl specials)))

(defmacro bind (&rest forms)
  (cond ( (null forms) (error "Missing IN keyword") )
        ( (eq (car forms) 'in)
          `(progn ,@(cdr forms)) )
        ( (eq (car forms) 'and)
          `(bind ,@(cdr forms)) )
        (t (let ( (var (pop forms))
                  (val (pop forms)) )
             (if (eq val '=) (setf val (pop forms)))
             (cond
              ( (atom var)
                `(let ( (,var ,val) ) (bind ,@forms)) )
              ( (eq (car var) '%symval)
                (setf var (second var))
                `(let ( (,var ,val) )
                   (declare (special ,var))
                   (bind ,@forms)) )
              ( (eq (car var) 'values)
                (receive (vars specials) (process-vartree (cdr var))
                  `(multiple-value-bind ,vars ,val
                     (declare (special ,@specials))
                     (bind ,@forms))) )
              (t (receive (vars specials) (process-vartree var)
                   `(destructuring-bind ,vars ,val
                      (declare (special ,@specials))
                      (bind ,@forms)))))))))

#|
; Example:
(defun baz () (values 1 2 3 4))

(defun foo ()
  (declare (special x y))
  (list x y $z $w))     ; Look Ma, no warnings!

(setf x nil y nil z nil w nil)

  it seems to me that this the fundamental question.  the answer lies in
  the implementation's excellence in implementing the function `describe'
  and any other environment-querying functions.

  however, there _is_ something we can and should do: add explicit support
  for retrieving this important piece of information about a symbol, apart
  from the rather obvious user interface issues like querying the system
  when at a symbol name in Emacs.  the compiler and the interpreter could
  also be asked to produce warnings about special variables for those who
  need them.  (I'd favor declaring variables special locally as a nice way
  to document the known special effects and also silence such warnings.)

  removing special variables because they confuse a few people is a typical
  "modern" reaction to the lack of diligence and effort that "modern" users
  are no longer expected to expend in learning anything.  this is simply a
  beginner's issue.  Common Lisp used to cater to experienced programmers
  at the cost of having to learn it, like a _skill_, something people of
  reasonable competence levels would _value_.  such is of course terribly
  politically incorrect in this day and age, where blathering idiots get to
  vote as many times as they can by virtue of forgetting the question and
  anyone with any experience at all is considered prejudiced by virtue of
  not answering all questions up for vote with a blank stare.

  I vote that Common Lisp remain a language that needs to be learned and
  studied, and instead focus our attention on stuff that actually affects
  users of all categories much more than this trifling issue, like being
  compatible with the notion in languages with which we would like to
  communicate of what constitutes a symbol name: the actual, literal
  sequence of characters (dollar signs included), not some case-mangled
  version of same.

  I also vote that somebody write "the complete idiot's guide to special
  variables" instead of proposing silly language changes.

| $x == (symbol-value 'x)

  I think "Common Perl" would be a good name for your modified language,
  with syntax in macros and equal signs and all.  yuck.

Janos Blazi

Erann Gat <g...@jpl.nasa.gov> schrieb in im Newsbeitrag:
gat-2902001653240...@milo.jpl.nasa.gov...

**

I don't think this is not a CL specific problem. And programmers
have already been advised to avoid the usage of 'global' variables
with indefinite scope unless it is necessary.

Is there anything particularly dangerous when a beginner equates
CL's special variables to global vars in other languages?

  I can't answer for Erann, but my take on this is that beginners who get
  confused about this will remain confused for a few days, and then get it
  or get over it, as in: not worrying about it even if they don't get it.

  if we change the semantics of the language from what said confused people
  will find described in textbooks and other reference materials and when
  searching the net, the number of days of confusion can only increase, not
  the least because half the vendors will think this is a lame idea and not
  implement it, and the other half will do it better than the lame code and
  so the only thing we will succeed in is in destroying a very powerful
  mechanism in Common Lisp that every other language is sadly lacking:
  transparent, safe, and convenient global, dynamic variables.  all for the
  purported, but obviously unrealizable benefit of reducing the number of
  confused people and their posting frequency to comp.lang.lisp.

  still, it would be nice if we had some simple programmatic access to the
  specialness of a symbol.  this would have been covered by the environment
  access functions that were not included in the standard.

[...elided excellent reasons for not changing the sematics of the language...]

Best Regards,

:) will

I'd also like to gratuitously point out that your technique doesn't
actually solve the whole problem: symbol-macros can mean that what
looks like an innocent variable is actually something entirely
different.

I think a much more useful meta-solution to this problem is better
environmental information, and a compiler MOP and/or code walker.
Given that I can instrument the compiler in such a way that it will
warn me if it ever sees a special variable that does not adhere to
whatever naming convention I choose, or in general check any other
thing I like about the code I'm compiling.

And this solution has the advantage that it's useful for other stuff
too!

A little help from the programming environment would go a long way to
solving the problem.  Wrapping '*' around a variable is the
de-facto standard for special variables.  If the compiler issued a
warning when you declared or used a special variable without '*', or
if you declared or used a non-special (lexical) variable _with_ a '*',
it would probably solve the bulk of the problem without introducing
changes to the language.

in CLISP:
(defvar x)
(system::special-variable-p 'x) ==> T

BTW does anyone
know of a reference that has a GOOD description of hopefully
general methods to come up with a naming standard for symbols,
variables, functions etc.?  Is there something approaching a standard.

My best coding experience was one in C, where the name of a variable or
function told you in what file it was define, whether it was global,
or a constant.  It didn't include type, but that wasn't too much of
a problem since almost every thing was an Integer, and on the
DSP we were using short, int, long integer were all 32 bits and
strings only existed in 2 out of 200 files.  

Muddy

- Use * to surround global special variables
- Use + to surround global constants
- Let yourself be guided by the (newer parts of the) ANSI CL standard
  for ideas on how to name accessors, functions and variables.

Regs, Pierre.

Muddy

I don't think it's a bad enough problem to justify changing the
language to solve it, but it's a real problem, even for real,
experienced programmers. I recently finished renaming
most of the special variables in the SBCL implementation of
Common Lisp to use the *FOO* convention. It was a considerable
amount of work. And even for the variables which use this convention,
I still have to worry about subtly breaking something a la
  (DEFVAR *FOO*)
  (DEFUN BAR (..)
    (LET ((*FOO* (..)))
       (BLETCH *FOO*)
       (ZUT)
  ;; Don't move the DEFVAR to here, or you will lose.
  (DEFUN ZUT () ; which I hope isn't something obscene in Italian:-)
    (IF *FOO* (ZUT1) (ZUT2)))
when *FOO*, BAR, and ZUT are defined in different files and
the order of compilation might be rearranged. I'd need to be
fairly unlucky to rearrange just the right things in just the right
way to silently break something like this, but it could happen.

By and large, Common Lisp does a good job of letting you define things
in any order, which is a good thing. By and large, when you can't
define things in any order (e.g. DEFCLASS) at least it gives you
an error, which is a good thing. But the problem I described just
above won't give you an error in any implementation I'm aware of,
which is nasty.

(By the way, I've considered making SBCL issue STYLE-WARNINGs
for any SPECIAL use of non-*FOO*-style symbols, and any non-SPECIAL
use of *FOO*-style symbols, to detect problems like this. But I'm
a little uncomfortable embedding informal naming conventions in the
compiler, so I've avoided doing this so far. Does anyone have
any opinions on whether such STYLE-WARNINGs would be The Right
Thing?)

  Bill Newman

Regs, Pierre.

The $ macro is optional.  So is the BIND macro.  The core of my
proposal is to move the specification of dynamic bindings out of
the declarations and into the lambda list.  It's a 100% backwards-
compatible change, and it doesn't change the semantics of the language
at all.  It's also very much in the spirit of Common Lisp.  All I am
saying is that we should extend lambda-list syntax to allow variables
whose names are lists beginning with SYMBOL-VALUE (or SPECIAL or
DYNAMIC, I don't really care which keyword we choose).  It's much
like extending the syntax of function names to include lists whose
first element is SETF.

BTW, all the keywords in the BIND macro are optional except IN, so you
can dispense with all the = signs and ANDs and just write:

(bind x 1 (symbol-value q) 2 (values y z) (foo) in ...)

It's also pretty simple to change BIND to use parens instead of a
keyword to separate the bindings from the body:

(bind (x 1 (symbol-value q) 2 (values x z) (foo)) ...)

or:

(bind ( (x 1)
        ((symbol-value q) 2)  ; implied (declare (special q))
        ((values y z) (foo))  ; multiple-value-bind
        ((a (symbol-value b) c) (baz)) )     ; destructuring (with a special)
  ...

Without BIND this code becomes:

(let ( (x 1)
       (q 2) )
  (declare (special q))
  (multple-value-bind (y z) (foo)
    (destructuring-bind (a b c) (baz)
       (declare (special b))

Personally, I think:

   (bind x 1 $q 2 (values y z) (foo) (a $b c) (baz) in ...

is a win.  Notice also that you can implement this without a reader macro.
BIND could simply declare special those variables whose print names start
with $, or whose print names start and end with *, thus:

  (bind ( (x 1) (*y* 2) ) ; *Y* is automatically declared special

We are already using a typographical convention to distinguish lexical and
special variables.  We already know that we get into trouble when we don't
adhere to that convention.  Why not move the burden of applying that
convention from the programmer to the compiler?

I also wonder how many of you BIND haters are also LOOP haters.  I am
really beginning to worry that people have lost sight of the fact that
an S-expression with just one level of parens is still an S-expression.

I also think that if you are going to make a change to the language
like this, you should think it out a good deal first.  For instance my
suggestion of better environmental access would enable you to detect
these kinds of problems yourself, but it would also enable a whole
bunch of other cool stuff.

If you want to stick to souping up lambda lists, why just fix this one
issue?  Why not let general declarations be in there? Step back from
the problem.

--tim

Muddy

  I see no evidence of that.  I think we have a fairly good understanding
  of your issue.  all the silly syntax detracted from its delivery, but
  you've been clear enough.

| We are already using a typographical convention to distinguish lexical
| and special variables.  We already know that we get into trouble when we
| don't adhere to that convention.  Why not move the burden of applying
| that convention from the programmer to the compiler?

  because we would want to make our own conventions in code where it is
  safe to make them.  and people don't _actually_ get into trouble in the
  first place, except the first few days they are confused about this.

| I also wonder how many of you BIND haters are also LOOP haters.  I am
| really beginning to worry that people have lost sight of the fact that
| an S-expression with just one level of parens is still an S-expression.

  no need to worry about that.  I like loop, I don't if*, and I don't like
  your bind or any of the numerous other _gratuitous_ syntax-heavy ideas
  people seem to get with an alarming regularity whenever they see a need
  for some miniscule improvement to the language.  it's as if they don't
  like simple syntax to begin with, and rush to solve any semantic issue
  with syntax.  I find this disturbing, but nonetheless indicative of
  something much more important: the language changers don't really grok
  Common Lisp.

| But symbol macros can only be established with lexical scope, so if
| there is a symbol macro it must be lexically manifested.  Global symbol
| macros would be truly problematic.

  take a look at define-symbol-macro some day and weep, then.  I love the
  fact that we have global symbol macros!  I also love the fact that we
  have constants, which also causes pervasive differences in behavior, and
  there's no truly established convention for them.  I happen to think that
  special variables is one of Common Lisp's truly great idea.  what I want,
  in response to your "it's EVIL!" is a programmatic means to query the
  system for the status of a symbol.  `describe' and friends help me as a
  user who gawks at the screen, but that is clearly insufficient.

  again, there is a need to change the language to make it more amenable to
  beginners and experienced users alike: unlike all other languages now in
  current and widespread use, Common Lisp violates the notion that what you
  see is what you get with respect to the _names_ of the symbols.  if
  symbol-value is such a problem for beginners that it needs language-smith
  attention, wouldn't you be interested in solving a _real_ problem that
  would have far-reaching consequences for our interoperability with other
  languages, other textbooks, other people?  it's not a question of case,
  it's a matter of making (setf (readtable-case *readtable*) :preserve)
  work the way people _actually_ expect it to.  think about it.  please.

  I maintain that this is a beginner problem, only.  real Lisp programmers
  don't call their global variables "X".  real Lisp programmers use
  packages if they want their symbols to stay of other people's face.  real
  Lisp programmers know about unintern, too.

| I'd say this is a design flaw that makes them not particularly safe.

  it's a design flaw to you because your notion of safe is wrong.

| (By the way, I've considered making SBCL issue STYLE-WARNINGs for any
| SPECIAL use of non-*FOO*-style symbols, and any non-SPECIAL use of
| *FOO*-style symbols, to detect problems like this. But I'm a little
| uncomfortable embedding informal naming conventions in the compiler, so
| I've avoided doing this so far. Does anyone have any opinions on whether
| such STYLE-WARNINGs would be The Right Thing?)

  I have already said what I think is the right thing here: demand that
  there be lexically apparent declarations that reiterate the special
  status of symbols so declared globally.  lacking such a declaration, you
  might issue a style-warning for free variables even if you know they are
  globally declared special.  it will lead to slightly more verbose code,
  but the excuse to be making invisible, pervasive changes would go away.

| Yes, I also really wish there was a standard way to query "is symbol FOO
| special?" And "what's the value of the optimization property BAR?" too..

  precisely, and this is the _only_ problem worth solving as I see it.

(lambda (x) (lambda (x) (lambda (foo::x) ...

(Actually, we have three variables ONLY if X has not been DEFVARed.
If X has been defvared then we only have two variables.)

When you bind a lexical variable you make a new variable.  When you
bind a special/dynamic variable you make a new binding for an existing
variable.

Specialness can be associated with variables (via SPECIAL declarations)
or with symbols (via DEFVAR) but NOT with names.  Once specialness has
been associated with a symbol, it is no longer possible to create new
variables using that symbol, only new bindings for the (one) special variable
associated with that symbol (also known as the symbol's symbol-value).

The point is that none of this has anything to do with symbol names.
In fact, you can make examples using symbols that don't even have names:

(defmacro weird-lambda (&body body)
  (let ( (arg1 (make-symbol ""))
         (arg2 (make-symbol "")) )
    `(lambda (,arg1 ,arg2) (declare (special ,arg1)) ...

Here there is one lexical binding and one dynamic binding.  They are
associated with two uninterned symbols, neither of which has a name!

Here's an even more pathological example:

(progn
  (defvar #1=#.(make-symbol "") 1)
  (defun #1# () #1#)
  (funcall (lambda (#1#) (#1#)) 2))

This wouldn't work in Scheme ;-)

? (setf (readtable-case *readtable*) :preserve)
:PRESERVE
? 'foo
foo
? 'foO
foO
? 'FOO
FOO
?

Works the way I'd expect it to.

The programmers who wrote the Python compiler (part of CMU CL,
available at http://www.cons.org/cmucl/) weren't beginners. But
they put somewhat over 10K lines of Common Lisp into the "C"
package (the compiler). And they defined special variables with
names like
      C::TOP-LEVEL-LAMBDA-MAX
      C::NO-LOADS
      C::PACK-OPTIMIZE-SAVES
      C::PACK-SAVE-ONCE
      C::MAX-OPTIMIZE-ITERATIONS
      C::PACK-ASSIGN-COSTS
      C::VM-SUPPORT-ROUTINES
      C::PRIMITIVE-TYPE-SLOT-ALIST
      C::NO-COSTS
      C::SYSTEM-CONSTANTS
I'm not trying to claim this was good style, or a good idea. (As I
said in my earlier article, I've since renamed as many special
variables as possible in the *FOO* style.) All I'm trying to claim is
that this example shows that the problem is not confined to beginners.
It was created by experienced programmers, and it bites experienced
programmers now who have to worry about not naming local variables
NO-LOADS. And even now that the special variables have been renamed
in the *FOO* style in my version (SBCL), I still need to worry about
respecting constraints on the order of compilation of code, with
no help from the compiler even to tell me when something goes wrong.

Incidentally, experienced C and C++ programmers can avoid many of the
problems of C and C++, such as
   * using the preprocessor to define constants
   * having to manually free memory resources
   * having wild pointers be able to corrupt the system
But that because these problems mostly trouble beginners (or maintainers
of large programs written by others) doesn't mean that these problems
aren't shortcomings of C and C++. Why should the fact that the special
variable problem mostly troubles beginners (or maintainers of large
programs written by others) mean that this problem isn't a shortcoming
of Common Lisp?

  Bill Newman

There's a real difference between problems which just go away, and
problems that you eventually realise you can live with, but will just
cost you 30% of your time for ever.

The CMUCL thing is a fine example of the former.  Whoever did that was
being silly, and there's a once off fix involving DO-SYMBOLS and
tags-query-replace in emacs which really should not take that long.

  but your example doesn't show any such thing!  that people are not
  following the asterisk convention is _not_ the problem.  the problem as
  stated is that there's a lambda list or let binding somewhere with a
  symbol in it that has special binding which comes as a _surprise_ to
  people who are moderately (but not overly) intimate with the code.  do
  you have evidence of that, or is this only more unfounded fear that there
  _might_ be a problem?  incidentally, I recognize that there _might_ be a
  problem, the solution to which is to make the system easier to query for
  such information.  I don't see any other problems that need solving.

  my philosophy, in case you need to have it stated to avoid speculating
  about it, is that in order to serve the needs of any community, one must
  never, _ever_ to cater to the needs of ignorants and novices except in
  carefully controlled settings where the express purpose is to make them
  non-ignorants and non-novices, such as school or training courses.