In core structure editor, anyone?

[Simon Brooke]

I note people seem mainly to be using Emacs as an editing/development
environment for Clojure. But as people keep pointing out, Clojure is
homoiconic; the canonical source of a function is not a stream of
bytes read from a file, but is a structure in core.

Now, in-core editing does raise some issues - there needs to be some
means of associating the structure which is the canonical source of a
function with the function itself, so that when the user invokes (edit
froboz) (or (edit 'froboz), or whatever), the source structure for
froboz is displayed in the editor. But in-core structure editors are
extremely powerful and useful when editing homoiconic languages, so...
is anyone working on an in-core editor for Clojure?

[Stuart Sierra]

On Dec 10, 7:15 am, Simon Brooke <still...@googlemail.com> wrote:
> But in-core structure editors are
> extremely powerful and useful when editing homoiconic languages, so...
> is anyone working on an in-core editor for Clojure?

Not that I've heard, but Emacs + Paredit <http://www.emacswiki.org/cgi-
bin/wiki/ParEdit> is a powerful combination. Paredit is like a
structured editor for Lisp expressions. There's a patch to handle
Clojure data structures: <http://github.com/jochu/clojure-mode/tree/
master/clojure-paredit.el>

-Stuart Sierra

[Scott Fleckenstein]

Seems like having something like this would be a good step towards
supporting image-based development similar to Smalltalk. Whether that
is a good thing or not is a different discussion ;)

-Scott Fleckenstein

[Chouser]

On Wed, Dec 10, 2008 at 7:15 AM, Simon Brooke <stil...@googlemail.com> wrote:
>
> I note people seem mainly to be using Emacs as an editing/development
> environment for Clojure. But as people keep pointing out, Clojure is
> homoiconic; the canonical source of a function is not a stream of
> bytes read from a file, but is a structure in core.
>
> Now, in-core editing does raise some issues - there needs to be some
> means of associating the structure which is the canonical source of a
> function with the function itself

There is sufficient information provided in most Var metadata to get
the source code where the Var defined (assuming the source code is
available in the classpath).

I'm working on (or perhaps more accurately, poking at) a small program
to leverage Swing to provide an environment for editing Clojure and
interacting with a running REPL. It's by no means emacs or even vim
yet, but it will have no dependencies beyond what Clojure requires and
may provide a sufficiently comfortable environment for spot-editing.
It will also attempt to be an acceptable starting place for people
wanting to play with Clojure who aren't too deeply committed to emacs
or vim. It already has an 'open' function to pull up the source of a
function in the edit pane.

I'm not ready to provide a link yet, but go ahead and ask me in a few
weeks how "textjure" is coming along.

--Chouser

[falcon]

Could you describe in-core editing a bit more? Sounds interesting.

On Dec 10, 7:15 am, Simon Brooke <still...@googlemail.com> wrote:

[J. McConnell]

On Wed, Dec 10, 2008 at 3:35 PM, Chouser <cho...@gmail.com> wrote:
>
> It's by no means emacs *or even vim* yet

Ouch, that hurts ;)

- J.

[Chouser]

I'm a vim user, but although I don't envy the multitude of
configuration issues it sound like emacs/slime/swank users seem to
have, I'm certainly a bit envious of the deep integration between repl
and editor.

I wish Gorilla all the best as well.

--Chouser

[J. McConnell]

On Wed, Dec 10, 2008 at 4:06 PM, Chouser <cho...@gmail.com> wrote:
>
> I'm a vim user, but although I don't envy the multitude of
> configuration issues it sound like emacs/slime/swank users seem to
> have, I'm certainly a bit envious of the deep integration between repl
> and editor.
>
> I wish Gorilla all the best as well.

Agreed on all counts ... still, the truth sometimes hurts :)

- J.

[Drew Olson]

I had the same response to the initial email :) I've actually had a really great experience with gorilla so far, hope it keeps moving forward.

- Drew

 

[RZez...@gmail.com]

On Dec 10, 3:59 pm, falcon <shahb...@gmail.com> wrote:
> Could you describe in-core editing a bit more?  Sounds interesting.
>

+1

[John Newman]

Some Noobian critique:

Thread-local bindings also allow us to monkey-patch for the span of a local context. Say we have a function cat which calls a function stored in a Var; if a function goat is root-bound to the Var, then cat will normally call goat; however, if we call cat in a scope where we have thread-locally bound a function moose to that Var, then cat will invoke moose instead of goat. (The key idea here is that the function call is bound to a Var, not directly to a function, so the actual function called is up for grabs every time the call is executed.)

 You lost me at "span of a local context."

I pretty much understand what you're saying, since I've been playing around with clojure for a while and "thread-local bindings" is somewhat self-explanatory, if a little bit cryptic.

Basically, a var is a container.  Different threads can have vars (containers) of the same name but with different values (contents).  If a var is called but not found in a thread, clojure will search the global/root bindings/namespace.

Is that accurate?

[evins...@gmail.com]

On Dec 10, 2:59 pm, falcon <shahb...@gmail.com> wrote:
> Could you describe in-core editing a bit more?  Sounds interesting.

The canonical structure editor (not "structured editor") is probably
Interlisp-D's SEDIT, or its predecessor DEDIT. (See
http://bitsavers.org/pdf/xerox/interlisp/3102300_interlDprimer_Nov86.pdf
for an archive of Interlisp docs, including docs on SEDIT.) Interlisp
was for a while one of a couple major dialects of Lisp, along with
MACLISP, that were perhaps the most widely used dialects. There were
lots of dialects of lisp in fairly wide use; Common Lisp was intended
to merge the most common idioms in use across a lot of dialects
without losing *too* much of the facilities of any one popular
dialect, but Common Lisp might perhaps resemble MACLISP a little more
than it resembles Interlisp.

Interlisp was the system programming language for the Xerox D-machine
series.

A structure editor, as exemplified by SEDIT, does not operate on text
in a file buffer; instead, it operates on the lisp data structures
represented by the text. When you edit lisp code in emacs, even with a
utility like Paredit, which helps you with the syntactic structure,
you are operating on text characters stored in some sort of array.
There is a connection between the text and what the lisp reader
produces in memory, but you operate on the text, not on the in-memory
lisp objects that are represented by the text.

Structure editors are different: they operate on the in-memory data
structures.

Consider a lisp (or Clojure) list, (a b). This list is a chain of cons
cells, where a cons cell is a pair of pointers. The left pointer
points to the symbol 'a'; the right one points to a second cons cell.
In the second cons cell, the left pointer points to the symbol 'b' and
the right one points to nil.

Now consider the text "(a b)", as seen by some editor. It's a row of
character values: '(', 'a', ' ', 'b', ')'.

The lisp (or Clojure) reader converts that row of characters into the
chain of cons cells described above; the printer converts the chain of
cons cells into the row of characters. There is necessarily a close
correspondence between the text and the in-memory objects, but they
are not the same thing.

Text editors edit text. Structure editors edit the in-memory data
structures.

This might be a good place to repeat that in lisp, and perhaps in
Clojure, I'm not sure, the code is not the text; it's the in-memory
objects. This differs from most programming languages. In most
languages, when you say "source code," you mean text. Strictly
speaking, the text in a ".lisp" text file is not the source code for a
lisp program; it is a text description of lisp source code; the actual
source code is constructed by the lisp reader when it reads the file.
Indeed, given the syntactic rules of Common Lisp, it's possible to
have two very different text files that contain (text specifications
for) exactly the same source code.

Most of the time, this distinction is pretty much academic, because of
the close and necessary correspondence between lisp source code and
its textual representation. There are a few places where the
difference is noticeable, though, and some of them illustrate how a
structure editor differs from a text editor.

For example, consider a circular list: suppose we used RPLACD to make
the nil at the end of (a b) point at the first cons cell, the one
whose left side points to the symbol 'a'. If that value is returned at
the repl (and *print-circle* is not configured to inhibit it) then the
printer will be stuck in an infinite loop printing (a b a b a b a
b....

Now, of course, the circular data structure is not infinite, it's just
circular; it points to itself. But the textual representation is a
little inconvenient for printing such a structure, and indeed, it's
inconvenient for constructing it in the first place (the Common Lisp
reader and printer have a sort of a hack specified in them for these
purposes).

It's quite a bit more convenient to work with such a data structure in
a structure editor, which makes it clear that there isn't really
anything weird or magical about the circular list. It just has an
arrow in the rightmost box of the chain that points to the leftmost
box. No big deal.

Structure editors are not in common use, maybe because, while they're
a valid and maybe cool alternative to text editors for lisp code, it's
not clear how useful they are for other kinds of code. In most
programming languages a lexer and parser construct an abstract syntax
tree from the source code. In lisp, the abstract syntax tree *is* the
source code. That being the case, it makes a lot of sense with lisp to
operate directly on the parse tree. In other languages, not so much.

Some folks love structure editors. I never really used one, but my old
friend and longtime lisp hacker Philip McBride really likes them. I'd
be very glad to see one for Clojure just because it would be fun to
have the option to play around with the code that way.

I don't know whether Rich regards the text of a Clojure program as the
source, or whether he thinks of the source as the data structure
created by the reader when it reads the text. In Common Lisp and some
older dialects, it's pretty clear that the latter is the case.

[Stuart Sierra]

On Dec 11, 9:32 am, "evins.mi...@gmail.com" <evins.mi...@gmail.com>
wrote:

> I don't know whether Rich regards the text of a Clojure program as the
> source, or whether he thinks of the source as the data structure
> created by the reader when it reads the text. In Common Lisp and some
> older dialects, it's pretty clear that the latter is the case.

I think Clojure, by avoiding reader macros and reader side-effects,
actually comes closer to the code-is-data ideal than Common Lisp
does. It might not lure me away from Emacs, but it would be
interesting to see an editor that goes all the way and abandons text
sources altogether.

-Stuart Sierra

[falcon]

Can't say I understood all of it (being a LISP noob) but appreciate
the explanation.

Does the following fit into the current discussion?

Barista editor: http://faculty.washington.edu/ajko/barista.shtml

Barista editor is based on Citrus http://faculty.washington.edu/ajko/citrus.shtml

The citrus page has some _VERY_ cool videos. They are worth watching,
just for entertainment value (entertainment for developers only
though).

On Dec 11, 9:32 am, "evins.mi...@gmail.com" <evins.mi...@gmail.com>
wrote:

> On Dec 10, 2:59 pm, falcon <shahb...@gmail.com> wrote:
>
> > Could you describe in-core editing a bit more?  Sounds interesting.
>
> The canonical structure editor (not "structured editor") is probably

> Interlisp-D's SEDIT, or its predecessor DEDIT. (Seehttp://bitsavers.org/pdf/xerox/interlisp/3102300_interlDprimer_Nov86.pdf

[TNeste]

Do you mean structured editing combined with image based development?
I haven't really looked that closely at the old Interlisp documents
but I suppose the source code is managed the same way it is in most
Smalltalk systems but I have always been a bit puzzled about how that
works with Lisp.

In Smalltalk it is fairly straightforward since there is an obvious
mapping from classes and methods to source code. With lisp it is not
that simple as there is no limit to the syntactic forms that the user
might want to query and edit in different forms. For example, if I
have (define-foo my-foo ...) macro that is used to define some complex
function named my-foo what happens when I do (edit 'my-foo)? Do I get
the original define-foo form or the macroexpansion? What about when
define-foo does something more complex like defines a bunch of methods
and datastructures or binds my-foo to a closure? There needs to be
mechanism that associates evaluated forms with the name(s) of the
resulting object(s) and it probably couldn't, in general, be done
automatically.

[evins...@gmail.com]

Depends. Using a decent lisp environment nowadays, the lisp will
remember the files and line numbers for all the forms it has read, so
you can get those back pretty easily, as long as you don't turn that
feature off, and as long as you have those text files. If you don't
have the original text, but only the data produced from it by the
reader and compiler, then the best you can usually do is some
plausible lisp forms that correspond to what's in memory--generally a
bunch of fully-expanded s-expressions that don't look all that much
like code a human would write. It'll be accurate, but not necessarily
congenial.

In older Lisp systems, like Interlisp and Genera, the systems
remembered absolutely everything about how a given hunk of data was
created, including versions of everything, including mouse-and-window
direct-manipulation operations, so you could view the contents of a
structure editor in any form you found convenient.

[Simon Brooke]

I am indeed looking back at structure editors such as DEdit on
InterLISP - which I rate as the best programmer's editor I've ever
used, superior even to its successor SEdit - but also to the editor in
the versions of Portable Standard LISP I was exposed to. I don't know
what the genesis of those was. I also wrote my own structure editors
for other variants of LISP I worked with - in a REPL on a TTY, a
primitive structure editor is pretty simple to write (you basically
only need a pretty printer). However, the DEdit style editor where you
can see the whole of the function or other data-structure you're
working on while operating on some small part of it is vastly
preferable, but rather more complex to write. Not /that/ difficult,
however, if you can pretty-print source into a Swing textbox.

Chouser, when you have some code you'd be willing to share I would be
extremely interested in poking at it.

And yes, Scott, this is a step towards Smalltalk-style (or old-
fashioned Lisp style) image based development. You may well be right
that this is a bad idea - but we won't know until we try... and in the
end we may disagree.

The next step, of course, it to write SYSOUT and its counterpart,
SYSIN ;-)

[falcon]

How's textjure coming along?

On Dec 10, 3:35 pm, Chouser <chou...@gmail.com> wrote:

[Chouser]

On Tue, Dec 30, 2008 at 1:24 AM, falcon <shah...@gmail.com> wrote:
>
> How's textjure coming along?

Not ready yet. :-/ Current sub-project: setting up sufficiently
flexible system to handle vi-style keybindings.

--Chouser

[chris]

I would be super interested in trying things out with textjure
whenever you are ready.

I don't want to give up emacs + slime, but I do think it is an
interesting concept to connect to a running process and change it.

One other thing that LISP had that was useful was saving the state of
the system. We can't really save the state of the jvm, but we could
re-jar any files with changed code, or perhaps (maybe a better idea)
create add-on jars that overrode old code with new code?

Chris

On Dec 30, 6:46 am, Chouser <chou...@gmail.com> wrote:

[Tom Ayerst]

Time to bug Chouser again ;-)

How's textjure coming along?

2008/12/30 Chouser <cho...@gmail.com>

[Chouser]

On Fri, Jan 9, 2009 at 1:47 PM, Tom Ayerst <tom.a...@gmail.com> wrote:
> Time to bug Chouser again ;-)
>
> How's textjure coming along?

Vaporware is such an ugly word. :-P I've definitely had less time to
work on it over the last month or so than I was expecting.

Discussion from yesterday:
http://clojure-log.n01se.net/date/2009-01-08.html#10:02

--Chouser

[Greg Fodor]

Funny, it's this concept that has drawn me into looking into clojure
more in the
last few days. I worked on a highly structured editor at intentional
software
for some time but at the time had little exposure to LISP.

I feel a modern JVM based LISP such as Clojure with full access to
Java drawing
APIs could provide the platform for a revolutionary structured editing
environment
for both LISP and custom DSLs implemented in LISP. With the generative
techniques
of LISP there should be an easy way for developers to not only build
DSLs like
they currently do in LISP using macros, but also provide a way to
generate and
project those DSLs using domain specific notation in a business-user
friendly way.

This is definitely something I hope to be learning more about and
working on in
the future -- I think Clojure could provide both a watershed moment
for concurrent
programming but also one for domain specific languages/structured
editing for
domain experts.

[roman]

Here is a link to an article comparing structure editors with other
approaches to code editing. Seems apropos to this discussion.

http://web.media.mit.edu/~lieber/Lieberary/Softviz/CACM-Debugging/Already-Full/Already-Full.html

Also, anyone wanting to play with Interlisp's structure editor can do
so without too much trouble. Just create a virtual machine (I used
vmware player and it works fine), install damn small linux (with a 2.4
kernel) to it and download the software from:

ftp://ftp.parc.xerox.com/pub/lfg/

*note that the license prohibits exporting the software to among
others, Yugoslavia. So please refrain from travelling back in time and
trafficking in the software*

(it is a piece of software written inside a lisp development
environment which runs among other lisps, Interlisp). Then either
install csh (I didn't) or just modify the startup shell script to use
the included
bash shell.

Now, you'll be able to interactively follow along the Interlisp Primer
mentioned before in this discussion; specifically section "11.3 Using
The List Structure Editor".

On Dec 11 2008, 8:08 am, falcon <shahb...@gmail.com> wrote:
> Can't say I understood all of it (being a LISP noob) but appreciate
> the explanation.
>
> Does the following fit into the current discussion?
>
> Barista editor:http://faculty.washington.edu/ajko/barista.shtml
>

> Barista editor is based on Citrushttp://faculty.washington.edu/ajko/citrus.shtml