About a networked REPL...
Martin Blais
After Rich's suggestion at the recent NYC meetup, I had a
detailed look at inferior-lisp vs. Slime, and nREPL, read
Chas' document, wrote a bit of code, tried to figure out the
issues for myself; here are my conclusions on nREPL:
- In the Slime/Swank model, there is a single IDE that
connections to N possible types of LISPs via a rich
protocol (and a lot of corresponding work on the
workflow/GUI side in Emacs in the various fancy Slime
functions I discussed at the meetup).
Emacs/Slime -> Clojure, sbcl, clisp, Allegro, etc.
My understanding of nREPL is that it inverts the problem
by making M different IDEs able to connect to one type of
LISP VM (Clojure).
Emacs, Eclipse, NetBeans, whatever, etc. -> Clojure
One of the assumptions is that by keeping nREPL simple the
various clients will send whatever necessary Clojure code
they want in order to perform the same common tasks that
the Swank protocol already describes. I think that this is
touted to be "powerful" because you can eval anything you
like, i.e., "just send it some clojure expression you
want". But from my POV, it only serves to move the
complexity from the server/VM (currently in swank-clojure
and the protocol) to the client/IDE.
I think this is wrong, because every single IDE client
will have to adapt itself to the various changes occurring
in the Clojure VM as it evolves. This is just creating M
times the amount of porting problems (for each client). By
keeping the complexity on the server, every IDE that
speaks a common protocol (*) works between versions.
One could argue that a rich client library could be
included that provides similar functionality that is
currently provided by swank-clojure, i.e. the problem is
lessened, because all the clients now share the same
client library, so you port over the client lib and
recompile and that should be enouhg. But what language do
you do this in? Clojure itself? Pure Java? What about the
Emacs-LISP client? Then you have a problem of having
multiple client libraries that need be ported.
- I take another issue at nREPL: if someone is going to
implement a request/response protocol that supports only
generic requests (because the complexity should live in
the client), why bother at all, why not instead just make
it a really, really simple network REPL, without any
protocol at all? e.g. all input means "eval", output is
just output, no ids no nothing.... just a socket that you
could connect to with telnet and have a repl right there,
no protocol. telnet localhost 4005, type "(+ 2 2)" at the
terminal, see "4" appear. That's it.
I think it would be pretty straightforward to extend
Emacs' inferior-lisp-mode to support talking to a socket
like that instead of having to start a subprocess. This
would give you "vanilla" communication with a live Clojure
VM without having to start the VM from a parent process. I
rather like this idea; it's the simplicity you currently
enjoy with inferior-lisp, but over a socket; simple, can't
break.
IMHO if people will bother implementing a request/response
protocol, why not just stick with a rich protocol like
Swank's? It'll just be reinventing the wheel otherwise.
- Another problem with Slime that turns people off is the
weirdness in where the output goes. To detail the
weirdness:
1. Return values get rendered in the Emacs minibuffer
2. stdout of the main thread goes to the *slime-repl* buffer
3. stdout of all the other threads go to the console of
the VM, because they inherit System/out
4. Side-effects from converting return values to string
(i.e., from lazy seqs, e.g., (map println (range 10)))
appear in the minibuffer too, interspersed with the
return values.
I think a fair bit of that weirdness can be addressed
easily:
- I already coded a fix for (4) in my clone of
swank-clojure.
- (2) is normal, that's what you expect.
- (3) can be addressed in 1.3 by setting up a suitable
binding to override *out* so that child threads inherit
that that instead of the default *out*. The stream
object that would replace *out* to should send a swank
message back to emacs to have the output printed in the
repl.
Note that this isn't going to work always; pure Java
code running in child threads could still output to
System/out, the console (is there any way to hijack
System/out in Java? I spent 10 minutes on it, it looks
like not). Also, nREPL and even the simplistic
telnet-like networked repl would suffer from the same
problem.
- (1) could be changed by adapting swank-clojure to return
the result as an output command from swank, and to
always return 'nil' as the result. IMHO this could be an
option to Swank--I personally find it neat to have two
separate places to print stdout and the result of the
expression, and the slime repl vs. the minibuffer
provide that, but I can see how that's a little weird.
Those changes would make Slime behave a lot more like
inferior-lisp-mode; a lot less strangeness, one place for
all output, the *repl* buffer.
I'm curious to hear people's thoughts. I don't have a ton of
experience with Clojure and I also might be misled regarding
the aims of the nREPL project. What's bothering people with
Swank's protocol anyway?
In any case, I think it would be really cool to implement a
simplistic networked repl and adapt inferior-lisp-mode to
work with it (without a protocol), and fix the couple of
warts in Slime so that at least output doesn't go to three
different places.
--
M.
(*) Please note: I am aware that only Emacs supports the
Swank protocol right now but I don't see why other IDEs
couldn't support it too--it's just made up of LISP forms
after all; in other words, if someone wants to replace
the swank protocol by an equivalent one with a different
encoding I don't really see the point. Anyway, for this
discussion I'll assume emacs-rex is a suitable enough
protocol; in fact, I think it probably is, because it
has already proved itself with a large number of LISP
environments, it has "history". In any case, that's a
separate discussion. I think of emacs-rex as just "the
protocol" and that other IDEs could implement it to talk
with swank-clojure.
Sean Corfield
> I'm curious to hear people's thoughts. I don't have a ton of
> experience with Clojure and I also might be misled regarding
> the aims of the nREPL project. What's bothering people with
> Swank's protocol anyway?
I have zero experience with Emacs (at least for Clojure - I last used
Emacs in the 80's and it looked much the same when I looked at it
briefly last year). I have no familiarity with Slime/Swank either. To
me, all that stuff (Emacs / Slime / Swank) seems to be tied together
and harks back to older Lisps - no offense intended to folks who live
and die in Emacs: it just doesn't feel like a good Java IDE to me.
I use Eclipse for all my development because I work entirely in a Java
environment (even the web scripting language I use these days, CFML,
compiles down to JVM bytecode and runs alongside all my other JVM
languages). My current Clojure IDE of choice is therefore CCW +
Eclipse. CCW uses nREPL.
As far as I understand it, none of CCW, Enclojure or LaClojure use
Slime/Swank? That means that all of those would need to implement the
rich protocol you mention in order to support Slime/Swank... Given
that nREPL is now a Clojure Contrib project -
https://github.com/clojure/tools.nrepl - I would imagine it would be
more likely to see IDE tools embracing that?
I guess I would need to be persuaded that either a) Slime/Swank is so
unassailably good that all IDEs should implement it or b) there are
inherent problems with nREPL and no IDEs should use it?
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/
Railo Technologies, Inc. -- http://www.getrailo.com/
"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)
Sean Corfield
> I guess I would need to be persuaded that either a) Slime/Swank is so
> unassailably good that all IDEs should implement it or b) there are
> inherent problems with nREPL and no IDEs should use it?
As a follow on, the "Why another REPL implementation?" section of
https://github.com/clojure/tools.nrepl is good reading (and confirms
my suspicions about non-Emacs IDEs and Slime/Swank).
Gary Schiltz
On Mar 19, 7:30 pm, Martin Blais <bl...@furius.ca> wrote:
> (*) Please note: I am aware that only Emacs supports the
> Swank protocol right now but I don't see why other IDEs
> couldn't support it too--it's just made up of LISP forms
> after all; in other words, if someone wants to replace
> the swank protocol by an equivalent one with a different
> encoding I don't really see the point. Anyway, for this
> discussion I'll assume emacs-rex is a suitable enough
> protocol; in fact, I think it probably is, because it
> has already proved itself with a large number of LISP
> environments, it has "history". In any case, that's a
> separate discussion. I think of emacs-rex as just "the
> protocol" and that other IDEs could implement it to talk
> with swank-clojure.
SBCL - see www.bitfauna.com/projects/cusp. It appears
to have no activity since 2009, and was forked off to
lispdef - see bitbucket.org/skolos/lispdev. That project
also seems rather abandoned, or at least no activity
for six months. Looking at the source could be helpful
in trying to adapt swank.
Kevin Downey
and the "integer" that prefixes messages is really just a variable
length string and is not useful for allocating buffers to receive data
in a client because it is a lines / 2 instead of a byte count. this
makes writing a client that uses anything but a BufferedReader
challenging. I am not advocating the slime protocol, but at least the
slime protocol prefixes message with a fixed width representation of
the count of bytes in the rest of the message.
I have been toying with a slime<->nrepl and using nio and polling to
make it all run in single thread, the slime protocol is very easy to
process like this, while I haven't been able to figure out a good way
to parse the nrepl protocol without having a thread continuously loop
around reading lines from the socket.
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?
Chas Emerick
Thank you for the pushback. :-)
There's no reason why the server couldn't provide a standard set of introspection / tooling utilities available -- it would be as easy as adding a proper dependency e.g. in your web project's pom/project.clj. A few people have been discussing the desired capabilities, and have started documenting desired semantics, and where each capability has already been implemented.
> One could argue that a rich client library could be
> included that provides similar functionality that is
> currently provided by swank-clojure, i.e. the problem is
> lessened, because all the clients now share the same
> client library, so you port over the client lib and
> recompile and that should be enouhg. But what language do
> you do this in? Clojure itself? Pure Java? What about the
> Emacs-LISP client? Then you have a problem of having
> multiple client libraries that need be ported.
If you want to load in the aforementioned standard set of utilities from the client side (perhaps if the nREPL server doesn't have them available already), doing so from any client should be straightforward. It's just loading Clojure code, after all. No porting.
> - I take another issue at nREPL: if someone is going to
> implement a request/response protocol that supports only
> generic requests (because the complexity should live in
> the client), why bother at all, why not instead just make
> it a really, really simple network REPL, without any
> protocol at all? e.g. all input means "eval", output is
> just output, no ids no nothing.... just a socket that you
> could connect to with telnet and have a repl right there,
> no protocol. telnet localhost 4005, type "(+ 2 2)" at the
> terminal, see "4" appear. That's it.
>
> I think it would be pretty straightforward to extend
> Emacs' inferior-lisp-mode to support talking to a socket
> like that instead of having to start a subprocess. This
> would give you "vanilla" communication with a live Clojure
> VM without having to start the VM from a parent process. I
> rather like this idea; it's the simplicity you currently
> enjoy with inferior-lisp, but over a socket; simple, can't
> break.
Because a synchronous client is unsuitable for REPL tasks that require or desire asynchronous evaluation of expressions.
> IMHO if people will bother implementing a request/response
> protocol, why not just stick with a rich protocol like
> Swank's? It'll just be reinventing the wheel otherwise.
Swank is fundamentally emacs-only, and is defined outside of the Clojure community. I discuss the issues with swank in the readme here: https://github.com/clojure/tools.nrepl
> - Another problem with Slime that turns people off is the
> weirdness in where the output goes. To detail the
> weirdness:
How SLIME works (or its problems) is entirely orthogonal to network REPL design IMO.
nREPL breaks up *err*, *out*, and a printed representation of expression results into separate slots in its responses, so clients can display response data however is appropriate for their environment. Note that I'm planning on adding optional multiplexing of System/out and System/err (much like is done in Cake) to nREPL so that clients can subscribe to those streams as desired.
> (*) Please note: I am aware that only Emacs supports the
> Swank protocol right now but I don't see why other IDEs
> couldn't support it too--it's just made up of LISP forms
> after all; in other words, if someone wants to replace
> the swank protocol by an equivalent one with a different
> encoding I don't really see the point. Anyway, for this
> discussion I'll assume emacs-rex is a suitable enough
> protocol; in fact, I think it probably is, because it
> has already proved itself with a large number of LISP
> environments, it has "history". In any case, that's a
> separate discussion. I think of emacs-rex as just "the
> protocol" and that other IDEs could implement it to talk
> with swank-clojure.
Again, the nREPL readme goes into the motivation for discounting swank. As far as I can tell, its big advantage is that it is what SLIME uses.
I've said this before at ClojureNYC, in irc, and elsewhere…I'm not an emacs/SLIME user, so I'm perhaps the last person those who do use those tools should listen to. That said, it seems to me that moving away from SLIME and putting together an emacs toolchain for Clojure that is *built for Clojure* could yield a lot of benefits, assuming compatibility with Common Lisp environments is unimportant to you. Clojure isn't just another Lisp -- while it's great that emacs users have been able to use pre-existing Lisp tooling with Clojure, that's surely not an optimal situation (even leaving aside what, from an outside observer, seems to be a lot of pain associated with breakage due to upstream changes and such). If you were to consider what a dedicated Clojure toolchain for emacs would look like, I'd be very surprised if it involved swank.
Cheers,
- Chas
Chas Emerick
On Mar 20, 2011, at 3:13 AM, Kevin Downey wrote:
> nrepl's protocol is also very line reader centric, which is a drag,
> and the "integer" that prefixes messages is really just a variable
> length string and is not useful for allocating buffers to receive data
> in a client because it is a lines / 2 instead of a byte count. this
> makes writing a client that uses anything but a BufferedReader
> challenging. I am not advocating the slime protocol, but at least the
> slime protocol prefixes message with a fixed width representation of
> the count of bytes in the rest of the message.
>
> I have been toying with a slime<->nrepl and using nio and polling to
> make it all run in single thread, the slime protocol is very easy to
> process like this, while I haven't been able to figure out a good way
> to parse the nrepl protocol without having a thread continuously loop
> around reading lines from the socket.
My apologies for not getting back to you privately about this earlier. It's been a hell of a week.
When I was designing the protocol, my aim was to make it simple enough to implement from, e.g., Python, and able to be hoisted up onto something like STOMP with relative ease. Thus, line orientation and all-strings made a lot of sense.
I would not pretend to be an expert at designing network protocols. My question would be: in what context is allocating response buffers efficiently a must-have?
- Chas
Kevin Downey
http://docs.python.org/library/socket.html you see it passes in the
number of bytes you wish to receive on a call to the receiv method on
a socket. With that in mind parsing nrepl messages becomes a huge
pain. At no time when parsing a nrepl message do you know how many
bytes you need. The messages start with a string representation of a
number (variable width) and the rest of the message is some set of new
delimited strings.
I thought you were just advocating for ditching slime because it's not
clojure centric enough, how does python fit into this?
Stuart Sierra
Kevin Downey
Messages used in wire protocols exist on a continuum between fixed
width and variable width. The happy medium there, which almost all
protocols follow is a fixed width header that also provides the bye
count of the following variable width body. The nrepl protocol appears
to have aimed at that, but missed.
Fixed width messages have the advantage of being extractable from the
byte stream without having to do any parsing, while with a variable
width message format you must parse and extract messages from the
stream in the same step.
After trying to write a nrepl client it is fairly obvious to me that
in the creation of the protocol similar protocols were examined and
because they were prefixed by numbers, the nrepl messages were,
without a clear understanding of why and what the purpose of the
number is. As implemented the number may as well be replaced with some
kind of START and END tag.
Chas Emerick
> I am no python programmer, but if you look at
> http://docs.python.org/library/socket.html you see it passes in the
> number of bytes you wish to receive on a call to the receiv method on
> a socket. With that in mind parsing nrepl messages becomes a huge
> pain. At no time when parsing a nrepl message do you know how many
> bytes you need. The messages start with a string representation of a
> number (variable width) and the rest of the message is some set of new
> delimited strings.
FWIW, a hello-world-level interaction with an nREPL server from python:
https://gist.github.com/de3b8d0ecdccf6655a63
You don't need to know how many bytes you need when parsing an nrepl message, which would seem to be an advantage to me.
> I thought you were just advocating for ditching slime because it's not
> clojure centric enough, how does python fit into this?
I've never advocated "ditching slime", I've been advocating for a network REPL suitable for use by any and all Clojure tooling. My mentioning python was just an example of a non-Clojure runtime from which one might want to talk to a Clojure/JVM process running a network REPL server.
- Chas
Chas Emerick
In any case, my objective with nREPL was to get something working well that had what I thought were the right semantics for the use cases I was concerned with (i.e. point-to-point Clojure tooling backends). Lifting those semantics onto other transports in the future shouldn't be difficult (I think Rich was pretty unhappy with the protocol as well -- IIRC, he would have preferred using STOMP directly, but it's important to some users of nREPL that external dependencies be minimal/nonexistent, and a Clojure STOMP broker has yet to come along).
Is the protocol as-is a blocker for your attempt to build a SLIME client for nREPL?
- Chas
James Reeves
> In any case, my objective with nREPL was to get something working well that had what I thought were the right semantics for the use cases I was concerned with (i.e. point-to-point Clojure tooling backends). Lifting those semantics onto other transports in the future shouldn't be difficult (I think Rich was pretty unhappy with the protocol as well -- IIRC, he would have preferred using STOMP directly, but it's important to some users of nREPL that external dependencies be minimal/nonexistent, and a Clojure STOMP broker has yet to come along).
FWIW, my personal preference would be construct a protocol for passing
around Clojure data-structures securely over a socket, and then build
your REPL protocol on top of that.
For instance, you could print your data structure to a string, then
encode it as a netstring (http://cr.yp.to/proto/netstrings.txt):
"12:{:foo \"bar\"},"
=> {:foo "bar"}
I'm not sure we need a protocol like STOMP, because there doesn't seem
to be much streaming involved in a REPL.
- James
blais
> Martin,
>
> Thank you for the pushback. :-)
> On Mar 19, 2011, at 8:30 PM, Martin Blais wrote:
> > I think this is wrong, because every single IDE client
> > will have to adapt itself to the various changes occurring
> > in the Clojure VM as it evolves. This is just creating M
> > times the amount of porting problems (for each client). By
> > keeping the complexity on the server, every IDE that
> > speaks a common protocol (*) works between versions.
>
> There's no reason why the server couldn't provide a standard set of
> introspection / tooling utilities available -- it would be as easy
> as adding a proper dependency e.g. in your web project's
> pom/project.clj.
a different protocol to support the same kinds of operations. I don't
really see the point, unless the current protocol is unsuitable
somehow.
> A few people have been discussing the desired capabilities, and
> have started documenting desired semantics, and where each
> capability has already been implemented.
> > - I take another issue at nREPL: if someone is going to
> > implement a request/response protocol that supports only
> > generic requests (because the complexity should live in
> > the client), why bother at all, why not instead just make
> > it a really, really simple network REPL, without any
> > protocol at all? e.g. all input means "eval", output is
> > just output, no ids no nothing.... just a socket that you
> > could connect to with telnet and have a repl right there,
> > no protocol. telnet localhost 4005, type "(+ 2 2)" at the
> > terminal, see "4" appear. That's it.
>
> > I think it would be pretty straightforward to extend
> > Emacs' inferior-lisp-mode to support talking to a socket
> > like that instead of having to start a subprocess. This
> > would give you "vanilla" communication with a live Clojure
> > VM without having to start the VM from a parent process. I
> > rather like this idea; it's the simplicity you currently
> > enjoy with inferior-lisp, but over a socket; simple, can't
> > break.
>
> Because a synchronous client is unsuitable for REPL tasks that
> require or desire asynchronous evaluation of expressions.
necessary for rich interaction and I understand why. What I'm saying
above is that a dead simple option with no async support is also
really valuable, because it is unlikely to break between versions, and
it's already good enough for a lot of users via stdin/stdout pipes
(i.e., inferior lisp).
> > IMHO if people will bother implementing a request/response
> > protocol, why not just stick with a rich protocol like
> > Swank's? It'll just be reinventing the wheel otherwise.
>
> Swank is fundamentally emacs-only, and is defined outside of the
> Clojure community. I discuss the issues with swank in the readme
> here:https://github.com/clojure/tools.nrepl
to; they state that swank is unsuitable somehow, but it doesn't say
why.
Why is swank an unsuitable protocol?
What are its shortcomings?
What features that it doesn't support require redesigning another
protocol?
(Keep in mind: Swank != Emacs.)
> > - Another problem with Slime that turns people off is the
> > weirdness in where the output goes. To detail the
> > weirdness:
>
> How SLIME works (or its problems) is entirely orthogonal to network REPL design IMO.
enough for a section of folks which may think that it's broken right
now. The problem is not that Slime/Swank is broken, the problem is
that it has a few warts that make it look radically different from a
simple REPL with pipes, which I think is why some people dislike it.
It probably just needs a bit more polishing.
> nREPL breaks up *err*, *out*, and a printed representation of
> expression results into separate slots in its responses, so clients
> can display response data however is appropriate for their
> environment. Note that I'm planning on adding optional multiplexing
> of System/out and System/err (much like is done in Cake) to nREPL so
> that clients can subscribe to those streams as desired.
>
> > (*) Please note: I am aware that only Emacs supports the
> > Swank protocol right now but I don't see why other IDEs
> > couldn't support it too--it's just made up of LISP forms
> > after all; in other words, if someone wants to replace
> > the swank protocol by an equivalent one with a different
> > encoding I don't really see the point. Anyway, for this
> > discussion I'll assume emacs-rex is a suitable enough
> > protocol; in fact, I think it probably is, because it
> > has already proved itself with a large number of LISP
> > environments, it has "history". In any case, that's a
> > separate discussion. I think of emacs-rex as just "the
> > protocol" and that other IDEs could implement it to talk
> > with swank-clojure.
>
> Again, the nREPL readme goes into the motivation for discounting
> swank. As far as I can tell, its big advantage is that it is what
> SLIME uses.
already available in Swank, how it fails to reach the desired goals of
a client/server connection to Clojure, and what those goals should be,
and why simply extending Swank would not work.
I also fail to see how Clojure's differences with other LISPs, however
fundamental, require anything else that Slime already provides or that
could be added with only minor changes. AFAIK minor additions to Swank
could be added--it's open source, and other clients could speak Swank
as well.
Remember: Swank != Emacs.
Swank just transfers LISP data.
> If you were to consider what a dedicated Clojure toolchain for
> emacs would look like, I'd be very surprised if it involved swank.
blais
> really valuable, because it is unlikely to break between versions, and
> it's already good enough for a lot of users via stdin/stdout pipes
> (i.e., inferior lisp).
see "4" printed on stdout. Not good enough for an IDE, but still
useful.
Meikel Brandmeyer
just to drop some comments from someone who has rather heavy
requirements due to the environment he has to live with. But to re-
state what I said before: I'm a pretty extreme case and don't want to
be an obstacle for nREPL or any similar project. Nevertheless you can
learn a lot from VimClojure if you want to do async connections.
I don't know SLIME en detail. So please correct me in case I'm saying
something terribly wrong.
On 21 Mrz., 03:48, blais <martin.bl...@gmail.com> wrote:
> Then you'd just be redoing Swank all over again. You'd be redesigning
> a different protocol to support the same kinds of operations. I don't
> really see the point, unless the current protocol is unsuitable
> somehow.
the server and a possible common IDE backend.
The backend has already been re-implemented by almost every Clojure
IDE out there. So getting things together should be fairly easy.
Although, I'm way behind my schedule to work on this... A common
backend would make things easier for other toolsmiths (example of the
day: bluefish). Independent of how they connect to it.
As for the connection: Vim requires async connections and doesn't come
with a parser generator. So the SLIME/swank protocol is unsuitable in
two ways: it requires a synchronuous connection and is to complicated
to parse.
> Where is this information?
Chas probably refers to this: http://dev.clojure.org/display/design/IDE+tooling+backend
> I think you're missing my point; I understand that request/response is
> necessary for rich interaction and I understand why. What I'm saying
> above is that a dead simple option with no async support is also
> really valuable, because it is unlikely to break between versions, and
> it's already good enough for a lot of users via stdin/stdout pipes
> (i.e., inferior lisp).
besides: it's good enough for a lot of users, but it is not good
enough for me.
> Why is swank an unsuitable protocol?
> What are its shortcomings?
I have to tie my project to some other project with ominous release
cycle based on random CVS (o.O) versions? No, thank you. (Ok. This is
now ranting. But it is the impression what I get from the various SLIME
+swank+emacs+doesn't work threads on this list.)
> What features that it doesn't support require redesigning another
> protocol?
> Swank just transfers LISP data.
transfer Vim data structures.
I'm not getting the idea of this whole thread. It started as some
critical feedback to some idea. Which is a Good Thing to begin with.
But now things starts going downwards. No one has to justify nREPL in
any way here. If you don't want it, don't use it. I'm happy that Chas
puts this together and I plan to support it in VimClojure hopefully
sooner than later. Swank is not an option for me for various technical
(and some non-technical) reasons. If other people are happy with it,
they should use it. Neither has to justify his choice.
Sincerely
Meikel
Kevin Downey
source for the last few days, the protocol is not synchronuous.
swank-clojure is completely built around async message passing (to the
point where it can be difficult to trace an evaluation request and a
response), and it is a mirror of the implementation of slime.el
Laurent PETIT
Because it seems to be opiniotatedly undocumented, for one.
Everytime I tried to get info, the final answer was something like "reverse engineer the protocol".
No, thank you.
Chas Emerick
On Mar 20, 2011, at 10:48 PM, blais wrote:
On Mar 20, 12:16 pm, Chas Emerick <cemer...@snowtide.com> wrote:Martin,Thank you for the pushback. :-)
I'm not pushing back, I'm really just trying to understand...
On Mar 19, 2011, at 8:30 PM, Martin Blais wrote:I think this is wrong, because every single IDE clientwill have to adapt itself to the various changes occurringin the Clojure VM as it evolves. This is just creating Mtimes the amount of porting problems (for each client). Bykeeping the complexity on the server, every IDE thatspeaks a common protocol (*) works between versions.There's no reason why the server couldn't provide a standard set ofintrospection / tooling utilities available -- it would be as easyas adding a proper dependency e.g. in your web project'spom/project.clj.
Then you'd just be redoing Swank all over again. You'd be redesigning
a different protocol to support the same kinds of operations. I don't
really see the point, unless the current protocol is unsuitable
somehow.
Because a synchronous client is unsuitable for REPL tasks thatrequire or desire asynchronous evaluation of expressions.
I think you're missing my point; I understand that request/response is
necessary for rich interaction and I understand why. What I'm saying
above is that a dead simple option with no async support is also
really valuable, because it is unlikely to break between versions, and
it's already good enough for a lot of users via stdin/stdout pipes
(i.e., inferior lisp).
IMHO if people will bother implementing a request/responseprotocol, why not just stick with a rich protocol likeSwank's? It'll just be reinventing the wheel otherwise.Swank is fundamentally emacs-only, and is defined outside of theClojure community. I discuss the issues with swank in the readmehere:https://github.com/clojure/tools.nrepl
You don't. I have read both the nREPL readme and the document you link
to; they state that swank is unsuitable somehow, but it doesn't say
why.
Why is swank an unsuitable protocol?
What are its shortcomings?
What features that it doesn't support require redesigning another
protocol?
(Keep in mind: Swank != Emacs.)
If you are not someone who likes to read documentation, you can avoid 99% of the problems that people have with SLIME by following one simple rule:Always use the latest CVS version of SLIME with the latest development version of the Lisp implementation.Due to the "symbiotic" relationship between SLIME and SWANK, it is not uncommon for changes (in either SLIME or the underlying Lisp) to "break" things.
- assumes an emacs client
- assumes a SLIME client
- assumes a Common Lisp, or at least CL idioms
- defines a fixed set of tooling-related operations (many of which are simply not germane – e.g. restarts and such – while it's not clear how one would support other tooling operations that might be useful given the JVM host)
- includes its own authentication mechanism (ack!)
Again, the nREPL readme goes into the motivation for discountingswank. As far as I can tell, its big advantage is that it is whatSLIME uses.
It does not. There is no detail that describes what functionality is
already available in Swank, how it fails to reach the desired goals of
a client/server connection to Clojure, and what those goals should be,
and why simply extending Swank would not work.
I also fail to see how Clojure's differences with other LISPs, however
fundamental, require anything else that Slime already provides or that
could be added with only minor changes. AFAIK minor additions to Swank
could be added--it's open source, and other clients could speak Swank
as well.
Remember: Swank != Emacs.
Swank just transfers LISP data.
If you were to consider what a dedicated Clojure toolchain foremacs would look like, I'd be very surprised if it involved swank.
So, what features not supported in Swank would be supported?
Chas Emerick
The notion of "passing around Clojure data structures" does get watered down because of the non-sexpr-nature of e.g. *out* and *err*, and it's all message-oriented so as to support asynchronous evaluation from the same client, but that's pretty much what nREPL does. Expressions are sent, results are printed and sent back.
STOMP would be valuable in enabling one to separately control routing of the messages in question. The prototypical example of this is sending a message to a "broadcast" queue to do some computationally-intensive data processing on N Clojure VMs. Stuff like that is baked into STOMP, but would require some nontrivial client orchestration/configuration given how nREPL is designed right now.
- Chas
George Jahad
On Mar 19, 5:30 pm, Martin Blais <bl...@furius.ca> wrote:
> Hi,
>
> After Rich's suggestion at the recent NYC meetup,
the NYC meetup?
James Reeves
> The notion of "passing around Clojure data structures" does get watered down because of the non-sexpr-nature of e.g. *out* and *err*, and it's all message-oriented so as to support asynchronous evaluation from the same client, but that's pretty much what nREPL does. Expressions are sent, results are printed and sent back.
That wasn't quite what I meant :)
You could use a Clojure data structure protocol (henceforth known as
CDSP) to send the just data you want evaluated by the REPL, e.g.
>>> (+ 1 1)
<<< 2
But that's probably too simplistic. Instead, you'd probably encase
messages in a map, e.g.
>>> {:id 1, :eval (+ 1 1)}
<<< {:id 1, :return 2}
Which would allow you to handle more sophisticated streaming stuff as well:
>>> {:id 2, :eval (do (Thread/sleep 1) (print "world"))}
>>> {:id 3, :eval (print "hello")}
<<< {:id 3, :out "hello"}
<<< {:id 3, :return nil}
<<< {:id 2, :out "world"}
<<< {:id 2, :return nil}
My point was more that network protocols should be designed in layers.
Take a streaming protocol like TCP, layer netstrings on top of it so
you have a safe way of passing fixed-length messages. Then use the
Clojure read-string and pr-str functions to encode data.
You then have a very basic protocol that allows you to communicate in
more complex data structures, like maps, vectors and lists, which you
can use as a basis for a more sophisticated protocol.
In essence, we want to build a compound network protocol out of simple
components. You don't have to use netstrings and Clojure data
structures to do it, but they seem like a good fit.
- James
Meikel Brandmeyer
On 21 Mrz., 13:52, James Reeves <jree...@weavejester.com> wrote:
> Then use the Clojure read-string and pr-str functions to encode data.
cannot build clients without read-string. By hard-coding this I'm not
free to encode things as I want. Eg. with the current nREPL I can do:
>>> (with-vim-output (generic-function-returning-a-map))
<<< "{ 'foo': 'bar', 'baz': 5 }"
Which I can just stuff into the eval of my vim. It is a priori not
clear, what a reasonable encoding is. The current nREPL approach
allows to choose an appropriate one by the user.
Sincerely
Meikel
Chas Emerick
I think we're in violent agreement. Here's a sample nREPL exchange from https://github.com/clojure/tools.nrepl:
> [catapult:~] chas% telnet localhost <port-number>
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> 2
> "id"
> "foo"
> "code"
> "(println 5)"
> 3
> "ns"
> "user"
> "id"
> "foo"
> "value"
> "nil\n"
> 3
> "ns"
> "user"
> "out"
> "5\n"
> "id"
> "foo"
> 3
> "status"
> "done"
> "ns"
> "user"
> "id"
> "foo"
It's probably not clear from the listing, but only the first message is being sent; the next three are received. There's really not a lot of sophistication here; indeed, it's even less sophisticated than if the messages were being sent using sexpressions, that that would require a lisp reader and printer (of some diminished capacity, presumably) on the client side.
As Kevin was saying elsewhere, the entry count that precedes each message could be replaced with a terminator, or start and end tags if one so wishes. Those that really, really like sexpressions might use braces:
{"id"
"foo"
"code"
"(println 5)"}
I considered doing this, but thought it better to not imply that the protocol allowed for any ol' readable data structure to be passed along.
- Chas
Kevin Downey
what I meant to imply is that the number of lines is not any better
than START and BEGIN tags, while a fixed width count of bytes (even a
number string padded to a constant number of bytes with zeroes) is
better.
James Reeves
> Then we need a clear definition of the clojure syntax. Otherwise you
> cannot build clients without read-string.
True. I guess bencoded data structures, or JSON inside a netstring
would perhaps be more cross-platform solutions.
- James
James Reeves
> I think we're in violent agreement. Here's a sample nREPL exchange from https://github.com/clojure/tools.nrepl:
Ah, I did look through that REPL exchange, but I misunderstood the
syntax the first time I read it. I now realise that the integer at the
beginning of each exchange refers to the number of key/value pairs
being transferred.
This seems a little odd, as it's halfway between explicitly specifying
the size of the input (as in bencode or netstrings), and using an
explicit delimiter (as in Stomp or HTTP). I can't think of any
advantage to doing it this way, instead of using either a delimiter or
an initial byte-count.
But now I understand the protocol, I can see the gist of what you're
doing is the same as what I had in mind. So we're certainly in
agreement in that respect :)
- James
Kevin Downey
> On 21 March 2011 13:14, Chas Emerick <ceme...@snowtide.com> wrote:
>> I think we're in violent agreement. Here's a sample nREPL exchange from https://github.com/clojure/tools.nrepl:
>
> Ah, I did look through that REPL exchange, but I misunderstood the
> syntax the first time I read it. I now realise that the integer at the
> beginning of each exchange refers to the number of key/value pairs
> being transferred.
>
> This seems a little odd, as it's halfway between explicitly specifying
> the size of the input (as in bencode or netstrings), and using an
> explicit delimiter (as in Stomp or HTTP). I can't think of any
> advantage to doing it this way, instead of using either a delimiter or
> an initial byte-count.
and there are numerous disadvantages, including the inability to
separate possibly blocking extraction of a single message from the
byte-stream from parsing of a single message
Chas Emerick
Kevin was patient enough to explain his use case in irc to me, so I understand better what he's aiming for.
It seems to me that the protocol, as it sits, is probably the easiest possible thing to work with assuming you have character streams available (all implementations of which I'm familiar with provide a convenient read-line operation). This makes sense, as such an idiom is what I was assuming as the best/simplest approach for a non-Clojure client implementation.
However, there appears to be complete unanimity that this is a less-than-optimal approach that will make life harder for some potential client implementers in some contexts. So, I'm willing to change the wire protocol.
I'll read more about netstrings, bencode, and other schemes. In the meantime, those who care should pipe up now as to their preferences and/or points of advice. The only significant preferences I have at the moment are that:
1. The result must be UTF-8, through and through.
2. The result should be trivially implementable in e.g. Clojure/Python/Ruby/bash, given character streams/readers.
3. This be a productive exchange with a notable lack of bikeshedding. ;-)
- Chas
Martin Blais
To summarize everyone's points:
- Several people consider the "transport" to be
orthogonal/separte to the "tooling" or "IDE backend."
a) By transport, it refers to a generic ability to transfer
key-value pairs, with a 'id' key for associating responses
to requests (asynchronous responses).
b) "Tooling," or "IDE backend," is a reference to the library
of functions implemented on top of this. I called this
"rich protocol" in my original email.
* These two layers are one in swank/emacs-rex; some people see
benefits in keeping those separate;
* nREPL aims only at the first goal;
* It's not clear whether people are interested in a common
"tooling" library at this point.
- The document that contains some of the juice I'd been looking
for is
http://dev.clojure.org/display/design/IDE+tooling+backend
(Thanks Meikel.)
* (Chas) A suggestion: adding a link to this document from the
nREPL README, or to the "original design notes" would be
useful. (I only found this later.)
- Swank is unsuitable, because...
* Its development and release cycle isn't well organized and
stable ("use the CVS HEAD")
* It is not managed and developed separately from its client
(Slime)
* It is undocumented (and it won't be)--you need to reverse
engineer it from the source code, which is more painful than
it needs to be
* It is more complicated to parse for non-lisp readers
* Evolving it--if even possible-- would cause too much
friction while breaking existing code for CL users
* Despite this, at least one project did use that approach:
CUSP // www.bitfauna.com/projects/cusp
- Clojure requires support beyond that offered for Common LISP
because...
* Looking up JavaDocs or integrating Java-specific
introspection would be useful
(nothing else so far...)
(Meikel Brandmeyer)
> Why does a synchronuous version break less than an async one? And
> besides: it's good enough for a lot of users, but it is not good
> enough for me.
Why does Slime with Clojure-1.3 not work right now?
(I don't know either: I'd have to dig in--and that's my point.
No time to dig in... though you could argue rightfully I should
try to fix the bug instead of spending this time writing an
email.)
IMO something really, really simple will almost never break,
even between versions; because if you keep it really simple it
never changes.
Being able to telnet to a VM on a remote server--no editor, just
telnet--is useful in real-world cases where you can't install
stuff on boxes; being able to shell in via telnet and eval a few
things is _really_ powerful. Again, that's not to say that that
could replace an asynchronous protocol, but it's a really nice
fallback.
Actually, I don't see why both could not be provided
simultaneously (separate ports).
> As for the connection: Vim requires async connections and doesn't come
> with a parser generator. So the SLIME/swank protocol is unsuitable in
> two ways: it requires a synchronuous connection and is to complicated
> to parse.
Swank is asynchronous, e.g.::
(:emacs-rex
(swank:create-repl nil)
"clojure.tools.nrepl-test" t 4)
(:return
(:ok
("user" "user"))
4)
AFAIK that final "4" in both the request and response is a
request id.
(Chas Emerick)
> In any case, I appreciate public thrashing of design and
> implementation choices (as long as the process is productive).
I apologize if it sounded like thrashing, I didn't mean it that way.
(Also, apologies if my email tone was a bit square BTW, I had
written a much longer and more nuanced one right before it and
it got lost in Gmail or my web browser textarea and I think in
my rewriting I was still a bit miffed at losing a lot of time on
the first one.)
> A few people have been discussing the desired capabilities,
> and have started documenting desired semantics, and where each
> capability has already been implemented.
Is there anything something beyond this page?
http://dev.clojure.org/display/design/IDE+tooling+backend
> The operations that would be provided by a tooling library would
> have zero impact on the nREPL protocol. There's no reason why a
> particular client would have to either (a) use the server's
> provided set of tooling functions or (b) use any particular
> standard set of tooling functions. It's just code that's being
> loaded somewhere, and then called later on -- no protocol changes.
Well, no low-level "protocol" changes that's true, but on top of
it, something higher-level will be built for invoking that
"tooling" library, if anything, just function calls on it, and
though the nREPL transport won't break, access to the tooling
library will likely break on version mismatch.
But I understand that's not the goal of nREPL anyway, so it's beside
the point.
Is there an effort somewhere for a common tooling library?
> apply. After going through the above, I think it's worth asking:
> what about swank is so great that it should wag the dog here?
It's a wheel that has been riding for a while. I've learned
not to ignore "working code", especially "old working code";
it's often too easy to dismiss something and restart from
scratch without looking at it in enough detail (I'm not
saying that this is the case here, but I'd have loved to see
an exhaustive list of swank functions--I think I'll produce
one; if anything that could be useful in helping define a
complete "tooling" library).
> However, there appears to be complete unanimity that this is a
> less-than-optimal approach that will make life harder for some
> potential client implementers in some contexts. So, I'm willing
> to change the wire protocol. I'll read more about netstrings,
> bencode, and other schemes. In the meantime, those who care
> should pipe up now as to their preferences and/or points of
> advice. The only significant preferences I have at the moment
> are that:
Your nREPL protocol is unusual indeed, I haven't seen one like
that before (but I don't think it's bad).
Netstrings are very common in binary protocols. I think for this
application we don't care about performance, so a text protocol
makes more sense to me.
Also, your keys probably don't need to be fully general
(e.g., be able to contain \n), you could simplify there.
I don't know; to me, the transport for this application is
uninteresting. You write a parser, and then it's what
happens on top I really care about.
(George Jahad)
> For those of us who weren't there, what exactly did Rich suggest at
> the NYC meetup?
I did a presentation on Slime and Emacs; (slides here:
http://furius.ca/tmp/emaclj/emacs-and-clojure.martin-blais.pdf)
we discussed why he and a few others were using inferior-lisp
instead of Slime (it didn't make sense to me since I like the
fancer Slime features, but I can see why now). I said I'd be
curious to implement support for the newer effort, Rich
recommended I have a look at nREPL, that that was the right
approach, and I had already discussed it with Chas in a previous
meetup, but I hadn't had time to look at the code yet. That's
all.
I'm interested in doing some Emacs work on making it work
with the new new thing, but I'm not sure I want to rewrite
Slime, thus the long long thread to try to convince myself
if it's worth it or not to start from scratch. (I find
writing Emacs-Lisp fun, but sometimes really thorny and
time-consuming--I cannot say I'm really good at it.)
One more idea:
IMHO a client library should consider integrating the work
of George on debugging with JDI. Because of the nature of
JDI, it is certain to dictate a client written in Java. For
Emacs users, this means some external/child process used to
indirectly connect to one or more target VMs will be
necessary.
I wonder if there would be any reasons to generalize this,
e.g. a client/IDE always connects to a long-running "JDI/IDE
server" instead of the target; say, that server never stops
on your machine, even if you restart your target VM(s). It
always connects to the target VMs.
Thanks for all the ideas,
Meikel Brandmeyer
On 22 Mrz., 06:44, Martin Blais <bl...@furius.ca> wrote:
> > Why does a synchronuous version break less than an async one? And
> > besides: it's good enough for a lot of users, but it is not good
> > enough for me.
>
> Why does Slime with Clojure-1.3 not work right now?
puts transport and backend in one thing, IIUC. The IDE backend breaks.
Not the transport layer.
> IMO something really, really simple will almost never break,
> even between versions; because if you keep it really simple it
> never changes.
versions? What you do on top is the problem. And that is dictated by
necessity. I managed to get a working backend for 1.2 and 1.3 (so
far). But I had to deprecate usage with 1.1. At the moment this is not
very painful (who uses still 1.1?), but for the future I will probably
have to fork for different clojure versions.
> Being able to telnet to a VM on a remote server--no editor, just
> telnet--is useful in real-world cases where you can't install
> stuff on boxes; being able to shell in via telnet and eval a few
> things is _really_ powerful. Again, that's not to say that that
> could replace an asynchronous protocol, but it's a really nice
> fallback.
repl on a network socket.
> Swank is asynchronous, e.g.::
>
> (:emacs-rex
> (swank:create-repl nil)
> "clojure.tools.nrepl-test" t 4)
> (:return
> (:ok
> ("user" "user"))
> 4)
>
> AFAIK that final "4" in both the request and response is a
> request id.
close the connection, will SLIME keep repl state like the value of
*warn-on-reflection* and the like?
> Is there an effort somewhere for a common tooling library?
the major toolsmiths: Laurent, Eric and myself. And of course all
future IDEs supporters should be. (eg. bluefish is a possible
example).
Unfortunately I haven't had the chance to dive in, yet. Real life
wants its share.
> It's a wheel that has been riding for a while. I've learned
> not to ignore "working code", especially "old working code";
possible. But make independent from any transport or other system. We
can't know now, which environment an IDE developer does face. So it's
better not to couple things now needlessly.
> it's often too easy to dismiss something and restart from
> scratch without looking at it in enough detail (I'm not
> saying that this is the case here, but I'd have loved to see
> an exhaustive list of swank functions--I think I'll produce
> one; if anything that could be useful in helping define a
> complete "tooling" library).
currently contains only my experiences from VimClojure and some
distant observations from other systems + some corrections and
additions by Laurent.
Sincerely
Meikel
Chas Emerick
On Mar 22, 2011, at 1:44 AM, Martin Blais wrote:
>> The operations that would be provided by a tooling library would
>> have zero impact on the nREPL protocol. There's no reason why a
>> particular client would have to either (a) use the server's
>> provided set of tooling functions or (b) use any particular
>> standard set of tooling functions. It's just code that's being
>> loaded somewhere, and then called later on -- no protocol changes.
>
> Well, no low-level "protocol" changes that's true, but on top of
> it, something higher-level will be built for invoking that
> "tooling" library, if anything, just function calls on it, and
> though the nREPL transport won't break, access to the tooling
> library will likely break on version mismatch.
>
> But I understand that's not the goal of nREPL anyway, so it's beside
> the point.
Each of the development environments could easily load their own local versions of the tooling libraries, even in VMs that they didn't start. Even today, when I connect to a running Clojure VM with ccw, the first thing it does is load up its tooling-support libraries. As long it eventually takes care to load those libraries into a version-specific namespace, tooling shouldn't break in such an approach. We'll deal with the namespace unloading / class GC when we need to down the road.
> Is there an effort somewhere for a common tooling library?
Nothing concrete yet. All it requires is someone who is ready to take on the inevitable cat-herding. The vast majority of the actual code is already written, and is sitting in the repositories for ccw/swank-clojure/enclojure/vimclojure/etc.
>> However, there appears to be complete unanimity that this is a
>> less-than-optimal approach that will make life harder for some
>> potential client implementers in some contexts. So, I'm willing
>> to change the wire protocol. I'll read more about netstrings,
>> bencode, and other schemes. In the meantime, those who care
>> should pipe up now as to their preferences and/or points of
>> advice. The only significant preferences I have at the moment
>> are that:
>
> Your nREPL protocol is unusual indeed, I haven't seen one like
> that before (but I don't think it's bad).
>
> Netstrings are very common in binary protocols. I think for this
> application we don't care about performance, so a text protocol
> makes more sense to me.
>
> Also, your keys probably don't need to be fully general
> (e.g., be able to contain \n), you could simplify there.
>
> I don't know; to me, the transport for this application is
> uninteresting. You write a parser, and then it's what
> happens on top I really care about.
I fundamentally agree, but if a middle path can be found that will make writing clients and adapters as simple as possible in all known contexts, I'm willing to put the work in for that – especially now, when nREPL is very young and hasn't been widely adopted yet. A common, easily-targetable transport is the first step in enabling positive network effects; the sooner we can do away with the balkanization that exists among Clojure tooling, the sooner all our lives will get a lot better.
> One more idea:
>
> IMHO a client library should consider integrating the work
> of George on debugging with JDI. Because of the nature of
> JDI, it is certain to dictate a client written in Java. For
> Emacs users, this means some external/child process used to
> indirectly connect to one or more target VMs will be
> necessary.
>
> I wonder if there would be any reasons to generalize this,
> e.g. a client/IDE always connects to a long-running "JDI/IDE
> server" instead of the target; say, that server never stops
> on your machine, even if you restart your target VM(s). It
> always connects to the target VMs.
IMO, one thing at a time. :-) The VM has a number of purpose-built protocols/services (JDI, JMX, JVMPI, probably others), many of which we may want to orchestrate/interact with within a REPL, or using tools that use the REPL connection as a rich datasource. I wouldn't want to run into that stuff until we've learned to crawl. And, as you say, things like JDI require a "host" and target VM, but many environments already have a host (IDEs like Eclipse, etc., which would likely still want to use as much of CDT as possible). Someone will need to puzzle out the truly common bits and ensure that they're suitable for everyone, while allowing environments like vimclojure and emacs to opt into the parts that manage the "host" VM.
- Chas