Potential REPL library issue
Chas Emerick
Counterclockwise, I've discovered either a significant flaw or an
unfortunate design decision.
In the :result-fn, the .await call will return false, even when more
content from the last evaluation has yet to be read. This can happen
as a result of a race condition, or more frequently, when the
PipedReader's buffer wraps around (which leads to a lot of larger
responses being broken up into 1024-byte segments).
This is not a problem in an interactive context where REPL results are
being read off as available, and simply being spit into a buffer.
However, if some tooling is aiming to use the REPL in a systematic way
(e.g. to do some introspection about the REPL server's environment),
this means there's no way (or, no straightforward, obvious way) to
reliably retrieve and pair responses with evaluated expressions.
It seems that it would be far simpler to have *out* and *err* drain
into the same StringBuilder, and drop the result of that into a queue
that :result-fn can pop responses off of as available. This would
make evaluate->read-response usages simple, though I seem to recall
that the stream-oriented architecture came after issues with a message-
based REPL architecture.
Thoughts? Am I missing something? FWIW, this is definitely a blocker
on the re-use of the library in ccw.
Thanks,
- Chas
Laurent PETIT
Maybe I don't understand the issue to its full extent, but could one
possibility be to not use the same "server" machinery for the "remote
REPL" usecase and the "remote commands" (from IDE, etc.) usecase ?
For each new client connection to the "server", some parameter could
be used to specify which mode, and a dispatch be done by the "server"
for the client to be serverd by either the "remote REPL" logic, or the
"remote commands" logic.
I'm saying that 'cause I "feel" (totally subjective) that those 2
usecases of using the backend server may diverge even more in the
future, with different expectations concerning e.g. smartness of the
"remote REPL" logic (keeping out/err streams open with the client,
optionnally doing smart stuff on the output - pretty print, etc.-,
special treatments for special return values such as images, urls, who
knows what ...) and the "remote commands" logic (more strict, even
more reliable, option to inject new code to the server / update the
code of the server, ...)
Maybe the above content does not make sense, you'll tell me.
Regards,
--
Laurent
2010/9/2 Chas Emerick <ceme...@snowtide.com>:
> In the process of integrating the enclojure repl library into
> Counterclockwise, I've discovered either a significant flaw or an
> unfortunate design decision.
>
> In the :result-fn, the .await call will return false, even when more content
> from the last evaluation has yet to be read. This can happen as a result of
> a race condition, or more frequently, when the PipedReader's buffer wraps
> around (which leads to a lot of larger responses being broken up into
> 1024-byte segments).
>
> This is not a problem in an interactive context where REPL results are being
> read off as available, and simply being spit into a buffer. However, if
> some tooling is aiming to use the REPL in a systematic way (e.g. to do some
> introspection about the REPL server's environment), this means there's no
> way (or, no straightforward, obvious way) to reliably retrieve and pair
> responses with evaluated expressions.
>
> It seems that it would be far simpler to have *out* and *err* drain into the
> same StringBuilder, and drop the result of that into a queue that :result-fn
> can pop responses off of as available. This would make
> evaluate->read-response usages simple, though I seem to recall that the
> stream-oriented architecture came after issues with a message-based REPL
> architecture.
>
> Thoughts? Am I missing something? FWIW, this is definitely a blocker on
> the re-use of the library in ccw.
>
> Thanks,
>
> - Chas
>
> --
> You received this message because you are subscribed to the Google Groups
> "enclojure" group.
> To post to this group, send email to encl...@googlegroups.com.
> To unsubscribe from this group, send email to
> enclojure+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/enclojure?hl=en.
>
>
Chas Emerick
-- do you mean the latter to be strictly in support of IDE features?
Regardless, I see no reason at all to have a separate REPL library in
either case. There's nothing special at all about any use case I can
think of that would imply that one would require special functionality
vs. another.
Just to pluck a few mentioned characteristics: reliability should be a
given in any case, updating/loading new server-side code is really
just pushing expressions to be evaluated, controlling pretty print is
a matter of set!'ing or binding a var (talking now in general, not
specifically about the enclojure repl lib), and treating different
response data types differently is just a matter of encoding, plus
some metadata around the response indicating what encoding is being
used.
Having now seen a pretty wide range of REPL usage in two IDEs, I
actually think the ideal would be to run REPLs over HTTP. A message-
based protocol seems like a must (the concept having been proven in
scads of other scenarios, as well as by swank for years), and given
the freedom to choose, HTTP seems like a no-brainer: tons of tool
support, flexibly extensible (e.g. just add headers or params as
necessary to support/enable new features), and easily portable (to
Clojure.CLR and any other host that Clojure might run on in the future).
Speaking of swank, I looked at swank-clojure briefly (http://github.com/technomancy/swank-clojure
being the canonical source at the moment, I think). I'd much rather
use it (or any other existing REPL library) in conjunction with ccw or
enclojure before thinking of writing another one (or even of having to
break the enclojure REPL library in order to fix it), but there's a
variety of issues that would prevent me from recommending that:
there's no clojure client that I'm aware of (and I wouldn't look
forward to writing one), the "spec" of the protocol is driven by SLIME
(which, AFAIK, is CL-centric, and therefore likely not to be as
responsive as we'd like to Clojure-specific issues), and there are
oddities in the server-side code that are worrisome (e.g. <http://github.com/technomancy/swank-clojure/blob/master/src/swank/core/protocol.clj#L25
> appears to replace 't symbols with boolean true?).
Anyway, hopefully Eric has a solution for the issue I raised in my
first message.
- Chas
Laurent PETIT
> I don't quite grok the "remote REPL" vs. "remote commands" distinction -- do
> you mean the latter to be strictly in support of IDE features?
>
> Regardless, I see no reason at all to have a separate REPL library in either
> case. There's nothing special at all about any use case I can think of that
> would imply that one would require special functionality vs. another.
>
> Just to pluck a few mentioned characteristics: reliability should be a given
> in any case, updating/loading new server-side code is really just pushing
> expressions to be evaluated, controlling pretty print is a matter of
> set!'ing or binding a var (talking now in general, not specifically about
> the enclojure repl lib), and treating different response data types
> differently is just a matter of encoding, plus some metadata around the
> response indicating what encoding is being used.
np, my message was just for brainstorming.
>
> Having now seen a pretty wide range of REPL usage in two IDEs, I actually
> think the ideal would be to run REPLs over HTTP. A message-based protocol
> seems like a must (the concept having been proven in scads of other
> scenarios, as well as by swank for years), and given the freedom to choose,
> HTTP seems like a no-brainer: tons of tool support, flexibly extensible
> (e.g. just add headers or params as necessary to support/enable new
> features), and easily portable (to Clojure.CLR and any other host that
> Clojure might run on in the future).
message-based is what ccw's current implementation does.
> Speaking of swank, I looked at swank-clojure briefly
> (http://github.com/technomancy/swank-clojure being the canonical source at
> the moment, I think). I'd much rather use it (or any other existing REPL
> library) in conjunction with ccw or enclojure before thinking of writing
> another one (or even of having to break the enclojure REPL library in order
> to fix it), but there's a variety of issues that would prevent me from
> recommending that: there's no clojure client that I'm aware of (and I
> wouldn't look forward to writing one), the "spec" of the protocol is driven
> by SLIME (which, AFAIK, is CL-centric, and therefore likely not to be as
> responsive as we'd like to Clojure-specific issues), and there are oddities
> in the server-side code that are worrisome (e.g.
> <http://github.com/technomancy/swank-clojure/blob/master/src/swank/core/protocol.clj#L25>
> appears to replace 't symbols with boolean true?).
Yes, I tried to follow this road before rolling my own for ccw (back
in end of 2008/early 2009) and gave up for the same exact reasons :
uncontrollable "spec", in fact inexistant spec (must retro-engineer),
lack of clojure client, and no will to be the hero which would come up
with a clojure client :)
>
> Anyway, hopefully Eric has a solution for the issue I raised in my first
> message.
This is also my preferred solution :-)
Eric Thorsen
This is something that I've wanted for quite a while. There is some
history here that most likely drove where the existing architecture.
There were several hands in that code...
We definitely need to be able to pair requests with responses to
support several repl-clients talking to the same server as well. I
also started looking at the swank server for the same reasons of
wanting a consistent protocol (I occasionally dabble in emacs). html
might work very nicely as I also want lazy Clojure data viewers,
something along the lines of the javascript console in Chrome (which
is not lazy...but the utility is very nice)...but i digress...
It might make sense for us to have a quick phone call to see if we can
get on the same page and not duplicate any efforts.
Eric
On Sep 2, 8:42 am, Laurent PETIT <laurent.pe...@gmail.com> wrote:
> 2010/9/2 Chas Emerick <cemer...@snowtide.com>:
> > I don't quite grok the "remote REPL" vs. "remote commands" distinction -- do
> > you mean the latter to be strictly in support of IDE features?
>
> > Regardless, I see no reason at all to have a separate REPL library in either
> > case. There's nothing special at all about any use case I can think of that
> > would imply that one would require special functionality vs. another.
>
> > Just to pluck a few mentioned characteristics: reliability should be a given
> > in any case, updating/loading new server-side code is really just pushing
> > expressions to be evaluated, controlling pretty print is a matter of
> > set!'ing or binding a var (talking now in general, not specifically about
> > the enclojure repl lib), and treating different response data types
> > differently is just a matter of encoding, plus some metadata around the
> > response indicating what encoding is being used.
>
> np, my message was just for brainstorming.
>
>
>
> > Having now seen a pretty wide range of REPL usage in two IDEs, I actually
> > think the ideal would be to run REPLs over HTTP. A message-based protocol
> > seems like a must (the concept having been proven in scads of other
> > scenarios, as well as by swank for years), and given the freedom to choose,
> > HTTP seems like a no-brainer: tons of tool support, flexibly extensible
> > (e.g. just add headers or params as necessary to support/enable new
> > features), and easily portable (to Clojure.CLR and any other host that
> > Clojure might run on in the future).
>
> message-based is what ccw's current implementation does.
>
> > Speaking of swank, I looked at swank-clojure briefly
> > library) in conjunction with ccw or enclojure before thinking of writing
> > another one (or even of having to break the enclojure REPL library in order
> > to fix it), but there's a variety of issues that would prevent me from
> > recommending that: there's no clojure client that I'm aware of (and I
> > wouldn't look forward to writing one), the "spec" of the protocol is driven
> > by SLIME (which, AFAIK, is CL-centric, and therefore likely not to be as
> > responsive as we'd like to Clojure-specific issues), and there are oddities
> > in the server-side code that are worrisome (e.g.
Chas Emerick
have been obvious. (I should stop doubting myself all the time so
much. :-)
You know where to reach me. IM or ring up anytime.
- Chas