Clojure stream socket repl

[Alex Miller]

Just wanted to drop a note re CLJ-1671 which we've added to 1.7. Text from the ticket is below. This is also related to the commit that Rich made today to print Object and Throwable via tagged literal formats.

-----

Programs often want to provide REPLs to users in contexts when a) network communication is desired, b) capturing stdio is difficult, or c) when more than one REPL session is desired. In addition, tools that want to support REPLs and simultaneous conversations with the host are difficult with a single stdio REPL as currently provided by Clojure.

The goal is to provide a simple streaming socket repl as part of Clojure. The socket repl should support both program-to-program communication (by exchanging data) and human-readable printing (which mirrors current behavior). The socket repl server will be started only when supplying a port to clojure.main or by explicitly starting it by calling the provided function.

Each socket connection will be given its own repl context and unique starting user namespace (user, user1, user2, etc) with separate repl stack and proper bindings. On session termination, the namespace will be removed. in and out will be bound to incoming and outgoing streams. Tools can communicate with the runtime while also providing a user repl by opening two connections to the same server.

There are two known cases where the repl interprets non-readable objects and prints them for human consumption in a non-readable way: objects with no specific print-method and when handling Throwables. Both cases (Object and Throwable) now have a print-method implementation to return a tagged-literal representation. By default the socket repl will print exceptions with the print-method data form. Optionally, the user can set a custom repl exception printer. A function that provides the current human readable exception printing will be provided.

[Colin Fleming]

This is very interesting, and I think it's an improvement. It'll make building tooling over a simpler REPL layer than nREPL a lot easier. I'd like to see this going further, though.

For those who aren't aware, the main complaints I'm aware of with nREPL are:

1. General complexity, see https://gist.github.com/levand/b1012bb7bdb5fcc6486f. As Luke states, most of his complaints are really issues with nREPL.
2. I have also heard reports of a lot of difficulty with the load-file operation, although I don't have enough detail to know what's happening there.
3. Rich feels very strongly that a REPL interaction is a streaming interaction - you stream forms to the REPL and you get some results and output streamed back. I don't agree, I think that most users think of it as request/response - I send a form to the REPL, and I get a result back.

Cursive inherited its REPL from La Clojure originally, which was basically inferior lisp-like. I wanted to add more sophisticated functionality, e.g. syntax highlighting returned values, but that was basically impossible since you're trying to extract values from a stream of text. A long time ago I switched to nREPL for all Cursive's REPL types and a lot of things became much easier:

1. Response values can be parsed much more easily, which allows them to be highlighted or formatted.
2. I have much better information about what happened to the execution, i.e. success, failure, exceptions and the current ns after evaluation.
   
3. I could implement functionality which requires reliably identifying responses and tying them to their originating requests. This includes completion, test integration, macroexpansion, etc. In the case of Cider this is also required for navigation and so forth. All of this is basically impossible to implement reliably with a fully streaming model.

The main problems with the nREPL model for tool implementers are:

1. nREPL tries to return *out* and *err* from the execution of the form. This mostly works except when the form spawns something in another thread, and at that point you have to capture stdout/stderr from the actual process yourself and integrate that into your output by hand. Any exceptions on those threads are just dumped to stderr and don't get returned in any useful form to the client. And if you're working over a remote connection rather than a JVM you started yourself then you're just totally out of luck and all that output basically goes into the toilet.
2. A lot of this breaks down if you have really huge data items being returned - the streaming model handles this easily.
3. The encoding issues Luke mentioned, although I do think that there must be a way to escape values so this isn't such a problem.
   

Users of nREPL have other problems like middleware ordering, but they're generally not a problem for implementers. I do think that middleware is generally not worth the complexity it introduces, though - very few things actually require it. Cursive doesn't use it at all.

I feel pretty strongly that actual user evaluations are not streaming interactions, and should be request/response. Not doing this basically rules out highlighting the response in any way, and also makes it difficult for the editor to identify the current namespace of the REPL. I do think that I/O really is streaming. However this proposal will suffer the same I/O problem as nREPL (I/O from threads other than the REPL handling thread will go nowhere) while not providing the benefit of the message framing that nREPL provides. I do think that some sort of message framing is very useful, but that then also forces you to interleave your I/O into your message frames somehow. It's tricky.

I think the ideal integration would probably be to use this new feature, create the process from the editor so you can capture true stdio, then make connections to it over a data-only connection for the evaluations and other functionality. It seems like that's a good middle ground between the two models. It doesn't solve all the problems with encoding or huge data structures though. There's a basic tension between adding more sophisticated functionality, and having the basic stuff (but only that) work all the time.

Is there any more detail on what the data exchange connections were planned to look like?

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

[Rich Hickey]

Let's please keep this discussion to the proposed feature and its suitability to various tasks. I don't want this to become a discussion about nREPL.

REPL stands for something - Read, Eval, Print, Loop.

It does not stand for - Eval RPC Server/Window.

It is not something I "feel" about. But I do think we should be precise about what we call REPLs.

While framing and RPC orientation might make things easier for someone who just wants to implement an eval window, it makes the resulting service strictly less powerful than a REPL.

Let's look at some of that power:

'read' is a protocol. It works on character streams, one of the most widely available transports. It uses delimiters (spaces, [], (), {} etc) to separate forms/messages. Readers implement this protocol and return data upon encountering complete forms. read is shipped with Clojure and multiple implementations are available. It is available for use in programs, including programs you invoke in a REPL. A substantial subset of evaluable+read-able forms are supported by edn, which has even wider support. The reader format is extensible. So, when you get read, you are getting a lot.

When embedded in a REPL, even more is implied by read. There must be some source (stream) of things read. That source is the same source that should be used should the code being evaluated call 'read'. If that source is a terminal with a human, they can continue their interaction *in place*.

'eval' implies the full power of Clojure. Take a form as data, evaluate it, return a result as data.

'print' takes the returned data and prints it on an output stream as characters. It is true that not all things might print read-ably, which can pose challenges for *programs* sitting on the other end of a REPL that would like to call 'read' on its output. But we have improved Clojure in this area - now the default print-method prints something that is read-able (with an appropriate tag handler installed). Another area where many REPLs presume a human consumer and print un-read-ably is when exceptions occur. This socket REPL will default to printing exceptions read-ably, but you can switch to a human oriented formatter of your choice.

Again, as with read, when embedded in a REPL, even more is implied by print. There must be a destination (stream) of things printed. That destination is the same destination that should be used should the code being evaluated call print. If that source is a terminal with a human, they can leave their reading focus *in place* and see all output targeted towards them.

'loop' - do it again. Note however that a REPL will not attempt to read/print while evaluating.

The stream binding and nestability is important. It means that you can, in the code you evaluate in your REPL, launch a nested interpreter, or Eliza-like thing, or a text-adventure etc, and those things will *get input from the same source and print to the same target*. This is what someone who knows what REPLs are expects and demands.

You can, if you desire, even launch an eval RPC server in a REPL, but not vice-versa.

REPLs work over the most primitive connections, terminals, and can be pipe, rerouted, etc with all tools. They accomplish a lot over a single connection, which will often be all you can get.

But, we now have nice editors/IDEs often running on the same machine, with the ability to use sockets. Multiplexing their needs with those of a human REPL consumer over the same connection is going to make things bad for one or both of them.

With the socket REPL, the recommendation is that tools will open more than one connection, and use a dedicated channel for most editor/IDE operations that might want to interact with the running process. With the improved read-ability of REPL output, I think an edn-based REPL makes a fine RPC server for program-to-program communication, especially since the set of 'verbs' supported is bounded only by what you can evaluate, and the types of things you might discover is open.

These are the intentions of this feature.

Thanks for the feedback,

Rich

[Colin Fleming]

Sure, I agree that a base REPL is a powerful substrate. I suspect that most people will end up building an RPC mechanism on top of it, since I believe that's what nearly everyone wants from their REPL interactions. Cursive has never supported need-input, which is the nREPL response saying that the server needs some input that the user has to supply. There's only one issue in the tracker about that and that was relating to CLJS REPLs - once Piggieback came out as far as I can tell no-one has ever needed it. If they have, they're pretty quiet about it.

Anyway, as long as it's possible to build these layers on top of this I think this is a good change. The readable object change will be helpful too. The one thing I can't see a solution for is I/O from async threads - once you're not capturing true I/O from the REPL server process that becomes difficult, but I don't think there's a good solution for that.

[Gary Fredericks]

I don't know if the intention with the new #object printing is to actually make everything print quasi-readably, but just in case I thought I should point out that currently any IDeref (and local vars especially) still print with #<...>.

Gary Fredericks
(803)-295-0195
frederi...@gmail.com
gfredericks.com

--

[Rich Hickey]

We'll fix that, thanks.

[Alex Miller]

Gary - Rich made a commit this morning in regards to your comment.

I dropped a socket repl patch and info on the ticket at http://dev.clojure.org/jira/browse/CLJ-1671. Lot of fun to test it via telnet! :) 

Alex

[Daniel Solano Gómez]

Hello,

This is a very interesting idea. I have just one comment that comes to
mind, inline below.

On Mon Mar 9 14:34 2015, Alex Miller wrote:
> Just wanted to drop a note re CLJ-1671

> <http://dev.clojure.org/jira/browse/CLJ-1671> which we've added to 1.7.

> Text from the ticket is below. This is also related to the commit

> <https://github.com/clojure/clojure/commit/692645c73c86d12c93a97c858dc6e8b0f4280a0b>

> that Rich made today to print Object and Throwable via tagged literal
> formats.
>
> -----
>
> Programs often want to provide REPLs to users in contexts when a) network
> communication is desired, b) capturing stdio is difficult, or c) when more
> than one REPL session is desired. In addition, tools that want to support
> REPLs and simultaneous conversations with the host are difficult with a
> single stdio REPL as currently provided by Clojure.
>
> The goal is to provide a simple streaming socket repl as part of Clojure.
> The socket repl should support both program-to-program communication (by
> exchanging data) and human-readable printing (which mirrors current
> behavior). The socket repl server will be started only when supplying a
> port to clojure.main or by explicitly starting it by calling the provided
> function.
>
> Each socket connection will be given its own repl context and unique
> starting user namespace (user, user1, user2, etc) with separate repl stack
> and proper bindings. On session termination, the namespace will be removed.
> in and out will be bound to incoming and outgoing streams. Tools can
> communicate with the runtime while also providing a user repl by opening
> two connections to the same server.

What about making the creation of the namespace configurable? I am
guessing that the rationale behind having each connection have its own
namespace is facilitate multiplexing in a sane fashion. However, what
about in contexts where generating extra namespaces is considered
expensive in terms of memory/runtime costs? In particular, I am
thinking about REPLs embedded in Android applications where we would
prefer the overhead of the REPL to be as minimal as possible. In such
cases, it might be nice to be able for incoming connections to connect
to an existing namespace, understanding that multiple connections would
not be as isolated in such a configuration. Should this be a valid way
to use a socket REPL?

Sincerely,

Daniel

[Rich Hickey]

People rarely ask for things that they don't know they are missing.

Absolutely no one asked for Clojure, for instance.

I know I'd really appreciate Cursive supporting Clojure's native REPL (soon REPLs).

[Colin Fleming]

Sure, but in this case I think that applications with interactive console I/O are probably just not a very common use case for Clojure. Mike Fikes pointed one out to me yesterday (a menu at startup of his Ambly REPL). I definitely want to support it, but it's tricky in Cursive because the REPL editor is a "strongly typed" Clojure editor in order to provide all the Cursive support. So to support this sort of thing it needs to transparently switch to a plain text editor somehow. Anyway, that's my problem, unrelated to this change.

Cursive will definitely support this soon, I'm working on clojure.main support now actually, which will be easy to adapt to this.

I'm also interested in Daniel's question, although I'm not so worried about the performance. I suspect that users will expect to always be connected to the same user namespace. Is there a particular reason for providing multiple user namespaces?

[Rich Hickey]

I think it is important, with tooling, to have the broadest, open notion of the types of 'use cases' for Clojure, and for workflows of people using Clojure, lest you actually limit them. I hope people are able to write their languages and DSLs, test embedded control/debug consoles, design next-generation REPLs and debuggers, grab data dumps etc, all from their REPL. That's the heritage of Lisp. Many people are doing data science, languages, genetics, rule-systems etc type things with Clojure, not just web apps.

In any case, it's great to hear support might be coming. I think you should also question your presumptions about needing to completely give up your Cursive functionality in such a REPL. You can examine/prepare anything you want on the way to sending it over the REPL. You can toggle per-character and per-form transmission. People will only need the former when utilizing a non-reader-based stream consumer. You can package any form in a wrapper expression that wraps the result in something you can detect on the return, thus being able to distinguish responses to IDE-issued requests from other output. You can provide IDE controls for switching namespaces that allow you to track that. Etc. Of course you need do little more than support eval-form and load-file, while staying out of the way of the streaming, in order to win me over from inferior lisp mode :)

As far as the namespaces, again yes, it is important to think broadly about the power of multiple sessions, even from the same user. At first we are talking about 2 sessions, for human and IDE. But it is quite common in rich Common Lisp IDEs like Lispworks for people to have multiple listener (REPL) sessions open at the same time (against the same runtime). You can launch long running processes that produce streaming output, switch to another listener and continue interaction and development, have separate listeners for separate contexts/state/command-history etc. There's a lot to Lisp development that's not RPC. And of course a socket REPL server could support multiple simultaneous users, where 'users' might be programs.

Have you ever worked with an environment like Lispworks or Allegro? If not, I would definitely recommend trying it. I like IntelliJ, and am a long time customer, but it still can't touch some aspects of the experience of a rich Lisp environment. IntelliJ is much more of an editor/code management system than a live programming environment. Hopefully Cursive won't settle for giving Clojure programmers the power of Java :)

http://franz.com/images/ide.png
http://www.lispworks.com/images/lw-ide-gtk.png

Note multiple listeners, data inspectors and more. Maybe not the latest GUIs, but the capabilities should be a source of inspiration for any Clojure IDE.

We have options and tradeoffs around ns per session and are considering them. Sharing by default can lead to conflict by default. Having session/ns affinity allows for automatic cleanup, else that also becomes a user problem. People who want to create namespaces that survive sessions can do so, and can then manage them. The biggest argument against per-session namespaces might be memory use in constrained environments. Even then, there would likely be at least one user ns, so it might just be a matter of configuration to ensure that the first socket REPL session creates it. Still thinking about this.

If you think users will want to end up in 'user' every time, just have Cursive send an in-ns on connect before handing it over to the user. But please make that something I can turn off :)

[Bozhidar Batsov]

Btw, there's still one fundamental tooling problem that still has to be solved. While I'm pretty convinced there's nothing fundamentally wrong with nREPL and I'm not sure what will be the exact benefits for the most users from the new solution, I'm pretty sure they'll still have to use libraries that provide tooling functionality in their apps (e.g. completion, inspection, debugging, etc). With no easy way to isolate the tooling deps from the app's deps tool writers are in a pretty tight spot... This is the only real blocker to making something like CIDER much more powerful.

Excuse me for this detour of the discussion, but I couldn't help myself, as the this socket based REPL will be only as useful as the tooling support it provides IMO.

[Colin Fleming]

Sure - I'm certainly not trying to limit what people can do with Clojure. But I have to make pragmatic decisions too - there's only me working on Cursive, I only have 24 hours in a day and I have over 300 open issues in the tracker. So I have to prioritise things that are real day to day problems for the majority of users, and try to maximise the bang for the buck of the time that I spend on it. The ability to implement text based adventure games seldom comes up :-). I don't have a problem saying that it's unlikely anyone will be able to easily build curses-based interactive apps any time soon in a Cursive REPL window if ruling that out (for the moment) means that the editing support for Clojure is much better.

It's also worth reiterating that basically all of these issues in Cursive only relate to streaming input - it's always fully supported output, that's relatively easy to implement even over nREPL. And the interactive input thing is mostly a UI problem, albeit a pretty serious one. I'm sure there's a good solution there, I just haven't spent long thinking about it. But the use case you mentioned (multiple REPL windows with streaming output) would work fine today, with the caveat that multiple REPLs are shown in tabs in the same toolwindow - that's relatively easily fixable if required, though.

I haven't used any of the CL environments, no - I'd like to try but the bar for starting is pretty high given that I don't know CL either. This would be a fantastic topic for an unsession at Clojure/West, if anyone is listening :). Geiser (the Scheme mode for Emacs) is very interesting also. It gets extra bonus points for the following phrase in the doc: "Because I prefer growing and healing to poking at corpses, the continuously running Scheme interpreter takes the center of the stage in Geiser", which sounds similar to what you're talking about.

And yes, I don't think any of the ns management stuff is really a serious blocking point. Like you say, either you can issue an in-ns to get back to user, or (ns user2) to get a new one - that's not a big deal.

[Colin Fleming]

Can you elaborate on what you mean by isolating the tooling deps from the app deps? How does this block CIDER's development?

[Bozhidar Batsov]

Libraries used in the middleware can potentially conflict with different versions of the same libs used by the app in question. This means we have to be extra careful about our dependencies and pick only libraries that are unlikely to be used in actual apps. More details:

- https://github.com/clojure-emacs/cider-nrepl/issues/143

- https://github.com/clojure-emacs/refactor-nrepl/issues/20 

[Andy Fingerhut]

Another possibility, not necessarily a good one, used by Eastwood is to copy in the source code of license-compatible libraries, and rename them.  Eastwood contains a copy of tools.reader, for example, with all clojure.tools.reader.* namespaces renamed eastwood.copieddeps.clojure.tools.reader.*

This enables Eastwood to read and eval any dependencies that use clojure.tools.reader.*, including a different version, without problems.

Andy

[Bozhidar Batsov]

The work on nrepl-refactor actually spawned https://github.com/benedekfazekas/mranderson, which does exactly this. Unfortunately this is an option only for Clojure deps and when dealing with Java deps you're kind of screwed. Not to mention dealing with this renaming makes the dev process way more involved than it has to be IMO. 

[Andy Fingerhut]

Java reps with license-compatible source code cannot be copied in and their packages renamed?

Or do you mean that the mranderson tool doesn't handle Java source code?

I agree that this technique makes it significantly more time consuming to update versions of dependencies in a project, and automation tools can help make it faster.  Hence my "not necessarily a good one" qualifying phrase.

If anyone knows another method that is as effective, or nearly so, I'm curious to know what it is.

Thanks,

Andy

[Bozhidar Batsov]

mranderson can't do it. Sorry for not being clear.

However, renaming a pretty big Java package with many deps is a operation that's extremely involved. Not sure you can easily develop generic tooling to help you with that.

On 12 March 2015 at 17:04, Andy Fingerhut <andy.fi...@gmail.com> wrote:

Java reps with license-compatible source code cannot be copied in and their packages renamed?

Or do you mean that the mranderson tool doesn't handle Java source code?

I agree that this technique makes it significantly more time consuming to update versions of dependencies in a project, and automation tools can help make it faster.  Hence my "not necessarily a good one" qualifying phrase.

If anyone knows another method that is as effective, or nearly so, I'm curious to know what it is.

You and me both. :-)

[Rich Hickey]

Please move this discussion of how to fix nREPL off of this thread about socket REPL

[Chas Emerick]

On 03/12/2015 07:31 AM, Bozhidar Batsov wrote:

Libraries used in the middleware can potentially conflict with different versions of the same libs used by the app in question. This means we have to be extra careful about our dependencies and pick only libraries that are unlikely to be used in actual apps. More details:

- https://github.com/clojure-emacs/cider-nrepl/issues/143

- https://github.com/clojure-emacs/refactor-nrepl/issues/20 

On 12 March 2015 at 10:11, Colin Fleming <colin.ma...@gmail.com> wrote:

Can you elaborate on what you mean by isolating the tooling deps from the app deps? How does this block CIDER's development?

There's only two ways to skin this cat within a single JVM: classloader isolation, and source-level rewriting. There are many implementations of both floating about.

Source rewriting implies a compile/run cycle that is mostly incompatible with the interactivity expectations Clojure developers have, so few go down this route.

Boot's pods (via shimdandy) are a recent incarnation of classloader isolation that looks particularly attractive on paper (I've not used either yet). Classloader isolation is notoriously something that is hard to get right, and can subtly change program semantics (e.g. the "same" class loaded into different classloaders yields two Class objects, which are not equal to each other).

In any case, this is a challenge in the JVM, unrelated to REPL mechanisms.

- Chas

[Toby Crawley]

On 03/12/2015 12:00 PM, Chas Emerick wrote:
> Boot's pods (via shimdandy) are a recent incarnation of classloader
> isolation that looks particularly attractive on paper (I've not used
> either yet). Classloader isolation is notoriously something that is hard
> to get right, and can subtly change program semantics (e.g. the "same"
> class loaded into different classloaders yields two Class objects, which
> are not equal to each other).
>

I should point out that shimdandy[1] isn't intended to provide
dependency isolation in a single Clojure runtime, but instead provide
multiple, isolated Clojure runtimes in a single JVM. There are two
caveats with using it:

1) Clojure can't be loaded in the parent classloader tree, and
clojure.jar cannot be visible to that tree. Shimdandy doesn't invert the
standard parent classloader delegation model, so if clojure.jar is
visible to the parent loader, the shims created by Shimdandy will all
use the single static RT.class from the parent tree instead of providing
multiple, isolated runtimes. This means that you can't create Shimdandy
shims from within Clojure, it must be done from some other language.

2) As you mention, sharing objects between classloader trees is a recipe
for pain. The recommended approach for sharing data between shims is
either by pr-str/read-string serialization/deserialization, or by
carefully passing/treating data as java.lang.* classes/interfaces
(though it's easy to mess that up if you are not careful, and end up
with confusing errors). Caveat emptor.

- Toby

[1]: https://github.com/projectodd/shimdandy

[Colin Fleming]

It seems like source rewriting is really the only way to achieve this for CIDER's needs. If the middleware deps are isolated in a shimdandy runtime, then the CIDER code also needs to exist in that runtime and it won't have access to the host app, which is the whole point of a REPL. Is there a way around this that I've missed?

[Colin Fleming]

Coming back to the socket REPL: for machine-machine communication (which an IDE control channel basically is), is there prior art for a way to obtain the responses from an output stream? The issue that I see is that I write my command to the input stream, perhaps with some operation ID so I can tie it to a reply, and I probably have something reading forms repeatedly from the output stream and handling them. However if the server-side code prints any output in the REPL thread, that will mess up the response parsing. I can't always control that, for example during macroexpansion - even if I control all the code, this seems brittle. Is the preferred method to do what interruptible-eval does, and evaluate a form that rebinds *out* and *err* before doing what it needs to?

[Rich Hickey]

I'm not trying to set your priorities, and I know you are working on this (thanks!). But I will continue to object to this characterization of which people require nested read and the things it can be used for. It is not about text adventure games. I've given several serious use cases, including languages and DSLs and control/debug interaction prompts.

Here's a wonderful demonstration of the power of nestability:

"Programming Should Eat Itself" by Nada Amin
https://www.youtube.com/watch?v=SrKj4hYic5A

And it's not just about hardcore Lispers who know its heritage.

Consider students. Here's one of my daughter's first high school programming assignments:

prompt user for their gender
read the gender
prompt user for their height
read the height
print "your ideal weight is:" the result of a calculation using gender and height

In Java this requires defining, compiling and running a whole program with a main. How can we claim that Clojure is dynamic and interactive if we can't run this in a REPL, and test its component functions there?

=====
Connecting to local nREPL server...
Clojure 1.7.0-master-SNAPSHOT
nREPL server started on port 53446 on host 127.0.0.1

(defn homework []
(let [_ (println "enter your gender (m/f):")
gender (read)
_ (println "enter your height:")
height (read)]
(println "your ideal weight is" 42)))
=> #'user/homework

(homework)
enter your gender (m/f):
DEBUG: unknown status need-input
=====

Note that nested interactive input has 2 modes - form-oriented 'read'ing and arbitrary eager character-by-character reading.

Ideally, you would continue to maintain form orientation (allowing completion and structural editing) as a default. Structural editing and form correctness checking will still be a win for those whose nested reader is calling 'read' (which won't care if the form was fully prepared before being sent). Only in a (necessary, but could be primitive) mode would you have to allow character-granularity transmission, for arbitrary stream-consuming functions.

[Rich Hickey]

I don't see why you don't just send your request form over and then call read?

On a dedicated channel, why would you encounter unexpected output? You can test all the expressions/forms you use.

[David Nolen]

On Fri, Mar 13, 2015 at 8:39 AM, Rich Hickey <richh...@gmail.com> wrote:

I'm not trying to set your priorities, and I know you are working on this (thanks!). But I will continue to object to this characterization of which people require nested read and the things it can be used for. It is not about text adventure games. I've given several serious use cases, including languages and DSLs and control/debug interaction prompts.

Nothing to add other than noting that this is how ClojureScript REPLs are embedded in the Clojure REPL and how it has worked now for years.

David

[Chas Emerick]

Clojure has many REPLs; clojure.main provides one today, nREPL provides
ways to build others, and there are yet more. Each have been built with
different tradeoffs in mind, and I look forward to seeing how the new
stream socket REPL in clojure.main skins the cat.

You are right in implying that nREPL subjugates stdin/out in favor of a
discrete message-based protocol (which some people have found useful).
Reading from stdin is thus turned into an explicit "needs input" signal,
one of the few things nREPL borrowed from swank (used by many other
lispers via SLIME and other compatible tools). This is obviously just a
shadow of stdin, but as 80/20 solutions go, it's a decent one given
nREPL's other objectives and constraints.

Regarding your `homework` example, it seems you're using some nREPL
client that doesn't understand the aforementioned "needs input" signal.
(I wonder which one it is?) That's unfortunate, but isn't itself an
indictment of the approach. Both `lein repl` (versions 2.4.3 and 2.5.1)
and my very old copy of cider (some pre-0.5.0 checkout) appear to work
as expected:

$ lein repl
nREPL server started on port 54061 on host 127.0.0.1 -
nrepl://127.0.0.1:54061
REPL-y 0.3.2, nREPL 0.2.3
Clojure 1.6.0
Java HotSpot(TM) 64-Bit Server VM 1.8.0_40-b25
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e

user=> (defn homework []
#_=> (let [_ (println "enter your gender (m/f):")
#_=> gender (read)
#_=> _ (println "enter your height:")
#_=> height (read)]
#_=> (println "your ideal weight is" 42)))
#'user/homework
user=> (homework)
enter your gender (m/f):
m
enter your height:
74
your ideal weight is 42
nil

Of course, the above working (or not) doesn't negate your objectives,
etc. In any case, I agree with your original sentiment that clojure-core
stream socket REPL discussion shouldn't center on nREPL.

- Chas

[Rich Hickey]

That was Cursive we were talking, about and whose REPL I was demonstrating.

[Colin Fleming]

Sure, I was being facetious with the text adventure dig, my apologies. I do understand that there are great use cases for this - again, not having fixed it before now has purely been because it's tricky in Cursive's case for IntelliJ reasons and there hasn't been enough demand to warrant the investigation. I do consider this a bug in Cursive, it's just been a very low priority one.

[Colin Fleming]

The two possibilities I can think of are
a) executing code that calls libraries, which might print something unexpected - it's harder to ensure I've tested all code paths that library code might take.
b) for Cursive to provide macroexpansion using the REPL, I have to execute essentially arbitrary code - it's entirely plausible that a macro would print something during expansion, for debugging or other purposes.

[Colin Fleming]

There's one other thing that I'm still not sure of. I can't see a way for the IDE to get information about the current user channel for use over the tooling channel. I'm thinking specifically of *ns* and *e here, but there may be more cases. *e is useful for having a UI element that displays the last exception in some way, or at least prints it out in the REPL. That's not so critical, but *ns* is vital for completion and various other operations. The only way I can think of to get it is to parse it out of the prompt but that's pretty brittle - is there a better solution for this?

[Rich Hickey]

While these might be possible, I don't consider either plausible. Sitting around with REPLs all day, I never see spurious printing. Most lib developers would get quickly harassed for spurious printing - it would render a library pretty worthless/harmful.

When you truly need to run arbitrary unknown code (not, I imagine, the case for most programmatic interactions), you can always sink the streams.

[Rich Hickey]

You can:

a) issue an (ns-name *ns*) when connecting the user's socket before returning it to them
b) have IDE facilities for changing the REPL ns that you can track
c) watch REPL output forms for in-ns and ns
d) issue calls yourself on the user's channel
etc

It might be interesting to provide some socket REPL registry, but so many of the interesting things are stack bindings, so you do need the participation of the nested loop.

[Laurent PETIT]

Hello,

 Rich, Colin:

I also am thinking hard (hammock time currently) about how to add proper support for socket REPLs.

The most basic think to do, and which is probably already working out of the box for all IDEs, is to use their integrated Console. They are stream oriented (in and out).

Of course, this would provide absolutely no other interesting Clojure editing support out of the box.

So I'm tempted to reuse the current Counterclockwise REPL UI.

Colin, Rich:

For programmatic use, I think the following `rpc` macro, that you can easily install/send in the socket session namespace as a first command, should suffice to catch arbitrary prints to stdout, stderr, and even sent exceptions:

(defmacro rpc [& body]

  `(let [out# (java.io.StringWriter.)

         err# (java.io.StringWriter.)]

     (try

       (let [res# (binding [*out* out# *err* err#]

                    ~@body)]

         {:eval res#, :out (.toString out#), :err (.toString err#)})

       (catch Exception e#

         {:exception e#, :out (.toString out#), :err (.toString err#)}))))

which is a way of 'sinking the streams", what Rich mentioned later on this same thread.

WRT multiplexing rpc calls, I would probably either not bother at all with that and just use a single IDE-dedicated socket session per socket server, or a dynamic pool of socket sessions if the IDE has too many concurrent calls to the socket server, or some calls can block for a too long time.

Another idea is to use an agent to ensure correct serialization of concurrent rpc responses, and start all server side rpc calls in futures:

(defmacro rpc [rpc-id & body]

  `(future

     (let [r# (let [out# (java.io.StringWriter.)

                    err# (java.io.StringWriter.)]

                (try

                  (let [res# (binding [*out* out# *err* err#]

                               ~@body)]

                    {:eval res#, :out (.toString out#), :err (.toString err#)})

                  (catch Exception e#

                    {:exception e#, :out (.toString out#), :err (.toString err#)})))]

       (send rpc-agent pr (assoc r# :id '~rpc-id)))))

WRT a reliable way of detecting the namespace the session currently is in, I have an idea belong the following line: instrument every code the user sends for evaluation in a stream-unobtrusive way. This will require out-of-the-band communication of the current ns, but is not hard to achieve:

- store the current ns inside the socket session ns (this provides easy garbage collection - when user closes session, everything about it goes away-, and a straight-forward place for the IDE to look for. Only drawback: the user can notice that the IDE sneakily created a var in its user namespace. Note that if users complain about this, it's still possible to store this information in a global { session-ns -> current-ns } map somewhere else.

The wrapping code would be straightforward, something like:

(try

  .... user form ....

  (finally (reset! user4/current-session-ns *ns*)))

WRT a way of detecting nesting of reads on stdin, it's also possible to use a non-intrusive method, along the same lines: use an out-of-the-band var that can be consulted from the outside to know whether the evaluation of the sent form has finished or not:

(try

  (reset! user4/evaluation-in-progress true)

  .... user form which may include a call to read-line, or even a switch to a clojurescript repl ....

  (finally (reset! user4/evaluation-in-progress false)))

In the IDE, you can then react by watching evaluation-in-progress changes:

- use some heuristics on the form sent by the user to try to guess if you should continue to display a clojure editor for user input or a raw text editor

- consider not user the above mentioned code wraps if the user sends something for evaluation while evaluation-in-progress is still set to true

In all cases, I agree with Rich that we should enhance the UI of our REPLs so that the user can explicitly mention what it is doing:

- typing arbitrary characters to be sent over the socket stream (maybe not on the fly character by character as mentioned by Rich, or maybe in a separate even more basic text editor which should prevent deleting typed chars, etc.)

- typing clojure code (and thus we can wrap the user code. Note that this won't work perfectly either: if the user is playing the Inception movie with clojure repls, or jumping from Repl to Repl, the IDE will lose track of it and, by wrapping the user code for what would then be a totally unrelated socket session in a maybe totally unrelated JVM, will damage the user experience ... This problem can probably be mitigated by adding proper checks and more dynamicity to the above wrappers.

I've been hammocking some of these ideas for quite some time now, and would be pleased to hear your reactions to them.

Cheers,

-- 

Laurent

Laurent Petit

[Christophe Grand]

On Mon, Apr 27, 2015 at 12:27 AM, Laurent PETIT <lauren...@gmail.com> wrote:

For programmatic use, I think the following `rpc` macro, that you can easily install/send in the socket session namespace as a first command, should suffice to catch arbitrary prints to stdout, stderr, and even sent exceptions:

(defmacro rpc [& body]

  `(let [out# (java.io.StringWriter.)

         err# (java.io.StringWriter.)]

     (try

       (let [res# (binding [*out* out# *err* err#]

                    ~@body)]

         {:eval res#, :out (.toString out#), :err (.toString err#)})

       (catch Exception e#

         {:exception e#, :out (.toString out#), :err (.toString err#)}))))

which is a way of 'sinking the streams", what Rich mentioned later on this same thread.

I think it's not enough since you only capture stdout, stderr and exceptions during the E of REPL. Lazy evaluation forced while printing in itself may trigger spurious printing and exceptions. You need to print yourself and not let the REPL print.

--

On Clojure http://clj-me.cgrand.net/
Clojure Programming http://clojurebook.com
Training, Consulting & Contracting http://lambdanext.eu/