Extending the REPL

[Thomas Heller]

Hello,

so REPLs usually incite heated debates over what is best in life and I don't want to start one of those again. I'm only looking for feedback. I obviously like this idea but as long as no tool supports it it is not that useful.

I want to propose a very small (about 100 LoC) extension (library) on top of clojure.main/repl. The only purpose of this is to simplify better tool support. The streaming model is kept as is, it just becomes extensible and inspectable from the outside.

To do this I introduce two new concepts: REPL "root" and REPL "level"

The "root" is whoever created the *in* and *out* streams. There will usually be one for the JVM but you can get additional ones via sockets but this is optional.

A "level" is created whenever a new read-eval-print-loop is started. Any loop aware of the above model can register itself as a level.

Each root can expose additional features as can every level but the basic contract is you read from *in* and print to *out*. From outside the REPL you get a very simple API to query and interact with these.

Let me try a brief example, this is inside a clojure.main REPL:

user=> (shadow.repl-demo/main)

REPL ready, type :repl/quit to exit

[1:0] user=> (shadow.repl-demo/repl)

[1:1] user=> :repl/quit

nil

[1:0] user=> :repl/quit

REPL stop. Goodbye ...

:repl/quit

user=> 

shadow.repl-demo/main just properly sets up the root and enters a new level which is just another clojure.main/repl. It modifies the prompt to show the current root/level. shadow.repl-demo/repl starts another level which you can see in the prompt. You can nest as many as you like and it is an important property of a REPL. Maybe not so useful for a CLJ REPL inside a CLJ REPL but very useful if you start a CLJS REPL for example.

For demo purposes I start a Socket REPL (extended to be aware of this API) and start a CLJS REPL inside that. The client connecting to this looks like:

rlwrap telnet localhost 5001

Trying ::1...

telnet: connect to address ::1: Connection refused

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

[2:0] user=> (shadow.devtools.api/node-repl)

[:node-repl] Build started.

[:node-repl] Build completed. (22 files, 1 compiled, 0 warnings)

cljs.user=> 

The [2:0] shows that the id of the root is 2 at level 0. Starting the node REPL added another level [2:1] (don't mind that the CLJS prompt doesn't show this, it is there).

From the other window I can query what is going on.

user=> (shadow.repl/roots)

{2

 {:shadow.repl/type :remote,

  :shadow.repl/levels

  [{:shadow.repl/lang :clj,

    :shadow.repl/get-current-ns

    #object[shadow.repl_demo$repl$fn__176 0x35c395cc "shadow.repl_demo$repl$fn__176@35c395cc"],

    :shadow.repl/root-id 2,

    :shadow.repl/level-id 0}

   {:shadow.repl/lang :cljs,

    :shadow.repl/get-current-ns

    #object[shadow.devtools.api$repl_level$fn__26443 0x651d2d15 "shadow.devtools.api$repl_level$fn__26443@651d2d15"],

    :shadow.repl/root-id 2,

    :shadow.repl/level-id 1}]}}

I skipped the other levels because they are not relevant now, but this is just a simple data structure. The root is a map and each level is a map. Each can contain namespaced keywords where each keyword would have a defined (as in clojure.spec'd maybe) meaning. The information exposed is limited for demo purposes but can be extended easily.

So both levels export a :shadow.repl/lang property to indicate which language it is. In addition both provide get-current-ns which is a function with no arguments that returns the current namespace of the loop.

user=> ((:shadow.repl/get-current-ns (shadow.repl/level 2 0)))

"user"

user=> ((:shadow.repl/get-current-ns (shadow.repl/level 2 1))) 

{:ns cljs.user, :name "cljs/user.cljs", ...}

Note that they both should return the same kind of data structure but I'm not sure how much information is necessary so I just made two examples. This query happened from outside the actual REPL so the loop is unaffected. In the actual CLJS REPL I can do

cljs.user=> (in-ns 'cljs.core)

cljs.core=> 

and query it again somewhere else

user=> ((:shadow.repl/get-current-ns (shadow.repl/level 2 1))) 

{:ns cljs.core, :name "cljs/core.cljs", ...}

I used two REPLs in this demo but access to the API shadow.repl provides could also be accessed remotely via an RPC API and whatever the tool in question prefers really. The REPL does not care how the features it provides are consumed. It could provide any feature that would be useful to a tool that would need to be executed in the context of the REPL (in a true lisp-ish fashion I guess). The loop should never be affected by what a tool does in the background as the user should be in control of that. Some things you may just send to *in* but others maybe are better placed somewhere else.

I hope this is somewhat understandable, the implementation currently resides here:

https://github.com/thheller/shadow-devtools/blob/master/src/main/shadow/repl.clj

The API does not have any dependencies and is very small. I would turn this into a standalone library if anyone agrees with me that this could be useful.

The demo code just shows the minimal implementation necessary:

https://github.com/thheller/shadow-devtools/blob/master/src/dev/shadow/repl_demo.clj

The interesting bits are repl/enter-root and repl/takeover.

I'd be happy to go into more detail about my motivation behind all this but that really isn't important. Yes this would directly compete with nREPL and this was born out of my frustrations with nREPL. I do not claim in any way that this would be better than nREPL. I just like this idea a lot and wanted to share it.

Cheers,

/thomas

PS: I'm bad with names and all this probably has prior art in the LISP world, so if there are better names than root and level I'd be happy to adopt those.

PS: This discussion somewhat inspired this:

https://groups.google.com/d/topic/clojure-dev/Dl3Stw5iRVA/discussion

PS: As did this https://www.youtube.com/watch?v=SrKj4hYic5A

[Alex Miller]

Rich has spoken in this group in the past in favor of stream-based (and nested) repls. See https://groups.google.com/d/msg/clojure-dev/Dl3Stw5iRVA/IHoVWiJz5UIJ (and other posts in that thread) for his thoughts.

You say "I'd be happy to go into more detail about my motivation behind all this but that really isn't important. " but I'd say that's probably more important than anything else in your post. You're pushing a solution without discussing the problem. What is the problem you are trying to solve? Why do you need nested repls? What do you need to do that isn't supported by the existing repls?

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

[Thomas Heller]

FWIW I think this is is very much the direction Rich was talking about, ie. streaming and nestable. Just also inspectable.

The problem I'm trying to solve is that tool support relies on having the tool implement support for every different implementation.

Let me pick up on the example above of "get-current-ns". It is a requirement for any tool that wants to show a "fancier" prompt or show any contextual information. Getting information about the current ns is a completely different implementation for CLJ compared to CLJS. I wrote a "new" CLJS REPL with some implementation differences, thereby it would require any tool to support yet another implementation. In my proposed solution a feature is either provided by the REPL or not. How it is implemented is of no interest to the tool as the keywords basically only provide the interface.

Also as a REPL author the example I want to show in my README is which dependency to add and what to run, ie. (my.ns/start-that-repl). Should the tool support any of its exposed features it can start using them, if the tool does not I just get the basics. I do not want to worry about the users REPL configuration. The current instructions for figwheel [1] to get nREPL to work are long and complex. Bruce told me that he is working on making this much easier. That would mean that I need to do the exact same work for my REPL. Then tools would need to agree on the new protocol and add support for that. In my proposal I would still need to implement the feature but I do not need to worry about the protocol used.

Also what tools currently cannot do is to detect that I entered a CLJS REPL. Cursive has an option to toggle between CLJ and CLJS mode. I always send at least one form to eval before I notice than I'm in the wrong mode. It could easily detect which mode I'm in based on what I suggested.

I don't need the nesting support really as CLJS eval happens outside the JVM. So that would work just as well without being nested inside another CLJ REPL. It is just convenient to run things this way and play around with things.

I'm really not looking for much, most of the features in CIDER go too far for me. I have been quite happy on a clojure.main REPL for a while now, Cursive already does the rest without a REPL. Still some things could be better but aren't mostly because I don't want to use nREPL. I understand the Cursive's perspective that adding features on top of clojure.main is really hard at the moment which sort of started all this.

/thomas

[1] https://github.com/bhauman/lein-figwheel/wiki/Using-the-Figwheel-REPL-within-NRepl

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

[Bozhidar Batsov]

On 27 February 2017 at 19:26, Thomas Heller <th.h...@gmail.com> wrote:

FWIW I think this is is very much the direction Rich was talking about, ie. streaming and nestable. Just also inspectable.

The problem I'm trying to solve is that tool support relies on having the tool implement support for every different implementation.

Let me pick up on the example above of "get-current-ns". It is a requirement for any tool that wants to show a "fancier" prompt or show any contextual information. Getting information about the current ns is a completely different implementation for CLJ compared to CLJS. I wrote a "new" CLJS REPL with some implementation differences, thereby it would require any tool to support yet another implementation. In my proposed solution a feature is either provided by the REPL or not. How it is implemented is of no interest to the tool as the keywords basically only provide the interface.

I'm still curious how this REPL helps the tooling for anything but really basic cases. E.g. how does it help with completion? 

Btw, have you seen https://github.com/cgrand/unrepl ?

 

Also as a REPL author the example I want to show in my README is which dependency to add and what to run, ie. (my.ns/start-that-repl). Should the tool support any of its exposed features it can start using them, if the tool does not I just get the basics. I do not want to worry about the users REPL configuration. The current instructions for figwheel [1] to get nREPL to work are long and complex. Bruce told me that he is working on making this much easier. That would mean that I need to do the exact same work for my REPL. Then tools would need to agree on the new protocol and add support for that. In my proposal I would still need to implement the feature but I do not need to worry about the protocol used.

Btw, why not try to enhance nREPL instead? I'm assuming that getting changes there would be way easier than in Clojure's REPL and so many tooling is already relying on nREPL. I don't think it has some fundamental unsolvable issues - it just needs love as it's basically unmaintained lately. 

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.

[Thomas Heller]

Yes, I'm aware of unrepl but it does somewhat solve a different issue than I was trying to solve.

shadow.repl was born out of the idea that I wanted to be able to inspect the state of any running REPL. unrepl does not cover that. You still rely on it sending you messages. In my example I used "get-current-ns" which is somewhat similar to the :prompt unrepl would send but instead anyone can ask for it. The :prompt message would only be received by whoever is listening to the *out* of the unrepl.

I made shadow.repl extensible so that other things like completion could be done over that. I changed my mind on that subject though.

Here is my current thought process about how completion or other features should work. First of all for completion you need the current NS (covered above) and everything else that follows is not related to the REPL at all. You just hijacked the REPL connection for your RPC-ish request to get some information about the runtime. Do you really need to do that over the REPL? Wouldn't it be simpler to have another endpoint somewhere that does real RPC and does not go over the REPL at all? Heck you could curl http://localhost:64223/complete.edn?ns=user&prefix=foo.

Visual Studio Code introduced the concept of a language server [1] which really appeals to me. The basic idea is that you start a server with a defined protocol the editor can interact with. Completions is just a RPC request away, as are many other features I don't think any of the current Clojure tooling do.

The point here is though: This does not use the REPL, nothing of that is a REPL. From a tooling perspective the only thing I imagine you really need from a REPL is its current NS and the language it represents (ie. CLJS, CLJ or others). Everything else is just hijacking a REPL for something that isn't a REPL in the first place.

As Rich said: 

REPL stands for something - Read, Eval, Print, Loop. 

It does not stand for - Eval RPC Server/Window. 

That is my main issue with nREPL as well, you patch many other features into it and sacrifice the basic promise of a REPL in the process. Many of the features of CIDER, Cursive, lighttable or other editors do not need to use a REPL. Basically every time your editor sends something into the REPL the user did not type you do not really need a REPL but would be better off with something RPC-ish (or remote messaging, some things just aren't RPC).

Say you instead have a running language server JVM process. Nothing is keeping you from also starting a Socket REPL in that process. The editor talks to the language server for its editor things (completions) while each REPL connection remains separate from that. shadow.repl would provide a way for the editor to query what is going on in each REPL. Again: it would not need to hijack that REPL or inject commands into it. It can query it out-of-band through its already existing tooling connection. Since everything has equal access to the runtime they all can modify and query it.

unrepl tries to solve the issue of how to better present the flow of a REPL interaction to the user, although Christophe also hinted at some other things that would go past that.

My issue with that again however is that I would really want to be able to do different things in my editor.

- Sometimes I just want a REPL nothing more, clojure.main/repl is enough. I don't need more.

- Sometimes I want to eval something and modify the runtime by doing so, seeing the result in some other place (ie. by doing a HTTP request somewhere else). load-file is probably the best example for this, the result really isn't all that useful. The only thing that matters that it completed and the runtime now has the updated state. I may care more about some printed output in the process rather than the actual result.

- Sometimes I really want to inspect the resulting value in detail. Might just be macroexpand but might also be some deeply nested object that just plainly is too large to print. I have on numerous occasions blown up my REPL by accidentally printing something that was several hundred megabytes big. If I had the option I would not use a REPL for these instances, instead some kind of inspector would be cool.

To some extent nREPL is kind of the language server I have in mind, it just also pretends to be a REPL. By complecting these things together you end up with a system that is much more complicated than it would otherwise need to be (IMHO, YMMV). I really don't want to use VSCode but I'm very curious about the effect a true language server would have which the editor already understands. So I will probably start implementing a basic one for clojure whenever I find the time. There is also a client for emacs [2] already, which might be interesting.

Cheers,

/thomas

[1] https://github.com/Microsoft/language-server-protocol (JSON-RPC is not great, ignore that)

[2] https://github.com/Microsoft/language-server-protocol/wiki/Protocol-Implementations#editors-ides-supporting-the-protocol

[Andrea Richiardi]

On Thursday, March 9, 2017 at 3:48:52 AM UTC-8, Thomas Heller wrote:

As Rich said: 

REPL stands for something - Read, Eval, Print, Loop. 

It does not stand for - Eval RPC Server/Window. 

That is my main issue with nREPL as well, you patch many other features into it and sacrifice the basic promise of a REPL in the process. Many of the features of CIDER, Cursive, lighttable or other editors do not need to use a REPL. Basically every time your editor sends something into the REPL the user did not type you do not really need a REPL but would be better off with something RPC-ish (or remote messaging, some things just aren't RPC).

Hi Thomas! Just to understand your train of thoughts better, in case of completion you actually did press a character there. Would it still be considered a REPL thing?

I definitely agree that some operations are not concerning the REPL per se, but you see the line there is very thin. To me the problem you solved of nesting CLJ and CLJS REPL is the hardest one to solve and the one that nRepl fails big time at. Upgradable REPLs like yours and unrepl one would solve this. The IDE interaction is a different one imho. I would even go as far as proposing one REPL that communicates with one or more external services (one for completion, linting, ...). The external service can do static analysis (like Cursive does not), and asks/receives updates from the REPL about the runtime state.

It is definitely a thorny problem we need to address, I just wish we could all sit at a table and discuss this (the unrepl session at EuroClojure was definitely too short.

Thanks for sharing your ideas!

Andrea. 

[Thomas Heller]

In case of completion you did press a character but it did not cross the wire which means it did not enter the REPL yet. Any tool will buffer the REPL input and attempt to provide completion based on what it has buffered. It can obtain that completion results by sending something else over the REPL before sending the actual input which is what many tools do. However there are other ways the information could be obtained.

The problem is that the tool needs to know which state the REPL is in: Which language is it? Which namespace is it? etc.

Many tools achieve this by basically spamming the REPL with extra commands but that means that it needs to parse the REPL character stream to get the data it needs. That is a hard problem to solve since it may be interleaved with all kinds of other characters. It is also hard since CLJS may not understand the commands if they were intended for CLJ. unrepl tries to solve this by structuring the printed characters by "upgrading" and then downgrading (:bye) if something tries to Read in Eval.

In my proposed solution you instead just need a regexp to look for "[1:0]~shadow.user=> ". Parsing that is simple and conveys enough context to do everything else. The tool can use those two numbers (root id, level id) to query the REPL state over either a second connection or some other protocol. The tool never sends anything over the actual REPL connection other that what the user inputs.

Since the data the tool requires uses a proper protocol it does not need to worry about actually parsing the REPL output. The tool just assumes that it will find a prompt at some point in the character stream. If it doesn't find a prompt after sending something it "downgrades" and doesn't provide completion until it does.

The user may either switch the Print in REPL or call things like clojure.inspector directly to get pretty "formatted" output. Not something the tool needs to worry about.

The gist of it is to leave the REPL alone and just treat it as a stream of characters in both directions. By adding a tiny bit of structure to the prompt we can get enough information to do everything else. If the user uses a supported REPL the tool can provide additional support. If it never finds the "prompt" it just doesn't attempt to provide anything and treats it as text in/text out, which is good enough most of the time.

Everything the tool wants to do is not part of the REPL but it can use the REPL to bootstrap itself and run its own loop, just not over the connection the user is using.

Thats how I think about it at least, whether or not that is actually better than what we have now I don't know.

[Ewen Grosjean]

Hi Thomas,

I believe replique has similarities with what you are describing.