Some thoughts on the socket REPL

[Bozhidar Batsov]

Hi everyone,

With 1.8.0-alpha5 out I guess it's finally time to start a discussion regarding the new socket repl that'll be part of 1.8. As some of you know I'm the main CIDER maintainer and my remarks here are mostly driven my desire to be able to leverage the new REPL as an alternative to nREPL in future CIDER versions. So, let's start:

1. The name - why call it socket repl instead of a repl server or something like this? After all, pretty much everything involving some networking will be socket something. Clearly that's a minor nitpick, but still...

2. Dealing with tooling Clojure code

A tool like CIDER derives much of its functionality from evaluating some Clojure code (e.g. for completions, documentation, etc). The problem is that this code can't the evaluated in main user REPL, as it'd interfere with the evaluations the users performed by themselves (it will mess up special vars, lock their main session while those evaluations take place, etc). To make this more concrete - you don't want you're autocompletion code results to leak in the user's REPL. nREPL has different sessions per connection (and middleware which operates outside the scope of regular evaluations), so this was easy to achieve there. It'd be nice if the new REPL has some way to separate user-initiated evaluations from tooling-initiated ones. 

Maybe there's some provision for this, but I certainly can't find it. Without the notion of a several sessions per connection I don't see how something like CIDER can work with the new REPL.

3. Interrupting evalutions

I can't understand how are supposed to interrupt an evaluation which is currently in progress. Perhaps this is something considered obvious and therefore not part of the design spec, but it's not obvious to me. In particular it seems to me that all evaluations are synchronous which makes it harder for me to grasp how can you interrupt something that has blocked the REPL process.

We can live without this, but it will provide terrible user experience, as every long evaluation would force you to restart the server. 

4. EDN

The use of EDN will be a boon for Java-based editors and IDEs, but handling EDN in Emacs will be pretty hard. I implore you to make it possible to use another data format as well (if it's bencode you'll make our lives way easier). 

I'm guessing this will be problem for many tools as well.

I'm aware we can drop down to dealing with plain text for requests and results, but I'd rather not do this. 

5. Async evaluation

Any plans for this? I doubt so, but still I wanted to ask.

===========

Probably there are other things to consider as well, but this is all I have for now. I'd love to hear what more people think about these issues I outlined.

P.S. nREPL support in CIDER is not going anywhere. We want to support both REPLs going forward, but whether this will happen depends mostly on whether it'd be technically possible to use the socket REPL. 

[Laurent PETIT]

Hi, 

I'll try to answer these questions, with at least how I understand things could be done.

First remark: I'm certain it is by design that it doesn't do all the things you're asking for. Because they can be provided either by plugging another function than the one provided by default with clojure 1.8. So the socket repl is just that: the possibility to startup *several* socket severals at the same time as Clojure, and a default server provided in clojure.socket.repl which reproduces what you're used to with stdin/stdout socket. From there, everything else can be built.

Let's now answer inline more specifically:

 

2015-09-13 11:37 GMT+02:00 Bozhidar Batsov <bozh...@batsov.com>:

Hi everyone,

With 1.8.0-alpha5 out I guess it's finally time to start a discussion regarding the new socket repl that'll be part of 1.8. As some of you know I'm the main CIDER maintainer and my remarks here are mostly driven my desire to be able to leverage the new REPL as an alternative to nREPL in future CIDER versions. So, let's start:

1. The name - why call it socket repl instead of a repl server or something like this? After all, pretty much everything involving some networking will be socket something. Clearly that's a minor nitpick, but still...

2. Dealing with tooling Clojure code

A tool like CIDER derives much of its functionality from evaluating some Clojure code (e.g. for completions, documentation, etc). The problem is that this code can't the evaluated in main user REPL, as it'd interfere with the evaluations the users performed by themselves (it will mess up special vars, lock their main session while those evaluations take place, etc). To make this more concrete - you don't want you're autocompletion code results to leak in the user's REPL. nREPL has different sessions per connection (and middleware which operates outside the scope of regular evaluations), so this was easy to achieve there. It'd be nice if the new REPL has some way to separate user-initiated evaluations from tooling-initiated ones. 

Maybe there's some provision for this, but I certainly can't find it. Without the notion of a several sessions per connection I don't see how something like CIDER can work with the new REPL.

There are several ways: 

- the simplest = you use the provided repl server, and initiate one connection for the user, and as many connection as you see fit for the tooling.

- the more powerful one = you start one server for the user with the provided repl server, and you start another server with your own implementation for tooling (this would be the replacement for nrepl).

There's a paradigm shift: it's not like nrepl when nrepl was in control of the server / socket side. Now you plug in a socket implementation below the server side. It's more composable.

 

3. Interrupting evalutions

I can't understand how are supposed to interrupt an evaluation which is currently in progress. Perhaps this is something considered obvious and therefore not part of the design spec, but it's not obvious to me. In particular it seems to me that all evaluations are synchronous which makes it harder for me to grasp how can you interrupt something that has blocked the REPL process.

You simply close the client side connection?

 

We can live without this, but it will provide terrible user experience, as every long evaluation would force you to restart the server. 

4. EDN

The use of EDN will be a boon for Java-based editors and IDEs, but handling EDN in Emacs will be pretty hard. I implore you to make it possible to use another data format as well (if it's bencode you'll make our lives way easier). 

I'm guessing this will be problem for many tools as well.

I'm aware we can drop down to dealing with plain text for requests and results, but I'd rather not do this. 

I don't understand. EDN is not imposed. And anyway, with nREPL your client needs to know 2 things: bencode, AND EDN since generally you get serialized clojure datastructures back.

With the socket repl, you're back to just EDN, or whatever you want.

 

5. Async evaluation

Any plans for this? I doubt so, but still I wanted to ask.

I also doubt so, but of course there are 2 easy ways to achieve this:

- on the client-side : you create one connection per call, on separate threads, and manage the asynchronicity directly in the client.

- on the server-side : with an enhanced tooling repl which accepts advanced commands. You can either install this server side code by injecting it through the connection "live", or because you started the project JVM via the editor/IDE and took care of adding its jar to the classpath. Then you're just one call to some "enhanced-repl" function away from upgrading the regular repl to a more advanced one.

I'm very excited by the socket repl stuff overall, because it seems to turn the problem upside-down for the better.

 

===========

Probably there are other things to consider as well, but this is all I have for now. I'd love to hear what more people think about these issues I outlined.

P.S. nREPL support in CIDER is not going anywhere. We want to support both REPLs going forward, but whether this will happen depends mostly on whether it'd be technically possible to use the socket REPL. 

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--

Laurent Petit

[Alex Miller]

Hey Bozhidar!

You have many good questions. I had a great conversation with Colin on Friday and we covered some of the same territory. An earlier version of the patch included the ability to attach from one session to another and eval in its environment. We ended up pulling that part out of the patch and I think after talking to Colin that it is not needed as an explicit feature (but is still achievable with more hooks on setup). I am working on a proof of concept for how this can be done and will follow-up in the next couple days.

Alex

[Laurent PETIT]

Wow, how come I had totally overlooked the "session" part of your question, Bozhidar ? :-)

--

You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--

Laurent Petit

[Bozhidar Batsov]

On 13 September 2015 at 15:29, Laurent PETIT <lauren...@gmail.com> wrote:

Hi, 

I'll try to answer these questions, with at least how I understand things could be done.

First remark: I'm certain it is by design that it doesn't do all the things you're asking for. Because they can be provided either by plugging another function than the one provided by default with clojure 1.8. So the socket repl is just that: the possibility to startup *several* socket severals at the same time as Clojure, and a default server provided in clojure.socket.repl which reproduces what you're used to with stdin/stdout socket. From there, everything else can be built.

Let's now answer inline more specifically:

 

2015-09-13 11:37 GMT+02:00 Bozhidar Batsov <bozh...@batsov.com>:

Hi everyone,

With 1.8.0-alpha5 out I guess it's finally time to start a discussion regarding the new socket repl that'll be part of 1.8. As some of you know I'm the main CIDER maintainer and my remarks here are mostly driven my desire to be able to leverage the new REPL as an alternative to nREPL in future CIDER versions. So, let's start:

1. The name - why call it socket repl instead of a repl server or something like this? After all, pretty much everything involving some networking will be socket something. Clearly that's a minor nitpick, but still...

2. Dealing with tooling Clojure code

A tool like CIDER derives much of its functionality from evaluating some Clojure code (e.g. for completions, documentation, etc). The problem is that this code can't the evaluated in main user REPL, as it'd interfere with the evaluations the users performed by themselves (it will mess up special vars, lock their main session while those evaluations take place, etc). To make this more concrete - you don't want you're autocompletion code results to leak in the user's REPL. nREPL has different sessions per connection (and middleware which operates outside the scope of regular evaluations), so this was easy to achieve there. It'd be nice if the new REPL has some way to separate user-initiated evaluations from tooling-initiated ones. 

Maybe there's some provision for this, but I certainly can't find it. Without the notion of a several sessions per connection I don't see how something like CIDER can work with the new REPL.

There are several ways: 

- the simplest = you use the provided repl server, and initiate one connection for the user, and as many connection as you see fit for the tooling.

- the more powerful one = you start one server for the user with the provided repl server, and you start another server with your own implementation for tooling (this would be the replacement for nrepl).

There's a paradigm shift: it's not like nrepl when nrepl was in control of the server / socket side. Now you plug in a socket implementation below the server side. It's more composable.

 

3. Interrupting evalutions

I can't understand how are supposed to interrupt an evaluation which is currently in progress. Perhaps this is something considered obvious and therefore not part of the design spec, but it's not obvious to me. In particular it seems to me that all evaluations are synchronous which makes it harder for me to grasp how can you interrupt something that has blocked the REPL process.

You simply close the client side connection?

I thought of this, but it seems to be that re-establishing the connection will introduce a bit of networking overhead. I can live with this, I just wondered if we can do better somehow. 

 

 

We can live without this, but it will provide terrible user experience, as every long evaluation would force you to restart the server. 

4. EDN

The use of EDN will be a boon for Java-based editors and IDEs, but handling EDN in Emacs will be pretty hard. I implore you to make it possible to use another data format as well (if it's bencode you'll make our lives way easier). 

I'm guessing this will be problem for many tools as well.

I'm aware we can drop down to dealing with plain text for requests and results, but I'd rather not do this. 

I don't understand. EDN is not imposed. And anyway, with nREPL your client needs to know 2 things: bencode, AND EDN since generally you get serialized clojure datastructures back.

With the socket repl, you're back to just EDN, or whatever you want.

On second thought - you're right. I just realized that not much will change - we'll send the EDN as a string (which is easy) and we'll either dump the EDN as a string or will return some form which would compatible with the Emacs Lisp reader (that's what we do right now). Ideally, we'd have some proper support for EDN some day, but we can probably live without it.

 

 

5. Async evaluation

Any plans for this? I doubt so, but still I wanted to ask.

I also doubt so, but of course there are 2 easy ways to achieve this:

- on the client-side : you create one connection per call, on separate threads, and manage the asynchronicity directly in the client.

- on the server-side : with an enhanced tooling repl which accepts advanced commands. You can either install this server side code by injecting it through the connection "live", or because you started the project JVM via the editor/IDE and took care of adding its jar to the classpath. Then you're just one call to some "enhanced-repl" function away from upgrading the regular repl to a more advanced one.

I'm very excited by the socket repl stuff overall, because it seems to turn the problem upside-down for the better.

Let's hope you're right! 

[Bozhidar Batsov]

On 14 September 2015 at 05:33, Alex Miller <al...@puredanger.com> wrote:

Hey Bozhidar!

You have many good questions. I had a great conversation with Colin on Friday and we covered some of the same territory. An earlier version of the patch included the ability to attach from one session to another and eval in its environment. We ended up pulling that part out of the patch and I think after talking to Colin that it is not needed as an explicit feature (but is still achievable with more hooks on setup). I am working on a proof of concept for how this can be done and will follow-up in the next couple days.

Great! I'm looking forward to this.

 

Alex

On Sunday, September 13, 2015 at 4:37:36 AM UTC-5, Bozhidar Batsov wrote:

Hi everyone,

With 1.8.0-alpha5 out I guess it's finally time to start a discussion regarding the new socket repl that'll be part of 1.8. As some of you know I'm the main CIDER maintainer and my remarks here are mostly driven my desire to be able to leverage the new REPL as an alternative to nREPL in future CIDER versions. So, let's start:

1. The name - why call it socket repl instead of a repl server or something like this? After all, pretty much everything involving some networking will be socket something. Clearly that's a minor nitpick, but still...

2. Dealing with tooling Clojure code

A tool like CIDER derives much of its functionality from evaluating some Clojure code (e.g. for completions, documentation, etc). The problem is that this code can't the evaluated in main user REPL, as it'd interfere with the evaluations the users performed by themselves (it will mess up special vars, lock their main session while those evaluations take place, etc). To make this more concrete - you don't want you're autocompletion code results to leak in the user's REPL. nREPL has different sessions per connection (and middleware which operates outside the scope of regular evaluations), so this was easy to achieve there. It'd be nice if the new REPL has some way to separate user-initiated evaluations from tooling-initiated ones. 

Maybe there's some provision for this, but I certainly can't find it. Without the notion of a several sessions per connection I don't see how something like CIDER can work with the new REPL.

3. Interrupting evalutions

I can't understand how are supposed to interrupt an evaluation which is currently in progress. Perhaps this is something considered obvious and therefore not part of the design spec, but it's not obvious to me. In particular it seems to me that all evaluations are synchronous which makes it harder for me to grasp how can you interrupt something that has blocked the REPL process.

We can live without this, but it will provide terrible user experience, as every long evaluation would force you to restart the server. 

4. EDN

The use of EDN will be a boon for Java-based editors and IDEs, but handling EDN in Emacs will be pretty hard. I implore you to make it possible to use another data format as well (if it's bencode you'll make our lives way easier). 

I'm guessing this will be problem for many tools as well.

I'm aware we can drop down to dealing with plain text for requests and results, but I'd rather not do this. 

5. Async evaluation

Any plans for this? I doubt so, but still I wanted to ask.

===========

Probably there are other things to consider as well, but this is all I have for now. I'd love to hear what more people think about these issues I outlined.

P.S. nREPL support in CIDER is not going anywhere. We want to support both REPLs going forward, but whether this will happen depends mostly on whether it'd be technically possible to use the socket REPL. 

--

[Alex Miller]

To help demonstrate a bit what is possible with the combination of the new socket server and the existing normal clojure.main/repl (which is almost excessively customizable), I created a repo with a readme and some bits and pieces of code. This is just a sketch, not a library - you will likely need to modify this to use it in a real tool like CIDER, Cursive, etc:

https://github.com/puredanger/replicant (because Blade Runner) 

The readme should cover the main idea, so I will not repeat that here. The basic idea is that you have the opportunity to do anything you want in the accept function of the socket server. And if you want to run a REPL, you have the opportunity to customize every aspect of that REPL, including how it reads, evaluates, prints, etc. Included in the repo are facilities for two different repl customizations.

The first is a data-centric repl that is designed for tooling use. This repl turns off the normal "user>" prompt and expects to simply receive a string that is a form to evalute. It will then evaluate the form, capture the result data (or the exception data) and return it in a map. The *out* and *err* buffers are captured and sent along as additional keys in the map (so they cannot be intermingled with the result data). The key takeaway here is - you can encode that data in whatever way you like. You could insert an entire middleware layer in there.

The second is a repl that is designed to be what the user sees, with one important specialization - the user's entire dynamic var environment is saved off after every evaluation. (I haven't done it here, but there is a bit more work to actually save off *1, *2, *3 because of where those get modified in clojure.main/repl, but that is also doable.) Those bindings are then stashed in an atom.

We stitch these together by having the tooling repl connection programatically start the socket server the client will connect to and create the evaluation function with an atom that the tooling repl connection also can see.

Once you have created both servers and both client connections, you then having a tooling repl that can evaluate things in the context of the user's repl environment.

Taking the questions/points in the first email:

1) Name - I don't really care much about the name of whatever here. In my mind, the "feature" here is really the socket server (which happens to compose with the existing repl function). 

2) Tooling - one approach is demonstrated above (there are others too I think)

3) Interruption - this is an interesting question I haven't previously thought about much. Because you are saving binding state, you could theoretically kill the socket from the client side, start a new client connection, and restore the original bindings. I haven't tried this to see whether that will interrupt activity on the server side - you might need something more to actually kill the thread doing the bad activity. You could certainly interrupt the thread or even stop it (although this releases all held monitor which can have some bad effects).

4) EDN - as I demonstrated on the data repl, you can handle input and output paths in ways that are appropriate for you.

5) Async - curious what your goals are exactly? Certainly seems plausible to make such a thing given the tools at hand, but the devil is in the details. Since you control eval, you could spin a future, add it to some global map of pending and return 

[Ewen Grosjean]

For EDN in emacs, you might be interested by https://github.com/EwenG/replique.el/blob/master/replique-edn.el

[Bozhidar Batsov]

I'll take a look at this. You might consider developing it as a separate package (and replacing the usage of dash with seq.el), as an EDN parser would useful in general.