nREPL middleware command generation

[Phil Hagelberg]

I've been thinking a bit about nREPL tooling and how I feel like I'm
doing a bit of wheel-reinventing whenever I want to expose some Clojure
functionality in nrepl.el.

Most of the commands I've added to nrepl.el simply embed a string of
Clojure code inside the elisp to be evaluated server-side whenever the
command is invoked. I feel like most of this functionality could be
implemented server side and a simple client-specific UI for it could be
generated from a declarative description of the operation. Middleware
already has a mechanism and convention for self-describing upon which we
can build.

For instance, looking up a docstring could be implemented by a
middleware that self-describes as such:

{:op "doc"
:doc "Display the docstring of a given var."
:arglist [{:prompt "Var: " :type :var}]
:client-handler [:message]}

This tells the client that the "doc" operation takes a single var
argument (and how to prompt the user) and that the response should be
handled by displaying a multi-line message to the user. In Emacs, that
would be done by opening a new temporary buffer for longer messages or
showing in the status area for one-liners; other clients would behave
as expected based on their expected UI paradigms. Clients could
optionally offer completion based on type.

Similar operations could be easily described for bread-and-butter
operations like jump-to-definition, macroexpand, and pprint. Of course,
this is amounts to inventing a new language, with all the perils and
promise you'd expect with such an endeavour. I think it makes sense to
keep the language very limited and require fancier operations to
implement their own clients, but I feel like you could get a lot of
benefit from a simple list of argument types and client handlers.

Ideas for arglist types to support:

* string
* var
* namespace
* file
* file/line/column tuple (maybe call this a "position"?)
* choose from N predefined options

Supported client handlers could include:

* message
* insertion/deletion
* color overlays?

Proposing inventing an entirely new language feels a bit crazy, but
continuing to write boilerplate Elisp commands which just wrap
well-defined Clojure functionality is getting me down. Is it a good idea
to begin with? Is this a good set of types and handlers to start with?
Is it even possible to get Emacs, Eclipse, and Vim users to agree on
anything UI-related?

-Phil

[Chas Emerick]

A couple of thoughts:

* this is awesome
* you don't need a "new language", we have Clojure
* each of these operations doesn't need to be a separate middleware, esp. insofar as some have pointed out that, for simple things, writing middleware isn't the easiest of endeavors.

The "doc" descriptor looks to me like function metadata that provides something approximating static types for that function (:client-handler really is the return type). I'd suggest that that's where it should stay.

Step two is to have a single middleware that can find vars in the runtime that have the appropriate middleware, and wire them up as dispatch endpoints for ops. This means that, to implement a simple op, one needs only to decorate the entry point fn that implements the op, not all the other stuff that goes along with implementing middleware (which, honestly, is only useful/worth it if you're doing middleware-esque things that a regular function can't, e.g. transforming the nREPL request or response messages). Phil, perhaps this is what you meant when you mentioned classpath scanning in irc?

This also means that clients will no longer need to cons up Clojure forms to invoke these commands (an error-prone PITA thing to do dynamically anywhere you don't have syntax-quote): rather, if every argument is named (i.e. no positional arguments allowed), then calling one of these commands only involves sending an nREPL message with the right :op, and all of the arguments to it named as specified in the metadata.

Finally, this all flows nicely into what's already there around "describe" (each function will appear to be a separate op in its output), so discovery falls out naturally.

Potential downsides/pitfalls:

* We'd be deepening the commitment to a separate namespace system...though it will largely be populated with "normal" functions based on var metadata, and will cut out a large portion of the code generation being done by clients currently, which I'm all for.
* Figuring out truly "portable" return types seems hard. This is essentially already in progress with various middleware/command efforts (clojure-complete, ritz, etc), but most of the consumption has been from Emacs, and...emacs. :-P But, it's a process that has to happen regardless of how the operations in question are wired up.
* Somewhere, there needs to be support for toggling the "context" (what a horrible term) of a session. i.e. piggieback effectively changes the "context" to use a different eval once a ClojureScript REPL is started on a session. That action should trigger a wholesale swap of operations, so everything from documentation lookup to code completion to jump-to-definition talks to the ClojureScript compiler and the session's ClojureScript environment rather than looking at Clojure namespaces. This may be a bigger, hairier topic for another day, but I thought I'd throw it out there since it's at least tangentially related.
* ...I'm sure there's more, but that's what I've got off the top of my head.

Cheers,

- Chas

[Phil Hagelberg]

Chas Emerick writes:

> * you don't need a "new language", we have Clojure

Well... we're describing a language with new semantics which happens to
borrow Clojure's syntax but is designed to be compiled into N different
target runtimes. For most clients, Clojure code is not going to be
running client-side, and even when it is, it won't be the same Clojure
code that describes the ops.

> The "doc" descriptor looks to me like function metadata that provides
> something approximating static types for that function
> (:client-handler really is the return type). I'd suggest that that's
> where it should stay.

I intended the :doc key to be for human-readable description of an
operation that the client could attach to a given command. But if ops
are just vars then the var's docstring could be used instead? Though I
could see in some cases it'd be good to have one docstring server side
and a separate one client-side.

> Step two is to have a single middleware that can find vars in the
> runtime that have the appropriate middleware, and wire them up as
> dispatch endpoints for ops. This means that, to implement a simple
> op, one needs only to decorate the entry point fn that implements the
> op, not all the other stuff that goes along with implementing

> middleware.

Good point; one middleware can spider all known vars (or vars under a
certain namespace prefix) in order to find which have been annotated
with (say) :nrepl/op metadata.

An open question here is whether the spidering would be limited to
already-loaded namespaces or whether it would attempt to require
namespaces by scanning the classpath. Probably best to stick to the
former at first; auto-requiring can be added after the fact without
affecting things much, and we don't need to have this all figured out
(scanning the classpath at runtime vs having whatever launches the repl
be responsible for finding manifests and ensuring things get loaded)
to proceed.

> * Figuring out truly "portable" return types seems hard. This is
> essentially already in progress with various middleware/command
> efforts (clojure-complete, ritz, etc), but most of the consumption has
> been from Emacs, and...emacs. :-P But, it's a process that has to
> happen regardless of how the operations in question are wired up.

Yep, definitely. Hugo mentioned that URLs should be a return type as
well, which makes lots of sense. I'm not sure how common support for
colored overlays is, but messaging and insertion/deletion should
definitely be lowest-common-denominator material. Though of course
only messages and URLs could be supported by clients like reply.

-Phil

[Phil Hagelberg]

Another question I meant to bring up: should this be part of nREPL
itself or a third-party middleware?

There's no technical reason it couldn't live in a third-party library,
but given that these conventions won't be useful without widespread
adoption maybe keeping it in nREPL proper would be prudent as long as
Jira could be avoided.

-Phil

[Phil Hagelberg]

I've implemented a proof-of-concept here:

https://github.com/technomancy/nrepl-discover

It contains a reference implementation of a `toggle-trace` command
which uses tools.trace as well as a simple elisp implementation that
only supports strings as args and return values. Check out the readme
for an overview of open questions.

Feedback welcome, especially from people who might want to implement
clients for other runtimes.

-Phil

[Phil Hagelberg]

Phil Hagelberg writes:

> I've implemented a proof-of-concept here:
>
> https://github.com/technomancy/nrepl-discover

I've documented the interface proposed for this functionality in greater
detail here:

https://github.com/technomancy/nrepl-discover/blob/master/Proposal.md

The argument types listed in the proposal are all fairly straightforward
with the exception of `eval`; the idea for this came about when I
thought about how to implement "run this one test". We have an argument
type for vars, which would be sufficient, but it would be great to
narrow it down to just the vars in the current namespace which have
`:test` metadata. So that serves as a sort of escape hatch for when
the existing types aren't rich enough.

I'm fully aware that I'm coming at this from an Emacs-biased angle, but
I was able to implement support for nearly everything in the proposal in
a matter of hours. I'm definitely interested in hearing from others
who are considering implementing it in other runtimes. My next plan is
to look at moving certain features in nrepl.el over to the server side;
I think the `tools.nrepl` library should probably ship with things like
docstring support and jump-to-definition. And load-file should be
auto-discovered rather than hard-coded into each client too.

After review, I believe the second half of the proposal (the Responses
section) is only tangentially related to auto-discovery of ops; it's
simply some response types I have found to be necessary for implementing
certain ops that I would like to use with auto-discovery. I believe the
"message", "text", "position", and "url" responses should become
standardized as a regular part of the nREPL protocol, though perhaps
some distinction should be made between clients which intend to provide
a full environment vs clients like reply which only intend to offer a
CLI.

Support for overlays isn't as much of a no-brainer, but it's really
great to have for implementing things like test runners which can
indicate where failures occur or possibly giving feedback about
instrumentation. I've also thought of an `edit` response which could
indicate that the editor should make some change to the namespace, but I
haven't really thought this through; it may not be necessary.

These should probably be evaluated independently from the auto-discovery
proposal, which simply consists of standardizing on the metadata to
attach to operations: the `:nrepl/op` map containing `:name`, `:doc`,
and `:args` keys and deciding which operations to ship with
`tools.nrepl` itself.

-Phil

[Phil Hagelberg]

I've attached a patch to tools.nrepl which adds this middleware but
doesn't add it to the default stack or implement any built-in
ops. There's a docstring with a summary, but I thought I'd wait to go
into detail describing the completion types until I got a bit more
feedback on the topic.

-Phil

[Phil Hagelberg]

> feedback on the topic. 

I'm planning on making a new release of Leiningen soon.

If there's a chance that a new nrepl release containing this small patch could be released soon, I would love to bring it in.

-Phil 

[Chas Emerick]

Seems like a perfectly nice middleware that can exist in a library? ;-)

More seriously, I've been trolling around some lately, looking at the current state of nREPL tooling (the various middlewares that are out there, custom libs like clojure-complete, etc), and mulling over the detailed proposal you put together over at nrepl-discover.  Something _like_ the suggested `clojure.tools.nrepl.middleware.discover` namespace will definitely find its way into nREPL proper eventually, but I'm currently very unsure of the particular path outlined by nrepl-discover (where ops end up essentially driving user interfaces, very explicitly in some places).  On the other hand, I think it might not be doing enough, insofar as classpath-discovery may be a far superior alternative in many cases for building middleware stacks than the fully-explicit approach currently in place (i.e., what's the point of discovering ops if you can't rope them into the stack that's sitting on the endpoint to which you're connected?).

http://dev.clojure.org/jira/browse/NREPL-29 is tangentially related to the above, and there's some other bug-squashing that I need to attend to before continuing to scale the tooling abstractions in nREPL proper…

- Chas

[Phil Hagelberg]

Chas Emerick writes:

> Something _like_ the suggested
> `clojure.tools.nrepl.middleware.discover` namespace will definitely
> find its way into nREPL proper eventually, but I'm currently very
> unsure of the particular path outlined by nrepl-discover (where ops
> end up essentially driving user interfaces, very explicitly in some
> places).

Can you give some more detail about your hesitations? The server-driven
UI aspects were specifically motivated by a desire to avoid having to
reinvent the wheel for every client, since this kind of drudgery has
repeatedly put me off from implementing certain features in the past.

> On the other hand, I think it might not be doing enough, insofar as
> classpath-discovery may be a far superior alternative in many cases
> for building middleware stacks than the fully-explicit approach
> currently in place

I agree; here I specifically kept it from doing too much because I
wasn't sure how the classpath discovery would look and would prefer to
experiment with in Leiningen where I can find the right answer by
iterating more quickly.

-Phil

[Chas Emerick]

On Aug 6, 2013, at 7:45 PM, Phil Hagelberg wrote:

>
> Chas Emerick writes:
>
>> Something _like_ the suggested
>> `clojure.tools.nrepl.middleware.discover` namespace will definitely
>> find its way into nREPL proper eventually, but I'm currently very
>> unsure of the particular path outlined by nrepl-discover (where ops
>> end up essentially driving user interfaces, very explicitly in some
>> places).
>
> Can you give some more detail about your hesitations? The server-driven
> UI aspects were specifically motivated by a desire to avoid having to
> reinvent the wheel for every client, since this kind of drudgery has
> repeatedly put me off from implementing certain features in the past.

There's a lot of conflation of concerns; if they are to be portable w.r.t. clients/tools, op requests and responses should be all about the information involved, not transient, tool-specific concerns like prompts, UI models (the whole "overlay" thing?), colors, etc. Even if it weren't an architectural tangle, the long-term internationalization, localization, and accessibility complexity would scare me (or, any op author thinking long-term).

Anyway, I ended up thinking out loud a bit on this topic over here: https://github.com/pallet/ritz/issues/105 ...the (maybe too abstract) tl;dr might be that I'd like to see ops authors (and tooling authors generally) move away from providing APIs, towards implementing functionality that generically handles data shaped in particular ways. That's what I was aiming for when I originally put in the mechanisms for supporting discovery, self-documentation, etc. (Not that the default ops are necessarily exemplars of what I'm talking about now -- especially since they're so intimately involved with the operational semantics of the whole stack -- but they're hopefully pointing the way.)

>> On the other hand, I think it might not be doing enough, insofar as
>> classpath-discovery may be a far superior alternative in many cases
>> for building middleware stacks than the fully-explicit approach
>> currently in place
>
> I agree; here I specifically kept it from doing too much because I
> wasn't sure how the classpath discovery would look and would prefer to
> experiment with in Leiningen where I can find the right answer by
> iterating more quickly.

Very sensible. Of course, everything we're talking about here can safely percolate and compete out in library-space, negatively impacting anyone AFAICT. nREPL "promotion" should very explicitly _not_ be a goal, at least until there's something that looks like a solution out in the world that can be productively canonicalized.

I hope the above is productive/helpful/comprehensible/etc. :-)

Thanks,

- Chas