REPL pretty-printing troubles

[Chas Emerick]

The way REPL output is being pretty-printed is making it difficult to work with some datasets. Immediate issues I'm seeing include:

• reader tags are always included in the output. This will make any pretty-printing less so, especially for nested structures of records (which result in a lot of #record.Name noise). This also exacerbates issue #2:
  
• very wide output (especially compared to clojure.pprint-formatted output), with lines up to ~165 characters long

I assume at least the latter is a bug? Certainly the soft-wrap option in the output area doesn't help, since that makes apparent indentation unhelpful in tracking structure as you're reading output. FWIW, I have the intellij code style right margin set at 100 (which cursive theoretically uses for formatting, viz https://cursive-ide.com/archive/2077.html), but that's definitely not being respected.

It seems that Cursive is using fipp to print REPL results (at least judging by other forum/ML posts). At least, the default REPL output and the result of (fipp.clojure/pprint …) appears to be equivalent. However, fipp isn't on my REPL's classpath AFAICT, but it seems to ignore the documented :width option when I invoke it explictly.

Does cursive provide any knobs to control REPL output printing? I'd love to recover clojure.pprint-powered output if that's at all possible, esp. since I have print-method implementations that it respects and also dedicated clojure.pprint/simple-dispatch methods I'd prefer to not refactor or otherwise touch.

Thanks,

- Chas

[Maarten Truyens]

Hi Chas, 

I asked a very similar question in the past — see Colin's reply here:

https://cursive-ide.com/archive/2077.html

[Chas Emerick]

Yeah, I cited that thread in my message. The difference in my case is that I'm strictly talking about the formatting of expression results, not arbitrary output/printing.

- Chas

[Colin Fleming]

Right, Maarten’s problem seems different since that was about the width of arbitrary output. There was also a request to be able to configure the REPL output formatting width independently of the general editor width, for which I’ve filed #1640.

Cursive does use fipp, but it just uses the layout engine and creates its own formatting events. So you don’t need it in your deps, Cursive just injects a vendored namespace on REPL start.

I’ll have a look at supporting pprint as well. It’s tricky for various reasons - the fipp support doesn’t require the form returned from the REPL to be read, I just lex the string output returned, which means that I can even pretty print most unreadable forms (i.e. containing objects). It’s also very efficient, which helps memory use and speed when very large forms are returned. As you note, it’s not as flexible, though. The fact that I’m lexing the returned string also allows me to syntax highlight the output - I’d have to check whether pprint would support that.

[Chas Emerick]

Yeah, I saw that too. To be clear though, I'm seeing eval results printed up to 170 columns wide...it's definitely not respecting my global editor width preference.

- Chas

[Colin Fleming]

Do you see that when using fipp directly? You said you’d compared the results, right?

[Chas Emerick]

Yes, the output is consistent between printing a value through fipp.clojure/pprint explicitly, and how cursive pprints the same value as a result of evaluation.

[Chas Emerick]

I tinkered with this a bit more this morning because the current REPL output is just completely useless for certain areas of my primary project, which happens to be record-heavy.

Here's a sample of the REPL output, which is identical to what fipp outputs if used directly (gisted because I figure the wide lines would be scrambled in email):

https://gist.github.com/cemerick/8959b068a7e41c13219a42f130f6441f

I looked at fipp's implementation some. Leaving aside the fact that there's no option to disable writing those record tags (a custom fipp printer could do something different), the printing width option only applies to the width of each "entry" in a data structure (element in sequences, entries in maps/records). The net result is that if you have records that contain other records as their first printed entry will end up printing very wide, regardless of the width of the contents of the first non-record entry.

Reading some about fipp's design goals (e.g. https://github.com/brandonbloom/fipp/issues/21#issuecomment-64693415), I don't know that it's a good choice for a default interactive output printer; I'd be surprised if it ever ended up supporting the sorts of hooks that make for such a thing (like print-method, or what puget seems to be building on top of fipp?). Those just aren't relevant (and potentially actively harmful) for fipp's core use cases of e.g. macroexpansion printing, viewing large datasets, and so on.

Of course, I don't grok all the constraints you have for REPL output features, but I desparately want pprint output at this point. Working with nested records as nodes in loom graphs (themselves records) creates output that is effectively noise at the moment. I'm coping with a `(clojure.pprint/pprint *1)` REPL command keybinding, but that obviously doesn't help with the noise.

Hope the above helps,

- Chas

[Colin Fleming]

Contrary to appearances, I have actually been thinking about this as well - sorry for the delay.

You’re correct that fipp’s implementation can’t guarantee to fit it’s output within the margin. In general, doing that requires having the whole data structure that you want to print in memory, and also requires a constraint-based approach with multiple passes over it. I read a really interesting article about this recently: http://journal.stuffwithstuff.com/2015/09/08/the-hardest-program-ive-ever-written/ - this is in the context of code formatting, but the principle is the same. There are some conflicting goals here - fipp is really really fast and works in a single pass over the document you’re formatting. This matters a lot to people printing tons of output, but it’s less flexible. Anecdotally, I’ve received numerous issues complaining about REPL output performance, but you’re the first to ever complain about flexibility. Which is strange in retrospect, but it’s true.

I realised something else the other day which might not be obvious - right now, Cursive pprints in Cursive itself, i.e. in the REPL client. What you’re talking about requires doing the output pprinting on the REPL server, since your customisations live there. I feel kind of silly suggesting this given the context, but wouldn’t what you want be better achieved with a nREPL middleware? Then you could turn of Cursive’s pprinting but you’d still get colourisation, and pprinting would work as you want from the server?

[Colin Fleming]

In fact, it seems like this is what you want: https://github.com/clojure-emacs/cider-nrepl/blob/master/src/cider/nrepl/middleware/pprint.clj, and you could just turn off Cursive’s pprinting. 

I’ll think about how best to support this. I like the pprinting on the client side, in general Cursive tries to be as minimally invasive on the user’s running process as possible. But you’re right that currently this is much less flexible than it might be. I’ll see what I can come up with.

[Chas Emerick]

I had never even contemplated the possibility that cursive was doing this client-side. It sounded like you had vendored fipp, were injecting it into every REPL's classpath, and had set up middleware or some other mechanism to intercept and format evaluation results.

CIDER's nREPL pprinting middleware is particularly complicated for what I care about, and depends upon a fair bit of cooperation from the client that I can't rig together in cursive. I got a bare-bones pprinting middleware in place instead, but there are a couple of issues:

1. It looks like cursive's tooling REPL connection(s) don't appreciate :value slots being pretty-printed. Completion suggestions disappeared and CPU went through the roof once I attempted to complete something in the REPL.
2. Multiline pprinted output isn't syntax-highlighted

I got around #1 by only pprinting when (.startsWith (:code msg) "(cursive.repl.runtime/")  :-P Absent pushing the REPL backend into actual middleware so it's out of the :op :eval path (which was fundamentally intended to be for interactive purposes), it'd be nice if user-invoked evaluations were marked vs. those related to tooling. CIDER and ccw likely have something, and maybe they follow a convention, if this is something you'd be willing to do…

Thanks!

- Chas