next generation terminal

[Mark]

The last time I said this has been a long time coming was... September of 2020. So yes, this really has been a long time coming!

Over the past year and a half we have been slowly chipping away at improvements to Urbit's terminal stack. Now the time for release is finally upon us! Attached is a document that goes over the changes and roadmap in detail. The work done so far is mostly internal, but there are a few visible changes.

For users:
Web terminal will support multiple sessions! Both dojo and chat-cli have been updated to take advantage of this. Notably, system output and printfs will only appear in the default sessions, leaving your other sessions less-cluttered.
Also you can now click the mouse... to move the cursor around in the prompt. Exciting!

For developers:
No superpowers just yet, but there is a breaking change to the sole protocol. The subscription path has changed from /sole/drum_~someship to /sole/~someship/sessionname. A +path-to-id helper is available in /lib/sole to aid in parsing this. The interface of /lib/shoe has been changed to match.
For further details and examples, see commit 3120681. Feel free to reach out if you have questions about updating your code.
https://github.com/urbit/urbit/commit/3120681

So, when does this release? Very soon! It's getting bundled alongside a lot of other work, and the upgrade will require a kelvin bump. We have a lot of goodies in the pipeline right now, so it's hard to tell exactly when. But: it's never been this close.

Happy to answer any and all questions about this!

—
~palfun-foslup
https://urbit.org

[Juanjo Presa]

Thanks for your in depth report. Im excited with coming CLI apps.

> --
> To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@urbit.org.
> # next-generation terminal, phases one and two
>
> In September of 2020 I posted command-line-evolution.txt to the urbit-dev
> mailing list[1], outlining the state of Urbit's terminal stack, its problems,
> and proposed solutions.
>
> Since then, a lot of sporadic work has happened... and a lot of time has
> passed! I had originally written up a document similar to this one for the
> release of the first phase of terminal work, back in June of 2021, but for
> various reasons the release just didn't pan out at that time. So instead,
> what follows is a slightly rewritten document for a slightly bigger release!
>
> If you aren't familiar with Urbit's terminal stack, the overview section of
> the earlier mailing list post might be worth checking out. I will not be
> repeating that here.
>
> The current view on terminal work divides it into four distinct phases:
>
> 1. Phase one lays the technical groundwork on the backend for further,
> user-facing improvements.
> 2. Phase two surfaces some of that to the user, by realizing the idea of
> fully separate terminal sessions.
> 3. Phase three expands the capabilities available to userspace developers,
> and delivers a richer, more fully-featured terminal experience.
> 4. Phase four is as of yet vaguely defined, but is aimed towards achieving
> the mythical "ui protocol".
>
> Below, we'll dive into what each of these phases entail in more detail.
>
>
> ## phase one: groundwork
>
> PR urbit/urbit#4463, "term: next-gen, phase one"[2] laid the groundwork for
> more visible improvements. Most importantly, it added a few primitives that
> enable new kinds of terminal behavior. However, the existing terminal stack
> made very strong assumptions about only writing to the bottom of the screen,
> and the prompt always being there. Clearly, to allow a more free-form
> experience, this had to be refactored.
>
> ### un-prompt
>
> The default (and currently only) terminal handler, drum (part of hood),
> implements a prompt and lets you cycle through different applications
> within it. This prompt had leaked into the implementations of both dill and
> the runtime, adding friction when trying to deviate from the prompt-centric
> experience.
>
> We have now removed "updating the prompt" from the dill interface. In fact,
> dill no longer keeps track of the prompt at all. The runtime unfortunately
> still tracks the prompt, sort of, for... reasons.
>
> The problem is, both dill and the runtime may want to write output to the
> screen without involving the terminal app. Think system output for example, or
> debug printfs. Both of those would be much better at home in a logging system
> of sorts, rather than intertwined with formal terminal output, but that is a
> whole separate project.
>
> What we ended up doing now, for the short term, is having dill and the runtime
> meddle exclusively in the *default* terminal session (more on sessions below),
> letting them assume prompt semantics for just that session, not the terminal
> experience as a whole.
>
> For dill, this means always drawing system output to the default session,
> above the bottom line of the screen, and then telling the terminal handler
> to restore the cursor position.
>
> For the runtime, this means inserting debug and log output between the last
> and second-to-last lines of the screen, but only if we are connected to the
> default session. For other sessions, we won't print it at all.
> For drawing the spinner, we must remember the contents of the bottom line of
> the screen, and ensure the spinner always gets drawn there, so that we then
> may redraw its original contents once the spinner disappears.
>
> This is not the ideal implementation, but it is enough to let us move on
> for the time being. Further improving upon this is definitely still on the
> list, but should now be largely independent of other terminal work.
>
> ### sessions
>
> We mentioned "sessions" in the previous section, and how they let us punt on
> some problems by restricting certain behaviors to the "default session". Now
> we must ask: what exactly does that mean?
>
> Dill has technically always supported multiple sessions living side-by-side.
> That is, it can keep track of separate connections, potentially to different
> terminal handlers, and route everything accordingly. This functionality,
> however, was never actually used.
>
> These new changes intend to revivify the sessions implementation. Sessions now
> have names, instead of being identified by duct, and inputs are sent to target
> sessions explicitly. This lets you connect to the same session from multiple
> clients/processes.
>
> The default session, then, is the dill session created during the boot process.
> It uses the identifier `%$` (empty string constant) and is always connected
> to drum. This lets dill and the runtime make the assumptions described above.
>
> Notably, this means that outside the default session, system output will no
> longer interleave itself with application output. This is nice for the current
> append-to-bottom class of terminal applications, since it lets you isolate
> their output. But it is absolutely *essential* for full-screen terminal
> applications like text editors.
>
> We should mention here that actually extending userspace and the various Urbit
> terminal clients to make use of sessions is work that happens in phase two.
>
> ### 2d cursor movement
>
> You cannot reasonably implement fancy terminal applications without being
> able to do free-form screen drawing. So the dill interface was updated to
> support that.
>
> Now, instead of content always replacing an entire line, it simply draws
> at the last selected cursor location, regardless of what is presently on-screen
> there. Additionally, the cursor can now be moved in two dimensions, instead of
> only to columns at the bottom of the screen.
>
> ### mouse support?
>
> Oh, there is one more thing. Mouse-click support! Sort of.
>
> The catch here is that we are (currently) only interested in click events, and
> do not want to hijack scrolling or other input events, as this could interfere
> with things like the tmux session Urbit might be running in. So the runtime
> asks the terminal emulator it is running in for precisely just mouse clicks.
> From light testing, it seems most terminal emulators do not support this
> reporting mode, and only support full reporting (including click release,
> scroll events, etc.). As such, mouse clicks may remain absent from your Urbit
> experience for now.
>
> The good news is that the web terminal has been reimplemented, and does indeed
> support mouse-clicks in the way we want.
>
> Drum, however, has not yet had its grand facelift. It only reacts to click
> events in the prompt (to move the cursor around), and does not pass clicks on
> to the currently selected application. In fact, the sole protocol has not been
> changed at all (besides some trivial changes to accomodate sessions, see phase
> two). It also does not support the new primitives that have been added.
> More on that in phase three.
>
>
> ## phase two: reifying sessions
>
> PR urbit/urbit#5663, "term: extended session support, other improvements"[3]
> carried on where the first phase of work had left off. In order to be able to
> use all the new primitives to their full potential, we need an environment
> that lets them flourish. The missing ingredient was mentioned above:
> userspace and client-side support for sessions.
>
> ### userspace
>
> Drum was implemented as a single-session terminal. Even if we were to open a
> new session in our client, drum would just mirror the contents between it and
> the default session.
>
> So drum needs to be updated to account for sessions. While this does not seem
> like too complex a change on the surface, there were some interesting
> challenges around the `|link`ing of applications. Do new drum sessions still
> automatically connect to dojo? How does `|link` know which session to link
> the app into?
>
> We ended up admitting that drum and dojo are coupled to a certain extent. Dojo
> will now take note of what session it's running in and, if a generator takes
> an optional named `drum-session` argument, auto-fills that with the session
> that ran the generator. This lets `|link` work on individual sessions.
>
> Applications themselves might also want to account for drum session identifiers
> if they are not already doing so. Chat-cli was already upgraded to account for
> this a long time ago.[4] Dojo has now been updated as well, and third party
> developers who have made sole (or shoe) applications may want to follow suit.
>
> Notably, the session identifiers in the sole protocol have changed slightly,
> and as such constitute a breaking change for existing sole applications.[5]
> Luckily, that's the only breaking change for userspace, and it should be
> fairly straightforward to resolve in most cases.
>
> ### clients
>
> With session support propagated into userspace, the last step was updating
> the terminal clients to let you create, delete, and switch between sessions.
> Webterm was the ideal candidate for a first stab at this, and it came out
> really well.
>
> Sessions support in the runtime is a slightly more involved endeavor, and as
> such we have punted on it for the time being. There are some open questions
> here around what session management looks like in a pure tui context, and how
> connecting to a running urbit's terminal would work.
>
> While we do want to answer those questions and expose non-default sessions to
> the "real" terminal sooner rather than later, the availability of sessions in
> the web terminal is enough to unblock us for the next phases of work.
>
>
> ## phase three: full native terminal
>
> Now that sessions are in, we can start going wild. The first order of business
> is to make a second terminal handler app, an alternative to (fork of?) drum.
> It should track and display buffers and scrollback on a per-app basis, giving
> each app its own visual real-estate. Its protocol, if we decide it's
> appropriate to add one, should support the newly added primitives, or perhaps
> provide a slightly higher-level layer on top of them.
>
> In the above, "scrollback" means the full thing. Built-in shortcuts to scroll
> back up the buffer to see previous output and copy parts of it out. Think
> tmux, but native to Urbit. Throw in support for mouse -drag and -scroll events,
> and we suddenly have a complete, fully-featured in-arvo terminal experience.
>
> It will probably be necessary to build a small application or two on top of
> this, to see if it is up to snuff. Examples that immediately come to mind are
> a simple text editor, and a scry namespace explorer. A souped-up chat-cli
> might be nice too, but really we want the full suite of landscape applications
> to have terminal equivalents. And of course, many third party applications
> as well.
>
>
> ## phase four: a future yet unseen
>
> At this point we should have enough affordances in the terminal to start
> experimenting with apps and libraries in the direction of "ui protocol".
> Generic ways of describing interfaces and binding behavior to them. Graphical
> clients that deliver richer or more accessible experiences outside of the
> terminal, while being compatible with originally text-only applications.
>
> Many unanswered questions here. Many unasked ones, too. If any readers are
> aware of prior art or literature that we must know about and consider when
> doing this work, please reach out with it, we would love to see what is
> already out there!
>
>
> ## closing thoughts
>
> The current release has been a long time coming. But with this foundational
> work complete, we can move on to more exciting parts of terminal improvement.
> Before we know it, native interface development will be more than just a
> distant dream!
>
>
> [1] https://groups.google.com/a/urbit.org/g/dev/c/ubp3UNxkQBc/m/zsB_5rxOBAAJ
> [2] https://github.com/urbit/urbit/pull/4463
> [3] https://github.com/urbit/urbit/pull/5663
> [4] https://github.com/urbit/urbit/pull/4000
> [5] https://github.com/urbit/urbit/commit/3120681

--
Juanjo Presa
https://nadanix.com