A couple issues with scrollback

[dho...@google.com]

There are a couple of issues I’m having with scrollback which I haven’t seen reported before.

1. (Debatable between WAI, a bug, and an enhancement) The scrollback is not updated on CSI 2J. This sequence is usually sent by readline when you type ^L, and people often use ^L as a way to clear the screen while keeping scrollback. Hterm shows the same behavior as xterm and the Linux virtual console (on the virtual console ^L sends a parameterless CSI J, which has the same scrollback behavior), but it seems less useful than the alternatives. Konsole (and derivatives like QTerm) and VTE-based emulators like Gnome Terminal push the entire current screen into the scrollback buffer, which leads to a lot of blank lines, but still leaves everything findable. The ideal behavior seems to be what screen and tmux do: push only up to the last line with content. I have a workaround for this when using zsh, which uses its own ZLE instead of the standard Readline, but Readline doesn’t seem to have similar customization options.
2. (Bug) The alternate screen sharing a scrollback buffer with the main screen causes the scrollback to quickly become hard to use (if they don’t trigger https://crbug.com/740607 instead). Most emulators don’t seem to actually store alternate screen scrollback at all, but if it is saved it shouldn’t be interleaved in the way it is. For example, consider the following session:
   
   
   $ echo one
   one
   $ echo two
   two
   $ seq 60 # Here I assume that $LINES < 60
   [Over a page of output skipped for readability]
   $ echo three
   $ man less
   # Scroll around the man page for `less`
   $ echo four
   four
   $ echo five
   five
   
   
   Here, if you try to scroll up and read the session, in the best case* you’ll see:
   
   
   $ echo one
   one
   $ echo two
   two
   $ seq 60
   # several numbers
   # $((LINES - 1)) blank lines
   # Excerpts from the manpage of `less`, interspersed with `...skipping`, not even including the section you were looking at when you quit `less`
   # The numbers from $((65 - LINES)) to 60
   $ echo four
   four
   $ echo five
   five
   
   
   There are several issues with this output:
   
1. Output from the alternate buffer is interrupting output from another command. This can be confusing in cases more complicated than `seq`, such as if I had `cat`ted a long file with similar contents to what was shown in `less`.
2. Output from the alternate buffer is not adjacent to the command that created it, but up to a screen above that command.
3. With the way hterm stores the scrollback data, the last screenful isn’t saved at all, and if you quit `less` because you found what you wanted this is the most important part.
   
4. Output from the alternate buffer often isn’t useful anyway. A pager like `less` might be the best-case scenario for usefulness of the alternate buffer, and even then it gets confusing if you do anything more complicated than scrolling down. This is probably why most terminal emulators don’t store any alternate screen scrollback; GNU Screen does, but it seems to discard that as soon as you switch back to the main screen.
5. Apparently the first use of the alternate buffer instead erases whatever’s on the screen instead of adding it to scrollback and the output is even worse: you completely lose a screenful of regular output unexpectedly. That might or might not be related to the first scrollback issue.
3. (Feature request) CSI 3 J should erase the entire scrollback buffer (as it does in Xterm). This is specified in the nonstandard E3 terminfo capability and sent by the `clear` command. I don’t actually use this, but a lot of people do so it seemed appropriate to report it while I was reporting other scrollback issues. I think it should preferably work, or at least be documented as Ignored (TBD) on the control sequences page, although if you do implement this I would love an option to turn it off (to go with the options to turn off blink, bell, DEC 12, and other potentially-annoying features).

Issues 1 and 2 make scrollback a lot more confusing to use, and although issue 3 probably doesn’t make scrollback more confusing for the people who are affected by it, it does make scrollback less useful for them.

[Mike Frysinger]

(1) since scrollback isn't really covered by any spec, and every terminal emulator has done their own thing, i'd say not a bug since we have the freedom to be "implementation defined".  behaving like tmux/screen is a reasonable, so FR here if you want to star it: https://crbug.com/881088

(2) similarly, we're free to define whatever behavior we want here, so technically none of these are bugs ;).  i understand that some people might not like the current behavior, but i've also heard from people where having no scrollback at all is annoying.  which is precisely the scenario you get when using `screen` as it always runs under the alternative screen.  even with some of the examples you highlighted wrt loading/exiting editors, having the scrollback can be useful in case they crash/misbehave, but you had previous output you wanted to refer to (which has happened to me).  screen doesn't seem to behave as you describe ... if i load up `man man` and scroll a few pages and exit, then screen's scrollback includes everything i had looked at.  i vaguely recall some comments in our scrollback logic talking about edge cases like this and how we purposefully push unlike some other terminal emulators.

to reiterate, i don't think any of these are "wrong".  they're entirely personal preference as to how the scrollback is managed.  i don't want hterm exploding a matrix of configuration options here though that nobody is going to change from the default, although i would entertain one or two high level knobs.  maybe something like "scrollback-save-alternate-screen" where you control whether anything in the alternate screen is retained.

i don't think we want to try to manage two scrollbacks (one per screen).  otherwise things pushed in one are, practically speaking, inaccessible.  especially when you consider how many people even know terminal emulators have a concept (from xterm) of an "alternate screen".

i don't think we've documented expected scrollback behavior anywhere.  so basically "whatever the code is doing is what is expected" which means if things change or "regress", no one notices.  i'll try to hash out a few principles we can strive for.

(3) we actually support CSI 3 J today, but treat it like CSI 2 J:

  } else if (arg == 3) {

    // The xterm docs say this means "Erase saved lines", but we'll just clear

    // the display since killing the scrollback seems rude.

    this.terminal.clear();

making a knob to control this seems excessive, but does seem like one of those esoteric cases most people don't notice/care, and the few who do strongly care :).

-mike

--
You received this message because you are subscribed to the Google Groups "chromium-hterm" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-hterm/a84f9987-32eb-4ce7-b8d6-bd18c92593fe%40chromium.org.

[dho...@google.com]

I intended to mention that GNU Screen by default does not support the alternate screen (in what it emulates; as you mentioned it runs itself in the alternate screen); you need to explicitly say `altscreen on` at the Screen prompt (^a :) or in .screenrc, or it just ignores the alternate screen control codes. That’s one way to keep scrollback (in the correct location and without overwriting other scrollback), at the cost of not showing your previous shell commands when you quit a full-screen program.

Most people might not know that terminal emulators have alternate screens, but they do know “When I scroll up I see sections of the man page I wasn’t even looking at in between my other output”. I agree that losing data from your file and not being able to scroll up to see it is also bad. It seems that the issues is that the reasonable expectations contradict. If you want alternate screens, then the order things scroll off the screen is not the order you see them in. Possible behaviors seem to be:

1. No alternate screens: This gives the nicest scrollback (if you want your alternate-screen apps in scrollback at all), but makes it harder to see what you’ve recently run. This what the Linux console does always and what GNU Screen does by default.
   
2. Alternate screens without scrollback: This is annoying if you want to scroll up to see what was on the alternate screen, but is what everything I’ve tested which supported alternate screens used (except Screen).
   
3. Alternate screens with separate scrollback: Separate buffers for main and alternate screen, with the alternate screen scrollback buffer potentially erased on return to the main screen. This is what Screen does when alternate screens are enabled, and except in terms of implementation effort seems strictly better than 2.
   
4. Alternate screens with interleaved shared scrollback*: Insert the alternate screen scrollback buffer right above the main screen. This is what hterm does, at the expense of out-of-order scrollback. With this approach I think the scrollback should also include the last screenful of data, and shouldn’t erase data from the scrollback when enabling alternate screens for the first time.
   
   * This isn’t quite correct because the entire alternate screen buffer is put in one place in the scrollback, but it’s better than the only other name I could think of, “Alternate screens with misplaced scrollback”.
   
5. Alternate screen with in-place shared scrollback: Save the alternate screen scrollback in a temporary buffer, along with what line you’re on when leaving the alternate screen, and push it into the main scrollback buffer when that line reaches the top of the visible screen. This gives almost the best scrollback buffer (equivalent to “no alternate screens”), but also gives the benefit of alternate screens; however, alternate screen scrollback isn’t available immediately. I’m not aware of anything which does this, probably because it would be next to impossible to implement, but otherwise seems to be the best option if you want both alternate screens and their scrollback.
   

Depending on how difficult 3 is from a technical point of view, I think there should be a knob that either switches between 1, 2, and 4 or between 1, 3, and 4. (If I’m wrong about how hard 5 would be then I’d want that to be an option in addition to/instead of 4).

For CSI 3 J, I still think the behavior difference should be documented. I understand why people want to use it (it’s easier to scroll up to the top and read down, then figure out exactly how much you need to scroll up, so running `clear` before something that generates a lot of output, can be easier), but I like not being able to accidentally clear my scroll buffer, which is why I proposed a knob.

(This is a semantic argument, which is why I moved it to the end, but I’d say simply not documenting something doesn’t mean it’s not a bug. As an extreme example, the documentation also doesn’t say how bracketed paste mode works and it isn’t in any spec except the xterm documentation (which incidentally also specifies no scrollback for alternate screen buffer), but if you got \e[200~ and \e[201~ switched it would completely break any shell using the feature and I wouldn’t hesitate to call that a bug. In fact, if you did that and explicitly said in the documentation “This puts \e[201~ before pasted text and \e[200~ after pasted text, which is the opposite of how most terminal emulators work”, I would still call it a bug. The implementation not matching the documentation is a type of bug, but it’s not the only type of bug.)

[Mike Frysinger]

once i have a doc that outlines current/expected scrollback behavior, i'll circle back on those topics

wrt CSI 3 J, it wasn't an intentional omission in the control sequence document. as you can imagine, putting that together from scratch years after the project started was non-trivial, so it's going to have bugs.

wrt bracketed paste documentation, xterm invented it, so it's not surprising that it's the only place currently to really explain what it means. there are a ton of things we don't define when we aren't the source and instead expect people to leverage the references linked earlier. that's why it starts off by explicitly stating:

> It's not meant to be a comprehensive guide to a terminal. For that, please see the References section.

xterm is one of those references.  the extended portions of the doc are there when i find other sources to be underspecified, unclear, or hterm has its own extensions.  like mouse reporting and color processing (e.g. the ISO 8613-6 craziness).

wrt terminal emulator differences, that's frankly just how this world works.  while there are some core standards (like the ECMA specs) that one can arguably say "everyone should be following those baselines", there's a huge space left over.  ECMA-48 defines CSI [012] J and that's it.  xterm decided to extend things and recognize CSI 3 J to do <something>.  but that doesn't mean xterm "owns" the CSI 3 J sequence and thus every other terminal out there now has to match.

as our docs say, hterm aims to be compliant with ECMA/ISO standards, although it's not a hard rule as you quickly run into conflicts: as a modern terminal, we only care about UTF-8, but those dusty standards still have NRCS (codesets) that we have no interest in supporting.  we look at xterm/VTxxx for guidance, but they aren't "standards".  actively deviating from any of these isn't a bug, it's a project choice.  if you want xterm behavior, then run xterm ;).

-mike

--

You received this message because you are subscribed to the Google Groups "chromium-hterm" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-hterm/a4bde961-6b8f-41e2-84d5-b3d8b6140e96%40chromium.org.

[Daniel Houck]

I understand CSI 3J probably wasn’t an intended omission from the docs, which is why I brought it up.

You seem to be saying that bracketed paste is specified by xterm so it doesn’t need to be specified by hterm, but also that hterm can behave differently from xterm with things like CSI 3J and alt-screen scrollback. I agree with both, although I think it isn’t quite clear when you’re using xterm as a reference and when you’re deliberately deviating from it. Is it that the docs are incomplete but ideally you’d like to document all deliberate differences from what you have in the references section?

WRT terminal emulator differences, I only was bringing those up for comparison. Clearly you shouldn’t try to match everything else out there, especially because they don’t match each other so it would be impossible, but I find it useful to have a general survey of other approaches.

[Mike Frysinger]

if you've found deviations in behavior between hterm and xterm for the same sequence, and xterm is the origin of the feature/standard, and the hterm control sequences doc doesn't cover it, then it's a bug.  whether it's a bug in the docs or the code is up for debate :).

certainly feel free to highlight any such discrepancies in this group or file bugs.  we appreciate anyone taking the time to read/review such dense esoteric documentation :).

wrt CIS 3J, i've filed https://crbug.com/881507 to add a config knob and change hterm so it behaves like xterm by default.

-mike