Shell integration documentation

[Aaron Meurer]

Is there any documentation on the shell integration, both on what the escape sequences are, and how the bash integration works (e.g., I don't get why PROMPT_COMMAND is needed instead of just putting something in the PS1, or why DEBUG is used)? I'm interested in writing integrations for other shells, if that's possible.

Aaron Meurer

[George Nachman]

Looks like the escape sequence docs went down with the googlecode site, but I can document it here easily enough.

There are four escape sequences called the "FinalTerm" sequences (since FinalTerm invented them):

OSC 1 3 3 ; A ST

  Sent just before start of shell prompt

OSC 1 3 3 ; B ST

  Sent just after end of shell prompt, before command user enters

OSC 1 3 3 ; C ST

 Sent just before start of command output

OSC 1 3 3 ; D ; Ps ST

 Sent just after end of command prompt. Ps is command status, 0-255.

These enable the mark, command history, and select output of last command.

Additionally, there are a few other sequences that iTerm2 defined. These are typically sent within the prompt. These enable the rest of the shell integration feature (recent directories, per-host command history, scp upload and download, user-defined badge variables, whatever else I'm forgetting).

OSC 1 3 3 7 ; S e t U s e r V a r = Ps1 = Ps2 ST

  User-defined vars. Ps1 is the key, Ps2 is the value. Not sent by default. Typically the user defines a shell function that invokes function like "iterm2_set_user_var key value", which causes this code to be sent.

OSC 1 3 3 7 ; S h e l l I n t e g r a t i o n V e r s i o n = Pn ST

  Reports the current version of the shell integration script. Pn is the version. Currently only "1" is defined. iTerm2 has a baked-in notion of the "current" version and if it sees a lower number the user will be prompted to upgrade.

O S C 1 3 3 7 ; R e m o t e H o s t = Ps1 @ Ps2 ST

  Reports the user name and hostname Ps1 is username, Ps2 is fully-qualified hostname. New shell integration scripts should allow the user to provide the hostname in case the DNS server is slow.

O S C 1 3 3 7 ; C u r r e n t D i r = Ps1 ST

  Reports the current directory.

The zsh script is the easiest to understand. bash is a mess because it doesn't have support for a proper preexec function, fish is weird, and nobody understands tcsh.  Everything lives in github.com/gnachman/iterm2-website in source/misc.

TBH we probably don't need to use PROMPT_COMMAND. The bash code is derived from this: https://github.com/rcaloras/bash-preexec (actually, an older version). I am certain it can be simplified. DEBUG is used because it lets us send the FinalTerm C code at the right time, which is otherwise impossible.

On Mon, Aug 3, 2015 at 6:13 PM, Aaron Meurer <asme...@gmail.com> wrote:

Is there any documentation on the shell integration, both on what the escape sequences are, and how the bash integration works (e.g., I don't get why PROMPT_COMMAND is needed instead of just putting something in the PS1, or why DEBUG is used)? I'm interested in writing integrations for other shells, if that's possible.

Aaron Meurer

--
You received this message because you are subscribed to the Google Groups "iterm2-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iterm2-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aaron Meurer]

Thanks! 

My goal is to write the integration in Python so that it can be integrated with IPython and xonsh. 

On Tue, Aug 4, 2015 at 11:57 AM, George Nachman <gnac...@llamas.org> wrote:

Looks like the escape sequence docs went down with the googlecode site, but I can document it here easily enough.

There are four escape sequences called the "FinalTerm" sequences (since FinalTerm invented them):

OSC 1 3 3 ; A ST

  Sent just before start of shell prompt

OSC 1 3 3 ; B ST

  Sent just after end of shell prompt, before command user enters

OSC 1 3 3 ; C ST

 Sent just before start of command output

OSC 1 3 3 ; D ; Ps ST

 Sent just after end of command prompt. Ps is command status, 0-255.

You mean command output?

 

These enable the mark, command history, and select output of last command.

Additionally, there are a few other sequences that iTerm2 defined. These are typically sent within the prompt.

Is the order important here? I notice that the zsh integration passes these after D.

 

These enable the rest of the shell integration feature (recent directories, per-host command history, scp upload and download, user-defined badge variables, whatever else I'm forgetting).

OSC 1 3 3 7 ; S e t U s e r V a r = Ps1 = Ps2 ST

  User-defined vars. Ps1 is the key, Ps2 is the value. Not sent by default. Typically the user defines a shell function that invokes function like "iterm2_set_user_var key value", which causes this code to be sent.

So, for instance, if I want to include the IPython prompt number, I would use this with something like ipython_prompt_number = N, right?

OSC 1 3 3 7 ; S h e l l I n t e g r a t i o n V e r s i o n = Pn ST

  Reports the current version of the shell integration script. Pn is the version. Currently only "1" is defined. iTerm2 has a baked-in notion of the "current" version and if it sees a lower number the user will be prompted to upgrade.

So what should a custom shell integration use? Is it still a good idea to include this to be warned of compatibility breaks (although I would hope those don't happen). 

 

O S C 1 3 3 7 ; R e m o t e H o s t = Ps1 @ Ps2 ST

  Reports the user name and hostname Ps1 is username, Ps2 is fully-qualified hostname. New shell integration scripts should allow the user to provide the hostname in case the DNS server is slow.

O S C 1 3 3 7 ; C u r r e n t D i r = Ps1 ST

  Reports the current directory.

The zsh script is the easiest to understand. bash is a mess because it doesn't have support for a proper preexec function, fish is weird, and nobody understands tcsh.  Everything lives in github.com/gnachman/iterm2-website in source/misc.

TBH we probably don't need to use PROMPT_COMMAND. The bash code is derived from this: https://github.com/rcaloras/bash-preexec (actually, an older version). I am certain it can be simplified. DEBUG is used because it lets us send the FinalTerm C code at the right time, which is otherwise impossible.

I see. Hopefully the shells/REPLs I want to integrate this with make this easier.

Aaron Meurer

[George Nachman]

Cool! Someone has done this for Julia as well. Seems to work well.

OSC 1 3 3 ; D ; Ps ST

 Sent just after end of command prompt. Ps is command status, 0-255.

You mean command output?

Yes, command output.

 

 

These enable the mark, command history, and select output of last command.

Additionally, there are a few other sequences that iTerm2 defined. These are typically sent within the prompt.

Is the order important here? I notice that the zsh integration passes these after D.

Can go before or after D as long as there's no newline between them. I suppose after D would be preferred in case something changes in the future but right now it doesn't matter.

 

 

These enable the rest of the shell integration feature (recent directories, per-host command history, scp upload and download, user-defined badge variables, whatever else I'm forgetting).

OSC 1 3 3 7 ; S e t U s e r V a r = Ps1 = Ps2 ST

  User-defined vars. Ps1 is the key, Ps2 is the value. Not sent by default. Typically the user defines a shell function that invokes function like "iterm2_set_user_var key value", which causes this code to be sent.

So, for instance, if I want to include the IPython prompt number, I would use this with something like ipython_prompt_number = N, right?

Yep. ATM it's only used by badges but that will expand in the future.

SetUserVar=ipython_prompt_number=N

 

OSC 1 3 3 7 ; S h e l l I n t e g r a t i o n V e r s i o n = Pn ST

  Reports the current version of the shell integration script. Pn is the version. Currently only "1" is defined. iTerm2 has a baked-in notion of the "current" version and if it sees a lower number the user will be prompted to upgrade.

So what should a custom shell integration use? Is it still a good idea to include this to be warned of compatibility breaks (although I would hope those don't happen). 

I'd leave this out in your script, since the upgrade button won't do what you want.

 

 

O S C 1 3 3 7 ; R e m o t e H o s t = Ps1 @ Ps2 ST

  Reports the user name and hostname Ps1 is username, Ps2 is fully-qualified hostname. New shell integration scripts should allow the user to provide the hostname in case the DNS server is slow.

O S C 1 3 3 7 ; C u r r e n t D i r = Ps1 ST

  Reports the current directory.

The zsh script is the easiest to understand. bash is a mess because it doesn't have support for a proper preexec function, fish is weird, and nobody understands tcsh.  Everything lives in github.com/gnachman/iterm2-website in source/misc.

TBH we probably don't need to use PROMPT_COMMAND. The bash code is derived from this: https://github.com/rcaloras/bash-preexec (actually, an older version). I am certain it can be simplified. DEBUG is used because it lets us send the FinalTerm C code at the right time, which is otherwise impossible.

I see. Hopefully the shells/REPLs I want to integrate this with make this easier.

This is the risky part.

[Aaron Meurer]

Cool, been playing around with this. 

It seems like the "select output of last command" is ignoring the D sequence, and selecting between C and A. Is this a bug? I'm not fully clear on why D is needed instead of just A, other than that you don't have a real command status for the first prompt, but it seems that if you're going to have separate codes, you might as well use them as documented.

Second, it seems iTerm2 borks if there is no command (text between B and C), or the command is just a newline. I get that you want to support the user typing enter at the prompt without typing a command, but isn't this already handled by the shell repeating the A and B codes? 

It also doesn't handle multiline commands, which is probably fine (it wouldn't do the right thing with continuation prompts like PS2 anyway).

Aaron Meurer

[George Nachman]

I'd classify it as a harmless bug. D and A always go together, so I don't bother to track D's location. There isn't really a need for D separate from A as far as I can tell, but then I didn't invent the protocol. You could argue that the first prompt is different so separate codes are justified, but it's not a very important distinction.

What goes wrong for you when there's no command? It works ok if I just hit enter at the prompt. That doesn't get entered into command history, of course. It should be forgiving if you forget C or D, but there might be bugs there.

Multiline commands aren't supported because every shell prefixes the continued lines differently (?, >, or indentation, or something fancy like PS2). I suppose I could customize PS2, at least for bash and zsh, adding a new escape sequence to indicate a continuation.

[Aaron Meurer]

On Tue, Aug 4, 2015 at 11:06 PM, George Nachman <gnac...@llamas.org> wrote:

I'd classify it as a harmless bug. D and A always go together, so I don't bother to track D's location. There isn't really a need for D separate from A as far as I can tell, but then I didn't invent the protocol. You could argue that the first prompt is different so separate codes are justified, but it's not a very important distinction.

OK, I'll just be sure to document that in my library.

I guess I could see this being a problem if someone has a wacky multiline prompt and they want the arrow to appear on something other than the first line. 

What goes wrong for you when there's no command? It works ok if I just hit enter at the prompt. That doesn't get entered into command history, of course. It should be forgiving if you forget C or D, but there might be bugs there.

This is coming from testing printing the escape sequences directly. If I do

A prompt B \n C output D

then the arrow shows up, and I can select it with Cmd-Shift-Up, but I can't right click on it. It's basically the same as

A prompt B \n A prompt B

Maybe it's intended behavior? 

Anyway, these things probably won't happen under normal prompt customization in a REPL, but I was trying to figure out the behavior so I could document it. 

Multiline commands aren't supported because every shell prefixes the continued lines differently (?, >, or indentation, or something fancy like PS2). I suppose I could customize PS2, at least for bash and zsh, adding a new escape sequence to indicate a continuation.

It's not a feature I care strongly about. I don't really use the command capture from the shell integration. I mainly use it for navigation, timing, exit status reporting, and most importantly, alert on next mark. I rely on the shell itself to keep track of its own history, and navigating readline is much faster than whatever GUI iTerm2 could implement. 

[George Nachman]

On Tue, Aug 4, 2015 at 11:18 PM, Aaron Meurer <asme...@gmail.com> wrote:

On Tue, Aug 4, 2015 at 11:06 PM, George Nachman <gnac...@llamas.org> wrote:

I'd classify it as a harmless bug. D and A always go together, so I don't bother to track D's location. There isn't really a need for D separate from A as far as I can tell, but then I didn't invent the protocol. You could argue that the first prompt is different so separate codes are justified, but it's not a very important distinction.

OK, I'll just be sure to document that in my library.

I guess I could see this being a problem if someone has a wacky multiline prompt and they want the arrow to appear on something other than the first line. 

You can't please everyone, especially people with multiline prompts, so the arrow always goes on the first line.

 

What goes wrong for you when there's no command? It works ok if I just hit enter at the prompt. That doesn't get entered into command history, of course. It should be forgiving if you forget C or D, but there might be bugs there.

This is coming from testing printing the escape sequences directly. If I do

A prompt B \n C output D

then the arrow shows up, and I can select it with Cmd-Shift-Up, but I can't right click on it. It's basically the same as

A prompt B \n A prompt B

Maybe it's intended behavior? 

The command is supposed to start on the same line as the B code. If you have a multiline prompt, do

A prompt \n B command C output D

This is because multiline commands can't be properly supported, so the first newline after B is where we stop looking at commands.

 

I wonder if it would be a good idea to include some kind of application identifier in the hostname so your REPL's commands don't get mixed in with your shell commands in command history. Something that would play nice with automatic profile switching. Maybe username@hostname(application) would make sense.

[Aaron Meurer]

That sounds like a good idea, but I wouldn't mix it with the hostname. That requires every REPL to know what the username and hostname are. 

Aaron Meurer

[George Nachman]

Agreed. A proper solution would be a sequence that says "this repl is starting" and "this repl is ending" (to handle nested REPLs, just in case), and if we see the RemoteHost sequence then we know we're back at a shell prompt.

[Aaron Meurer]

That's right. Nested REPLs are common because the shell itself is a REPL. So if I add integration to IPython, for instance, every time I start ipython in the shell, that's a nested REPL, meaning most of the shell integration metadata for the command "ipython" in the shell is going to be wrong.

I'm not super concerned with it personally because I don't tend to care about the metadata for commands that start interactive sessions, but it's something to think about.  

Aaron Meurer