uv_spawn a terminal program?

[Thiago Arruda]

Hi

Is it currently possible to use uv_spawn to open a terminal program?

On UNIX it would probably require connecting the child process stdin/stdout/stderr to a pseudo-terminal(I have no idea how that works on windows).

If not supported natively by libuv(I think it's not since there's no UV_CREATE_PTY flag for uv_spawn), is there any workaround I can use?

Thanks in advance

[Ben Noordhuis]

I'm afraid not. You can make the child process inherit the parent
process's TTY but I suspect that is not what you want.

There is an old feature request[0] for PTY support but (to the best of
my knowledge) no one has worked on it so far.

[0] https://github.com/joyent/libuv/issues/573

[Thiago Arruda]

The ability to automate interactive programs using libuv would be a killer feature, libuv could be used as a replacement for libexpect and node.js a replacement for expect :)

I imagine that beyond the system calls used to open the pty, there's not much difference between ptys and pipes from the programmer POV(at least on UNIX). If I managed to implement a UV_CREATE_TERMINAL spawn option by the end of next week, is there any chance it would make into 0.12?  I'm particularly interested in this, since it's a feature Neovim could benefit from.

The patch would include:

- The UV_CREATE_TERMINAL option, which would use the `pty()` instead of `pipe()` on UNIX

- A windows implementation(I would need to study winpty source code and adapt the solution to libuv)

- A good set of tests

[Saúl Ibarra Corretgé]

Hi Thiago,

On 07/25/2014 11:59 AM, Thiago Arruda wrote:
> The ability to automate interactive programs using libuv would be a
> killer feature, libuv could be used as a replacement for libexpect and
> node.js a replacement for expect :)
>
> I imagine that beyond the system calls used to open the pty, there's not
> much difference between ptys and pipes from the programmer POV(at least
> on UNIX). If I managed to implement a UV_CREATE_TERMINAL spawn option by
> the end of next week, is there any chance it would make into 0.12? I'm
> particularly interested in this, since it's a feature Neovim could
> benefit from.
>
> The patch would include:
>
> - The UV_CREATE_TERMINAL option, which would use the `pty()` instead of
> `pipe()` on UNIX
> - A windows implementation(I would need to study winpty source code and
> adapt the solution to libuv)
> - A good set of tests
>

This would indeed be a very welcome addition. Unfortunately I'm not sure
about adding it to 0.12, not because you can't make it happen before we
release it (you probably will) but the line needs to be drawn somewhere,
otherwise there will always be "one more patch" and we'll never ship it.

I'm open to being convinced otherwise, though. Fedor?

--
Saúl Ibarra Corretgé
bettercallsaghul.com

[Thiago Arruda]

I completely understand your point of view Saghul, my reason for proposing this as last-minute feature of 0.12 is because I believe that at least for the supported unixes, the necessary changes will be very small.

For example, here's a relevant snippet of code from the st terminal emulator

void

ttynew(void) {

int m, s;

struct winsize w = {term.row, term.col, 0, 0};

/* seems to work fine on linux, openbsd and freebsd */

if(openpty(&m, &s, NULL, NULL, &w) < 0)

die("openpty failed: %s\n", SERRNO);

switch(pid = fork()) {

As you can see, this is very similar to the pipe/fork combination already used by UNIX implementation of uv_spawn. The only difference is the `openpty` call, which also returns a pair of file descriptors to be wrapped into libuv's uv_pipe_t/uv_tty_t abstractions.

Other than that, I think two problems remain:

- Create compatibility layer for solaris, which may not have the the `openpty` system call(which is an abstraction to the OS-specific terminal API)

- Figure out how that fits into windows, which may use completely different method for communicating with terminal programs

If you guys think there's a chance of this happening before 0.12, it will be enough motivation for me to give it a shot, even if it's not ultimately merged.

[Saúl Ibarra Corretgé]

You are more than welcome to give it a try and see how deep the
rabbit-hole goes :-)

Now, assuming we merge this after 0.12, wouldn't the following work with
current master? (at least on Unixes):

* call openpty yourself
* create a pipe or tty handle with the given fd (uv_pipe_init +
uv_pipe_open / uv_tty_init)
* use UV_INHERIT_STREAM when calling uv_spawn
* profit?

[Thiago Arruda]

Yes it seems that could work, though a cross-platform API would be way nicer :)

I will start looking into this tomorrow, and will open a PR until next sunday if the implementation is sufficiently simple to be merged before 0.12.

Thanks for the opportunity.

[Thiago Arruda]

I'm giving up on this, mainly because Windows has nothing equivalent to pseudo terminals and it seems to be impossible to achieve something similar with conventional API call. It's not surprising that there aren't many custom terminal emulators for Windows out there.

With that said, I found a way achieve some level of terminal emulation on Windows after reading parts of Console2 source code: http://sourceforge.net/projects/console/

The idea is fairly simple: Besides the executable, Console2 also builds a dll that is injected into the child process running in a new hidden console created with CREATE_NEW_CONSOLE(actually, what is injected is a call to LoadLibrary passing the dll path). This dll creates a thread to watch for changes in 'CONOUT$' (which is the console screen buffer) and send updates with the parent Console2 process through shared memory. Likewise, it will watch shared memory for input coming from Console2, and forward it to 'CONIN$' which is the console input buffer.

To avoid requiring an extra dll to be shipped, one could use MemoryModule(https://github.com/fancycode/MemoryModule) to inject an embedded dll, but that would be too hacky for a library like libuv

[Saúl Ibarra Corretgé]

What a mess :-S I suspected it was not going to be a walk in the park.
Thanks for taking the time to share your findings though!

[Geert Jansen]

Hi,

Windows has very little terminal emulation support but the basics of starting a console program and redirecting it standard input / output are there and possible without any DLL injection hacks. I've done it in a past project of mine called winpexpect See the link here:

https://bitbucket.org/geertj/winpexpect

(there's also a repo on Github with the same name but that's an abandoned WIP - don't use it)

The builtin support won't allow you to build a decent terminal emulator but it does allow you to automate most command-line programs including cmd.exe and powershell. Out of curiosity what do you require in addition to this?

I would love to see PTY support in uv_process, but I think it will have to be a Unix specific flag.

Regards,
Geert

[Thiago Arruda]

On Tuesday, August 5, 2014 6:51:06 AM UTC-3, Geert Jansen wrote:

Hi,

Windows has very little terminal emulation support but the basics of starting a console program and redirecting it standard input / output are there and possible without any DLL injection hacks. I've done it in a past project of mine called winpexpect See the link here:

https://bitbucket.org/geertj/winpexpect

(there's also a repo on Github with the same name but that's an abandoned WIP - don't use it)

The builtin support won't allow you to build a decent terminal emulator but it does allow you to automate most command-line programs including cmd.exe and powershell. Out of curiosity what do you require in addition to this?

I would love to see PTY support in uv_process, but I think it will have to be a Unix specific flag.

Regards,
Geert

The problem is that some programs behave differently when stdio is connected to

a terminal(eg: python interpreter). As I said, it would be possible to implement

a full terminal emulation library for windows, but it would be far from trivial.

I can see two possible ways to attack the problem:

- Inject code into the child process to poll for $CONOUT/$CONIN data in a

  separate thread and communicate with the emulator using a pipe. This is what

  Console2 does.

- Take winpty approach which I haven't investigated, but I guess it involves

  starting a 'pty agent'(a proxy daemon that somehow manages to get the data

  from the console program)

I'm not sure if libuv devs would accept adding a flag that only works on UNIX

and is ignored on windows, but here's the code I have so far if you want to

continue from where I stopped: https://gist.github.com/tarruda/08e3a5cb5ebade8acc97

It still needs some improvements:

- An API function to resize the child process terminal(eg:

  uv_resize_terminal(uv_process_t *, int cols, int rows))

- Refactor to use `posix_openpt` which is seems to be standard posix way of

  opening pseudo-terminals. The `openpty` function is non-standard and requires

  an extra `-lutil` link.

- Add more tests

Note that my goal was to provide the simplest API that would behave similarly on

windows or unixes, so it has these limitations:

- All std file descriptors have to be redirected when using the

  UV_PROCESS_CREATE_TERMINAL flag

- The UV_PROCESS_CREATE_TERMINAL flag also implies that the child process is

  started in a new session(so it receives signals independently from the parent)

  and has a controlling terminal. In windows terms it means the process

  owns the console.