Interacting with sys.stdin/sys.stdout/sys.stderr

[Nathan Hoad]

Hello,

I was wondering if anyone had tried doing asynchronous interaction with stdio, and if so, had they had any success?

I had to make some small changes to _UnixWritePipeTransport and _UnixReadPipeTransport to check for stat.S_ISCHR (supplementary to S_ISFIFO and S_ISSOCK calls). After I did this, I was able to read from stdin and write to stdout separately, but if I mix the two in a given process, then writing to stdout breaks. I've traced this down to a _UnixWritePipeTransport._read_ready call, indicating the pipe was closed on the other end, which I'm inclined to think is not true.

Here's an example showing what I've tried:

import asyncio
import sys

from asyncio.unix_events import _set_nonblocking
from asyncio.streams import StreamWriter

EVENT_LOOP = asyncio.get_event_loop()

@asyncio.coroutine
def stdio(stdin=sys.stdin, stdout=sys.stdout):
    class Stdout(asyncio.Protocol): pass  # unused, just a placeholder

    _set_nonblocking(stdin.fileno())
    _set_nonblocking(stdout.fileno())

    reader = asyncio.StreamReader()
    reader_protocol = asyncio.StreamReaderProtocol(reader)

    yield from EVENT_LOOP.connect_read_pipe(lambda: reader_protocol, stdin)

    transport, protocol = yield from EVENT_LOOP.connect_write_pipe(Stdout, stdout)
    writer = StreamWriter(transport, protocol, reader, EVENT_LOOP)

    while True:

        # toggle these two assignments to line to see stdout working vs not

        # line = b'test line\n'

        line = yield from reader.readline()
        print(line, file=sys.stderr)
        writer.write(b'data received ')
        writer.write(line)

 

EVENT_LOOP.run_until_complete(stdio())

 

Terminal output for that reading/writing:

nathan@nathan $ python asyncio
1                                                     
b'1\n'
2
b'2\n'
3
b'3\n'
pipe closed by peer or os.write(pipe, data) raised exception.
4
b'4\n'
pipe closed by peer or os.write(pipe, data) raised exception.
pipe closed by peer or os.write(pipe, data) raised exception.
5
b'5\n'
pipe closed by peer or os.write(pipe, data) raised exception.
pipe closed by peer or os.write(pipe, data) raised exception.

Terminal output for just writing:

nathan@nathan $ python asyncio
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line

If anyone has any ideas, that would be good.

Thank you,

Nathan.

[Victor Stinner]

Hi,

2014-02-12 10:01 GMT+01:00 Nathan Hoad <nat...@getoffmalawn.com>:
> Hello,
>
> I was wondering if anyone had tried doing asynchronous interaction with
> stdio, and if so, had they had any success?

Tulip now has multiple examples showing how to interact with stdin and stdout:
- examples/child_process.py (low-level API, as you did)
- examples/shell.py
- examples/subprocess_shell.py

> I had to make some small changes to _UnixWritePipeTransport and
> _UnixReadPipeTransport to check for stat.S_ISCHR (supplementary to S_ISFIFO
> and S_ISSOCK calls).

It looks like you are using an old version, please try the latest
version (0.3.1).

> After I did this, I was able to read from stdin and
> write to stdout separately, but if I mix the two in a given process, then
> writing to stdout breaks. I've traced this down to a
> _UnixWritePipeTransport._read_ready call, indicating the pipe was closed on
> the other end, which I'm inclined to think is not true.

It's hard to tell you what is the problem. First, what is the program
you are running (the one which reads stdin and writes stdout)?

>> _set_nonblocking(stdin.fileno())
>> _set_nonblocking(stdout.fileno())
>> ...

>> yield from EVENT_LOOP.connect_read_pipe(lambda: reader_protocol,
>> stdin)
>> transport, protocol = yield from EVENT_LOOP.connect_write_pipe(Stdout,
>> stdout)

I don't think that _set_nonblocking() is needed. It's done by
connect_read/write_pipe() if I remember correctly. Where does
_set_nonblocking() come from? asyncio.unix_events? If yes, please
don't use private methods.

>> writer = StreamWriter(transport, protocol, reader, EVENT_LOOP)

Here you connect stdin to stdout with the reader parameter. In the
latest version you can pass None for the reader. But it should not be
an issue.

If you would like to make sure that all data are written, you may add:
yield from writer.drain()

Victor

[Nathan Hoad]

Hello,

On Wednesday, 12 February 2014 20:14:25 UTC+11, Victor Stinner wrote:
>
> Hi,

>
> Tulip now has multiple examples showing how to interact with stdin and
> stdout:
> - examples/child_process.py (low-level API, as you did)
> - examples/shell.py
> - examples/subprocess_shell.py

Maybe I'm misunderstanding - all of these examples demonstrate communicating
with stdin/stdout of a subprocess - I am looking to communicate over
stdin/stdout in the current running process, communicating with a parent
process, e.g. a terminal.

> It looks like you are using an old version, please try the latest
> version (0.3.1).

Thank you! I've done this now. I feel much better having gotten rid of that.

>
>
> It's hard to tell you what is the problem. First, what is the program
> you are running (the one which reads stdin and writes stdout)?

I am running the given example in a terminal, interacting with it directly
myself.

>
>
> I don't think that _set_nonblocking() is needed. It's done by
> connect_read/write_pipe() if I remember correctly. Where does
> _set_nonblocking() come from? asyncio.unix_events? If yes, please
> don't use private methods.

Indeed, it would appear that _set_nonblocking is not needed. For the record,
I know that private methods shouldn't be used - I just thought it would help
clarify that blocking vs non-blocking wasn't the issue. Sorry for the
confusion.

>
>
> >> writer = StreamWriter(transport, protocol, reader, EVENT_LOOP)
>
> Here you connect stdin to stdout with the reader parameter. In the
> latest version you can pass None for the reader. But it should not be
> an issue.

Yep, it doesn't seem to be an issue - I have tried both versions to
completeness.

>
> If you would like to make sure that all data are written, you may add:
> yield from writer.drain()

Okay, so calling this meant changing that placeholder Stdio class to inherit
from asyncio.streams.FlowControlMixin, because the drain method assumes the
protocol has a _make_drain_waiter method.

After reading from stdin, I get a ConnectionResetError.

test
b'test\n'
Traceback (most recent call last):
File "stdio.py", line 31, in <module>
EVENT_LOOP.run_until_complete(stdio())
File
"/home/nathan/.local/lib/python3.3/site-packages/asyncio/base_events.py",
line 177, in run_until_complete
return future.result()
File "/home/nathan/.local/lib/python3.3/site-packages/asyncio/futures.py",
line 236, in result
raise self._exception
File "/home/nathan/.local/lib/python3.3/site-packages/asyncio/tasks.py",
line 279, in _step
result = coro.send(value)
File "stdio.py", line 28, in stdio
yield from writer.drain()
File "/home/nathan/.local/lib/python3.3/site-packages/asyncio/streams.py",
line 247, in drain
raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost

But if I don't read from stdin, then no error occurs:

b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line
b'test line\n'
data received test line

Thank you,

Nathan.

[Jonathan Slenders]

If it helps, I got reading/writing to stdin/out work like this:

output_transport, output_protocol = yield from loop.connect_write_pipe(BaseProtocol, os.fdopen(0, 'wb'))
input_transport, input_protocol = yield from loop.connect_read_pipe(
                                lambda:OurInputProtocol(session), sys.stdin)

Not sure whether that's the best way.

See the example in https://github.com/jonathanslenders/libpymux and take a look at https://github.com/jonathanslenders/pymux

Hope that helps,

Jonathan

[Jonathan Slenders]

One thing that could be also confusing is that (as far as I understand) stdin and stdout are actually the same file if your process is attached to a pseudo terminal. In that case, stdin is opened as read, while stdout is opened as write. (Internally, often os.dup2 is used by the parent process to copy the file descriptor from '0' to '1' and '2' before running exec, making them identical. See pexpect for instance.) But if you open a subprocess, it's possible to use pipes, and I think in that case, they are really different file descriptors.

Further.

StreamReader and StreamWriter are often used in pairs. In Python2.7.3, you can do create a bidirectional pipe on stdin/out as follows:

>>> import os
>>> fd = os.fdopen(0, 'r+')
>>> 

Now 'fd' supports both read and write operations.
However, in Python 3.3, you can't create such an object:

>>> import os
>>> os.fdopen(0, 'r+')

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.3/os.py", line 1032, in fdopen
    return io.open(fd, *args, **kwargs)
io.UnsupportedOperation: File or stream is not seekable.
>>> 

I think that could be a bug in Python 3, but I'm not sure. 

 

[Nathan Hoad]

Hello Jonathan,

The snippet you've sent through has worked perfectly for me, thank
you. I found the os.fdopen(0, 'wb') call was important, as was the
order - connecting the read pipe before the write pipe resulted in
silence.

Thank you for the explanation on stdin/stdout in a pseudo terminal -
that helped make more sense of it all to me.

What follows is a rough example of asynchronous stdin/stdout,
implemented as an async version of input(). Hopefully this will be of
use to someone in the future.

As a bonus, the os.fdopen() call means that regular print() calls will
still work.

import os
import asyncio
import sys

from asyncio.streams import StreamWriter, FlowControlMixin

reader, writer = None, None

@asyncio.coroutine
def stdio(loop=None):
if loop is None:
loop = asyncio.get_event_loop()

reader = asyncio.StreamReader()
reader_protocol = asyncio.StreamReaderProtocol(reader)

writer_transport, writer_protocol = yield from
loop.connect_write_pipe(FlowControlMixin, os.fdopen(0, 'wb'))
writer = StreamWriter(writer_transport, writer_protocol, None, loop)

yield from loop.connect_read_pipe(lambda: reader_protocol, sys.stdin)

return reader, writer

@asyncio.coroutine
def async_input(message):
if isinstance(message, str):
message = message.encode('utf8')

global reader, writer
if (reader, writer) == (None, None):
reader, writer = yield from stdio()

writer.write(message)

line = yield from reader.readline()

return line.decode('utf8').strip()

@asyncio.coroutine
def main():
name = yield from async_input("What's your name? ")

print("Hello, {}!".format(name))

asyncio.get_event_loop().run_until_complete(main())

Thank you,

Nathan.

[Guido van Rossum]

I wonder if we shouldn't have a separate concept of a two-way pipe transport in UNIX? It's clear that the write-only pipe implementation doesn't work when the pipe is also readable.

--
--Guido van Rossum (python.org/~guido)

[Jonathan Slenders]

Late reply, but two-way pipe transport or something that feels like a better API would be really nice.

There will be more people that are going to try asynchronous stdin/stdout operations.

Honestly, I'm not sure that I have the time myself to write a patch...

[Guido van Rossum]

I'm confident that you *can* write working code using the low-level API (event loop methods, possibly writing a custom protocol, which isn't verboten). Yes, it will be complicated (as the discussion in this thread has shown), but once someone figures it out it can live for a long time as a recipe or PyPI package. We can add it to the stdlib in 3.5.

[Victor Stinner]

Hi,

2014-02-19 16:46 GMT+01:00 Jonathan Slenders <jonathan...@gmail.com>:
> Late reply, but two-way pipe transport or something that feels like a better
> API would be really nice.

I'm not sure that I understand correctly. stdin are stdout have a
different file descriptor: 0 and 1. But in case of PTY, it looks like
0 and 1 are the same file: os.path.samestat(os.fstat(0), os.fstat(1))
is True.

Do you mean that we need a duplex pipe transport when stdin and stdout
are the same PTY?

Does it make sense to wait for read event on the PTY in the write pipe
transport? It looks like writing data into the PTY raises a read
event, and os.read() on the file descriptor returns the written data.

What do you think of the following patch? It fixes the initial example.

diff -r f7b20eaefa3c asyncio/unix_events.py
--- a/asyncio/unix_events.py Tue Feb 18 10:02:52 2014 +0100
+++ b/asyncio/unix_events.py Wed Feb 19 17:29:20 2014 +0100
@@ -266,7 +270,7 @@ class _UnixWritePipeTransport(selector_e
# On AIX, the reader trick only works for sockets.
# On other platforms it works for pipes and sockets.
# (Exception: OS X 10.4? Issue #19294.)
- if is_socket or not sys.platform.startswith("aix"):
+ if is_socket or not sys.platform.startswith("aix") and not
os.isatty(self._fileno):
self._loop.add_reader(self._fileno, self._read_ready)

self._loop.call_soon(self._protocol.connection_made, self)

Victor

[Guido van Rossum]

Ah, clever. It needs a little more research--is it always the case that a tty FD works this way? But tentatively worth getting into RC2.

[Jonathan Slenders]

Le mercredi 19 février 2014 17:32:25 UTC+1, Victor Stinner a écrit :

I'm not sure that I understand correctly. stdin are stdout have a
different file descriptor: 0 and 1. But in case of PTY, it looks like
0 and 1 are the same file: os.path.samestat(os.fstat(0), os.fstat(1))
is True.

This is because the file descriptor is copied, this is how forkpty works.

Maybe you can take a look at this line of pexpect:

https://github.com/noahspurrier/pexpect/blob/master/pexpect.py#L630

It looks like writing data into the PTY raises a read
event, and os.read() on the file descriptor returns the written data.

This is only the case when the pty is in canonical mode. (The pty implements some line editing/buffering functionality itself.)

Maybe you want to set the pty in raw mode first. Everything seems more logical then:

http://docs.python.org/2/library/tty.html#tty.setraw

To the questions considering asyncio itself, I don't have a real answer.

[Guido van Rossum]

On Wed, Feb 19, 2014 at 11:08 AM, Jonathan Slenders <jonathan...@gmail.com> wrote:

Le mercredi 19 février 2014 17:32:25 UTC+1, Victor Stinner a écrit :

I'm not sure that I understand correctly. stdin are stdout have a
different file descriptor: 0 and 1. But in case of PTY, it looks like
0 and 1 are the same file: os.path.samestat(os.fstat(0), os.fstat(1))
is True.

This is because the file descriptor is copied, this is how forkpty works.

To clarify, this is a fundamental property of pseudo ttys. I assume it's similar to sockets: a socket always represents a bidirectional channel, and if you use dup() on a socket, all FDs produced that way will become ready for reading or writing at the same time. (I wouldn't be surprised if real ttys work this way too, but I haven't seen one in over a decade. :-)

In any case, the consequence is that even though stdin and stdout (and stderr) use different FDs, they refer to the same object in the kernel, and that object doesn't know which FD you were using to refer to it. So if there's data to be read, this information is available equally through all FDs.

 

Maybe you can take a look at this line of pexpect:

https://github.com/noahspurrier/pexpect/blob/master/pexpect.py#L630

 

It looks like writing data into the PTY raises a read
event, and os.read() on the file descriptor returns the written data.

This is only the case when the pty is in canonical mode. (The pty implements some line editing/buffering functionality itself.)

Hm, I would presume that what you're really observing here is the behavior of whatever process controls the other end (the master). That would be your terminal emulator program.

 

Maybe you want to set the pty in raw mode first. Everything seems more logical then:

http://docs.python.org/2/library/tty.html#tty.setraw

This documentation isn't very helpful. It would be more useful to look up the theory and operation of pseudo ttys in some UNIX internals book.

 

To the questions considering asyncio itself, I don't have a real answer.

Having thought more about it, I am fine with Victor's fix.

[Victor Stinner]

> Having thought more about it, I am fine with Victor's fix.

I opened an issue with a better patch (adding an example):
http://code.google.com/p/tulip/issues/detail?id=147

Victor

[Jonathan Slenders]

Thanks, the example is very clear and helpful!

[Victor Stinner]

2014-02-20 8:57 GMT+01:00 Jonathan Slenders <jonathan...@gmail.com>:
> Thanks, the example is very clear and helpful!

Sorry, but I didn't understand your opinion on the patch. Does it look
correct? It just found a new issue with the non-blocking mode when the
cat program is used as input. IMO the pipe should not be set to
non-blocking if it's a TTY.

The worst case with my patch would be that the write protocol is not
notified immediatly when the PTY is closed or if the file descriptor
is closed, but only at the next write.

Or would it be better to use a duplex pipe transport for PTY?

Victor

[Jonathan Slenders]

For the patch, honestly I'm not completely sure.

If the patch doesn't break anything else, I'm okay, but I don't know what caveats there are.

The example was nice, if that could ever work.

I got a copy of "The linux programming interface", still so much to learn :)