Reading from character special device file

[Jonathan Slenders]

Hi all,

Why does  loop.connect_read_pipe not support the pipes that openpty() creates, but is does for os.pipe()

master, slave = os.openpty()
shell_out = io.open(master, 'rb', 0)
transport, protocol = yield from loop.connect_read_pipe(MyProtocol, shell_out)

In unix_events.py, there's the following check that fails.

if not (stat.S_ISFIFO(mode) or stat.S_ISSOCK(mode)):

I think it should be something like this:

if not (stat.S_ISFIFO(mode) or stat.S_ISSOCK(mode) or stat.S_ISCHR(mode)):

I tried this patch and it resolves the issue for me. Correct me if I'm wrong, but I don't think there is any reason to block character devices like this. Actually they're not that different from a unix pipe.


For those interested in my use case:
I'm building something like "tmux" or "gnu screen", but in Pure Python. I use the "pyte" library as output interpreter, but will write a custom renderer.
Calling the subprocess_exec wrapper does not work, because it uses normal pipes, while I want the programs that run to be aware that they are running in a real pseudo terminal.


Cheers,
Jonathan

[Guido van Rossum]

Why do you report a bug using a rhetorical question? :-)

Please file a bug either at http://code.google.com/p/tulip/issues/list
or bugs.python.org (the former gets my attention, the latter might get
other developers' attention). If you want to submit a patch with a
test that would be super. (See
http://code.google.com/p/tulip/wiki/Contributing .)

--
--Guido van Rossum (python.org/~guido)

[Jonathan Slenders]

I created a patch: https://codereview.appspot.com/38500044/

That works, but the unit test still fails, I'm not sure how to correctly close this pipe.


[tulip] > python runtests.py
Skipping 'test_windows_events': Windows only
Skipping 'test_windows_utils': Windows only
......................................................................................................................
............................Fatal error for <asyncio.unix_events._UnixReadPipeTransport object at 0x7fb6a5951e10>
Traceback (most recent call last):
  File "/home/jonathan/hg/tulip/asyncio/unix_events.py", line 204, in _read_ready
    data = os.read(self._fileno, self.max_size)
OSError: [Errno 5] Input/output error
Exception in callback <bound method _UnixReadPipeTransport._call_connection_lost of <asyncio.unix_events._UnixReadPipe
Transport object at 0x7fb6a5951e10>> (OSError(5, 'Input/output error'),)
Traceback (most recent call last):
  File "/home/jonathan/hg/tulip/asyncio/events.py", line 38, in _run
    self._callback(*self._args)
  File "/home/jonathan/hg/tulip/asyncio/unix_events.py", line 240, in _call_connection_lost
    self._protocol.connection_lost(exc)
  File "tests/test_events.py", line 135, in connection_lost
    assert self.state == ['INITIAL', 'CONNECTED', 'EOF'], self.state
AssertionError: ['INITIAL', 'CONNECTED']

[Guido van Rossum]

I looked into this. The tests pass on OSX (10.8), but fail indeed on
my Ubuntu VM. I can make them pass by catching the OSError with
errno==EIO and pretending it's an EOF *and* using
test_utils.run_until() instead of run_briefly() in the test (in both
places).

Here's a revised version of the patch:
http://codereview.appspot.com/48350043 I also folded some long lines
and made master_read_obj unbuffered. I also think that technically the
master should be set in nonblocking mode (using e.g.
unix_events._set_nonblocking()) but it doesn't seem to affect the
test. I also wonder if the test ought to be in test_unix_events.py
instead of in test_events.py.

When you wrote "That works, but [...]" what did you mean by "that
works"? What code did you run to verify it? (Since clearly the test
you wrote *doesn't* work. :-)

On Sun, Jan 5, 2014 at 3:01 AM, Jonathan Slenders

[Jonathan Slenders]

It was the code I meant, not the test.

But I just tested your revised version, and now the unit tests succeed here as well. Thanks.

The master should be non blocking indeed. I my project i called ""io.open(self.master, 'wb', 0)""

Something related about blocking vs. non blocking. I don't know how these file descriptors work exactly. But I was now also able to use connect_read_pipe to read from stdin which was really nice for me. (I really didn't like the executor solution that much.) However, if you make stdin non blocking, stdout will automatically also become non blocking. But writing to non blocking stdout seems like a bad idea, (you get "write could not complete without blocking" errors everywhere.) So what I do right now is to make stdout blocking again before writing, during a repaint of the app, and unblocking after writing. (Because this is all in the same thread as the event loop, it will be non blocking again when we get there.)

It works nice, but what would be nice was to have a _sec_blocking method available in unix_events which is symmetrical.  (I why wouldn't we make them public?)

[Guido van Rossum]

On Mon, Jan 6, 2014 at 10:57 PM, Jonathan Slenders
<jonathan...@gmail.com> wrote:
> It was the code I meant, not the test.
>
> But I just tested your revised version, and now the unit tests succeed here
> as well. Thanks.

OK, but I'm not at all convinced that catching EIO and treating it the
same as EOF is correct. I don't even understand why I get that EIO.

Could you show me some synchronous code (maybe using threads) showing
how pyts are expected to work?

> The master should be non blocking indeed. I my project i called
> ""io.open(self.master, 'wb', 0)""

There seems to be confusion here -- the 0 means non-*buffering*, it
has no effect on *blocking*.

> Something related about blocking vs. non blocking. I don't know how these
> file descriptors work exactly.

Then how can you write working code using them? (Not a rhetorical question.)

> But I was now also able to use
> connect_read_pipe to read from stdin which was really nice for me.

Hm, this is really falling quite far outside the intended use case for
asyncio. (Though well within that for the selectors module.)

> (I really didn't like the executor solution that much.)

I'm not aware of that solution -- is it in the docs somewhere?

> However, if you make stdin non
> blocking, stdout will automatically also become non blocking.

Yeah, I see that too. I can't explain it; I suspect it's some obscure
implementation detail of tty (or pty) devices. :-(

> But writing to
> non blocking stdout seems like a bad idea, (you get "write could not
> complete without blocking" errors everywhere.)

So wrap a transport/protocol around stdin/stdout.That would seem to be
the asyncio way to do it. You should probably wrap those in a
StreamReader/StreamWriter pair -- the source code (streams.py) shows
how to create those by hand, which is an intended use.

> So what I do right now is to
> make stdout blocking again before writing, during a repaint of the app, and
> unblocking after writing. (Because this is all in the same thread as the
> event loop, it will be non blocking again when we get there.)

Eew! :-(

> It works nice, but what would be nice was to have a _set_blocking method

> available in unix_events which is symmetrical. (I why wouldn't we make them
> public?)

By the time you are passing your own file descriptors you should be
mature enough to know how to make them non-blocking. Plus this is all
UNIX-specific... Plus, create_pipe_transport() already calls it for
you.

Overall, perhaps you should just use the selectors module instead of
asyncio? You might be happier with that...

[Jonathan Slenders]

2014/1/7 Guido van Rossum <gu...@python.org>

On Mon, Jan 6, 2014 at 10:57 PM, Jonathan Slenders
<jonathan...@gmail.com> wrote:
> It was the code I meant, not the test.
>
> But I just tested your revised version, and now the unit tests succeed here
> as well. Thanks.

OK, but I'm not at all convinced that catching EIO and treating it the
same as EOF is correct. I don't even understand why I get that EIO.

Could you show me some synchronous code (maybe using threads) showing
how pyts are expected to work?

My guess is that we should usually close the master side first.

I found at books.google.com, the linux programming interface, p1388:

"""

If we close all file descriptors referring to the pseudoterminal master, then:

- if the slave device has a controlling process, a SIGHUP signal is sent to that process.

- a read() from the slave device returns end-of-of

- a write to the slave device failes with error EIO (on some other UNIX implementations, write fails with the error ENXIO in this case.)

If we close all file descriptors referring to the pseudoterminal slave, then:

- a read() from the master device fails with error EIO (on some other UNIX implementations, a read() returns end-of-file in this case.

"""

You are always going to attach an client application (e.g. an editor) on the slave side and have the terminal application (e.g. xterm) on the master side.

By default, you'll have the pty in line buffered mode. This means that written characters on the master will be echo'ed back on the master (to be displayed). Only after enter has been pressed, becomes the whole line available to be read on the slave. This way, the pseudo terminal implements some line editing functionality itself. A character written on the slave is always immediately available on the master.

In raw mode, every key press on the master is immediately send to the slave side. The application on the slave is in that case also responsible for displaying it, and should probably send some feedback by echoing back the received characters.

> The master should be non blocking indeed. I my project i called
> ""io.open(self.master, 'wb', 0)""

There seems to be confusion here -- the 0 means non-*buffering*, it
has no effect on *blocking*.

Sorry, you're right. I was confused.

 

> Something related about blocking vs. non blocking. I don't know how these
> file descriptors work exactly.

Then how can you write working code using them? (Not a rhetorical question.)

What I meant is that often I feel there a still missing parts I don't understand, but I know enough to build something useful.

> But I was now also able to use
> connect_read_pipe to read from stdin which was really nice for me.

Hm, this is really falling quite far outside the intended use case for
asyncio. (Though well within that for the selectors module.)

> (I really didn't like the executor solution that much.)

I'm not aware of that solution -- is it in the docs somewhere?

Oh, just an executor running a blocking read in a while loop:

def in_executor():

   while True:

      c = stdin.read(1)

      process_char(c)

> However, if you make stdin non
> blocking, stdout will automatically also become non blocking.

Yeah, I see that too. I can't explain it; I suspect it's some obscure
implementation detail of tty (or pty) devices. :-(

 

Maybe, like openpty(), stdin and stdout can actually be the same file descriptor underneath if they are attached to a pseudo terminal, even if they have different numbers. I learned that if you want to attach a child process to a newly created pseudo terminal, you do it by calling openpty, take the slave number, fork you own process, and in the fork use os.dup2 to copy that file descriptor to 0 and 1 (for stdout and stdin respectively.)

> But writing to
> non blocking stdout seems like a bad idea, (you get "write could not
> complete without blocking" errors everywhere.)

So wrap a transport/protocol around stdin/stdout.That would seem to be
the asyncio way to do it. You should probably wrap those in a
StreamReader/StreamWriter pair -- the source code (streams.py) shows
how to create those by hand, which is an intended use.

Thanks! That sounds really helpful. 

> So what I do right now is to
> make stdout blocking again before writing, during a repaint of the app, and
> unblocking after writing. (Because this is all in the same thread as the
> event loop, it will be non blocking again when we get there.)

Eew! :-(

Yes, I know.

[Guido van Rossum]

OK, I think I get it now. I've worked up a simpler version of the
patch, because I realized that the issue in _read_ready() is just that
we don't want the EIO exception to be logged. Your test still passes.
Let me know if your real code works with this version.

https://codereview.appspot.com/48350043/ (patch set 2)

raw download (for hg import):
https://codereview.appspot.com/download/issue48350043_20001.diff

[Jonathan Slenders]

Super. It's working on my real application code.

The tests also run successfully on my ubuntu machine.

Thank you!

2014/1/9 Guido van Rossum <gu...@python.org>

[Guido van Rossum]

One last question. Have you filled out a PSF contribution form? If
not, would you mind doing so? (Ideally you'd also tell me your
bugs.python.org username, so I can check.)

[Victor Stinner]

Hi,

The new test fails on Mac OS X with the kqueue selector (it pass with
the select selector).

======================================================================
ERROR: test_read_pty_output (__main__.KqueueEventLoopTests)
----------------------------------------------------------------------

Traceback (most recent call last):

File "/Users/haypo/prog/HG/tulip/asyncio/selector_events.py", line
135, in add_reader
key = self._selector.get_key(fd)
File "/Users/haypo/prog/HG/tulip/asyncio/selectors.py", line 180, in get_key
raise KeyError("{!r} is not registered".format(fileobj)) from None
KeyError: '7 is not registered'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "tests/test_events.py", line 980, in test_read_pty_output
self.loop.run_until_complete(connect())
File "/Users/haypo/prog/HG/tulip/asyncio/base_events.py", line 177,
in run_until_complete
return future.result()
File "/Users/haypo/prog/HG/tulip/asyncio/futures.py", line 236, in result
raise self._exception
File "/Users/haypo/prog/HG/tulip/asyncio/tasks.py", line 281, in _step
result = next(coro)
File "tests/test_events.py", line 974, in connect
master_read_obj)
File "/Users/haypo/prog/HG/tulip/asyncio/base_events.py", line 537,
in connect_read_pipe
transport = self._make_read_pipe_transport(pipe, protocol, waiter)
File "/Users/haypo/prog/HG/tulip/asyncio/unix_events.py", line 149,
in _make_read_pipe_transport
return _UnixReadPipeTransport(self, pipe, protocol, waiter, extra)
File "/Users/haypo/prog/HG/tulip/asyncio/unix_events.py", line 200,
in __init__
self._loop.add_reader(self._fileno, self._read_ready)
File "/Users/haypo/prog/HG/tulip/asyncio/selector_events.py", line
138, in add_reader
(handle, None))
File "/Users/haypo/prog/HG/tulip/asyncio/selectors.py", line 447, in register
self._kqueue.control([kev], 0, 0)
OSError: [Errno 22] Invalid argument
----------------------------------------------------------------------

The test fails in Trollius, Tulip and CPython. These 3 projects share
at least the same bugs :-)

Victor

[Tobias Oberstein]

Am 11.01.2014 00:35, schrieb Victor Stinner:
> Hi,
>
> The new test fails on Mac OS X with the kqueue selector (it pass with
> the select selector).

kqueue for pipes/processes is notoriously buggy on osx. in general: osx
sucks big regarding networking, and in particular Apple's kqueue
implementation is crappy.

does it fail on kqueue/freebsd?

/Tobias

[Guido van Rossum]

On Fri, Jan 10, 2014 at 1:35 PM, Victor Stinner
<victor....@gmail.com> wrote:
> The new test fails on Mac OS X with the kqueue selector (it pass with
> the select selector).

What OSX version? Other info? It passes with all three types of
selectors for me on OS X 10.8.5.

They pass for me in CPython 3, Tulip and Trollius (with Python 2.7 as
well as 2.6).

[Victor Stinner]

2014/1/11 Guido van Rossum <gu...@python.org>:

> On Fri, Jan 10, 2014 at 1:35 PM, Victor Stinner
> <victor....@gmail.com> wrote:
>> The new test fails on Mac OS X with the kqueue selector (it pass with
>> the select selector).
>
> What OSX version? Other info? It passes with all three types of
> selectors for me on OS X 10.8.5.

$ uname -a
Darwin Imac-Photo.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun
7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
$ ./python.exe -m platform
Darwin-10.8.0-i386-64bit

CPython:
Mercurial @ 52fbc7bb78ad
Python 3.4.0b2+ (default:f1f707dd7cae, Jan 10 2014, 23:24:41)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
./python.exe -m test -v test_asyncio
=> ERROR: test_read_pty_output

Tulip:
Mercurial @ 0b3e835df369
PYTHONPATH=$PWD ~/prog/python/default/python.exe tests/test_events.py -v
=> ERROR: test_read_pty_output

We don't have exactly the same OS X version (10.8.5 vs 10.8.0).

Victor

[Guido van Rossum]

Can you upgrade your OSX? 10.8.0 seems really old. Your uname -a
output says June 2011; mine has

Darwin guido-mba.corp.dropbox.com 12.5.0 Darwin Kernel Version 12.5.0:
Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64
x86_64

On Fri, Jan 10, 2014 at 2:10 PM, Victor Stinner

[Guido van Rossum]

Oh wait, maybe we're not even talking about versions of the same
thing. The 10.8.5 I quote is from the "About this Mac" dialog. My
uname has 12.5.0. Your 10.8.0 is from uname, so that's a completely
unrelated sequence of versions.

I doubt the Python or Tulip versions matter.

[Guido van Rossum]

It's also possible that it's a 32-bit vs. 64-bit issue? My mac is 64 bit.

[Saúl Ibarra Corretgé]

On 01/11/2014 12:52 AM, Tobias Oberstein wrote:
> Am 11.01.2014 00:35, schrieb Victor Stinner:
>> Hi,
>>
>> The new test fails on Mac OS X with the kqueue selector (it pass with
>> the select selector).
>
> kqueue for pipes/processes is notoriously buggy on osx. in general: osx
> sucks big regarding networking, and in particular Apple's kqueue
> implementation is crappy.
>
> does it fail on kqueue/freebsd?
>

FWIW, in libuv we fallback to using select() in a thread for special fds
on OSX: https://github.com/joyent/libuv/blob/master/src/unix/stream.c#L290

Not sure if a similar workaround could be implemented here, or if it
would be desirable.


Cheers,

--
Saúl Ibarra Corretgé
bettercallsaghul.com

[Glyph Lefkowitz]

'uname' is asking about the version of the kernel; you want the version of Mac OS X.

10.8.0 is pretty old; up-to-date Darwin kernel is 13.0.2.

If you want the whole-OS product version, the relevant command line is 'sw_vers -productVersion'.

-glyph