Mac OS X, kqueue and PTY

[Victor Stinner]

Hi,

I search on the WWW for info about kqueue() and PTY: on Mac OS X, nor
kqueue() nor poll() support PTY or /dev/null, but select() does.

There are 3 options:

* just skip the test and document that you must explicitly use the
select selector on OSX to watch devices (is it possible to choose
explicitly the selector in asyncio?)
* switch from kqueue/poll to select when a device is added
* use a custom watcher to workaround the limitation, Saúl Ibarra
Corretgé wrote "libuv we fallback to using select() in a thread for
special fds on OSX"
https://github.com/joyent/libuv/blob/master/src/unix/stream.c#L290

What do you think?

Victor

[Guido van Rossum]

I'm still confused, because those tests pass with kqueue for me. Is
this perhaps a new development in OS X?

--
--Guido van Rossum (python.org/~guido)

[Victor Stinner]

The test fails on my Mac OS 10.6 (Snow Leopard).

Maybe devices are supported in newer Mac OS versions?

Victor

[Tobias Oberstein]

Am 13.01.2014 18:46, schrieb Victor Stinner:
> The test fails on my Mac OS 10.6 (Snow Leopard).
>

Expected. FWIW, see http://twistedmatrix.com/trac/ticket/5458

> Maybe devices are supported in newer Mac OS versions?

Maybe. I haven't found/heard anything in that direction though.

>
> Victor
>

[Guido van Rossum]

But how *else* would you explain that the PTY tests pass for me on my
64-bit MacBook Air running OS X 10.8.5?

I'm happy to do something like skip those tests based on feature
detection (does kqueue support PTYs on this box).

If someone really wants to use PTYs on affected OS X boxes they can
start by forcing the affected apps to use the select selector -- a
select-based work-around in the kqueue selector sounds messy, although
if someone is motivated enough to contribute a patch I will take it.

[Tobias Oberstein]

>> Expected. FWIW, see http://twistedmatrix.com/trac/ticket/5458
>>> Maybe devices are supported in newer Mac OS versions?
>> Maybe. I haven't found/heard anything in that direction though.
> But how *else* would you explain that the PTY tests pass for me on my
> 64-bit MacBook Air running OS X 10.8.5?

Yes, it seems to be the most likely explanation (that Apple indeed did
something to kqueue/PTY on newer OSX versions).

I once wrote a test:

https://github.com/oberstet/txkqreactor/blob/master/test/test_pty.py

I don't have a recent OSX system. Would be interesting what above test
outputs on your system ..

/Tobias

[Guido van Rossum]

$ python oberstein.py
[<select.kevent ident=0 filter=-1 flags=0x4000 fflags=0x0 data=0x22 udata=0x0>]
kqueue has no support for PTY
$

I am immersed in the PEP 460 debate, not sure when I'll have time to
investigate more thoroughly why the PTY tests don't fail for me.

[Jonathan Slenders]

For me, on OS X, Tobias his script tells me I seem to have PTY support.

Jonathans-MacBook-Pro:~ jonathan$ python /tmp/test_pty.py 

[]
kqueue seems to support PTY

Jonathans-MacBook-Pro:~ jonathan$ python -V
Python 2.7.5

Jonathans-MacBook-Pro:~ jonathan$ sw_vers -productVersion
10.9

Jonathans-MacBook-Pro:~ jonathan$ uname -a
Darwin Jonathans-MacBook-Pro.local 13.0.0 Darwin Kernel Version 13.0.0: Thu Sep 19 22:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64 x86_64

[Victor Stinner]

2014/1/14 Jonathan Slenders <jonathan...@gmail.com>:

> Jonathans-MacBook-Pro:~ jonathan$ python /tmp/test_pty.py
> kqueue seems to support PTY

> Jonathans-MacBook-Pro:~ jonathan$ sw_vers -productVersion
> 10.9

So kqueue was improved in OS X Maverick (10.9).

Guido wrote:
> If someone really wants to use PTYs on affected OS X boxes
> they can start by forcing the affected apps to use the select selector

The workaround is not so easy:
---
import sys, platform
if sys.platform == 'darwin':
version_txt = platform.mac_ver()[0]
version = tuple(map(int, version_txt.split('.')))
if version < (10, 9):
import selectors, asyncio
from asyncio.unix_events import SelectorEventLoop
selector = selectors.SelectSelector()
asyncio.set_event_loop(SelectorEventLoop(selector))
---

What do you think of the opposite? Use select.select() on OS X < 10.9
by default, but explain how to force select.epoll if you don't care of
PTY?

This discussion remembers me a Python issue proposing to add a
function to get the best selector depending on different criteria like
"don't use a file descriptor for the poller" (use poll instead of
epoll).

FYI Apple provides the Maverick upgrade for free (!) for Mac OS X
users. I will upgrade my iMac (which currently runs Mac OS 10.6.8) :-)

Victor

[Guido van Rossum]

No, they would just always call
set_event_loop(SelectorEventLoop(selector)). You are proposing to make
that the default for everyone which I think would be a really bad
step, since few people need PTYs.

Let's just skip the PTY unit tests on OS X before Mavericks.

[Victor Stinner]

2014/1/23 Guido van Rossum <gu...@python.org>:

> No, they would just always call
> set_event_loop(SelectorEventLoop(selector)). You are proposing to make
> that the default for everyone which I think would be a really bad
> step, since few people need PTYs.

Ok, I agree. Kqueue is more efficient. Skipping the test is fair :-)

I modified the test to skip it on Mac OS X < 10.9:

- Tulip: changeset 5f733b6064c8.
- Python: http://hg.python.org/cpython/rev/424564cffad0

Victor

[Jonathan Slenders]

Fine for me too to skip older OS X versions.

Could you also take a look at the following patch? This is the fix for pty writes, while the other was for pty reads only.

Probably, you should do the same here with: support.requires_mac_ver(10, 9)

https://codereview.appspot.com/54130043

Jonathan

[Victor Stinner]

Hi,

2014/1/23 Jonathan Slenders <jonathan...@gmail.com>:

> Fine for me too to skip older OS X versions.

I improved the skip to only skip PTY tests with the kqueue selector:
the test should pass with select and poll selector.

> Could you also take a look at the following patch? This is the fix for pty
> writes, while the other was for pty reads only.

> https://codereview.appspot.com/54130043

The patch is simple and has a simple, I applied it, thanks. Changeset
d5986dcbef0b.

> Probably, you should do the same here with: support.requires_mac_ver(10, 9)

I upgraded my iMac to Maverick: the test pass with select, poll and
kqueue. I added the decorator to skip the test on older Mac OS X
versions.

Victor

[Guido van Rossum]

Is there anything left to do here (besides copying the changes to the
cpython repo)?

[Victor Stinner]

Copy to CPython and that's all.

Victor

[Victor Stinner]

2014-01-25 Victor Stinner <victor....@gmail.com>:

> Copy to CPython and that's all.

Done: I copied all files from CPython to Tulip, so PTY changes with
other Tulip changes.

Victor