On 26 Mar 2014 08:22, "Antoine Pitrou" <soli...@pitrou.net> wrote:
>
>
> Hi,
>
> On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for
> subprocess.popen. I answered that asyncio now has subprocess support
> (including non-blocking I/O on the three standard stream pipes), so
> it's not obvious anything else is needed.
>
> Should we change the PEP's status to Rejected or Superseded?
Yes. I think we'd typically use Rejected in this case, as Superseded normally relates to the evolution of interface definition PEPs.
Cheers,
Nick.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
<mailto:josiah.carlson@gmail.com>>:
> * Because it is example docs, maybe a multi-week bikeshedding
discussion
> about API doesn't need to happen (as long as "read line", "read X
bytes",
> "read what is available", and "write this data" - all with
timeouts - are
> shown, people can build everything else they want/need)
I don't understand this point. Using asyncio, you can read and write a
single byte or a whole line. Using functions like asyncio.wait_for(),
it's easy to add a timeout on such operation.
Victor
--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
2014-03-28 2:16 GMT+01:00 Josiah Carlson <josiah....@gmail.com>:
> def do_login(...):I don't understand this example. How is it "asynchronous"? It looks
> proc = subprocess.Popen(...)
> current = proc.recv(timeout=5)
> last_line = current.rstrip().rpartition('\n')[-1]
> if last_line.endswith('login:'):
> proc.send(username)
> if proc.readline(timeout=5).rstrip().endswith('password:'):
> proc.send(password)
> if 'welcome' in proc.recv(timeout=5).lower():
> return proc
> proc.kill()
like blocking calls. In my definition, asynchronous means that you can
call this function twice on two processes, and they will run in
parallel.
Using greenlet/eventlet, you can write code which looks blocking, but
runs asynchronously. But I don't think that you are using greenlet or
eventlet here.
I take a look at the implementation:
http://code.google.com/p/subprocdev/source/browse/subprocess.py
It doesn't look portable. On Windows, WriteFile() is used. This
function is blocking, or I missed something huge :-) It's much better
if a PEP is portable. Adding time.monotonic() only to Linux would make
the PEP 418 much shorter (4 sentences instead of 10 pages? :-))!
The implementation doesn't look reliable:
def get_conn_maxsize(self, which, maxsize):
# Not 100% certain if I get how this works yet.
if maxsize is None:
maxsize = 1024
...
This constant 1024 looks arbitrary. On UNIX, a write into a pipe may
block with less bytes (512 bytes).
asyncio has a completly different design. On Windows, it uses
overlapped operations with IOCP event loop. Such operation can be
cancelled. Windows cares of the buffering. On UNIX, non-blocking mode
is used with select() (or something faster like epoll) and asyncio
retries to write more data when the pipe (or any file descriptor used
for process stdin/stdoud/stderr) becomes ready (for reading/writing).
asyncio design is more reliable and portable.
I don't see how you can implement asynchronous communication with a
subprocess without the complex machinery of an event loop.
> The API above can be very awkward (as shown :P ), but that's okay. FromAs I wrote, it's complex to handle non-blocking file descriptors. You
> those building blocks a (minimally) enterprising user would add
> functionality to suit their needs. The existing subprocess module only
> offers two methods for *any* amount of communication over pipes with the
> subprocess: check_output() and communicate(), only the latter of which
> supports sending data (once, limited by system-level pipe buffer lengths).
have to catch EWOULDBLOCK and retries later when the file descriptor
becomes ready. The main thread has to watch for such event on the file
descriptor, or you need a dedicated thread. By the way,
subprocess.communicate() is currently implemented using threads on
Windows.
> Neither allow for nontrivial interactions from a single subprocess.Popen()I call this "non-blocking functions", not "async functions".
> invocation. The purpose was to be able to communicate in a bidirectional
> manner with a subprocess without blocking, or practically speaking, blocking
> with a timeout. That's where the "async" term comes from.
It's quite simple to check if a read will block on not on UNIX. It's
more complex to implement it on Windows. And even more complex to
handle to add a buffer to write().
> Your next questions will be: But why bother at all? Why not just build theYou don't have to rewrite your whole application. If you only want to
> piece you need *inside* asyncio? Why does this need anything more? The
> answer to those questions are wants and needs. If I'm a user that needs
> interactive subprocess handling, I want to be able to do something like the
> code snippet above. The last thing I need is to have to rewrite the way my
> application/script/whatever handles *everything* just because a new
> asynchronous IO library has been included in the Python standard library -
> it's a bit like selling you a $300 bicycle when you need a $20 wheel for
> your scooter.
use asyncio event loop in a single function, you can use
loop.run_until_complete(do_login) which blocks until the function
completes. The "function" is an asynchronous coroutine in fact.
Even if eval_python_async() is asynchronous, eval_python() function is
blocking so you can write: print("1+1 = %r" % eval_python("1+1"))
without callback nor "yield from".
Running tasks in parallel is faster than running them in sequence
(almost 5 times faster on my PC).
The syntax in eval_python_async() is close to the API you proposed,
except that you have to add "yield from" in front of "blocking"
functions like read() or drain() (it's the function to flush the stdin
buffer, I'm not sure that it is needed in this example).
The timeout is on the whole eval_python_async(), but you can as well
using finer timeout on each read/write.
I agree that writing explicit asynchronous code is more complex than
> But here's the thing: I can build enough using asyncio in 30-40 lines of
> Python to offer something like the above API. The problem is that it really
> has no natural home.
using eventlet. Asynchronous programming is hard.
> But in the docs? It would show an atypical, but notThe asyncio documentation is still a work-in-progress. I tried to
> wholly unreasonable use of asyncio (the existing example already shows what
> I would consider to be an atypical use of asyncio).
document all APIs, but there are too few examples and the
documentation is still focused on the API instead of being oriented to
the user of the API.
Don't hesitate to contribute to the documentation!
If it makes you feel any better, I spent an hour this morning building a 2-function API for Linux and Windows, both tested, not using ctypes, and not even using any part of asyncio (the Windows bits are in msvcrt and _winapi). It works in Python 3.3+. You can see it here: http://pastebin.com/0LpyQtU5
On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson <josiah....@gmail.com> wrote:
If it makes you feel any better, I spent an hour this morning building a 2-function API for Linux and Windows, both tested, not using ctypes, and not even using any part of asyncio (the Windows bits are in msvcrt and _winapi). It works in Python 3.3+. You can see it here: http://pastebin.com/0LpyQtU5Seeing this makes *me* feel better. I think it's reasonable to add (some variant) of that to the subprocess module in Python 3.5. It also belongs in the Activestate cookbook. And no, the asyncio module hasn't made it obsolete.
Josiah, you sound upset about the whole thing -- to the point of writing unintelligible sentences and passive-aggressive digs at everyone reading this list. I'm sorry that something happened that led you feel that way (if you indeed feel upset or frustrated) but I'm glad that you wrote that code snippet -- it is completely clear what you want and why you want it, and also what should happen next (a few rounds of code review on the tracker).
But that PEP? It's just a terrible PEP. It doesn't contain a single line of example code. It doesn't specify the proposed interface, it just describes in way too many sentences how it should work, and contains a whole lot of references to various rants from which the reader is apparently meant to become enlightened. I don't know which of the three authors *really* wrote it, and I don't want to know -- I think the PEP is irrelevant to the proposed feature, which is of "put it in the bug tracker and work from there" category -- presumably the PEP was written based on the misunderstanding that having a PEP would make acceptance of the patch easier, or because during an earlier bikeshedding round someone said "please write a PEP" (someone always says that). I propose to scrap the PEP (set the status to Withdrawn) and just work on adding the methods to the subprocess module.
If it were me, I'd define three methods, with longer names to clarify what they do, e.g.
proc.write_nonblocking(data)data = proc.read_nonblocking()
data = proc.read_stderr_nonblocking()
I.e. add _nonblocking to the method names to clarify that they may return '' when there's nothing available, and have a separate method for reading stderr instead of a flag. And I'd wonder if there should be an unambiguous way to detect EOF or whether the caller should just check for proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable when the other end is closed, and then the write() will fail. But maybe I forget.)
But that's all bikeshedding and it can happen on the tracker or directly on the list just as easily; I don't see the need for a PEP.
On 3/28/2014 12:45 PM, Josiah Carlson wrote:Thank you. The docs gave me the impression that I could simply write proc.stdin and read proc.stdout. I failed with even a simple echo server (on Windows) and your code suggests why. So it does not get lost, I attached your code to
If it makes you feel any better, I spent an hour this morning building a
2-function API for Linux and Windows, both tested, not using ctypes, and
not even using any part of asyncio (the Windows bits are in msvcrt and
_winapi). It works in Python 3.3+. You can see it here:
http://pastebin.com/0LpyQtU5
http://bugs.python.org/issue18823
My interest is with Idle. It originally ran user code in the same process as the Shell and Editor code. Then Guido added an option to os.spawn a separate process and communicate through a socket connection and the option became the default with same process (requested by -N on the command line) as a backup option. 3.2 switched to using subprocess, but still with a socket. The problem is that the socket connection intermittently fails. Firewalls are, or at least used to be one possible cause, but there are others -- unknown. (While it works, the suggestion to restart with -N is a mystery to people who have never seen a command line.) This is one of the biggest sources of complaints about Idle. A pipe connection method that always worked on Windows, *x, and Mac would be great in itself and would also allow code simplification by removing the -n option. (Roger Serwy has suggested the latter as having two modes makes patching trickier.)
The current socket connection must be non-blocking. Even though the exec loop part of the Shell window waits for a response after sending a user statement, everything else is responsive. One can select text in the window, use the menus, or switch to another window. So Idle definitely needs non-blocking write and read.
In my ignorance, I have no idea whether the approach in your code or that in Viktor's code is better. Either way, I will appreciate any help you give, whether by writing, reviewing, or testing, to make communication with subprocesses easier and more dependable.
If it were me, I'd define three methods, with longer names to clarify what they do, e.g.
proc.write_nonblocking(data)
data = proc.read_nonblocking()
data = proc.read_stderr_nonblocking()
Easily doable.
Le 28 mars 2014 21:59, "Terry Reedy" <tjr...@udel.edu> a écrit :
>
> On 3/28/2014 6:20 AM, Victor Stinner wrote:
>
>> Full example of asynchronous communication with a subprocess (the
>> python interactive interpreter) using asyncio high-level API:
>
> However, the code below creates a subprocess for one command and one response, which can apparently be done now with subprocess.communicate. What I and others need is a continuing (non-blocking) conversion with 1 and only 1 subprocess (see my response to Josiah), and that is much more difficult. So this code does not do what he claims his will do.
I tried to write the shortest example showing how to read and send data and how to make the call blocking. It's different to communicate() because write occurs after the first read.
It should be quite easy to enhance my example to execute more commands.
Victor
If it were me, I'd define three methods, with longer names to clarify what they do, e.g.
proc.write_nonblocking(data)
data = proc.read_nonblocking()
data = proc.read_stderr_nonblocking()
Easily doable.