How to do gevent-friendly stdin/stdout I/O?

[vitaly]

By default, when my code reads from sys.stdin (raw_input,
sys.stdin.readline(), etc.), the thread blocks in a gevent-unfriendly
way, and other greenlets are starved. What's the right way to make
stdin/stdout reads/writes gevent-friendly?

Thank you.

[vitaly]

I should add that my environments of interest are Linux and MacOS.

[Ian Epperson]

I was curious about this too, and there doesn't seem to be any cleanly pre-defined replacements to input() or raw_input().  However, the tools are there to write your own.  This works for me (OSX, gevent 1.0b1):

#!/usr/bin/python

import gevent

import gevent.select

import sys

# gevent replacement for input()

def ginput( prompt ):

sys.stdout.write( prompt )

sys.stdout.flush()

gevent.select.select([sys.stdin], [], [])

return sys.stdin.readline()

# Test the gInput function

def getInput():

while True:

print "received: " + ginput( "give me data: ")

# Prove that the reactor is running

def count():

x = 1

while True:

gevent.sleep(5)

x = x + 1

print "count: %d" % x

gevent.spawn( getInput )

count()

--
This email is intended for the use of the individual addressee(s) named above and may contain information that is confidential, privileged or unsuitable for overly sensitive persons with low self-esteem, no sense of humor or irrational religious beliefs. If you are not the intended recipient, any dissemination, distribution or copying of this email is not authorized (either explicitly or implicitly) and constitutes an irritating social faux pas. Unless the word absquatulation has been used in its correct context somewhere other than in this warning, it does not have any legal or grammatical use and may be ignored. No animals were harmed in the transmission of this email, although the yorkshire terrier next door is living on borrowed time, let me tell you. Those of you with an overwhelming fear of the unknown will be gratified to learn that there is no hidden message revealed by reading this warning backwards, so just ignore that Alert Notice from Microsoft: However, by pouring a complete circle of salt around yourself and your computer you can ensure that no harm befalls you and your pets. If you have received this email in error, please add some nutmeg and egg whites and place it in a warm oven for 40 minutes. Whisk briefly and let it stand for 2 hours before icing.

[Ian Epperson]

I experimented more with this solution and found that it's not bullet proof.  If the user enters some data then EOF (ctrl-D on OSX, ctrl-Z on Windows) the entire program halts on the unsatisfied readline() until the user presses return.  I may hack on it a bit tonight to come up with something better.

Ian E.

[vitaly]

On Jun 7, 7:30 am, Ian Epperson <i...@epperson.com> wrote:
> I experimented more with this solution and found that it's not bullet
> proof.  If the user enters some data then EOF (ctrl-D on OSX, ctrl-Z on
> Windows) the entire program halts on the unsatisfied readline() until the
> user presses return.  I may hack on it a bit tonight to come up with
> something better.
>
> Ian E.

Thank you. That looked promising -- too bad EOF didn't behave. I
ended up coding a conceptually similar, but lower-level solution that
works with EOF; this one is generalized to work with file-descriptors.

I would use it like this for stdin:
```python
reader = GLineReader(sys.stdin.fileno(), makeNonBlocking=False)

line = reader.readline()

```

And here is GLineReader:

```python
import errno
import fcntl
import os
import socket

from gevent.select import select as gevent_select


class GLineReader(object):
""" Read a line of text from a file descriptor up to and including
the very
first '\n', using gevent-friendly API.

NOTE: We implement this as a class in order to faclitate performance
improvements, such as buffered reads.

TODO: next release of gevent v1.0 should have a FileObject class
that might
do what we're trying to accomplish here.
"""

def __init__(self, fd, makeNonBlocking=True):
""" Construct the reader; configure the given file descriptor for
non-blocking operation

WARNING: if the fd is from sys.stdin.fileno(),
then changing this fd to non-blocking also has
the side-
effect of making the file desriptor in sys.stdout
non-blocking, so print and sys.stdout.write/flush
will
sometimes fail with "IOError: [Errno 35] Resource
temporarily unavailable" (35=errno.EWOULDBLOCK).
"""

if makeNonBlocking:
fl = fcntl.fcntl(fd, fcntl.F_GETFL)
fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

self._fd = fd

self._blocking = not makeNonBlocking

return


def readline(self):
""" Read a line of text from a file descriptor up to and including
the very
first '\n', using gevent-friendly API.

NOTE: Lines are assumed to be terminated with '\n'

retval: The read string, including the terminating '\n',
unless EOF is
encountered first, in which case the string read so
far will be
returned without the terminating '\n'; empty string
is returned
on EOF if no line characters preceded it.
"""

fd = self._fd

data = ""

rlist = [fd]
wlist = xlist = []

while not data.endswith('\n'):
# Wait for readability
rready = gevent_select(rlist, wlist, xlist)[0]
assert fd in rready

# Read
newData = ""
while newData != '\n':
try:
newData = os.read(fd, 1)
except socket.error as e:
if e.args[0] == errno.EWOULDBLOCK:
break
else:
raise
else:
if not newData:
# EOF reached
return data

data += newData

if self._blocking:
# Don't want to risk blocking on next char if the fd is
blocking
break


return data
```

[Vitaly]

Found another solution at https://gist.github.com/2915875

[Danil "Eleneldil G. Arilou" Lavrentyuk]

It looks good, but I've got a strange thing trying it.

Using GeventFD to wrap sys.stdout and, also, to wrap a bulk file
writting I've found, that if I use 'ptint' after one bulk file write,
my next file write (even a small string) lock for long time (I hasn't
waited to the end),

Here is my test script:
=======
#!/usr/bin/python
import sys
import gevent
from gfio import GeventFD
import struct
import random
import os

sys.stdout = GeventFD(sys.stdout.fileno())
data = ''.join( struct.pack('I', random.randint(0, 1000000000)) for x
in xrange(256 * 1024))
print "%d bytes of data prepared" % len(data)

def ticker():
print "Start\n",
for x in xrange(20):
gevent.sleep(0.1)
print "%d\n" % x,

def writor(name, data):
f = open(name, 'w')
out = GeventFD(os.dup(sys.__stdout__.fileno()))
gevent.sleep(0)
print "File %s opened\n" % (name,),
g = GeventFD(f.fileno())
N = random.randint(30,80)
for i in range(N):
g.write(data)
# :1:
#out.write("%d data bytes written to %s\n" % (len(data), name))
# :2:
#print "%d data bytes written to %s\n" % (len(data)*N, name),
g.write('test')
f.close()
print "%s closed\n" % (name,),


gr = []
gr.append( gevent.spawn(writor, 'testfile', data) )
gr.append( gevent.spawn(writor, 'testfile1', data) )
gr.append( gevent.spawn(ticker) )
gr.append( gevent.spawn(writor, 'testfile2', data) )

gevent.joinall(gr)
===

It works good, but if you uncomment the line after :1: or :2:, the
next g.write() call wont return. And if you add timeout in the
wait_write() call of the GeventFD.write(), you'll get it's timeout
exception.
(You can see, it does not matter if I user os.dup() for
sys.stdout.fileno() or not.)

If I don't wrap sys.stdout with GeventFD, any 'print' blocks all
greenlets, but there is no problems with g.write() calls and prints
betwen them.

Is it possible to improve GeventFD class to avoid this problem?

Some more atempts show me, that I can't use more then one
GeventFD-wrapped file in the same Greenlet, even if I close the first
file before open thesecond one.

I think, something wrong with get_hub().wait(io) call where io =
get_hub().loop.io(fileno, 2) for two different fileno in the same
Greenlet.
But I hasn't dug it deepper.

I've found it with gevent 1.0b1 and 1.0b2.

2012/6/16 Vitaly <vital...@gmail.com>:

[Danil "Eleneldil G. Arilou" Lavrentyuk]

Also I have to note, that prints to GeventFD-wrapped stdout do not
hang (even beeng done after writes to another GeventFD-wrapped file).
But they make to hang other writes late in this greenlet.

2012/6/29 Danil "Eleneldil G. Arilou" Lavrentyuk <elen...@gmail.com>:

[Denis Bilenko]

FYI, 1.0dev has a fileobject implementation:
https://bitbucket.org/denis/gevent/src/tip/gevent/fileobject.py

[CryptWizard]

Great to see increasingly more of the shortcomings of gevent being addressed
with standard implementations.

[vitaly]

On Monday, July 9, 2012 4:54:43 AM UTC-7, Denis Bilenko wrote:

FYI, 1.0dev has a fileobject implementation:
https://bitbucket.org/denis/gevent/src/tip/gevent/fileobject.py

Looking forward to the release.  Thanks!