New to async programming. Is gevent right for this application?
Jamie
like, trying to get a grasp on async programming on Python. I'm at
the point in which I'd like to build something practical but I'm
unsure if gevent is right for this particular application:
I need to write a client that uses raw sockets. The client
authenticates to the server and a persistent TCP connection is
established. Occasionally the server transmits a small amount of data
on that connection, which I then read via the client and then act on
the resulting data.
What I'd normally do is just write this using Python's standard
sockets library and then have the app sit in a loop, reading from the
input buffer when data was present. However, it would be cleaner if I
could use an async design instead that used callbacks rather than a
loop.
Would gevent be a good library for this? If so, could someone point
me at some sample code that might be handy? Thanks!
Andrey Popp
starting point for your application:
http://bitbucket.org/denis/gevent/src/tip/examples/echoserver.py
--
С уважением, Андрей Попп.
+7 911 740 24 91
Jamie
understand async programming yet, but I don't know how to use
something like the echo server example for a client app. I don't want
to bind/listen on a specific port (like a server), I need to initiate
the connection, log-in, and then await data. What would I use rather
than the bind() and listen() methods?
Andrey Popp
echoserver I've attached. I've also commented it.
You probably need first to study python socket module
(http://docs.python.org/library/socket.html) or better to read book
UNIX programming (http://www.kohala.com/start/unpv12e.html).
gevent.socket module mimics standard python socket module, but
provides non-blocking version of it, so there is no difference in
using them.
Jamie
handler that will process the replies from the server? As an example,
here is how the asyncore module does it:
http://docs.python.org/library/asyncore.html#asyncore-example-basic-http-client
In this application, does gevent offer any real advantages to just
using standard sockets or asyncore?
> echoclient.py
> < 1KViewDownload
Bob Van Zant
Luckily, this is way better than asyncore and you'll find the programming
model to be very easy to use. As an example, here's how you might write a
simple HTTP client:
import gevent
import gevent.socket
import socket
def client():
s = gevent.socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('66.102.7.147', 80))
s.send('GET / HTTP/1.0\r\nHost: www.google.com\r\n\r\n')
block = s.recv(65536)
while block:
print block
block = s.recv(65536)
s.close()
g = gevent.spawn(client)
g.join()
Say you wanted to write a silly HTTP load tester. Take that call to spawn()
and wrap it in a loop. Put the result of all those calls to spawn() into a
list and then call gevent.joinall(). Now you've got a client app that can do
thousands of HTTP connections in parallel and, if you're careful with
python, gigabits of traffic (yes, plural, I'm doing it right now). All from
a single machine and a single CPU.
What happens under the covers is that when a call needs to block, like
connect, send and recv, that file descriptor is put onto a list to be
watched until it is ready. Once on this list your green thread (in this case
the client function) is context switched out and another thread is allowed
to run.
There are several wonderful things about this programming model. First, the
threads that get created are very light weight. They involve little more
overhead than the stack frame that contains them. They're way lighter than
normal threads and, though bound to a single CPU, allow you to create
thousands upon thousands of them without destroying your machine.
The second wonderful thing about this programming model is that
synchronization becomes a thing of the past. Because a green thread never
gets swapped out at just the wrong time you don't need to lock access to
shared state. No two threads can ever run at the same time.
Perhaps the third wonderful thing is that you still get to use the thread
abstraction that the human mind is capable of understanding. It's all
self-contained and the flow is from top to bottom without much fanfare. You
program as though you only had a single thread even though you might have
thousands.
-Bob
Jamie
appreciated. I'm making progress here, have one additional question:
How do I yield execution to the rest of application? Using your
example, Bob:
g = gevent.spawn(client)
g.join()
print "Hello!"
"Hello!" isn't printed unless an exception is thrown in "client".
> On 2/27/10 2:21 PM, "Jamie" <jjbe...@gmail.com> wrote:
>
> > I guess the basic question that I have is: How do I set up a callback
> > handler that will process the replies from the server? As an example,
> > here is how the asyncore module does it:
>
> http://docs.python.org/library/asyncore.html#asyncore-example-basic-h...>
> >>>> Ñ óâàæåíèåì, Àíäðåé Ïîïï.
> >>>> +7 911 740 24 91
>
> >> --
> >> Ñ óâàæåíèåì, Àíäðåé Ïîïï.
Bob Van Zant
Be careful though, in that case you're in a race to see who finishes first. I believe that if the main thread gets to the end the application will exit, regardless of if there's another thread still running.
Keep in mind that threads only context switch when they would otherwise block. In really fast network environments or if you're CPU bound you need to occasionally gevent.sleep() (pass no arguments in this case) which tells the gevent scheduler to let someone else run for a while.
These things become important quickly but you might be able to ignore them for now. Soon you'll understand it better.
-Bob
Bob Van Zant
statement after the join() "Hello!" is printed after the HTTP GET request
finishes. Which is what I'd expect.
I'm not getting any exceptions.
-Bob
Jamie
been great!
On Feb 27, 9:54 pm, Bob Van Zant <b...@veznat.com> wrote:
> I should note that I don't have this same problem. When I put the print
> statement after the join() "Hello!" is printed after the HTTP GET request
> finishes. Which is what I'd expect.
>
> I'm not getting any exceptions.
>
> -Bob
>
Denis Bilenko
that you're running and gevent/libevent versions.
Easy way to find those out, add this line at the beginning of your
script:
import gevent; print gevent.__version__, gevent.core.get_version()