gevent in production
Denis Bilenko
I'm looking for examples of gevent in production for my EuroPython presentation.
If you don't mind free advertisement, please share the details about your app.
Would be cool if you mentioned
- the avg/peak number of concurrent users/connections/requests the app
handles (if server-side)
- how much hardware (memory) does it need to do that
- if it was re-written in gevent, what did you use before and how much
gevent improved resource usage / reduced the complexity / ...
- any other detail that you think is impressive
If you are going to respond privately, please mention which
information is OK to share.
My presentation is scheduled on Thursday 22 July 2010, Recital Hall.
Would love to chat about gevent with anyone who is going to be at EuroPython.
Thanks,
Denis.
Dan Korostelev
work using a custom AMF3-based protocol through sockets to provide
fast full-duplex communication needed for real-time multiplayer
gaming. Initially I used Tornado's IOLoop, but since I grokked
coroutine-style and liked it alot, I switched to gevent. Also, using
gevent+psycopg2 I can easilly make my DB connection asynchronous
without complexity of callbacks, also it makes it easy to use
SQLAlchemy in async environment. I started not so long time ago and I
currently don't have info on number of concurrent connections and
hardware requirements though.
2010/7/11 Denis Bilenko <denis....@gmail.com>:
--
WBR, Dan Korostelev
Daniele Varrazzo
> [...] Also, using
> gevent+psycopg2 I can easilly make my DB connection asynchronous
> without complexity of callbacks, also it makes it easy to use
> SQLAlchemy in async environment.
Sweet! Are you using psycopg 2.2 with a wait callback for this? I'd be
interested in some feedback.
-- Daniele
Marcin Bachry
> Yep, i'm using this wait callback -
> http://bitbucket.org/denis/psycogreen/src/tip/gevent/psyco_gevent.py
>
> BTW, I used to get some kind of deadlocks with sqlalchemy's connection
> pool until i specify max_overflow=-1 in engine arguments.
> Unfortunately I didn't have time to dig it out and understand the
> reason, so if anyone have an explanation, I'd like to learn about
> this. :-)
Check out this thread on eventlet mailing list:
https://lists.secondlife.com/pipermail/eventletdev/2010-March/000793.html
m.
Peter Schuller
> interested in some feedback.
I'm not the OP to which you addressed that question, but: At Spotify
we have been slowly trying out gevent lately; one of the applications
included use of psycopg.
Your support for the wait callback was very much appreciated. You
literally posted about it the very day I was going to sit down and
figure out how to best make psycopg work with gevent and it exactly
fit the bill.
We have used gevent+psycopg in production a little bit. However, the
usage was such as to not exercise the feature set of psycopg, nor
under any kind of high load (except during offline benchmarking).
However even so, I can happily report that we have not yet seen any
problems that could be traced back to psycopg or the
gevent/wait_callback integration.
--
/ Peter Schuller
Cyril B.
can't post in it anymore, so I'll post here instead, as Denis
suggested to me.)
> a. what are you using gevent for?
I run a web hosting company (http://www.alwaysdata.com) and we've
developed a frontend HTTP proxy to handle all incoming requests. It
had to be very fast with a low overhead, but packed with many features
(connecting to external databases, having a XML-RPC interface, etc.),
so we thought Python was perfect for this task. It's been running in
production for a few weeks, handling several millions requests each
day, in both regular HTTP and HTTPS.
> b. what other options did you consider and why have chosen gevent? /
> if you switched from another network library, why?
Our application was initially powered by Tornado, as gevent didn't
exist back when we started. Unfortunately, we had several issues with
it:
- Tornado had a few important bugs related to epoll
- we had to patch Tornado quite a lot to do what we wanted (including
to fix really bad performance issues)
- our own code was in async mode, which makes it more difficult to
read and write that simple synchronous mode (à la gevent)
So we ditched Tornado a few months ago and rewrote our application to
use a library that patches Python's socket code to make it async. We
considered 3 libraries: eventlet, gevent and syncless. Since all of
them patch Python's socket library, we could easily switch from one to
another.
We first tried gevent and eventlet, and soon figured out they both had
the very same annoying bug. I reported it on both bug trackers, and in
the meantime tried syncless, which didn't have that specific bug.
syncless is quite promising, with nice unique features (can use Python
stackless or greenlet, libevent or libev, written mostly in Cython to
be very fast), but it really lacks a community (mailing-list, IRC
channel). The pace of development also seems to have really slowed
down lately, I even wonder if it's still alive. I also experienced
severe memory leaks as well as a few other bugs, which led me to
switch back to eventlet/gevent. I'll keep an eye on syncless, though,
as it has potential to be a great library.
Eventlet marked my bug as wontfix, and gevent had it fixed quickly, so
I went for gevent. gevent's API also seems nicer. The only downside
compared to eventlet, as far as I can tell, is that there's no easy
way to run processes asynchronously. I ended up launching my processes
in a separate Python thread to make sure it doesn't block the rest of
my code.
Denis was very helpful all along, and very quick to investigate and
fix bugs I reported to him. Talking directly to him on IRC is a big
plus, too, as I tend to favor this kind of fast communication over
mailing-lists or plain emails.
> c. are you planning to use gevent in future projects, if so what kind
> of tasks will you use it for?
There's no doubt I'd use gevent again if I ever need to. I'm actually
glad to see Python can be used to write very fast daemons, as I'm
reluctant to write code in C unless I really have to. Thanks to
gevent, I don't think it'll happen anytime soon.
--
Cyril
Nicholas Piël
Thanks for sharing!
On Aug 14, 2010, at 12:56 PM, Cyril B. wrote:
...
> - Tornado had a few important bugs related to epoll
> - we had to patch Tornado quite a lot to do what we wanted (including
> to fix really bad performance issues)
Would you be so kind to explain these issues a little bit more or point to the relevant tickets?
> We first tried gevent and eventlet, and soon figured out they both had
> the very same annoying bug. I reported it on both bug trackers, and in
> the meantime tried syncless, which didn't have that specific bug.
Which bug was this?
Cheers,
Nicholas
Cyril B.
> > - Tornado had a few important bugs related to epoll
> > - we had to patch Tornado quite a lot to do what we wanted (including
> > to fix really bad performance issues)
>
> Would you be so kind to explain these issues a little bit more or point to the relevant tickets?
- performance wise, using string buffers was awful as soon as the
buffers started growing more than few MB. I fixed it using a cStringIO
(which also limits memory fragmentation), but the easiest way is a
list of strings. I also added a method to stop reading a socket when
its buffer exceeds a certain size.
- features wise, I added a few things. For instance, I tweaked
read_bytes to have my callback called as soon as it received
something. I could also set num_bytes to -1 to have it return
"forever".
- as far as bugs are concerned, here are a few I remember:
- in iostream, I sometimes had a recursion limit exception when my
callback called read_bytes (or read_until), and so on. It happened
with HTTP chunked encoding with very small chunks. I used a trick
where the callback would unwind the stack when the recursion reached a
specific number of occurrences (before the recursion limit exception
would be raised)
- ioloop had nasty epoll bugs in it. One of them is that sometimes
self._impl.unregister would fail with an IOError (no such file
directory). I've also seen KeyError exceptions in the 'while
self._events' loop. gevent had similar issues
I haven't reported these issues, neither sent my patches upstream. I
was planning to try to merge my changes into trunk when my application
would be finished, but I finally decided to switch to a pseudo-
synchronous library (gevent/eventlet).
Honestly, all of this wasn't the main reason I switched. The ease of
use of synchronous libraries was my primary concern. Also, your very
own benchmarks showed that gevent performed better than Tornado, which
was a small plus :)
> > We first tried gevent and eventlet, and soon figured out they both had
> > the very same annoying bug. I reported it on both bug trackers, and in
> > the meantime tried syncless, which didn't have that specific bug.
>
> Which bug was this?
http://bitbucket.org/which_linden/eventlet/issue/56/deadblock-on-socket-close
Actually, syncless' documentation recommends against doing that, and
says it's implementation-dependent. It worked for me (Linux/libevent).
--
Cyril
Season Lee
- as far as bugs are concerned, here are a few I remember:
- in iostream, I sometimes had a recursion limit exception when my
callback called read_bytes (or read_until), and so on. It happened
with HTTP chunked encoding with very small chunks. I used a trick
where the callback would unwind the stack when the recursion reached a
specific number of occurrences (before the recursion limit exception
would be raised)
--
Cyril