First impressions of gevent & introduction to rust, a web bot corrosive
Tim McNamara
$ curl http://rust.io/corrode/
That URL will attempt to swamp the client's URL request queue, eventually exhausting concurrency available in there app.
I'm very interested in hearing people's views on whether this makes for geventish code. In particular, I would like to swap out the WSGI server with my own HTTP response parser/request generator. I would like to stream malformed HTTP response headers to implement the slowloris. Streaming headers is prohibited by PEP-333.
I thought I would share some of the thoughts that I had as a new-comer to the project:
- IPython consistently crashed on me
- I wasn't quite certain whether monkey patching really works, e.g. will this work in a gevent compatible way?
from gevent import monkey; monkey.patch_all();
import gevent
import requests
requests.get('http://gevent.org/')
Or should I do something like this?
gevent.spawn(requests.get, 'http://gevent/org')
- It sounds silly, but I really wanted to use a callback-like style for
long request/responses, e.g. provide callbacks for when timeouts
occur, a line is received, connection is dropped. I guess I should
just use normal try:/except:
- I found some of the documentation a little difficult to understand.
For example, the "Implementing Servers" page contains this
paragraph:
The pywsgi.WSGIServer does not have these limitations.
In addition, gunicorn is a stand-alone server that supports
gevent. Gunicorn has its own HTTP parser but can also use
gevent.wsgi module.
This made me think, why is gunicorn being mentioned? What
benefit is there in using gunicorn behind NGINX when I could
just have gevent being NGINX?
The API docs seem to be mixed in with the introduction quite
a lot. This made it fairly hard to understand
In the end, the web.py example was probably the easiest. That is where I have taken inspiration from with rust. I have gained quite a bit of confidence with gevent, but I'm still fairly unsure what the 'gevent way' is.
Tim
[0] https://github.com/timClicks/rust
Denis Bilenko
> In other to learn gevent, I have built a project for attacking web crawlers
> which disobey robots.txt: rust[0].
Interesting project, thanks for sharing!
> I thought I would share some of the thoughts that I had as a new-comer to
> the project:
>
> - IPython consistently crashed on me
Is this related to gevent? Have you got a backtrace?
> - I wasn't quite certain whether monkey patching really works, e.g.
> will this work in a gevent compatible way?
>
> from gevent import monkey; monkey.patch_all();
> import gevent
> import requests
>
> requests.get('http://gevent.org/')
>
> Or should I do something like this?
>
> gevent.spawn(requests.get, 'http://gevent/org')
The former will execute the request in the current greenlet. The
latter creates a new greenlet and schedules it's execution as soon as
the current greenlet yields. It does not actually run requests.get
immediately.
>
> - It sounds silly, but I really wanted to use a callback-like style for
> long request/responses, e.g. provide callbacks for when timeouts
> occur, a line is received, connection is dropped. I guess I should
> just use normal try:/except:
You can actually use callback-like style with gevent, at the core it's
just an event loop. The green thread way is easier though.
> - I found some of the documentation a little difficult to understand.
> For example, the "Implementing Servers" page contains this
> paragraph:
>
> The pywsgi.WSGIServer does not have these limitations.
> In addition, gunicorn is a stand-alone server that supports
> gevent. Gunicorn has its own HTTP parser but can also use
> gevent.wsgi module.
>
> This made me think, why is gunicorn being mentioned? What
> benefit is there in using gunicorn behind NGINX when I could
> just have gevent being NGINX?
You could, if you don't need any of gunicorn features (e.g. multiple
process workers).