does gevent work with Flask and the uWSGI gevent loop?

[David Montgomery]

Hi,

I am using uWSGI 2.0.1 with flask and requests. 

Is the below code blocking when using the gevent loop?  How do I do a yield in flask? and and can the requests lib be monkey patched?

Thanks



from gevent import monkey; monkey.patch_all()
import gevent
import gevent.socket
from flask import render_template,flash, redirect,request, url_for, g, session,flash, jsonify,Response,jsonify,make_

response,send_file
from flask.ext.login import login_user, logout_user, current_user, login_required, UserMixin,confirm_login,fresh_login_required
from flask.ext import restful
from flask.ext.restful import reqparse, abort, Api, Resource
import sys
from yaml import load, Loader
try:
   import uwsgi
except:
   pass
import traceback
import sys
import os
import requests
try:
    from app import app #, login_manager
except:
    print 'ImportError: No module named app'
   

def fetch_url(url):
    r = requests.get(url)
    return r.status_code


@app.route('/test')
def test():
    """
    when run with the uWSGI gevent loop ....while the web page is being fetched
    other requests will be accepted.  This increased thoughput. THe is one proccess per core.
    Need more scale..add more cores....................
    QED.....
    """
    url = 'http://www.cnn.com'
    jt = gevent.spawn(fetch_url,url)
    jt.join()
    status = jt.value
   
    return render_template("test.html",title = 'Home', status=status)

[Shahaf Abileah]

I don't have an exact answer, but here's some info you may find useful...

Flask supports yields for streaming responses.  http://flask.pocoo.org/docs/patterns/streaming/  I've done this successfully.  One gotcha is that you may have wsgi filters that buffer responses before sending them to the browser (e.g. I had a this issue with a gzip filter).  Another gotcha is that some browsers don't start processing responses until they get ~256 characters, so it helps to begin the response with that many whitespace characters.

Separately, I've tried spawning greenlets from a flask request and not waiting for them to finish (with a join() call) and that did not work.  I don't know exactly why.  It seems that Flask doesn't wait for spawned stuff to finish.

I've been using Celery for background tasks, and issuing a long-poll ajax request to wait for the task to finish.  It's a more complicated setup, but it works.

--S

--
You received this message because you are subscribed to the Google Groups "gevent: coroutine-based Python network library" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gevent+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

http://twitter.com/shahafabileah

[Roberto De Ioris]

> Hi,
>
> I am using uWSGI 2.0.1 with flask and requests.
>
> Is the below code blocking when using the gevent loop? How do I do a
> yield
> in flask? and and can the requests lib be monkey patched?
>
> Thanks

It is blocking as you are using a non-gevent friendly 'requests'
implementation, try this:

https://github.com/kennethreitz/grequests

as a side note, i would probably remove the gevent.spawn() and will
directly call grequests in the main request greenlet.

Regarding yield, i suppose you want to stream the response in various
chunks right ?


--
Roberto De Ioris
http://unbit.it

[David Montgomery]

Assume I dont use requests or grequests(which has not been mainated so it seems)....

I will use good ole fashion monkey patch of urllib2.  Will that work?  Just wondering if flask supports the uWSGI gevent loop.

[David Montgomery]

PS  I will have 

uwsgi_buffering off;

in the nginx config.  Per uWSGI docs that helps with buffering.  I dont need streaming.  Just want to increase concurrency of rest requests with gevent and flask via uWSGI. 

On Thu, Jun 5, 2014 at 5:56 AM, Shahaf Abileah <sha...@gmail.com> wrote:

[Roberto De Ioris]

> PS I will have
>
> uwsgi_buffering off;
>
> in the nginx config. Per uWSGI docs that helps with buffering. I dont
> need streaming. Just want to increase concurrency of rest requests with
> gevent and flask via uWSGI.

there is no problem in flask by itself, you need to check your db adapters
(if any) are gevent-friendly (as well as other blocking modules you could
have in your code)

regarding urllib2, yes, once monkeypatched it works with gevent flawlessly

[David Montgomery]

So..but does gevent loop work with Flask then?  Maybee best posted in Flask or uWSGI forums. 

Because in flask I will spawn a urllib2 requests and return as follows.  I have used bottle with great success but dont know about the stack of uWSGI/Flask/Gevent

render_template("test.html",title = 'Home', status=status)

There is no yield I gather in Flask of a web page

[Matthias Urlichs]

Hi,

David Montgomery:

> There is no yield I gather in Flask of a web page
>

? Why would you need one?

Flask passes the return value of the template renderer up to WSGI,
packaged in a Response object.
If that is an iterator, WSGI is supposed to stream it.

With Jinja2 you call .stream() or .generate() on the template
instead of .render().

--
-- Matthias Urlichs

[Matias Guijarro]

Hi,

If you have the choice, prefer bottle instead of flask with gevent.

The same example with bottle would work perfectly.

Cheers,
Matias.

[INADA Naoki]

uWSGI have --gevent-monkey-patch option.
It does gevent.monkey_patch.patch_all() at the beginning.

requests is compatible with gevent.
But DNS lookup may be blocked.

gevent's default DNS resolver is threadpool based.
You should use uWSGI's --enable-thread option to use default DNS resolver.
You can also choose nonblocking DNS resolver with
GEVENT_RESOLVER=ares environment variable.

http://www.gevent.org/whatsnew_1_0.html#new-dns-resolver

> --
> You received this message because you are subscribed to the Google Groups
> "gevent: coroutine-based Python network library" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gevent+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
INADA Naoki <songof...@gmail.com>