gevent 1.0b1 available

[Denis Bilenko]

Hi.

I'm happy to announce the first beta release of gevent 1.0.

Changes compared to 1.0a3:

- Removed "link to greenlet" feature of Greenlet.
- Added thread pool: gevent.threadpool.ThreadPool.
- Added thread pool-based resolver. Enable with GEVENT_RESOLVER=thread.
- Added UDP server: gevent.server.DatagramServer
- A "configure" is now run on libev. This fixes a problem of 'kqueue'
not being available on Mac OS X.
- Gevent recognizes some environment variables now:
- GEVENT_BACKEND allows passing argument to loop, e.g.
"GEVENT_BACKEND=select" for force select backend
- GEVENT_RESOLVER allows choosing resolver class.
- GEVENT_THREADPOOL allows choosing thread pool class.
- Added new examples: portforwarder, psycopg2_pool.py, threadpool.py,
udp_server.py
- Fixed non-embedding build. To build against system libev, remove or
rename 'libev' directory. To build against system c-ares, remove or
rename 'c-ares'.
- Fixed a number of issues in pywsgi, core, socket

See the full changelog here:
https://bitbucket.org/denis/gevent/raw/tip/changelog.rst
In the issue tracker:
http://code.google.com/p/gevent/issues/list?can=1&q=Milestone%3DRelease1.0b1

Download the release tarballs and windows installers from
http://code.google.com/p/gevent/downloads/list

Thanks to Ralf Schmitt who is co-maintaining gevent with me.

Thanks to Örjan Persson, Shaun Lindsay, Shaun Cutts, David
LaBissoniere, tck420 for sending in patches.

(If I forgot you please let me know.)

Cheers,
Denis.

[Benoit Chesneau]

Really nice list of features! What about the compatibility with py3
right now? Are there anything still needed for that ? Where ?

- benoît

[Matt Joiner]

Has anyone tested this with py3?

[Ralf Schmitt]

Benoit Chesneau <bche...@gmail.com> writes:

>
> Really nice list of features! What about the compatibility with py3
> right now? Are there anything still needed for that ? Where ?
>

It's currently not compatible with python 3. I've started working on
that. The current code already ships with six as gevent.six
(http://pypi.python.org/pypi/six). Some of the modules have already been
ported (or at least the tests do pass). If time permits I will continue
to work on py 3 compatibility bit by bit.

I didn't use any of the forks that add python 3 compatibility. Some of
them did use 2to3, which we don't want to use. Some of them were
branched from rather old versions of gevent (at least at the time I did
look at them). If you think I'm missing some important work, please
speak up!

I'm doing my work in https://github.com/schmir/gevent/commits/wip-py3
and when I consider a module finished, I commit to the main branch in
the bitbucket repo. We decided not to use a dedicated branch for that
work.

Help would be appreciated!

--
Cheers
Ralf

[Benoit Chesneau]

Thanks, I will have a closer look on it during the week. For gunicorn
this is a must-have feature and would allow us to ship it for py3 with
async support. Like on py2.7 . I'm following the same path anyway,
there won't be a special branch or repo for py3 in gunicorn.

Thanks for your work anyway,

- benoît

[Executier Godlike]

> Really nice list of features! What about the compatibility with py3
> right now?

At least socket module doesn't work a few weaks ago :(.

>>> import gevent.socket
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "gevent/socket.py", line 233, in <module>
class socket(object):
File "gevent/socket.py", line 546, in socket
for _m in set(__socket__._socketmethods) - set(locals()):
AttributeError: 'module' object has no attribute '_socketmethods'

I tried to fix this and other "trivial" errors, but didn't succeed.

--
Alexandre

[Denis Bilenko]

On Wed, Jan 11, 2012 at 1:28 AM, Executier Godlike <sp...@messir.net> wrote:
>>>> import gevent.socket
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File "gevent/socket.py", line 233, in <module>
>    class socket(object):
>  File "gevent/socket.py", line 546, in socket
>    for _m in set(__socket__._socketmethods) - set(locals()):
> AttributeError: 'module' object has no attribute '_socketmethods'

You probably have a file called socket.py[c] lying in your PYTHONPATH somewhere.

find / -iname 'socket.py*'

And you're not using 1.0b1 (line numbers don't match).

[Sean Talts]

Thanks! This is awesome :) Out of curiosity, have you switched
sitesupport.com to it? We'll start running a service on this new beta
this week (lot18.com).

Thanks,
Sean

On Jan 10, 1:38 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:

[Executier Godlike]

I'm using python3.3 trunk (tip).
I think this is the source of the problem.

There is no stalled socket.py*.

[Ralf Schmitt]

Denis Bilenko <denis....@gmail.com> writes:

> Hi.
>
> I'm happy to announce the first beta release of gevent 1.0.
>
> Changes compared to 1.0a3:
>
> - Removed "link to greenlet" feature of Greenlet.

If you have been using this feature, please add an assert to make your
life easier, like in:
https://github.com/schmir/gevent/commit/4f2fce96feaefd84bd498b30d957b65e606bd054

I'll do that later on the bitbucket repo with some extra fixes to the testsuite
from https://github.com/schmir/gevent/tree/wip-fixes

--
Cheers
Ralf

[Denis Bilenko]

On Wed, Jan 11, 2012 at 4:16 AM, Sean Talts <xit...@gmail.com> wrote:
> Thanks!  This is awesome :)  Out of curiosity, have you switched
> sitesupport.com to it?

No, we still on 1.0a3, but plan on upgrading soon.

> We'll start running a service on this new beta this week (lot18.com).

On Wed, Jan 11, 2012 at 9:16 AM, Ralf Schmitt <ra...@systemexit.de> wrote:
> If you have been using this feature, please add an assert to make your
> life easier, like in:
> https://github.com/schmir/gevent/commit/4f2fce96feaefd84bd498b30d957b65e606bd054

I agree, we should check for that. Only let's raise TypeError and
let's do hasattr(..., '__call__') instead of callable() to be
future-compatible with python3.

[Ralf Schmitt]

Denis Bilenko <denis....@gmail.com> writes:

> I agree, we should check for that. Only let's raise TypeError and
> let's do hasattr(..., '__call__') instead of callable() to be
> future-compatible with python3.

callable is back in 3.2, if we care about 3.0 or 3.1 we should use
gevent.six.callable. I plan to commit the changes today.

[Denis Bilenko]

On Wed, Jan 11, 2012 at 6:41 PM, Ralf Schmitt <ra...@systemexit.de> wrote:
>> I agree, we should check for that. Only let's raise TypeError and
>> let's do hasattr(..., '__call__') instead of callable() to be
>> future-compatible with python3.
>
> callable is back in 3.2, if we care about 3.0 or 3.1 we should use
> gevent.six.callable. I plan to commit the changes today.

Should we care about 3.0 and 3.1?

Also any idea why is callable() in six defined as

def callable(obj):
return any("__call__" in klass.__dict__ for klass in type(obj).__mro__)

instead of

def callable(obj):
return hasattr(obj, '__call__')

[Denis Bilenko]

On Fri, Jan 13, 2012 at 3:25 AM, Denis Bilenko <denis....@gmail.com> wrote:
>> callable is back in 3.2, if we care about 3.0 or 3.1 we should use
>> gevent.six.callable. I plan to commit the changes today.
>
> Should we care about 3.0 and 3.1?
>
> Also any idea why is callable() in six defined as
>
>    def callable(obj):
>        return any("__call__" in klass.__dict__ for klass in type(obj).__mro__)
>
> instead of
>
>    def callable(obj):
>        return hasattr(obj, '__call__')

Also I think gevent/six.py will be removed or trimmed eventually,
since we seem to need only a small fraction of it. So far, we've used
this in gevent package:

- integer_types
- string_types
- callable (not really, since it's back in 3.2)
- _meth_self
- xrange
- PY3

That is, it's OK to use it while porting is in progress, but I don't
want gevent users to depend on it too, since I don't want to maintain
it beyond what's needed for gevent.

[Ralf Schmitt]

Denis Bilenko <denis....@gmail.com> writes:

>> Should we care about 3.0 and 3.1?

I think supporting only 3.2 and up would be fine.

>>
>> Also any idea why is callable() in six defined as
>>
>>    def callable(obj):
>>        return any("__call__" in klass.__dict__ for klass in type(obj).__mro__)
>>
>> instead of
>>
>>    def callable(obj):
>>        return hasattr(obj, '__call__')

I don't know.

>
> Also I think gevent/six.py will be removed or trimmed eventually,
> since we seem to need only a small fraction of it. So far, we've used
> this in gevent package:
>
> - integer_types
> - string_types
> - callable (not really, since it's back in 3.2)
> - _meth_self
> - xrange
> - PY3
>
> That is, it's OK to use it while porting is in progress, but I don't
> want gevent users to depend on it too, since I don't want to maintain
> it beyond what's needed for gevent.

[six.exec_ and six.advance_iterator in the tests]

why not make it a dependency then? gevent already depends on the
greenlet module and a pure python dependency doesn't hurt IMHO.

trimming unused functionality from the six module is what makes you
maintain it; it's a bad idea IMHO. if you don't do that, you only need
to upgrade it once in a while and forget about it. it's only 350 lines
of code.
Also remember that porting has just begun; at the end a larger fraction
of functionality might be in use.

btw is there any reason we don't add the greenlet module to the
dependencies and let easy_install/pip take care of it?

--
Cheers
Ralf

[Denis Bilenko]

On Fri, Jan 13, 2012 at 4:31 AM, Ralf Schmitt <ra...@systemexit.de> wrote:
> I think supporting only 3.2 and up would be fine.

OK.

> [six.exec_ and six.advance_iterator in the tests]

Yeah, I noticed. Tests can have dependencies that gevent itself does
not have. It's not critical there.

> why not make it a dependency then? gevent already depends on the
> greenlet module and a pure python dependency doesn't hurt IMHO.

Might create more work for packagers.

> trimming unused functionality from the six module is what makes you
> maintain it; it's a bad idea IMHO. if you don't do that, you only need
> to upgrade it once in a while and forget about it. it's only 350 lines
> of code.

Every line counts. If you keep it there, people might start depending
on it and that's when it becomes painful to remove so you end up
maintaining it.

> Also remember that porting has just begun; at the end a larger fraction
> of functionality might be in use.

OK, let's keep it for now until it's obvious what we need exactly.

> btw is there any reason we don't add the greenlet module to the
> dependencies and let easy_install/pip take care of it?

I thought we do that:
https://bitbucket.org/denis/gevent/src/b045e5f368f4/setup.py#cl-261

or did you mean something else?

Cheers,
Denis.

[Ralf Schmitt]

Denis Bilenko <denis....@gmail.com> writes:

>> btw is there any reason we don't add the greenlet module to the
>> dependencies and let easy_install/pip take care of it?
>
> I thought we do that:
> https://bitbucket.org/denis/gevent/src/b045e5f368f4/setup.py#cl-261
>
> or did you mean something else?

That's exactly what I meant. Seems like I need more sleep!

--
Cheers
Ralf

[Floris Bruynooghe]

On 12 January 2012 22:00, Denis Bilenko <denis....@gmail.com> wrote:
> On Fri, Jan 13, 2012 at 4:31 AM, Ralf Schmitt <ra...@systemexit.de> wrote:
>> why not make it a dependency then? gevent already depends on the
>> greenlet module and a pure python dependency doesn't hurt IMHO.
>
> Might create more work for packagers.

Actually, packagers (the ones creating GNU/Linux distributions)
generally frown more on bundling packages as that is what creates
extra work for them. They don't like bundling since it means they
need to fix lots of things instead of just one place. A dependency is
much nicer for them.

Direct users however are a different matter, but most probably use pip
by now. Not to say bundling isn't tempting for this scenario. Maybe
ideally setup.py detect if it's available and only uses the bundled
version if it's not available?

Regards,
Floris

--
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

[AnilG]

> It's currently not compatible with python 3.

> I didn't use any of the forks that add python 3 compatibility.

> I'm doing my work in https://github.com/schmir/gevent/commits/wip-py3

Ralf, I've been hanging around since 2010 trying to help gevent move
to Py3k.
My last attempt came a cropper and since then I've been short on time
http://groups.google.com/group/gevent/browse_thread/thread/e4bcb639e9b239e5

I've also tried to establish where help may be wanted
http://groups.google.com/group/gevent/browse_thread/thread/987b4bf2738b34fc

> Help would be appreciated!

If you can indicate what help you want and how it should be delivered
I'll see if I can comply.
I just want to avoid doing work that gets left out in the cold.

[Adam LeVasseur-Arribas]

Right now our application works great with gunicorn-0.13.1 +
gevent-1.0a2, but upgrading to 1.0b1 causes errors as soon as loading
the app:

2012-01-13 12:26:29,412 - ERROR - base.py:handle_uncaught_exception:
209 - Internal Server Error: /

Traceback (most recent call last):

File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/usr/local/www/player-site/ihr/django/views/decorators.py",
line 63, in wrapper
response = render_html(request, tpl, context=output,
mimetype=MIME_HTML)
File "/usr/local/www/player-site/ihr/django/views/shortcuts.py",
line 16, in render_html
response = render_to_string(request, template, context)
File "/usr/local/www/player-site/ihr/django/views/shortcuts.py",
line 13, in render_to_string
return shortcuts.render_to_string(template, context,
context_instance)
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
coffin/template/loader.py", line 50, in render_to_string
template = get_template(template_name)
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
coffin/template/loader.py", line 24, in get_template
return env.get_template(template_name)
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
jinja2/environment.py", line 719, in get_template
return self._load_template(name, self.make_globals(globals))
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
jinja2/environment.py", line 693, in _load_template
template = self.loader.load(self, name, globals)
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
jinja2/loaders.py", line 121, in load
bucket = bcc.get_bucket(environment, name, filename, source)
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
jinja2/bccache.py", line 184, in get_bucket
self.load_bytecode(bucket)
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
jinja2/bccache.py", line 293, in load_bytecode
code = self.client.get(self.prefix + bucket.key)
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
memcache.py", line 862, in get
return self._get('get', key)
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
memcache.py", line 814, in _get
server, key = self._get_server(key)
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
memcache.py", line 337, in _get_server
if server.connect():
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
memcache.py", line 1077, in connect
if self._get_socket():
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
memcache.py", line 1094, in _get_socket
s.connect(self.address)
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
gevent/socket.py", line 349, in connect
return EAGAIN
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
gevent/socket.py", line 653, in getaddrinfo
name = gethostname()
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
gevent/hub.py", line 361, in _get_resolver
if switch is not None:
File "/usr/local/www/player-site/ve/lib/python2.7/site-packages/
gevent/resolver_ares.py", line 24, in __init__
self.fork_watcher = hub.loop.fork(ref=False)
TypeError: fork() takes no keyword arguments

[Denis Bilenko]

On Fri, Jan 13, 2012 at 11:31 PM, Adam LeVasseur-Arribas
<adam....@gmail.com> wrote:
> Right now our application works great with gunicorn-0.13.1 +
> gevent-1.0a2, but upgrading to 1.0b1 causes errors as soon as loading
> the app:
>

>    self.fork_watcher = hub.loop.fork(ref=False)
> TypeError: fork() takes no keyword arguments

Somehow you still seem to be using old core.so (from 1.0a2). Not sure
how it happened, but I'm pretty sure that wiping out gevent completely
and the doing fresh install of 1.0b1 would solve the issue.

[bdew]

>> Also any idea why is callable() in six defined as

>>
>>    def callable(obj):
>>        return any("__call__" in klass.__dict__ for klass in type(obj).__mro__)
>>
>> instead of
>>
>>    def callable(obj):
>>        return hasattr(obj, '__call__')

I don't know.

Just for the record - because CPython doesn't check the instance __dict__ for many magic methodes, including __call__. http://docs.python.org/reference/datamodel.html#special-method-lookup-for-new-style-classes

[Adam LeVasseur-Arribas]

I tried "pip uninstall gevent greenlet" and reinstalled, no dice.
Any ideas?
Thanks!

On Jan 16, 1:53 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:
> On Fri, Jan 13, 2012 at 11:31 PM, Adam LeVasseur-Arribas
>

[Fantix King]

Hi Ralf,

Thanks for the work!

I see your fixes are now under https://github.com/schmir/gevent-old-mirror/commits/wip-py3.

Do you have any plan to go on with Gevent py3k? Which repo/branch shall I start from

if I would like to help with this?

BR,

Fantix

[Ralf Schmitt]

Fantix King <fanti...@gmail.com> writes:

> Hi Ralf,
>
> Thanks for the work!
>
> I see your fixes are now under
> https://github.com/schmir/gevent-old-mirror/commits/wip-py3.
> Do you have any plan to go on with Gevent py3k?

probably after 1.0 is released. lack of time is the limiting factor here...

> Which repo/branch shall I start from if I would like to help with
> this?

I'd just use git master.

[Fantix King]

Hi Ralf (and Denis),

Thanks for the reply!

I don't know how much will an RC change, but I've already started

from scratch to fix it on git master according to the comments in

Issue 38 by Denis, which is generally to support both 2 and 3 with

single source. I've also "ported" part of your fix 1 year ago.

I didn't reuse the work to split into socket2.py and socket3.py, instead

I made an adaptable _fileobject in Python 3 for the new I/O objects.

Here's the Travis report link:

https://travis-ci.org/fantix/gevent/builds/3413038

It is more or less working in Python 3.3, but still lot's of things to fix.

In Python 2, I broke the test for the fix to Issue 116, because I changed

the implementation of HTTP header parsing from "mimetools" module

to "email" module in pywsgi.py. Is it suitable to make this change please?

Please advice. :)

One thing to mention is that, with Cython built with Python 3, sometimes

Gevent doesn't compile correctly. I had to force using Cython built with

Python 2.

And, I didn't yet add the "six" module back completely, just added the

necessary "six" code to gevent/hub.py and greentest/six.py according to

previous discussions. If you feel like that it is suitable to fully depend on

"six", I can make the change.

BR,

Fantix

[Ralf Schmitt]

Fantix King <fanti...@gmail.com> writes:

> In Python 2, I broke the test for the fix to Issue

> 116<https://github.com/SiteSupport/gevent/issues/116>,

> because I changed
> the implementation of HTTP header parsing from "mimetools" module
> to "email" module in pywsgi.py. Is it suitable to make this change please?
> Please advice. :)

I wasn't very happy with the fix for the above issue and would rather
have replaced the whole implementation. I didn't do it because 1.0
should happen rather soon and I think it might have broken some code
(gevent-socketio probably...). If the email module's API works like
mimetools's API, I would rather replace it.

>
> One thing to mention is that, with Cython built with Python 3, sometimes
> Gevent doesn't compile correctly. I had to force using Cython built with
> Python 2.

that's rather strange, you should probably report that to the cython
developers.

>
> And, I didn't yet add the "six" module back completely, just added the
> necessary "six" code to gevent/hub.py and greentest/six.py according to
> previous discussions. If you feel like that it is suitable to fully depend
> on
> "six", I can make the change.
>

I'd probably just depend on six, you already seem to do so in the tests.
but then denis removed it some time ago, he might disagree with me here.

It might also be a good idea to drop support for python 2.5 since python
2.6 allows 'except Exception as err'.

btw, your code seems to live in the issue38 branch (see
https://github.com/fantix/gevent/commits/issue38).

The very first commit on that branch is rather large, and has a rather
short log message!

Denis, what's your take on this?

--
Cheers
Ralf

[Fantix King]

On Thu, Nov 29, 2012 at 5:04 PM, Ralf Schmitt <ra...@systemexit.de> wrote:

Fantix King <fanti...@gmail.com> writes:

> One thing to mention is that, with Cython built with Python 3, sometimes
> Gevent doesn't compile correctly. I had to force using Cython built with
> Python 2.

that's rather strange, you should probably report that to the cython
developers.

Will do - I'll try to collect some details.

 

btw, your code seems to live in the issue38 branch (see
https://github.com/fantix/gevent/commits/issue38).

The very first commit on that branch is rather large, and has a rather
short log message!

I have actually squash-merged into issue38_2 with another big patch. I would

like to commit everything back to master in one changeset with commit

message in detail when the progress comes to some milestone. :)

Thanks for the comments.

BR,

Fantix