Is it possible to execute code in main thread when called from another thread?

[André Cruz]

Hello.

I'm running an application that uses the Zookeeper client library which is written in C. Even though I execute the monkey_patch() as early as I can, when the Zookeeper library is initialized it still spawns a separate thread for its processing (probably because it is written in C the call to spawn a new thread bypasses the python stdlib).

This Zookeeper library thread makes some callbacks to my code, but since it is a different thread it has problems with gevent whenever the hub is called. For example, even the logging poses problems:

Traceback (most recent call last):

  File "/servers/disco/site/utils/zoo.py", line 28, in watchdog

    log.warn('Got event: TYPE {0} STATE {1} PATH {2}'.format(TYPE_NAME_MAPPING[event_type], STATE_NAME_MAPPING[conn_state], path))

  File "/usr/lib/python2.7/logging/__init__.py", line 1152, in warning

    self._log(WARNING, msg, args, **kwargs)

  File "/usr/lib/python2.7/logging/__init__.py", line 1258, in _log

    self.handle(record)

  File "/usr/lib/python2.7/logging/__init__.py", line 1268, in handle

    self.callHandlers(record)

  File "/usr/lib/python2.7/logging/__init__.py", line 1308, in callHandlers

    hdlr.handle(record)

  File "/usr/lib/python2.7/logging/__init__.py", line 746, in handle

    self.acquire()

  File "/usr/lib/python2.7/logging/__init__.py", line 697, in acquire

    self.lock.acquire()

  File "/usr/lib/python2.7/threading.py", line 128, in acquire

    rc = self.__block.acquire(blocking)

  File "_semaphore.pyx", line 111, in gevent._semaphore.Semaphore.acquire (gevent/gevent._semaphore.c:2547)

  File "/servers/python-environments/discosite/local/lib/python2.7/site-packages/gevent/hub.py", line 373, in switch

    return greenlet.switch(self)

greenlet.error: cannot switch to a different thread

Is is possible to invoke the GEvent library and execute code in the main thread from this separate thread? This way I would just call this code and the main thread would do the logging with no problems. Btw, I'm using the beta 2 of Gevent 1.0.

Thank you and best regards,

André Cruz

[Denis Bilenko]

On Wed, May 9, 2012 at 6:37 PM, André Cruz <andre...@co.sapo.pt> wrote:
> Is is possible to invoke the GEvent library and execute code in the main
> thread from this separate thread? This way I would just call this code and
> the main thread would do the logging with no problems. Btw, I'm using the
> beta 2 of Gevent 1.0.

Yes, use the async watcher. See gevent/threadpool.py for how to use it.

[Matthias Urlichs]

Hi,

Andr� Cruz:

> Is is possible to invoke the GEvent library and execute code in the main
> thread from this separate thread? This way I would just call this code and
> the main thread would do the logging with no problems. Btw, I'm using the
> beta 2 of Gevent 1.0.
>

I would use a separate process. Get the two to talk to each other via RPyC
(there is a gevent-compatibility layer for it which I use myself and which
I *really* should package up, along with the Twisted stuff I'm using *sigh*)
or some other "transparent" Python RPC library.

Alternately, do not monkeypatch anything.

I don't think there's a safe way to use gevent+monkey-patching *and*
native threads in the same process.

--
-- Matthias Urlichs

[Jim Fulton]

There's an example, specifically for ZooKeeper, here:

http://svn.zope.org/zc.resumelb/trunk/src/zc/resumelb/zk.py?rev=125635&view=auto

(Search for the comment: "Set up notification of address changes".)

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://www.dublinstore.com/

[André Cruz]

Thanks, all. Problem solved.

Best regards,

André

[Neilen Marais]

On May 9, 7:04 pm, Jim Fulton <j...@zope.com> wrote:

> On Wed, May 9, 2012 at 11:09 AM, Denis Bilenko <denis.bile...@gmail.com> wrote:

> > Yes, use the async watcher. See gevent/threadpool.py for how to use it.
>
> There's an example, specifically for ZooKeeper, here:
>

> http://svn.zope.org/zc.resumelb/trunk/src/zc/resumelb/zk.py?rev=12563...

>
> (Search for the comment: "Set up notification of address changes".)

Is there any documentation describing how this works? Looking at the
source of get_hub() it looks like a new hub is created for each
thread. In which thread context are the greenlets run? I'm new to
gevent, so please enlighten me if I am misunderstanding.

Thanks
Neilen