Using Janus to wait for a coroutine to finish in a standard python function

[daniel.jam...@gmail.com]

I am encouraging a common problem when mixing sync and async code: calling and waiting for an async function to finish from a standard synchronous function.

def notification(self, news):
result = await process(news) # <- not valid Python

Unfortunately, I cannot make the synchronous function async because it an external library’s function that is calling into my code. Furthermore, this for a desktop application that already has an event loop running, so it’s impossible to do “run_until_complete” on the asyncio event loop.

I wanted to ask for advice here to see if anyone has used Janus to solve this problem?

I was think of starting a thread running a Janus queue when my application starts. When I need to wait for a Coroutine to finish in a synchronous function, my idea was to put that into the queue and wait for it. It will be processed on the other thread and block the calling function until completed.

I really cannot see another way around this without nesting event loops, which I would rather not do, sounds scary.

Any ideas?

[Gustavo Carneiro]

On Tue, 5 Mar 2019 at 18:21, <daniel.jam...@gmail.com> wrote:

I am encouraging a common problem when mixing sync and async code: calling and waiting for an async function to finish from a standard synchronous function.

def notification(self, news):
    result = await process(news) # <- not valid Python

It sounds like you just need to use asyncio.run_coroutine_threadsafe(), see:

https://docs.python.org/3/library/asyncio-dev.html#concurrency-and-multithreading

Your example would become:

---------

loop = asyncio.get_event_loop()

def notification(self, news):

    future = asyncio.run_coroutine_threadsafe(process(news), loop)

    result = future.result()

---------

 

Unfortunately, I cannot make the synchronous function async because it an external library’s function that is calling into my code. Furthermore, this for a desktop application that already has an event loop running, so it’s impossible to do “run_until_complete” on the asyncio event loop.

I wanted to ask for advice here to see if anyone has used Janus to solve this problem?

I was think of starting a thread running a Janus queue when my application starts. When I need to wait for a Coroutine to finish in a synchronous function, my idea was to put that into the queue and wait for it. It will be processed on the other thread and block the calling function until completed.

I really cannot see another way around this without nesting event loops, which I would rather not do, sounds scary.

Any ideas?

--
You received this message because you are subscribed to the Google Groups "aio-libs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aio-libs+u...@googlegroups.com.
To post to this group, send email to aio-...@googlegroups.com.
Visit this group at https://groups.google.com/group/aio-libs.
To view this discussion on the web visit https://groups.google.com/d/msgid/aio-libs/4f0cb5e9-9d6b-4472-9f9b-5228cfaf28bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Gustavo J. A. M. Carneiro

Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert

[daniel.jam...@gmail.com]

Thanks!

I’ll try that. I cannot understand how this works but will try.

If this is called from the UI thread and I pass in the UI thread’s event loop, then does this immediately execute the coroutine on the UI threads loop and block until it is complete?

I think that would be acceptable compromise. Obviously, a better way of doing things would be to block at the calling function but not block the event loop, but I think that might be impossible, without another thread or nested event loop.

What do you think?

[Gustavo Carneiro]

On Tue, 5 Mar 2019 at 22:21, <daniel.jam...@gmail.com> wrote:

Thanks!

I’ll try that. I cannot understand how this works but will try.

If this is called from the UI thread and I pass in the UI thread’s event loop, then does this immediately execute the coroutine on the UI threads loop and block until it is complete?

OK, maybe I need to break it down a little:

----
def notification(self, news):
    # this line schedules the coroutine to be executed, and returns a Future:
    future = asyncio.run_coroutine_threadsafe(process(news), loop)
    # this line will block until the coroutine has finished executing, and gets the result:
    result = future.result()
----

If you comment out the line "result = future.result()", then you do not block the event loop until the coroutine has finished.  The coroutine will be executed asynchronously.

The downside is that this way you cannot get the result this way.  Or if the coroutine raises an exception, you don't find out about it.

If you really want to get the coroutine result/exception, but still not block, then you need to store the Future somewhere, and periodically check the result.

Note that this future is a concurrent.futures.Future, not asyncio.Future.  They have similar interface, but they are different objects.

https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future

 

I think that would be acceptable compromise. Obviously, a better way of doing things would be to block at the calling function but not block the event loop, but I think that might be impossible, without another thread or nested event loop.

Yes, store the future somewhere, and in another thread check if the future has completed.

Or maybe you don't even need another thread.  If the UI event loop has a mechanism for periodically calling a callback function of yours, have a callback called from the UI event loop every 100ms, and this callback would check any pending Futures, (e.g. call future.done() just to see if the future has finished, if it has then call future.result()).

 

What do you think?

--
You received this message because you are subscribed to the Google Groups "aio-libs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aio-libs+u...@googlegroups.com.
To post to this group, send email to aio-...@googlegroups.com.
Visit this group at https://groups.google.com/group/aio-libs.

To view this discussion on the web visit https://groups.google.com/d/msgid/aio-libs/da2bee38-f6e5-49e9-baf6-b35f6c7409ae%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[daniel.jam...@gmail.com]

That’s really helpful! Thanks so much. It was the concurrent.futures.Future part, then it all made sense!

Also, thanks for the comments on storing the future object somewhere and polling it. That seems like a nice way of doing things.

I’ll have a play around.