Nested asynchronous (wamp) calls

[sieben tupel]

Hi everybody, I got the following situation:

Inside some wamp procedure i have other asynchronous calls, which are called in a chain using callbacks. Now, at the end, i need to return the result from my last inner asynchronous call to the original caller of my method (called through wamp rpc from another client). But as twisted is non blocking, the execution of my outer method ends returning nothing before my inner calls have finished. Chaining the inner calls together (because they depend on each others results) works fine. The asynchronous calls may be calls to other registered wamp procedures or any other asynchronous methods (e.g. in my case it is database interaction using a non blocking driver)

How can i wait/make my outer method return the result of the inner method as soon as it is ready?

Before you say "don't do this, this breaks all the advantages of asynchronous programming", i got my reason why i need this. 

Code example to illustrate what i want:

class MyComponent(ApplicationSession):
    @inlineCallbacks
    def onJoin(self, details):

        def some_method(x, y):
            # ...
            async_result_1 = do_some_async_call(x)
            async_result_1.addCallback(fire_on_1)

            def fire_on_1(result_from_1):
                async_result_2 = do_some_other_async_call(y, result_from_1)
                async_result_2.addCallback(fire_on_2)

            def fire_on_2(result_from_2):
                return result_from_2 # <--result needs to get to caller of some_method()

Bringing nested calls even a step further, how would the following work:

Again i have a remote procedure with inner asynchronous calls. But now those internal calls do not depend on each other in a linear way. Lets say i have 4 inner calls (inner_1,...,inner_4). The calls inner_1, inner_2 and inner_3 do not depend on each other, but to make call inner_4 i have to wait for their results. Thinking in the "old" thread based parallel execution i would start calls inner_1..3 and wait for them to finish and then run inner_4. This does not work with the asynchronous model and simple chaining would be bad because i would potentially waste performance (e.g. inner_1 needs to wait for some network input, then inner_2 and inner_3 would be blocked as well for no good reason even when they could already perform their work)

Is there any way to do this? How would i access the results from inner_1..3 to use them in inner_4?

By the way i'm using python 3.5 and the current version of autobahn from pip.

hope you can enlighten my way a little :)

[Tobias Oberstein]

Hi,

yes, the implementation of a procedure in a callee can itself issue
asynchronous calls.

When you want to issue multiple calls that should run parallel and do
something when _all_ calls have finished, without chaining (which would
lead to non-parallel, though still asynch. execution), have a look at:

https://twistedmatrix.com/documents/current/core/howto/defer.html#gatherresults
http://txaio.readthedocs.org/en/latest/api.html#txaio.gather

Cheers,
/Tobias

> returnresult_from_2 # <--result needs to get to caller of some_method()

> |
>
>
> Bringing nested calls even a step further, how would the following work:
>
> Again i have a remote procedure with inner asynchronous calls. But now
> those internal calls do not depend on each other in a linear way. Lets

> say i have 4 inner calls (/inner_1,...,inner_4/). The calls /inner_1,
> inner_2 /and/inner_3/ do not depend on each other, but to make call
> /inner_4/ i have to wait for their results. Thinking in the "old" thread
> based parallel execution i would start calls /inner_1..3/ and wait for
> them to finish and then run /inner_4/. This does not work with the

> asynchronous model and simple chaining would be bad because i would

> potentially waste performance (e.g. /inner_1/ needs to wait for some
> network input, then /inner_2/ and /inner_3/ would be blocked as well for

> no good reason even when they could already perform their work)
>
> Is there any way to do this? How would i access the results from

> /inner_1..3 /to use them in/inner_4/?

>
> By the way i'm using python 3.5 and the current version of autobahn from
> pip.
>
> hope you can enlighten my way a little :)
>

> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/4260a7d7-8dc4-48de-a564-b7044753a172%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/4260a7d7-8dc4-48de-a564-b7044753a172%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Elvis Stansvik]

Den 27 jan 2016 2:44 fm skrev "sieben tupel" <7tu...@googlemail.com>:
>
> Hi everybody, I got the following situation:
>
> Inside some wamp procedure i have other asynchronous calls, which are called in a chain using callbacks. Now, at the end, i need to return the result from my last inner asynchronous call to the original caller of my method (called through wamp rpc from another client). But as twisted is non blocking, the execution of my outer method ends returning nothing before my inner calls have finished. Chaining the inner calls together (because they depend on each others results) works fine. The asynchronous calls may be calls to other registered wamp procedures or any other asynchronous methods (e.g. in my case it is database interaction using a non blocking driver)
>
> How can i wait/make my outer method return the result of the inner method as soon as it is ready?
>
> Before you say "don't do this, this breaks all the advantages of asynchronous programming", i got my reason why i need this. 
>
> Code example to illustrate what i want:
> class MyComponent(ApplicationSession):
>     @inlineCallbacks
>     def onJoin(self, details):
>
>         def some_method(x, y):
>             # ...
>             async_result_1 = do_some_async_call(x)
>             async_result_1.addCallback(fire_on_1)
>
>             def fire_on_1(result_from_1):
>                 async_result_2 = do_some_other_async_call(y, result_from_1)
>                 async_result_2.addCallback(fire_on_2)
>
>             def fire_on_2(result_from_2):
>                 return result_from_2 # <--result needs to get to caller of some_method()
>

Can't you just yield on the calls to your inner methods? You're even using the inlineCallbacks decorator in your example above, but you're not yielding anywhere.

The other approach would be to manually construct a Deferred which you return immediately in your endpoint (outermost) method, and which you .callback(...) on when you have the final result in your last inner method.

But sounds like you should just use inline callbacks and yield.

For your other question, just like Tobias said, there are mechanism to "gather" results. (I think the plain twisted method is gatherResults).

Cheers,
Elvis

>
> Bringing nested calls even a step further, how would the following work:
>
> Again i have a remote procedure with inner asynchronous calls. But now those internal calls do not depend on each other in a linear way. Lets say i have 4 inner calls (inner_1,...,inner_4). The calls inner_1, inner_2 and inner_3 do not depend on each other, but to make call inner_4 i have to wait for their results. Thinking in the "old" thread based parallel execution i would start calls inner_1..3 and wait for them to finish and then run inner_4. This does not work with the asynchronous model and simple chaining would be bad because i would potentially waste performance (e.g. inner_1 needs to wait for some network input, then inner_2 and inner_3 would be blocked as well for no good reason even when they could already perform their work)
>
> Is there any way to do this? How would i access the results from inner_1..3 to use them in inner_4?
>
> By the way i'm using python 3.5 and the current version of autobahn from pip.
>
> hope you can enlighten my way a little :)
>

> --
> You received this message because you are subscribed to the Google Groups "Autobahn" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to autobahnws+...@googlegroups.com.
> To post to this group, send email to autob...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/4260a7d7-8dc4-48de-a564-b7044753a172%40googlegroups.com.

[sieben tupel]

hi, thank you very much, booth of you. The gatherresults is exactly what i need for the later case and in the first one i have tried using the @inlinecallbacks decorator before, but i made i mistake some place else so it didn't work. Fortunately i figured out this other mistake and now it works as it is supposed to.

thank you very much, cheers mo

[Elvis Stansvik]

Glad it worked out.

Elvis

>
> --
> You received this message because you are subscribed to the Google Groups
> "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to autobahnws+...@googlegroups.com.
> To post to this group, send email to autob...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/d991131b-b0e3-441d-befe-067341c7a362%40googlegroups.com.