how to do fully asynchronous publishing?
108 views
Skip to first unread message
David Ford
Dec 2, 2015, 5:31:32 PM12/2/15
to Crossbar
@Tobias? :)
how can i, from clientA, trigger a function on clientB, whereby clientB runs several functions and publishes to several endpoints, and each publish() is forwarded immediately through crossbar instead of waiting until clientB has finished performing all functions?
e.g.
browser js clientA does either an rpc call or publish to com.example.trigger
python backend clientB, having either a matching rpc function or subscription callback for com.example.trigger (doesn't matter which i use) then runs callback(). callback will look like the following:
@inlineCallbacks
def __yield(self, uri, data):
logger.info(log some stuff)
yield self.publish(uri, data)
@wamp.register(com.example.trigger)
at present, crossbar router does see published data immediately for the first three functions foo1(), foo2(), and foo3(); but it waits until after foo4() has completed before any of the published data is forwarded to the browser client. --loglevel debug and tcpdump confirms it.
how do i make crossbar forward the messages immediately to subscribed clients using a single trigger?
how can i, from clientA, trigger a function on clientB, whereby clientB runs several functions and publishes to several endpoints, and each publish() is forwarded immediately through crossbar instead of waiting until clientB has finished performing all functions?
e.g.
browser js clientA does either an rpc call or publish to com.example.trigger
python backend clientB, having either a matching rpc function or subscription callback for com.example.trigger (doesn't matter which i use) then runs callback(). callback will look like the following:
class Butterfly(ApplicationSession):
@inlineCallbacks
def __yield(self, uri, data):
logger.info(log some stuff)
yield self.publish(uri, data)
@wamp.register(com.example.trigger)
def callback(self, opts):
@asyncio.coroutine
def foo1(opts):
results=compute() # takes <1 second
__yield(com.example.something1, results)
@asyncio.coroutine
def foo2(opts):
results=compute() # takes <1 second
__yield(com.example.something2, results)
@asyncio.coroutine
def foo3(opts):
results=compute() # takes <1 second
__yield(com.example.something3, results)
@asyncio.coroutine
def foo4(opts):
results=compute() # takes 60 seconds
__yield(com.example.something4, results)
loop = asyncio.get_event.loop()
tasks = [asyncio.Task(foo1(opts)),
asyncio.Task(foo2(opts)),
asyncio.Task(foo3(opts)),
asyncio.Task(foo4(opts))]
loop.run_until_complete(asyncio.wait(tasks))
@asyncio.coroutine
def foo1(opts):
results=compute() # takes <1 second
__yield(com.example.something1, results)
@asyncio.coroutine
def foo2(opts):
results=compute() # takes <1 second
__yield(com.example.something2, results)
@asyncio.coroutine
def foo3(opts):
results=compute() # takes <1 second
__yield(com.example.something3, results)
@asyncio.coroutine
def foo4(opts):
results=compute() # takes 60 seconds
__yield(com.example.something4, results)
loop = asyncio.get_event.loop()
tasks = [asyncio.Task(foo1(opts)),
asyncio.Task(foo2(opts)),
asyncio.Task(foo3(opts)),
asyncio.Task(foo4(opts))]
loop.run_until_complete(asyncio.wait(tasks))
at present, crossbar router does see published data immediately for the first three functions foo1(), foo2(), and foo3(); but it waits until after foo4() has completed before any of the published data is forwarded to the browser client. --loglevel debug and tcpdump confirms it.
how do i make crossbar forward the messages immediately to subscribed clients using a single trigger?
Amber Brown
Dec 3, 2015, 4:56:41 AM12/3/15
to Crossbar
Hi,
This appears to be because the item that takes 60 seconds is doing so synchronously, and is blocking the entire reactor.
Twisted and asyncio can only do things when you let them - if you spend 60 seconds doing CPU time and not yielding to the reactor, it cannot run. They are both cooperative multitasking frameworks.
What is the nature of your processing here? Is it network I/O (like database access -- which should be done using Twisted or asyncio specific libraries), disk I/O (reading/writing a lot of files -- in which case you should use a threadpool/thread executor) or CPU-heavy (in which case you may want to use something like Twisted's process support to spin up a new Python to do the work specifically, and communicate with it asynchronously)?
- Amber
This appears to be because the item that takes 60 seconds is doing so synchronously, and is blocking the entire reactor.
Twisted and asyncio can only do things when you let them - if you spend 60 seconds doing CPU time and not yielding to the reactor, it cannot run. They are both cooperative multitasking frameworks.
What is the nature of your processing here? Is it network I/O (like database access -- which should be done using Twisted or asyncio specific libraries), disk I/O (reading/writing a lot of files -- in which case you should use a threadpool/thread executor) or CPU-heavy (in which case you may want to use something like Twisted's process support to spin up a new Python to do the work specifically, and communicate with it asynchronously)?
- Amber
David Ford
Dec 3, 2015, 1:30:05 PM12/3/15
to Crossbar
i considered this, but discarded this line of thought for two reasons.
1) this 60 second task yields in several places
2) running crossbar with loglevel debug (and confirmed by tcpdump), crossbar gets the published messages from the other tasks within a few dozen milliseconds
the long running task is doing a chain of DNS requests using async coroutines with yielded calls and internal yields. the actual run time is about 2-4 seconds but i stuffed a yielded asyncio.sleep(60) at the end of it
Tobias Oberstein
Dec 4, 2015, 6:23:26 AM12/4/15
to cross...@googlegroups.com
operations - they won't run concurrently.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com
> <mailto:crossbario+...@googlegroups.com>.
> To post to this group, send email to cross...@googlegroups.com
> <mailto:cross...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crossbario/1e40eecc-c0be-4910-a337-5b40687cafec%40googlegroups.com
> <https://groups.google.com/d/msgid/crossbario/1e40eecc-c0be-4910-a337-5b40687cafec%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
David Ford
Dec 5, 2015, 12:19:35 AM12/5/15
to Crossbar
ok, given the following structure of these 4 functions, let's presume that when using psycopg2 asynchronously, aio has yielded my best (it works) async wrapper for database lookups. twisted's adbapi seems to have vanished, it doesn't get installed when the package is built and installed and references for it missing are scarce at best. further, i must mix autobahn.twisted.wamp with asyncio. using ApplicationSession from autobahn.asyncio.wamp fails to finish startup. see https://github.com/crossbario/crossbar/issues/534.
so therefore:
- i must use autobahn.twisted.wamp.ApplicationSession and i use asyncio for aio and psycopg2
- therefore, i mix @asyncio.coroutine with @inlineCallbacks
- my sql queries are fast
- my pythonwhois takes between 1 and several seconds
- my dnspython queries take between 2 and several seconds
foo1 and foo2 run separate fast sql queries using aio and psycopg2
foo3 does a pythonwhois lookup
foo4 does several dozen DNS queries that can range from milliseconds to several seconds.
all of these functions are needed when a particular web page is opened to populate numerous DIVs with data.
how should i restructure this so each DIV is populated as fast as possible with as little impact to other functions as possible?
thank you very much in advance for helping,
-d
Tobias Oberstein
Dec 5, 2015, 4:53:10 AM12/5/15
to cross...@googlegroups.com
There is no need (and it doesnt make sense) to mix tx anf asyncio.
Simply dont yield things you want to run concurrently, but usr Deferreds (tx) or Futures (aio).
Sent from Mobile (Google Nexus 5)
--
You received this message because you are subscribed to the Google Groups "Crossbar" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/85acf967-5410-45e2-88b3-094fe1f721a3%40googlegroups.com.
David Ford
Dec 6, 2015, 3:37:58 PM12/6/15
to cross...@googlegroups.com
can't go pure asyncio as per the bug i reported. crossbar won't finish starting up. i require asyncio to use aio with psycopg2.
thank you
-d
To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/CABuE%2BY7ZNxCn2yiQbQ1u%3DvrbrATA8Rfw_kDC_Y_KLboR70sp6w%40mail.gmail.com.
Gay/FireRescue/Geek in 33484, USA
It's the ideals of Linux and Open Source that are amazing, it embodies what WE want, not what is marketed
It's the ideals of Linux and Open Source that are amazing, it embodies what WE want, not what is marketed
L. Daniel Burr
Dec 24, 2015, 3:01:57 PM12/24/15
to Crossbar
Have you considered using txpostgres (https://pypi.python.org/pypi/txpostgres) instead of aio? It is the twisted equivalent of aio.
Reply all
Reply to author
Forward
0 new messages