AutobahnPython + multiprocessing

179 views
Skip to first unread message

Oleg Korsak

unread,
Feb 15, 2016, 3:12:43 AM2/15/16
to Autobahn
Hello,
I have WAMP application which connects to crossbar.io route. WAMP application is running with asyncio which is pretty cool for I/O delays, but it is still single threaded. Is it possible to make my WAMP app running an many cores? One eventloop per core. So it will run ok for I/O bound task and CPU-bound too. Any example? Thanks

Oleg Korsak

unread,
Feb 15, 2016, 3:27:24 AM2/15/16
to Autobahn
Actually I need many independent workers in separate processes

понедельник, 15 февраля 2016 г., 10:12:43 UTC+2 пользователь Oleg Korsak написал:

Tobias Oberstein

unread,
Feb 15, 2016, 4:38:43 AM2/15/16
to autob...@googlegroups.com
Hi Oleg,

Crossbar.io already has a multi-process / worker architecture

http://crossbar.io/docs/Architecture/

Not everything is implemented already, but eg here is how to scale (wrt
to CPU) a worker on multiple processes

https://github.com/crossbario/crossbarexamples/tree/master/sharedregs/python

and here is how the Web service built into Crossbar.io scales on multi-core

https://github.com/crossbario/crossbarexamples/tree/master/benchmark/web

We are working on scaling router workers on multi-core too!

Cheers,
/Tobias

Am 15.02.2016 um 09:27 schrieb Oleg Korsak:
> Actually I need many independent workers in separate processes
>
> понедельник, 15 февраля 2016 г., 10:12:43 UTC+2 пользователь Oleg Korsak
> написал:
>
> Hello,
> I have WAMP application which connects to crossbar.io
> <http://crossbar.io> route. WAMP application is running with asyncio
> which is pretty cool for I/O delays, but it is still single
> threaded. Is it possible to make my WAMP app running an many cores?
> One eventloop per core. So it will run ok for I/O bound task and
> CPU-bound too. Any example? Thanks
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

Oleg Korsak

unread,
Feb 15, 2016, 4:51:09 AM2/15/16
to autob...@googlegroups.com
Thanks for an examples, but I'm talking here more about WAMP component scaling. These examples assume that crossbar launches components that are stored nearby. In my case WAMP app is connecting to crossbar and registers it's "endpoints". They even run different Python versions. So crossbar router and WAMP app are different and separate instances. And I want to run wamp app on multiple cores


--
You received this message because you are subscribed to a topic in the Google Groups "Autobahn" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to autobahnws+...@googlegroups.com.
To post to this group, send email to autob...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/56C19CA0.2010903%40gmail.com.

Tobias Oberstein

unread,
Feb 15, 2016, 5:00:56 AM2/15/16
to autob...@googlegroups.com
Am 15.02.2016 um 10:51 schrieb Oleg Korsak:
> Thanks for an examples, but I'm talking here more about WAMP component
> scaling. These examples assume that crossbar launches components that
> are stored nearby. In my case WAMP app is connecting to crossbar and
> registers it's "endpoints". They even run different Python versions. So
> crossbar router and WAMP app are different and separate instances. And I
> want to run wamp app on multiple cores

Sure. It's not required that Crossbar.io starts your components to use
"shared registrations".

Just start multiple instances of your component and use shared regs.

Crossbar.io does not care _how_ your components are started, and what
languages/run-times they use. You can even mix (!) languages and have 1
procedure being registered (shared) by components in different
languages/run-times.

Cheers,
/Tobias

>
> 2016-02-15 11:38 GMT+02:00 Tobias Oberstein <tobias.o...@gmail.com
> <mailto:tobias.o...@gmail.com>>:
> <mailto:autobahnws%2Bunsu...@googlegroups.com>
> <mailto:autobahnws+...@googlegroups.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com>>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>
> <mailto:autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "Autobahn" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> autobahnws+...@googlegroups.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

Oleg Korsak

unread,
Feb 15, 2016, 9:46:30 AM2/15/16
to autob...@googlegroups.com
Yeah! That's an amazing feature! Thanks! :)

To unsubscribe from this group and all its topics, send an email to autobahnws+...@googlegroups.com.
To post to this group, send email to autob...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/56C1A1D5.2020401%40gmail.com.
Message has been deleted
Message has been deleted

Tobias Oberstein

unread,
Feb 16, 2016, 3:02:38 PM2/16/16
to autob...@googlegroups.com
At least for Twisted, that doesn't blend well will stdlib
multiprocessing - and it's not needed anyway.

Simply fork your workers, and each will run a separate event loop.

On Twisted:

https://twistedmatrix.com/documents/current/api/twisted.internet.utils.getProcessOutput.html

with

excutable=sys.executable
args=['myworker.py']

where myworker.py would fire up your component using ApplicationRunner.

Am 16.02.2016 um 11:16 schrieb Oleg Korsak:
> ok, looks like I've found a way to fix that:
> https://bitbucket.org/snippets/oleg_korsak/5rRGr/revisions/
>
> вторник, 16 февраля 2016 г., 12:10:59 UTC+2 пользователь Oleg Korsak
> написал:
>
> Here is my playground:
> https://bitbucket.org/snippets/oleg_korsak/5rRGr
> <https://bitbucket.org/snippets/oleg_korsak/5rRGr>
> While it still spawns multiple processes and they handle requests -
> I get random count of following errors on startup:
> Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future
> finished result=None>, <socket.socke...6.245', 8090)>,
> ('144.76.226.245', 8090))
> handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future
> finished result=None>, <socket.socke...6.245', 8090)>,
> ('144.76.226.245', 8090))>
> Traceback (most recent call last):
> File "/usr/lib64/python3.5/asyncio/events.py", line 125, in _run
> self._callback(*self._args)
> File "/usr/lib64/python3.5/asyncio/selector_events.py", line 443,
> in _sock_connect_cb
> fut.set_result(None)
> File "/usr/lib64/python3.5/asyncio/futures.py", line 329, in
> set_result
> raise InvalidStateError('{}: {!r}'.format(self._state, self))
> asyncio.futures.InvalidStateError: FINISHED: <Future finished
> result=None>
>
> And I'm not sure that asyncio`s event loops are separate and being
> handled correctly in all processes in my case.
> Does this code (
> http://autobahn.ws/python/_modules/autobahn/asyncio/wamp.html#ApplicationRunner
> <http://autobahn.ws/python/_modules/autobahn/asyncio/wamp.html#ApplicationRunner>
> ) mean that there is only one global event-loop which fails to be
> used in multiple processes? (I'm weak at asyncio stuff)
>
> понедельник, 15 февраля 2016 г., 16:46:30 UTC+2 пользователь Oleg
> Korsak написал:
>
> Yeah! That's an amazing feature! Thanks! :)
>
> 2016-02-15 12:00 GMT+02:00 Tobias Oberstein
> <tobias.o...@gmail.com <mailto:tobias.o...@gmail.com>>:
> <mailto:tobias.o...@gmail.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com
> <mailto:autobahnws%252Buns...@googlegroups.com>>
> <mailto:autobahnws+...@googlegroups.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com>
>
> <mailto:autobahnws%2Bunsu...@googlegroups.com
> <mailto:autobahnws%252Buns...@googlegroups.com>>>.
> <https://groups.google.com/d/msgid/autobahnws/9114c821-f788-4546-b515-833c44c212de%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are
> subscribed to a topic in
> the Google Groups "Autobahn" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe
> <https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe>.
> To unsubscribe from this group and all its topics,
> send an email to
> autobahnws+...@googlegroups.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com>
> <mailto:autobahnws%2Bunsu...@googlegroups.com
> <mailto:autobahnws%252Buns...@googlegroups.com>>.
> To post to this group, send email to
> autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>
> <mailto:autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/56C19CA0.2010903%40gmail.com
> <https://groups.google.com/d/msgid/autobahnws/56C19CA0.2010903%40gmail.com>.
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to
> the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails
> from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com>
> <mailto:autobahnws+...@googlegroups.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com>>.
> To post to this group, send email to
> autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>
> <mailto:autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com>
> <https://groups.google.com/d/msgid/autobahnws/CALGa3ynr-caTRcP425-O-x4zVsedThXgXFZWSubhgtdBTmk2eQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to a
> topic in the Google Groups "Autobahn" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe
> <https://groups.google.com/d/topic/autobahnws/6awZsc8bqXw/unsubscribe>.
> To unsubscribe from this group and all its topics, send an
> email to autobahnws+...@googlegroups.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com>.
> To post to this group, send email to
> autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/56C1A1D5.2020401%40gmail.com
> <https://groups.google.com/d/msgid/autobahnws/56C1A1D5.2020401%40gmail.com>.
>
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/f267e0bd-d723-4119-8203-374e7d5fcc9f%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/f267e0bd-d723-4119-8203-374e7d5fcc9f%40googlegroups.com?utm_medium=email&utm_source=footer>.
Reply all
Reply to author
Forward
0 new messages