Celery + Autobahn

[James Piechota]

Hi,

I've set up a python websocket client with Autobahn and would like to control it via Celery dispatched tasks. For example, I'd like to be able to dispatch a task to Celery and have it send a message on the open websocket. 

Since Celery and Twisted each have their own event loops I'm having trouble getting the two to talk to each other. Would my best approach be to create a Twisted protocol that opens a (unix) socket and then use that socket to talk between the Celery process and the Autobahn process? (i.e. when my Celery worker handles a task it sends a message along the unix socket to the Twisted protocol, the Twisted protocol then talks to Autobahn and tells it to send a message on the websocket).

Alternatively, if anyone has a suggestion for a better way to control the websocket client from a remote process or server using message queues or some other technology, I'm all ears.

Thanks,

James

[James Piechota]

It looks like the accepted approach is indeed to control Twisted via a separate control connection (e.g. Twisted listens on TCP). There are a couple of examples on SO:

http://stackoverflow.com/questions/23582706/twisted-and-autobahn-pub-sub-listen-on-secondary-port-for-messages

http://stackoverflow.com/questions/23584554/understanding-autobahn-and-twisted-integration

[Tobias Oberstein]

James,

I haven't worked with Celery, but the the approach you mention below
seems sane (since as you mention, Twisted or asyncio come with their own
event loop).

And yes, you could talk over Unix domain sockets or TCP loopback.

Probably, another approach would be using

http://docs.celeryproject.org/en/latest/userguide/remote-tasks.html

Don't know if you can make Celery issue a HTTP/GET or HTTP/POST to a
preconfigured URL upon tasks being enqueued or in general tasks changing
their state.

If so, this would make integration even easier: have Twisted Web running
side-by-side with Autobahn under same event loop. Forward stuff received
as HTTP/GET/POST in Twisted Web to Autobahn.

Cheers,
/Tobias

> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[James Piechota]

Thanks, Tobias. That's a neat idea - particularly cause it looks like it would retain all the celery task management functionality (like aynchronously querying results) which I'd probably have to re-implement were I to use the raw socket approach.

Great, I have few things to look into now. Appreciate it!

James

an email to autobahnws+unsubscribe@googlegroups.com
<mailto:autobahnws+unsub...@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Autobahn" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/autobahnws/ilC6kBYO_p4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to autobahnws+unsubscribe@googlegroups.com.