Sending celluloid notifications between different processes

[Josh Adams]

Hi,

So I brought this up in IRC, and Tony and Grant both suggested the ML is probably the place for me to have this sort of discussion.

Anyway, I have an app that will involve a reel app that provides a websocket feed, and another app (call it proxy) that for now will be proxying another websocket feed into that.  I'll ultimately have other actors in proxy that do things like http polling and turning that data into websocket-type data as well so that my frontend can represent those polling-services as websockets/realtime updates as well.

The way I plan on implementing this is having the reel app subscribe to notifications on a channel, and have the proxy publish on that channel.

I'm keeping them in separate processes because this description is a simplification, and there are ultimately around 5 different processes that will need to be scaled using different strategies in the future conceivably.  Obviously if they were in the same process then notification is simple.

Anyway, I wondered if anyone else felt like joining a discussion about the best way to do it.  Short of the notification / cross-process pubsub, there's no other need to do actor cross-talk.  I could use dcell to just access the notifier actor, and start it up explicitly or link to the remote one rather than using celluloid/autostart.  That doesn't sound great at present (dcell feels like overkill for this)

Grant sent a link to https://github.com/grantr/telecine/blob/master/lib/celluloid/zmq/notifier.rb which he described as a zmq notifier implementation that's very experimental.  That sounds like the sort of thing I was planning, but wondered if anyone else has dealt with this.

Anyway, thanks for any responses :)

--
Josh Adams
CTO | isotope|eleven
[cell] (205) 215-3957
[work] (877) 476-8671 x201

[Ben Langfeld]

I take it you've considered real queueing systems like AMQP and discounted them. Can you say a little about that just for my own curiosity?

Regards,
Ben Langfeld

--
You received this message because you are subscribed to the Google Groups "Celluloid" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celluloid-rub...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Josh Adams]

Is it actually crazy that I wanted to go zeromq/celluloid pubsub before real queuing systems?

[Tony Arcieri]

0MQ provides brokerless, ephemeral messaging. If that's what you need, perfect. Brokers are simpler in some ways since they work like a "server" (or servers) you can connect to, instead of expecting services to rendezvous without a broker. They're also nice if you need durability.

Tony Arcieri

[Josh Adams]

Durability's not a requirement in this component (it's realtime data to the browser).  Simplicity of design is on up there, inasmuch as I want to just publish the new data as it comes in and get it to the browser.  If I drop an update, so be it - I don't much care.  This is user interface update, one way read-only as far as the browser's concerned.  All writes happen off-socket, via traditional restful services.  They'll be placed into the system elsewhere.

[cootcraig]

On Wednesday, April 24, 2013 1:05:10 PM UTC-6, Josh Adams wrote:

Durability's not a requirement in this component (it's realtime data to the browser).  Simplicity of design is on up there, inasmuch as I want to just publish the new data as it comes in and get it to the browser.  If I drop an update, so be it - I don't much care.  This is user interface update, one way read-only as far as the browser's concerned.  All writes happen off-socket, via traditional restful services.  They'll be placed into the system elsewhere.

On Wed, Apr 24, 2013 at 1:28 PM, Tony Arcieri <tony.a...@gmail.com> wrote:

0MQ provides brokerless, ephemeral messaging. If that's what you need, perfect. Brokers are simpler in some ways since they work like a "server" (or servers) you can connect to, instead of expecting services to rendezvous without a broker. They're also nice if you need durability.

0mq pub/sub is a good fit for my application.  I have a Trinidad/JRuby/celluloid-zmq thread that publishes phone switch events. I currently have 2 JRuby/celluloid-zmq applications/threads that subscribe.  One of them is a Trinidad hosted HTTP server that provides

AJAX long poll events.

 

[Tim Carey-Smith]

Hola,

There was a similar discussion around the Gossip and PAXOS stuff which could exist on top of DCell.
I suggested that this be something that is done on top of core Celluloid.

The same could be true of this.
You could very simply use the notifications actors of another DCell node.

Also, grantr did have a PR for notifications directly in Celluloid ZMQ.
https://github.com/celluloid/celluloid-zmq/pull/16 which I guess he pulled into his telecine project.

At some level, exposing the transport layer to the actors is problematic.
This PR solves this by using a ZMQ PUB socket instead of the internal notifications.

One thing I did suggest to AJ (fujin) was to ignore the transport layer to begin with and focus on the Actors in the same node.
This means that you can add the distribution of messages at a later stage.

He might be able to chime in with his experiences.

Ciao,
Tim

> *Josh Adams*
> CTO | isotope|eleven <http://www.isotope11.com>

> [cell] (205) 215-3957
> [work] (877) 476-8671 x201
>

[Grant Rodgers]

FWIW my goal for telecine is to come up with useful abstractions for tunneling celluloid messages across processes independent of the transport mechanism. My early work was zeromq-focused but lately I've been working on demonstrators that use internal pubsub in the hope that eventually zeromq, websockets, tcp, or really any socket-oriented transport will work.

I like the zeromq notifier for the way it extends internal notifications to external mechanisms. The abstraction does leak a bit though. Celluloid notifications were never designed to support multiple notifiers which makes it difficult to replace the internal notifier completely as the pull request recommends. If you decide to experiment, I'd recommend just starting up the notifier independently and calling methods on it directly. 

Also you'll probably want to remove the SocketMonitor stuff since it relies on zeromq 3.2.