Connecting DistributedPubSubMediators between different ActorSystems

[Brian Dunlap]

Is it possible to connect DistributedPubSub mediators between  **different** ActorSystems?

In ClusterSystemA we'd like to subscribe to events from the ClusterSystemB mediator.

We need **separate** clusters - that's why can't use the same cluster name.

Thanks!

Brian -

[Konrad Malawski]

Hi Brian,

No, PubSub works within a cluster – it needs to know which nodes to send messages to, right?

However you could have a subscriber that will mediate the messages to the other cluster via Cluster Client – http://doc.akka.io/docs/akka/2.3.4/contrib/cluster-client.html

Would that help in your case?

I'm interested why you need separate clusters – is it that they're "local" to some resource or something like that?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

Konrad 'ktoso' Malawski

hAkker @ Typesafe

[Brian Dunlap]

We're deploying separate wars that contain functional behaviors like:

- UI (not play)

- Command Handlers

- JMS adapter for inbound A

- JMS adapter for inbound B

Today...

- each one of these is a separate buildable / deployable artifact and each one has it's own ActorSystem.  (Applications A, B, C, D)

- each artifact runs across multiple VMs for fault tolerance and performance scaling 

- we're not using node roles yet

- some apps are using cluster singletons

We're trying to get the best of these behaviors.

- a single pub/sub message bus that all ActorSystems can use and minimize a single point of failure

- no startup dependencies on the other apps

- happy cluster singletons for each application - I can't run behaviors from App A on an App B VM.  (separate builds / log files / support focus / testing regression)

What I'm thinking is to try:

- Use a single ActorSystem name across all applications

- On the Application-A nodes, list Application-A nodes first in the seed node list - followed by seed nodes for Application B, C, and D.

- On the Application-B nodes, list Application-B nodes first in the seed node list - followed by seed nodes for Application A, C, and D.

- Same idea for Application-C and Application-D seed nodes listed first in their own deployment...

- Use node roles to get the ClusterSingletons happy for Application A, B, C, D

Does this seem like a good plan?

:)

Thanks!

Brian -

[Konrad Malawski]

Hello Brian,

yeah this sounds OK to me.

You may want to tweak when the cluster determines to be "up": http://doc.akka.io/docs/akka/snapshot/scala/cluster-usage.html#how-to-startup-when-cluster-size-reached

Happy hakking!

[Konrad Malawski]

Oh, and I'd add roles to these nodes – so A nodes actually know they are "app-a" nodes (when starting up actors remotely etc).

[Patrik Nordwall]

It looks good apart from the seed nodes, see inline...

When you use different nodes as first element in the seed-nodes list on different nodes there is a chance that you will form different separate clusters instead of one uniform cluster when you boot all nodes at the same time.

Therefore you should always use the same first seed node in the configuration of all nodes. See the documentation: http://doc.akka.io/docs/akka/2.3.4/scala/cluster-usage.html#Joining_to_Seed_Nodes

/Patrik

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Rodrigo Boavida]

Hi Konrad,

We have the same requirement. The reason why we need this approach is because we need two types of cluster: Web and Processing cluster. Both have different lifecycles and concerns, but the Web cluster needs to subscribe to data sources processed by the backend cluster. The publisher/subscriber would be ideal as I would only need to connect through the contact points of a cluster client and subscribe to a given topic without being concerned how is the backend implemented in terms of actors and paths.

Does this make sense or would you have any other suggestion?

Many thanks in advance,

Rod

[Patrik Nordwall]

On Fri, Sep 12, 2014 at 6:26 PM, Rodrigo Boavida <rodrigo...@gmail.com> wrote:

Hi Konrad,

We have the same requirement. The reason why we need this approach is because we need two types of cluster: Web and Processing cluster. Both have different lifecycles and concerns, but the Web cluster needs to subscribe to data sources processed by the backend cluster. The publisher/subscriber would be ideal as I would only need to connect through the contact points of a cluster client and subscribe to a given topic without being concerned how is the backend implemented in terms of actors and paths.

Does this make sense or would you have any other suggestion?

Have you considered using cluster node roles instead of separate clusters?

Regards,

Patrik

--

[Rodrigo Boavida]

Tnks for the reply Patrik.

Yes we have that option on the table as well, but would each node in the cluster still participate in the gossip control of all the other nodes, or would it be confined to nodes within the same role?

Let me expand a bit on what we have. Both backend and front end (web) are very much decoupled in terms of purpose. The FrontEnd doesn't always need the backend layer, could even be offline as the front end has it's own data access nodes. Depends on what is the user requesting. Backend is also for other processing functions requested by other entities through an API. They could also live in different geographies. Bottom line is they don't depend on each other to exist.

I guess the only reason I still struggle on accepting having everything inside the same cluster is because of the "community" concept of the cluster, where there's common agreement on classifying each node state and gossiping it around. Wouldn't the backend worker nodes influence judgement on availability of frontend nodes to other frontend nodes?

And in terms of seed nodes, I imagine we would need to have four seed nodes, two for each layer (be and fe) to make sure nodes on both layers start successfully.

Looking forward on your comments.

Tnks,

Rod

[Patrik Nordwall]

On Mon, Sep 15, 2014 at 3:02 PM, Rodrigo Boavida <rodrigo...@gmail.com> wrote:

Tnks for the reply Patrik.

Yes we have that option on the table as well, but would each node in the cluster still participate in the gossip control of all the other nodes, or would it be confined to nodes within the same role?

They would all be part the same gossip.

 

Let me expand a bit on what we have. Both backend and front end (web) are very much decoupled in terms of purpose. The FrontEnd doesn't always need the backend layer, could even be offline as the front end has it's own data access nodes. Depends on what is the user requesting. Backend is also for other processing functions requested by other entities through an API. They could also live in different geographies. Bottom line is they don't depend on each other to exist.

I guess the only reason I still struggle on accepting having everything inside the same cluster is because of the "community" concept of the cluster, where there's common agreement on classifying each node state and gossiping it around. Wouldn't the backend worker nodes influence judgement on availability of frontend nodes to other frontend nodes?

And in terms of seed nodes, I imagine we would need to have four seed nodes, two for each layer (be and fe) to make sure nodes on both layers start successfully.

Thanks for clarifying.  You have presented several good reason for why they should be separated.

The ClusterClient doesn't support publish-subscribe where the subscriber is on the client side. It is designed for the opposite direction. It has been discussed a few times but I'm not sure we came to a working solution. See here: https://groups.google.com/d/msg/akka-user/5sWghsQF9oc/DeQxl6VQfuEJ

Looks like a created a small sample: https://gist.github.com/patriknw/6271877

That is extremely untested and only intended as inspiration.

Please read that thread and see if it makes any sense for you. I'm happy to try to answer further questions.

Cheers,

Patrik

[Rodrigo Boavida]

Tnks again Patrik. I'm happy not to be the only one having this kind of requirement.

I will definitely have a look at this and will give you some feedback.

Cheers,

Rod

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/oecXFlSS9MY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Patrik Nordwall]

On Wed, Sep 17, 2014 at 11:41 AM, Rodrigo Boavida <rodrigo...@gmail.com> wrote:

Tnks again Patrik. I'm happy not to be the only one having this kind of requirement.

You're welcome

 

I will definitely have a look at this and will give you some feedback.

Thanks, looking forward to that

/Patrik