Pub/Sub performance for app-level wildcard filering

[Behrad Zari]

I want to dispatch messages to wildcard registered (can be remote) subscribers. I see two possibilities:

1) Can I use remote actors as akka-camel endpoints? + routes using .when with regex on messages would gimme wildcard pushing at the sender

2) Distributed Pub/Sub but unfortunately it doesn't support wildcard subscribers. So can I accept the overhead to subscribe all subscribers to a single topic, and let them filter messages by their own regex? Would this send-to-all semantic become a bottleneck in 1000 msg/sec deployment? 

[Martynas Mickevičius]

Hi Behrad,

On Sat, Oct 11, 2014 at 10:12 PM, Behrad Zari <beh...@gmail.com> wrote:

I want to dispatch messages to wildcard registered (can be remote) subscribers. I see two possibilities:

1) Can I use remote actors as akka-camel endpoints? + routes using .when with regex on messages would gimme wildcard pushing at the sender

you can't specify an actor as a camel endpoint. 

You can use ActorSelection to select multiple actors (even remote ones) with a wildcard. However a wildcard in ActorSelection can only select all actors at the specified depth.

 

2) Distributed Pub/Sub but unfortunately it doesn't support wildcard subscribers. So can I accept the overhead to subscribe all subscribers to a single topic, and let them filter messages by their own regex? Would this send-to-all semantic become a bottleneck in 1000 msg/sec deployment?

This needs a benchmark to see if it would be a bottleneck or not. However Akka Distributed PubSub uses optimization and sends a message over the wire only once per node (if that node has a matching topic).

Another approach could be using a EventBus with custom classification rules. However EvenBus is local ActorSystem only. You would need to subscribe remote actors to the EventBus explicitly to carry messages from one node to another.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Martynas Mickevičius

Typesafe – Reactive Apps on the JVM

[Behrad]

Thank you Martynas,

​​

​

2014-10-13 16:30 GMT+03:30 Martynas Mickevičius <martynas.m...@typesafe.com>:

Hi Behrad,

On Sat, Oct 11, 2014 at 10:12 PM, Behrad Zari <beh...@gmail.com> wrote:

I want to dispatch messages to wildcard registered (can be remote) subscribers. I see two possibilities:

1) Can I use remote actors as akka-camel endpoints? + routes using .when with regex on messages would gimme wildcard pushing at the sender

​​

​

you can't specify an actor as a camel endpoint. 

​

I found that actors can not be used as camel consumer endpoints, when reading about akka-camel, however I can route to them. So why couldn't I

from( "timer://foo" ).to( "akka.tcp://beh...@127.0.0.1:2553/user/remote" ) 

instead of

from( "timer://foo" ).to( "akka://behrad/user/normal" ) ?

Couldn't akka-camel  transform a CamelMessage to a serializable subset, send it to the actorPath finally and let the akka-remote/akka-cluster do the rest, and at that actor I would receive the serialized message.

​

​

You can use ActorSelection to select multiple actors (even remote ones) with a wildcard. However a wildcard in ActorSelection can only select all actors at the specified depth.

 

2) Distributed Pub/Sub but unfortunately it doesn't support wildcard subscribers. So can I accept the overhead to subscribe all subscribers to a single topic, and let them filter messages by their own regex? Would this send-to-all semantic become a bottleneck in 1000 msg/sec deployment?

This needs a benchmark to see if it would be a bottleneck or not. However Akka Distributed PubSub uses optimization and sends a message over the wire only once per node (if that node has a matching topic).

Another approach could be using a EventBus with custom classification rules. However EvenBus is local ActorSystem only. You would need to subscribe remote actors to the EventBus explicitly to carry messages from one node to another.

​

​

EventBus is not my desired solution since the sender part would/should know about the sender. That is not a true pub/sub​.

The best solution I've came to so far, is this:

when a remote node (R) wants to subscribe itself to a wildcard content, it sends a home-made subscribe message to my local node (L) subscriber actor, he then spawns a local proxy-actor that would relay all messages to the target actor in R, then adds a new local filtered (R's provided wildcard) camel route to the proxy-actor :)

Another question hits my mind, about seed nodes on akka-cluster. In my application, I have split up my logic to 3 layers (My Rest API + Core + ServicePlugins) which any could be remote or not. So my actual usage is remoting, not clustering [of the same app logic]. However If I use akka-cluster, should I have 3 seed nodes in each of my apps? or just one (=core) in each?

​

​​

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Martynas Mickevičius

Typesafe – Reactive Apps on the JVM

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/ui5GpGK9nr4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

--Behrad

[Patrik Nordwall]

On Mon, Oct 13, 2014 at 9:11 PM, Behrad <beh...@gmail.com> wrote:

Thank you Martynas,

​​

​

2014-10-13 16:30 GMT+03:30 Martynas Mickevičius <martynas.m...@typesafe.com>:

Hi Behrad,

On Sat, Oct 11, 2014 at 10:12 PM, Behrad Zari <beh...@gmail.com> wrote:

I want to dispatch messages to wildcard registered (can be remote) subscribers. I see two possibilities:

1) Can I use remote actors as akka-camel endpoints? + routes using .when with regex on messages would gimme wildcard pushing at the sender

​​

​

you can't specify an actor as a camel endpoint. 

​

I found that actors can not be used as camel consumer endpoints, when reading about akka-camel, however I can route to them. So why couldn't I

from( "timer://foo" ).to( "akka.tcp://beh...@127.0.0.1:2553/user/remote" ) 

instead of

from( "timer://foo" ).to( "akka://behrad/user/normal" ) ?

Couldn't akka-camel  transform a CamelMessage to a serializable subset, send it to the actorPath finally and let the akka-remote/akka-cluster do the rest, and at that actor I would receive the serialized message.

CamelMessage is not serializable and it would be hard for akka to know what a serializable subset would be. I think you have to first send it to local consumer, which sends it to remote paths.

 

​

​

You can use ActorSelection to select multiple actors (even remote ones) with a wildcard. However a wildcard in ActorSelection can only select all actors at the specified depth.

 

2) Distributed Pub/Sub but unfortunately it doesn't support wildcard subscribers. So can I accept the overhead to subscribe all subscribers to a single topic, and let them filter messages by their own regex? Would this send-to-all semantic become a bottleneck in 1000 msg/sec deployment?

This needs a benchmark to see if it would be a bottleneck or not. However Akka Distributed PubSub uses optimization and sends a message over the wire only once per node (if that node has a matching topic).

Another approach could be using a EventBus with custom classification rules. However EvenBus is local ActorSystem only. You would need to subscribe remote actors to the EventBus explicitly to carry messages from one node to another.

​

​

EventBus is not my desired solution since the sender part would/should know about the sender. That is not a true pub/sub​.

The best solution I've came to so far, is this:

when a remote node (R) wants to subscribe itself to a wildcard content, it sends a home-made subscribe message to my local node (L) subscriber actor, he then spawns a local proxy-actor that would relay all messages to the target actor in R, then adds a new local filtered (R's provided wildcard) camel route to the proxy-actor :)

Another question hits my mind, about seed nodes on akka-cluster. In my application, I have split up my logic to 3 layers (My Rest API + Core + ServicePlugins) which any could be remote or not. So my actual usage is remoting, not clustering [of the same app logic]. However If I use akka-cluster, should I have 3 seed nodes in each of my apps? or just one (=core) in each?

You can have as many seed nodes as you want. The configured seed node addresses are only used as initial contact points when joining the cluster. Members of an Akka cluster may play different roles, see http://doc.akka.io/docs/akka/2.3.6/java/cluster-usage.html#Node_Roles

Regards,

Patrik

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw