Efficiently subscribing for numerous topics

[Zaar Hai]

Hi guys,

Lets say I have 1000 machines that report temperature every now and then (not very often) and 1000 users that want to get temperature updates - each user about its own subset of machines (the subset may overlap between clients).

The data is sensitive - each user has a set of machines he is allowed to receive updates about.

I thought about topic-per-machine approach - it aligns perfectly with WAMP dynamic auth where I can allow machines only publish to their own topic, e.g. temperature_<machine_id> and users only to subscribe to topics for machines they are eligible to see.

The only problem seems to be on the user side of code - if a user wants (and allowed) to receive updates for 500 machines, it (e.g. browser client) has to issue 500 subscription requests, which sounds grossly inefficient.

Any other/better ideas on how to do it?

Thanks in advance,

Zaar

[Konstantin Burkalev]

Hi, Zaar Hai!

Well, why not to do so? 

Client need to send 500 subscription requests only once, and then client will constantly receive updates for needed sensors.

Another solution could be: to create 1 topic for 1 client, and implement logic on publisher to send messages only to clients (client topics) that are allowed to receive data.  

среда, 10 октября 2018 г., 9:56:35 UTC+3 пользователь Zaar Hai написал:

[Zaar Hai]

Please see below inline.

On Wed, 10 Oct 2018 at 20:53, Konstantin Burkalev <kostik...@gmail.com> wrote:

Hi, Zaar Hai!

Well, why not to do so? 

Client need to send 500 subscription requests only once, and then client will constantly receive updates for needed sensors.

Well, this way, each time a client (browser) loads it needs to issue 500 requests, which may take time. Also it's sub-optimal user experience wise - client needs first to subscribe for updates, and only then to fetch the latest status (in order not to have the complete picture).

 

Another solution could be: to create 1 topic for 1 client, and implement logic on publisher to send messages only to clients (client topics) that are allowed to receive data.  

Yes, that will obviously work, but that will mean I'm lifting up routing logic from router to my app; and ideally I would live to leave routing to the router :)

 

среда, 10 октября 2018 г., 9:56:35 UTC+3 пользователь Zaar Hai написал:

Hi guys,

Lets say I have 1000 machines that report temperature every now and then (not very often) and 1000 users that want to get temperature updates - each user about its own subset of machines (the subset may overlap between clients).

The data is sensitive - each user has a set of machines he is allowed to receive updates about.

I thought about topic-per-machine approach - it aligns perfectly with WAMP dynamic auth where I can allow machines only publish to their own topic, e.g. temperature_<machine_id> and users only to subscribe to topics for machines they are eligible to see.

The only problem seems to be on the user side of code - if a user wants (and allowed) to receive updates for 500 machines, it (e.g. browser client) has to issue 500 subscription requests, which sounds grossly inefficient.

Any other/better ideas on how to do it?

Thanks in advance,

Zaar

--
You received this message because you are subscribed to a topic in the Google Groups "WAMP" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wampws/7SWHNvUjjb0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wampws+un...@googlegroups.com.
To post to this group, send email to wam...@googlegroups.com.
Visit this group at https://groups.google.com/group/wampws.
To view this discussion on the web visit https://groups.google.com/d/msgid/wampws/da1ddeb0-45fc-4088-9505-7ff3ec66c4d8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Zaar

[Zaar Hai]

Well, my hesitation is not about a need to subscribe to 500 topics, but that I need to issue 500 subscription requests to do that.

Did anyone do such massive subscriptions? Can you please share your experience?

Also, was "one SUBSCRIBE request for multiple topics" feature ever considered?

[Laurence Armstrong]

Hi there!

One possibility would be to have an intermediary aggregation client.

This aggregation client is subscribed to all 1000 temperature sensing machines.

The client offers a procedure "aggregateToTopic" which takes a URI and a subset of machines.

When it receives an event from a machine, it publishes it to all URIs it has that are linked to that machine.

It could even batch publish the event maybe with argumentsKw like:

{
    "machine_<id>": <temp>,
    "machine_<id>": <temp>,
    ...
}

So from the browser perspective it can subscribe to a single topic and then call the "aggregateToTopic" RPC. It should then get all events applicable to it.

There are a bunch of technical details you may need to sort out there, but it's just a suggestion.

Regarding your last question about subscribing to multiple topics in one request, there is this PubSub advanced feature where you can subscribe based on patterns.

[Zaar Hai]

Hi Laurence,

Thanks for the suggestion. That would be the ultimate solution of course. The only downside with this approach is practically you do routing in application and not WAMP level; and it incurs another round of message de/serialization. It would work I believe, but I wanted to make sure there is no other way I'm missing before I jump on this approach. 

Pattern subscriptions are great, but requires organizing my machines into some hierarchy, which is not clear in the current stage.

To sum up all of the options suggested:

1. Use/try-to-design for pattern subscriptions

2. Take a plunge with subscription storm

3. Implement your own routing logic by building a catch-all subscriber/distributor (on a server side)

Thanks everyone for the answers!

To view this discussion on the web visit https://groups.google.com/d/msgid/wampws/d436a1f5-6b7c-4f59-b097-78f65ca0cd45%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zaar

[Konstantin Burkalev]

Mine 2 cents in addition to Laurence Armstrong answer:

Server side client, which collects sensor data can register an RPC, that browser client calls, providing for example sensors list of interest (or just a fact of interest) → than server RPC callee creates unique topic id for this client and returns it back as RPC result → client than subscribe to this topic and awaits for sensor data. Some server side peer (may be one that collects sensor data) needs to save mapping of sensors to client urls and use this information to publish data.

So, anyway, you need to implement somewhere logic for checking user permissions or mapping.

Regarding additional round trip — you can install such peer locally on router and connect it through unix socket (raw socket transport) to minimize websocket overhead.

[Zaar Hai]

Thanks Konstantin!

That will be a path to take if 500 subscriptions wouldn't fly :)

--

You received this message because you are subscribed to a topic in the Google Groups "WAMP" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wampws/7SWHNvUjjb0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wampws+un...@googlegroups.com.
To post to this group, send email to wam...@googlegroups.com.
Visit this group at https://groups.google.com/group/wampws.

To view this discussion on the web visit https://groups.google.com/d/msgid/wampws/e495b41c-7594-45b3-9bd3-6d975316c00d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zaar