Client notifications/triggers

[George Porter]

Hello,

I was wondering if Hyperdex supported the ability for clients to register their interest in certain changes to the data store, similar to how Redis works.  For example, a client might want to "monitor" a value and get notified whenever that value changes.

The specific use case I was thinking of was a set of clients submitting work to a central work queue (maybe implemented as a list).  One of a few workers would monitoring that queue, and when a new item is put into it a single worker atomically dequeues work and executes it.  Presumably this could be done by polling, but didn't know if there was some feature in Hyperdex that might make this work without polling.

Thanks, George

[Robert Escriva]

Hi George,

Thanks for requesting this feature again. People have requested
pub/sub-like functionality in the past. It's a popular feature request
and we've put it on our TODO list.

From what I've seen, the primary motivation for the pub/sub feature is
usually a high-performance work queue.

If you had available a dedicated work-queue service which was more
efficient than a key-value store, would you still need pub/sub in the
key-value store?

-Robert

[George Porter]

On Jul 27, 2012, at 10:31 AM, Robert Escriva <esc...@cs.cornell.edu> wrote:
>
> Thanks for requesting this feature again. People have requested
> pub/sub-like functionality in the past. It's a popular feature request
> and we've put it on our TODO list.

Ah, apologies for asking multiple times. I did a search through the Google Groups archive and didn't see anything about this topic, but hadn't checked the TODO.

> From what I've seen, the primary motivation for the pub/sub feature is
> usually a high-performance work queue.
>
> If you had available a dedicated work-queue service which was more
> efficient than a key-value store, would you still need pub/sub in the
> key-value store?

I would say yes. The model I'm considering is where a variety of systems are pushing/pulling data to/from the key-value store, but there is still a need for a "control plane." For example, imagine that nodes push status information like system utilization, systems metrics, etc. But every now and then they want to receive a control message from some central controller. In that case I could have each machine poll for new commands from a queue, but it would also be good to have each machine just subscribe to a queue and get notified when there is a new message for it.

You're right that these could be two separate systems, but the control and key/value content isn't unrelated, and so having a single data store would be convenient.

Thanks, George

>
> -Robert
>
> --
>
>

[Robert Escriva]

On Fri, Jul 27, 2012 at 01:47:14PM -0400, George Porter wrote:
> On Jul 27, 2012, at 10:31 AM, Robert Escriva <esc...@cs.cornell.edu> wrote:
> >
> > Thanks for requesting this feature again. People have requested
> > pub/sub-like functionality in the past. It's a popular feature request
> > and we've put it on our TODO list.
>
> Ah, apologies for asking multiple times. I did a search through the
> Google Groups archive and didn't see anything about this topic, but
> hadn't checked the TODO.

No worries. The repeated requests help us understand what new features
would be most useful so we can direct our efforts to add them.

> > From what I've seen, the primary motivation for the pub/sub feature is
> > usually a high-performance work queue.
> >
> > If you had available a dedicated work-queue service which was more
> > efficient than a key-value store, would you still need pub/sub in the
> > key-value store?
>
> I would say yes. The model I'm considering is where a variety of
> systems are pushing/pulling data to/from the key-value store, but
> there is still a need for a "control plane." For example, imagine
> that nodes push status information like system utilization, systems
> metrics, etc. But every now and then they want to receive a control
> message from some central controller. In that case I could have each
> machine poll for new commands from a queue, but it would also be good
> to have each machine just subscribe to a queue and get notified when
> there is a new message for it.
>
> You're right that these could be two separate systems, but the control
> and key/value content isn't unrelated, and so having a single data
> store would be convenient.
>
> Thanks, George
>
> > -Robert

I see the utility in having the key-value store notify on update. I can
also imagine many different optimizations and conveniences that a
standalone service would provide above and beyond those that come from
pub/sub in a key-value store. It is always possible to refer to content
in the key-value store from a separate service.

In any case, we are considering both options and will keep you and the
community updated with our ideas.

Thanks,
Robert

[Holger Winkelmann]

HI, We just considering to use HyperDex as a datastore for our distributed applications.

The Change Notification Support is on the feature list as well. This could work similar

to postgres NOTIFY / LISTEN support:

http://www.postgresql.org/docs/9.1/static/sql-notify.html

We are not looking to model a Message Queue on top of a KVS.

Is there an progress on notifictaion or a hint how to do this best?

Many thanks,

Holger

[Robert Escriva]

Hi Holger,

HyperDex doesn't support notifications at the moment. The nature of
notify/listen is very much like a 1-1 or m-n pub/sub system. Given that
this differs significantly from a key-value store, we've not added this
feature yet, but have not ruled it out entirely.

HyperDex does work well with an existing pub/sub system, many of which
have message size limitations, or have performance degrade with complex
messages. You can add the large payload to HyperDex, and pass the key
around via the pub/sub system.

Best,
Robert

> --
> You received this message because you are subscribed to the Google Groups
> "hyperdex-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to hyperdex-discu...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.