Features from left field

[Demis Bellot]

I'm curious on peoples thoughts are on a new feature where we can set-up the subscriber of a key (aka topic) to be a list / set so a copy of each key will be added to a list/set turning Redis Pub/Sub into a kind of durable Pub/Sub.

This is something I've discussed a little at the Redis London Meetup, but only now have had a chance to focus on it...

The initial use-case for this would be to keep a full-text search index (or potentially RDBMS) in sync with modified entities in redis.

Basically it would work something like this: A client could setup a rule with any of the following commands:

PSUBSCRIBESET    urn:customer:*    unique-modified-customer-keys-set-name

PSUBSCRIBELIST    urn:customer:*   queue-of-modified-customer-keys-list-name

PSUBSCRIBEZSET  urn:customer:*    time-ordered-unique-set-of-modified-customer-keys 

Then even after the client closes the connection the target set/list will get a copy of all the keys that have been modified by other clients.

Then somewhere in a external process far, far away it reads the keys, grab the corresponding entities, populate the search index and pops them from the list.

It is already possible to achieve a similar result with the current feature set however I think this approach is:

- the most loosely-coupled, transparent and re-usable amongst all clients (i.e. clients don't ever know there is an external process maintaining a full-text index).

- the most efficient - since the key never leaves the server

- the most resilient - survives net splits, as long as the server is up it should be accurate.

The compensating operations to 'delete the rule', would be something like:

PUNSUBSCRIBESET    unique-modified-customer-keys-set-name

PUNSUBSCRIBELIST    queue-of-modified-customer-keys-list-name

PUNSUBSCRIBEZSET  time-ordered-unique-set-of-modified-customer-keys 

Just so the redis-server doesn't deteriorate because of orphaned rules, we could also have a optional max size or expiry date that can dispose or put limits on the rule.

PSUBSCRIBESET    urn:customer:*    unique-modified-customer-keys-set-name  MAXSIZE 10000 

PSUBSCRIBESET    urn:customer:*    unique-modified-customer-keys-set-name  EXPIRESIN {some time in the future}

PSUBSCRIBESET    urn:customer:*    unique-modified-customer-keys-set-name  EXPIRESAT {some time in the future}

Also we don't need all three of them, I'd be happy with just PSUBSCRIBESET/PUNSUBSCRIBESET for the time being....

Thoughts?

--
- Demis

http://twitter.com/demisbellot

http://www.servicestack.net/mythz_blog

[Xiangrong Fang]

That is a very neat feature, but it seems too specific, for that kind of problem.  It does not "stand out" in the crowd of feature requests. 

Also you said it is already doable through current feature set, so I would vote a "0" on this idea, i.e. nice to have, but has no priority, and may unnecessarily complicate redis vocabulary (list of commands).

Alternatively I think a "server side trigger" would be very useful for a much wider range of problems.  I haven't think about it, just pop in my mind:

WATCH key TRIGGER proc ON [modify|delete]

where proc is a server-side script.  Which could be a lua script or others.  There might be other commands such as register a proc into redis...

Regards,

Shannon

2010/12/16 Demis Bellot <demis....@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Demis Bellot]

I would not say this is a specific use-case at all: a 'durable' Pub/Sub like this is a very powerful combination that is useful in many Enterprise Integration patterns where even a lot of Message Brokers today fail to implement properly.

You get the durability of a Message Queue with the loose-coupling offered by Pub/Sub - i.e. best of both worlds.

Note: being loosely coupled whilst maintaining data integrity is not currently possible with any of the other solutions.

Which pretty much negates anyone attempting to build a full-text / RDBMS-backed if its not going to maintain a 'true and accurate' mirror of the data in Redis.

Personally I still think its useful (and I would say necessary for some use-cases/tooling) even if only one of the commands were available, i.e:

PSUBSCRIBESET    urn:customer:*    unique-modified-customer-keys-set-name

Note: this is a non-invasive feature that will transparently benefit all clients (since its a once-off command that can be fired with redis-cli).

I would think this would be trivial to implement since the Pub/Sub messaging and SET functionality already exists.

- Demis

[Jak Sprats]

On the topic of durable pub-sub (and this is brainstorming)

Maybe a simpler approach is to have the following constructs
ZPUBLISH channel zzz msg
which would publish "msg" to the channel "channel" and do a ZADD to
the list "zzz" w/ score timestamp
Then users could do a
ZSUBSCRIBE zzz
which would dump the zset and also publish any additions to this zzz

The goal here is durable pub-sub, maybe just supporting ZSETs and
where the score is a timestamp and the value is a message is a way to
cut down on the requirements for durable pub-sub. The nice thing w/
ZSETs whose score is a timestamp is a "ZRANGE zzz 0 -1" will give the
last timestamp and then subscribing to pubs (into the ZSET) after that
timestamp would be possible ...

Just brainstorming, which on a mailing list can cause other people to
come up w/ better ideas :)

> > 2010/12/16 Demis Bellot <demis.bel...@gmail.com>

> >> redis-db+u...@googlegroups.com<redis-db%2Bunsu...@googlegroups.com>
> >> .

> >> For more options, visit this group at
> >>http://groups.google.com/group/redis-db?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "Redis DB" group.
> > To post to this group, send email to redi...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > redis-db+u...@googlegroups.com<redis-db%2Bunsu...@googlegroups.com>
> > .

[Demis Bellot]

Hi Jak,

 

ZPUBLISH channel zzz msg

This would provide a more explicit durable Pub/Sub like feature although it doesn't help the initial use-case much (of transparently maintaining a back-end RDBMS/index in sync) since it requires an explicit additional ZPUBLISH operation so is not triggered by a normal write operations (i.e. SET). We could already achieve a similar effect by forcing each client to do an explicit ZADD after each write operation, unfortunately this would be invasive as clients would be required to *know and perform* the additional required operation.

Although *it is* a more explicit Pub/Sub operation, so useful for other use-cases that need it.

Maybe a better syntax for my initial feature above would be better off with the following format:

ONWRITE {pattern} TRIGGER {command}

e.g:

ONWRITE urn:customer:* TRIGGER SADD updatedcustomerset $KEY 

Where $KEY is a special constant to represent the key that was modified.

This should open up a lot more flexibility on what can be achieved. Where {command} could be any existing operation with the results discarded.

Thoughts?

To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Pedro Melo]

Hi,

On Wed, Jan 5, 2011 at 9:57 PM, Demis Bellot <demis....@gmail.com> wrote:
> This would provide a more explicit durable Pub/Sub like feature although it
> doesn't help the initial use-case much (of transparently maintaining a
> back-end RDBMS/index in sync) since it requires an explicit additional
> ZPUBLISH operation so is not triggered by a normal write operations (i.e.
> SET). We could already achieve a similar effect by forcing each client to do
> an explicit ZADD after each write operation, unfortunately this would be
> invasive as clients would be required to *know and perform* the
> additional required operation.
> Although *it is* a more explicit Pub/Sub operation, so useful for other
> use-cases that need it.
> Maybe a better syntax for my initial feature above would be better off with
> the following format:

If you want to monitor writes, and given that we already have a
MONITOR command, I believe MONITOR WRITES is the best solution.

Still, same problems as other solutions: if the processing MONITOR
WRITES dies, you'll loose events; if you have multiple MONITOR WRITES
running for performance/reliability, all of them receive all the
events.

I suppose a "MONITOR WRITES shared_secret", where "shared_secret"
would be shared by multiple workers and only one of them receives the
event would help, if you don't have ordering issues that is.

Basically (from AMQP world), you want a binding that listens to writes
and feeds a single mailbox, and that mailbox can be consumed by a set
of workers. Thats the most reliable way of doing it.

Bye,
--
Pedro Melo
http://www.simplicidade.org/
xmpp:me...@simplicidade.org
mailto:me...@simplicidade.org

[Jak Sprats]

>> ONWRITE urn:customer:* TRIGGER SADD updatedcustomerset $KEY

If you look at the pub-sub C code, this is a much cleaner fit...

on publish "$KEY" is already a sds, so doing the trigger's command
substitution is a no-brainer.
Adding a flag "trigger" to a channel is pretty easy.
Running the "SADD updatedcustomerset $KEY" can be done by using
"call()"

If you want we can offline this, and come up w/ a minimalistic patch
(one new command, and a couple of entry points is all that should be
needed). you know C? email me if you want to.

> > <redis-db%2Bunsu...@googlegroups.com<redis-db%252Buns...@googlegroups.com>

>
> > > >> .
> > > >> For more options, visit this group at
> > > >>http://groups.google.com/group/redis-db?hl=en.
>
> > > >  --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Redis DB" group.
> > > > To post to this group, send email to redi...@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > > > redis-db+u...@googlegroups.com<redis-db%2Bunsu...@googlegroups.com>

> > <redis-db%2Bunsu...@googlegroups.com<redis-db%252Buns...@googlegroups.com>

[Demis Bellot]

Hey Jak,

Sounds like a plan, tho my free time is tied up for the next couple of weeks on a another project. 

I'll drop you a line when I come into some free time again.

Learnt C at Uni (Many moons ago :) only recent exposure was some C libs for a couple of recent iPhone apps, so its pretty rusty atm, will definitely need peer review :)

To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.