Question about data model

[Jesús Gabriel y Galán]

Hi,

I need to implement an API for a platform that allows users to
subscribe to services, and I have a question about what would be the
best way to implement this in Redis. Basically there are many
services. Each service has several subscription types. A user can
subscribe to one or more subscription types in a service. When he does
so, he is assigned a subscriptionId. The API has three calls:

- subscribe, which receives user_id, service_id, subscription_type and
subscription_id and stores the subscription
- get subscriptions for a user in a service, which receives a user_id
and a service_id and returns an array of subscriptions (one entry per
subscription within that service)
- unsubscribe, which receives user_id, service_id and subscription_id
and removes that subscription.

A subscription also has other data associated like period,
billing_status, date_of_next_billing, etc that are read and updated by
a recurring job. There should be a job for each subscription (I'm
planning to do this with beanstalkd).

What I have now is a hash for each subscription keyed by the
subscription_id, with all the relevant fields of that subscription
(billing_status, date, etc) and a list of subscription_id, keyed by
user:service

With this model, I have the following queries (sinatra app using redis
ruby client):

post '/subscribe' do
[... params from request...]
settings.redis.rpush "u:#{user_id}:#{service_id}", subscription_id
setting.redis.hmset "si:#{subscription_id}", "m", user_id, "st",
subscription_type, "s", billing_status
settings.beanstalk.put(subscription_id)
end

post '/unsubscribe' do
[... params from request...]
settings.redis.lrem "u:#{user_id}:#{service_id}", 0, subscription_id
settings.redis.del "si:#{subscription_id}"
end

get '/subscriptions' do
[... params from request...]
key = "u:#{user_id}:#{service_id}"
subs_list = settings.redis.lrange key, 0, -1
result = settings.redis.pipelined do
subs_list.each do |sub|
settings.redis.hgetall("si:#{sub}")
end
end
end

The first two are pretty straightforward, but getting all the
subscriptions in a service involves reading all entries in the list
with the subscription ids, then iterating through it getting all the
hash entries for each of them. I pipeline the hgetalls, but still I
feel there must be a better way to model this.

The beanstalkd job is basically the subscription_id, it goes to Redis
to read the info from the hash, performs some work, updates the Redis
hash and reschedules the job for the next time.

Is there a better way to model this, or with this model, is there a
more efficient way to perform the queries?

Thanks,

Jesus.

[Felix Gallo]

Are you having a performance problem, or are you just worried about cleanliness?

If it's actual measured performance, then what I tend to do for highly static data is to trade space for time and run the query pre-fork (cf. http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-before_fork) and cache the data (here, the subscriptions metadata) in ruby data structures.

The downside of this is that you have to signal the app server (e.g. http://unicorn.bogomips.org/SIGNALS.html) whenever your static data changes significantly enough, in order to force a cache invalidation.  If you are feeling sufficiently frisky you could even have the app server detect this condition and revalidate itself, but in practice I haven't found that to be worth the time spent.

Generally, even for a fairly heavily loaded app server doing upwards of thousands of requests per second, as long as the redis server is in the same data center, this optimization is of little discernible benefit.

F.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Jesús Gabriel y Galán]

On Mon, Nov 19, 2012 at 6:46 PM, Felix Gallo <felix...@gmail.com> wrote:
> Are you having a performance problem, or are you just worried about
> cleanliness?

The second. I'm still in the exploratory phase, in fact the final
product might not even be in Ruby.
I was trying to wrap my head around best ways of modelling data with
Redis' datastructures, so that I'm not missing something obvious in
how to store this information for the type of queries I need.

> If it's actual measured performance, then what I tend to do for highly
> static data is to trade space for time and run the query pre-fork (cf.
> http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-before_fork)
> and cache the data (here, the subscriptions metadata) in ruby data
> structures.
>
> The downside of this is that you have to signal the app server (e.g.
> http://unicorn.bogomips.org/SIGNALS.html) whenever your static data changes
> significantly enough, in order to force a cache invalidation. If you are
> feeling sufficiently frisky you could even have the app server detect this
> condition and revalidate itself, but in practice I haven't found that to be
> worth the time spent.
>
> Generally, even for a fairly heavily loaded app server doing upwards of
> thousands of requests per second, as long as the redis server is in the same
> data center, this optimization is of little discernible benefit.

In this case the beanstalkd job updates the subscription info so the
appserver should be notified everytime a job completes, which is very
frequently, so I'm not sure this calls for such complexity.
Interesting strategy though, for other cases, thanks.

Thanks for your answer.

Jesus.