How to store tens of billions counters in redis?

[唐福林]

we now have tens of billions counters that we want to store in redis.

but redis treat all keys and values as string , which cost too much
ext memories !

eg, key is a long type num (id) , and value is just a int type num .
but in redis, it cost more then 80 bytes to store one counter like
this .

but we have too much counters, so , any suggestion?

[Salvatore Sanfilippo]

Hello, there are two possibilities.

1) Use hashes to group, let's say, 256 counters per hash. This way you
can reduce the memory usage by 10 times more or less.
2) Use 2.2 GETRANGE / SETRANGE together with MULTI/EXEC if you found a
way to avoid too much contention. So in every key you have N counters
as a single array of bytes. But this requires you to be able to index
N counters with just one key and an offset. Too little details to
understand if you can do this or not.

Cheers,
Salvatore

>
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware

http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[唐福林]

if we use hash, how to do multi get (not only get multi fields in a
hash, but also get multi hashes) ?

can we store all counters in one big hash ? any performance defect of
doing this ?


On Jan 13, 4:46 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:

> On Thu, Jan 13, 2011 at 3:32 AM, 唐福林 <tangfu...@gmail.com> wrote:
> > we now have tens of billions counters that we want to store in redis.
>
> > but redis treat all keys and values as string , which cost too much
> > ext memories !
>
> > eg, key is a long type num (id) , and value is just a int type num .
> > but in redis, it cost more then 80 bytes to store one counter like
> > this .
>
> > but we have too much counters, so , any suggestion?
>
> Hello, there are two possibilities.
>
> 1) Use hashes to group, let's say, 256 counters per hash. This way you
> can reduce the memory usage by 10 times more or less.
> 2) Use 2.2 GETRANGE / SETRANGE together with MULTI/EXEC if you found a
> way to avoid too much contention. So in every key you have N counters
> as a single array of bytes. But this requires you to be able to index
> N counters with just one key and an offset. Too little details to
> understand if you can do this or not.
>
> Cheers,
> Salvatore
>
>
>
> > --
> > You received this message because you are subscribed to the Google Groups "Redis DB" group.
> > To post to this group, send email to redi...@googlegroups.com.
> > To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/redis-db?hl=en.

[Pieter Noordhuis]

You can only get fields in one hash at a time. There's HMGET or HGETALL for that. You can send multiple commands that fetch counters from separate hashes in a pipeline ( http://redis.io/topics/pipelining ) to minimize impact on the throughput.

Are your currently using MGET to fetch multiple counters in one go, and if so, how many keys do you fetch per command?

Cheers,

Pieter

[catwell]

You could also implement a command that does what you want in Redis
itself if you are willing to maintain a fork. For instance I
implemented a command that adds a value for the same key in several
hashes (MHSET key hash_key1 value1 hash_key2 value2 ...). See:

https://github.com/catwell/redis/commit/39a72ac98f3d127f1e1743bbcfecd9a77228c73f
https://github.com/catwell/redis/commit/f543041424adbf12dc345108f982fab0c5dadc47

It is not atomic yet (meaning it could fail in the middle of the
process) but I don't really care for now, as long as it is fast... If
you implement its READ version (MHGET?) then you could probably
achieve what you're trying to do.

By the way, I'm new in this group so hello everybody ;)

[Arrix]

While a big hash can store a lot of keys without problems, it doesn't
save memory for you.
The first option suggested by @antirez only groups a few (e.g. 256)
counters in one hash.
Doing this will save a lot of memory because redis optimizes memory
usage for small hashes by using a compact encoding rather than real
hash tables.
I think future redis should do this by default transparently.
For now, you can use this trick at the client side.
See this thread for a python hash layer implementation which simulates
MGET using pipeline. http://groups.google.com/group/redis-db/browse_thread/thread/39ba2023be2694bb/b117c564a083b204

You should also make sure the hash-max-zipmap-entries setting is
larger than your group size.

[唐福林]

We fetch about 200 keys per command. Even worse, we have 3 counters
for one id!

On Jan 14, 5:57 pm, Pieter Noordhuis <pcnoordh...@gmail.com> wrote:
> You can only get fields in one hash at a time. There's HMGET or HGETALL for
> that. You can send multiple commands that fetch counters from separate

> hashes in a pipeline (http://redis.io/topics/pipelining) to minimize

> > redis-db+u...@googlegroups.com<redis-db%2Bunsubscribe@googlegroups.c om>
> > .

> > > > For more options, visit this group athttp://
> > groups.google.com/group/redis-db?hl=en.
>
> > > --
> > > Salvatore 'antirez' Sanfilippo
> > > open source developer - VMware
>
> > >http://invece.org
> > > "We are what we repeatedly do. Excellence, therefore, is not an act,
> > > but a habit." -- Aristotele
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Redis DB" group.
> > To post to this group, send email to redi...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > redis-db+u...@googlegroups.com<redis-db%2Bunsubscribe@googlegroups.c om>
> > .