Redis as caching engine for large datasets that do not fit in memory?
Vincent Chavelle
few (50 000) can fit in memory (the one that are the most asked). Is
Redis fast enough for this use? With virtual memory or not?
Thanks
Salvatore Sanfilippo
this is a use case for VM, but not when the difference between in-ram
and not-in-ram is so big.
Do you need persistence in this application?
Btw in Redis unstable the VM is replaced by a new idea called
"diskstore" that address your use case, but unfortunately it is not
production ready at this stage.
Cheers,
Salvatore
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele
Vincent Chavelle
this is a use case for VM, but not when the difference between in-ram
and not-in-ram is so big.
Do you need persistence in this application?
Btw in Redis unstable the VM is replaced by a new idea called
"diskstore" that address your use case, but unfortunately it is not
production ready at this stage.
pretty well, without obvious crashes. But warning, it will crash with
an assert if you try to BGSAVE.
a.
that exist in memcached too), or when you have some millions of items
then you're better off with membase.
Redis has is obscenely slow at startup and when snapshotting,
especially if you enable vm as well.
On Feb 8, 11:24 am, Vincent Chavelle <vincent.chave...@gmail.com>
wrote:
Salvatore Sanfilippo
> If you don't use "redis specific" commands, (as opposed to the ones
> that exist in memcached too), or when you have some millions of items
> then you're better off with membase.
>
> Redis has is obscenely slow at startup and when snapshotting,
> especially if you enable vm as well.
>
Redis without VM enabled is very fast to start up even with millions of keys.
Example, 40 millions of keys in my desktop took less than 60 seconds
to start (.rdb persistence).
We have a problem with VM, but on the other hand with Diskstore that
is the replacement startup time is zero, but we need a few months to
get this stable.
Cheers,
Salvatore
> On Feb 8, 11:24 am, Vincent Chavelle <vincent.chave...@gmail.com>
> wrote:
>> Thanks for your reply
>>
>> this is a use case for VM, but not when the difference between in-ram
>>
>> > and not-in-ram is so big.
>> > Do you need persistence in this application?
>>
>> Yes, because third-party applications are slow to create pages...
>>
>> > Btw in Redis unstable the VM is replaced by a new idea called
>> > "diskstore" that address your use case, but unfortunately it is not
>> > production ready at this stage.
>>
>> Stability: the system is obviously in alpha stage, however it works
>> pretty well, without obvious crashes. But warning, it will crash with
>> an assert if you try to BGSAVE.
>>
>> It's still valid? I realy want to use Redis for that (and not membase).
>> I only use the Keys section of the doc : GET, SET, KEYS, DEL, EXPIRE etc...
>
Vincent Chavelle
If you don't use "redis specific" commands, (as opposed to the ones
that exist in memcached too), or when you have some millions of items
then you're better off with membase.
Redis has is obscenely slow at startup and when snapshotting,
especially if you enable vm as well.
On Feb 8, 11:24 am, Vincent Chavelle <vincent.chave...@gmail.com>
wrote:
> Thanks for your reply
>
> this is a use case for VM, but not when the difference between in-ram
>
> > and not-in-ram is so big.
> > Do you need persistence in this application?
>
> Yes, because third-party applications are slow to create pages...
>
> > Btw in Redis unstable the VM is replaced by a new idea called
> > "diskstore" that address your use case, but unfortunately it is not
> > production ready at this stage.
>
> Stability: the system is obviously in alpha stage, however it works
> pretty well, without obvious crashes. But warning, it will crash with
> an assert if you try to BGSAVE.
>
> It's still valid? I realy want to use Redis for that (and not membase).
> I only use the Keys section of the doc : GET, SET, KEYS, DEL, EXPIRE etc...
Tim Lossen
be careful with the KEYS command:
> Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using sets.
tim
Salvatore Sanfilippo
> vincent,
>
> be careful with the KEYS command:
Good advice. How to do this instead? Adding the keys you want to track
into a set or a sorted set or a list (depends on the use case).
Cheers,
Salvatore
a.
> Redis without VM enabled is very fast to start up even with millions of keys.
>
> Example, 40 millions of keys in my desktop took less than 60 seconds
> to start (.rdb persistence).
> We have a problem with VM, but on the other hand with Diskstore that
> is the replacement startup time is zero, but we need a few months to
> get this stable.
Salvatore Sanfilippo
>
> That's why i said that currently he's better off with something else.
You said that Redis is slow saving/loading *especially* with VM, that
implies that also without VM there are problems, that is not true.
Salvatore
a.
loading times with snapshot mode on databases having ~5M zesets (each
having 1-50 items). For me 10 minutes on a small database is "slow".
(With VM this was even slower.)
a.
On Feb 9, 10:58 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> On Wed, Feb 9, 2011 at 11:55 PM, a. <a...@enyim.com> wrote:
>
> > That's why i said that currently he's better off with something else.
>
> You said that Redis is slow saving/loading *especially* with VM, that
> implies that also without VM there are problems, that is not true.
>
> Salvatore
>
> > --
> > You received this message because you are subscribed to the Google Groups "Redis DB" group.
> > To post to this group, send email to redi...@googlegroups.com.
> > To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
Salvatore Sanfilippo
> Well, back in december when i was testing redis i had ~8-10 minutes of
> loading times with snapshot mode on databases having ~5M zesets (each
> having 1-50 items). For me 10 minutes on a small database is "slow".
> (With VM this was even slower.)
5M zsets means 5M keys, but how much objects there are inside? That's the point.
If there are 100 elements inside every sorted set, guess what, you
have 500 millions of objects to load from disk.
We can improve in this side, but if you think at it, with in memory
databases you can't escape I/O to load and save, ever.
For instance, with diskstore, or with any other solution involving a
cache and a store (membase uses this maybe) so that with the working
set it will be as fast as an in ram DB, what happens when you just
started, if the application is hitting the DB at a rate that is faster
than the time it takes for the disk to transfer things in RAM?
You are going to busy wait the loading.
The difference is that since Redis is *designed* for this, you get a
lot of extras that will help about that, that is, a CONFIG SET command
to hack the configuration while the DB is running without any need to
restart and things like that.
No free lunch basically.
Cheers,
Salvatore