Re: Restarting when AOF and RDB are both present

[Matt Palmer]

On Wed, Apr 24, 2013 at 11:40:23PM -0700, Rajiv Kurian wrote:
> From the Redis documentation it seems like when both AOF and RDB options
> are enabled the AOF file is used when restarting. Would it not be faster to
> use both the RDB and the AOF file for recovery. We could associate the a
> change number with every mutation of a key on the AOF file. when a RDB
> snapshot is made each key could also store the change number corresponding
> to the version being snapshotted. On a restart the RDB could be loaded
> first and then for each key we would only need to replay the events post
> the change number for the key in RDB instead of all the events. AOF and RDB
> could still work independently but if both are enabled recovery would be
> faster. It also seems like Redis could get away with having only one
> persistence option that is a blend of the current AOF and RDB options.

This was discussed about a year ago:

https://groups.google.com/d/msg/redis-db/viEBw0hDOas/jv9S5ER-v_kJ

There are some implementation ideas and caveats scattered throughout that
thread that you'll want to make yourself aware of if you're looking to
implement this yourself. I don't believe there are any specific plans by
anyone else to build this.

I've been working on an alternate persistence mechanism recently, which
incorporates fast start up, incremental updating (which means you can flush
modified keys to disk very frequently), and the ability to keep only a
subset of keys in memory (with the least-used keys residing on disk). It's
very, very experimental, but it'll be quicker to wring the bugs out of this
than write an AOF/RDB hybrid. An introductory blog post about it, with
links to source, etc, is available at:

http://www.anchor.com.au/blog/2013/04/redis-rethought-exciting-extremes-with-larger-than-memory-datasets/

- Matt

--
"I have a cat, so I know that when she digs her very sharp claws into my
chest or stomach it's really a sign of affection, but I don't see any reason
for programming languages to show affection with pain."
-- Erik Naggum, comp.lang.lisp

[Salvatore Sanfilippo]

Hello,

the development plan about that is to rewrite the AOF using an RDB
pre-amble, in a single file like today.
It is also possible that the AOF appended entries will use a binary
format, using an opcode for every available command so that we can use
a format like that:

opcode - argc - len - arg - len -arg - ...

Cheers,
Salvatore

On Thu, Apr 25, 2013 at 8:40 AM, Rajiv Kurian <geet...@gmail.com> wrote:
> From the Redis documentation it seems like when both AOF and RDB options are
> enabled the AOF file is used when restarting. Would it not be faster to use
> both the RDB and the AOF file for recovery. We could associate the a change
> number with every mutation of a key on the AOF file. when a RDB snapshot is
> made each key could also store the change number corresponding to the
> version being snapshotted. On a restart the RDB could be loaded first and
> then for each key we would only need to replay the events post the change
> number for the key in RDB instead of all the events. AOF and RDB could still
> work independently but if both are enabled recovery would be faster. It also
> seems like Redis could get away with having only one persistence option that
> is a blend of the current AOF and RDB options.
>

> Thanks,
> Rajiv
>
> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Rajiv Kurian]

Thanks for the update Salvatore!