Common Backup Procedures?

[Chris Chew]

Hello.

We’re preparing to deploy a Voldemort cluster with a 3x or 4x
replication factor, but we want to be prepared for the disaster
situation where we happen to lose all the 3 or 4 nodes which contain a
particular keyset.

I am hoping people might share some information on how they are
backing up their Voldemort clusters.

For backups, do you simply perform hot copies of the bdb/mysql data
filesystems (if bdb, copied in alphabetical order)? Or do you
maintain a separate, canonical storage system that can be used to
reload Voldemort when recovering from a disaster?

For recovery, what is your procedure if you lose just one node? Do
you just clear out the data, restart it, and let it eventually become
consistent again or do you prime the data files with the most recent
backup before starting the node back up? And is the procedure the
same if you lose multiple nodes?

Thank you in advance,

Chris Chew

[Alex Feinberg]

Hot copies of BDB files would suffice. As you've said, alphanumeric
order is important. Default rsync options, for example, would mean the
files won't be copied in order; this can result in bad copies of data.

There's also an Admin interface available to Voldemort, which allows
for programmatic bulk transfers of specific stores and partitions from
specific nodes:

http://project-voldemort.com/blog/2009/12/introducing-adminclient-apis/

You could use this API, for example, to perform hot backups via a
Hadoop Map/Reduce job (to HDFS)

As for recovery, provided there are enough in machines with *correct*
data to constitute a quorum (i.e. R + W > N), read-repair will be
performed. That, however, may take an indefinite amount of time.
There's an issue open for finishing our Hinted Hand off implementation
(where data is written to the first N healthy nodes in the hash ring
for key K, even if these nodes aren't originally responsible for the
key), which should allow transient failures (e.g. node goes down for a
short period of time and then comes back) to be handled in a more
graceful manner:

http://code.google.com/p/project-voldemort/issues/detail?id=118

Thanks,
- Alex

> --
>
> You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> To post to this group, send email to project-...@googlegroups.com.
> To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
>
>
>

[Chris Chew]

Alex,

Thank you for your detailed response, I apologize for the long delay
for mine. Also, thank you for pointing out the issue with Hinted
Hand-off. We'll keep an eye on it.

We're going to start with taking hot copies of the BDB files since
we're really still just dipping our toes in the water here and it's
the path of least resistance for the organization. We have plans to
eventually set up a Hadoop/BashReduce/etc environment that will
process the data and store it in some other system for query-ability
-- perhaps that will become a better backup scenario.

I'll be sure to post back to the group with any lessons learned.

Thank you all for such a great project!

Chris