Fast consistent backup solution?

[Blair Zajac]

I was thinking about getting a consistent backup from Gerrit and the SQL
db without shutting Gerrit down so that users would be minimally
impacted. I don't want to stop Gerrit and then restart it for the downtime.

I haven't looked at the code to see how feasible this is or how much
work. I'm also writing this after a long day. Is it worth the effort
now given that the SQL db will be removed?

Gerrit would add another thread that would wait for a write to git or
SQL and then wait for Gerrit to become quiescent, then it will block
writes to git and SQL and signal an external process to take a backup,
when the backup completes, it will release the write lock in Gerrit.

Details:

The hardest part would be putting in the appropriate locking in Gerrit.
I haven't looked at the code, how hard would this be? I haven't
totally designed this part, but it would have a ReentrantReadWriteLock
and some other locking primitive. How much work and how many places
would this have to be done? Are there a few places that one could
"inject" this code?

A fast backup could be accomplished by putting the git repos on a LVM2
volume, btrfs or zfs filesystem. The SQL backup could be started and
then the Gerrit write lock could be released. I'm assuming/hoping that
once a SQL backup has started, one could release Gerrit and the backup
process wouldn't see any new writes.

A simple way for the external process to run would be to use
ProcessBuilder.start(), but I'm concerned with the cost in forking a
process with a decent amount of memory, so having a child process listen
on a pipe would be more efficient.

Feedback welcome, I'll probably start on this soon if its worth the
effort, assuming that the SQL removal work is a decent ways away.

Blair

[Luca Milanesio]

Doesn't seem complex and it make sense to me.

SQL will disappear, but in the meantime hot-backup is definitely needed.
We could give you a hand in getting this done.

Luca.

> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en

[Shawn Pearce]

On Thu, Jan 10, 2013 at 10:37 PM, Blair Zajac <bl...@orcaware.com> wrote:
> I was thinking about getting a consistent backup from Gerrit and the SQL db
> without shutting Gerrit down so that users would be minimally impacted. I
> don't want to stop Gerrit and then restart it for the downtime.
>
> I haven't looked at the code to see how feasible this is or how much work.
> I'm also writing this after a long day. Is it worth the effort now given
> that the SQL db will be removed?
>
> Gerrit would add another thread that would wait for a write to git or SQL
> and then wait for Gerrit to become quiescent, then it will block writes to
> git and SQL and signal an external process to take a backup, when the backup
> completes, it will release the write lock in Gerrit.
>
> Details:
>
> The hardest part would be putting in the appropriate locking in Gerrit. I
> haven't looked at the code, how hard would this be? I haven't totally
> designed this part, but it would have a ReentrantReadWriteLock and some
> other locking primitive. How much work and how many places would this have
> to be done? Are there a few places that one could "inject" this code?

This probably needs to be in 3 places:

- Filter on /* in the HTTP servlet stack. Put it early enough before
we attempt any database access for this request, e.g. in the top of
WebModule's configure method after wantSSL. Grab the lock while the
request is running and release the lock in a finally when the request
is done.

- SSH command execution. We already track what commands are "running"
so they are visible to admins. The lock should be in similar parts of
the code so that the lock is held while the command is "running".

- MergeOp. The merge queue does writes (obviously). Anytime a MergeOp
is running the lock should be held.

That might cover everything in the server.

> A fast backup could be accomplished by putting the git repos on a LVM2
> volume, btrfs or zfs filesystem. The SQL backup could be started and then
> the Gerrit write lock could be released. I'm assuming/hoping that once a
> SQL backup has started, one could release Gerrit and the backup process
> wouldn't see any new writes.

Depends on your database. For a PostgreSQL or H2, yes. The backup
should be able to execute in a transaction. The SQL metadata tends to
be very small, so the backup will easily complete before the database
would run out of snapshot storage. On MySQL... well, its MySQL. If you
wanted transaction safety, you wouldn't have installed it, so you also
probably don't want consistent backups. Maybe InnoDB can do a
consistent backup, but I just always say no to MySQL.

> A simple way for the external process to run would be to use
> ProcessBuilder.start(), but I'm concerned with the cost in forking a process
> with a decent amount of memory, so having a child process listen on a pipe
> would be more efficient.

Java 7 does this better. Java 6 yes its slow to fork the large server
process to start a small child. But I think I would flip this around
and use an SSH command.

Have your backup script SSH into gerrit and run something like `gerrit
backup begin` to freeze the system and take a write lock. So your
backup is run out of cron and looks like:

#!/bin/sh
label=$(date)

ssh -P 29418 'Gerrit Code Review@localhost' gerrit backup begin;
snapshot zfs
psql -c "SELECT pg_start_backup('$label');"
ssh -P 29418 'Gerrit Code Review@localhost' gerrit backup end;

backup postgres
psql -c "SELECT pg_stop_backup();"

backup zfs snapshot

Between the period of backup begin/backup end, Gerrit will stall user
requests. The HTTP filter might want to use a timeout on the lock and
report a 503 error response to clients if it goes on for too long. SSH
could do something similar and report an error over stderr and abort
the command if the period goes longer than something reasonable, e.g.
1 minute or 5 minutes.

You could in the SSH code try to annotate commands that are read-only
and thus "safe". E.g. git upload-pack is used for clone/fetch and
won't cause database or Git modifications. There is no reason these
can't run concurrently with a backup. Likewise HTTP REST API calls
that are using method GET are also safe, these won't cause state
changes on the server.