Tonight I'll present on backups; what to backup and how to backup, and
how to recreate Apple’s Time Machine with a one line command. The
presentation will be about 30-45 minutes and we’ll socialize after —
stay as long as you want to — if you have any computer questions,
bring ‘em!
See you tonight.
- Seth
Weber County Main Library, Board Room
2464 Jefferson Ave.
Ogden, UT 84401
------Original Message------
From: Seth House
Sender: oalug@googlegroups.com
To: Ogden Area Linux Users Group
ReplyTo: oalug@googlegroups.com
Subject: [oalug] Reminder: Meeting tonight on Backups
Sent: Jun 30, 2009 8:59 AM
Hey, all. Sorry for the late reminder.
Tonight I'll present on backups; what to backup and how to backup, and
how to recreate Apple’s Time Machine with a one line command. The
presentation will be about 30-45 minutes and we’ll socialize after —
stay as long as you want to — if you have any computer questions,
bring ‘em!
See you tonight.
- Seth
Weber County Main Library, Board Room
2464 Jefferson Ave.
Ogden, UT 84401
On Tue, Jun 30, 2009 at 8:59 AM, Seth House<whitei...@gmail.com> wrote:
> Tonight I'll present on backups; what to backup and how to backup, and > how to recreate Apple’s Time Machine with a one line command. The > presentation will be about 30-45 minutes and we’ll socialize after — > stay as long as you want to — if you have any computer questions, > bring ‘em!
For those of us who couldn't attend, what is the magic one line command? And are there any slides or materials from the presentation available anywhere?
On Jul 1, 3:53 pm, Michael Heath <mike.thomas.he...@gmail.com> wrote:
> For those of us who couldn't attend, what is the magic one line
> command?
Run this command:
rsync -a $HOME/ /var/timemachine/$USER/`date '+%FT%R'`
Then put this in your crontab (all on one line):
1 */2 * * * BACKUPDIR=/var/timemachine/$USER; rsync -a
--link-dest=$BACKUPDIR/`/bin/ls -t $BACKUPDIR | head -1` $HOME/
$BACKUPDIR/`date '+%FT%R'`
> And are there any slides or materials from the presentation available
> anywhere?
The slides are available in the group Files section. Be sure to read
through the handout notes (print the HTML file) as they’re pretty
conversational and will give you a good feel for what was said at the
meeting.
How often does that cron job run? I finally started creating backups when I
accidentally recursively changed the permissions of my friend's home to 000.
(it took me about 1 hour to get everything back to the way it was...)
On Wed, Jul 1, 2009 at 4:54 PM, Seth House <whitei...@gmail.com> wrote:
> On Jul 1, 3:53 pm, Michael Heath <mike.thomas.he...@gmail.com> wrote:
> > For those of us who couldn't attend, what is the magic one line
> > command?
> Run this command:
> rsync -a $HOME/ /var/timemachine/$USER/`date '+%FT%R'`
> Then put this in your crontab (all on one line):
> 1 */2 * * * BACKUPDIR=/var/timemachine/$USER; rsync -a
> --link-dest=$BACKUPDIR/`/bin/ls -t $BACKUPDIR | head -1` $HOME/
> $BACKUPDIR/`date '+%FT%R'`
> > And are there any slides or materials from the presentation available
> > anywhere?
> The slides are available in the group Files section. Be sure to read
> through the handout notes (print the HTML file) as they’re pretty
> conversational and will give you a good feel for what was said at the
> meeting.
On Jul 3, 6:49 am, "Roly C." <rol...@gmail.com> wrote:
> How often does that cron job run?
1 */2 * * * somecommand
``somecommand`` runs every two hours, one minute after the top of the
hour. The crontab manpage has a bunch of easy to understand examples.
(It's actually a typo, the one in my crontab runs at the top of the
hour.)
I'm playing with an idea similar to this for incrimental backups. I read an article recently that discussed unionfs for backing up like this. Basically, to snapshot a folder, you create a union ontop of the directory, and then create another directory, like /.snapshots/home-since-20090702. You add this new directory to the union, and set the original RO and the new directory to RW. You can then add any further snapshots by simply creating a new directory, adding it to the union, and setting the older dirs read only.
I've been working on some scripts for this, and I'm going to work on a python plugin for nautilus that adds right click functionality like "revert to version as of DATE". If anyone is interested, I'll share the finished products here.
(Please pardon any bizarre formatting in this message. I'm out of town and messaging from my phone.)
What would be really interesting is to benchmark it after each step and plot out the performance degradation--it has to exist because you are keeping snapshots live, the question is how much of an impact does it have.
FYI, for incrementals, I just use gnu tar with --newer-mtime (infact I have attached an old script I made in solaris to make it behave like the old unix 'dump').
I recommend monthly fulls, weekly differential and daily incremental (for this I use a different script, which is part of a bigger infrastructure, but uses the same --newer-mtime method). On one site I do this to a DataDomain de-dup device / nfs share. It has 40TB of "real" storage but currently we have 300TB of actual data stored only in about 20TB of real disk (it is a cool device... I want an open source version :)
Michael Heath wrote:
> I'm playing with an idea similar to this for incrimental backups. I
> read an article recently that discussed unionfs for backing up like
> this. Basically, to snapshot a folder, you create a union ontop of the
> directory, and then create another directory, like
> /.snapshots/home-since-20090702. You add this new directory to the
> union, and set the original RO and the new directory to RW. You can
> then add any further snapshots by simply creating a new directory,
> adding it to the union, and setting the older dirs read only.
> I've been working on some scripts for this, and I'm going to work on a
> python plugin for nautilus that adds right click functionality like
> "revert to version as of DATE". If anyone is interested, I'll share
> the finished products here.
> (Please pardon any bizarre formatting in this message. I'm out of town
> and messaging from my phone.)
[
incrtar 5K ] #!/usr/local/bin/perl # # Copyright 2002 Protos LLC, used with permission #
while (!flock($fd, $LOCK_EX|$LOCK_NB)) { if (!$retry) { print STDERR "Cannot lock file: $fname ($!)\n"; return 0; } $retry--; sleep(1); }
if ($fname =~ /^>>/) { ## make sure we are at the end--now that we have a lock seek($fd, 0, $SEEK_END); } else { ## make sure we are at the start--now that we have a lock seek($fd, 0, $SEEK_SET); }
return $fd;
}
sub lclose { my ($file) = $_[0];
close($file); flock($file, $LOCK_UN);
}
sub cmd { my $c = $_[0]; # if ($strip_logdirs) { # $c .= " | egrep -v '/\$'"; # } print STDERR "$c\n"; if ($writeout) { system($c); }
I don't have the original source (the site has died) but on the first
site I read about the unionfs snapshot method, the author said that hr
benchmarked filesystem performance at 80% of original with 4 snapshots
(original + 2 frozen + live).
This article was about 5 years old, and from what I understand,
unionfs has been vastly improved performance wise.
> What would be really interesting is to benchmark it after each step and
> plot out the performance degradation--it has to exist because you are
> keeping snapshots live, the question is how much of an impact does it have.
> FYI, for incrementals, I just use gnu tar with --newer-mtime (infact I
> have attached an old script I made in solaris to make it behave like the
> old unix 'dump').
> I recommend monthly fulls, weekly differential and daily incremental
> (for this I use a different script, which is part of a bigger
> infrastructure, but uses the same --newer-mtime method). On one site I
> do this to a DataDomain de-dup device / nfs share. It has 40TB of
> "real" storage but currently we have 300TB of actual data stored only in
> about 20TB of real disk (it is a cool device... I want an open source
> version :)
> -Brandon
> Michael Heath wrote:
>> I'm playing with an idea similar to this for incrimental backups. I
>> read an article recently that discussed unionfs for backing up like
>> this. Basically, to snapshot a folder, you create a union ontop of the
>> directory, and then create another directory, like
>> /.snapshots/home-since-20090702. You add this new directory to the
>> union, and set the original RO and the new directory to RW. You can
>> then add any further snapshots by simply creating a new directory,
>> adding it to the union, and setting the older dirs read only.
>> I've been working on some scripts for this, and I'm going to work on a
>> python plugin for nautilus that adds right click functionality like
>> "revert to version as of DATE". If anyone is interested, I'll share
>> the finished products here.
>> (Please pardon any bizarre formatting in this message. I'm out of town
>> and messaging from my phone.)
I'm going to post the full stats later, but I ran some benchmarks using bonnie++. I actually used aufs rather than unionfs.
The big surprise came from new file creation. The speed of that was only a small fraction of the original performance, though still high - bonnie++ still measured several thousand files per second. I guess the lesson here is that while this may be suitable for desktops, on many server systems with many dynamic files it may not work.
Other performance indicators like read and write speeds were a touch slower but very close to original.
The snapshot tools I'm working on will be another week or so at least - it's not complex, but I had a change of heart and switched from bash to python.
On Thu, 2009-07-09 at 12:07 -0600, Michael Heath wrote: > I'm going to post the full stats later, but I ran some benchmarks > using bonnie++. I actually used aufs rather than unionfs.
> The big surprise came from new file creation. The speed of that was > only a small fraction of the original performance, though still high - > bonnie++ still measured several thousand files per second. I guess the > lesson here is that while this may be suitable for desktops, on many > server systems with many dynamic files it may not work.
> Other performance indicators like read and write speeds were a touch > slower but very close to original.
> The snapshot tools I'm working on will be another week or so at least > - it's not complex, but I had a change of heart and switched from bash > to python.
Have you seen the LWN review of union filesystems on Linux? If not, I highly recommend it. The lesson I took away was: Don't use a union filesystem unless you absolutely have to.
-- "XML is like violence: if it doesn't solve your problem, you aren't using enough of it." - Chris Maden
Stuart Jansen wrote: > Have you seen the LWN review of union filesystems on Linux? If not, I > highly recommend it. The lesson I took away was: Don't use a union > filesystem unless you absolutely have to.
And those in the BSD camp? FFS for NetBSD seems to be remarkably stable, support a vast array of options, and even hold it's own with I/O benchmarks.
-- . O . O . O . . O O . . . O . . . O . O O O . O . O O . . O O O O . O . . O O O O . O O O