Gmail Calendar Documents Reader Web more »
Recently Visited Groups | Help | Sign in
Google Groups Home
Reminder: Meeting tonight on Backups
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  13 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Seth House  
View profile  
 More options Jun 30, 10:59 am
From: Seth House <whitei...@gmail.com>
Date: Tue, 30 Jun 2009 07:59:14 -0700 (PDT)
Local: Tues, Jun 30 2009 10:59 am
Subject: Reminder: Meeting tonight on Backups
Hey, all. Sorry for the late reminder.

Tonight I'll present on backups; what to backup and how to backup, and
how to recreate Apple’s Time Machine with a one line command. The
presentation will be about 30-45 minutes and we’ll socialize after —
stay as long as you want to — if you have any computer questions,
bring ‘em!

See you tonight.

- Seth

Weber County Main Library, Board Room
2464 Jefferson Ave.
Ogden, UT 84401

http://maps.google.com/maps?q=Weber+County+Library,2464+Jefferson+Ave...


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
edf...@gmail.com  
View profile  
 More options Jun 30, 11:22 am
From: edf...@gmail.com
Date: Tue, 30 Jun 2009 15:22:51 +0000
Local: Tues, Jun 30 2009 11:22 am
Subject: Re: [oalug] Reminder: Meeting tonight on Backups
Sorry. I forgot what time the meeting is at.


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Seth House  
View profile  
 More options Jun 30, 11:27 am
From: Seth House <whitei...@gmail.com>
Date: Tue, 30 Jun 2009 08:27:40 -0700 (PDT)
Local: Tues, Jun 30 2009 11:27 am
Subject: Re: Reminder: Meeting tonight on Backups
On Jun 30, 9:22 am, edf...@gmail.com wrote:

> Sorry. I forgot what time the meeting is at.

Oh, that would’ve been a good thing to include in the first
message!  :)

From 7:30 to 8:45 PM tonight.


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Heath  
View profile  
 More options Jul 1, 5:53 pm
From: Michael Heath <mike.thomas.he...@gmail.com>
Date: Wed, 1 Jul 2009 15:53:53 -0600
Local: Wed, Jul 1 2009 5:53 pm
Subject: Re: [oalug] Reminder: Meeting tonight on Backups

On Tue, Jun 30, 2009 at 8:59 AM, Seth House<whitei...@gmail.com> wrote:

> Tonight I'll present on backups; what to backup and how to backup, and
> how to recreate Apple’s Time Machine with a one line command. The
> presentation will be about 30-45 minutes and we’ll socialize after —
> stay as long as you want to — if you have any computer questions,
> bring ‘em!

For those of us who couldn't attend, what is the magic one line
command? And are there any slides or materials from the presentation
available anywhere?

    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Discussion subject changed to "Backups Presentation Slides" by Seth House
Seth House  
View profile  
 More options Jul 1, 6:54 pm
From: Seth House <whitei...@gmail.com>
Date: Wed, 1 Jul 2009 15:54:58 -0700 (PDT)
Local: Wed, Jul 1 2009 6:54 pm
Subject: Backups Presentation Slides
On Jul 1, 3:53 pm, Michael Heath <mike.thomas.he...@gmail.com> wrote:

> For those of us who couldn't attend, what is the magic one line
> command?

Run this command:

rsync -a $HOME/ /var/timemachine/$USER/`date '+%FT%R'`

Then put this in your crontab (all on one line):

1   */2   *   *   *   BACKUPDIR=/var/timemachine/$USER; rsync -a
--link-dest=$BACKUPDIR/`/bin/ls -t $BACKUPDIR | head -1` $HOME/
$BACKUPDIR/`date '+%FT%R'`

> And are there any slides or materials from the presentation available
> anywhere?

The slides are available in the group Files section. Be sure to read
through the handout notes (print the HTML file) as they’re pretty
conversational and will give you a good feel for what was said at the
meeting.

http://groups.google.com/group/oalug/web/oalug_2009-06-30_backups.tar...


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Roly C.  
View profile  
 More options Jul 3, 8:49 am
From: "Roly C." <rol...@gmail.com>
Date: Fri, 3 Jul 2009 06:49:51 -0600
Local: Fri, Jul 3 2009 8:49 am
Subject: Re: [oalug] Backups Presentation Slides

How often does that cron job run?  I finally started creating backups when I
accidentally recursively changed the permissions of my friend's home to 000.
 (it took me about 1 hour to get everything back to the way it was...)


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Seth House  
View profile  
 More options Jul 3, 4:38 pm
From: Seth House <whitei...@gmail.com>
Date: Fri, 3 Jul 2009 13:38:02 -0700 (PDT)
Local: Fri, Jul 3 2009 4:38 pm
Subject: Re: Backups Presentation Slides
On Jul 3, 6:49 am, "Roly C." <rol...@gmail.com> wrote:

> How often does that cron job run?

1   */2   *   *   *   somecommand

``somecommand`` runs every two hours, one minute after the top of the
hour. The crontab manpage has a bunch of easy to understand examples.
(It's actually a typo, the one in my crontab runs at the top of the
hour.)


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Heath  
View profile  
 More options Jul 4, 11:35 am
From: Michael Heath <mike.thomas.he...@gmail.com>
Date: Sat, 4 Jul 2009 15:35:51 +0000
Local: Sat, Jul 4 2009 11:35 am
Subject: Re: [oalug] Backups Presentation Slides
I'm playing with an idea similar to this for incrimental backups. I
read an article recently that discussed unionfs for backing up like
this. Basically, to snapshot a folder, you create a union ontop of the
directory, and then create another directory, like
/.snapshots/home-since-20090702. You add this new directory to the
union, and set the original RO and the new directory to RW. You can
then add any further snapshots by simply creating a new directory,
adding it to the union, and setting the older dirs read only.

I've been working on some scripts for this, and I'm going to work on a
python plugin for nautilus that adds right click functionality like
"revert to version as of DATE". If anyone is interested, I'll share
the finished products here.

(Please pardon any bizarre formatting in this message. I'm out of town
and messaging from my phone.)


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
brandon  
View profile  
 More options Jul 4, 12:30 pm
From: brandon <b...@solv.com>
Date: Sat, 04 Jul 2009 10:30:33 -0600
Local: Sat, Jul 4 2009 12:30 pm
Subject: Re: [oalug] Re: Backups Presentation Slides

What would be really interesting is to benchmark it after each step and
plot out the performance degradation--it has to exist because you are
keeping snapshots live, the question is how much of an impact does it have.

FYI, for incrementals, I just use gnu tar with --newer-mtime (infact I
have attached an old script I made in solaris to make it behave like the
old unix 'dump').

I recommend monthly fulls, weekly differential and daily incremental
(for this I use a different script, which is part of a bigger
infrastructure, but uses the same --newer-mtime method).  On one site I
do this to a DataDomain de-dup device / nfs share.  It has 40TB of
"real" storage but currently we have 300TB of actual data stored only in
about 20TB of real disk (it is a cool device... I want an open source
version :)

-Brandon

[ incrtar 5K ]
#!/usr/local/bin/perl
#
# Copyright 2002 Protos LLC, used with permission
#

$locked = 0;
$LOCK_SH = 1;
$LOCK_EX = 2;
$LOCK_NB = 4;
$LOCK_UN = 8;
$SEEK_SET = 0;
$SEEK_CUR = 1;
$SEEK_END = 1;

$datefile = "/usr/local/etc/incrtar.dates";
$excludefile = "";
$outputfile = "";
$writeout = 1;
$level = 0;
@args = ();
$targs = "";
$strip_logdirs = 0;

while ($#ARGV >= 0) {
    $_ = shift(@ARGV);
    if (/^-([0-9])/) {
        $level = $1 + 0;
    } elsif (/^-u(.*)$/) {
        $datefile = $1;
        if (!$datefile) {
            $datefile = shift(@ARGV);
        }
    } elsif (/^-o(.*)$/) {
        if (length($1) == 0) {
            die("ABORT: no tar option specified with -o\n");
        }
        if ("$1" eq "v") {
            $strip_logdirs = 1;
        }
        $targs .= " -$1";
    } elsif (/^-f(.*)$/) {
        $outputfile = $1;
        if (!$outputfile) {
            $outputfile = shift(@ARGV);
        }
    } elsif (/^-x(.*)$/) {
        $excludefile = $1;
        if (!$excludefile) {
            $excludefile = shift(@ARGV);
        }
    } elsif (/^-N(.*)$/) {
        $writeout = 0;
    } elsif (/^-/) {
        die("ABORT: unrecognized option: $_\n");
    } else {
        push(@args, $_);
    }

}

if ($excludefile) {
    $targs .= " --exclude-from $excludefile";

}

if ($outputfile) {
    $targs .= " -f $outputfile";
} else {

    die("ABORT: No output file/device (-f) specified.\n");

}

$dir = $args[0];
if (!$dir || ! -d $dir) {
    die("ABORT: '$dir' is not a directory.\n");

}

## error here if more than one fs?  or loop later

@months = ("Jan", "Feb", "Mar", "Apr", "May", "Jun",
           "Jul", "Aug", "Sep", "Oct", "Nov", "Dec");
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
$year += 1900;
$now = sprintf("%02d $months[$mon] $year %02d:%02d:00",$mday,$hour,$min);

&read_datefile();

if ($level > 0) {
    $prev = $level - 1;

    while ($prev > 0) {
        if ($levels{"$fs:$prev"}) {
            last;
        }
        $prev--;
    }

} else {
    $prev = 0;
}

print STDERR <<END;
Output File:    $outputfile
Date File:      $datefile
Archive Level:  $level
Directory Base: $dir
END
if ($excludefile) {
    print STDERR "Exclude Files:  $excludefile\n";
}

print STDERR "Starting Time:  $now\n";

if ($prev < $level && ($prevd = $dates{"$dir:$prev"})) {
    $targs .= " --newer-mtime '$prevd'";
    print STDERR "Previous Level: $prev\n";
    print STDERR "Previous Time:  $prevd\n";

} else {

    print STDERR "Previous Level: NONE\n";
    print STDERR "Previous Time:  NONE\n";
    print STDERR "** FULL ARCHIVE **\n";

}

&cmd("/usr/local/bin/tar -v -c$targs $dir");

if (!$writeout) {
    exit(0);

}

&get_lock();
&read_datefile();

open(DATES, ">$datefile") || die("open(>$datefile): $!\n");
$dates{"$dir:$level"} = $now;
for $k (sort keys (%dates)) {
    ($fs, $lv) = split(/:/, $k);
    print DATES "$fs $lv $dates{$k}\n";

}

close(DATES);

&release_lock();

sub lopen {
    my ($fd, $fname) = @_;
    my $retry;

    if (!open($fd, $fname)) {
        print(STDERR "open($fd, \"$fname\"): $!\n");
        return 0;
    } # || return 0;
    $retry = $lopen_retry || 5;

    return $fd;

    while (!flock($fd, $LOCK_EX|$LOCK_NB)) {
        if (!$retry) {
            print STDERR "Cannot lock file: $fname ($!)\n";
            return 0;
        }
        $retry--;
        sleep(1);
    }

    if ($fname =~ /^>>/) {
        ## make sure we are at the end--now that we have a lock
        seek($fd, 0, $SEEK_END);
    } else {
        ## make sure we are at the start--now that we have a lock
        seek($fd, 0, $SEEK_SET);
    }

    return $fd;

}

sub lclose {
    my ($file) = $_[0];

    close($file);
    flock($file, $LOCK_UN);

}

sub cmd {
    my $c = $_[0];
#    if ($strip_logdirs) {
#        $c .= " | egrep -v '/\$'";
#    }
    print STDERR "$c\n";
    if ($writeout) {
        system($c);
    }

}

sub read_datefile {
    %dates = ();
    &get_lock();
    if (lopen(DATES, "$datefile")) {
        while (<DATES>) {
            ($fs, $lv, @date) = split(/\s+/);
            $date = join(" ", @date);
            $key = "$fs:$lv";
            $dates{$key} = $date;
        }
        lclose(DATES);
    } else {
        print STDERR "** no datefile ($datefile) **\n";
    }
    &release_lock();

}

sub get_lock {
    if ($locked > 0) {
        return;
    }
    lopen(LOCK, ">$datefile._L") || die("lopen(>$datefile._L): $!\n");
    $locked = 1;

}

sub release_lock {
    $locked = 0;
    lclose(LOCK, "$datefile._L");
    unlink("$datefile._L");


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Heath  
View profile  
 More options Jul 4, 2:47 pm
From: Michael Heath <mike.thomas.he...@gmail.com>
Date: Sat, 4 Jul 2009 18:47:53 +0000
Local: Sat, Jul 4 2009 2:47 pm
Subject: Re: [oalug] Re: Backups Presentation Slides
I don't have the original source (the site has died) but on the first
site I read about the unionfs snapshot method, the author said that hr
benchmarked filesystem performance at 80% of original with 4 snapshots
(original + 2 frozen + live).

This article was about 5 years old, and from what I understand,
unionfs has been vastly improved performance wise.

I'll run some benchmarks later this week.

On 7/4/09, brandon <b...@solv.com> wrote:


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Heath  
View profile  
 More options Jul 9, 2:07 pm
From: Michael Heath <mike.thomas.he...@gmail.com>
Date: Thu, 9 Jul 2009 12:07:00 -0600
Subject: Re: [oalug] Re: Backups Presentation Slides
I'm going to post the full stats later, but I ran some benchmarks
using bonnie++. I actually used aufs rather than unionfs.

The big surprise came from new file creation. The speed of that was
only a small fraction of the original performance, though still high -
bonnie++ still measured several thousand files per second. I guess the
lesson here is that while this may be suitable for desktops, on many
server systems with many dynamic files it may not work.

Other performance indicators like read and write speeds were a touch
slower but very close to original.

The snapshot tools I'm working on will be another week or so at least
- it's not complex, but I had a change of heart and switched from bash
to python.


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Stuart Jansen  
View profile  
 More options Jul 9, 2:36 pm
From: Stuart Jansen <sjan...@buscaluz.org>
Date: Thu, 09 Jul 2009 12:36:08 -0600
Local: Thurs, Jul 9 2009 2:36 pm
Subject: Re: [oalug] Re: Backups Presentation Slides

Have you seen the LWN review of union filesystems on Linux? If not, I
highly recommend it. The lesson I took away was: Don't use a union
filesystem unless you absolutely have to.

--
"XML is like violence: if it doesn't solve your problem, you aren't
using enough of it." - Chris Maden


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Aaron Toponce  
View profile  
 More options Jul 9, 8:41 pm
From: Aaron Toponce <aaron.topo...@gmail.com>
Date: Thu, 09 Jul 2009 18:41:36 -0600
Local: Thurs, Jul 9 2009 8:41 pm
Subject: Re: [oalug] Re: Backups Presentation Slides

Stuart Jansen wrote:
> Have you seen the LWN review of union filesystems on Linux? If not, I
> highly recommend it. The lesson I took away was: Don't use a union
> filesystem unless you absolutely have to.

And those in the BSD camp? FFS for NetBSD seems to be remarkably stable,
support a vast array of options, and even hold it's own with I/O benchmarks.

--
. O .   O . O   . . O   O . .   . O .
. . O   . O O   O . O   . O O   . . O
O O O   . O .   . O O   O O .   O O O

  signature.asc
< 1K Download

    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2009 Google