Proper format when rsyncing to thumb drive

[pH]

I'm going to use a thumb (usb) drive to back up my entire home
directory via rsync.

I know VFAT won't preserve all the directory info, what file system to
folks here recommend to minimize 'wear' on the usb drive or other
considerations I should have.

(A suggested command line would not be bad, either to make sure suck
up all hidden files, links, etc.)

A recent need to use my external drive backup got me thinking about
this.

Pureheart

[GangGreene]

ext2

[Robert Heller]

Right. Unless there is some need to 'share' the thumb drive, there is
no reason not to perform a mkfs.ext2 on it. I would NOT bother with
ext3 or ext4, since journaling a thumb drive is not worthwhile and will
only increase 'wear' needlessly.

>

--
Robert Heller -- 978-544-6933 / hel...@deepsoft.com
Deepwoods Software -- http://www.deepsoft.com/
() ascii ribbon campaign -- against html e-mail
/\ www.asciiribbon.org -- against proprietary attachments

[The Natural Philosopher]

Ive used that - still working.


--
Ineptocracy

(in-ep-toc’-ra-cy) – a system of government where the least capable to
lead are elected by the least capable of producing, and where the
members of society least likely to sustain themselves or succeed, are
rewarded with goods and services paid for by the confiscated wealth of a
diminishing number of producers.

[pH]

On Nov 11, 9:54 am, The Natural Philosopher <t...@invalid.invalid>
wrote:

> On 11/11/12 17:02, GangGreene wrote:
>
> > On Sun, 11 Nov 2012 08:07:10 -0800, pH wrote:
>
> >> I'm going to use a thumb (usb) drive to back up my entire home directory
> >> via rsync.
>
> >> I know VFAT won't preserve all the directory info, what file system to
> >> folks here recommend to minimize 'wear' on the usb drive or other
> >> considerations I should have.
>
> >> (A suggested command line would not be bad, either to make sure  suck up
> >> all hidden files, links, etc.)
>
> >> A recent need to use my external drive backup got me thinking about
> >> this.
>
> >> Pureheart
>
> > ext2
>
> Ive used that - still working.
>
> --

Thank-you, gentlemen. ext2 it shall be.

I intend to use "rsync -av --progress /home /media/
whatever_name_shows_up
and run it with 'nice' first time so I can still do other things.

Any other suggested command line toggles?

pH

[unruh]

On 2012-11-11, pH <pure...@pacbell.net> wrote:
> I'm going to use a thumb (usb) drive to back up my entire home
> directory via rsync.
>
> I know VFAT won't preserve all the directory info, what file system to
> folks here recommend to minimize 'wear' on the usb drive or other
> considerations I should have.

Format it as ext3. Theonlyproblem is that you will not be able to use it
to transfer files to Windows.
Us cfdisk or fdisk to repartition it ( leave the first small amount vfat
so that you can use it to transfer stuff) it to ext filesystem and then
format it.

Then rsync -av yourdirectory/ /media/tunmbdrive2
Note that I have found that rsync sometimes crashes on writing to usb
thumbdrives. Maybe it is writing to fast or something.

[pH]

On Nov 11, 2:17 pm, unruh <un...@invalid.ca> wrote:

Hi Mr. Unruh.
You disagree with the others in your recommendation to use ext3.

The idea to actually partition the drive is an interesting one as
well.
This particular drive's only mission in life will be to be a Linux
(mint) backup, so I think I will still go ahead and make the whole
thing ext2/3.

Looks like I'm not missing any important toggles on the command by the
lack of comments, so I'm relieved there.

Thanks for the help folks.

Pureheart in Aptos, CA

[Matthew of Boswell]

On Sun, 11 Nov 2012 16:43:18 -0800 (PST)
pH <pure...@pacbell.net> wrote:

> Hi Mr. Unruh.
> You disagree with the others in your recommendation to use ext3.

The main difference between ext2 and ext3 is the journal. Note that
filesystems can be converted back and forth between ext2 and ext3, so
you can change your mind later (ex: add a journal to ext2 to make it
ext3, or force ext3 to mount as ext2 to ignore the journal). Just make
sure you know what you're doing if you switch.

The journal, in my opinion, is unnecessary for a simple backup
solution. The main benefit is that the data recovery and filesystem
structure will be more reliable if you happen to lose power or crash
your system at the exact moment you happen to be writing to the drive.
Since you'd probably rather start your backup over again from scratch
at that point, I'd say journaling is extra overhead.

> Looks like I'm not missing any important toggles on the command by the
> lack of comments, so I'm relieved there.

Have you considered rdiff? If you want to be able to go back in time to
a previous system state (maybe 3-5 backups back or so), it may be a
consideration. However, it will probably do more writing to the USB
drive than plain rsync.

I'm pretty happy with ZFS, RAIDz, and daily snapshots for backups at
work, but that's way overkill for a USB drive and most home backup
solutions.

FWIW, my home backup is just dropbox and a SATA drive that I plug in
once in a while for big stuff.

--
Matt Boswell

() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - and proprietary attachments

[Robert Heller]

At Sun, 11 Nov 2012 19:58:37 -0500 Matthew of Boswell <mordervomu...@lockmail.us> wrote:

>
> On Sun, 11 Nov 2012 16:43:18 -0800 (PST)
> pH <pure...@pacbell.net> wrote:
>
> > Hi Mr. Unruh.
> > You disagree with the others in your recommendation to use ext3.
>
> The main difference between ext2 and ext3 is the journal. Note that
> filesystems can be converted back and forth between ext2 and ext3, so
> you can change your mind later (ex: add a journal to ext2 to make it
> ext3, or force ext3 to mount as ext2 to ignore the journal). Just make
> sure you know what you're doing if you switch.
>
> The journal, in my opinion, is unnecessary for a simple backup
> solution. The main benefit is that the data recovery and filesystem
> structure will be more reliable if you happen to lose power or crash
> your system at the exact moment you happen to be writing to the drive.
> Since you'd probably rather start your backup over again from scratch
> at that point, I'd say journaling is extra overhead.

For a thumb drive, a journal adds more 'wear'. Since the thumb drive is
a backup device and will likely only ever *manually* be mounted and
unmounted and will not likely be mounted for longer that the duration of
the backup process, the journal adds nothing but overhead and *needless*
wear.

>
> > Looks like I'm not missing any important toggles on the command by the
> > lack of comments, so I'm relieved there.
>
> Have you considered rdiff? If you want to be able to go back in time to
> a previous system state (maybe 3-5 backups back or so), it may be a
> consideration. However, it will probably do more writing to the USB
> drive than plain rsync.
>
> I'm pretty happy with ZFS, RAIDz, and daily snapshots for backups at
> work, but that's way overkill for a USB drive and most home backup
> solutions.
>
> FWIW, my home backup is just dropbox and a SATA drive that I plug in
> once in a while for big stuff.
>

--

Robert Heller -- 978-544-6933 / hel...@deepsoft.com
Deepwoods Software -- http://www.deepsoft.com/

() ascii ribbon campaign -- against html e-mail
/\ www.asciiribbon.org -- against proprietary attachments

[unruh]

On 2012-11-12, Matthew of Boswell <mordervomu...@lockmail.us> wrote:
> On Sun, 11 Nov 2012 16:43:18 -0800 (PST)
> pH <pure...@pacbell.net> wrote:
>
>> Hi Mr. Unruh.
>> You disagree with the others in your recommendation to use ext3.
>
> The main difference between ext2 and ext3 is the journal. Note that
> filesystems can be converted back and forth between ext2 and ext3, so
> you can change your mind later (ex: add a journal to ext2 to make it
> ext3, or force ext3 to mount as ext2 to ignore the journal). Just make
> sure you know what you're doing if you switch.
>
> The journal, in my opinion, is unnecessary for a simple backup
> solution. The main benefit is that the data recovery and filesystem
> structure will be more reliable if you happen to lose power or crash
> your system at the exact moment you happen to be writing to the drive.
> Since you'd probably rather start your backup over again from scratch
> at that point, I'd say journaling is extra overhead.
>
>> Looks like I'm not missing any important toggles on the command by the
>> lack of comments, so I'm relieved there.
>
> Have you considered rdiff? If you want to be able to go back in time to
> a previous system state (maybe 3-5 backups back or so), it may be a
> consideration. However, it will probably do more writing to the USB
> drive than plain rsync.

You could use rsnapshot (or rsync's --link-dest) which will both
preserve previous states, and cause far less writing during the backups,
at the expense of greater use of disk space than a single backup )(but
far less than multiple backups)

[Richard Kettlewell]

Robert Heller <hel...@deepsoft.com> writes:
> Matthew of Boswell <mordervomu...@lockmail.us> wrote:

>> The main difference between ext2 and ext3 is the journal. Note that
>> filesystems can be converted back and forth between ext2 and ext3, so
>> you can change your mind later (ex: add a journal to ext2 to make it
>> ext3, or force ext3 to mount as ext2 to ignore the journal). Just
>> make sure you know what you're doing if you switch.
>>
>> The journal, in my opinion, is unnecessary for a simple backup
>> solution. The main benefit is that the data recovery and filesystem
>> structure will be more reliable if you happen to lose power or crash
>> your system at the exact moment you happen to be writing to the
>> drive. Since you'd probably rather start your backup over again from
>> scratch at that point, I'd say journaling is extra overhead.

A key practical advantage is not having to run fsck - a potentially
substantial timesaving at boot time (or whenever it is one mounts the
device).

> For a thumb drive, a journal adds more 'wear'. Since the thumb drive
> is a backup device and will likely only ever *manually* be mounted and
> unmounted and will not likely be mounted for longer that the duration
> of the backup process, the journal adds nothing but overhead and
> *needless* wear.

Could you quantify how much difference you think this will make in
practice? There’s an awful lot of hot air around about flash wear.

(Last time I attempted to answer this question for a particular device
it turned out that it would be 75-150 years before I had anything to
worry about; realistically speaking something else is going to go wrong
first.)

--
http://www.greenend.org.uk/rjk/

[Chris Davies]

unruh <un...@invalid.ca> wrote:
> You could use rsnapshot (or rsync's --link-dest) which will both
> preserve previous states, and cause far less writing during the backups,
> at the expense of greater use of disk space than a single backup )(but
> far less than multiple backups)

I too would recommend rsnapshot for backups. The configuration file's
actually not too scary once you start reading it. A quick 'du' suggests
that for my main PC (laptop) the backups are using about 160% of my
actual disk usage. For this overhead I get a couple of backups a day,
two or three per week, and several ranging across the last few
weeks. (It's not as precise as you might expect because my laptop isn't
on 24x7.)

Chris

[pH]

On Nov 11, 4:58 pm, Matthew of Boswell <mordervomubel

+use...@lockmail.us> wrote:
> On Sun, 11 Nov 2012 16:43:18 -0800 (PST)
>

> pH <purehe...@pacbell.net> wrote:
> > Hi Mr. Unruh.
> > You disagree with the others in your recommendation to use ext3.
>
> The main difference between ext2 and ext3 is the journal. Note that
> filesystems can be converted back and forth between ext2 and ext3, so
> you can change your mind later (ex: add a journal to ext2 to make it
> ext3, or force ext3 to mount as ext2 to ignore the journal). Just make
> sure you know what you're doing if you switch.

That's sure not me!
I think I'm fairly competent when choosing command line toggles,
but when it comes to anything that's actually going to do any
significant
writing then I come here for expert help.

I still fear reversing my arguments and overwriting my real data by
accident or
something equally horrible.

Looking further down the thread I see that I am going to have to read
up on 'rsnapshot'
and the --link-dest toggle of rsync.

Fortunately, or unfortunately, it looks like I've already outgrown the
16GB drive I was going to use so will
be finding a nice, cheap 32GB before I proceed. (any brands to
*avoid*?)

Thanks, everyone.

pH


<snip rest of post>

[Chick Tower]

On 2012-11-12, pH <pure...@pacbell.net> wrote:
> Looks like I'm not missing any important toggles on the command by the
> lack of comments, so I'm relieved there.

If I recall correctly, you should make sure the "backup source" has a
slash at the end of it, but the "backup destination" does not. i.e.,

rsync -av /home/ /media/usb

The slash will force rsync to treat the source as a directory and copy
everything in and under it, rather than treating it as only the files
immediately below that directory, without diving down into
subdirectories.

See the man page for rsync for authoritative information.
--
Chick Tower

For e-mail: aols2 DOT sent DOT towerboy AT xoxy DOT net

[unruh]

On 2012-11-12, pH <pure...@pacbell.net> wrote:

> On Nov 11, 4:58?pm, Matthew of Boswell <mordervomubel

> +use...@lockmail.us> wrote:
>> On Sun, 11 Nov 2012 16:43:18 -0800 (PST)
>>
>> pH <purehe...@pacbell.net> wrote:
>> > Hi Mr. Unruh.
>> > You disagree with the others in your recommendation to use ext3.
>>
>> The main difference between ext2 and ext3 is the journal. Note that
>> filesystems can be converted back and forth between ext2 and ext3, so
>> you can change your mind later (ex: add a journal to ext2 to make it
>> ext3, or force ext3 to mount as ext2 to ignore the journal). Just make
>> sure you know what you're doing if you switch.
>
> That's sure not me!
> I think I'm fairly competent when choosing command line toggles,
> but when it comes to anything that's actually going to do any
> significant
> writing then I come here for expert help.
>
> I still fear reversing my arguments and overwriting my real data by
> accident or
> something equally horrible.

That is why it is a good idea to automate it-- write a script file to
run the rsync command for you.

>
> Looking further down the thread I see that I am going to have to read
> up on 'rsnapshot'
> and the --link-dest toggle of rsync.
>
> Fortunately, or unfortunately, it looks like I've already outgrown the
> 16GB drive I was going to use so will
> be finding a nice, cheap 32GB before I proceed. (any brands to
> *avoid*?)
>
> Thanks, everyone.
>
> pH
>
>
><snip rest of post>
>>
>> FWIW, my home backup is just dropbox and a SATA drive that I plug in
>> once in a while for big stuff.
>>
>> --
>> Matt Boswell
>>

>> () ?ascii ribbon campaign - against html e-mail
>> /\ ?www.asciiribbon.org? - and proprietary attachments
>

[unruh]

On 2012-11-12, Chick Tower <c.t...@deadspam.com> wrote:
> On 2012-11-12, pH <pure...@pacbell.net> wrote:
>> Looks like I'm not missing any important toggles on the command by the
>> lack of comments, so I'm relieved there.
>
> If I recall correctly, you should make sure the "backup source" has a
> slash at the end of it, but the "backup destination" does not. i.e.,
>
> rsync -av /home/ /media/usb

That depends on if you want just the files in /media/usb, or you want
them all under the subdirectory "home" on /media/usb. If you have the /
only the files get copied. If you do not have the /, then directory
called home will be created/used on /media/usb and the files in /home
will be placed there.

>
> The slash will force rsync to treat the source as a directory and copy
> everything in and under it, rather than treating it as only the files
> immediately below that directory, without diving down into
> subdirectories.

Wrong way around. / does the files only, without a / it also does the
directory. And no, rsync always does all subdirectories as well if you
use the -a option (it includes the -r option which means "recurse")

[pH]

Well, as I write this the backup is in progress 4GB down and 13 to go
to a USB thumbdrive.

Man, I am *not* calm at all when I type in that mkfs.ext2
command....whew!

One thing I don't like is once I was done the partition mounts with a
looong string of hexadecimal digits.

the format is 8-4-4-4-12 where this notation means eight hexadecimal
digits, a hyphen, four hex digits, hyphen, etcetera.

Is there a way I can rename that to something nicer like "usb_drive"
or should I just not worry about it, build a script to automate the
backup process and live with it?

Special thanks to Mr. Unruh for his kindness in taking the time for
personal consultation.

pH

[Jerry Peters]

I'll never understand why anyone would want to mount with UUID, except
in very special circumstances. Yet I've still seen poeple recommend it.

See 'man mount' and the -L option, also 'man e2label'.

Except for backup drives, which get mounted as '/bkup', I mount all
USB sticks as '/usd1', I *know* which stick I've stuck in the socket,
and the computer doesn't care.

Jerry

[pure...@pacbell.net]

On Sunday, November 18, 2012 1:56:30 PM UTC-8, Jerry Peters wrote:
<snip>

>
>
> I'll never understand why anyone would want to mount with UUID, except
>
> in very special circumstances. Yet I've still seen poeple recommend it.
>
>
>
> See 'man mount' and the -L option, also 'man e2label'.
>
>
>
> Except for backup drives, which get mounted as '/bkup', I mount all
>
> USB sticks as '/usd1', I *know* which stick I've stuck in the socket,
>
> and the computer doesn't care.
>
>
>
> Jerry

Oh, Jerry, that worked very slickly.
Wish I had known this a long time ago.

I had more than a little trepidation running it...but the result was fine.

I'll read up on your suggestions more.

Thanks!
pH

[Stuart Barkley]

On Sun, 11 Nov 2012 at 19:43 -0000, pH wrote:

> This particular drive's only mission in life will be to be a Linux
> (mint) backup, so I think I will still go ahead and make the whole
> thing ext2/3.

I leave a FAT partition on my thumb drives with a 00README.txt file in
it with contact information.

Others are unlikely to be able to read the ext2 partition. So just in
case I lose the little thing, I want to hope someone might try to read
it then return it to me.

Stuart
--
I've never been lost; I was once bewildered for three days, but never lost!
-- Daniel Boone

[Michael Black]

On Sat, 8 Dec 2012, Stuart Barkley wrote:

> On Sun, 11 Nov 2012 at 19:43 -0000, pH wrote:
>
>> This particular drive's only mission in life will be to be a Linux
>> (mint) backup, so I think I will still go ahead and make the whole
>> thing ext2/3.
>
> I leave a FAT partition on my thumb drives with a 00README.txt file in
> it with contact information.
>
> Others are unlikely to be able to read the ext2 partition. So just in
> case I lose the little thing, I want to hope someone might try to read
> it then return it to me.
>

That's an interestng idea. Just enough for the file to hold your contact
information. And you're right, the rest won't be "encrypted" but unless
Windows comes with the means to read ext2 partitions, for most people it
will be "unreadable".

Michael

[The Natural Philosopher]

On 08/12/12 19:42, Stuart Barkley wrote:
> On Sun, 11 Nov 2012 at 19:43 -0000, pH wrote:
>
>> This particular drive's only mission in life will be to be a Linux
>> (mint) backup, so I think I will still go ahead and make the whole
>> thing ext2/3.
>
> I leave a FAT partition on my thumb drives with a 00README.txt file in
> it with contact information.
>
> Others are unlikely to be able to read the ext2 partition. So just in
> case I lose the little thing, I want to hope someone might try to read
> it then return it to me.
>

If I lose mine, I would really NOT want anyone to read it :-)

Ext2 and then the wintards will reformat it.

> Stuart

[Aragorn]

On Sunday 09 December 2012 01:09, The Natural Philosopher conveyed the
following to comp.os.linux.misc...

> On 08/12/12 19:42, Stuart Barkley wrote:
>
>> On Sun, 11 Nov 2012 at 19:43 -0000, pH wrote:
>>
>>> This particular drive's only mission in life will be to be a Linux
>>> (mint) backup, so I think I will still go ahead and make the whole
>>> thing ext2/3.
>>
>> I leave a FAT partition on my thumb drives with a 00README.txt file
>> in it with contact information.
>>
>> Others are unlikely to be able to read the ext2 partition. So just
>> in case I lose the little thing, I want to hope someone might try to
>> read it then return it to me.
>
> If I lose mine, I would really NOT want anyone to read it :-)

Ditto.

> Ext2 and then the wintards will reformat it.

Not necessarily. There is an ext2 driver for Windows, albeit that I
don't know whether it can handle ext3, and I doubt that it would be able
to handle ext4. And of course, the average Win-droid won't even know
that such a thing exists, but you should never exclude the possibility
that they've already heard of a search engine called Google. :p

Also, please do bear in mind that the Linux kernel can read from and
write to several other usable UNIX-style filesystems beside the ext?
ones, such as reiserfs, XFS, JFS and btrfs. I am using XFS for all my
Mageia partitions on this hard disk, and reiserfs for all the partitions
on the hard disk which has my PCLinuxOS installation. [*]

In other words, if one has no need for Microsoft Windows, then there is
no need to stick with ext2 (or even ext3 or ext4) for a thumb drive.
ext2 and ext3 are fairly slow by comparison anyway, even when compared
to ext4.


[*] reiserfs may have fallen out of grace with many by now due to the
fact that it's not actively being maintained anymore (as a result
of Hans Reiser's conviction over having killed his wife), but XFS
is an industry standard POSIX-compliant filesystem and was already
mature long before reiserfs and even ext3 came along, let alone
ext4 or btrfs.

--
= Aragorn =
(registered GNU/Linux user #223157)

[J G Miller]

Op zondag, 09 december, 2012, om 02:14:48u +0100, schreef Aragorn:

> There is an ext2 driver for Windows, albeit that I don't know whether
> it can handle ext3, and I doubt that it would be able to handle ext4.

Trying to write to ext3 and ext4 file systems from Windoze is dangerous
or not possible with the various add on softwares that are available,
but reading ext2/ext3/ext4 and even LVM is simple and easy with such
interfaces as ext2read

<http://sourceforge.NET/projects/ext2read/>

so your doubt is misplaced.

> And of course, the average Win-droid won't even know
> that such a thing exists

The difference between disks, partitions, and file system
types is totally lost on Windroids, but then, why should
they care "if it just works"? ;)

> but you should never exclude the possibility
> that they've already heard of a search engine called Google. :p

Really? I though Windroids used Bing? ;)

> Also, please do bear in mind that the Linux kernel can read from and
> write to several other usable UNIX-style filesystems beside the ext?
> ones, such as reiserfs, XFS, JFS and btrfs.

openSuse is almost saying btrfs is ready for prime time.

> In other words, if one has no need for Microsoft Windows, then there is
> no need to stick with ext2 (or even ext3 or ext4) for a thumb drive.

So which would actually be the best file system to use for performance
and longevity of the USB device in your humble opinion?

> [*] reiserfs may have fallen out of grace with many by now due to the
> fact that it's not actively being maintained anymore

Sorry, this is just not true. reiserfs is still under active (but
limited and slow) development.

<http://sourceforge.NET/projects/reiser4/files/>

The problem for reiserfs is that it is being eclipsed by btrfs which
also does not carry the skeletons found in the reiserfs closet

[Aragorn]

On Sunday 09 December 2012 16:50, J G Miller conveyed the following to
comp.os.linux.misc...

> Op zondag, 09 december, 2012, om 02:14:48u +0100, schreef Aragorn:
>

>> Also, please do bear in mind that the Linux kernel can read from and
>> write to several other usable UNIX-style filesystems beside the ext?
>> ones, such as reiserfs, XFS, JFS and btrfs.
>
> openSuse is almost saying btrfs is ready for prime time.

As I understand it, btrfs as a filesystem is pretty stable, but it still
lacks the supporting userland tools for crash recovery and such. And
crashes do happen, even if only because of the occasional power outage -
not everyone connects their machines to a UPS (although everyone
should).

>> In other words, if one has no need for Microsoft Windows, then there
>> is no need to stick with ext2 (or even ext3 or ext4) for a thumb
>> drive.
>
> So which would actually be the best file system to use for performance
> and longevity of the USB device in your humble opinion?

ext4 is pretty good - it is the default filesystem in Android 2.3.3 and
above - but you could also go with reiserfs or XFS. Essentially
anything that uses B-trees or H-trees. ext2 doesn't use those and ext3
has them but they're not turned on by default.

>> [*] reiserfs may have fallen out of grace with many by now due to the
>> fact that it's not actively being maintained anymore
>
> Sorry, this is just not true. reiserfs is still under active (but
> limited and slow) development.
>
> <http://sourceforge.NET/projects/reiser4/files/>

That's reiser4, not reiserfs. reiser4 is the 4th generation filesystem
from Hans Reiser and NameSys. reiserfs is version 3.6 - and thus the
3th generation filesystem from said makers. reiserfs is older dan ext3,
albeit newer than ext2.

> The problem for reiserfs is that it is being eclipsed by btrfs which
> also does not carry the skeletons found in the reiserfs closet

btrfs attempts to eclipse a lot more than that, because it's both a
filesystem and a volume manager. It's the GPL'd equivalent of Solaris's
ZFS.