rsync - excluding all files older than ?
Geico Caveman
This is a somewhat unusual situation. I want rsync to only upload files
with modification times newer than 10 days ago. The server where I am
uploading files has less disk space than the entire tree that I need
backups of, so I am placing this restriction.
I have additional backups for this tree on (two) other geographically
separated servers, so no worries.
So what do I do to make rsync to consider files only newer than 10 days ago
and delete all files that are older than that on the server ?
This is not the usual update situation. I am just using this server as a
snapshot of the newest files that will permit me to travel from home to
work and back without lugging my laptop.
Thanks.
Chris Davies
> This is a somewhat unusual situation. I want rsync to only upload files
> with modification times newer than 10 days ago.
You can't do that directly, but you could do something along those lines
with a combination of find/rsync.
> So what do I do to make rsync to consider files only newer than 10 days ago
> and delete all files that are older than that on the server ?
I'm not sure on which side you want to delete the files, so I'm going to
assume you actually only want them deleted on the remote server rather
than the local system. Accordingly, something along these *untested*
lines might help you:
cd /base/of/tree/to/copy
mkdir .tmp # Same filesystem, but omitted by "find"
find * -depth -mtime -10 -type f |
cpio -pmdlv .tmp # Build tree of links < 10 days old
(
cd .tmp &&
rsync -avHP --delete . remotehost:/path/to/backup/
)
rm -rf .tmp # Clean up
If you could have files starting with "." you'll need to tweak the
temporary directory name and the * wildcard to "find" accordingly
Chris
F8BOE
Hello,
Do you really need it?
RSYNC only sends updated parts of files unless your command line tells to
reload the whole new file and backing up the older files.
I suggest you take a closer look to the RSYNC manual... If needed print it;
it can really help!
You can also run RSYNC on the subfolders you need.
Perhaps is CP better than RSYNC (for your application)?
It could be possible to add a time filtering argument; you'll have to search
in the MAN RSYNC... Are the files backed up? Renamed? Inflating the
server's HDDs? A wrong server RSYNC command?
Ciao @+
Geico Caveman
> Hello,
>
> Do you really need it?
Wouldn't be asking otherwise.
> RSYNC only sends updated parts of files unless your command line tells to
> reload the whole new file and backing up the older files.
>
> I suggest you take a closer look to the RSYNC manual... If needed print
> it; it can really help!
I can read it perfectly fine on the screen.
>
> You can also run RSYNC on the subfolders you need.
I have used rsync for backup purposes for about 8-9 years. I can reasonably
claim a passing familiarity with its capabilities. This particular
requirement stumped me, hence my question.
>
> Perhaps is CP better than RSYNC (for your application)?
Other than taking longer and wasting bandwidth, no.
>
> It could be possible to add a time filtering argument; you'll have to
> search in the MAN RSYNC... Are the files backed up? Renamed? Inflating the
> server's HDDs? A wrong server RSYNC command?
If you could point me to this elusive time filtering argument, I would
appreciate it.
pk
You could use find to get all the files newer than 10 days (eg modification
time), and feed the list to rsync. Something like
find . -type f -mtime -240 | rsync --files-from=- ....
this could be a starting point to build the exact command that suits your
needs. See the find and rsync man pages (option --files-from, which has
some side-effects) for more details.
Geico Caveman
I found something similar asked and answered using google. That solution
consists of dumping file paths from find to a file in /tmp and then using
it as an argument to the --exclude-from argument.
The problem is that the exclude-from argument expects patterns and not file
names per se. So, this suggested solution does not work. I even tried to
use some printf pipes to get the correct format (with directories only),
but rsync does not like and ends up ignoring the --exclude-from argument
(effectively as it then attempts to upload everything to the server -
something I definitely do not want) :
find /path/to/files -mtime +30 -type d -printf '- %p\n'> /tmp/exclude.list
The other option is to use the --files-from argument. However, this option
is new, and does not exist on the version of rsync used on the server I am
trying to upload to (I checked).
That brings us to something like your solution. I have to admit that it
might be the only thing that might ultimately work. However, it involves
making a temporary archive of links using cpio, and I am trying to keep
this as simple as possible. I want to first find out why the find (above)
does not generate an acceptable exclude list for rsync.
Grant
>I found something similar asked and answered using google. That solution
>consists of dumping file paths from find to a file in /tmp and then using
>it as an argument to the --exclude-from argument.
You may exclude from a file list
>
>The problem is that the exclude-from argument expects patterns and not file
>names per se. So, this suggested solution does not work. I even tried to
>use some printf pipes to get the correct format (with directories only),
>but rsync does not like and ends up ignoring the --exclude-from argument
>(effectively as it then attempts to upload everything to the server -
>something I definitely do not want) :
>
>find /path/to/files -mtime +30 -type d -printf '- %p\n'> /tmp/exclude.list
>
>The other option is to use the --files-from argument. However, this option
>is new, and does not exist on the version of rsync used on the server I am
>trying to upload to (I checked).
>
>That brings us to something like your solution. I have to admit that it
>might be the only thing that might ultimately work. However, it involves
>making a temporary archive of links using cpio, and I am trying to keep
>this as simple as possible. I want to first find out why the find (above)
>does not generate an acceptable exclude list for rsync.
There's always the dinosaur approach, copy the lot over then prune stale
files from the backup ;)
Grant.
--
http://bugsplatter.id.au/
Chris Davies
> I found something similar asked and answered using google. That solution
> consists of dumping file paths from find to a file in /tmp and then using
> it as an argument to the --exclude-from argument.
Ah, now that's an interesting approach. Mind you, I'd be inclined to
use the --files-from list rather than mucking around with --include
and --exclude.
> That brings us to something like your solution. I have to admit that it
> might be the only thing that might ultimately work. However, it involves
> making a temporary archive of links using cpio, and I am trying to keep
> this as simple as possible.
Fair enough, but see below.
I want to first find out why the find (above)
> does not generate an acceptable exclude list for rsync.
There are all sorts of strange (to me) interactions between the --include
and --exclude flags. Try --files-from instead. This (seems to) work
for me with version 3.0.3 of rsync:
DIR=.
find "$DIR" -mtime -10 -depth >/tmp/files
rsync -avHP --files-from=/tmp/files "$DIR" remote:/path/to/backup/
rm -f /tmp/files
Chris