Rsync problem

[Tom Friedetzky]

Hi all, I'm trying to achieve something rather generic-seeming with
rsync (3.1.5) but am struggling a little. I've got a (file) server and
a client (actually, two of them). What can happen is that

- files may get added to the server at arbitrary times
- files may get deleted from the client at arbitrary times

When I say "arbitrary times" we may safely assume that no modifications
take place during synchronisations. Anyway, what I'd like to achieve is
that

- the client-side deletions since the last sync get synced to the server,
but *without* the server-side additions that may have occurred since that
last sync being deleted, and

- the server-side additions since the last sync get synced to the client,
but *without* the client-side deletions that may have occurred since last
sync being copied there again.

Would the best be to write a script for actually adding stuff
server-side that logs exactly what has been added since last sync, and
use that as an exclude file, so that a subsequent sync would correctly
sync client-side deletions to the server, but would not delete those
files recently added to the server? (I can't easily log client-side
deletions.) Am I missing something obvious and trivial?

[Ian Zimmerman]

On 2016-11-14 06:18, Tom Friedetzky wrote:

> Hi all, I'm trying to achieve something rather generic-seeming with
> rsync (3.1.5) but am struggling a little. I've got a (file) server and
> a client (actually, two of them). What can happen is that
>
> - files may get added to the server at arbitrary times
> - files may get deleted from the client at arbitrary times

unison, or git/hg if your files are text, or maybe something like git-annex.

--
Please *no* private Cc: on mailing lists and newsgroups
Personal signed mail: please _encrypt_ and sign
Don't clear-text sign: http://cr.yp.to/smtp/8bitmime.html

[Thomas 'PointedEars' Lahn]

Tom Friedetzky wrote:

> When I say "arbitrary times" we may safely assume that no modifications
> take place during synchronisations. Anyway, what I'd like to achieve is
> that
>
> - the client-side deletions since the last sync get synced to the server,
> but *without* the server-side additions that may have occurred since
> that last sync being deleted, and
>
> - the server-side additions since the last sync get synced to the client,
> but *without* the client-side deletions that may have occurred since
> last sync being copied there again.
>
> Would the best be to write a script for actually adding stuff
> server-side that logs exactly what has been added since last sync, and
> use that as an exclude file, so that a subsequent sync would correctly
> sync client-side deletions to the server, but would not delete those
> files recently added to the server?

No.

> (I can't easily log client-side deletions.) Am I missing something
> obvious and trivial?

No.

--
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.

[Tom Friedetzky]

On Mon Nov 14 2016 at 16:41:37 UTC, Michael Vilain <mev9...@yahoo.com> wrote:
> In article <slrno2ilqh....@friedetzky.org>,

> Sounds like you're talking about a "backup mode" on updates during
> rsync, akin to Oracle setting aside all transactions while backups are
> ongoing, then applying updates after the backups are complete.
>
> AFAIK, rsync won't do that. It takes a snapshot of the files on the
> remote system and the local system and does the syncing based on those
> snapshots. Of course, you're welcome to roll you own version of rsync
> but why bother.
>
> You could futz with the mount points, essentially moving the location of
> where the files are stored during backups to an alternative location.
> Updates would occur there while the sync happens, then you'd have to
> apply the updated files in the backup directory to the parent directory
> to get everything in sync.
>
> Another method is to double mirror (e.g. there are three copies of the
> filesystem, all syncronized) the volume where the backups are happening,
> break a mirror and sync that, then reconnect the mirror when rsync is
> done. Let the mirror "re-silver" to sync everything up again.
>
> I don't see how to do this with just rsync.

Thanks. I was afraid it wasn't going to be simple. Maybe I'll just chuck
the lot into Dropbox and be done with it...

[Tom Friedetzky]

On Mon Nov 14 2016 at 17:50:48 UTC, Ian Zimmerman <i...@primate.net> wrote:
> On 2016-11-14 06:18, Tom Friedetzky wrote:
>
> > Hi all, I'm trying to achieve something rather generic-seeming with
> > rsync (3.1.5) but am struggling a little. I've got a (file) server and
> > a client (actually, two of them). What can happen is that
> >
> > - files may get added to the server at arbitrary times
> > - files may get deleted from the client at arbitrary times
>
> unison, or git/hg if your files are text, or maybe something like git-annex.

They aren't text unfortunately, but binary. And there's rather a lot of
the stuff, some 700GB in total (though typically the updates are just a
few MB at a time).

[Tom Friedetzky]

Feeling better now?

[Thomas 'PointedEars' Lahn]

Tom Friedetzky wrote:

> […] Thomas 'PointedEars' Lahn […] wrote:

>> Tom Friedetzky wrote:
>> > Would the best be to write a script for actually adding stuff
>> > server-side that logs exactly what has been added since last sync, and
>> > use that as an exclude file, so that a subsequent sync would correctly
>> > sync client-side deletions to the server, but would not delete those
>> > files recently added to the server?
>>
>> No.
>>
>> > (I can't easily log client-side deletions.) Am I missing something
>> > obvious and trivial?
>>
>> No.
>
> Feeling better now?

<http://catb.org/esr/faqs/smart-questions.html>

HTH

[Ben Bacarisse]

Tom Friedetzky <tom-...@friedetzky.org> writes:

> On Mon Nov 14 2016 at 17:50:48 UTC, Ian Zimmerman <i...@primate.net> wrote:
>> On 2016-11-14 06:18, Tom Friedetzky wrote:
>>
>> > Hi all, I'm trying to achieve something rather generic-seeming with
>> > rsync (3.1.5) but am struggling a little. I've got a (file) server and
>> > a client (actually, two of them). What can happen is that
>> >
>> > - files may get added to the server at arbitrary times
>> > - files may get deleted from the client at arbitrary times
>>
>> unison, or git/hg if your files are text, or maybe something like git-annex.
>
> They aren't text unfortunately, but binary.

So do consider unison. I find it very useful.

> And there's rather a lot of
> the stuff, some 700GB in total (though typically the updates are just a
> few MB at a time).

It may take a while for it to set up all the meta-data, but it should be
reasonably fast after that. I found the man page a little on the light
side, but there is other documentation.

--
Ben.

[Thomas 'PointedEars' Lahn]

Ben Bacarisse wrote:

> Tom Friedetzky <tom-...@friedetzky.org> writes:
>> [The files] aren't text unfortunately, but binary.

>
> So do consider unison. I find it very useful.
>
>> And there's rather a lot of the stuff, some 700GB in total (though
>> typically the updates are just a few MB at a time).
>
> It may take a while for it to set up all the meta-data, but it should be
> reasonably fast after that. I found the man page a little on the light
> side, but there is other documentation.

However, I think that, if I understand the OP correctly, this can be done
with two invocations of rsync(1).