Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
two-way rsync again, with ZFS snapshots maybe?
41 views
Skip to first unread message
andyw...@gmail.com
Jun 3, 2014, 4:02:23 AM6/3/14
to
Dear All,
Main purpose for this post is to make sure that I am not re-inventing the wheel, and, possibly, get some fresh ideas or constructive criticism.
I am breaking my head trying to sync Samba servers in two remote sites.
Of course I've got the worst scenario possible - users can remove, create, rename and modify files and rename/move large directory subtrees randomly on both sides.
Good news is only that users rarely work on the same directories concurrently, usually everyone has his own subtree to mess with.
I know rsync is not recommended even by it's developers for such scenarios,
but my choice is quite limited - Unison is not available on one peer, which is some ancient PowerPC server running ancient Linux with no development tools. Only basics like rsync, bash and ssh. Distributed filesystems don't seem to be an option either, partially for the same reason, and also for local Samba performance. Objections are more than welcome, though.
I've also looked at various rsync wrappers such as csync and osync, but, since quite complicated wrappers are already in place, I'd prefer to implement the same ideas in my own bash code, possibly utilising the luxury of ZFS snapshots (other side, luckily, runs Solaris 11).
At the moment, temporarily, the PowerPC side is announced to users as
sort-of-master, where changes are fully allowed, and guaranteed to propagate to Solaris side, whereas Solaris side is more like backup, and new files may be lost at forward sync (PPC-Sol) and only manually recovered from ZFS snapshot.
However, I can not keep this limitation for too long and need to come up with a more transparent solution.
Basically, now I do the following (triggered by cron on Solaris side)
zfs snapshot $FS@`timestamp`-before
rsync -avzru --delete --exclude-from=$EXFILE --rsh=$TunnOpts $PPC:$FS $FS
rsync -avzru --exclude-from=$EXFILE --rsh=$TunnOpts $FS $PPC:$FS
zfs snapshot $FS@`timestamp`-after
My idea would be, on Solaris side:
diff the file lists between
snapshot-after-previos-rsync
and
snapshot-before-new-rsync
This should give me the list of files *created* on Solaris side since last sync.
Then run 1st rsync (PPC-to-Sol with --delete) - this *may* delete new files on Solaris side, since they don't exist on PPC. Then, restore wrongly removed files, using the snapshot-before (cd ...zfs/.. && cat list | cpio -dump ?).
Solution is still very far from ideal, specifically:
- diff list may contain not only deleted, but moved files/subdirs, which makes consequent delete/restore-from-snapshot-using-cpio quite risky.
- I am not sure if I can rely on file timestamps in snapdir and play with "find -newer" options there. I've seen that ls -lc, ls -lm sometimes report quite strange file timestamps under .zfs tree.
Any comments, suggestions?
Thank you
Andrei
Main purpose for this post is to make sure that I am not re-inventing the wheel, and, possibly, get some fresh ideas or constructive criticism.
I am breaking my head trying to sync Samba servers in two remote sites.
Of course I've got the worst scenario possible - users can remove, create, rename and modify files and rename/move large directory subtrees randomly on both sides.
Good news is only that users rarely work on the same directories concurrently, usually everyone has his own subtree to mess with.
I know rsync is not recommended even by it's developers for such scenarios,
but my choice is quite limited - Unison is not available on one peer, which is some ancient PowerPC server running ancient Linux with no development tools. Only basics like rsync, bash and ssh. Distributed filesystems don't seem to be an option either, partially for the same reason, and also for local Samba performance. Objections are more than welcome, though.
I've also looked at various rsync wrappers such as csync and osync, but, since quite complicated wrappers are already in place, I'd prefer to implement the same ideas in my own bash code, possibly utilising the luxury of ZFS snapshots (other side, luckily, runs Solaris 11).
At the moment, temporarily, the PowerPC side is announced to users as
sort-of-master, where changes are fully allowed, and guaranteed to propagate to Solaris side, whereas Solaris side is more like backup, and new files may be lost at forward sync (PPC-Sol) and only manually recovered from ZFS snapshot.
However, I can not keep this limitation for too long and need to come up with a more transparent solution.
Basically, now I do the following (triggered by cron on Solaris side)
zfs snapshot $FS@`timestamp`-before
rsync -avzru --delete --exclude-from=$EXFILE --rsh=$TunnOpts $PPC:$FS $FS
rsync -avzru --exclude-from=$EXFILE --rsh=$TunnOpts $FS $PPC:$FS
zfs snapshot $FS@`timestamp`-after
My idea would be, on Solaris side:
diff the file lists between
snapshot-after-previos-rsync
and
snapshot-before-new-rsync
This should give me the list of files *created* on Solaris side since last sync.
Then run 1st rsync (PPC-to-Sol with --delete) - this *may* delete new files on Solaris side, since they don't exist on PPC. Then, restore wrongly removed files, using the snapshot-before (cd ...zfs/.. && cat list | cpio -dump ?).
Solution is still very far from ideal, specifically:
- diff list may contain not only deleted, but moved files/subdirs, which makes consequent delete/restore-from-snapshot-using-cpio quite risky.
- I am not sure if I can rely on file timestamps in snapdir and play with "find -newer" options there. I've seen that ls -lc, ls -lm sometimes report quite strange file timestamps under .zfs tree.
Any comments, suggestions?
Thank you
Andrei
Dirk Heinrichs
Jun 7, 2014, 3:25:44 PM6/7/14
to
andyw...@gmail.com wrote:
> Good news is only that users rarely work on the same directories
> concurrently, usually everyone has his own subtree to mess with. I know
> rsync is not recommended even by it's developers for such scenarios, but
> my choice is quite limited - Unison is not available on one peer, which is
> some ancient PowerPC server running ancient Linux with no development
> tools. Only basics like rsync, bash and ssh. Distributed filesystems don't
> seem to be an option either, partially for the same reason, and also for
> local Samba performance. Objections are more than welcome, though.
Which distributed fs did you look at? Especially if everyone only works in
> Good news is only that users rarely work on the same directories
> concurrently, usually everyone has his own subtree to mess with. I know
> rsync is not recommended even by it's developers for such scenarios, but
> my choice is quite limited - Unison is not available on one peer, which is
> some ancient PowerPC server running ancient Linux with no development
> tools. Only basics like rsync, bash and ssh. Distributed filesystems don't
> seem to be an option either, partially for the same reason, and also for
> local Samba performance. Objections are more than welcome, though.
his/her own subtree, OpenAFS could be an option, due to its read-only
replication capabilities. However, I'm afraid you'd need to look into adding
more packages to (or even replace) your ancient PPC box.
HTH...
Dirk
--
Dirk Heinrichs <dirk.he...@altum.de>
Tel: +49 (0)2471 209385 | Mobil: +49 (0)176 34473913
GPG Public Key CB614542 | Jabber: dirk.he...@altum.de
Sichere Internetkommunikation: http://www.retroshare.org
0 new messages