Atomicity idea.
Kaz Kylheku
commits and tagging atomic.
One idea is to do it in a lower layer, and perform only minimal
instrumentation of the CVS logic: basically insert a few calls to the
transactional layer into a few key places in CVS.
That lower layer would replace the POSIX filesystem functions with
ones that support transactions. The idea is that the application can
call a special function which starts a transaction. It then executes
arbitrary filesystem mutating calls: writing to files, renaming
temporary files to permanent ones, and so on. The transaction is then
terminated with an abort or commit. If the machine dies during the
transaction, then it's as if the transaction never happened. If the
client socket dies (client was killed by SIGINT or whatever), then the
transaction is aborted. Again, as if it never happened.
This transactional filesystem could be done in a kernel, or in user
space.
Maybe some of the existing journalling filesystems out there already
have interfaces for doing things like this; I'm not up to date in this
area.
The nice thing about this approach is that it's decoupled from CVS;
you can test it quite independently, and use it to add atomicity to
any software that stores data in multiple files whose contents,
properties and directory structure must be kept coherent as a set.
I know that atomicity is not a real problem; basically the only reason
for doing this would be to squash anti-CVS FUD spread by people who
promote other systems. I also realize that without changing some of
the actual algorithms in CVS, such a transactional system would
require lots of temporary space. (Though compression techniques could
help, exploiting the vast similarity between the uncommited material,
and the old material).
Kaz Kylheku
> I'm looking at the CVS source code, and thinking about ways to make
> commits and tagging atomic.
>
> One idea is to do it in a lower layer, and perform only minimal
> instrumentation of the CVS logic: basically insert a few calls to the
> transactional layer into a few key places in CVS.
I have come up with a very simple, low-risk way for doing this. It
assumes that files contents are never destructively manipulated, which
is true: CVS always writes to temporary files and then moves on top.
Here is the scheme. Before doing a commit or tagging operation on a
directory tree, you make a virtual backup of that tree. How? By
faithfully replacing that tree's directory structure under a temporary
directory, and populating it with a farm of hard links to the objects
in the original.
The CVS operation then takes place in the orignal tree. If everything
goes well, the hard-link-based backup tree is blown away. If the
transaction must be rolled back, then the backup tree is installed in
place of the original, and the original is blown away. This is done
using renaming operations on the top level directory names.
Atomic renames at the top level can handle the logic. The backup tree,
while it is constructed, can be named
.#cvs.backup.<dir-name>.incomplete. When constructed, and the logic
moves on to doing the commit operation it is then called
.#cvs.backup.<dir-name>.complete. Then when it is being deleted, it
can go back to the .incomplete name.
During rollback, the original directory is renamed to some special
name before it is recursively deleted, and then the backup directory
is renamed to the original directory name.
Crash recovery can be performed by simple shell scripts, which look
for the presence of these directory names and decide what to blow away
and what to keep.
Derek Robert Price
>Crash recovery can be performed by simple shell scripts, which look
>for the presence of these directory names and decide what to blow away
>and what to keep.
>
>
Better yet, always perform the operation on the duplicate tree. Then
the duplicate copies can always be blown away when it is discovered
their supervising process is missing.
Derek
--
*8^)
Email: de...@ximbiot.com
Get CVS support at <http://ximbiot.com>!
--
Barium: what you do with dead chemists.
Kaz Kylheku
> Kaz Kylheku wrote:
>
> >Crash recovery can be performed by simple shell scripts, which look
> >for the presence of these directory names and decide what to blow away
> >and what to keep.
> >
> >
>
> Better yet, always perform the operation on the duplicate tree. Then
> the duplicate copies can always be blown away when it is discovered
> their supervising process is missing.
True; even if the transaction completed fully in the duplicate just
before the interruption, you can blow it away.
However, this adds more complexity to CVS, because it has to shunt its
repository access to the duplicate. I'd rather add a little
complexitity to the simple shell scripts that do the crash recovery.
The virtue of working on the original while maintaining a backup is
that, in theory, we can modify CVS simply by instrumenting it with
advice before and after a commit or tagging operation, but otherwise
completely leave it alone.
It's like aspect oriented programming. Atomicity is modeled as an
orthogonal concern to everything else, and hacked using auxiliary
(before and after) methods on selected operations, which aren't aware
of anything.
Low-risk preservation of stability is the key.
I have some more detailed ideas about the design.
The backup and rollback actions should be nested within the protection
of the CVS lock. This handles the case of concurrent commits being
tried at different levels of the same tree, and other problems.
The acquisition of the lock means that there won't be any concurrent
backing up going on. When you mkdir() the top-level backup directory,
you can be confident that nobody is trying the same thing. Moreover,
the checkout or update -d logic does not have to recognize the backup
directories and skip over them.
The backup procedure has to watch out for lock files and directories
and skip over them. The backup will not have any locks. That is
convenient---if you have to roll back, there will be no locks to
remove: just do some top-level renames, and the backup is installed in
place of the scrapped tree, unlocked and ready to go.
However, the following concern occured to me now. What if a process is
trying to acquire a lock while a transaction is rolled back? The
backup directory is then renamed to the original name, and the process
that is trying to get a lock will hang in the discarded original tree,
hammering away on a lock that doesn't exist.
I think that this case may just evaporate when that tree is
recursively deleted. The problem is that the process will be chdir'ed
into there, and hang on to the directory i-node. Maybe it can peek at
the directory link count to detect that it's in an orphaned directory
and bail out.
I don't know how various filesystems handle this. On Linux's ext2, for
instance, if your current directory has been deleted, you get EPERM
errors trying to create new directory entries. (Hence my ``evaporate''
hope in the above paragraph). But I'm not sure about this because I
tried it only interactively. There could be a race in the kernel
whereby file creation can work briefly in a blown-away directory. Yes,
in fact you have to blow away the files first, and then the directory.
This means blowing away the lock files, which could just give the
other process the opportunity to seize the lock. Ah but in that case
the recursive removal won't be able to blow away the directory. Argh.