redo doesn't work well on virtual file systems (VirtualBox shared folders)

[Mildred Ki'Lya]

Hi,

Redo has a problem when running both inside and outside virtualbox
shared folders. This is an issue I found a year ago, and only recently
understood. And it is the reason of the --overwrite flag I introduced.
The setup is quite simple, I have a project that uses redo as its build
system, and the project is build both inside and outside a virtual
machine (VirtualBox). The project directory is shared using VirtualBox
shared folders.

The problem: Files generated on one side are always marked as overrides
in the other side.

Example: I build a target on the VM, if I try to build it again on the
host, the file is found modified and won't be rebuilt. Unless
--overwrite option is given. The same is true if the file is built on
the host and later rebuilt on the VM.

Why: Because the file stamp contains the file inode, and when in the VM,
the file has a different inode (because of the virtual file system). I
believe it could also happen with fuse file systems.

My question is the following: why is there the inode number in the file
stamp? I am tempted to drop it entirely. Perhaps there should be a
special comparison that marks the file as dirty and not override if the
inode is the only thing different between the recorded stamp and the
actual stamp.

An extract from the debug log:

redo: ?./config
redo: expect: 1357722535.73-1357722535.73-84-11017654+74 (recorded
stamp)
redo: old: 1357722535.73-1357722535.73-84-11017654+74 (recorded
stamp)
redo: new: 1357722535.73-1357722535.73-84-397+74 (the actual
stamp computed from the file)
redo: -- CLEAN (override)


Avery, what is your opinion?

Mildred

[Alan Falloon]

I use vagrant with NFS, and I found a different issue. Due to caching, the ctime field is often changed between calls to stat(). I added a fix here: https://github.com/alanfalloon/redo-rust/commit/a27887fb0d706499168f7d767f6cb9ed14c0b0b1

I'm not sure about the default shared folders. Since I rarely build the same tree both inside and out of my vagrant box (I only need vagrant when I can't build on my host usually), but if you switch to NFS mounted folders, this should fix the builds for you.

[Gyepi SAM]

I think it's not a good idea to use timestamps or inodes to determine file currency.
All of those can change independently of the file contents.

Redux (https://github.com/gyepisam/redux) only uses a hash of the file contents (sha1)
to determine currency and only uses timestamps for small efficiencies.

-Gyepi

[Mildred Ki'Lya]

On mar., 2014-04-08 at 22:05 +0200, Gyepi SAM wrote:
> I think it's not a good idea to use timestamps or inodes to determine file currency.
> All of those can change independently of the file contents.
>
> Redux (https://github.com/gyepisam/redux) only uses a hash of the file contents (sha1)
> to determine currency and only uses timestamps for small efficiencies.
>
> -Gyepi
>
> On Tue, Apr 08, 2014 at 10:46:29AM -0700, Alan Falloon wrote:
> > I use vagrant with NFS, and I found a different issue. Due to caching, the
> > ctime field is often changed between calls to stat(). I added a fix here:
> > https://github.com/alanfalloon/redo-rust/commit/a27887fb0d706499168f7d767f6cb9ed14c0b0b1
> >

I myself worked around this in
https://github.com/mildred/redo/commit/15974a6b6c34dae9a93bdf62aaaaa5dfc7ec6511

The solution I found was to check the device id and inode number stored
in the .deps file, with the actual file. if the information do not
match, I consider the target old and in need of rebuild.

I also implemented a format version numer in the .deps file that would
also trigger the build if not matching.

I think it's acceptable to rebuild the file, even if it didn't changed,
but if the filesystem changed. Checksuming for every file would decrease
performance too much I think.

Mildred

[Gyepi SAM]

Comparing file dependencies based on modification times and inodes, etc does
not work 100% of the time and leads to workarounds that never quite close the
gap.

I have not found checksumming to be a problem in redux. Like you, I initially
thought it would decrease performance so I added a check for the mtime first
and skipped the checksum if the mtime had not changed.
However, when I compared the two methods, it turned out that the difference
was not significant so I removed the mtime check. redux always checksums
dependencies now and it's still fast.

On the issue of performance; on my circa 2008 laptop, it takes 9.9 seconds to
run sha1sum on 5 files of size (22M 22M 154M 506M 786M).
That's a long time to wait, but those are large files.

On the other hand, sha1sum on the .go files in redux (32 files
ranging from 4k to 16k), takes less than 0.01 seconds (time (1) doesn't track
smaller units).

Also, keep in mind that git checksums files on commit but no one complains that git is too slow.

A simple solution is always preferable to a complicated solution that requires workarounds.

I think you would be pleasantly surprised if you try it.

-Gyepi