Torus Consistency Checking

[Fred Whipple]

Hi all,

Apologies if this is a more user question but since Torus is quite new I thought this group may be more appropriate.

Understanding that Torus is still quite new, and also understanding that comparing a filesystem to distributed block storage is quite different, I'm trying to determine if Torus' consistency checking is similar to ZFS' consistency checking and at what times that consistency is checked.  From the early docs, I believe I read that Torus does a CRC check at write time, which will ensure the data is written correctly to all storage nodes.  Similar with ZFS.  When reading, ZFS will check the data it's reading and, even better, if you have RAID1/Z, it will transparently give you the "correct" block and then auto-repair the "incorrect" one.  Does or will Torus do something similar?

The context is obviously to provide not only high availability but also high reliability.  It occurred to me that you could always create two volumes and run ZFS on them, but assuming redundancy at the Torus level was 2, you'd have 4 copies of the same data.  I suppose you could run two Torus networks on separate hardware with redundancy at 1 and run ZFS across them.

Thanks in advance!

     -Fred

[Brandon Philips]

cc'ing Barak.

[Barak Michener]

Understanding that Torus is still quite new, and also understanding that comparing a filesystem to distributed block storage is quite different, I'm trying to determine if Torus' consistency checking is similar to ZFS' consistency checking and at what times that consistency is checked.  From the early docs, I believe I read that Torus does a CRC check at write time, which will ensure the data is written correctly to all storage nodes.  Similar with ZFS.  When reading, ZFS will check the data it's reading and, even better, if you have RAID1/Z, it will transparently give you the "correct" block and then auto-repair the "incorrect" one.  Does or will Torus do something similar?

Yep, it checks (and saves) CRC at write time, and checks it again at read time. Right now, when CRC check fails, the block fails, but it needn't end there. Xiang's https://github.com/coreos/torus/pull/227 expands the interface in question to transparently repair the correct one, by any of the usual error correction schemes.

The context is obviously to provide not only high availability but also high reliability.  It occurred to me that you could always create two volumes and run ZFS on them, but assuming redundancy at the Torus level was 2, you'd have 4 copies of the same data.  I suppose you could run two Torus networks on separate hardware with redundancy at 1 and run ZFS across them.

You could, but your intuition on how that would play out is exactly correct :)

[Fred Whipple]

Awesome!  I'm very excited about this project.