ZFS vs. Opendedup

[Oscar Segarra]

Hi, 

I'm deploying a VDI solution and I'd like to dedup the file system. 

I'd like to know the difference between zfs and opendedup in terms of resilience, dedup ratio, performance, etc...

Anybody has experience with theese two systems and can share its own expericence?

Thanks in advance!

[Chip Burke]

I am actually using BOTH. I use ZFS as the base file system and also to compress the SDFS metadata which I put in its own dataset. So for example:

mkfs.sdfs --volume-name=sdfsvol1 --volume-capacity=50GB --base-path=/somezfsvol1/sdfsvol1/ --chunk-store-hashdb-location=/somezfsvol2/sdfsvol1/hdb/ --dedup-db-store=/somezfsvol2/sdfsvol1/ddb/ --io-log=/somezfsvol2/sdfsvol1/ 

Then, on /somezfsvol2/sdfsvol1 I run

# zfs set compression=lz4

SDFS already compresses its chunk data via lz4, so I leave compression=off (which is default) on /somezfsvol1/sdfsvol1/

I also let ZFS run RAID on my underlying JBOD.

I find SDFS to use many fewer resources than ZFS deduplication. So my full layout looks like:

RAID 1 boot HDDs encrypted with LUKS (encrypts the SDFS keys in the XML)

Pair of SSDs with:

ZFS mirrored ZIL partition

ZFS mirrored pool partition for the SDFS metadata with 'zfs set compression=lz4' (see --chunk-store-hashdb-location, --dedup-db-store and --io-log)

ZFS L2ARC partitions (not mirrored)

12 HDDs in a ZFS RAID 1+0 for the actual SDFS chunk data. Chunk data is encrypted via the native SDFS encryption. Chunk data is compressed via the native SDFS compression. 128k chunk sizes in SDFS and 128k record size in ZFS.

My actual SDFS creation command looks like:

mkfs.sdfs --volume-name=repo1 --volume-capacity=10752GB --base-path=/veeamrepo/repo1/ --chunk-store-hashdb-location=/sdfs_ddb/repo1/hdb/ --dedup-db-store=/sdfs_ddb/repo1/ddb/ --io-log=/sdfs_ddb/repo1/ --chunk-store-encrypt=true --hash-type=VARIABLE_MURMUR3 --io-safe-close=false --io-safe-sync=true --io-chunk-size=128

[Pavel Tolstokorov]

I'm using ZFS without deduplication, because it's need a LOT of ram . If something goes wrong it's need a lot + more ram. So until ill get at least 100GB of ram I will not thinking about dedup with ZFS. But I like zfs for stability , it was survive 10 powerloss in one week without corruption of data.

That's why I'm looking to SDFS.
But I have only 1Tb license, so I can't tell you much about it, I'm just start testing.
For ~850 Gb of misc .ovf files I get about 480GB of used space.
But couple years ago data on SDFS was damaged on powerfail tests.
But I'm shure you have UPS.

If I can get larger license key , I will able to test vm with hba passthrough.

4 июля 2016 г. 3:41 пользователь "Chip Burke" <burke...@gmail.com> написал:

--
You received this message because you are subscribed to the Google Groups "dedupfilesystem-sdfs-user-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedupfilesystem-sdfs-user-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.