OpenDeDup Filesystem seems very unreliable

[Louis van Dyk]

Hi

I've been using OpenDeDup on Linux for a long time now. I use it to make a nightly copy of my /var/spool/mail/louis mail file, as well as my /home/louis/mail folder (which has my sent mail and other folders).  As a result, there is a LOT of duplicated data.

I am using a 500 GB external drive to host the sdfs volume files.  Because of the data being on a small drive, when I formatted the SDFS file system, I used the size of 150 GB.

My nightly copy of data is about 17 GB.  I'd been merrily copying data before, hitting 60 days of backups, and then on a whim thought: can I read the data?  Only to find then I try to read any of the files I get an error message.

I figured maybe an unclean shutdown had caused something, so trashed it all because it is technically useless, and formatted again.  It now seems to me that after 6 or 7 nightly backups, that my data goes corrupt. I'm still doing testing on this theory.  I'm wondering if the 150 GB size is exceeded by my logical disk copies, and then loses data.  I've just formatted with a "size" of 1500 GB, so let's see what happens.

OpenDeDup doesn't claim that the filesystem is full though - it's been writing and writing and writing.

Anyone else experience any similar weirdness?

Regards

Louis

[Rinat Camal]

https://github.com/opendedup/sdfs/issues/88

[Rinat Camal]

I have a similar issue.
At this point I actually have 2 issues.
1. SDFS cannot return diskspace.

2. SDFS has an issues with NFS backend.

on NFS the issue is totally reproducible, however on local FS (ext4, xfs) it is not always reproducible.

also you have to monitor DSE size, Volume size, etc.

I usually use this chain command

export port=6442
sdfscli --port $port --debug-info
sdfscli --port $port  --volume-info
sdfscli --port $port  --dse-info

to get all 3 info from the system

[Rinat Camal]

I use SDFS  only for backups.

I'm currently just rotating sdfs volumes when I think it is time to do it.

Pay attention to value DSE Percent Full . If it hits 100% you cannot write to volume, even if on the real disk you have a lot of free space

[Louis van Dyk]

It would really be a good thing if Sam could code in some checks for DSE Percent Full and then not allow writes once it gets exceeded.

I am guessing that's what happened in my case.  I formatted this morning to 1500GB.  I think I will erase it again, and drop it to 150 GB and then test to see if that being exceeded is when my data becomes unreadable.  At least knowing WHY it happens can help me prevent it in future.

I really love the concept of this and really want it to work well.  I have just not had great success with it thus far.  This tip should help immensely if this is in fact the cause.

Thanks Rinat !!

[Rinat Camal]

Wait.
I had such issue several times. Especially when I was playing with memory settings.

DSE can be damaged during shutdown.

Before you recreate volume you should try and rename/delete file "BucketInfo" in folder "chunks".

If this didn't worked you also can  change in configuration  and set "closed-gracefully=false"

[Louis van Dyk]

I had actually erased and formatted the system by the time you wrote this message, so I have waited until an issue occurred .... which was tonight.

Unfortunately I had to hit the reset button on my PC a few days ago because of an issue with the nvidia driver - somehow it locked up my keyboard and mouse. 

Even though I have the sdfs filesystem in my /etc/fstab for some weird reason it never mounts it on boot.  I'd forgotten about my 4 AM backups of my mail spool file (it's 18 GB .... don't judge)!  I needed to mount the /media/opendedup mount point, which failed with the usual java errors that I've come to recognise so frequently.

I followed through with my usual solution:

rm -rf /media/500GB-BKUP/sdfs-volumes/chunkstore/chunks/ ; mount /media/opendedup

The filesystem mounted, and the backup ran.  I then (being awake at 04h30) thought "Let me check if the files are still working". I've had seven mail folder backups so far on the newly formatted system.  And again ....

[root@fedora chunks]# head /media/opendedup/mailbkup/mail-2020-08-04/louis
head: error reading '/media/opendedup/mailbkup/mail-2020-08-04/louis': No data available

What's worse is that a) the backup ran successfully, and b) the other jobs are all also corrupt - the below one worked when it was backed up:

[root@fedora mailbkup]# head mail-2020-07-20/louis
head: error reading 'mail-2020-07-20/louis': No data available

I've tried your suggestions of deleting the BucketInfo file and remounting, and then of changing closed-gracefully to false and remounting.  Neither helped.

For info sake, the output to your three checks for my folder is:

[root@fedora ~]# export port=6442
[root@fedora ~]# sdfscli --port $port --debug-info
Active SDFS Threads : 54
CPU Load : 183%
SDFS CPU Load : 1%
Total Memory : 8.98 GB
Free Memory : 8.49 GB
Free Disk Space for Chunk Store: 151.53 GB
Total Space for Chunk Store: 457.45 GB
[root@fedora ~]# sdfscli --port $port  --volume-info
Files : 6657
Volume Capacity : 1.46 TB
Volume Current Logical Size : 106.45 GB
Volume Max Percentage Full : 95.0%
Volume Duplicate Data Written : 345.71 GB
Unique Blocks Stored: 0 B
Unique Blocks Stored after Compression : 0 B
Cluster Block Copies : 2
Volume Virtual Dedup Rate (Unique Blocks Stored/Current Size) : 0%
Volume Actual Storage Savings (Compressed Unique Blocks Stored/Current Size): 0%
Compression Rate: 0%
[root@fedora ~]# sdfscli --port $port  --dse-info
DSE Max Size : 1.46 TB
DSE Current Size : 0 B
DSE Compressed Size : 0 B
DSE Percent Full : 0.0%
DSE Page Size : 41943040
DSE Blocks Available for Reuse : 0
Total DSE Blocks : 5144090
Average DSE Block Size : 0
DSE Current Cache Size : 73.15 MB
DSE Max Cache Size : 10 GB
Trottled Read Speed : 0 B/s
Trottled Write Speed : 0 B/s
Encryption Key : DmVSVm2wNY7DJnHPf1CYMHGVDtx5KRpNtLgbceVxFqjy=2JFhD2xgzQ9KPWaZHkl
Encryption IV : e4f8d885ba637b103c61f9a3d94f7cac

Incidentally, the folder structures are all there - but I get that read error on any file I try to read.

So, like I originally stated, it seems OpenDeDup is very unreliable with my data!