OpenDedup biggest problem

[Rinat Camal]

Hello!

I believe this is biggest problem I was facing with OpenDedup in all my tests.

Now I found a way to reproduce it.

1. create new local sdfs volume - “sdfs01”

2. mount and umount it.

3. check size of chunkstore

4. mount it again.

5. copy files on SDFS volume.

debian-10.0.0-amd64-xfce-CD-1.iso (641M, you can get it from https://debian.org) three times.

CentOS-7-x86_64-DVD-1810.iso 90 (4Gb, you can get it from https://www.centos.org) once.

6. Chunk store grown as expected and deduplicated this copies.

7. remove all files from sdfs volume.

8. check sizes of chunkstore.

9. run command “sdfscli --compactcleanstore=0 –cleanstore=0” or  “sdfscli --compactcleanstore=1 –cleanstore=1”

10. check size of chunkstore again

Result: folder chunks is not reducing in size, whatever i tried.

Expected: if chunk is not used it must be removed from physical  disk after some time(define in config) or immediately(preferred, but maybe 1 minute later can be accepted).

PS.
my test environment VM Debian 10, 4CPU, 12GB RAM, 24GB disk, 12GB sdfs volume, 

[Mike]

Rinat - I can confirm I have this same problem with Ubuntu Server. the cleanstore command says that it was successful but the size is still the same, I don't have a solution yet unfortunately

[Sam Silverberg]

The size should reduce over time (less that 1 hour) let me know if this is not the case.

On Tue, Jul 16, 2019 at 12:25 PM Mike <mousem...@gmail.com> wrote:

Rinat - I can confirm I have this same problem with Ubuntu Server. the cleanstore command says that it was successful but the size is still the same, I don't have a solution yet unfortunately

--
You received this message because you are subscribed to the Google Groups "dedupfilesystem-sdfs-user-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedupfilesystem-sdfs-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedupfilesystem-sdfs-user-discuss/f0d26a23-a76a-4dd2-92e9-91a7f3a9967b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[richman...@gmail.com]

Hello!

nope. It is running for 3 days now - still same size.

I also changed GC to run every minute(from once a week). Volume is empty anyway.

To view this discussion on the web visit https://groups.google.com/d/msgid/dedupfilesystem-sdfs-user-discuss/CAE9URKGYb41ZvHWKx83jeGW7f2n%3DGh8YeRWEpdarBZT%2BDd32Cg%40mail.gmail.com.

[bin Sun]

Hi Sam,

I have the same issue with CentOS Server. 

On Wednesday, July 17, 2019 at 3:52:41 AM UTC+8, Sam Silverberg wrote:

The size should reduce over time (less that 1 hour) let me know if this is not the case.

On Tue, Jul 16, 2019 at 12:25 PM Mike <mousem...@gmail.com> wrote:

Rinat - I can confirm I have this same problem with Ubuntu Server. the cleanstore command says that it was successful but the size is still the same, I don't have a solution yet unfortunately

--
You received this message because you are subscribed to the Google Groups "dedupfilesystem-sdfs-user-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dedupfilesystem-sdfs-user-discuss+unsubscribe@googlegroups.com.

[liu lei]

same problem,same sdfs version.

在 2019年7月15日星期一 UTC+8下午11:48:17，Rinat Camal写道：

[Rinat Camal]

As additional test I tried to reduce size of SDFS volume after.

I looks like Garbage-collector is runs but not removing any data.  chunkstore not reducing in size

[Rinat Camal]

Maybe I can generate debug info and send it to you?

PS. I understand that Windwos 2019 already have a de-duplication, but I want to use Open-source variant.

[Paul Soucy]

I have the same problem with Ubuntu 18.04!

[Rinat Camal]

I found it!!
On version SDFS "3.5.4.0"
command "sdfscli --password  --cleanstore" with password was working and cleaning garbage chunks!!

but on newer version chunk GC is broken.

[Louis van Dyk]

I'm running

Version     : 3.7.8

I found that if checked the disk used by the chunkstore:

# du -hs /media/500GB-BKUP/sdfs-volumes/chunkstore/
24G /media/500GB-BKUP/sdfs-volumes/chunkstore/

Then I ran:

# sdfscli --cleanstore  --password admin
Cleaning store
Garbage Collection Task Completed Successfully : SDFS Volume Cleanup Finished for pool0

and then checked the disk usage again there was no change:

# du -hs /media/500GB-BKUP/sdfs-volumes/chunkstore/
24G /media/500GB-BKUP/sdfs-volumes/chunkstore/

But, if I then ran:

# sdfscli --compactcleanstore  --password admin
Cleaning store
Garbage Collection Task Completed Successfully : SDFS Volume Cleanup Finished for pool0

when I checked the disk usage again, there WAS a change:

[root@fedora ~]# du -hs /media/500GB-BKUP/sdfs-volumes/chunkstore/
23G /media/500GB-BKUP/sdfs-volumes/chunkstore/

I haven't tested with only using the compactcleanstore option yet, but I thought I would share this result so long ...

--
You received this message because you are subscribed to the Google Groups "dedupfilesystem-sdfs-user-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dedupfilesystem-sdfs-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedupfilesystem-sdfs-user-discuss/aaa45da6-0f65-4a58-80b3-3a555105fed7%40googlegroups.com.

[Rinat Camal]

Hi!
no. you making comparison of wrong folder.

"hdb-..." cleared successfully

"chunks " is not cleared at all

[Rinat Camal]

I think in new version 3.10.8  it shows what is wrong with cleanup.

command “sdfscli --compactcleanstore=0 –cleanstore=0” or  “sdfscli --compactcleanstore=1 –cleanstore=1”  causes this error in log files. and sometimes space is returned a little bit

[Rinat Camal]

I think I found it!

in SDFS xml config change set
<local-chunkstore sync-files="true" />
<volume sync-files="true" />

after this run

sdfscli --compactcleanstore
sdfscli --cleanstore
sdfscli --compactcleanstore
sdfscli --sync-fs-size

PS. If someone able to reduce chunk store size successfully, please, respond to the chat.