No space left on device - DSE Current Cache Size : 5.49 GB

[bin Sun]

Hi Guys,

I am testing SDFS for two weeks. I got a problem. Here is the details:

I created a SDFS pool named pool2. The size is 6GB. The physical disk space is 40GB. 

this is the command what I used to creat the pool. 

mkfs.sdfs --volume-name=pool2 --volume-capacity=6GB --backup-volume --sdfscli-listen-addr 0.0.0.0 --sdfscli-listen-port 6444 --sdfscli-require-auth --sdfscli-disable-ssl

/dev/sdd                                   40G  7.8G   33G  20% /opt/sdfs/volumes/pool2

sdfs:/etc/sdfs/pool2-volume-cfg.xml:6444  6.1G  5.4G  752M  88% /opendedupe/volumes/pool2

I copied two big files (one is 4GB and the other one is 1.5GB). everything was fine. after that I remove these two files and then I tried to copy another big file (4GB) to this pool. I got "No space left on device" error. 

Here is the output of  sdfscli --dse-info

# sdfscli --dse-info --nossl --port=6444

DSE Max Size : 6.06 GB

DSE Current Size : 6.4 GB

DSE Compressed Size : 5.33 GB

DSE Percent Full : 105.58%

DSE Page Size : 41943040

DSE Blocks Available for Reuse : 0

Total DSE Blocks : 1390765

Average DSE Block Size : 4940

DSE Current Cache Size : 5.49 GB

DSE Max Cache Size : 10 GB

Trottled Read Speed : 0 B/s

Trottled Write Speed : 0 B/s

Encryption Key : VmJss3mr-1q5h1xwj-L2cmB8r10DDNgEgSte4sxxpKCjSIxThwmFT-xXTDW8sC8G

Encryption IV : 4d66c0e70cff41ab7e0b5c7da1cb64da

and then I tried to run command "sdfscli --cleanstore=1 --port=6444 --nossl" to clean the GC. 

[root@nbumed01 pool2]# sdfscli --cleanstore=1 --port=6444 --nossl

Cleaning store

Garbage Collection Task Completed Successfully : SDFS Volume Cleanup Finished for pool2

[root@nbumed01 pool2]# sdfscli --dse-info --nossl --port=6444

DSE Max Size : 6.06 GB

DSE Current Size : 6.4 GB

DSE Compressed Size : 5.33 GB

DSE Percent Full : 105.58%

DSE Page Size : 41943040

DSE Blocks Available for Reuse : 0

Total DSE Blocks : 690549

Average DSE Block Size : 9950

DSE Current Cache Size : 5.49 GB

DSE Max Cache Size : 10 GB

Trottled Read Speed : 0 B/s

Trottled Write Speed : 0 B/s

Encryption Key : VmJss3mr-1q5h1xwj-L2cmB8r10DDNgEgSte4sxxpKCjSIxThwmFT-xXTDW8sC8G

Encryption IV : 4d66c0e70cff41ab7e0b5c7da1cb64da

DSE Current Cache Size is still 5.49GB. and I still cannot copy the last one (4GB) file to this volume with "No space left on device". 

and then I tried to copy the first (4GB) file to this volume again. I got the same error "No space left on device". 

So I have the question. How to clean DSE Current Cache manually? What is DSE Current Cache? I cannot accept to remove the chucks under "/opt/sdfs/volumes/pool2/chunkstore/chunks". because it will remove everything not just the cache. For example, I copy two files (4GB and 1.5GB) to this volume. then I delete the 4GB file. and then I want to copy another 4GB file to this volume will be failed. I cannot remove the chunks. Because I still have 1.5GB file in this volume.

Please help me if you know how to fix this problem. Thanks in advance!

BTW. I tried to change DSE Max Cache Size to 1GB But did not fix this issue. 

BR

Bin

Here is the xml file. 

<?xml version="1.0" encoding="UTF-8" standalone="no"?><subsystem-config version="3.7.8">

<locations dedup-db-store="/opt/sdfs/volumes/pool2/ddb" io-log="/opt/sdfs/volumes/pool2/ioperf.log"/>

<io chunk-size="40960" claim-hash-schedule="0 59 23 * * ?" dedup-files="true" hash-type="VARIABLE_MD5" log-level="1" max-file-inactive="900" max-file-write-buffers="80" max-open-files="20" max-variable-segment-size="128" meta-file-cache="512" read-ahead="false" safe-close="true" safe-sync="true" variable-window-size="48" volume-type="backup" write-threads="8"/>

<permissions default-file="0644" default-folder="0755" default-group="0" default-owner="0"/>

<sdfscli enable="true" enable-auth="true" listen-address="0.0.0.0" password="eb534c87b545545954387cd2ff8afeaee9ef444f9ce3fdb8d5f53a691d4102ba" port="6444" salt="InNTqs" use-ssl="false"/>

<local-chunkstore allocation-size="6442450944" average-chunk-size="8192" chunk-store="/opt/sdfs/volumes/pool2/chunkstore/chunks" chunkstore-class="org.opendedup.sdfs.filestore.BatchFileChunkStore" cluster-config="/etc/sdfs/jgroups.cfg.xml" cluster-dse-password="admin" cluster-id="sdfscluster" compress="true" disable-auto-gc="true" encrypt="false" encryption-iv="4d66c0e70cff41ab7e0b5c7da1cb64da" encryption-key="VmJss3mr-1q5h1xwj-L2cmB8r10DDNgEgSte4sxxpKCjSIxThwmFT-xXTDW8sC8G" fpp=".001" gc-class="org.opendedup.sdfs.filestore.gc.PFullGC" hash-db-store="/opt/sdfs/volumes/pool2/chunkstore/hdb-1125108407374687410" hashdb-class="org.opendedup.collections.RocksDBMap" io-threads="8" low-memory="false" max-repl-batch-sz="128">

<extended-config allow-sync="false" block-size="60 MB" delete-unclaimed="true" io-threads="16" local-cache-size="10 GB" map-cache-size="100" read-speed="0" sync-files="false" upload-thread-sleep-time="1500" write-speed="0"/>

</local-chunkstore>

<volume allow-external-links="true" capacity="6 GB" closed-gracefully="false" cluster-block-copies="2" cluster-datacenter="datacenter1" cluster-id="sdfscluster" cluster-rack-aware="false" cluster-response-timeout="40960000" compress-metadata="true" current-size="0" dse-comp-size="4270694591" dse-size="4321315829" duplicate-bytes="151872532" files="0" maximum-percentage-full="0.95" name="pool2" offline="false" path="/opt/sdfs/volumes/pool2/files" perf-mon-file="/opt/sdfs//logs/volume-pool2-perf.json" read-bytes="0.0" read-timeout-seconds="-1" serial-number="1125108407374687410" sync-files="false" use-dse-capacity="true" use-dse-size="true" use-perf-mon="false" volume-clustered="false" write-bytes="99785708" write-timeout-seconds="-1"/></subsystem-config>

[Rinat Camal]

Hi!

This is known issue

https://github.com/opendedup/sdfs/issues/88

[bin Sun]

Hi Rinat,

All version of SDFS has this issue? or just 3.7.8.

[richman...@gmail.com]

Hi

updated https://github.com/opendedup/sdfs/issues/88 the issue fixed in 3.12