Why unused space of wiredTiger stay unused?

[Yos Tj]

I believe that wiredTiger reuses unused space after removing documents.
But I can see very large unused space stays unused.

I did a test like below:
- insert 1.0 million documents with random _id to an empty collection,
  each document size is approximately 1KB.
- remove roughly 0.5 million documents picked up randomly
- insert 0.5 million documents with random _id
- remove roughly 0.5 million documents again
- insert 0.5 million documents again
   :
and I got a result of stats() like below:

> db.test1.stats()
{
        "ns" : "test.test1",
        "count" : 999358,
        "size" : 1185238588,
        "avgObjSize" : 1186,
        "storageSize" : 1520132096,

        "wiredTiger" : {
                "block-manager" : {
                        "file allocation unit size" : 4096,
                        "blocks allocated" : 932278,
                        "checkpoint size" : 902328320,
                        "allocations requiring file extension" : 110579,
                        "blocks freed" : 819125,
                        "file magic number" : 120897,
                        "file major version number" : 1,
                        "minor version number" : 0,
                        "file bytes available for reuse" : 591319040,
                        "file size in bytes" : 1520132096

Why 591m still stays unused(available for reuse)?
If they were used efficiently, file size would not go up to 1520m.

Regards,

[michael...@10gen.com]

Hi Yos,

 

Why 591m still stays unused(available for reuse)?
If they were used efficiently, file size would not go up to 1520m.

It's difficult to be certain, but space does not become available in WiredTiger immediately.  At least one checkpoint has to complete in between the removes and the space becoming available.

If you sleep for ~2 minutes after the removes, before inserting again, do you still see the file grow?

Regards,

Michael.

[Yos Tj]

Hi,

 > If you sleep for ~2 minutes after the removes, before inserting again, do you still see the file grow?

Yes, I still see it. As an additional test, I tried gradual removal like this:

- insert 1.0 million docs, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.

- insert 0.5 million docs, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.

- insert 0.5 million docs, then sleep 3min.
- remove 0.1 million docs randomly, then sleep 3min.
  :

But I've got the same result.
  "file bytes available for reuse" : 923234304,  <-- too large!!
  "file size in bytes" : 1483091968

Why wiredTiger won't reuse such large unused space...
Should we know wiredTiger's page allocation and management in depth?

Regards,


2016年4月29日金曜日 11時58分22秒 UTC+9 michael...@10gen.com:

[Yos Tj]

Hi,

 > But I've got the same result.
 >  "file bytes available for reuse" : 923234304,  <-- too large!!
 >  "file size in bytes" : 1483091968

Above was the status after the last removal. The status after the last insertion was:

    "file bytes available for reuse" : 543006720,  <-- too large!!
    "file size in bytes" : 1483046912

Regards,

[Stephen Steneker]

On Wednesday, 11 May 2016 17:47:13 UTC-7, Yos Tj wrote:

Why wiredTiger won't reuse such large unused space...
Should we know wiredTiger's page allocation and management in depth?

Hi,

What specific version of MongoDB are you using?

Thanks,

Stephen

[Yos Tj]

Hi,

 > What specific version of MongoDB are you using?

I'm using 3.0.8. So, I tried with 3.2.6, but got a similar result.

3.0.8:
after the last insertion(1.0 million docs in total):
(data)  file size in bytes :             1,483,046,912
(data)  file bytes available for reuse :   543,006,720
(index) file size in bytes :                70,643,712
(index) file bytes available for reuse :    33,996,800

3.2.6:
after the last insertion(1.0 million docs in total):
(data)  file size in bytes :             1,414,909,952
(data)  file bytes available for reuse :   578,301,952
(index) file size in bytes :                71,667,712
(index) file bytes available for reuse :    35,389,440

Thanks,
Yos

[michael...@10gen.com]

Hi Yos,

3.2.6:
after the last insertion(1.0 million docs in total):
(data)  file size in bytes :             1,414,909,952
(data)  file bytes available for reuse :   578,301,952

 Also note that WiredTiger does not free space in the middle of a file back the operating system.  I think your workload is almost guaranteed to create fragmentation with the default settings in MongoDB.  You might want to try running mongod with:

--wiredTigerCollectionConfigString="leaf_page_max=8KB"

Also the compact operation may be required to free space that was allocated in the middle of a file.

Michael.