New replica set member uses more disk space

[Tony Nelson]

I have a relatively small 3 node replica set running on 2.6.7 on Ubuntu.

The 3 existing nodes are running on Ubuntu 12.04LTS.  Each node has a 185G mongodb partition which is slowly running out of physical space.

tnelson@ihdb2:~$ df -h /var/lib/mongodb/

Filesystem      Size  Used Avail Use% Mounted on

/dev/sda4       185G  162G   14G  93% /var/lib/mongodb

I purchase three new machines to replace the existing nodes.  My plan is simply to add all three new machines to the replica set, then remove the old ones.

Tonight I added a new system running Mongo 2.6.7 on Ubuntu 14.04LTS and let Mongo sync all of the data.  I was very surprised to see the new filesystem 90% full already.

tnelson@ny-ihmongo1:~$ df -h /var/lib/mongodb/

Filesystem                    Size  Used Avail Use% Mounted on

/dev/mapper/ubuntu--vg-mongo  296G  250G   31G  90% /var/lib/mongodb

I have much more disk space available, so I can make this partition larger, but I don't understand why this new node is using so much more space. 

What should I look at to understand the difference in the amount of disk space allocated.

Thank you very much for any help.

Tony Nelson

Starpoint Solutions

[s.molinari]

This might answer your question indirectly.  

http://docs.mongodb.org/manual/faq/storage/#why-are-the-files-in-my-data-directory-larger-than-the-data-in-my-database

Things like configuration of preallocation might be different or depending on when you looked at the directory, the journal files had yet to be deleted from syncing. Just some wild guesses.:)

Scott

[Tony Nelson]

I read the first paragraph and thought, ok, it just preallocated til the disk was 90% full, but then I kept reading.  I found the set of commands that display statistics about your database.

Old server DB stats:

{

"db" : "instihire",

"collections" : 13,

"objects" : 7206099,

"avgObjSize" : 21528.276979819457,

"dataSize" : 155134895216,

"storageSize" : 158105507600,

"numExtents" : 190,

"indexes" : 20,

"indexSize" : 496705392,

"fileSize" : 163063005184,

"nsSizeMB" : 16,

"dataFileVersion" : {

"major" : 4,

"minor" : 5

},

"extentFreeList" : {

"num" : 0,

"totalSize" : 0

},

"ok" : 1

}

The same data from the new replica member:

{

"db" : "instihire",

"collections" : 13,

"objects" : 7206100,

"avgObjSize" : 34161.69733864365,

"dataSize" : 246172607192,

"storageSize" : 249297176880,

"numExtents" : 212,

"indexes" : 20,

"indexSize" : 352524592,

"fileSize" : 251066843136,

"nsSizeMB" : 16,

"dataFileVersion" : {

"major" : 4,

"minor" : 5

},

"extentFreeList" : {

"num" : 0,

"totalSize" : 0

},

"ok" : 1

}

Notice, the data size is 90G more than the node it copied it from, but the number of objects is only off by one.

Then I looked at one of my larger collections, first from the old node:

{

"ns" : "instihire.attachments.chunks",

"count" : 2130508,

"size" : 132681651188,

"avgObjSize" : 62277,

"storageSize" : 134714560880,

"numExtents" : 94,

"nindexes" : 2,

"lastExtentSize" : 2146426864,

"paddingFactor" : 1,

"systemFlags" : 1,

"userFlags" : 0,

"totalIndexSize" : 204759040,

"indexSizes" : {

"_id_" : 95059968,

"files_id_1_n_1" : 109699072

},

"ok" : 1

}

and from the new node

{

"ns" : "instihire.attachments.chunks",

"count" : 2130508,

"size" : 214121293872,

"avgObjSize" : 100502,

"storageSize" : 215360260592,

"numExtents" : 118,

"nindexes" : 2,

"lastExtentSize" : 2146426864,

"paddingFactor" : 1,

"systemFlags" : 1,

"userFlags" : 1,

"totalIndexSize" : 144012064,

"indexSizes" : {

"_id_" : 62219360,

"files_id_1_n_1" : 81792704

},

"ok" : 1

}

Notice right there is 100G storage size difference.  Am I reading this wrong?

Maybe I should just wait until tonight, shut down an old cluster member, and the new cluster member, and copy the data files?  Any idea what might have caused this size inflation?  I don't know if it's helps, but this collection is a GridFS created with the Mongo Java Driver. This just doesn't feel right atm.

Again, thanks in advance for any help.

Tony

[Tony Nelson]

Another thing to mention, just in case this may play a role.  Yesterday, before I added this new node, I upgraded my cluster members from 2.4 to 2.6.  Would that have anything to do with this issue?

[s.molinari]

Ahhhh! That is the answer, I think. In 2.6 they changed the default storage allocation strategy to powerof2sizes. http://docs.mongodb.org/manual/release-notes/2.6/#storage

You need to check the config of the old servers and see what the allocation strategy is and if you want the same sizes, change the strategy in your 2.6 nodes to the one your old servers had/ have. 

I'd suggest this might be a bad idea too, because the powersof2sizes strategy lets Mongo manage the storage space better. So, if you can live with the size of the files being bigger (the actual size of the data should stay the same...), then keep it like this. Mongo should work...let's say....more comfortably with the new storage allocation strategy.

Scott   

[Tony Nelson]

I just found out how to determine if usePowerOf2Sizes, it's the userFlags in db.collection.stats.  userFlags = 1, then usePowerOf2Sizes is enabled, 0 it is disabled.

Looking at the values I posted earlier, the old server definitely has usePowerOf2Sizes disabled, and the new one it is enabled.

I'm going to have to do some more reading, but I think I want it disabled, as these collections very rarely see deletes, and even though disk space is extremely inexpensive I don't see any reason to waste it.  From what I've read, this feature is designed to make more predictable sized buckets that can more easily be reused when deletes happen.

This feels like a bug to me.  I think if a collection has some property, then when it is replicated it should have the same property.  What do you think?

Thanks for your help.

Tony

[Tony Nelson]

Just to keep the thread updated, I found this Jira item which describes a fix to the Mongo Java driver.  This really describes my situation in a nutshell.

https://jira.mongodb.org/browse/SERVER-13331

I think my best plan at this point is to shut down the new node, remove the data files, copy the existing files from an old node and proceed from there.  I also need to investigate updating to the latest version of the Java driver.

Thanks again all.

[s.molinari]

Interesting find. But according to the bug report, it was fixed in 2.6.0, before it went gold. So theoretically, that shouldn't be your issue.

Maybe there is a regression (asking someone from Mongo, Asya, Will, Steni)?

Scott

[Tony Nelson]

That bug report was about the Java driver, and I do believe that is fixed.  

I think the bug is replication changing collection parameters, specifically changing the allocation style from the 2.4 default to the 2.6 default without consideration of what the existing collection is using.  Basically, a new replication will auto-upgrade every collection to the new behavior, weather it makes sense or not.

For me, I think I will copy the data files to the new server, update my java driver to the latest, and then enable powerOf2Size allocation for future additions to the collection.

[Asya Kamsky]

Tony,

Looks like you figured it out - note that another thing that can make
a big difference in overall space usage is if you don't explicitly
specify the size of the oplog - this is because 5% of available disk
space will be used. If your new machines had much bigger disks than
the old ones (when the old ones were first configured) this can cause
unnecessarily large oplog to be allocated.

I recommend always specifying the size of the oplog explicitly once
you calculate the desired size based on your write-rate and your HA
needs.

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/3235b1fd-a9eb-4b64-8746-9bc1a220b099%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Stephen Steneker]

On Wednesday, 21 January 2015 02:27:33 UTC+11, Tony Nelson wrote:

That bug report was about the Java driver, and I do believe that is fixed.  

I think the bug is replication changing collection parameters, specifically changing the allocation style from the 2.4 default to the 2.6 default without consideration of what the existing collection is using.  Basically, a new replication will auto-upgrade every collection to the new behavior, weather it makes sense or not.

Hi Tony,

The outcome in this case is perhaps unexpected since the per-server default allocation strategy has changed between 2.4 and 2.6, but replication should not be overriding any explicit collection settings. If you have explicitly set powerOf2Sizes to true or false for some collections this should be preserved on replication to a new server in the replica set.

It's definitely expected that you could have different different defaults per mongod server instance. For example, you may want to tune/test allocation strategies, configuration parameters, or storage engines (new to the 2.8.0 RCs) per replica set node.

For me, I think I will copy the data files to the new server, update my java driver to the latest, and then enable powerOf2Size allocation for future additions to the collection.

This is a reasonable approach.

Alternatively, there is a 2.6 configuration option to continue using the 2.4 allocation strategy ("best fit") as a default for new collections:
  http://docs.mongodb.org/manual/reference/parameters/#param.newCollectionsUsePowerOf2Sizes

If you set newCollectionsUsePowerOf2Sizes to false for a replica set node, you will still have to take some steps to reduce the disk space usage. Generally that would be re-syncing via initial sync or copying the data files from another node. The allocation default only affects new collections (which includes those created during initial sync).

Regards,

Stephen

[Tony Nelson]

I just wanted to post the final outcome.

Last night I removed all the data files from the new node, shut down and existing node and copied the existing data over.  This obviously preserved the allocation flag, and my amount of disk space stayed the same, as expected.

Thank you again to everyone who helped me understand what was going on, and come up with this resolution.