understanding disk usage

[Lucas, Sascha]

Hi,

I use mongodb for syslog-messages. I have 2 collections (not capped): test.messages and test.messages.old. Messages older then X days are moved (save+remove) from messages into messages.old. Messages older then Y days are removed from messages.old (X<Y). [1] tells me: "Recovering Deleted Space: MongoDB maintains lists of deleted blocks within the datafiles when objects or collections are deleted. This space is reused by MongoDB but never freed to the operating system."

[1] http://www.mongodb.org/display/DOCS/Excessive+Disk+Space

While the above would be enough for me (deleted space is reused by mongodb), I run out of disk space. Can someone tell me why?

$ mongod --version
db version v2.0.6, pdfile version 4.5
#############################################################################
> db.messages.old.totalSize()
822605984
> db.messages.old.storageSize()
796524544
> db.messages.old.validate()
{
"ns" : "test.messages.old",
"firstExtent" : "5:249da000 ns:test.messages.old",
"lastExtent" : "5:3ae0d000 ns:test.messages.old",
"extentCount" : 20,
"datasize" : 157111764,
"nrecords" : 453133,
"lastExtentSize" : 139808768,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "5:249da000",
"xnext" : "4:a074000",
"xprev" : "null",
"nsdiag" : "test.messages.old",
"size" : 32768,
"firstRecord" : "null",
"lastRecord" : "null"
},
"deletedCount" : 1533630,
"deletedSize" : 632159132,
"nIndexes" : 1,
"keysPerIndex" : {
"test.messages.old.$_id_" : 453133
},
"valid" : true,
"errors" : [ ],
"warning" : "Some checks omitted for speed. use {full:true} option to do more thorough scan.",
"ok" : 1
}

#############################################################################
> db.messages.totalSize()
5614160304
> db.messages.storageSize()
5025185792
> db.messages.validate()
{
"ns" : "test.messages",
"firstExtent" : "0:3000 ns:test.messages",
"lastExtent" : "8:51f4c000 ns:test.messages",
"extentCount" : 33,
"datasize" : 733750044,
"nrecords" : 2113172,
"lastExtentSize" : 721412096,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:3000",
"xnext" : "0:12000",
"xprev" : "null",
"nsdiag" : "test.messages",
"size" : 20480,
"firstRecord" : "null",
"lastRecord" : "null"
},
"deletedCount" : 11950896,
"deletedSize" : 4257619188,
"nIndexes" : 6,
"keysPerIndex" : {
"test.messages.$_id_" : 2113172,
"test.messages.$HOST_1" : 2113172,
"test.messages.$DATE_-1" : 2113172,
"test.messages.$FACILITY_1" : 2113172,
"test.messages.$PRIORITY_1" : 2113172,
"test.messages.$PROGRAM_1" : 2113172
},
"valid" : true,
"errors" : [ ],
"warning" : "Some checks omitted for speed. use {full:true} option to do more thorough scan.",
"ok" : 1
}

I sum up (totalSize) about 6GB used on a 20GB data partition. What's this deletedSize in validate()? Should deletedSize be reused like said in [1]? Or should I run some periodically repair?

Thanks, Sascha.

Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp
Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher
Sitz der Gesellschaft: Halle/Saale
Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
UST-ID-Nr. DE 158253683
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

[Jason Rassi]

The validate field "deletedSize" does in fact refer to the space consumed by data deleted from the collection, which will be reclaimed by newly inserted documents (it doesn't seem necessary for you to run a repair).

As far as where the extra disk space is going: MongoDB uses disk space for journaling, as well (unless disabled explicitly, for 2.0.6).  You can check how much space the journal consumes by seeing the size of the "journal" subdirectory in the DB path.

Assuming you're on a Unix system (and the DB path in your configuration file is /data/db), can you send the output of running the following at a command prompt:

df -h /data/db

and

du -hsc /data/db/*

This will show the breakdown of disk space that mongod is using.

[Lucas, Sascha]

Hi Jason,

From: Jason Rassi
Date: Tue, 18. Sep 2012 23:47

> The validate field "deletedSize" does in fact refer to the space consumed by data deleted from the collection, which will be reclaimed by newly inserted documents (it doesn't seem necessary for you to run a repair).

Thanks for confirmation (deletedSize will be reclaimed). Let me summarize that I have:

db.messages.old.validate(){..., "deletedSize" : 632159132, ...}
db.messages.validate(){..., "deletedSize" : 4257619188, ...}

This is ~0.6GB for messages.old and ~4GB for messages of deleted space, which can be reused.

> As far as where the extra disk space is going: MongoDB uses disk space for journaling, as well (unless disabled explicitly, for 2.0.6).

$ df -h /data/
Filesystem Size Used Avail Use% Mounted on
/dev/vdb 16G 14G 1.4G 91% /data

$ du -hsc /data/mongodb/*
3.1G /data/mongodb/journal
4.0K /data/mongodb/mongod.lock
65M /data/mongodb/test.0
129M /data/mongodb/test.1
257M /data/mongodb/test.2
513M /data/mongodb/test.3
1.1G /data/mongodb/test.4
2.1G /data/mongodb/test.5
2.1G /data/mongodb/test.6
2.1G /data/mongodb/test.7
2.1G /data/mongodb/test.8
17M /data/mongodb/test.ns
13G total

$ du -hsc /data/
14G /data/
14G total

So I have a 16GB data partition with 1.4GB free, 3.1GB journal, 1GB non-mongodb-data (static) and:

> db.messages.old.totalSize()
822605984

> db.messages.totalSize()
5614160304

about 6GB used by data/index. This is: 16-1.4-3.1-1-6=4.5GB, which is approx. the sum of deletedSize (of by 100MB).

> On Monday, September 17, 2012 3:28:10 AM UTC-4, sascha wrote:
>> I use mongodb for syslog-messages. I have 2 collections (not capped): test.messages and test.messages.old. Messages older then X days are moved (save+remove) from messages into messages.old. Messages older then Y days are removed from messages.old (X<Y).

@midnight I run the following mongo-shell code (via cron, generated by bash, piped into mongo, showed DATE is just a example):

db.messages.old.remove({DATE: {$lt: "1347524223"}})
for (var c = db.messages.find({DATE: {$lt: "1347610623"}}); c.hasNext();) {
db.messages.old.save(c.next());
}
db.messages.remove({DATE: {$lt: "1347610623"}})

So in theory mongo shouldn't run out of disk space. It seems that I need to further investigate what's happening at my cronjob.

Following question arises: Is the free space of "deletedSize" shared between collections within the same db? If not, then "deletedSize" of messages.old indeed can be to small.

THX Sascha.

[Jason Rassi]

A collection's "deletedSize" is calculated by totaling the amount of space occupied by its free list.  These are maintained independently, per collection.  Thus, a newly-inserted document from one collection won't occupy the space used by a document deleted from another collection.

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

[tingkun --]

why don't show my post? sorry to sascha
i try to show my problem

mongo 2.2,
when the query key contain xxx.xxx 

var uo = {"bag":{"bm1":10,"bm2":1,"bm3":1,"bm4":1},"money":50,_id:1}

> db.s1_users.save(uo)

> db.s1_users.findAndModify({"query":{"money":{"$gte":50},"bag.bm1":{"$gte":10},"_id":1},"update":{"$inc":{"money":-50,"bag.bm1":-10}},"new":true,"upsert":true})

Fri Sep 21 13:22:55 uncaught exception: findAndModifyFailed failed: {

        "errmsg" : "exception: assertion src/mongo/db/commands/find_and_modify.cpp:184",

        "code" : 0,

        "ok" : 0

}

> uo

{

        "bag" : {

                "bm1" : 10,

                "bm2" : 1,

                "bm3" : 1,

                "bm4" : 1

        },

        "money" : 50,

        "_id" : 1

}

>  db.s1_users.findOne({_id:1});

{

        "_id" : 1,

        "bag" : {

                "bm1" : 0,

                "bm2" : 1,

                "bm3" : 1,

                "bm4" : 1

        },

        "money" : 0

}

在 2012年9月17日星期一UTC+8下午3时28分10秒，sascha写道：

[Jason Rassi]

Did you have trouble creating a new thread on the mailing list?

The findAndModify() problem you saw is a known issue that has been fixed in the development branch (I verified this with the code snippet you included).  See <https://jira.mongodb.org/browse/SERVER-6909>.  The fix is slated to be included in the 2.2.1 release.

[Lucas, Sascha]

Hi Jason,

From: Jason Rassi
Date: Fri, 21. Sep 2012 06:33
> A collection's "deletedSize" is calculated by totaling the amount of space occupied by its free list.  These are maintained independently, per collection.  Thus, a newly-inserted document from one collection won't occupy the space used by a document deleted from another collection.

Thanks for clarification. As I thought I understood and solved my problem with doubling my disk space (because deleted space is not shared among collections), here it is again:

Mon Oct 8 12:24:01 [FileAllocator] allocating new datafile /data/mongodb/test.11, filling with zeroes...
Mon Oct 8 12:24:12 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Mon Oct 8 12:24:13 [FileAllocator] error failed to allocate new file: /data/mongodb/test.11 size: 2146435072 errno:28 No space left on
device
...
Mon Oct 8 12:42:14 [initandlisten] db version v2.0.6, pdfile version 4.5

> show dbs
local (empty)
test 13.947265625GB

> show collections
messages
messages.old
mojology.mr.hosts
mojology.mr.programs
mojology.mr.time
system.indexes

I would swear that messages is the only collection, which is written to (messages.old just at midnight via cron, mojology.mr* by hand). So if I have ~17e6 records deleted (~7GB), why was there a new datafile allocated? Should I consider padding?

> db.messages.validate()
{
"ns" : "test.messages",
"firstExtent" : "0:3000 ns:test.messages",

"lastExtent" : "a:2000 ns:test.messages",
"extentCount" : 36,
"datasize" : 817401456,
"nrecords" : 2359189,
"lastExtentSize" : 1246609408,

"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:3000",
"xnext" : "0:12000",
"xprev" : "null",
"nsdiag" : "test.messages",
"size" : 20480,
"firstRecord" : "null",
"lastRecord" : "null"
},

"deletedCount" : 17053034,
"deletedSize" : 7321177984,
"nIndexes" : 6,
"keysPerIndex" : {
"test.messages.$_id_" : 2359189,
"test.messages.$HOST_1" : 2359189,
"test.messages.$DATE_-1" : 2359189,
"test.messages.$FACILITY_1" : 2359189,
"test.messages.$PRIORITY_1" : 2359189,
"test.messages.$PROGRAM_1" : 2359189

},
"valid" : true,
"errors" : [ ],
"warning" : "Some checks omitted for speed. use {full:true} option to do more thorough scan.",
"ok" : 1
}

> db.messages.stats()
{
"ns" : "test.messages",
"count" : 2359189,
"size" : 817401456,
"avgObjSize" : 346.47561344173783,
"storageSize" : 8176332800,
"numExtents" : 36,
"nindexes" : 6,
"lastExtentSize" : 1246609408,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 675353952,
"indexSizes" : {
"_id_" : 76568240,
"HOST_1" : 125795936,
"DATE_-1" : 131445552,
"FACILITY_1" : 106835792,
"PRIORITY_1" : 104808144,
"PROGRAM_1" : 129900288
},
"ok" : 1
}

I would be grateful if someone can explain me what's wrong or how to investigate further. Sascha.

[Jason Rassi]

It looks like you've come across a known issue in the server:
https://jira.mongodb.org/browse/SERVER-2958 .  Sorry I didn't catch
this from your first two posts.

As a result, messages and messages.old are both likely in a state
where they're seeing nearly no space reuse at all.  You need to
compact both of them to remedy this.  You can do this either via
individual compactions on each, or with a full database repair.  For
either method, you'll need to take the server offline, and you'll need
temporary extra disk space for the operations to work with.  I'd guess
that the individual compactions would need at most 2GB total, while
the full repair would need about 7GB total (but the repair will
eventually result in the deletion of nearly all unused space).  See
http://docs.mongodb.org/manual/reference/command/compact/ (reference
for compaction) and http://docs.mongodb.org/manual/reference/commands/#repairDatabase
(reference for repair).

To prevent this from happening in the future, you have a few options.
 Any of the following solutions should work:

1. Upgrade to MongoDB 2.2 and enable the "powerOf2Sizes" collection
flag on both collections.  You can do this for e.g. the messages
collection by running the following at the mongo shell:
db.runCommand({collMod:"messages", usePowerOf2Sizes:"true"}).
2. Upgrade to MongoDB 2.2, convert your "DATE" field to store
Javascript Date object values, and set a TTL index on "DATE" for both
collections.  This will set the "powerOf2Sizes" flag on the
collections for you, and will also keep you from needing to expire
your records via your own cron script.  See http://docs.mongodb.org/manual/tutorial/expire-data/
for details.
3. If you aren't able to upgrade MongoDB, you could pad your documents
to a fixed size (perhaps to the length of the longest syslog message
you generally see, ignoring outliers) before inserting them.
4. Alternatively if you aren't able to upgrade MongoDB, you could run
the compactions above on a regular basis.  You'll have to determine
yourself experimentally how often you need to do this, likely by
monitoring "deletedSize" over time and factoring in how much extra
disk usage you can tolerate.

Let me know how this goes for you.

[Kenny Gorman]

Also, if you are using a replica set, then you can delete/resync a slave and then promote it to master after it's up to sync. This (in my opinion) is the best way to 'compact' with a minimal operational 'blip'. Have done this in the past on an automated/known schedule to keep things tidy.

-kg

[Lucas, Sascha]

Hi Jason,

From: Jason Rassi
Date: Thu, 11. Oct 2012 02:49

> It looks like you've come across a known issue in the server:
> https://jira.mongodb.org/browse/SERVER-2958 .  Sorry I didn't catch
> this from your first two posts.

Thank you very much for your help. Upgrading to mongodb 2.2 is definitely the way I'll go, because TTL collections is a must for syslog.

> 1. Upgrade to MongoDB 2.2 and enable the "powerOf2Sizes" collection

This will be my 1st step.

> 2. Upgrade to MongoDB 2.2, convert your "DATE" field to store
> Javascript Date object values, and set a TTL index on "DATE" for both
> collections.

And after syslog-ng is able to store other data types then string, this will be the second.

I'll report (in some days) if this helps (but I feel it in my bones, that this was my problem).

Thanks, Sascha.