> Hi,

> I use mongodb for syslog-messages. I have 2 collections (not capped):
> test.messages and test.messages.old. Messages older then X days are moved
> (save+remove) from messages into messages.old. Messages older then Y days
> are removed from messages.old (X<Y). [1] tells me: "Recovering Deleted
> Space: MongoDB maintains lists of deleted blocks within the datafiles when
> objects or collections are deleted. This space is reused by MongoDB but
> never freed to the operating system."

> [1] http://www.mongodb.org/display/DOCS/Excessive+Disk+Space

> While the above would be enough for me (deleted space is reused by
> mongodb), I run out of disk space. Can someone tell me why?

> $ mongod --version
> db version v2.0.6, pdfile version 4.5
> ########################################################################### ##

> > db.messages.old.totalSize()
> 822605984
> > db.messages.old.storageSize()
> 796524544
> > db.messages.old.validate()
> {
>         "ns" : "test.messages.old",
>         "firstExtent" : "5:249da000 ns:test.messages.old",
>         "lastExtent" : "5:3ae0d000 ns:test.messages.old",
>         "extentCount" : 20,
>         "datasize" : 157111764,
>         "nrecords" : 453133,
>         "lastExtentSize" : 139808768,
>         "padding" : 1,
>         "firstExtentDetails" : {
>                 "loc" : "5:249da000",
>                 "xnext" : "4:a074000",
>                 "xprev" : "null",
>                 "nsdiag" : "test.messages.old",
>                 "size" : 32768,
>                 "firstRecord" : "null",
>                 "lastRecord" : "null"
>         },
>         "deletedCount" : 1533630,
>         "deletedSize" : 632159132,
>         "nIndexes" : 1,
>         "keysPerIndex" : {
>                 "test.messages.old.$_id_" : 453133
>         },
>         "valid" : true,
>         "errors" : [ ],
>         "warning" : "Some checks omitted for speed. use {full:true} option
> to do more thorough scan.",
>         "ok" : 1
> }

> ########################################################################### ##

> > db.messages.totalSize()
> 5614160304
> > db.messages.storageSize()
> 5025185792
> > db.messages.validate()
> {
>         "ns" : "test.messages",
>         "firstExtent" : "0:3000 ns:test.messages",
>         "lastExtent" : "8:51f4c000 ns:test.messages",
>         "extentCount" : 33,
>         "datasize" : 733750044,
>         "nrecords" : 2113172,
>         "lastExtentSize" : 721412096,
>         "padding" : 1,
>         "firstExtentDetails" : {
>                 "loc" : "0:3000",
>                 "xnext" : "0:12000",
>                 "xprev" : "null",
>                 "nsdiag" : "test.messages",
>                 "size" : 20480,
>                 "firstRecord" : "null",
>                 "lastRecord" : "null"
>         },
>         "deletedCount" : 11950896,
>         "deletedSize" : 4257619188,
>         "nIndexes" : 6,
>         "keysPerIndex" : {
>                 "test.messages.$_id_" : 2113172,
>                 "test.messages.$HOST_1" : 2113172,
>                 "test.messages.$DATE_-1" : 2113172,
>                 "test.messages.$FACILITY_1" : 2113172,
>                 "test.messages.$PRIORITY_1" : 2113172,
>                 "test.messages.$PROGRAM_1" : 2113172
>         },
>         "valid" : true,
>         "errors" : [ ],
>         "warning" : "Some checks omitted for speed. use {full:true} option
> to do more thorough scan.",
>         "ok" : 1
> }

> I sum up (totalSize) about 6GB used on a 20GB data partition. What's this
> deletedSize in validate()? Should deletedSize be reused like said in [1]?
> Or should I run some periodically repair?

> Thanks, Sascha.

> Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp
> Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher
> Sitz der Gesellschaft: Halle/Saale
> Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
> UST-ID-Nr. DE 158253683
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
> Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail
> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
> Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet.
> Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte
> geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die
> Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden
> Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese
> E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit
> gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

> From: Jason Rassi
> Date: Tue, 18. Sep 2012 23:47

> > The validate field "deletedSize" does in fact refer to the space
> consumed by data deleted from the collection, which will be reclaimed by
> newly inserted documents (it doesn't seem necessary for you to run a
> repair).

> Thanks for confirmation (deletedSize will be reclaimed). Let me summarize
> that I have:

> db.messages.old.validate(){..., "deletedSize" : 632159132, ...}
> db.messages.validate(){..., "deletedSize" : 4257619188, ...}

> This is ~0.6GB for messages.old and ~4GB for messages of deleted space,
> which can be reused.

> > As far as where the extra disk space is going: MongoDB uses disk space
> for journaling, as well (unless disabled explicitly, for 2.0.6).

> $ df -h /data/
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vdb         16G   14G  1.4G  91% /data

> $ du -hsc /data/mongodb/*
> 3.1G    /data/mongodb/journal
> 4.0K    /data/mongodb/mongod.lock
> 65M     /data/mongodb/test.0
> 129M    /data/mongodb/test.1
> 257M    /data/mongodb/test.2
> 513M    /data/mongodb/test.3
> 1.1G    /data/mongodb/test.4
> 2.1G    /data/mongodb/test.5
> 2.1G    /data/mongodb/test.6
> 2.1G    /data/mongodb/test.7
> 2.1G    /data/mongodb/test.8
> 17M     /data/mongodb/test.ns
> 13G     total

> $ du -hsc /data/
> 14G     /data/
> 14G     total

> So I have a 16GB data partition with 1.4GB free, 3.1GB journal, 1GB
> non-mongodb-data (static) and:

> > db.messages.old.totalSize()
> 822605984

> > db.messages.totalSize()
> 5614160304

> about 6GB used by data/index. This is: 16-1.4-3.1-1-6=4.5GB, which is
> approx. the sum of deletedSize (of by 100MB).

> > On Monday, September 17, 2012 3:28:10 AM UTC-4, sascha wrote:
> >> I use mongodb for syslog-messages. I have 2 collections (not capped):
> test.messages and test.messages.old. Messages older then X days are moved
> (save+remove) from messages into messages.old. Messages older then Y days
> are removed from messages.old (X<Y).

> @midnight I run the following mongo-shell code (via cron, generated by
> bash, piped into mongo, showed DATE is just a example):

> db.messages.old.remove({DATE: {$lt: "1347524223"}})
> for (var c = db.messages.find({DATE: {$lt: "1347610623"}}); c.hasNext();) {
>         db.messages.old.save(c.next());
> }
> db.messages.remove({DATE: {$lt: "1347610623"}})

> So in theory mongo shouldn't run out of disk space. It seems that I need
> to further investigate what's happening at my cronjob.

> Following question arises: Is the free space of "deletedSize" shared
> between collections within the same db? If not, then "deletedSize" of
> messages.old indeed can be to small.

> THX Sascha.

> Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp
> Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher
> Sitz der Gesellschaft: Halle/Saale
> Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
> UST-ID-Nr. DE 158253683
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
> Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail
> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
> Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet.
> Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte
> geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die
> Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden
> Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese
> E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit
> gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com<javascript:;>
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com <javascript:;>
> See also the IRC channel -- freenode.net#mongodb

> Hi,

> I use mongodb for syslog-messages. I have 2 collections (not capped):
> test.messages and test.messages.old. Messages older then X days are moved
> (save+remove) from messages into messages.old. Messages older then Y days
> are removed from messages.old (X<Y). [1] tells me: "Recovering Deleted
> Space: MongoDB maintains lists of deleted blocks within the datafiles when
> objects or collections are deleted. This space is reused by MongoDB but
> never freed to the operating system."

> [1] http://www.mongodb.org/display/DOCS/Excessive+Disk+Space

> While the above would be enough for me (deleted space is reused by
> mongodb), I run out of disk space. Can someone tell me why?

> $ mongod --version
> db version v2.0.6, pdfile version 4.5
> ########################################################################### ##

> > db.messages.old.totalSize()
> 822605984
> > db.messages.old.storageSize()
> 796524544
> > db.messages.old.validate()
> {
>         "ns" : "test.messages.old",
>         "firstExtent" : "5:249da000 ns:test.messages.old",
>         "lastExtent" : "5:3ae0d000 ns:test.messages.old",
>         "extentCount" : 20,
>         "datasize" : 157111764,
>         "nrecords" : 453133,
>         "lastExtentSize" : 139808768,
>         "padding" : 1,
>         "firstExtentDetails" : {
>                 "loc" : "5:249da000",
>                 "xnext" : "4:a074000",
>                 "xprev" : "null",
>                 "nsdiag" : "test.messages.old",
>                 "size" : 32768,
>                 "firstRecord" : "null",
>                 "lastRecord" : "null"
>         },
>         "deletedCount" : 1533630,
>         "deletedSize" : 632159132,
>         "nIndexes" : 1,
>         "keysPerIndex" : {
>                 "test.messages.old.$_id_" : 453133
>         },
>         "valid" : true,
>         "errors" : [ ],
>         "warning" : "Some checks omitted for speed. use {full:true} option
> to do more thorough scan.",
>         "ok" : 1
> }

> ########################################################################### ##

> > db.messages.totalSize()
> 5614160304
> > db.messages.storageSize()
> 5025185792
> > db.messages.validate()
> {
>         "ns" : "test.messages",
>         "firstExtent" : "0:3000 ns:test.messages",
>         "lastExtent" : "8:51f4c000 ns:test.messages",
>         "extentCount" : 33,
>         "datasize" : 733750044,
>         "nrecords" : 2113172,
>         "lastExtentSize" : 721412096,
>         "padding" : 1,
>         "firstExtentDetails" : {
>                 "loc" : "0:3000",
>                 "xnext" : "0:12000",
>                 "xprev" : "null",
>                 "nsdiag" : "test.messages",
>                 "size" : 20480,
>                 "firstRecord" : "null",
>                 "lastRecord" : "null"
>         },
>         "deletedCount" : 11950896,
>         "deletedSize" : 4257619188,
>         "nIndexes" : 6,
>         "keysPerIndex" : {
>                 "test.messages.$_id_" : 2113172,
>                 "test.messages.$HOST_1" : 2113172,
>                 "test.messages.$DATE_-1" : 2113172,
>                 "test.messages.$FACILITY_1" : 2113172,
>                 "test.messages.$PRIORITY_1" : 2113172,
>                 "test.messages.$PROGRAM_1" : 2113172
>         },
>         "valid" : true,
>         "errors" : [ ],
>         "warning" : "Some checks omitted for speed. use {full:true} option
> to do more thorough scan.",
>         "ok" : 1
> }

> I sum up (totalSize) about 6GB used on a 20GB data partition. What's this
> deletedSize in validate()? Should deletedSize be reused like said in [1]?
> Or should I run some periodically repair?

> Thanks, Sascha.

> Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp
> Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher
> Sitz der Gesellschaft: Halle/Saale
> Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
> UST-ID-Nr. DE 158253683
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
> Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail
> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
> Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet.
> Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte
> geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die
> Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden
> Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese
> E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit
> gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

> why don't show my post? sorry to sascha
> i try to show my problem

> mongo 2.2,
> when the query key contain xxx.xxx

> var uo = {"bag":{"bm1":10,"bm2":1,"bm3":1,"bm4":1},"money":50,_id:1}
> > db.s1_users.save(uo)

> db.s1_users.findAndModify({"query":{"money":{"$gte":50},"bag.bm1":{"$gte":1 0},"_id":1},"update":{"$inc":{"money":-50,"bag.bm1":-10}},"new":true,"upser t":true})
> Fri Sep 21 13:22:55 uncaught exception: findAndModifyFailed failed: {
>         "errmsg" : "exception: assertion
> src/mongo/db/commands/find_and_modify.cpp:184",
>         "code" : 0,
>         "ok" : 0
> }
> > uo
> {
>         "bag" : {
>                 "bm1" : 10,
>                 "bm2" : 1,
>                 "bm3" : 1,
>                 "bm4" : 1
>         },
>         "money" : 50,
>         "_id" : 1
> }
> >  db.s1_users.findOne({_id:1});
> {
>         "_id" : 1,
>         "bag" : {
>                 "bm1" : 0,
>                 "bm2" : 1,
>                 "bm3" : 1,
>                 "bm4" : 1
>         },
>         "money" : 0
> }

> 在 2012年9月17日星期一UTC+8下午3时28分10秒，sascha写道：

>> Hi,

>> I use mongodb for syslog-messages. I have 2 collections (not capped):
>> test.messages and test.messages.old. Messages older then X days are moved
>> (save+remove) from messages into messages.old. Messages older then Y days
>> are removed from messages.old (X<Y). [1] tells me: "Recovering Deleted
>> Space: MongoDB maintains lists of deleted blocks within the datafiles when
>> objects or collections are deleted. This space is reused by MongoDB but
>> never freed to the operating system."

>> [1] http://www.mongodb.org/display/DOCS/Excessive+Disk+Space

>> While the above would be enough for me (deleted space is reused by
>> mongodb), I run out of disk space. Can someone tell me why?

>> $ mongod --version
>> db version v2.0.6, pdfile version 4.5
>> ########################################################################### ##

>> > db.messages.old.totalSize()
>> 822605984
>> > db.messages.old.storageSize()
>> 796524544
>> > db.messages.old.validate()
>> {
>>         "ns" : "test.messages.old",
>>         "firstExtent" : "5:249da000 ns:test.messages.old",
>>         "lastExtent" : "5:3ae0d000 ns:test.messages.old",
>>         "extentCount" : 20,
>>         "datasize" : 157111764,
>>         "nrecords" : 453133,
>>         "lastExtentSize" : 139808768,
>>         "padding" : 1,
>>         "firstExtentDetails" : {
>>                 "loc" : "5:249da000",
>>                 "xnext" : "4:a074000",
>>                 "xprev" : "null",
>>                 "nsdiag" : "test.messages.old",
>>                 "size" : 32768,
>>                 "firstRecord" : "null",
>>                 "lastRecord" : "null"
>>         },
>>         "deletedCount" : 1533630,
>>         "deletedSize" : 632159132,
>>         "nIndexes" : 1,
>>         "keysPerIndex" : {
>>                 "test.messages.old.$_id_" : 453133
>>         },
>>         "valid" : true,
>>         "errors" : [ ],
>>         "warning" : "Some checks omitted for speed. use {full:true}
>> option to do more thorough scan.",
>>         "ok" : 1
>> }

>> ########################################################################### ##

>> > db.messages.totalSize()
>> 5614160304
>> > db.messages.storageSize()
>> 5025185792
>> > db.messages.validate()
>> {
>>         "ns" : "test.messages",
>>         "firstExtent" : "0:3000 ns:test.messages",
>>         "lastExtent" : "8:51f4c000 ns:test.messages",
>>         "extentCount" : 33,
>>         "datasize" : 733750044,
>>         "nrecords" : 2113172,
>>         "lastExtentSize" : 721412096,
>>         "padding" : 1,
>>         "firstExtentDetails" : {
>>                 "loc" : "0:3000",
>>                 "xnext" : "0:12000",
>>                 "xprev" : "null",
>>                 "nsdiag" : "test.messages",
>>                 "size" : 20480,
>>                 "firstRecord" : "null",
>>                 "lastRecord" : "null"
>>         },
>>         "deletedCount" : 11950896,
>>         "deletedSize" : 4257619188,
>>         "nIndexes" : 6,
>>         "keysPerIndex" : {
>>                 "test.messages.$_id_" : 2113172,
>>                 "test.messages.$HOST_1" : 2113172,
>>                 "test.messages.$DATE_-1" : 2113172,
>>                 "test.messages.$FACILITY_1" : 2113172,
>>                 "test.messages.$PRIORITY_1" : 2113172,
>>                 "test.messages.$PROGRAM_1" : 2113172
>>         },
>>         "valid" : true,
>>         "errors" : [ ],
>>         "warning" : "Some checks omitted for speed. use {full:true}
>> option to do more thorough scan.",
>>         "ok" : 1
>> }

>> I sum up (totalSize) about 6GB used on a 20GB data partition. What's this
>> deletedSize in validate()? Should deletedSize be reused like said in [1]?
>> Or should I run some periodically repair?

>> Thanks, Sascha.

>> Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp
>> Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher
>> Sitz der Gesellschaft: Halle/Saale
>> Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
>> UST-ID-Nr. DE 158253683
>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
>> Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail
>> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
>> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
>> Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet.
>> Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte
>> geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die
>> Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden
>> Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese
>> E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit
>> gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

> From: Jason Rassi
> Date: Fri, 21. Sep 2012 06:33

> > A collection's "deletedSize" is calculated by totaling the amount of space occupied by its free list.  These are maintained independently, per collection.  Thus, a newly-inserted document from one collection won't occupy the space used by a document deleted from another collection.

> Thanks for clarification. As I thought I understood and solved my problem with doubling my disk space (because deleted space is not shared among collections), here it is again:

> Mon Oct  8 12:24:01 [FileAllocator] allocating new datafile /data/mongodb/test.11, filling with zeroes...
> Mon Oct  8 12:24:12 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
> Mon Oct  8 12:24:13 [FileAllocator] error failed to allocate new file: /data/mongodb/test.11 size: 2146435072 errno:28 No space left on
>  device
> ...
> Mon Oct  8 12:42:14 [initandlisten] db version v2.0.6, pdfile version 4.5

> > show dbs

> local   (empty)
> test    13.947265625GB

> > show collections

> messages
> messages.old
> mojology.mr.hosts
> mojology.mr.programs
> mojology.mr.time
> system.indexes

> I would swear that messages is the only collection, which is written to (messages.old just at midnight via cron, mojology.mr* by hand). So if I have ~17e6 records deleted (~7GB), why was there a new datafile allocated? Should I consider padding?

> > db.messages.validate()

> {
>         "ns" : "test.messages",
>         "firstExtent" : "0:3000 ns:test.messages",
>         "lastExtent" : "a:2000 ns:test.messages",
>         "extentCount" : 36,
>         "datasize" : 817401456,
>         "nrecords" : 2359189,
>         "lastExtentSize" : 1246609408,
>         "padding" : 1,
>         "firstExtentDetails" : {
>                 "loc" : "0:3000",
>                 "xnext" : "0:12000",
>                 "xprev" : "null",
>                 "nsdiag" : "test.messages",
>                 "size" : 20480,
>                 "firstRecord" : "null",
>                 "lastRecord" : "null"
>         },
>         "deletedCount" : 17053034,
>         "deletedSize" : 7321177984,
>         "nIndexes" : 6,
>         "keysPerIndex" : {
>                 "test.messages.$_id_" : 2359189,
>                 "test.messages.$HOST_1" : 2359189,
>                 "test.messages.$DATE_-1" : 2359189,
>                 "test.messages.$FACILITY_1" : 2359189,
>                 "test.messages.$PRIORITY_1" : 2359189,
>                 "test.messages.$PROGRAM_1" : 2359189
>         },
>         "valid" : true,
>         "errors" : [ ],
>         "warning" : "Some checks omitted for speed. use {full:true} option to do more thorough scan.",
>         "ok" : 1

> }
> > db.messages.stats()

> {
>         "ns" : "test.messages",
>         "count" : 2359189,
>         "size" : 817401456,
>         "avgObjSize" : 346.47561344173783,
>         "storageSize" : 8176332800,
>         "numExtents" : 36,
>         "nindexes" : 6,
>         "lastExtentSize" : 1246609408,
>         "paddingFactor" : 1,
>         "flags" : 1,
>         "totalIndexSize" : 675353952,
>         "indexSizes" : {
>                 "_id_" : 76568240,
>                 "HOST_1" : 125795936,
>                 "DATE_-1" : 131445552,
>                 "FACILITY_1" : 106835792,
>                 "PRIORITY_1" : 104808144,
>                 "PROGRAM_1" : 129900288
>         },
>         "ok" : 1

> }

> I would be grateful if someone can explain me what's wrong or how to investigate further. Sascha.

> Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp
> Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher
> Sitz der Gesellschaft: Halle/Saale
> Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
> UST-ID-Nr. DE 158253683
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

> As a result, messages and messages.old are both likely in a state
> where they're seeing nearly no space reuse at all.  You need to
> compact both of them to remedy this.  You can do this either via
> individual compactions on each, or with a full database repair.  For
> either method, you'll need to take the server offline, and you'll need
> temporary extra disk space for the operations to work with.  I'd guess
> that the individual compactions would need at most 2GB total, while
> the full repair would need about 7GB total (but the repair will
> eventually result in the deletion of nearly all unused space).  See
> http://docs.mongodb.org/manual/reference/command/compact/ (reference
> for compaction) and http://docs.mongodb.org/manual/reference/commands/#repairDatabase
> (reference for repair).

> To prevent this from happening in the future, you have a few options.
>  Any of the following solutions should work:

> 1. Upgrade to MongoDB 2.2 and enable the "powerOf2Sizes" collection
> flag on both collections.  You can do this for e.g. the messages
> collection by running the following at the mongo shell:
> db.runCommand({collMod:"messages", usePowerOf2Sizes:"true"}).
> 2. Upgrade to MongoDB 2.2, convert your "DATE" field to store
> Javascript Date object values, and set a TTL index on "DATE" for both
> collections.  This will set the "powerOf2Sizes" flag on the
> collections for you, and will also keep you from needing to expire
> your records via your own cron script.  See http://docs.mongodb.org/manual/tutorial/expire-data/
> for details.
> 3. If you aren't able to upgrade MongoDB, you could pad your documents
> to a fixed size (perhaps to the length of the longest syslog message
> you generally see, ignoring outliers) before inserting them.
> 4. Alternatively if you aren't able to upgrade MongoDB, you could run
> the compactions above on a regular basis.  You'll have to determine
> yourself experimentally how often you need to do this, likely by
> monitoring "deletedSize" over time and factoring in how much extra
> disk usage you can tolerate.

> Let me know how this goes for you.

> On Oct 8, 7:36 am, "Lucas, Sascha" <Sascha.Lu...@gisa.de> wrote:
>> Hi Jason,

>> From: Jason Rassi
>> Date: Fri, 21. Sep 2012 06:33

>>> A collection's "deletedSize" is calculated by totaling the amount of space occupied by its free list.  These are maintained independently, per collection.  Thus, a newly-inserted document from one collection won't occupy the space used by a document deleted from another collection.

>> Thanks for clarification. As I thought I understood and solved my problem with doubling my disk space (because deleted space is not shared among collections), here it is again:

>> Mon Oct  8 12:24:01 [FileAllocator] allocating new datafile /data/mongodb/test.11, filling with zeroes...
>> Mon Oct  8 12:24:12 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
>> Mon Oct  8 12:24:13 [FileAllocator] error failed to allocate new file: /data/mongodb/test.11 size: 2146435072 errno:28 No space left on
>>  device
>> ...
>> Mon Oct  8 12:42:14 [initandlisten] db version v2.0.6, pdfile version 4.5

>>> show dbs

>> local   (empty)
>> test    13.947265625GB

>>> show collections

>> messages
>> messages.old
>> mojology.mr.hosts
>> mojology.mr.programs
>> mojology.mr.time
>> system.indexes

>> I would swear that messages is the only collection, which is written to (messages.old just at midnight via cron, mojology.mr* by hand). So if I have ~17e6 records deleted (~7GB), why was there a new datafile allocated? Should I consider padding?

>>> db.messages.validate()

>> {
>>         "ns" : "test.messages",
>>         "firstExtent" : "0:3000 ns:test.messages",
>>         "lastExtent" : "a:2000 ns:test.messages",
>>         "extentCount" : 36,
>>         "datasize" : 817401456,
>>         "nrecords" : 2359189,
>>         "lastExtentSize" : 1246609408,
>>         "padding" : 1,
>>         "firstExtentDetails" : {
>>                 "loc" : "0:3000",
>>                 "xnext" : "0:12000",
>>                 "xprev" : "null",
>>                 "nsdiag" : "test.messages",
>>                 "size" : 20480,
>>                 "firstRecord" : "null",
>>                 "lastRecord" : "null"
>>         },
>>         "deletedCount" : 17053034,
>>         "deletedSize" : 7321177984,
>>         "nIndexes" : 6,
>>         "keysPerIndex" : {
>>                 "test.messages.$_id_" : 2359189,
>>                 "test.messages.$HOST_1" : 2359189,
>>                 "test.messages.$DATE_-1" : 2359189,
>>                 "test.messages.$FACILITY_1" : 2359189,
>>                 "test.messages.$PRIORITY_1" : 2359189,
>>                 "test.messages.$PROGRAM_1" : 2359189
>>         },
>>         "valid" : true,
>>         "errors" : [ ],
>>         "warning" : "Some checks omitted for speed. use {full:true} option to do more thorough scan.",
>>         "ok" : 1

>> }
>>> db.messages.stats()

>> {
>>         "ns" : "test.messages",
>>         "count" : 2359189,
>>         "size" : 817401456,
>>         "avgObjSize" : 346.47561344173783,
>>         "storageSize" : 8176332800,
>>         "numExtents" : 36,
>>         "nindexes" : 6,
>>         "lastExtentSize" : 1246609408,
>>         "paddingFactor" : 1,
>>         "flags" : 1,
>>         "totalIndexSize" : 675353952,
>>         "indexSizes" : {
>>                 "_id_" : 76568240,
>>                 "HOST_1" : 125795936,
>>                 "DATE_-1" : 131445552,
>>                 "FACILITY_1" : 106835792,
>>                 "PRIORITY_1" : 104808144,
>>                 "PROGRAM_1" : 129900288
>>         },
>>         "ok" : 1

>> }

>> I would be grateful if someone can explain me what's wrong or how to investigate further. Sascha.

>> Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp
>> Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher
>> Sitz der Gesellschaft: Halle/Saale
>> Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
>> UST-ID-Nr. DE 158253683
>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb