mongodb virtual memory usage is very high

[prashant]

Hi,

We have mongodb 1.6.5 running on SUSE linux box with 16G memory and
479G hard disk. We have the virtual memory limit set to 450G. We have
one database with multiple collections each containing around 8
million records. We do insertions/deletions and updates to the data.
Currently the collections (including indexes) are taking around 83G of
storage/disk space:

Somehow the mongod process is taking 428G of virtual memory as shown
below:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2356 demo 20 0 428g 3.0g 3.0g S 2 19.2 4100:46 mongod

> db.serverStatus().mem
{
"bits" : 64,
"resident" : 3073,
"virtual" : 438295,
"supported" : true,
"mapped" : 438011
}
>


When I try to create a new collection I get the following error:

> db.testcoll.insert({key:1})
can't map file memory

In the mongo log I get the following:

Thu Mar 31 20:44:57 [conn104] mmap() failed for /export/home/rail/
var/lib/mongodb/mis3.218 len:2146435072 errno:12 Cannot allocate
memory
Thu Mar 31 20:44:57 [conn104] mmap failed with out of memory, if
you're using 32-bits, then you probably need to upgrade to 64

1) What I don't understand is that if the amount of data currently
stored is only 83G, then why does mongod require so much virtual
memory.
2) Is there a way to free up some of this space without bringing the
database offline so that we can create new collections in the
database?

Thanks and Regards,
Prashant

[grahamhar]

Hi,

How many indexes do you have? Can you post output of db.stats()

Regards
Graham

[prashant]

Thanks for a quick reply Graham!!

here is an example for stats() for the collection for one day. We have
several collections similar to this in a single database

> db['collection-20110329'].stats()
{
"ns" : "testcol.collection-20110329",
"count" : 8495026,
"size" : 16235067108,
"avgObjSize" : 1911.1262411674784,
"storageSize" : 26093243392,
"numExtents" : 42,
"nindexes" : 11,
"lastExtentSize" : 1872502528,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 11777607328,
"indexSizes" : {
"_id_" : 355468224,
"index1" : 3673081520,
"index2" : 483459072,
"index3" : 370549136,
"index4" : 1295016896,
"index5" : 1327743040,
"index6" : 1256285120,
"index7" : 1155254176,
"index8" : 820462528,
"index9" : 544891840,
"index10" : 495395776
},
"ok" : 1
}

The indexes that we have a compound indexes consisting of 2-8 columns/
fields.

Just to reiterate again, we keep inserting/deleting/updating data very
frequently. Also we drop collections from the database after they are
7 days old.

Regards,
Prashant

[prashant]

Just realized that you asked for stats() on the database. The
following data is from a different server (because we had to drop the
earlier database from which I gave the data in my original post), but
the setup is similar and here also we are facing a similar problem:

> db.stats()
{
"collections" : 12,
"objects" : 44152689,
"avgObjSize" : 1896.623499239197,
"dataSize" : 83741027512,
"storageSize" : 116157463040,
"numExtents" : 250,
"indexes" : 40,
"indexSize" : 38495546736,
"fileSize" : 446392434688,
"ok" : 1

[Nat]

In your case, you should run repairDatabase to compact the database.
It will bring down the database size thus reducing virtual memory
required.

[prashant]

I know the repair database solution, but I thought that internally
mongodb should be able to reuse the space that is occupied even when
you delete documents from a collection. Shouldn't the space be reused
internally for newer collections?

Thanks!
Prashant

[Kent Sibilev]

I believe Mongo does reuse the free space. But it doesn't do on the
fly compaction of the db files. Consequently, it cannot delete already
allocated db file without running the database repair. Considering
that mongo uses memory-mapped files, the virtual size process occupies
should be around the total size of your db files.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
Kent

[Nat]

Unless you drop the collection entirely, the free list can only be
reused by the same collection.

[Manan Shah]

Use caution while doing a repairDatabase() operation. Although it would free up space, it might drop any collections including system.profile in which you might have a log of slow queries.
Always take a full backup before running this command. This is the result of the operation I ran on one of our test instances.

Before repairDatabase()

Name Size Storage Size Data Size Index Size Collections Objects

group 4.5g 2.11g 182.5m 350.12m 21    772933

After repairDatabase()

group 1.5g 231.31m        181.54m      315.02m 17    770081

So you can see it removed a few collections from the system db but be aware that it will remove any corrupt data and not repair it during the process.