How to keep it all in RAM ?

Laurent Vaills

unread,

Jul 4, 2012, 10:32:59 AM7/4/12

to mongod...@googlegroups.com

Hi,

I'am experiencing some performances problems in our mongo replicat set.

It is built with 3 hosts with 32 Gb RAM each with no sharding currently running on Debian 6, 64 bits.

Here is the statistics of our database :

> db.stats();

{

"db" : "bss",

"collections" : 3,

"objects" : 20402919,

"avgObjSize" : 4222.7186184486645,

"dataSize" : 86155785932,

"storageSize" : 83538116000,

"numExtents" : 42,

"indexes" : 20,

"indexSize" : 16388563072,

"fileSize" : 115841433600,

"nsSizeMB" : 16,

"ok" : 1

}

(3 collections are defined but we are really using only 1)

From top or htop, I see that only a little bit more than 1G of memory is used on my machine.

From mongostat :

insert query update delete getmore command flushes mapped vsize res

*0 1 *3 *0 0 3|0 0 128g 257g 581m

*0 *0 *7 *0 0 5|0 0 128g 257g 581m

*0 *0 *2 *0 0 7|0 0 128g 257g 581m

*0 *0 *3 *0 0 3|0 0 128g 257g 581m

*1 *0 *1 *0 0 5|0 0 128g 257g 581m

*0 *0 *1 *0 0 3|0 0 128g 257g 581m

*1 *0 *12 *0 0 3|0 0 128g 257g 581m

*1 *0 *4 *0 0 5|0 0 128g 257g 582m

*0 *0 *8 *0 0 9|0 0 128g 257g 581m

*0 1 *6 *0 0 1|0 0 128g 257g 581m

I understand that MongoDB has only 581m in resident memory which is not the size of all the indexes.

After some reading (http://answers.dotcloud.com/question/566/difference-in-mongodb-memory-usage-reported and http://blog.serverdensity.com/2010/12/13/mongodb-monitoring-keep-in-it-ram/ ) I understand that MongoDB is using mmap.

We have some batches that iterates a lot of documents of our collection and this is quite slow because we have a lot of page faults.

Everything that I read about MongoDB says to "keep all in RAM" but I never found the way to achieve that. Are there some special settings to tune in order that MongoDB really puts at least all its indexes in RAM and 'caches' more data in RAM than it currently does. I would be very happy to see the number of page faults down even even when my batches are running.

Regards,

Laurent

Eliot Horowitz

unread,

Jul 4, 2012, 8:35:28 PM7/4/12

to mongod...@googlegroups.com

Can you send the output of:
"free -m"
and
"sudo blockdev --report"
Most likely something at the storage level isn't tuned

> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb

Laurent Vaills

unread,

Jul 5, 2012, 7:06:24 AM7/5/12

to mongod...@googlegroups.com

Hi Eliott,

here is the output that you requested :

$ free -m

total used free shared buffers cached

Mem: 32230 24489 7740 0 51 23367

-/+ buffers/cache: 1070 31160

Swap: 4000 11 3989

$ sudo blockdev --report

RO RA SSZ BSZ StartSec Size Device

rw 256 512 4096 0 584652423168 /dev/sda

rw 256 512 4096 63 2097414144 /dev/sda1

rw 256 512 4096 4096575 4194892800 /dev/sda2

rw 256 512 4096 12289725 4194892800 /dev/sda3

rw 256 512 1024 20482875 1024 /dev/sda4

rw 256 512 4096 20482938 20974431744 /dev/sda5

rw 256 512 4096 61448688 1052803584 /dev/sda6

rw 256 512 4096 63505008 12584646144 /dev/sda7

rw 256 512 512 88084458 539545434624 /dev/sda8

I googled a little about the blockdev and the tunings which can be done. Assuming the current output, I would suggest to increase the RA from 256 to at least 1024 and maybe lot more (I read about setting to 16384 (8MB)).

But maybe, you (or someone else) have some other advices.

Regards,

Laurent

> mongodb-user+unsubscribe@googlegroups.com

> See also the IRC channel -- freenode.net#mongodb

> mongodb-user+unsubscribe@googlegroups.com

Eliot Horowitz

unread,

Jul 5, 2012, 8:57:31 AM7/5/12

to mongod...@googlegroups.com

I would do the opposite, and try setting read ahead to 8.
Need to do it all devices and then restart mongod.

>> > mongodb-user...@googlegroups.com

>> > See also the IRC channel -- freenode.net#mongodb
>

> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to

> mongodb-user...@googlegroups.com

Jorge Costa

unread,

Jul 5, 2012, 9:42:23 AM7/5/12

to mongod...@googlegroups.com, el...@10gen.com

large filesystem read-ahead values help with sequential IO, where smaller values reduce IO read-ahead wastage during random-read IO operations.

With a FRA value of 8 (512bytes) = 4Kb is roughly your "avgObjSize" : 4222. You should expect that for each document accessed from disk, you'd read-ahead the next block or document (as long the documents fits in a 4K block).
That's pretty much ideal based on your document size.

I don't think you need to restart the mongod after changing the filesystem read ahead, blockdev -setfra has immediate effect.

Eliot Horowitz

unread,

Jul 5, 2012, 10:20:47 AM7/5/12

to mongod...@googlegroups.com

You definitely need to restart mongod.

Praveen

unread,

Jul 9, 2012, 10:14:26 AM7/9/12

to mongodb-user

Hi All,

I am facing a similar situation, but on windows. Any idea how can I
set the READ ACCESS on windows? or is there some other parameter that
I will need modify for windows.

I am on Windows 7

Thanks

Praveen

Scott Hernandez

unread,

Jul 9, 2012, 10:36:36 AM7/9/12

to mongod...@googlegroups.com

On windows this is generally referred to as pre-fetching, which you can turn off.

Laurent Vaills

unread,

Jul 13, 2012, 3:15:07 AM7/13/12

to mongod...@googlegroups.com

Thanks for your advices, I'll try theses settings as soon as I can.

Laurent

> mongodb-user+unsubscribe@googlegroups.com

> See also the IRC channel -- freenode.net#mongodb
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to

> mongodb-user+unsubscribe@googlegroups.com

Rob Moore

unread,

Nov 8, 2012, 8:37:52 PM11/8/12

to mongod...@googlegroups.com

How is the "hh" collection used. That is the one that is locked based on the mongostat.

Have you tried turning on the built in profiler to see what the slow operations are? http://www.mongodb.org/display/DOCS/Database+Profiler

Also on indexes:

"_id_" : 228928,

"created_at_-1" : 212576,

"created_at_1" : 228928,

"updated_at_-1" : 212576,

"updated_at_1" : 228928,

"recent_search_index" : 376096

You don't need both the "created_at_-1" and "created_at_1". MongoDB knows how to reverse the direction of the index. Similar argument for updated_at.

Rob.

On Thursday, November 8, 2012 6:20:18 PM UTC-5, Jérémy Lecour wrote:

Hi everybody,

Is there anything new about this ?

I have the same issue, and was searching the group before posting my own topic.

Some facts :

> db.stats()
{
"db" : "hh",
"collections" : 15,
"objects" : 398769,
"avgObjSize" : 18754.794249302227,
"dataSize" : 7478830548,
"storageSize" : 12172083200,
"numExtents" : 122,
"indexes" : 46,
"indexSize" : 26302192,
"fileSize" : 14958985216,
"nsSizeMB" : 16,
"ok" : 1
}

The main collection is "searches" :

> db.searches.stats()
{
"ns" : "hh.searches",
"count" : 4365,
"size" : 1358614876,
"avgObjSize" : 311251.97617411223,
"storageSize" : 2159874048,
"numExtents" : 13,
"nindexes" : 6,
"lastExtentSize" : 597745664,
"paddingFactor" : 1.000000000012644,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 1488032,
"indexSizes" : {
"_id_" : 228928,
"created_at_-1" : 212576,
"created_at_1" : 228928,
"updated_at_-1" : 212576,
"updated_at_1" : 228928,
"recent_search_index" : 376096
},
"ok" : 1
}

The memory :

$ free -m
total used free shared buffers cached

Mem: 32230 27216 5013 0 2686 5535
-/+ buffers/cache: 18994 13235
Swap: 1905 1458 447

Blockdev :

$ sudo blockdev --report
RO RA SSZ BSZ StartSec Size Device

rw 256 512 4096 0 1999844147200 /dev/sdb
rw 256 512 4096 63 999996609024 /dev/sdb1
rw 256 512 4096 1953118440 900001912320 /dev/sdb2
rw 256 512 4096 0 299439751168 /dev/sda
rw 256 512 1024 2048 499122176 /dev/sda1
rw 256 512 4096 976896 1000341504 /dev/sda2
rw 256 512 4096 2930688 4999610368 /dev/sda3
rw 256 512 1024 12697598 1024 /dev/sda4
rw 256 512 4096 12697600 999292928 /dev/sda5
rw 256 512 4096 14651392 999292928 /dev/sda6
rw 256 512 4096 16605184 9999220736 /dev/sda7
rw 256 512 4096 36136960 9999220736 /dev/sda8
rw 256 512 4096 55668736 999292928 /dev/sda9
rw 256 512 4096 57622528 99997450240 /dev/sda10
rw 256 512 4096 252932096 169938518016 /dev/sda11

The main problem I have is shown by mongostat almost all the time ; the "lock db" is near 100% whenever I make a few dozen request at the same time :

insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time
0 18 149 0 0 18 0 12.2g 25g 801m 0 hh:97.3% 0 5|0 0|4 58k 17k 49 00:13:51
0 28 155 0 0 30 0 12.2g 25g 801m 0 hh:95.4% 0 7|0 0|2 59k 3m 49 00:13:52
0 28 180 0 0 31 0 12.2g 25g 801m 0 hh:96.9% 0 10|0 0|1 76k 32k 49 00:13:53
0 21 146 0 0 20 0 12.2g 25g 801m 0 hh:96.0% 0 11|0 0|1 66k 92k 50 00:13:54
0 26 101 0 0 29 0 12.2g 25g 801m 0 hh:95.9% 0 11|0 0|1 49k 110k 57 00:13:55
0 28 103 0 0 28 0 12.2g 25g 801m 0 hh:96.4% 0 9|0 0|1 56k 35k 57 00:13:56
0 34 117 0 0 35 0 12.2g 25g 801m 0 hh:97.6% 0 12|0 0|1 63k 44k 57 00:13:57
0 26 133 0 0 25 0 12.2g 25g 801m 0 hh:97.1% 0 10|1 0|1 61k 37k 57 00:13:58
0 41 145 0 0 46 0 12.2g 25g 801m 0 hh:95.9% 0 10|0 0|1 71k 72k 53 00:14:00
0 29 119 0 0 26 0 12.2g 25g 801m 0 hh:97.1% 0 9|0 0|1 64k 31k 51 00:14:01

My "searches" collection is used like this :

* <20K records, quite big, with deep BSON structure
* read/writes are almost 95% of the time using the primary index (which is not an ObjectID but an integer for an historic reason)
* a lot of updates are touching the same record at the same time, but on different parts of it

Thanks for your input on this

Rob Moore

unread,

Nov 13, 2012, 7:49:04 PM11/13/12

to mongod...@googlegroups.com

Sorry - Not sure how I missed the database name.

Did you get a chance to look for long running commands using the profiler?

For the queries: Are you expecting to scan a range? There are a lot of getmore commands so either the batch size is being set small or there is a query scanning large parts of the collection.

For the updates: What write concern are you using?

Rob.

On Friday, November 9, 2012 6:08:23 AM UTC-5, Jérémy Lecour wrote:

Le vendredi 9 novembre 2012 02:37:52 UTC+1, Rob Moore a écrit :

Hi Rob,

Many thanks for your answer.

How is the "hh" collection used. That is the one that is locked based on the mongostat.

"hh" is the database, "searches" is the mainly used collection.

The typical use is described in my original message

My "searches" collection is used like this :

* <20K records, quite big, with deep BSON structure
* read/writes are almost 95% of the time using the primary index (which is not an ObjectID but an integer for an historic reason)
* a lot of updates are touching the same record at the same time, but on different parts of it

Have you tried turning on the built in profiler to see what the slow operations are? http://www.mongodb.org/display/DOCS/Database+Profiler

Not yet, I'll do that.

Also on indexes:
"_id_" : 228928,
"created_at_-1" : 212576,
"created_at_1" : 228928,
"updated_at_-1" : 212576,
"updated_at_1" : 228928,
"recent_search_index" : 376096

You don't need both the "created_at_-1" and "created_at_1". MongoDB knows how to reverse the direction of the index. Similar argument for updated_at.

Thanks, I've changed that, but I doubt it's the cause of my problem.

--
Jérémy

Reply all

Reply to author

Forward