Estimating memory usage on shard config server

[Avery Fay]

Hi,

How can we estimate how much memory is required for the config server in a sharded cluster? We initially thought that the config server had fairly light memory requirements, but on a 1GB machine it would easily get into a swap storm right after balancing. Now, after upgrading the machine, we have a constant 1.1GB used and peaks > 2.2 GB. What happens as our dataset grows? Will the config server's memory requirement grow too? Honestly, the amount of memory it uses right now is sort of disappointing as it forces us to use a machine with 4GB on it, when 99% of the time it is completely idle.

Avery

[Scott Hernandez]

What does db.serverStatus() and "free -ltm" show? Is the server doing
anything other than being a config server?

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

[Avery Fay]

free -m

             total       used       free     shared    buffers     cached

Mem:          4096       1818       2278          0         75        454

-/+ buffers/cache:       1287       2808

Swap:         8191          0       8191

This of course is not the peak. Peak usage is during balancing.

> db.serverStatus()

{

"version" : "1.6.5",

"uptime" : 446007,

"uptimeEstimate" : 371659,

"localTime" : "Wed Feb 23 2011 01:59:10 GMT+0000 (UTC)",

"globalLock" : {

"totalTime" : 446006864144,

"lockTime" : 105882020,

"ratio" : 0.00023739997859273843,

"currentQueue" : {

"total" : 0,

"readers" : 0,

"writers" : 0

}

},

"mem" : {

"bits" : 64,

"resident" : 1035,

"virtual" : 21390,

"supported" : true,

"mapped" : 80

},

"connections" : {

"current" : 1259,

"available" : 6933

},

"extra_info" : {

"note" : "fields vary by platform",

"heap_usage_bytes" : 2591808,

"page_faults" : 0

},

"indexCounters" : {

"btree" : {

"accesses" : 26068,

"hits" : 26068,

"misses" : 0,

"resets" : 0,

"missRatio" : 0

}

},

"backgroundFlushing" : {

"flushes" : 7433,

"total_ms" : 44891,

"average_ms" : 6.039418808018297,

"last_ms" : 4,

"last_finished" : "Wed Feb 23 2011 01:58:38 GMT+0000 (UTC)"

},

"cursors" : {

"totalOpen" : 0,

"clientCursors_size" : 0,

"timedOut" : 2

},

"opcounters" : {

"insert" : 2659,

"query" : 1152157,

"update" : 640923,

"delete" : 239752,

"getmore" : 82940,

"command" : 74759

},

"asserts" : {

"regular" : 0,

"warning" : 0,

"msg" : 0,

"user" : 0,

"rollovers" : 0

},

"ok" : 1

}

[Avery Fay]

Oh and nothing else runs on the server.

[Scott Hernandez]

On Tue, Feb 22, 2011 at 6:00 PM, Avery Fay <av...@shadypixel.com> wrote:
> free -m
>              total       used       free     shared    buffers     cached
> Mem:          4096       1818       2278          0         75        454
> -/+ buffers/cache:       1287       2808
> Swap:         8191          0       8191
> This of course is not the peak. Peak usage is during balancing.

Are these numbers from the config server instance?

>> db.serverStatus()
> {
> "version" : "1.6.5",
> "uptime" : 446007,
> "uptimeEstimate" : 371659,
> "localTime" : "Wed Feb 23 2011 01:59:10 GMT+0000 (UTC)",
> "globalLock" : {
> "totalTime" : 446006864144,
> "lockTime" : 105882020,
> "ratio" : 0.00023739997859273843,
> "currentQueue" : {
> "total" : 0,
> "readers" : 0,
> "writers" : 0
> }
> },
> "mem" : {
> "bits" : 64,
> "resident" : 1035,
> "virtual" : 21390,
> "supported" : true,
> "mapped" : 80
> },
> "connections" : {
> "current" : 1259,
> "available" : 6933

1259 connections seems awfully high for a config server.

[Avery Fay]

On Tue, Feb 22, 2011 at 6:15 PM, Scott Hernandez <scotthe...@gmail.com> wrote:

On Tue, Feb 22, 2011 at 6:00 PM, Avery Fay <av...@shadypixel.com> wrote:
> free -m
>              total       used       free     shared    buffers     cached
> Mem:          4096       1818       2278          0         75        454
> -/+ buffers/cache:       1287       2808
> Swap:         8191          0       8191
> This of course is not the peak. Peak usage is during balancing.

Are these numbers from the config server instance?

Yes.

*shrug*. There are 412 alone between the config server and one of the shard's. Nothing in the logs indicates that there's a problem with that.

Avery

[Eliot Horowitz]

That looks like a connection leaking causing massive memory usage.
Can you send db.printShardingStatus()
How many mongos do you have?
Each connection will use a reasonable amount of space, so easily could
be causing the issues.

[Avery Fay]

It's attached. We have 13 mongos (one on each app server).

[Eliot Horowitz]

Can you run
db.runCommand( "connPoolStats" ) on all of the mongos.
Something definitely seems off, but not making sense.

[Avery Fay]

The results are attached.

Avery

[Eliot Horowitz]

That all looks fine.
Can you do a netstat -an | grep 27019 on the config server?
How many config servers do you have? If 3, is this happening on 1 or all?

[Avery Fay]

Attached.

We only have one config server right now because the cluster was initially setup wrong. Could that be the problem? I was planning on adding two more when we upgraded to 1.8.0. One of the reasons I wanted to figure out this memory issue is that I'd rather add them to data nodes, but if each instance uses > 1GB of memory that causes a certain amount of resource imbalance across the shards.

The only other thing I should mention is that I noticed memory spike right after a chunk migration completes.

We'll get a bunch of messages like:

Sat Feb 19 09:46:46 [conn1894] query config.chunks scanAndOrder  reslen:119257 nscanned:446 { query: { ns: "index.lookup_table", shard: { $in: [ "mpliveltmongo04/10.177.209.20,10.177.209.21", "mpliveltmongo04" ] } }, orderby: { min: 1 } }  nreturned:446 160ms

After it completes memory will go down somewhat, but still stays very high.

Avery

[Eliot Horowitz]

What kind of nodes are:
10.177.200.235
10.177.201.20
10.177.204.205
?

Also, can you send netstat at the same time as db.runCommand(
"connPoolStats" ) on all servers?

[Avery Fay]

Those three are shard servers that haven't been converted to replica sets yet. (the replica set shard servers are 10.177.209.2x)

I've attached the output from connPoolStats for the config server in config.txt, all mongos instances in mongos.txt, all shard servers in shard.txt, and finally netstat -tn run on the config server (run within a minute of everything else).

Avery

[Eliot Horowitz]

Can you try the 1.6 nightly?
There was an improvement made that may help you a lot.