MongoDB Performance Testing query

[sathish kumar]

Hi Team,

            I am doing performance testing on MongoDB...with RHEL 6.4....  I have generated MongoDB database with 35 million records

which is close to 25GB in Disk size. 

Hardware which i am running has 90GB of Main Memory, I used YCSB to stimulate workload... which reads around 15 million records in my test

but while running the test... i am seeing Mongod or MongoDB process uses only 47 to 50GB of Memory... I still have 40GB of memory left free.

Currently i have two database one is 25GB and other is 12GB in disk size.

I feel the system has more resource free, which can be utilized and run much faster..  Is there any settings in MongoDB side or System side to tell 

the MongoDB/system to use the rest of the available free memory also... 

Following are the runtime stats, I want to know is there any setting either in mongoDB or system level RHEL setting to leverage my entire hardware

while running my workload. 

Note: During my observation there wasn't much swapin and swapout also happening.

# free -g

             total       used       free     shared    buffers     cached

Mem:            90         48         42          0          0         43

-/+ buffers/cache:          3         86

Swap:            3          0          3

# vmstat 1 5

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

13  0      0 44354944 458624 46119360    0    0     2    81    0    1  1  0 99  0  0

 4  0      0 44353344 458624 46119424    0    0     0     0 68632 278273 10  9 80  0  0

 6  0      0 44354112 458624 46119424    0    0     0    48 64490 245212 10  9 81  0  0

 8  0      0 44354176 458624 46119424    0    0     0     0 53191 187589  9  7 84  0  0

10  0      0 44354112 458624 46119424    0    0     0     0 45025 153016  8  6 85  0  0

   > show dbs

bigdata350db    49.9296875GB

bigdatadb       23.9423828125GB

local   0.078125GB

testdb  0.203125GB

tutorial        0.203125GB

[Asya Kamsky]

Sathish:

Only if you are actually touching (reading) all of the data in both of
these databases will the pages be brought into RAM.

It's very possible that MongoDB *is* using all of the memory it can
possibly take advantage of. YCSB is hitting less than half of your
data, and I'm guessing that the actual documents are much smaller than
the full DB size (index sizes would be included in the totals).

Depending on how you have configured and are running YCSB, the test
may simply not be placing very high demand of the database - can you
elaborate on how you are running the workload (which workload) and how
many YCSB clients you are running, and how many threads are running in
each client?

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-dev...@googlegroups.com.
> To post to this group, send email to mongo...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-dev.
> For more options, visit https://groups.google.com/d/optout.

[sathish kumar]

Asya,

         Thank you for the response.   

         I am running YCSB workloada, Written few python wrapper scripts to

invoke the YCSB client with different operationcount, recordcount, ops/sec, threads via arguments... 

I have only one node as of now[Both client and server in same server] and one YCSB client which has 2000 & 2500 thread connection to db.

I want to determine what is the max limit for current 90GB RAM. Trying to find how far i can go with one client parallely increasing 

number of threads and find out a point[No of the threads] where i am able to exploit my entire hardware and get the maximum ops/sec.

  

Following are my command line arguments 

Command line: -db com.yahoo.ycsb.db.MongoDbClient -s -P workloads/workloada -p operationcount=15000000 -p recordcount=35000000 -p mongodb.url=mongodb://localhost:27017 -p mongodb.maxconnections=1000 -s -p mongodb.writeConcern=errors_ignored -p mongodb.database=my350db -target 10000 -threads 2500 -t

mongo connection created with localhost:27017/my350db.

[Asya Kamsky]

Running YCSB 2500 threads on the same host as mongod may be part of the problem - it will make it hard to know if the limiting factor is YCSB or mongod.

[sandeep]

Hi sathish I need to check performance test for MongoDB, and Cassandra using YCSB. Could you please let me know how did u setup this? is there any links you can provide me. Thanks.