Mongodb 3 is slower than Mongodb 2 ?

[Andrea Bozzetto]

Hi to all,

  we have converted our application to MongoDB 3. We have seen some operations like insert are slower.

We have made a test application. This webapp exposes a REST service called from a jmeter project.

The jmeter runs 1500 thread in 15 seconds

The results  are indicated in the next table.  

mongodb ver/ jdriver | Average | MAX

---------------------------------------------

2.6.9 / 2.13.1       |   68    | 977

3.0.0 / 2.13.1       |  137    | 1679

2.6.9 / 3.0.0        |   78    | 1202

3.0.0 / 3.0.0        |  119    | 1500

You can see, mongodb 2 is faster than mongodb 3.

I attach the maven project, with  jmeter test

It's something wrong with my test ?

Thank's a lot

bye

[Asya Kamsky]

You are using jmeter?   How exactly?   At one point jmeter would send everything wrapped in db.eval which was way wrong, and of course it would be limited by that.

Instead of attaching the project which I don't know if I can even read, why don't you describe exactly what you are testing?   

> The jmeter runs 1500 thread in 15 seconds

What do these threads do?

In addition, I would suggesting testing against 3.0.3 rather than 3.0.0.

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/7585b7c0-9520-4971-a00f-f4c612958720%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!

[Andrea Bozzetto]

The attachment is a maven project. The project create a webapp. 

This webapp exposes a REST service in POST. The REST service execute a query and save the json received in the POST in a MongoDB collection.

 I use jmeter only for send the json to REST service. 

To make some stress test jmeter runs 1500 thread in 15 seconds

 

 I did a new test using mongodb 3.03 and wiredtiger.

mongodb ver/ jdriver        | Average | MAX
---------------------------------------------

2.6.9 MMAPv1 / 2.13.1       |   68    | 977
3.0.0 MMAPv1 / 2.13.1       |  137    | 1679
2.6.9 MMAPv1 / 3.0.0        |   78    | 1202
3.0.0 MMAPv1 / 3.0.0        |  119    | 1500
3.0.3 wtiger / 2.13.1       |  184    | 1202
3.0.3 wtiger / 3.0.0        |  165    | 874

thanks a lot

bye

[Asya Kamsky]

Andrea:

So the test appears to be multithreaded - how many cores on the server machine?  (1500 threads seems like a lot if you have, for example, only a few cores)...

I'm a little surprised that mmap would be faster than wired tiger - however, regardless of threads, I would *not* expect 3.0 MMAP to ever be slower than 2.6 MMAP *if* the delay was in the database itself.

I suspect the delay is in the way jmeter interacts with mongod - do you happen to have the logs from mongod during the time the test is running?   It would be illuminating to see what operations are running on the server and how long they are each taking.   If no operation takes less than 100ms it wouldn't be logged but you can increase the logging verbosity starting mongod with -v flag or by running this command on the shell:

db.adminCommand({setParameter:1, logLevel:1})

(after, don't forget to run it again with logLevel:0).

Let me know what's in the logs after the test (probably don't need the full log but just showing some representative lines from it...)

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/0134d8f5-c23d-4c9d-b40c-f73058b51cb6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Stephen Dillon]

In my experience, 3.0 wiredTiger has been faster than 2.6.x when comparing inserts. I've not achieved the claimed 7+x performance boost in my use case but I vae seen better performance.

Here are some other thoughts; beyond what the Mongo rep has covered.

• Is this a sharded cluster or a standalone instance?

• If it is a sharded cluster; did you upgrade from 2.6.x to 3.0 WT including a resync of the data into the new dbpath for WT or did you start testing with an empty data path?

I ask because I wonder if you're 3.0 cluster is suffering from premature sharding. If you do not have any data in the new dbpath for 3.0 WT and you run insert tests, I can easily see how there could be overhead on the cluster once the balancer begins to move data. I've seen such performance degradation, across all versions, many times when the QA team wishes to perform tests against a multi shard cluster with no data.

[Andrea Bozzetto]

"I suspect the delay is in the way jmeter interacts with mongod"

Jmeter doesn't interacts with mongodb directly, JMeter call the REST Service. The REST Service interact with mongodb.

Here the java code:

    @RequestMapping(method = RequestMethod.POST, value = "/newDocument", headers = "content-type=application/json")

    public ResponseEntity<String> applicationUpsert(@RequestBody String json) {

        HttpHeaders responseHeaders = new HttpHeaders();

        responseHeaders.setContentType(MediaType.valueOf("application/json"));

        try {

            DB db = mongo.getDB("test");

            Jongo jongo = new Jongo(db);

            String query = "{'RegistryObjectList.ExtrinsicObject' : {$elemMatch : { 'ExternalIdentifier.value' : {$in: ['73387957557175793898']}, "

                    + "'Slot.name': 'hash', 'ExternalIdentifier.identificationScheme': 'urn:uuid:2e82c1f6-a085-4c72-9da3-8640a32e42ab'}}}";

            MongoCollection submission = jongo.getCollection("testPerf");

            Iterable<String> documentsAsJson = submission.find(query).map(new JSONResultHandler());

            Iterator<String> iter = documentsAsJson.iterator();

            String s;

            if (iter.hasNext()) {

                while (iter.hasNext()) {

                    s = iter.next();

                }

            }

            DBObject jsonColl = (DBObject) JSON.parse(json);

            WriteResult res = submission.insert(jsonColl);

            return new ResponseEntity<String>("{\"response\": \"OK\"}",

                    responseHeaders,

                    HttpStatus.CREATED);

        } catch (Exception e) {

            log.error(e.getMessage(), e);

            return new ResponseEntity<String>("{\"response\": \"KO\", \"errorMessage\": \"" + e.getMessage() + "\"}",

                    responseHeaders,

                    HttpStatus.INTERNAL_SERVER_ERROR);

        }

    }

Jmeter divides the threads in 15 seconds, so the maximun threads per seconds is 200.

A sample image of jmeter result

Thanks a lot

bye

[Asya Kamsky]

Unfortunately, without mongod logs that I asked about I wouldn't be able to guess where the bottleneck is.

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/37d805ee-026d-48a8-9fc1-fb9acc217066%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andrea Bozzetto]

Hi Asya,

   this is the log of mongodb with log level 0 and 1

Thanks

Bye

[Asya Kamsky]

Great!  Now that you know how to get a logLevel:1 log of your test (this one was done with 3.0) you can do the same thing with 2.6 and compare them.

Basically you are looking for groups of operations that are taking longer in 3.0 than 2.6.   If you see queries be slower, it could be because I see a couple of queries that are unindexed - are they taking less time in 2.6?   These sorts of details would usually make it possible to see what is responsible for the slower performance.

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/062708dc-93ba-4a63-9977-9f8ccee017b6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[henry oswald]

We are seeing something very similar after upgrading our replicaset to mongo3 and using the same mmap files we are seeing slower queries. Did you manage to find the root of the problem Andrea?

[Andrea Bozzetto]

Hi Henry,

  by now we have stopped the test with MongoDB 3 and we are still using mongoDB 2.6.3.

We don't known where is the problem with MongoDB 3.

bye

[Asya Kamsky]

Please describe your full setup, workload and observed performance issue in a new thread - that will make it easier to help you.

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/b5b88470-50aa-4da8-9272-7a3ea7e1c96f%40googlegroups.com.