Wired Tiger poor write performance compared to MMAP

[Jeff Olmstead]

I’ve started benching MongoDB’s MMAPv1 vs WiredTiger on my local setup and before even getting out of the gate, I’m seeing some performance results that I wouldn’t have expected.

 

I have a java program that does 2 batch inserts. The first batch is generally ~10K and the second ~1 million documents into 2 unindexed collection (except the default id index). This first test was single threaded and I’ve run the loop 7 times and it appears as if MMAP is outperforming the wired tiger engine by ~6-7 times. (File: PC_MMAP_Insert.txt & PC_WiredTiger_Insert.txt, show these results)

 

The java app being single threaded isn’t going to utilize wired tiger’s document level concurrency but I would expect the single threaded performance to be relatively equal. And this thought was backed up by benchmark performance results I’ve found online. http://www.acmebenchmarking.com/2015/02/mongodb-v30-compression-benchmarks.html shows wired tiger with snappy compression should be in the range of -10% with respect to performance.

 

Multiple other tests have all backed up my results, including using mongostat and analyzing the inserts/sec of the engine. File: mongoStat_WiredTiger_MMAP.txt shows WiredTiger getting ~15K writes / sec while MMAP is at ~70K (~5 times)

 

 

Interesting to note is that when I disabled Journaling with WiredTiger the insert speeds were approximately equal to MMAP. I’ve started in on the hunt for how MMAP vs WiredTiger write the journal. I see MMAP uses a 1 GB file while WiredTiger appears to write only 100 MB files. Watching the journal directory as WiredTiger performs appears to take ~30 seconds before the 100 MB PrepLog file can continue to the next entry. I believe the bottleneck is in this process for whatever reason…

 

 

So when it comes down to it does anyone have any idea why wired tiger is performing so poorly in comparison to MMAPv1?

 

 

My PC specs are:

 

Processor: I7-3770 C...@3.40 GHz 3401 MHz 4 Core, 8 Logical Processors

Hard drive: Samsung HE160HJ SCSI

                    160 GB

                    7.2 K spin speed

RAM: 16 GB

 

Java App is using the mongo-java-driver-2.13.1

MongoDB v. 3.0.4 and when using wired tiger the configuration file includes

wiredTiger:

    engineConfig:

        cacheSizeGB: 12

    collectionConfig:

        blockCompressor: none

    indexConfig:

        prefixCompression: false

 

(As a side note, removing the compression settings does not have an impact on improving performance with this simple test)

[MARK CALLAGHAN]

I have never run a test with only the _id index. Your results are interesting. 
* Was the oplog enabled?
* What is the write concern?

[Jeff Olmstead]

I am only using a single stand alone server. I'm not familiar with the opLog but it appears to be applicable when using a replica set. So i believe this is off by default.

The write concern is ACKNOWLEDGED.

If you add an index other than the primary _id index you will still see the large discrepancy between MMAP & WiredTiger's insert performance. 

I'm starting to wonder if the BulkWriteOperation performs more efficiency with MMAP than Wired Tiger.

If that's the case then I imagine I'll have to leverage Wired Tiger's concurrency to reign back in the performance. I'll test this out shortly

[Jeff Olmstead]

Yes that appears to be what is happening.

I believe the primary issue is with the BulkWriteOperation and Journaling. However MongoDB writes the journal it is more efficient with MMAP than it is with WiredTiger. Furthermore the BulkWriteOperation does not make efficiency use of Wired Tiger's concurrency. The insert performance can be gained back by sharing the load across multiple threads. E.g. Instead of one BulkWrite operation of 1 million documents, create threads to perform 10 BulkWrites of 100,000 inserts.

To mitigate the poor performance of the BulkWriteOperation do as I stated above. The following test creates 1 thread (constituting a pass) that creates 10 threads to insert ~1,000,000 documents (100,000 per thread) and the loop is run 7 times. Both MMAP and Wired Tiger were compared. (Files: PC_MMAP_3Pass_10Packet_Threads.txt & PC_WiredTiger_3Pass_10Packet_Threads.txt)

In this case:

MMAP took each pass (executing 10 threads) approx 28 seconds to complete. Total insert time: 195 seconds.

Wired Tiger took each pass (executing 10 threads) approx 60 seconds to complete. Total insert time: 430 seconds.

To further improve. The next test creates 3 pass threads that create 10 threads each for the bulk insert of the ~1,000,000 documents.  (Files: PC_MMAP_1Pass_10Packet_Threads.txt & PC_WiredTiger_1Pass_10Packet_Threads.txt)

MMAP took on avg 81 seconds per pass, but keep in mind 3 pass threads are running at once. This is approx. the same performance as one thread as seen by the total insert time of 214 seconds.

Wired Tiger took on avg 60 seconds. Total insert time 180 seconds.

Additional info:

A previous test that I performed was to create 3 threads each that inserted ~1,000,000 million documents via the bulk write operation. Before MMAP's performance degrades you can see that the insert performance is still 3 times faster than Wired Tiger. (Files: 2. PC_WiredTiger_MultiThread_Insert.txt & 3. PC_MMAP_MultiThread_Insert.txt).

Anyone have any thoughts on these results?

I'm also curious if this issue is resolved with the latest Mongo Java Driver 3.0 series.

[MARK CALLAGHAN]

If you start mongod with --master then it will use (write, manage) an oplog on the master 

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/595aab3b-1428-4e2f-8513-5660a5c08559%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mark Callaghan
mdca...@gmail.com

[Asya Kamsky]

I expect WiredTiger operations to be higher latency so a single threaded test should always be won by MMAP unless you create a scenario where your IOPS are extremely limited. 

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/595aab3b-1428-4e2f-8513-5660a5c08559%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers

[Jeff Olmstead]

Can you elaborate on the differences between the processes?

Essentially it's not the fact that MMAP is out performing it on some scale that bothers me. It's that MMAP is out performing Wired Tiger by 600 percent for a BulkWriteOperation inserting 1,000,000 documents.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/595aab3b-1428-4e2f-8513-5660a5c08559%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.