Excessively slow queries over the network

[mi...@zeroent.net]

Hey all,

Background - I'm supporting a group of developers writing an application that stores results to a mongo database.  The typical deployment has been a small mongo instance on the same VM as the application.  The amount of data necessitated moving to a mongodb cluster (for testing, 1 config server, 2 shards, and a router).  After trying both the cluster setup and just a separate (stand alone) mongo instance on a different VM, we're seeing very poor performance.  In short, as soon as we see move the mongo instance off the same vm/OS instance as the application (and thus accessing mongo over the network), performance plummets.

Details on the workload

     We ingest our data in 100MB chunks.  Each chunk gets broken up into records that are in the order of 1-30KB each, which then get written individually until the 100MB chunk is done processing.  As per one of my developers: "We are using Java to insert records, one at a time -- specifically MongoRepository.save(Object). We're using spring-boot-starter-data-mongodb 1.4.2.RELEASE. MongoDB version 3.2.8."

Expected performance:

     When we run a 100MB chunk on a local mongo instance, it takes 15-20 seconds to ingest all of the records.  This is roughly what we were expecting to get when using a mongo instance (standalone or clustered) on the network.

Actual performance:

     Processing a 100MB chunk takes on the order of 65-80 seconds (same dataset) when sending to a mongo system on the network (same subnet).

Resource utilization:

  I've checked the usual suspects (cpu, memory, disk i/o, network throughput) and none of them show more than 20% utilization.

What we (think we) have eliminated:

• limited resources - tested on local vmware cluster (20Gbps of network I/O, not even seeing 20 mb of throughput from application to mongo), increased standalone mongodb resrouces to 4 core/8GB of memory, no effect
• Issues in local environment - tested on AWS with RHEL t2.medium nodes, no performance difference

Has anybody run into this before?  Are there any thoughts on what might cause this?  

Much obliged,

Mike

[Rhys Campbell]

Network latency. Batch your inserts...

https://docs.mongodb.com/v3.2/reference/method/db.collection.insertMany/