Nearest ReadPreference throughput degradation

[Wenting Wang]

Hi everyone,

I'm using YCSB to test ReadPreference of MongoDB under read heavy workload(95%read, 5%write). From experiment results, it is very clear that nearest read preference improves read latency a lot. But meanwhile, the write latency goes up and the overall throughput decreases. Is there any reason why nearest hurts write latency and overall throughput?

My experiment settings: 5M/10M records, 3 ReplicaSets/6ReplicaSets, read-heavy workload and Zipf distribution in YCSB, 1 mongos, 800/1600 client threads. I attached a subset of the experiment results.

Thanks,

Wenting.

[William Berkeley]

Hey. Could you give us more information about your setup? How are your replica sets deployed in order to test nearest read preference?

-Will

[Wenting Wang]

Hi Will,

In my setup, each replica set has 1 primary and 2 secondary servers. For the experiments in the plots above, I have 9 mongod (3 replica sets x (1 primary+ 2 secondary)) + 1 mongos + 1 YCSB workload generator = 11 servers. Each config server stays with one primary servers.  All these servers are deployed in VMs which are all hosted on one physical machine. The cluster is enabled with hashed-sharding, journaling and smallfiles. Then in YCSB, I specifed read preference in each read operation and mongostat showed that queries were distributed differently under different read preference.

Thanks,
Wenting.

[William Berkeley]

Unless you have built-in some kind of realistic network delay, testing nearest read-preferences on one physical machine is not very meaningful (see the member selection process for details). What version of MongoDB are you using? Also, how are these VM's configured or deployed? They could be competing for resources since they share the same physical hardware. The single mongos could be the bottleneck.

You may also be interested in this recent blog post about benchmarking MongoDB with YCSB.

-Will

[Wenting Wang]

Thanks a lot, Will. I'll read that blog.

My VMs are configured 4vCPUs, 3GB, 16+ 50GB Disk, and the host is 48 CPUs x 2.2 GHz and 256GB memory on ESXi 5.5. The MongoDB version is 2.6.1. There is no network delay, so in my particular case, nearest is used as "random" read preference. The CPU usage for mongod is no more than 30%. Mongos CPU usage is high and close to 100%. But if mongos is the bottleneck for nearest, shouldn't it slow down both read and write latency? There may be resource contention, but how does nearest or secondary read preference make it worse? My understanding for nearest read preference is it should improve read performance(and it did) and have little influence on write performance. Is this correct? Or anything I'm missing?

Thanks,
Wenting.

[Asya Kamsky]

This is an interesting set-up that shows how a bottleneck might manifest in surprising ways.

Your mongod and mongos were VMs running on this physical box - where was YCSB running?   Your charts show testing with 200, 800 and 1600 "clients" - do you mean threads?  How many YCSB processes were running, and how many threads were utilized?

The other important question is which YCSB fork did you use and what was your workload configuration for the runs?

Asya