Combining Replica Sets and Shards to a single system to reduce server count?

[Colin Davis]

I'm working on a project where we use quite a few mongo servers to
store data that's spread across various disks.

Right now, we're running shards of 2 replica set servers each. If we
had (for example) 5 shards, this gives us 10 total servers.


This is very analogous to a RAID 0+1 setup. We have 1 redundant server
for each active server. It works, and it's reliable, but it's less
than ideal.

It would be very useful to useful to us to be able to distribute the
redundancy across multiple servers, more like a RAID5 setup.

This would allow us to run, say, 7-8 total servers, rather than 10,
and distribute the redundancy across the shards. Each shard would have
it's own data, plus the a little bit of the data from other shards,
such that the total system could recover even if shards were to go
missing entirely.

When scaling to large numbers of Mongo servers, this system would save
a dramatic amount of money.

I understand that this isn't the way replica sets and shards work now,
but this sort of change would be very useful
Is there any chance that we could see such a combination going
forward, perhaps post 1.9?

[Bernie Hackett]

You can run multiple mongod processes on one machine. In your example
you could have a setup like this:

Machine 0:
replica0/node0 replica1/node1

Machine 1:
replica0/node1 replica1/node0

You also need one arbiter for each of these replica sets or auto-
failover won't work. You could use a third machine for that. They are
really lightweight processes so they could be on your application
server.

The same idea applies to config servers for sharding. The basic idea
is that you want to have the mongods distributed in such a way that a
loss of an individual server doesn't take the database offline.

The thing to worry about with this setup is that the multiple mongod
processes are sharing memory and disk so proper sizing becomes more
important.

[Colin Davis]

Thanks, Bernie, that's a good suggestion, but that's not really
reducing the number of servers that we're running, it's just combining
two of them onto single HW.
Since doing that would take just as much in terms of RAM/CPU as doing
them on separate HW, it seems functionally equivalent to just running
fewer shards.

I'm not a mongo-dev, but it would seem like that doesn't really solve
the problem, it just masks it.
To take that answer to the next step, I could reduce server count by
running multiple shards on a single disk as well. ;)

We're still in RAID 0+1 land.

The RAID5 system would store a parity block, that allows the other
blocks to be reconstructed, using the remaining data, and XOR.

While I'm sure you know how RAID works, it might be helpful to quickly
review the exact mechanism-

http://www.scottklarr.com/topic/23/how-raid-5-really-works/

So in this system, with 5 shards, we'd use six servers.
Each shard would store 1/5th of the data, plus a parity block.

When any one of the systems went down, we could reconstruct it, by
using the data remaining on the other 5.
You can increase redundancy, by increasing the number of parity
blocks.
RAID6 uses the same system, but two parity blocks, to increase the
overall reliability of the system, and ensure it can handle two
failures at the same time.

If Mongo were to support such a system, companies could deploy
dramatically fewer servers, while maintaining a very high level of
reliability and failover.

[Bernie Hackett]

Ah, I misunderstood how you were using the word 'server'. Questions of
this sort usually are about how to limit cost by reducing the number
of physical machines.

MongoDB doesn't currently work the way you are describing it. You
could open an enhancement request in Jira but this would be a
fundamental change to the way sharding and replication currently
work.

[Colin Davis]

I apologize for my lack of clarity- I appreciate you at least hearing
me out.
I understand that this might not be possible in the near term, but
would help us quite a bit as we continue to scale.

I've opened https://jira.mongodb.org/browse/SERVER-3088

-Colin