MongoDB and Mesos

[Madhu Jahagirdar]

Can mongodb be run on the mesos framework ?

[Rob Moore]

Short answer: Yes.

I have, just this past couple of weeks, been setting up a set of MongoDB clusters to run on Mesos/Marathos/Chronos.

For replica sets and standalone instances you just need some careful scripting to get the instances started and, for the replica sets, add/remove the members as mesos starts stops members.  Nothing too hard but takes some planning to work through all of the scenarios. I have still not got the recovery from a total outage working by re-populating the replica set from the latest backup. The backups are done via Chronos.

For a sharded cluster I am using the On-prem version of MongoDB Operations (aka, MMS). For that I start/deploy the automator agents, monitoring agent, Operations web services, and the backing replica sets for MMS via marathon. I can then create my sharded clusters via the Operations web interface.

For operations web servers you will want some kind of proxy since mesos can start them anywhere and there is no easy way to update the agent with the new location(s). We are using a hardware load balancer (because we have one) but you can use HAProxy with something like Bamboo just as effectively.

In theory you should be able to directly start the configuration servers and monogs servers from mesos/marathon but you need to either lock the configuration servers to nodes or figure out a way to sync the new configuration server from an old member (manually) and then restart all of the mongos servers with the new configuration server list (and remember the order of the configuration servers matters).

HTH,

Rob.

[Jonathan Doklovic]

For replica sets and standalone instances you just need some careful scripting to get the instances started and, for the replica sets, add/remove the members as mesos starts stops members.  Nothing too hard but takes some planning to work through all of the scenarios.

Do you have any concrete examples/scripts for this? This is exactly what I'm trying to do but could use some help with adding th ereplicas based on the marathon assigned information.

Thanks,

- Jonathan

[Rob Moore]

Sorry no.  The script was not that long so I will try and get it printed out and copied over. For now here is an outline of what it did (from memory).

The general strategy I ended up using was to pull the expected members from the Marathon API, sort them and then use that list to:

0) Start a script in the background to seed the replica set.

1) Exec the mongod process (without --fork or putting it in the background so mesos tracks the mongod process).

For the forked script:

0) Wait for the socket to open for the member we are starting.

1) Iterate through the list looking for a primary.

1.a) If a primary is found add all of the other members to the primary in sorted order.

1.b) If all of the adds worked: goto 4. If not goto 1.

2) If no primary is found look for a secondary.

2.a) For each secondary we find that thinks it knows who the primary is try and add the members to that primary.

2.b) If all of the adds worked: goto 4. If not goto 1.

3) If we did not find a valid primary then we probably dropped quorum. For my system it is more important to get the replica set running with some data than to find the latest data or an optimal set.

3.a) Find the first node. If it thinks it is a secondary (was a member before then) force a configuration change where it is the only member. (This could be made smarter to find the set of running nodes and try to figure out which has the most up to date data.)

3.b) Goto 1.

4) Determine the members of the replica set that should be removed based on the list in Marathon's API and from the primary. Remove any members from the replica set that should not be there.

Other things I remember:

Make sure your marathon settings are set so that only one member of the replica set is restarted at a time and that a new member is added before the old member is removed.

Also I wrote a custom script health check to check if the member is a primary or secondary before declaring the member "alive" that ensures that the members are fully sync'd before restarting the next member. I'll grab that script too.

Rob.