Re: Restarting Secondary Stepped the Primary Down to Secondary

[Shaun]

Hi Kali,

Were you experiencing performance issues on your secondary?  Since mongodump is reading every document in your database, that means mongodb must load everything into memory at some point.  Increasing resident memory usage can happen as part of normal operation as your system warms up.  Are you normally working with only a small subset of documents on your primary?

For your second question, if the secondary goes down and you still have two nodes up, they should have enough votes to reelect the primary even if an election is triggered.  Have you made any changes to the number of votes each node has?

Thanks,
~Shaun

On Thursday, October 4, 2012 7:20:07 AM UTC-7, kali wrote:

Hi All,

We are running 3 server replica set(one primary, one secondary and an arbiter) on AWS. Primary & Secondary are EC2 extra large instances and arbiter is a small instance.

Our current server version is 2.0.7 and planning to upgrade to 2.2 soon. We are running with journalling enabled.

We scheduled Mongodump to run periodically on Secondary for daily backups.

I'm little baffled to see the resident memory on the secondary grows continuously whereas primary is almost constant. So what could be the reason behind this behavior? Is Mongodump loads the entire working set into memory during backups?! I also checked and made sure there were no memory leaks(comparing the 2*memory mapped with virtual).

Also, we restarted secondary when it started eating up lot of memory. This eventually caused the primary to step down. Ideally, believing this shouldn't be the behavior?

Thanks in advance.

[myCloudWatcher]

If the Majority of the array is "stable" a Primary can be elected.

Watch the mongoDB logs on all servers for "glitches" in  the network.  If you have these glitches going on the Majority may "flicker" at times to a lower number of votes.

When I see these UN-expected flips in role for the Primary they show up with a fill explanation in the logs.  Most of the time they are due to Network issues.

If you have a Cluster that has no votes that "select" one server over another as the Primary ( servers have equal votes ) they may flip flop over time due to these effects also.  Watch the logs for these Flips.  They have little effect on the clients of Mongo.

I tend to "want" to elect a server have my "preferred" Primary.  Even when the servers are equal in many ways.  The reason for this is Alerts.  I alert the Primary in special ways to help understand load.


Please call any time:
Cell:      916-202-1600
Skype:  EdwardMGoldberg

Edward M. Goldberg
http://myCloudWatcher.com/
e.m.g.

[Shaun]

Kali,

As long as you have journaling enabled, you can do any kind of filesystem snapshot.  http://www.mongodb.org/display/DOCS/Backups gives a good description of different backup strategies, and talks about backups with EBS.

You can also use mongodump directly on your data files with the --dbpath option, but only if your db instance is not running.  See http://docs.mongodb.org/manual/administration/backups/#database-dump-with-mongodump

For your second question, you can see all the situations that should trigger an election here: http://docs.mongodb.org/manual/core/replication-internals/#replica-set-election-internals

If your case doesn't fit any of these, and you can describe how to reproduce it, then I can file a bug report on this.

Thanks!
~Shaun

On Monday, October 22, 2012 6:08:18 AM UTC-7, kali wrote:

Shaun,

Thanks for the response. Yes, as you said, we are dealing with subset of record in primary and at any point of time, our working set on the memory won't even be close to the entire data stored in Mongo. Is there any other way we can take backups without letting the mongodump to increase the resident memory? I'm little familiar with ebs disk snapshots. Would that work well in RAID 10?

From your response to my second question, I assume that there will be an election if ever a node stepped down(even network outages would trigger an election regardless of whether the node went down is primary or not?!). 'm trying to understand why there is a need for election if the primary is still alive?

Thanks.