Adding member to replica set when Java Driver is connected causing problems

62 views

Skip to first unread message

Stephan

unread,

Mar 1, 2013, 4:06:10 AM3/1/13

to mongod...@googlegroups.com

Hi,

we just wanted to add additional nodes to our replica set. And there i found some strange behavior:

adding a node to the set (either with rs.reconfig() and changing the config, or by rs.add()) works fine, the new node is then in state "STARTUP2"
in the same instance, the mongo driver throws exceptions:

com.mongodb.MongoException: not talking to master and retries used up

at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:314)

at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:316)

at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)

at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)

....

Several Write request (obviously) are being sent to some secondary node (can't tell which one).
Some requests work fine though.
Setting priority to 0 alone does not avoid this problem. Our master has prioity set to 10.
the error does not occur, when the new node is set 'hidden:true'
Adding the node hidden and then setting back the hidden to false, does trigger the exception as well.
the error does not occur, when switching hidden to true when the new node reaches state "SECONDARY" (i think - I switched hidden:false after one day waiting and it worked)

This is really strange, is this a bug in the Java Driver (we're using 2.10.1 - mongodb Version 2.2.3).

Then there is an additional question: when starting up the connection to the mongo, you need to specify at least more than one host. Why is that so? the driver could easily read the config from one of the nodes?

What if you specify 2 nodes, but your replicaset has 4? Does the driver use all 4 or only 2? Or are those settings only used for startup?

Kind Regards,

Stephan

Asya Kamsky

unread,

Mar 1, 2013, 4:20:36 AM3/1/13

to mongod...@googlegroups.com

Hi Stephan:

It would be helpful if you would look in the logs for your replica nodes during the time that this happened and see what happened during the election.

Whenever you reconfigure your replica set, this will cause all connections to the primary to be severed and when they disconnect a new primary may not have yet been elected. If you check the logs you may be able to see why the election may be taking longer in some configurations of your replica set - is it possible that you are adding a node that's poorly connected (i.e. ping time to it is high or connectivity is intermittent?)

Multiple "seed" nodes are given in the startup to the driver in case the application/driver is restarted at the moment when not all the members are up (for example, you have three nodes, you give it one seed and during restart it just happens that the one seed it has is down for maintenance - result is application cannot connect to MongoDB).

Asya

Stephan

unread,

Mar 1, 2013, 6:30:38 AM3/1/13

to mongod...@googlegroups.com

I can reproduce the error locally. I installed locally a replicaset with 2 nodes + one Arbiter (localhost Port 27017-27019).

As soon as I added the 4th node (Port 27020), I get those exceptions from the Java Driver;:

[exec] com.mongodb.MongoException: not talking to master and retries used up

[exec] at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:314)

[exec] at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:316)

[exec] at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)

[exec] at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)

[exec] at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:295)

[exec] at com.mongodb.DBCursor._check(DBCursor.java:368)

[exec] at com.mongodb.DBCursor._hasNext(DBCursor.java:459)

[exec] at com.mongodb.DBCursor.hasNext(DBCursor.java:484)

.....

A lot of those - for some time (local DB is quite small, so syncing the new node only takes a couple of minutes - after that time, the errors are gone!)

This is the part of the mongod-log from the primary node:

Fri Mar 1 12:22:30 [conn57] replSet replSetReconfig config object parses ok, 4 members specified

Fri Mar 1 12:22:30 [conn57] replSet replSetReconfig [2]

Fri Mar 1 12:22:30 [conn57] replSet info saving a newer config version to local.system.replset

Fri Mar 1 12:22:30 [conn57] replSet saveConfigLocally done

Fri Mar 1 12:22:30 [conn57] replSet info : additive change to configuration

Fri Mar 1 12:22:30 [conn57] replSet replSetReconfig new config saved locally

Fri Mar 1 12:22:30 [rsHealthPoll] replSet member localhost:27020 is up

Fri Mar 1 12:22:39 [conn93] end connection 127.0.0.1:64209 (31 connections now open)

Fri Mar 1 12:22:39 [initandlisten] connection accepted from 127.0.0.1:64236 #95 (32 connections now open)

Fri Mar 1 12:22:39 [conn94] end connection 127.0.0.1:64210 (31 connections now open)

Fri Mar 1 12:22:39 [initandlisten] connection accepted from 127.0.0.1:64237 #96 (32 connections now open)

Fri Mar 1 12:22:40 [initandlisten] connection accepted from 127.0.0.1:64255 #97 (33 connections now open)

Fri Mar 1 12:22:42 [conn97] end connection 127.0.0.1:64255 (32 connections now open)

Fri Mar 1 12:22:42 [initandlisten] connection accepted from 127.0.0.1:64258 #98 (34 connections now open)

Fri Mar 1 12:22:42 [rsHealthPoll] replSet member localhost:27020 is now in state STARTUP2

Fri Mar 1 12:22:57 [initandlisten] connection accepted from 127.0.0.1:64268 #99 (34 connections now open)

Fri Mar 1 12:22:57 [initandlisten] connection accepted from 127.0.0.1:64269 #100 (35 connections now open)

Fri Mar 1 12:22:58 [conn100] end connection 127.0.0.1:64269 (34 connections now open)

Fri Mar 1 12:22:58 [initandlisten] connection accepted from 127.0.0.1:64270 #101 (35 connections now open)

Fri Mar 1 12:22:58 [conn101] end connection 127.0.0.1:64270 (34 connections now open)

Fri Mar 1 12:22:58 [initandlisten] connection accepted from 127.0.0.1:64271 #102 (35 connections now open)

Fri Mar 1 12:22:59 [conn102] end connection 127.0.0.1:64271 (34 connections now open)

Fri Mar 1 12:22:59 [initandlisten] connection accepted from 127.0.0.1:64272 #103 (35 connections now open)

Fri Mar 1 12:22:59 [conn103] end connection 127.0.0.1:64272 (34 connections now open)

Fri Mar 1 12:22:59 [initandlisten] connection accepted from 127.0.0.1:64273 #104 (35 connections now open)

Fri Mar 1 12:22:59 [conn104] end connection 127.0.0.1:64273 (34 connections now open)

Fri Mar 1 12:22:59 [initandlisten] connection accepted from 127.0.0.1:64274 #105 (35 connections now open)

Fri Mar 1 12:22:59 [initandlisten] connection accepted from 127.0.0.1:64275 #106 (36 connections now open)

Fri Mar 1 12:22:59 [conn105] end connection 127.0.0.1:64274 (35 connections now open)

Fri Mar 1 12:22:59 [initandlisten] connection accepted from 127.0.0.1:64276 #107 (36 connections now open)

Fri Mar 1 12:22:59 [conn107] end connection 127.0.0.1:64276 (35 connections now open)

Fri Mar 1 12:22:59 [initandlisten] connection accepted from 127.0.0.1:64277 #108 (36 connections now open)

Fri Mar 1 12:22:59 [conn108] end connection 127.0.0.1:64277 (35 connections now open)

Fri Mar 1 12:23:00 [conn99] end connection 127.0.0.1:64268 (34 connections now open)

Fri Mar 1 12:23:00 [rsHealthPoll] replSet member localhost:27020 is now in state RECOVERING

Fri Mar 1 12:23:00 [initandlisten] connection accepted from 127.0.0.1:64278 #109 (35 connections now open)

Fri Mar 1 12:23:02 [rsHealthPoll] replSet member localhost:27020 is now in state SECONDARY

Fri Mar 1 12:23:09 [conn95] end connection 127.0.0.1:64236 (34 connections now open)

the "new" node says:

Fri Mar 1 12:22:39 [conn12] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.config_element query:{ name: "after_campaign_timeout_seconds" }

Fri Mar 1 12:22:39 [conn12] problem detected during query over holidayinsider.config_element : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn12] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.campaign query:{ query: { $and: [ { timestamp: { $gt: 1362136560000 } }, { timestamp: { $lt: 1362137100000 } } ] }, orderby: { timestamp: -1 } }

Fri Mar 1 12:22:39 [conn12] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn12] problem detected during query over holidayinsider.campaign : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn12] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.config_element query:{ name: "after_campaign_timeout_seconds" }

Fri Mar 1 12:22:39 [conn12] problem detected during query over holidayinsider.config_element : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn12] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn12] problem detected during query over holidayinsider.campaign : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn12] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn12] problem detected during query over holidayinsider.campaign : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [initandlisten] connection accepted from 127.0.0.1:64250 #13 (10 connections now open)

Fri Mar 1 12:22:39 [conn13] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.config_element query:{ name: "after_campaign_timeout_seconds" }

Fri Mar 1 12:22:39 [conn13] problem detected during query over holidayinsider.config_element : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn13] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.config_element query:{ name: "after_campaign_timeout_seconds" }

Fri Mar 1 12:22:39 [conn13] problem detected during query over holidayinsider.config_element : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [initandlisten] connection accepted from 127.0.0.1:64251 #14 (11 connections now open)

Fri Mar 1 12:22:39 [conn13] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.config_element query:{ name: "after_campaign_timeout_seconds" }

Fri Mar 1 12:22:39 [conn14] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.hi_user query:{ _id: null }

Fri Mar 1 12:22:39 [conn13] problem detected during query over holidayinsider.config_element : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn12] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.hi_user query:{ _id: null }

Fri Mar 1 12:22:39 [conn14] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn12] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn14] problem detected during query over holidayinsider.hi_user : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn12] problem detected during query over holidayinsider.hi_user : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn14] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.config_element query:{ name: "before_campaign_timeout_seconds" }

Fri Mar 1 12:22:39 [conn14] problem detected during query over holidayinsider.config_element : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [initandlisten] connection accepted from 127.0.0.1:64252 #15 (12 connections now open)

Fri Mar 1 12:22:39 [conn15] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.config_element query:{ name: "after_campaign_timeout_seconds" }

Fri Mar 1 12:22:39 [conn14] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.config_element query:{ name: "after_campaign_timeout_seconds" }

Fri Mar 1 12:22:39 [conn15] problem detected during query over holidayinsider.config_element : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn14] problem detected during query over holidayinsider.config_element : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [initandlisten] connection accepted from 127.0.0.1:64253 #16 (13 connections now open)

Fri Mar 1 12:22:39 [conn16] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.campaign query:{ query: { $and: [ { timestamp: { $gt: 1362136560000 } }, { timestamp: { $lt: 1362137100000 } } ] }, orderby: { timestamp: -1 } }

Fri Mar 1 12:22:39 [conn16] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn16] problem detected during query over holidayinsider.campaign : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn14] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.campaign query:{ query: { $and: [ { timestamp: { $gt: 1362136560000 } }, { timestamp: { $lt: 1362137100000 } } ] }, orderby: { timestamp: -1 } }

Fri Mar 1 12:22:39 [conn14] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn14] problem detected during query over holidayinsider.campaign : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn16] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.hi_user query:{ _id: null }

Fri Mar 1 12:22:39 [conn14] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn16] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn14] problem detected during query over holidayinsider.campaign : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn15] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.hi_user query:{ _id: null }

Fri Mar 1 12:22:39 [conn16] problem detected during query over holidayinsider.hi_user : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn15] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn15] problem detected during query over holidayinsider.hi_user : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn14] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn14] problem detected during query over holidayinsider.campaign : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn16] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.config_element query:{ name: "before_campaign_timeout_seconds" }

Fri Mar 1 12:22:39 [conn16] problem detected during query over holidayinsider.config_element : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn16] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn16] problem detected during query over holidayinsider.campaign : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn16] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn16] problem detected during query over holidayinsider.campaign : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:39 [conn16] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.hi_user query:{ _id: null }

Fri Mar 1 12:22:39 [conn16] ntoskip:0 ntoreturn:-1

Fri Mar 1 12:22:39 [conn16] problem detected during query over holidayinsider.hi_user : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:40 [initandlisten] connection accepted from 127.0.0.1:64254 #17 (14 connections now open)

Fri Mar 1 12:22:40 [conn17] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:sessions.rest-sessions query:{}

Fri Mar 1 12:22:40 [conn17] problem detected during query over sessions.rest-sessions : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:40 [rsStart] trying to contact localhost:27017

Fri Mar 1 12:22:40 [rsStart] trying to contact localhost:27018

Fri Mar 1 12:22:40 [rsStart] trying to contact localhost:27019

Fri Mar 1 12:22:40 [rsStart] replSet I am localhost:27020

Fri Mar 1 12:22:40 [rsStart] replSet got config version 8 from a remote, saving locally

Fri Mar 1 12:22:40 [rsStart] replSet info saving a newer config version to local.system.replset

Fri Mar 1 12:22:40 [FileAllocator] allocating new datafile /Users/stephan/HiWeb/env/mongo/data4/local/local.ns, filling with zeroes...

Fri Mar 1 12:22:40 [FileAllocator] creating directory /Users/stephan/HiWeb/env/mongo/data4/local/_tmp

Fri Mar 1 12:22:40 [FileAllocator] done allocating datafile /Users/stephan/HiWeb/env/mongo/data4/local/local.ns, size: 16MB, took 0.021 secs

Fri Mar 1 12:22:40 [FileAllocator] allocating new datafile /Users/stephan/HiWeb/env/mongo/data4/local/local.0, filling with zeroes...

Fri Mar 1 12:22:40 [FileAllocator] done allocating datafile /Users/stephan/HiWeb/env/mongo/data4/local/local.0, size: 64MB, took 0.143 secs

Fri Mar 1 12:22:40 [FileAllocator] allocating new datafile /Users/stephan/HiWeb/env/mongo/data4/local/local.1, filling with zeroes...

Fri Mar 1 12:22:40 [rsStart] replSet saveConfigLocally done

Fri Mar 1 12:22:40 [rsStart] replSet STARTUP2

Fri Mar 1 12:22:40 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote

Fri Mar 1 12:22:40 [rsSync] ******

Fri Mar 1 12:22:40 [rsSync] creating replication oplog of size: 183MB...

Fri Mar 1 12:22:41 [FileAllocator] done allocating datafile /Users/stephan/HiWeb/env/mongo/data4/local/local.1, size: 128MB, took 0.301 secs

Fri Mar 1 12:22:41 [FileAllocator] allocating new datafile /Users/stephan/HiWeb/env/mongo/data4/local/local.2, filling with zeroes...

Fri Mar 1 12:22:41 [FileAllocator] done allocating datafile /Users/stephan/HiWeb/env/mongo/data4/local/local.2, size: 256MB, took 0.67 secs

Fri Mar 1 12:22:41 [rsSync] ******

Fri Mar 1 12:22:41 [rsSync] replSet initial sync pending

Fri Mar 1 12:22:41 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync

Fri Mar 1 12:22:42 [rsHealthPoll] replSet member localhost:27018 is up

Fri Mar 1 12:22:42 [rsHealthPoll] replSet member localhost:27018 is now in state SECONDARY

Fri Mar 1 12:22:42 [rsHealthPoll] replSet member localhost:27019 is up

Fri Mar 1 12:22:42 [rsHealthPoll] replSet member localhost:27019 is now in state ARBITER

Fri Mar 1 12:22:42 [rsHealthPoll] replSet member localhost:27017 is up

Fri Mar 1 12:22:42 [rsHealthPoll] replSet member localhost:27017 is now in state PRIMARY

Fri Mar 1 12:22:45 [initandlisten] connection accepted from 127.0.0.1:64259 #18 (15 connections now open)

Fri Mar 1 12:22:45 [conn18] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.msg query:{ query: { $or: [ { locked_by: "c58773f2-4451-4241-8f4f-3f8b14eaa919" }, { $and: [ { locked_by: "ALL" }, { processed_by: { $ne: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } }, { to: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } ] }, { $and: [ { locked_by: "ALL" }, { processed_by: { $ne: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } }, { to: null } ] } ] }, orderby: { timestamp: 1 } }

Fri Mar 1 12:22:45 [conn18] problem detected during query over holidayinsider.msg : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:48 [conn2] end connection 127.0.0.1:64217 (14 connections now open)

Fri Mar 1 12:22:48 [initandlisten] connection accepted from 127.0.0.1:64260 #19 (15 connections now open)

Fri Mar 1 12:22:49 [conn4] end connection 127.0.0.1:64219 (14 connections now open)

Fri Mar 1 12:22:49 [initandlisten] connection accepted from 127.0.0.1:64261 #20 (15 connections now open)

Fri Mar 1 12:22:49 [conn6] end connection 127.0.0.1:64221 (14 connections now open)

Fri Mar 1 12:22:49 [initandlisten] connection accepted from 127.0.0.1:64262 #21 (15 connections now open)

Fri Mar 1 12:22:51 [conn18] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.msg query:{ query: { $or: [ { locked_by: "c58773f2-4451-4241-8f4f-3f8b14eaa919" }, { $and: [ { locked_by: "ALL" }, { processed_by: { $ne: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } }, { to: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } ] }, { $and: [ { locked_by: "ALL" }, { processed_by: { $ne: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } }, { to: null } ] } ] }, orderby: { timestamp: 1 } }

Fri Mar 1 12:22:51 [conn18] problem detected during query over holidayinsider.msg : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:52 [initandlisten] connection accepted from 127.0.0.1:64264 #22 (16 connections now open)

Fri Mar 1 12:22:52 [conn22] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:sessions.rest-sessions query:{}

Fri Mar 1 12:22:52 [conn22] problem detected during query over sessions.rest-sessions : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:53 [conn18] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.msg query:{ query: { $or: [ { locked_by: "c58773f2-4451-4241-8f4f-3f8b14eaa919" }, { $and: [ { locked_by: "ALL" }, { processed_by: { $ne: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } }, { to: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } ] }, { $and: [ { locked_by: "ALL" }, { processed_by: { $ne: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } }, { to: null } ] } ] }, orderby: { timestamp: 1 } }

Fri Mar 1 12:22:53 [conn18] problem detected during query over holidayinsider.msg : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:56 [conn18] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.msg query:{ query: { $or: [ { locked_by: "c58773f2-4451-4241-8f4f-3f8b14eaa919" }, { $and: [ { locked_by: "ALL" }, { processed_by: { $ne: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } }, { to: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } ] }, { $and: [ { locked_by: "ALL" }, { processed_by: { $ne: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } }, { to: null } ] } ] }, orderby: { timestamp: 1 } }

Fri Mar 1 12:22:56 [conn18] problem detected during query over holidayinsider.msg : { $err: "not master or secondary; cannot currently read from this replSet member", code: 13436 }

Fri Mar 1 12:22:57 [rsSync] replSet initial sync pending

Fri Mar 1 12:22:57 [rsSync] replSet syncing to: localhost:27017

Fri Mar 1 12:22:57 [rsSync] build index local.me { _id: 1 }

Fri Mar 1 12:22:57 [rsSync] build index done. scanned 0 total records. 0 secs

Fri Mar 1 12:22:57 [rsSync] replSet initial sync drop all databases

Fri Mar 1 12:22:57 [rsSync] dropAllDatabasesExceptLocal 1

Fri Mar 1 12:22:57 [rsSync] replSet initial sync clone all databases

... (lots of syncing of DB successful messages)....

Fri Mar 1 12:22:58 [conn18] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:holidayinsider.msg query:{ query: { $or: [ { locked_by: "c58773f2-4451-4241-8f4f-3f8b14eaa919" }, { $and: [ { locked_by: "ALL" }, { processed_by: { $ne: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } }, { to: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } ] }, { $and: [ { locked_by: "ALL" }, { processed_by: { $ne: "c58773f2-4451-4241-8f4f-3f8b14eaa919" } }, { to: null } ] } ] }, orderby: { timestamp: 1 } }

Fri Mar 1 12:22:59 [rsSync] build index morphium_test.sequence { locked_by: 1 }

Fri Mar 1 12:22:59 [rsSync] build index done. scanned 20 total records. 0 secs

Fri Mar 1 12:22:59 [rsSync] replSet initial sync cloning indexes for : sessions

Fri Mar 1 12:22:59 [rsSync] build index sessions.sessions { lastmodified: 1 }

Fri Mar 1 12:22:59 [rsSync] build index done. scanned 1 total records. 0.027 secs

Fri Mar 1 12:22:59 [rsSync] oplog sync 3 of 3

Fri Mar 1 12:22:59 [rsSync] replSet initial sync finishing up

Fri Mar 1 12:22:59 [rsSync] replSet set minValid=51308f7f:8

Fri Mar 1 12:22:59 [rsSync] build index local.replset.minvalid { _id: 1 }

Fri Mar 1 12:22:59 [rsSync] build index done. scanned 0 total records. 0 secs

Fri Mar 1 12:23:00 [rsSync] replSet RECOVERING

Fri Mar 1 12:23:00 [rsSync] replSet initial sync done

Fri Mar 1 12:23:00 [FileAllocator] done allocating datafile /Users/stephan/HiWeb/env/mongo/data4/sessions/sessions.1, size: 128MB, took 0.305 secs

Fri Mar 1 12:23:00 [rsSyncNotifier] replset setting oplog notifier to localhost:27017

Fri Mar 1 12:23:02 [rsSync] replSet SECONDARY

This looks quite unspectacular - can't see the problem here...

Asya Kamsky

unread,

Mar 1, 2013, 11:37:10 AM3/1/13

to mongod...@googlegroups.com

What I see here is that you had no primary in this replica set for 12 seconds or so. How does that compare to the amount of time before errors stopped on the driver/application side?