[mongodb-user] db error [not master]

[Zsombor]

Hello,

I've tried to run mongodb in a replica pair configuration, and tried
to shut down one of the instances, manually to test automatic
failover. It seems that after the stop, the java client report the
following exceptions:
java.io.IOException: connection to server closed unexpectedly
at com.mongodb.DBPort._read(DBPort.java:221)
at com.mongodb.DBPort.go(DBPort.java:91)
at com.mongodb.DBPort.call(DBPort.java:58)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:158)
at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:297)
at com.mongodb.DBCollection.findOne(DBCollection.java:148)
at com.mongodb.DBCollection.findOne(DBCollection.java:135)

next time:
java.lang.IllegalStateException
at com.mongodb.DBMessage.prepare(DBMessage.java:85)
at com.mongodb.DBPort.go(DBPort.java:73)
at com.mongodb.DBPort.call(DBPort.java:58)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:158)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:180)
at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:297)
at com.mongodb.DBCollection.findOne(DBCollection.java:148)
at com.mongodb.DBCollection.findOne(DBCollection.java:135)

after it throws for a couple of seconds the following exception:
java.lang.RuntimeException: db error [not master]
at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:307)
at com.mongodb.DBCollection.findOne(DBCollection.java:148)
at com.mongodb.DBCollection.findOne(DBCollection.java:135)

I understand the first exceptions, and I think it's reasonable to
throw that. The second probably some client side error, related to
management of the connections - probably it tries to use the
previously closed connection ? The third type of exception is
mysterious for me : probably the second server report that error, but
why ? I thought that in this replica pair configuration, any server
can be queried, and updated any time, so why the problem to 'not being
master' ? After a couple of seconds, and 10-20 queries everything is
normalized.
Can anybody explains what happens in this scenario ? It would be
nice, if I can somehow detect this kind of problem in the client
program, to wait a half a second or retry automatically a couple of
time, or even better : it would be nice, if the connection manager
part is a little bit smarter. But because of the generic exceptions I
cant reliably detect this circumstances.

BR,
Zsombor

p.s. I really enjoy mongodb so far :)

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Eliot Horowitz]

Can you try with the new 2.0-rc0 java driver?
Should behave better - and if not will be able to debug easier.

[Zsombor]

On May 7, 9:33 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Can you try with the new 2.0-rc0 java driver?
> Should behave better - and if not will be able to debug easier.
>
>
>

I've tested with it, it's behaves similarly. I get the following
exception when I stop one of the server:

paired mode, switching master b/c of: java.net.SocketException:
Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:185)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at com.mongodb.Response.<init>(Response.java:38)
at com.mongodb.DBPort.go(DBPort.java:80)
at com.mongodb.DBPort.call(DBPort.java:57)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:169)
at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:277)
at com.mongodb.DBCollection.findOne(DBCollection.java:151)
at com.mongodb.DBCollection.findOne(DBCollection.java:138)

Sometimes I get this:

java.lang.IndexOutOfBoundsException
at java.io.BufferedInputStream.read(BufferedInputStream.java:327)
at com.mongodb.Response.<init>(Response.java:38)
at com.mongodb.DBPort.go(DBPort.java:80)
at com.mongodb.DBPort.call(DBPort.java:57)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:169)
at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:277)
at com.mongodb.DBCollection.findOne(DBCollection.java:151)
at com.mongodb.DBCollection.findOne(DBCollection.java:138)

After I get this:
paired mode, switching master b/c of: java.net.SocketException: Broken
pipe
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:
109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at org.bson.io.PoolOutputBuffer.pipe(PoolOutputBuffer.java:110)
at com.mongodb.OutMessage.pipe(OutMessage.java:128)
at com.mongodb.DBPort.go(DBPort.java:72)
at com.mongodb.DBPort.call(DBPort.java:57)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:169)
at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:277)
at com.mongodb.DBCollection.findOne(DBCollection.java:151)
at com.mongodb.DBCollection.findOne(DBCollection.java:138)

And the following comes for a couple seconds:
com.mongodb.MongoException: not talking to master and retries used up
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:178)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:180)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:189)
at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:277)
at com.mongodb.DBCollection.findOne(DBCollection.java:151)
at com.mongodb.DBCollection.findOne(DBCollection.java:138)

I can understand that in this case, for a couple of seconds, the
system is not available, but the exceptions are still strange.

BR,
Zsombor

[Zsombor]

Sometimes I get the following exception on startup :

May 10, 2010 2:07:48 PM com.mongodb.DBTCPConnector _pickInitial
SEVERE: switched to: localhost:25002 but isn't master
May 10, 2010 2:07:48 PM com.mongodb.DBTCPConnector _pickInitial
SEVERE: switched to: localhost:25001 but isn't master
May 10, 2010 2:07:48 PM com.mongodb.DBTCPConnector _pickInitial
SEVERE: can't pick initial master, using random one
com.mongodb.MongoException: can't find master
at com.mongodb.DBTCPConnector._pickInitial(DBTCPConnector.java:340)
at com.mongodb.Mongo.<init>(Mongo.java:159)
at com.mongodb.Mongo.<init>(Mongo.java:145)

[Zsombor]

Is there any way, I can help fixing this issues ? Or is it solved
yet?

[Eliot Horowitz]

Can you try building your own jar from source now?
Should be fixed.

[Zsombor]

On May 14, 7:39 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Can you try building your own jar from source now?
> Should be fixed.
>
>

I've tested it again, now I get the following exceptions after I shut
down the master server :

paired mode, switching master b/c of: java.net.SocketException: Broken
pipe
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:
109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at org.bson.io.PoolOutputBuffer.pipe(PoolOutputBuffer.java:110)
at com.mongodb.OutMessage.pipe(OutMessage.java:128)

at com.mongodb.DBPort.go(DBPort.java:71)

at com.mongodb.DBPort.call(DBPort.java:57)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:169)
at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:277)
at com.mongodb.DBCollection.findOne(DBCollection.java:151)
at com.mongodb.DBCollection.findOne(DBCollection.java:138)

After :

java.io.IOException: couldn't connect to [localhost/127.0.0.1:25002]
bc:java.net.ConnectException: Connection refused
at com.mongodb.DBPort._open(DBPort.java:113)
at com.mongodb.DBPort.go(DBPort.java:68)

at com.mongodb.DBPort.call(DBPort.java:57)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:169)

at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:192)

at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:277)
at com.mongodb.DBCollection.findOne(DBCollection.java:151)
at com.mongodb.DBCollection.findOne(DBCollection.java:138)

And finally:

paired mode, switching master b/c of: java.io.IOException: couldn't
connect to [fricipapa/127.0.0.1:25002] bc:java.net.ConnectException:
Connection refused
java.lang.RuntimeException: trying to put something back in the pool
that's already there
at com.mongodb.util.SimplePool.done(SimplePool.java:87)
at com.mongodb.DBTCPConnector$MyPort.done(DBTCPConnector.java:261)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:176)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:192)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:192)

at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:277)
at com.mongodb.DBCollection.findOne(DBCollection.java:151)
at com.mongodb.DBCollection.findOne(DBCollection.java:138)

For a couple of seconds.
The code wrotes the following to the stdout : "_internalStack < 0 :
-1" probably it's related to the issue.

So the symptoms stays the same : after the master is lost, for a
couple of seconds the system unable to do anything.

[Eliot Horowitz]

can you send the code you're testing with?

[Zsombor]

On May 17, 6:58 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> can you send the code you're testing with?
>
>

I've just filled one collection with 100000 objects, and after:
DB db = mongo.getDB("testdb");
for (int i =0;i<1000;i++) {
try {
DBObject obj =
db.getCollection("productInfo").findOne(createRandomId());
System.out.println("obj:"+obj);
Thread.sleep(200);
} catch (Exception e) {

[Eliot Horowitz]

I've cleaned up some of the exceptions.

But I think the general situation is correct.

On a replica pair failure, it takes 2 seconds for the slave to decide
the master is really down before taking over.