Java driver stacktraces on ensureIndex

[Robert Kuhar]

I am new to mongo and have inherited a codebase and am working to get it up and working on my dev box.  I am running OS X Snow Leopard and have installed mongodb using the instructions from http://www.mongodb.org/display/DOCS/Quickstart+OS+X with the straight-up curl of the tarball and deploy out from there.

It starts with the following mong.conf

# Only accept local connections (doesn't work with DM MongoSourceImpl)

bind_ip = 127.0.0.1

port = 27017

rest = true

noauth = true

The JavaScript shell works fine as does all the simple tutorials with the Java driver, but when I try to run my application, the first call to ensureIndex stacktraces out the wazoo.  The code looks like...

    DB db = mongoDBManager.getDb(mongoDBName);

    DBCollection profiles = db.getCollection(mongoCollectionName);

    // index: {_id, ModelId}

    BasicDBObject index = new BasicDBObject();

    index.put("_id", 1);

    index.put(ProfileFieldName.ModelID.name(), 1);

    profiles.ensureIndex(index);

...the stacktrace looks like...

2012-05-23T15:16:01.231 [ERROR] o.a.t.i.s.T.RegistryStartup DSN: RID: - Construction of service RegistryStartup failed: Error invoking service contribution method htc.cs.service.device.ui.services.DeviceManagementModule.ensureMongoDBSetup(MongoDBManager, String, String): can't find a master

org.apache.tapestry5.ioc.internal.OperationException: Error invoking service contribution method htc.cs.service.device.ui.services.DeviceManagementModule.ensureMongoDBSetup(MongoDBManager, String, String): can't find a master

at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.logAndRethrow(OperationTrackerImpl.java:121) [tapestry-ioc-5.3.1.jar:na]

...

Caused by: com.mongodb.MongoException: can't find a master

at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:406) ~[mongo-java-driver-2.6.5.jar:na]

...

If I change the mongo.conf to NOT bind to 127.0.0.1 via bind_ip (I just comment it out and bounce mongodb), the application starts without issue.  I'm a dev box...I'm not running a cluster.  Why is my application looking for one?

What I'm really asking is how do I troubleshoot this issue?  Is the problem my server configuration or my configuration of the driver?

Thanks in advance for any insights you may give.

[Eliot Horowitz]

Can you send the code where you create the Mongo object?

> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb

[Robert Kuhar]

Sure thing...

    public MongoSourceImpl(

            Collection<ServerAddress> replicaSet,

            @Symbol("mongo.slave-ok") Boolean slaveOk,

            @Symbol("mongo.write-concern-fsync") Boolean fsync,

            @Symbol("mongo.write-concern-w-timeout") Integer wTimeout,

            @Symbol("mongo.write-concern-w") Integer w,

            @Symbol("mongo.connections-per-host") Integer connectionsPerHost,

            @Symbol("mongo.blocked-threads-per-connection") Integer blockedThreadsPerConnection,

            @Symbol("mongo.auto-connect-retry") Boolean autoConnectRetry,

            @Symbol("mongo.connect-timeout") Integer connectTimeout,

            @Symbol("mongo.max-wait-time") Integer maxWaitTime,

            @Symbol("mongo.socket-time-out") Integer socketTimeout,

            @Symbol("mongo.socket-keep-alive") Boolean socketKeepAlive

    ) {

        MongoOptions mongoOptions = new MongoOptions();

        mongoOptions.slaveOk = slaveOk;

        mongoOptions.fsync = fsync;

        mongoOptions.wtimeout = wTimeout;

        mongoOptions.w = w;

        mongoOptions.connectionsPerHost = connectionsPerHost;

        mongoOptions.threadsAllowedToBlockForConnectionMultiplier = blockedThreadsPerConnection;

        mongoOptions.autoConnectRetry = autoConnectRetry;

        mongoOptions.connectTimeout = connectTimeout;

        mongoOptions.maxWaitTime = maxWaitTime;

        mongoOptions.socketTimeout = socketTimeout;

        mongoOptions.socketKeepAlive = socketKeepAlive;

        mongo = new Mongo(new ArrayList<ServerAddress>(replicaSet), mongoOptions);

    }

...it is the case, on my dev box, that the replicaSet collection is empty at the time of new Mongo.

Bob

> mongodb-user+unsubscribe@googlegroups.com

[Robert Kuhar]

I lied.  At the time I pass through the MongoSourceImpl constructor, the Collection<ServerAddress> replicaSet has exactly one member:  

[localhost:27017].  There should be at least 3 members there if I was actually doing Replication, no?  Please excuse my naivety, I'm new to Mongo.  I'm just trying to get this codebase to run on a dev box where there is no replication going on.  I am struggling to figure out which of the MongoOptions apply to the "dev box" use case and which apply to the "production" use case (were we do have 3 servers in the cluster).  My investigation is narrowing in on the MongoOptions as the actual root cause.

[Scott Hernandez]

If you are not connecting to a replicaset then don't use the
constructor which takes a list (even of 1) -- this is only to be used
when connecting to a replica set.

You could also run your dev instance as a replicaset of one member,
which will make it always the primary/master.

>>> > mongodb-user...@googlegroups.com

>>> > See also the IRC channel -- freenode.net#mongodb
>

> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to

> mongodb-user...@googlegroups.com

[Robert Kuhar]

Thanks for your reply, and I am confident that I now have my dev box running as worst ever single-node replSet.  Unfortunately I still get the same stacktrace;

    "com.mongodb.MongoException: can't find a master"

...from the troublesome ensureIndex call...

   collection.ensureIndex(new BasicDBObject("login", 1), "login_index", true);.

The conclusion I am reaching is that one cannot have mongo configured for bind_ip = 127.0.0.1 in the presence of replSets.  Is this a bug or just a total misunderstanding the bind_ip mongo startup parameter on my part?

So many questions, so little time.  Sigh.  For the time being, I am going to leave my setup as single-node replSet and just comment out the bind_ip=127.0.0.1 in my mongo.conf.  The effect of this is that my dev box is now a mongo-slut, accepting connections from anywhere, correct?

In any event, thanks so much for your help.  The ways of MongoDB are become more clear to me as I work my way through each of these challenges.

Bob

>>> > mongodb-user+unsubscribe@googlegroups.com

>>> > See also the IRC channel -- freenode.net#mongodb
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to

> mongodb-user+unsubscribe@googlegroups.com

> See also the IRC channel -- freenode.net#mongodb

>>> > mongodb-user+unsubscribe@googlegroups.com

>>> > See also the IRC channel -- freenode.net#mongodb
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to

> mongodb-user+unsubscribe@googlegroups.com

> See also the IRC channel -- freenode.net#mongodb

>>> > mongodb-user+unsubscribe@googlegroups.com

>>> > See also the IRC channel -- freenode.net#mongodb
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to

> mongodb-user+unsubscribe@googlegroups.com

[Scott Hernandez]

What does your rs.config() show?

>> >>> > mongodb-user...@googlegroups.com

>> >>> > See also the IRC channel -- freenode.net#mongodb
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > mongodb-user...@googlegroups.com

>> >>> > mongodb-user...@googlegroups.com

>> >>> > See also the IRC channel -- freenode.net#mongodb
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > mongodb-user...@googlegroups.com

>> >>> > mongodb-user...@googlegroups.com

>> >>> > See also the IRC channel -- freenode.net#mongodb
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > mongodb-user...@googlegroups.com

>> > See also the IRC channel -- freenode.net#mongodb
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to

> mongodb-user...@googlegroups.com

[Robert Kuhar]

bobk-mbp:DM_Server bobk$ mongo

MongoDB shell version: 2.0.4

connecting to: test

PRIMARY> rs.config();

{

"_id" : "dmReplSet",

"version" : 1,

"members" : [

{

"_id" : 0,

"host" : "localhost:27017"

}

]

}

PRIMARY> 

>> >>> > mongodb-user+unsubscribe@googlegroups.com

>> >>> > See also the IRC channel -- freenode.net#mongodb
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > mongodb-user+unsubscribe@googlegroups.com

>> >>> > mongodb-user+unsubscribe@googlegroups.com

>> >>> > See also the IRC channel -- freenode.net#mongodb
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > mongodb-user+unsubscribe@googlegroups.com

>> >>> > mongodb-user+unsubscribe@googlegroups.com

>> >>> > See also the IRC channel -- freenode.net#mongodb
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > mongodb-user+unsubscribe@googlegroups.com

>> > See also the IRC channel -- freenode.net#mongodb
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to

> mongodb-user+unsubscribe@googlegroups.com

[Scott Hernandez]

Does localhost resolve to 127.0.0.1? On some machines it does not.

You could change your bind_ip to localhost if they are different on
different systems.

>> >> >>> > mongodb-user...@googlegroups.com

>> >> >>> > See also the IRC channel -- freenode.net#mongodb
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user...@googlegroups.com

>> >> >>> > mongodb-user...@googlegroups.com

>> >> >>> > See also the IRC channel -- freenode.net#mongodb
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user...@googlegroups.com

>> >> >>> > mongodb-user...@googlegroups.com

>> >> >>> > See also the IRC channel -- freenode.net#mongodb
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user...@googlegroups.com

>> >> > See also the IRC channel -- freenode.net#mongodb
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > mongodb-user...@googlegroups.com

>> > See also the IRC channel -- freenode.net#mongodb
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to

> mongodb-user...@googlegroups.com

[Robert Kuhar]

That's a great question and the answer is not clear to me.  Localhost is bound to 127.0.0.1 in my /etc/hosts file and ping manages to resolve it to 127.0.0.1.

Further, if I tail -F the mongo console log I see "connection accepted" and "end connection" messages for each connection made.  From the JavaScript shell these messages look like...

    Thu May 24 16:52:53 [initandlisten] connection accepted from 127.0.0.1:27162 #21

    Thu May 24 16:52:56 [conn21] end connection 127.0.0.1:27162

From my initial Mongo Java Tutorial App (new Mongo();) they look like...

    Thu May 24 16:47:54 [initandlisten] connection accepted from 127.0.0.1:27098 #17

    Thu May 24 16:47:57 [conn17] end connection 127.0.0.1:27098

But, from the app that is giving me all the problems (new Mongo(new ArrayList<ServerAddress>(replicaSet), mongoOptions);), the log suggests that the connection(s) are coming from an IP address other than 127.0.0.1

    Thu May 24 16:49:15 [initandlisten] connection accepted from 10.78.4.124:27114 #18

    Thu May 24 16:54:57 [conn18] end connection 10.78.4.124:27114

Given what I see in the mongo log, I don't think trying to bind_ip=localhost would work either as the driver is forging that connection as if it is coming in from the outside (as little as I know of Mongo, my understanding of the network layer is equally shallow).

I think I need to refine my earlier assertion.  The Java Driver, in the presence of replSets, is incompatible with the mongo server running with bind_ip = 127.0.0.1.  This feels more like a java driver bug all the time, but I am also coming to understand that my "Replica Set cluster of 1 on localhost" is not representative of any production quality deployment.

Bob

>> >> >>> > mongodb-user+unsubscribe@googlegroups.com

>> >> >>> > See also the IRC channel -- freenode.net#mongodb
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user+unsubscribe@googlegroups.com

>> >> >>> > mongodb-user+unsubscribe@googlegroups.com

>> >> >>> > See also the IRC channel -- freenode.net#mongodb
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user+unsubscribe@googlegroups.com

>> >> >>> > mongodb-user+unsubscribe@googlegroups.com

>> >> >>> > See also the IRC channel -- freenode.net#mongodb
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user+unsubscribe@googlegroups.com

>> >> > See also the IRC channel -- freenode.net#mongodb
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com
>> > To unsubscribe from this group, send email to

>> > mongodb-user+unsubscribe@googlegroups.com

>> > See also the IRC channel -- freenode.net#mongodb
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to

> mongodb-user+unsubscribe@googlegroups.com

[Jeff Yemin]

I suspect that you're hitting https://jira.mongodb.org/browse/JAVA-249, which will be resolved in 2.8.0.  Does it work if you connect like:

new Mongo("127.0.0.1", options)

?

[Robert Kuhar]

Yes.  mongo = new Mongo( "127.0.0.1", mongoOptions ); works and the mongo log records it as a connection from 127.0.0.1 like "Fri May 25 10:41:42 [initandlisten] connection accepted from 127.0.0.1:40764 #2"

https://jira.mongodb.org/browse/JAVA-249 does indeed describe my observation.  It looks like I am eagerly awaiting 2.8.0 drivers.  Thanks.

Bob