Still having problems with failover and the java client

[Tony Nelson]

I wrapped my Mongo operation in a retry like this:

public String storeAttachment(AttachmentAbstract attachment, File
path) {

Exception lastException = null;

for (int i = 0; i < MAX_RETRIES; i++) {
try {
DB db = getDB();
db.requestStart();
GridFS gridFS = getGridFS(db);
GridFSInputFile dbFile = gridFS.createFile(path);
dbFile.put("attachmentId",
attachment.getAttachmentId());
dbFile.put("className",
attachment.getClass().getName());
dbFile.put("type", attachment.getAttachmentClass());
dbFile.put("origpath", path.getAbsolutePath());
dbFile.save();
dbFile.validate();
CommandResult result = db.getLastError();
db.requestDone();

if (!result.ok()) {
throw new
BusinessException(result.getErrorMessage(), result.getException());
}

return dbFile.getId().toString();
} catch (Exception e) {

logger.error(e.getMessage(), e);
lastException = e;

try {
Thread.sleep(SLEEP_TIME);
} catch (InterruptedException ie) {
// don't care
}
}
}

throw new BusinessException("Unable to store attachment in
Mongo", lastException);
}

This works fine until I force a failover with rs.stepDown(120).

No matter how long I wait, or how much I retry, the connection never
fixes itself. I always have to restart my app server. I have tried
this both with and without slaveOk(), that isn't making any
difference.

Should I throw away my Mongo object and create a fresh one? Will that
cause a memory leak?

Any other suggestions on what I should try?

Thanks in advance
Tony Nelson

[Scott Hernandez]

There have been some improvements for replicaset fail-over in the
current master branch. What version are you using now?

If you can try to build and use master you might have better luck.

You can create a new one if you want; as long as you don't keep any
references they will get cleaned up and should be no leak.

>
> Any other suggestions on what I should try?
>
> Thanks in advance
> Tony Nelson
>

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Tony Nelson]

Creating a new Mongo object fixes the problem but it feels like the
wrong thing to do. I couldn't find a method on the Mongo object that
I should be calling to release any allocated resources. Am I setting
myself up for a memory/connection/etc leak?

Is the code in the current master something you would consider putting
in a production environment? I certainly can download and build it,
but if it's not safe I can't risk it in production right now.

Thanks again
Tony

[Scott Hernandez]

On Sat, Feb 12, 2011 at 6:24 PM, Tony Nelson <hhu...@gmail.com> wrote:
> Creating a new Mongo object fixes the problem but it feels like the
> wrong thing to do.  I couldn't find a method on the Mongo object that
> I should be calling to release any allocated resources.  Am I setting
> myself up for a memory/connection/etc leak?

No --well, probably not--, when the Mongo instance gets garbage
collected all resources will be released. But there is a major
performance issue if you always create a new one; I would suggest just
doing that when you get in the failure case you are seeing.

> Is the code in the current master something you would consider putting
> in a production environment?  I certainly can download and build it,
> but if it's not safe I can't risk it in production right now.

It hasn't gone through final testing, no. But it might fix some of
your issues. For now you can create a new instance and if the new
version fixes that then it would be a better option than creating new
instances. It would good if you could test it and let us know if it
fixes your issue; if not we can work to create a reproducible test
case so we can fix it, and not have it come up again.

> Thanks again
> Tony

[Tony Nelson]

On Sat, Feb 12, 2011 at 9:36 PM, Scott Hernandez
<scotthe...@gmail.com> wrote:
>
> No --well, probably not--, when the Mongo instance gets garbage
> collected all resources will be released. But there is a major
> performance issue if you always create a new one; I would suggest just
> doing that when you get in the failure case you are seeing.
>

I only do it in the error case. I wrote a little help method.

private Exception handleError(final Exception e, final boolean sleep) {

setMongo(null);

if (e != null) {
logger.error(e.getMessage(), e);
}

if (sleep) {

try {
Thread.sleep(SLEEP_TIME);
} catch (InterruptedException ie) {
// don't care
}
}

return e;
}

> It hasn't gone through final testing, no. But it might fix some of
> your issues. For now you can create a new instance and if the new
> version fixes that then it would be a better option than creating new
> instances. It would good if you could test it and let us know if it
> fixes your issue; if not we can work to create a reproducible test
> case so we can fix it, and not have it come up again.
>

I cloned the repository, which I assume gives me the latest code.
(I'm an SVN guy, just learning GIT). Built it, ran the test cases and
dropped it in my project. I get the exact same behavior as with the
2.4 stable driver.

What info can I provide to help write a test case? There is nothing
especially exceptional about my Mongo provider class, I'm happy to
send it to you. It's only a couple of hundred lines.

In a nutshell, I log into my web app, and perform a few operations
that read or write from a Collection or GridFS. In a console I force
the replica set to fail over and repeat the same operations. They
fail with this exception (this exception is from a collection):

Caused by: com.mongodb.MongoException: not master
at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:136)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:157)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:141)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:225)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:180)
at com.mongodb.DBCollection.insert(DBCollection.java:72)
at com.mongodb.DBCollection.save(DBCollection.java:537)
at com.mongodb.DBCollection.save(DBCollection.java:517)

Thanks again for all of your help.

[Scott Hernandez]

Yep, that is it. You can do "git pull" in the future to keep it up to date.

> (I'm an SVN guy, just learning GIT).  Built it, ran the test cases and
> dropped it in my project.  I get the exact same behavior as with the
> 2.4 stable driver.
>
> What info can I provide to help write a test case?  There is nothing

If you can provide a class (source is best) that we can run to repro.
that would do it. The directions sound pretty clear. If you want to
wrap all that up in a jira issue (http://jira.mongodb.org/) then you
can track the progress.

> especially exceptional about my Mongo provider class, I'm happy to
> send it to you.  It's only a couple of hundred lines.

If it is private code you can create a support case instead of a public issue.

> In a nutshell, I log into my web app, and perform a few operations
> that read or write from a Collection or GridFS.  In a console I force
> the replica set to fail over and repeat the same operations.  They
> fail with this exception (this exception is from a collection):

Basically the mongo instance never finds the new master it seems.

> Caused by: com.mongodb.MongoException: not master
>            at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:136)
>            at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:157)
>            at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:141)
>            at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:225)
>            at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:180)
>            at com.mongodb.DBCollection.insert(DBCollection.java:72)
>            at com.mongodb.DBCollection.save(DBCollection.java:537)
>            at com.mongodb.DBCollection.save(DBCollection.java:517)
>
> Thanks again for all of your help.
>

[Tony Nelson]

On Sat, Feb 12, 2011 at 10:28 PM, Scott Hernandez

<scotthe...@gmail.com> wrote:
> On Sat, Feb 12, 2011 at 6:57 PM, Tony Nelson <hhu...@gmail.com> wrote:
>> On Sat, Feb 12, 2011 at 9:36 PM, Scott Hernandez
>> <scotthe...@gmail.com> wrote:
>
> If you can provide a class (source is best) that we can run to repro.
> that would do it. The directions sound pretty clear. If you want to
> wrap all that up in a jira issue (http://jira.mongodb.org/) then you
> can track the progress.
>

Done.

http://jira.mongodb.org/browse/JAVA-271

Thanks for all the help.