Cursor Not Found on Server

[Doug Jones]

We have a part of our Java app that tails the op log using a cursor. We've noticed that we frequently get exceptions stating the cursor could not be found on the server:

com.mongodb.MongoException$CursorNotFound: cursor not found on server

Our mongo logs show entries like:

Fri Jul 27 04:10:43 [conn921191] getMore: cursorid not found local.oplog.rs 6753601463703890990
Fri Jul 27 04:11:02 [conn921191] getMore: cursorid not found local.oplog.rs 7612769998655795512
Fri Jul 27 04:11:18 [conn921191] getMore: cursorid not found local.oplog.rs 3982945265377179786

I've read this can be caused if there is a cursor timeout, however I believe the default is a 10 min timeout and this happens much more frequently than 10 mins after cursor creation. Any thoughts on what could be causing this?

Here's some version info.

MongoDB: 2.0.4
Java Driver: 2.6.5

Here are the options applied to our cursor.

cursor.addOption( Bytes.QUERYOPTION_TAILABLE );
cursor.addOption( Bytes.QUERYOPTION_AWAITDATA );
cursor.addOption( Bytes.QUERYOPTION_OPLOGREPLAY );

[Jenna]

When using await data, the server hangs on to the server side cursor for a certain period of inactivity before the cursor is subject to clean up. If your write activity is low, the cursor may have been cleaned on the server.

If you would like to tail the oplog indefinitely, your application will need to start a loop and remember where it last read from the oplog so that it can continue tailing if the cursor is lost.

[Doug Jones]

We are using the loop + remember last location technique. However, I feel like the cursor is being cleaned up or lost in cases where it shouldn't be. This cursor happens to be looking through the oplog at some point in the past, so there's definitely enough activity/oplog entries that the cursor should be returning something rather than being marked as inactive.

[Doug Jones]

Hi Daniel,

That's a good point and after some looking into the logs, I think what you suggested is the most likely cause for many of the failures.

However, I was able to find instances of the same exception when the cursor should be well within the oplog window as reported in MMS.

On Monday, August 6, 2012 11:09:46 AM UTC-4, Daniel Gottlieb wrote:

Hey Doug,

Can you confirm that when the error occurs the oplog hasn't "rolled over"? By that I mean ensuring the oplog is not being written to much faster than the application is reading from the cursor.

For example if the oplog is 100MB and 5MB of data is being written to it every second, but only 1MB is being tailed and processed from it every second, the reading cursor will fall behind at a rate of 4MB/sec. If these throughput rates remain constant for 25 seconds (fell behind by 100MB), the capped collection nature of the oplog will write new data over the data that is currently pointed to by the cursor. The cursor becomes obsolete and I believe the server also reports this error as "cursorid not found".