Problem with the Cursor in the java driver. A mongo-db java driver Bug ?

[sebastien]

Hello,

Im a using the java driver for mongo-db and each night, i need to
iterate through all the items of my collection (aprox. 10000 items) to
make some special process (image caching).
My code is very simple :

DBCursor cur = this.mongoDBCollectionProducts.find(new
BasicDBObject("is_active", true));
System.out.println("NB items: "+cur.count());
int processed_items = 0;
while (cur.hasNext())
{
DBObject obj = cur.next();
// My custom process on the object.
//...
processed_items ++;
}
System.out.println("Processed items : "+processed_items
+" ("+cur.count()+")");

The problem is that the cusor did not iterate through all the items of
my data set... The "hasNext()" method is returning false, but all the
objects have not been processed.
The outpout is like :
NB items : 9799
Processed items : 1575 (9799)

No exception thrown, nothing else on the output.
I tried to "force" the cusor to go through the all the items by
replacing the "while" with a "for (int i=0; i<cur.count(); i++)" and i
get an error "Exception in thread "Thread-0"
java.lang.RuntimeException: no more
at com.mongodb.DBApiLayer$Result.next(DBApiLayer.java:347)
...
"

As a second test, i just removed the part of my process witch is a bit
"long" : downloading an image from a website, croping it and storing
it back to a cache server and ... miracle ! the cursor iterate through
all the data set...
I precise, the commented process has nothing to do with mongo-db. it's
just slowing down the process between two iterations.

I am really despaired, does anybody already has such a problem or has
an idea for me ?

Best regards,

[Eliot Horowitz]

Mongo cursors timeout after 10 minutes.
So if the downloading is slow you'l see that.

What version are you using? You should be getting a cursor timed out exception

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[sebastien]

Thanks you for your answer Eliot. After a lot of try and error i
finally find the issue.
You're right with the timeout. I just reduced the batch size in my
query : "DBCursor cur = this.mongoDBCollectionProducts.find(new
BasicDBObject("is_active", true)).batchSize(100);" and now everything
is working fine.
I belive the cursor connection get a timeout between two calls.
What is really stange is that i did not get any exception to warn me
about the expiration of the cursor...
An hint i'm going to keep in mind for all my db projects. really
touchy...

Thank you for your help

On 3 août, 18:10, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Mongo cursors timeout after 10 minutes.
> So if the downloading is slow you'l see that.
>
> What version are you using?  You should be getting a cursor timed out exception
>

[Eliot Horowitz]

What version of the driver are you using?

[sebastien]

Im am using the 2.0
I tried with the 1.4 and get the same problem.

On 3 août, 19:07, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> What version of thedriverare you using?

>
> On Tue, Aug 3, 2010 at 1:06 PM, sebastien <seb.b...@gmail.com> wrote:
> > Thanks you for your answer Eliot. After a lot of try and error i
> > finally find the issue.
> > You're right with the timeout. I just reduced the batch size in my
> > query : "DBCursor cur = this.mongoDBCollectionProducts.find(new
> > BasicDBObject("is_active", true)).batchSize(100);" and now everything
> > is working fine.
> > I belive the cursor connection get a timeout between two calls.
> > What is really stange is that i did not get any exception to warn me
> > about the expiration of the cursor...
> > An hint i'm going to keep in mind for all my db projects. really
> > touchy...
>
> > Thank you for your help
>
> > On 3 août, 18:10, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> >> Mongo cursors timeout after 10 minutes.
> >> So if the downloading is slow you'l see that.
>
> >> What version are you using?  You should be getting a cursor timed out exception
>
> >> On Tue, Aug 3, 2010 at 12:07 PM, sebastien <seb.b...@gmail.com> wrote:
> >> > Hello,
>

> >> > Im a using thejavadriverfor mongo-db and each night, i need to