Re: [mongodb-user] java driver: DBCursor cur = coll.find(); doesn't iterate over all documents

[Michael Dirolf]

Can you send example code that inserts the documents and does the
query that's only giving you 101 results? Will be easier to debug with
some code to work from.

On Wed, Aug 18, 2010 at 10:18 AM, ZedroS <zedros....@gmail.com> wrote:
> hi
>
> I've been struggling which a DBCursor not returning all the results.
>
> It happens as well when using  the way described in mongodb java
> documentation Using a Cursor to Get All the Documents (http://
> www.mongodb.org/display/DOCS/Java+Tutorial?focusedCommentId=4161628#JavaTutorial-UsingaCursortoGetAlltheDocuments)
> .
>
> Strangely enough, I always get 101 results, which is the number put
> out in this tutorial, whereas I've 195 documents.
>
> When looking closer, using the iterator from a DBCursor, it looks like
> it triggers the first query and then use the number of result of it
> for ever afterwards. In my case, it means 101 results:
> { "ts" : "Wed Aug 18 2010 15:24:54 GMT+0200 (CEST)", "info" : "query
> ion-cf.City reslen:42798 nscanned:101  \nquery: {}  nreturned:101
> bytes:42782 0ms", "millis" : 0 }
>
> A count query returns 195 and setting a "high" limit on the query
> backing the DBCursor does the trick as well:
> { "ts" : "Wed Aug 18 2010 16:16:37 GMT+0200 (CEST)", "info" : "query
> ion-cf.City ntoreturn:300 reslen:82516 nscanned:195  \nquery: {}
> nreturned:195 bytes:82500 0ms", "millis" : 0 }
>
> is it a bug or a feature ?
>
> read ya
> zedros
>
> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[ZedroS]

[ZedroS]

hi Michael

I had done a test even before writing to the mailing list, but the
issue didn't show up in it.

However, since, I put it a bit more in line with my context and...
going at a remote mongo server did it: my test fails ! :) Doing a
DBCursor.hasnext in this context makes com.mongodb.Response._num to be
false.

I copy the code just under, but it is built on top of morphia (from
the source repo), so I'm not sure it helps and I don't know much about
the driver. Let me know.

thanks
zedros

the test code:

import java.util.List;

import junit.framework.Assert;
import junit.framework.TestCase;

import org.junit.Before;
import org.junit.Test;

import com.google.code.morphia.Datastore;
import com.google.code.morphia.Morphia;
import com.google.code.morphia.TestMapping.BaseEntity;
import com.google.code.morphia.annotations.Entity;
import com.mongodb.DB;
import com.mongodb.Mongo;

public class TestAsList extends TestCase
{
private int documentsNb;
protected Mongo mongo;
protected DB db;
protected Datastore ds;
protected Morphia morphia = new Morphia();

public TestAsList() {
try {
this.mongo = new Mongo("mongo.dev");
} catch (Exception e) {
throw new RuntimeException(e);
}
}

@Entity
static class E extends BaseEntity
{
private final Integer index;
private final byte[] largeContent;

public E()
{
index = null;
largeContent = createLargeByteArray();
}

private byte[] createLargeByteArray()
{
int size = (int) (4000 + Math.random() * 100000);
byte[] arr = new byte[size];
for (int i = 0; i < arr.length; i++)
{
arr[i] = 'a';
}
return arr;
}

public E(int i)
{
this.index = i;
largeContent = createLargeByteArray();
}

}

@Override
@Before
public void setUp()
{
this.mongo.dropDatabase("morphia_test");
this.db = this.mongo.getDB("morphia_test");
this.ds = this.morphia.createDatastore(this.mongo,
this.db.getName());

morphia.map(E.class);
documentsNb = 1000;
for (int i = 0; i < documentsNb; i++)
{
ds.save(new E(i));
}
this.db = this.mongo.getDB("morphia_test");
this.ds = this.morphia.createDatastore(this.mongo,
this.db.getName());
}

@Test
public void testWithManyElementsInCollection() throws Exception
{

Query<E> query = ds.createQuery(E.class);
long countAll = query.countAll();
query = ds.createQuery(E.class);
List<E> list = query.asList();
Assert.assertEquals(documentsNb, countAll);
Assert.assertEquals(documentsNb, list.size());
}
}

[Scott Hernandez]

How long does this test take to run? Is it possible you are getting a timeout on the cursor while process data on the client?

I can track this down and will create an issue for the driver if I can reproduce the problem.

--

[Scott Hernandez]

I ran the tests against a remote server (1.6 over my slow wifi) and there was no problem.

I can't reproduce this :(

Are there any other environmental issues you can think of to try?

On Wed, Aug 18, 2010 at 8:41 AM, ZedroS <zedros....@gmail.com> wrote:

--

[ZedroS]

On Aug 18, 5:45 pm, Scott Hernandez <scotthernan...@gmail.com> wrote:
> How long does this test take to run? Is it possible you are getting a
> timeout on the cursor while process data on the client?

well, I'm at home right now, but it wasn't too long. A few seconds I
would say. I can check tomorrow if really needed.

> I can track this down and will create an issue for the driver if I can
> reproduce the problem.

very nice, thanks.

[ZedroS]

On Aug 18, 6:21 pm, Scott Hernandez <scotthernan...@gmail.com> wrote:
> I ran the tests against a remote server (1.6 over my slow wifi) and there
> was no problem.

I was working with mongodb 1.6 as well, on top of openvz debian,
accessing it remotely from an ubuntu 9.04 with latest java (sun/
oracle) update and morphia source.

> I can't reproduce this :(

I was fearing such an event. For me as well it was kind of elusive...

> Are there any other environmental issues you can think of to try?

hum, not really.

it here a way I could check the timeout on the cursor ?

In the end, however, the issue pop up on the very first use of the
iterator, where it retrieves the number of results. That where the
wrong number showed up and stayed afterwards. I could provide the
exact lines tomorrow (from mongodb driver, on the Response object).

in the end, currently, few would indicate a morphia issue here. I'll
try to have a mongodb java driver only test tomorrow (I just have to
found out how to define my document's content).

thanks again for your help
best
zedros

[ZedroS]

hi

I did a test case without morphia, as least for the reading part
(which is the faulty one), with the same outcome. Maybe I should open
a proper issue for it.

Here is the code:

import java.util.ArrayList;

import java.util.List;

import junit.framework.Assert;
import junit.framework.TestCase;

import org.junit.Before;
import org.junit.Test;

import com.google.code.morphia.Datastore;
import com.google.code.morphia.Morphia;
import com.google.code.morphia.TestMapping.BaseEntity;
import com.google.code.morphia.annotations.Entity;
import com.mongodb.DB;

import com.mongodb.DBCollection;
import com.mongodb.DBCursor;

DBCollection collection = db.getCollection("E");
int countAll = (int) collection.count();
List list = new ArrayList();
long startOfCursorUse = System.currentTimeMillis();
DBCursor cur = collection.find();
while (cur.hasNext())
{
list.add(cur.next());
}
long endOfCursorUse = System.currentTimeMillis();
System.out.println("Cursor used for " + (endOfCursorUse -
startOfCursorUse) + " ms");
Assert.assertEquals(documentsNb, countAll);
Assert.assertEquals(documentsNb, list.size());
}
}

[Eliot Horowitz]

Our tests of that always work.

If you can provide a full example that breaks would be great.

May also want to try java 2.1 driver (coming out shortly)

[ZedroS]

> If you can provide a full example that breaks would be great.

what do you mean by that ? The latest example breaks and does the
query bit only on the driver. Or would you like as well the data put
in the DB to have the full stuff (I guess I could export it)

[Eliot Horowitz]

Yes - either the data, or a java program that loads data and exhibits
the problem.
The code you sent works on my collections

[Scott Hernandez]

Here is a silly question, but is it possible you have corrupt data? Have you been able to reproduce this on another (fresh) system?

--

[ZedroS]

hi

so, we worked more on the issue.

New discovery: mongodump had the same issue as my test => only a
subpart of the collection is dumped.

The server is a debian one, with lastest mongo package, meaning 1.6.1
version. For the test, I used the latest morphia code, meaning the 2.1
mongo driver.

The data folder, with the dump folder in it, can be found there
http://rapidshare.com/files/414635149/mongo.7z (warning; only 10
downloads allowed). The content is the data dir in a tar file and then
put in a 7z file (size decreased from 208M to 42K !!).

Please be aware I changed a bit my test, with smaller docs, which made
me encounter the same 101 number again, both in the test and in
mongodump:

mongo1:/var/lib/mongodb# mongodump
connected to: 127.0.0.1
all dbs
DATABASE: morphia_test to dump/morphia_test
morphia_test.E to dump/morphia_test/E.bson
101 objects
morphia_test.system.indexes to dump/morphia_test/
system.indexes.bson
1 objects
DATABASE: admin to dump/admin

The new test code is under, I've commented out the lines to drop the
db and create the data.

let me know if I can help more in anyway

package com.google.code.morphia.query;

public E()

{

index = null;

}

public E(int i)

{

this.index = i;

}



}

@Override

@Before

public void setUp()

{

// this.mongo.dropDatabase("morphia_test");

this.db = this.mongo.getDB("morphia_test");

this.ds = this.morphia.createDatastore(this.mongo,
this.db.getName());



morphia.map(E.class);

documentsNb = 1000;

// for (int i = 0; i < documentsNb; i++)

// {

// ds.save(new E(i));

// }

this.db = this.mongo.getDB("morphia_test");

this.ds = this.morphia.createDatastore(this.mongo,
this.db.getName());

}



@Test

public void testWithManyElementsInCollection() throws Exception

{



Query<E> query = ds.createQuery(E.class);

DBCollection collection = db.getCollection("E");

int countAll = (int) collection.count();

List list = new ArrayList();

long startOfCursorUse = System.currentTimeMillis();

DBCursor cur = collection.find();

while (cur.hasNext())

{

list.add(cur.next());

}

long endOfCursorUse = System.currentTimeMillis();

System.out.println("Cursor used for " + (endOfCursorUse -
startOfCursorUse) + " ms");

Assert.assertEquals(documentsNb, countAll);

Assert.assertEquals(documentsNb, list.size());

}

}

On Aug 20, 2:35 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Yes - either the data, or a java program that loads data and exhibits
> the problem.
> The code you sent works on my collections
>

[ZedroS]

it's mongodb-unstable package, and the debian for 64 bits OS

[ZedroS]

the debian lenny is running on top of openvz => test env.
this issue is not reproduced on other boxes (a kvm one and an ubuntu
one)

I hope all is said.

[Eliot Horowitz]

So before I look at anything - openvz has known issues with mongo...

Are you sure this has never happened anywhere else?

[ZedroS]

On Aug 23, 5:04 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> So before I look at anything - openvz has known issues with mongo...

my admin told me so as well (having put it in place), but then he said
it was about data corruption, whereas our data is nicely readable

> Are you sure this has never happened anywhere else?

up to now yes, definitely

[Eliot Horowitz]

We know about corruption/crashes on openvz, but the oddities may
extend further than that.

Please test a bit more on non openvz systems.

[ZedroS]

> We know about corruption/crashes on openvz, but the oddities may
> extend further than that.

looks like it indeed

> Please test a bit more on non openvz systems.

we have decided to move to kvm for our test env, I guess it'll be ok

shall I open an issue for this openvz specific bug ?

thanks once more for your help

best
zedros

[Eliot Horowitz]

There already is a bug for openvz, maybe add a comment about this problem?

[ZedroS]

On Aug 24, 2:41 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> There already is a bug for openvz, maybe add a comment about this problem?
>

I'll do so