How to iterate large Collections(More than 20 million records)

[sanmukho]

Hi

I tried to iterate Large Collections using the below code however
getting exception
Cud u help?

Code:

import java.net.UnknownHostException;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBCursor;
import com.mongodb.DBObject;
import com.mongodb.Mongo;
import com.mongodb.MongoException;

public class TestUidChk {
public static void main(String[] args) throws UnknownHostException,
MongoException {
Mongo mg = new Mongo("172.16.4.18:37017");
DB mdb = mg.getDB("social");
DBCollection mcoll = mdb.getCollection("user");
long curSize = mcoll.count();
long i=0;
DBCursor cur = mcoll.find();
while(i < curSize)
{
cur = mcoll.find().limit(10).skip((int)i);
while(cur.hasNext()) {
try {
DBObject usr = cur.next();
System.out.println( usr.get("uid") );
} catch (Exception e) {
System.out.println( "Caught exception: "+
e.getMessage() );
}
}
i+=10;
if(i > curSize )
i=curSize;
}
}
}


Error:

16 Mar, 2011 6:00:59 PM com.mongodb.DBTCPConnector$MyPort error
SEVERE: MyPort.error called
java.lang.IllegalArgumentException: bad data. lengths don't match
read:204 != len:205
at org.bson.BSONDecoder._decode(BSONDecoder.java:67)
at org.bson.BSONDecoder.decode(BSONDecoder.java:42)
at com.mongodb.Response.<init>(Response.java:63)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:66)
at com.mongodb.DBPort.call(DBPort.java:56)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:266)
at com.mongodb.DBCursor._check(DBCursor.java:309)
at com.mongodb.DBCursor._hasNext(DBCursor.java:431)
at com.mongodb.DBCursor.hasNext(DBCursor.java:456)
at TestUidChk.main(TestUidChk.java:24)
Exception in thread "main" java.lang.IllegalArgumentException: bad
data. lengths don't match read:204 != len:205
at org.bson.BSONDecoder._decode(BSONDecoder.java:67)
at org.bson.BSONDecoder.decode(BSONDecoder.java:42)
at com.mongodb.Response.<init>(Response.java:63)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:66)
at com.mongodb.DBPort.call(DBPort.java:56)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:266)
at com.mongodb.DBCursor._check(DBCursor.java:309)
at com.mongodb.DBCursor._hasNext(DBCursor.java:431)
at com.mongodb.DBCursor.hasNext(DBCursor.java:456)
at TestUidChk.main(TestUidChk.java:24)

Thanks
Sandy

[Gates]

The error indicates that there may be some bad data here.

Is it possible to do a DB repair?

[Nat]

The data may have been corrupted. Was it crashed earlier? Did you try
to repair it?

[sanmukho]

> db.repairDatabase()
it worked like a charm.

Wht this above command does, Could u give link to source code?
When we shud use this command ?

As when I was using the same Code(JS) in shell it was working fine. so
didn't get this.

Thanks
Sandy

[Gaetan Voyer-Perrault]

Here are the docs on Durability and Repair:

http://www.mongodb.org/display/DOCS/Durability+and+Repair

- Gates

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.