Retrieval from Replica Set Takes Long Time

[gli]

Hi,

What is the best way to retrieve records from a busy replica set?

Our MongoDB replica set is consist of 3 MongoDBs. About 2 million records each day are inserted to MongoDBs, with the average record size of 32k bytes. The index one is “date” that the record is inserted, with the type of mongo::Date_t.

The problem we have is that it takes about 8 hours to retrieve the daily records and write to a file. The following is the retrieve code:

// connect to MongoDB
mongo::DBClientConnection *pConn = new mongo::DBClientConnection;
pConn->connect(mongo_db, strerror);

// create a query on index one “date”
mongo::BSONObjBuilder query;
mongo::BSONObjBuilder sub(query.subobjStart("date"));
sub.appendTimeT("$gte", start_timestamp);
sub.appendTimeT("$lte", stop_timestamp);
sub.done();

std::auto_ptr<mongo::DBClientCursor> cursor = pConn->query(collection_name, query.obj());
while (cursor->more())
{
    mongo::BSONObj obj = cursor->next();
    std::string strXml = obj["xml"].toString(false, true);
    // write the data to a file
    datfile << strXml << std::endl;
}

Is it possible to speed up the retrieval?

Thanks for your help!

GLI