Records are getting skipped while persisting into Mongo DB

152 views
Skip to first unread message

jagvandan nayak

unread,
Jan 28, 2016, 12:23:36 AM1/28/16
to mongodb-user
I am facing issue with Mongo DB document persist.I am able to read all the records from input file but while persisting into Mongo DB, documents are getting skipped. There is no error in logs.
For example, there are 100 records in the input file but after staging data into Mongo DB, only 44 records are getting persist.
As soon as the size is getting increased in mongo db , the document persistence frequency is getting decreased and after a point around 970 records, not even a single document is getting persisted in mongo db.

Ichecked max bson object size also, which is enough and greater than mongo db current storage size. 

I tried with mongo 3.0.4 version. 
I also cleaned all collections from existing mongo db and tried, so 1st tym 70 records persist out of 100 but over that only 45 records persist fro next 100 records.


Mongo DB Details::

rsdms01:PRIMARY> db.runCommand("buildinfo");
{
        "version" : "3.0.4",
        "gitVersion" : "0481c958daeb2969800511e7475dc66986fa9ed5",
        "OpenSSLVersion" : "OpenSSL 1.0.1e-fips 11 Feb 2013",
        "sysInfo" : "Linux ip-10-146-0-110 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011 x86_64 BOOST_LIB_VERSION=1_49",
        "loaderFlags" : "-fPIC -pthread -Wl,-z,now -rdynamic",
        "compilerFlags" : "-Wnon-virtual-dtor -Woverloaded-virtual -std=c++11 -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Werror -O3 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-missing-braces -fno-builtin-memcmp -std=c99",
        "allocator" : "tcmalloc",
        "versionArray" : [
                3,
                0,
                4,
                0
        ],
        "javascriptEngine" : "V8",
        "bits" : 64,
        "debug" : false,
        "maxBsonObjectSize" : 16777216,
        "ok" : 1
}

Stephen Steneker

unread,
Jan 28, 2016, 3:45:12 AM1/28/16
to mongodb-user

On Thursday, 28 January 2016 16:23:36 UTC+11, jagvandan nayak wrote:

I am facing issue with Mongo DB document persist.I am able to read all the records from input file but while persisting into Mongo DB, documents are getting skipped. There is no error in logs.
For example, there are 100 records in the input file but after staging data into Mongo DB, only 44 records are getting persist.
As soon as the size is getting increased in mongo db , the document persistence frequency is getting decreased and after a point around 970 records, not even a single document is getting persisted in mongo db.

Hi Jagvandan,

Can you please provide some further details:

  • How you are inserting documents into MongoDB (custom code, mongoimport, etc)

  • The specific version of the client/driver you are using

  • A usage example (code or command-line) for how you are inserting documents and checking that they are successfully inserted

Thanks,
Stephen

​

jagvandan nayak

unread,
Jan 29, 2016, 5:41:26 AM1/29/16
to mongodb-user
Hi Stephen,

Thanks for your reply. please find below requested details by you.

1) In order to insert documents into MongoDB we call the save method of class DBCollection(import com.mongodb.DBCollection) where DBObject
is passed as the argument(import com.mongodb.DBObject). following is the code for data insertion in mongoDB that we call

 public void persist(List<DBObject> object)
{
  for (DBObject object2 : object)
  {
  collection.save(object2);
}
} 

2. The specific version of the client/driver we are using are
a)mongo-java-driver-2.10.1.jar 
b)mongodb_client-1.0.jar  

3)In order to check the successful insertion of records we login to mongo DB shell and check the records by executing mongo command

Thanks
Jagvandan Nayak

Stephen Steneker

unread,
Jan 29, 2016, 7:35:38 PM1/29/16
to mongodb-user

On Friday, 29 January 2016 21:41:26 UTC+11, jagvandan nayak wrote:

1) In order to insert documents into MongoDB we call the save method of class DBCollection(import com.mongodb.DBCollection) where DBObject
is passed as the argument(import com.mongodb.DBObject).

…

collection.save(object2);

Hi Jagvandan,

Your code snippet isn’t checking the result of the save(). You should also be aware that the save() method does upserts if your objects have an _id field. If your intent is only to insert new documents, you should be using the insert() method instead. For an import use case, bulk write operations are generally more appropriate.

Are you using the Mongo class or MongoClient (which superceded the Mongo class in the 2.10 driver)? If you are using the older Mongo class and have not set any WriteConcern, the default will be unacknowledged writes.

2) The specific version of the client/driver we are using are
a)mongo-java-driver-2.10.1.jar
b)mongodb_client-1.0.jar

The 2.10 Java driver is very old (Dec 2012) and is not fully compatible with MongoDB 3.x (see: Java Driver Compatibility). For example, it does not include support for the newer bulk write operations API added in MongoDB 2.6.

As a starting point to eliminate driver compatibility problems please upgrade to a newer version of the Java driver. If this is a new project, you should use the latest Java driver (3.x series).

You also mentioned using MongoDB 3.0.4 server. I would strongly encourage upgrading to the latest 3.0.x (currently 3.0.9) as there have been some important bug fixes: Release Notes for MongoDB 3.0.

Regards,
Stephen

​
Reply all
Reply to author
Forward
0 new messages