mongorestore is not restoring complete db

[Gulab Pasha]

Hi Guys,

Kindly help me with mongodump and restore, we have running mongodb setup and we have planning to move our db setup on cloud.

I have successfully backdup my db using the following.

#mongodump --db dbname (dumped into current directory) which is also around 21gb of size, and I compressed the same and moved the compressed file into my new setup.

Where is have configured mongodb (Centos 7 and Mongodb 3.2.9).

Now the issues starts while restoring my db, I have not received any error but it is not complete, I mean mongorestore is not restoring my complete db. I used the following command to restore.

#mongodrestore --db dbname --drop dump/dbname

#mongorestore --db dbname --noIndexRestore --drop dump/dbname

I tried restoring with index and without indexing.

Kindly help

[Gulab Pasha]

Hey Guys,

Kindly help me restoring complete db on my new VM centOS 7 and Mongodb 3.2.9.

Thanks,
Gulab

[shakti shah]

mongorestore --db database_name path_to_bson_file

example:

>mongorestore --db shah H:\quiz.json\quiz

i have imported quiz database into new shah database..

worked successfully.. hope it will be useful....

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/86f63c8c-2ef8-4994-aea2-76bbc3f720a5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gulab Pasha]

Hi Shakti,

I tried to restore db using the following but still the same, it is not restoring complete db, I see there is no error while restoring but out 54gb it is restoring only 11gb data.

Is there any other way of restoring complete mongodb?

mongorestore --db database_name db_dump_path  (this is not worked)

Thanks,

Gulab Pasha

On Sunday, September 25, 2016 at 6:56:37 PM UTC+5:30, shakti shah wrote:

mongorestore --db database_name path_to_bson_file

example:

>mongorestore --db shah H:\quiz.json\quiz

i have imported quiz database into new shah database..

worked successfully.. hope it will be useful....

On Sun, Sep 25, 2016 at 4:12 PM, Gulab Pasha <gulab...@gmail.com> wrote:

Hey Guys,

Kindly help me restoring complete db on my new VM centOS 7 and Mongodb 3.2.9.

Thanks,
Gulab

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

[R Kumar]

Hi Gulab,

Is there a chance you were not using WiredTiger on the mongod you created the dump from?

mongorestore should output the number of files restored - can you compare that against the number of documents in the database you dumped from? You can use something like the dbstats command on the original and new database to compare the number of objects. If its the same, its possible you are just seeing WT compression in action.

[Kevin Adistambha]

Hi,

I tried to restore db using the following but still the same, it is not restoring complete db, I see there is no error while restoring but out 54gb it is restoring only 11gb data.

mongorestore should be able to restore the whole database. Could you please provide more details:

• Did mongodump correctly dumped the whole database?
• How did you determine that it’s only restoring 11GB out of 54GB?
• Are there actual missing documents in the restored database?
• What are the topologies and specifications of the source and destination databases (i.e. are both standalone nodes, replica sets, or sharded cluster, both using WiredTiger, etc.)?

The more detailed information you can give regarding your setup and the actual sequence of events, the more likely we could help you.

mongorestore —db database_name db_dump_path (this is not worked)

Could you please try running mongorestore with the -v flag? Note that you can pass multiple v e.g. -vvvv to increase the logging verbosity. Hopefully the increased logging verbosity will show some hint of the cause.

You could also check the mongodump page and the mongorestore page to see if there are any parameters that may be missed.

Best regards,
Kevin

​

[Gulab Pasha]

Hi Kevin,

Sorry for my late reply, mongorestore in unable to restore complete DB, I have captured the complete dump and restore log.

• Did mongodump correctly dumped the whole database?

• Yes - mongodump correctly dumped my complete db which is almost 21gb.
• root@server1 [~]# mongo --version
• MongoDB shell version: 3.2.9
• root@server1 [~]# mongo
• MongoDB shell version: 3.2.9
• connecting to: test
• Server has startup warnings:
• 2016-09-19T12:07:57.342+0530 I CONTROL  [initandlisten]
• 2016-09-19T12:07:57.342+0530 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
• 2016-09-19T12:07:57.342+0530 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
• 2016-09-19T12:07:57.342+0530 I CONTROL  [initandlisten]
• 2016-09-19T12:07:57.342+0530 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 1024 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.
• 2016-09-19T12:07:57.342+0530 I CONTROL  [initandlisten]
• > show dbs
• componentbuy  54.427GB
• local          3.952GB
• logintest      0.078GB
• > exit
• bye

• How did you determine that it’s only restoring 11GB out of 54GB?

• After restore completed I have logged into mongodb and checkt he DB size, which was showing only 7.93GB
• > show dbs
• componentbuy  7.923GB
• local         0.000GB
• >

• What are the topologies and specifications of the source and destination databases (i.e. are both standalone nodes, replica sets, or sharded cluster, both using WiredTiger, etc.)?

• Both are Standalone Nodes.

I have attached the complete dump and restore log with this post, looking forward to your support.

• Are there actual missing documents in the restored database?

• Yes - I believe DB is missing all the necessary documents and users

[Kevin Adistambha]

Hi,

Output of mongodump and mongorestore doesn’t show any error. However, data size is not a reliable metric when judging if two MongoDB deployments have the same content due to fragmentation, storage engine differences, etc.

Both are Standalone Nodes.

What about their storage engine? Are both of them using WiredTiger, MMAPv1? Are the output of db.serverCmdLineOpts() of both servers identical?

Yes - I believe DB is missing all the necessary documents and users

How did you find out that there are missing documents in the restored database? Is your application giving errors?

A couple of things you could check:

• Run db.stats() on the source server and the restored server. The output should look like the following:

    > db.stats()
    {
        "db": "test",
        "collections": 12,
        "objects": 10305,
        "avgObjSize": 149.719941775837,
        "dataSize": 1542864,
        "storageSize": 872448,
        "numExtents": 0,
        "indexes": 17,
        "indexSize": 774144,
        "ok": 1
    }
  

  Of particular interest is the collections and objects numbers, which will tell you the total number of documents that are present in the source and restored databases.

• Run db.<collection>.stats() on each collection. This command will have a similar number (i.e. document count) that you can compare on both source and restored databases.

Please note that the stats() numbers could be inaccurate if there was ever an unexpected shutdown (see https://docs.mongodb.com/v3.2/reference/method/db.collection.stats/#unexpected-shutdown-and-count). To ensure you have an accurate count, you can run db.<collection>.validate() on each collection.

Best regards,
Kevin

​

[Kevin Adistambha]

Hi,

Sorry, a small clarification for this sentence in my previous post:

However, data size is not a reliable metric when judging if two MongoDB deployments have the same content

I actually meant storage size (i.e. storageSize from the db.stats() output, which is the amount of disk space used) instead of data size (i.e. dataSize, which is the size of the actual data)

The output of show dbs shows the storage size on disk instead of the actual data size. Those values are the same values from the output of db.adminCommand({listDatabases:1}).

One possibility for the storage size discrepancy that you’re seeing is that mongodump were done on a MongoDB running the MMAPv1 storage engine, and mongorestore were done to a WiredTiger storage engine. The WiredTiger storage engine uses compression by default, which frequently results in a storage size (disk usage) that is smaller compared to MMAPv1 for the same data.

Best regards,
Kevin

​

[Gulab Pasha]

Hi Kevin,

I tried to restore my MongoDB dump in different Linux flavors but not satisfied with the outcome.

I have tried to restore my DB in CentOS 7 and Ubunut 16.04LTS and I have install MongoDB 3.2.10 on both the flavors. In both "Linux flavor VM's mongo 3.2.10" mongorestore is seems to be successful with Indexing but DB size seems to be very very less. However I have prepared one more VM with CentOS 7 and installed MongoDB 2.6.

In CentOS 7 MongoDB 2.6 I was unable to restore DB with Indexes, I have restored DB without indexes but the outcome is some what satisfactory.

Kindly find the attached screens shots and let me know where am I making mistake.

I have attached 3 screenshots

1) Main MongoDB Server

2) CentOS7 VM - MongDB2.6

3) Ubuntu 16.04 VM - MongoDB3.2.10

Sorry I'm very much new to MongoDB setup.

Looking forward to your help and support.

Thanks,

Gulab Pasha

[Gulab Pasha]

Hi Guys,

Any help on mongodb restore on 3.2.10 ?

On Thursday, September 22, 2016 at 12:18:52 PM UTC+5:30, Gulab Pasha wrote:

[Kevin Adistambha]

Hi Gulab,

I have tried to restore my DB in CentOS 7 and Ubunut 16.04LTS and I have install MongoDB 3.2.10 on both the flavors. In both “Linux flavor VM’s mongo 3.2.10” mongorestore is seems to be successful with Indexing but DB size seems to be very very less. However I have prepared one more VM with CentOS 7 and installed MongoDB 2.6.

In CentOS 7 MongoDB 2.6 I was unable to restore DB with Indexes, I have restored DB without indexes but the outcome is some what satisfactory.

What I can gather from your screenshots are:

• “CentOS7” and “Main Server” are using MMAPv1 (from the existence of dataFileVersion field), and “Ubuntu16” is using WiredTiger (from the lack of dataFileVersion field). See https://docs.mongodb.com/manual/reference/command/dbStats/#dbStats.dataFileVersion.
• “Main Server” contains 39 collections, “CentOS7” contains 38 collections, and “Ubuntu” contains 32 collections.

Some things you could try to determine what happened:

• Increase the verbosity of mongorestore using the -vvvvv option (see https://docs.mongodb.com/manual/reference/program/mongorestore/#cmdoption--verbose)
• Use the --stopOnError option to halt import when an error is encountered (see https://docs.mongodb.com/manual/reference/program/mongorestore/#cmdoption--stopOnError)
• Check the output of show collections between the three servers to find out what collections is not present in the restored deployment. Note that you can restore individual collection by running mongorestore --db <database> --collection <collection> <dump directory>/<database>/<collection>.bson (see https://docs.mongodb.com/manual/reference/program/mongorestore/#restore-a-collection)
• Are writes still being performed when mongodump was run?

Best regards,
Kevin

​