MongoDB 3.0 with WiredTiger restore Ends with unexpected EOF

[Boas Enkler]

I've a quite big collection in a mongo database. The Database runs in a cluster of 3 servers With

mongodumb -host localhost Z:\MongoDbBackup

i create a backup which i want to restore on my local machine, but the restore ends with unexpected EOF

Here some properties from the source database

  "count" : NumberInt(726859), 
  "size" : NumberInt(517082591), 
  "avgObjSize" : NumberInt(711), 
  "storageSize" : NumberInt(241795072), 
  "capped" : false, 

Any hints of what may be going wrong or i could try?

[Boas Enkler]

i also see before some E110000 DUPLICATE KEY Errors . as this collection doesn't have strong consistency conditions this isn't a problem for itself, but perhaps its related to the following EOF error ?

[Dmitry Romanenko]

Same trouble here. It seem to be happening on index creation stage - "createIndex error: EOF".

If I change storage engine it back to  mmapv1 then it passes successfully but takes awhile to do mongorestore.

mongorestore without indexes works perfect without any troubles and very fast. But well, we need indexes.

Probably worth to try create indexes manually till we find which index causes it.

This is how you can restore without indexes:

mongorestore -host localhost Z:\MongoDbBackup --noIndexRestore

[Dmitry Romanenko]

After some investigation it leaded me to conclusion that the problem is in multithreaded mongorestore. If I create indexes manually - no problem, if I run mongorestore for each db one-by-one  manually - no problem, if I run mongorestore for whole path with several databases - crash of mongorestore.

May be related to some OS resource limitation or something.

[Boas Enkler]

In my Case i get the error also when restoring without index and a single collection :-(

[Boas Enkler]

Code hier eingeben...

I now tried it with a complete newly created database.

i resotred with --NoIndexRestore, and only a given collection.

Same error :-(

As this error prevents restoring some parts our system in a emergency case i don't feel well with this sate :-( What can i do ?

[Dmitry Romanenko]

Seems like the trouble is related to space/RAM. Probably you're getting out of RAM or space.

For me it was RAM (index creation for 70GB data set requires like 7GB RAM to be used, if it uses more - it dies).

On Wednesday, 11 March 2015 08:30:07 UTC-4, Boas Enkler wrote:

[Dmitry Romanenko]

Try to check your RAM/space usage within running mongorestore.

On Wednesday, 11 March 2015 08:30:07 UTC-4, Boas Enkler wrote:

[Mike O'Brien]

Hi guys,

Can you try running mongorestore with the "-vvvv" flag (to turn on the extra debug logging output) and post its output here?

Also - can you post logs from the server during the time that the import was running?

"Unexpected EOF" might suggest that a connection or file handle was closed prematurely.

[Mike O'Brien]

Also, if you could provide details on the indexes that are being restored, that may be helpful as well.

Dmitry - you mentioned that after switching the server's storage engine from WT to mmapv1, the restore doesn't have any issues, which suggests that the problem has something to do with how the server itself is handling the inserts (or index creations).

Thanks for the report!

[Boas Enkler]

Well in my case i disable indexcreation on restore.

I Also used an complete new empty database.

i attched the output as screenshot

is there a way to force a non parallel import?

[Boas Enkler]

btw ramusage was only 1-4 MB ;)

[Mike O'Brien]

Boas,

You can force it to only restore a single collection at a time by using --numParallelCollections=1. You can turn off all the parallelism by using --numThreads=1 and --numInsertionWorkersPerCollection=1.

Does the same collection fail every time? Specifically, if you restore *just* GeolocationSet.bson by itself, does it ever succeed?

[Asya Kamsky]

Are you sure your dumped file for the collection it's failing on is
... whole? What is its size?

You can try bsondump on it (into /dev/null probably) and see if that
gives an error too - if your mongodump was interrupted you could
literally have an incomplete file in which case there is nothing that
restore can do.

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/153ce31d-8122-4085-a412-50a593f6f35e%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!

[Mohamed Ayoub Benjelloun]

Hi,

You have to check your mongo logs to know the exact error,
If it can help someone, here is what i found in my case using a Ubuntu-12.04 and a MongoDB-3.0.6:

1 - here is the retrieved error when executing the mongorestore
---
$ mongorestore DumpFolderPath --username admin
...
2015-09-10T15:48:21.701+0200    [#######################.]  DbName.CollectionName  17.2 GB/17.4 GB  (98.7%)
2015-09-10T15:48:24.702+0200    [#######################.]  DbName.CollectionName  17.3 GB/17.4 GB  (99.0%)
2015-09-10T15:48:27.703+0200    [#######################.]  DbName.CollectionName  17.3 GB/17.4 GB  (99.3%)
2015-09-10T15:48:30.704+0200    [#######################.]  DbName.CollectionName  17.3 GB/17.4 GB  (99.5%)
2015-09-10T15:48:33.705+0200    [#######################.]  DbName.CollectionName  17.4 GB/17.4 GB  (99.8%)
2015-09-10T15:48:35.984+0200    restoring indexes for collection DbName.CollectionName from metadata
2015-09-10T15:57:17.155+0200    Failed: restore error: DbName.CollectionName: error creating indexes for DbName.CollectionName: createIndex error: EOF

2 - here is the log details for the same date/hour
---
$ cat /var/log/mongodb/mongod.log
...
2015-09-10T13:56:58.004Z I -        [conn29]   Index Build: 6208000/26065549 23%
2015-09-10T13:57:01.008Z I -        [conn29]   Index Build: 6261000/26065549 24%
2015-09-10T13:57:04.002Z I -        [conn29]   Index Build: 6319100/26065549 24%
2015-09-10T13:57:11.088Z I -        [conn29]   Index Build: 6351300/26065549 24%
2015-09-10T13:57:16.352Z F -        [conn29] out of memory.
---

3 - I checked and found a MongoDB3 issue in the same context (#SERVER-17834), the issue was rejected because 1Go of SWAP was considered as very low and leading to this abnormal behavior

4 - my swap memory size was as low as described on this ticket
$ swapon -s
Filename                Type        Size    Used    Priority
/dev/mapper/vmname--vg-swap_1 partition    1044476    19872    -1

5 - Thus, increasing my swap memory size has solved the issue (here is a tutorial explaining how to do that on Ubuntu)

[Mohamed Ayoub Benjelloun]

PS : It can also be fixed by changing the storage.wiredTiger.engineConfig.cacheSizeGB value in your /etc/mongod.conf file