mongodump and mongoexport export only partial records

[thetuxracer]

I tried to get:

1. mongodump of the entire database.

2. mongoexport of a specific collection.

3. mongodump of a specific collection.

In all the three cases, only 101 records were dumped, even when the amount exceeds 10000 records. I tried different collections, all with the same result. For collections that have records less than 100, they export fine. Even collections with 126 records, in one case exported only 101 records. This is very frustrating, because I need to fix a bug on a production db, which is not possible unless I get the complete dump. The logs show the following:

http://paste.ubuntu.com/578339/

I run mongodb 1.6.5, on a 64 bit machine.

Thanks.

[Nat]

I've never seen it happened before
- Can you attach the output of mongodump/mongoexport?
- Can you make sure your data is not corrupted? Probably check it by
db.<collectionName>.validate()

[thetuxracer]

Hello Nat,

I tried validating the collection, and then taking a dump adn export, but that did not export all the records either, just 101 records.

This is the OP of db.collection.validate:

{
"ns" : "prolinkd.gnip",
"result" : "\nvalidate\n  firstExtent:2:60fe300 ns:prolinkd.gnip\n  lastExtent:5:12c2ff00 ns:prolinkd.gnip\n  # extents:28\n  datasize?:1861925724 nrecords?:973525 lastExtentSize:360318720\n  padding:1\n  first extent:\n    loc:2:60fe300 xnext:2:6123600 xprev:null\n    nsdiag:prolinkd.gnip\n    size:21248 firstRecord:2:60fe3b0 lastRecord:2:61030d8\n  973525 objects found, nobj:973525\n  1877502124 bytes data w/headers\n  1861925724 bytes data wout/headers\n  deletedList: 0100000000000000001\n  deleted: n: 4 size: 259026452\n  nIndexes:2\n    prolinkd.gnip.$_id_ keys:973525\n    prolinkd.gnip.$verb_1 keys:973525\n",
"ok" : 1,
"valid" : true,
"lastExtentSize" : 360318720
}

http://paste.ubuntu.com/578360/ is the output of 

$ mongodump -o ~/db_dump

This is the output of mongodump a particular collection only:

mongodump -d foo -c gnip -o ~/gnip
connected to: 127.0.0.1
DATABASE: foo to /root/gnip/foo
foo.gnip to /root/gnip/foo/gnip.bson
101 objects

This is the OP of mongoexport of a particular collection

$ mongoexport -d prolinkd -c gnip -o ~/gnip.bson
connected to: 127.0.0.1
exported 101 records

Why is the behavior so weird? 

Thanks.

[Nat]

- can you make sure you have the same version of mongodump/mongoexport
(1.6.5)?
- db.<collection>.validate is the output from prolinkd database while
the output in pastebin is from foo database. Is it intentional?

On Mar 10, 10:26 pm, thetuxracer <thetuxra...@gmail.com> wrote:
> Hello Nat,
>
> I tried validating the collection, and then taking a dump adn export, but
> that did not export all the records either, just 101 records.
>
> This is the OP of db.collection.validate:
>
> {
> "ns" : "prolinkd.gnip",
> "result" : "\nvalidate\n  firstExtent:2:60fe300 ns:prolinkd.gnip\n
>  lastExtent:5:12c2ff00 ns:prolinkd.gnip\n  # extents:28\n
>  datasize?:1861925724 nrecords?:973525 lastExtentSize:360318720\n
>  padding:1\n  first extent:\n    loc:2:60fe300 xnext:2:6123600 xprev:null\n
>    nsdiag:prolinkd.gnip\n    size:21248 firstRecord:2:60fe3b0
> lastRecord:2:61030d8\n  973525 objects found, nobj:973525\n  1877502124
> bytes data w/headers\n  1861925724 bytes data wout/headers\n  deletedList:
> 0100000000000000001\n  deleted: n: 4 size: 259026452\n  nIndexes:2\n  
>  prolinkd.gnip.$_id_ keys:973525\n    prolinkd.gnip.$verb_1 keys:973525\n",
> "ok" : 1,
> "valid" : true,
> "lastExtentSize" : 360318720
>
> }
>

> http://paste.ubuntu.com/578360/is the output of

[thetuxracer]

On Thursday, March 10, 2011 8:11:39 PM UTC+5:30, Nat wrote:

- can you make sure you have the same version of mongodump/mongoexport
(1.6.5)?

How do I check the version? I had installed it via apt-get, so I reckon it should be the same.

 

- db.<collection>.validate is the output from prolinkd database while
the output in pastebin is from foo database. Is it intentional?

They are the same :p I was trying to hide the name of the db, and forgot to change in one of them.

[thetuxracer]

Just as an aside, I got mongodump to work, by shutting down mongodb, and executing:

$ mongodump --dbpath /var/lib/mongodb/

It dumped all the collections correctly. 

But still, I would like to get to the root of the problem mentioned in the OP.

[Eliot Horowitz]

Few things:
- can you start mongod with -vv and try the mongodump and send the output
- from the shell, can you do
db.<collection name>.stats()
db.<collection name>.count()
db.<collection name>.find().itcount()
db.<collection name>.find().snapshot().itcount()

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

[thetuxracer]

Hi Nat,

I run mongod with an op logging level of 7, from the mongod.conf. Is it the same as running mongod with the -vv option? I am not sure, so I stopped it, and started mongod with --v option.

This is part of the mongodump:

http://paste.ubuntu.com/578744/

There is a 32 bit slave following the master, and I think it has exceeded the 2 G limit. So I stopped the slave, and this was what part of the logs contain:

http://paste.ubuntu.com/578745/

> db.collection.stats():

{
"ns" : "database.collection",
"count" : 990126,
"size" : 1895649704,
"avgObjSize" : 1914.5540102976793,
"storageSize" : 2136533504,
"numExtents" : 28,
"nindexes" : 1,
"lastExtentSize" : 360318720,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 42106880,
"indexSizes" : {
"_id_" : 42106880
},
"ok" : 1
}

> db.collection.count:

990255

> db.collection.find().itcount() (which took a _long_ time to run:

990910

> db.collection.find().snapshot().itcount()

991675

Thanks,

KDP. 

[Nat]

Can you just extract the part of log which is relevant to the mongoexport? I can't find it.

---

From: thetuxracer <thetu...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Fri, 11 Mar 2011 02:01:45 -0800 (PST)

To: <mongod...@googlegroups.com>

ReplyTo: mongod...@googlegroups.com

Subject: Re: [mongodb-user] Re: mongodump and mongoexport export only partial records

> db.collection.find().itcount() (which took a_long_ time to run:

990910

> db.collection.find().snapshot().itcount()

991675

Thanks,

KDP. 

--

[thetuxracer]

This is the logfile, commencing the dump: http://paste.ubuntu.com/578802/ and http://paste.ubuntu.com/578822/ oplogging is set at level 7.

I took a mongodump, by starting mongod manually from the command, and all the collections were dumped properly. Running mongod as a init script, however gives a problem.

This is the logfile of the dump, when mongod is started manually from the terminal. http://paste.ubuntu.com/578806/

[Nat]

mongod didn't get shutdown properly. So there are /tmp/mongodb-27017, /
tmp/mongodb-28017 lying around causing mongod not to start up
properly. You either need to delete those files or set proper
permission as you might start mongod from different users

On Mar 11, 9:30 pm, thetuxracer <thetuxra...@gmail.com> wrote:
> This is the logfile, commencing the dump:http://paste.ubuntu.com/578802/

>  andhttp://paste.ubuntu.com/578822/oplogging is set at level 7.

[thetuxracer]

I shutdown mongo, removed the two socket files, and restarted mongo by typing:

$ service mongodb start

and a mongodump still only exports 101 records, as mentioned in OP.

[Nat]

Can you post your config setting for your service?

---

From: thetuxracer <thetu...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Fri, 11 Mar 2011 07:00:48 -0800 (PST)

To: <mongod...@googlegroups.com>

ReplyTo: mongod...@googlegroups.com

Subject: [mongodb-user] Re: mongodump and mongoexport export only partial records

I shutdown mongo, removed the two socket files, and restarted mongo by typing:

$ service mongodb start

and a mongodump still only exports 101 records, as mentioned in OP.

--

[Tim Hawkins]

wild guess, when you run from commandline and from the service, have you checked they are using the same config file?

in particular are they using the same dbdir.

[thetuxracer]

Yes, they use the same database directory. Oh yes, your wild guess is correct. Using the same config file gives the error. I guess the problem lies in the mongodb.conf file This is my current mongodb.conf http://paste.ubuntu.com/578859/

I get the same problems, after commenting out maxConns, Running the server manually, without passing the config file as a parameter works as expected. passing the config file displays the same behavior, in both methods ie running it from terminal, or running it as an init script.

I tried commenting out the ${DAEMON_OPTS} --${CONF} but mongodb doesnt startup.

[Nat]

you should comment out "nocursors"

On Mar 11, 11:22 pm, thetuxracer <thetuxra...@gmail.com> wrote:
> Yes, they use the same database directory. Oh yes, your wild guess is
> correct. Using the same config file gives the error. I guess the problem

> lies in the mongodb.conf file This is my current mongodb.confhttp://paste.ubuntu.com/578859/

[Stanislav Prusac]

I tested and reproduced same problem in mongodb 2.2.3. /2.0.8

If you have wrong configured firewall for mongodump then mongodump only exports 101 documents. Try shutdown firewall or reconfigure it.