Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Message from discussion mongodump big bug
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Mark Fossen  
View profile  
 More options Nov 16 2012, 3:40 pm
From: Mark Fossen <m...@mover.io>
Date: Fri, 16 Nov 2012 12:40:11 -0800 (PST)
Local: Fri, Nov 16 2012 3:40 pm
Subject: Re: mongodump big bug

Unfortunately this is a bug in the mongodump code. When a collection is
created with the command line it creates the collection with all the
options for a capped table but the values are null. you can see this
using db.system.namespaces.find():

{ "name" : "mydatabase.mycollection", "options" : { "create" : "api_apps",
"capped" : null, "size" : null, "max" : null } }

The offending table should have and entry like above. There are a few ways
to fix this. The easiest one I found is to do the following:
1. 'mongoexport -d mydatabase --collection mycollection > mycollection.json'
2. mongo cli 'db.mycolleciton.renameCollection("myoldcollection")
3. 'mongoimport -d mydatabase --collection mycollection --file
mycollection.json

The result namespace entry will be
{ "name" : "mydatabase.mycollection"}

Of course, on an active collection, you may not have this opportunity.
Really this is a band-aid anyway. The real fix should be in the mongodump
code when it creates the metadata file. We can see the offender
in https://github.com/mongodb/mongo/blob/master/src/mongo/tools/dump.cpp

string sns = db + ".system.namespaces";
cursor = conn( true ).query( sns.c_str() , Query() , 0 , 0 , 0 ,
QueryOption_SlaveOk | QueryOption_NoCursorTimeout );
while ( cursor->more() ) {
   BSONObj obj = cursor->nextSafe();
   const string name = obj.getField( "name" ).valuestr();                  
                       <-------------- line 197 name is null here.
   if (obj.hasField("options")) {
     collectionOptions[name] =
obj.getField("options").embeddedObject().getOwned();  <----------------line
199 null name is undefined.
    }

I am pretty sure a small fix here could resolve this issue going forward.
If I had the time or the chops I would fix it. Hopefully there is someone
who can take a look into this.

--
Mark

On Thursday, November 15, 2012 6:43:21 PM UTC-7, Ketema Harris wrote:

> I found that if I manually edit the file manually to remove undefined, the
> restore still fails with:

> User Assertion: 15934:JSON object size didn't match file size

> On Thursday, September 27, 2012 4:27:12 PM UTC-4, Mathias Stearn wrote:

>> I've tracked this down to something that was recently fixed in master:
>> https://jira.mongodb.org/browse/SERVER-7104. I requested that it be
>> backported to 2.2.

>> For now, a possible workaround would be to manually edit the json files
>> to remove the offending undefineds.

>> On Friday, September 7, 2012 1:30:55 PM UTC-4, Thomas wrote:

>>> Indeed (i just thought that, because the completion on the console did
>>> not work for namespaces) :)

>>> Well for the collection message_queue i have this metadata genrated by
>>> mongodump:
>>> {options : { "create" : "message_queue", "capped" : true, "size" : 1024,
>>> undefined }, indexes:[{ "v" : 1, "key" : { "time" : 1 }, "unique" : true,
>>> "ns" : "nadb.message_queue", "sparse" : true, "name" : "time_1" }]}

>>> And in the db.system.namespaces.find() : (for our message_queue part
>>> only)
>>> { "name" : "nadb.message_queue", "options" : { "create" :
>>> "message_queue", "capped" : true, "size" : 1024, "max" : null } }
>>> { "name" : "nadb.message_queue.$time_1" }

>>> db.message_queue.stats() produces :
>>> > db.message_queue.stats()
>>> {
>>> "ns" : "nadb.message_queue",
>>> "count" : 63,
>>> "size" : 2796,
>>> "avgObjSize" : 44.38095238095238,
>>> "storageSize" : 4096,
>>> "numExtents" : 1,
>>> "nindexes" : 1,
>>> "lastExtentSize" : 4096,
>>> "paddingFactor" : 1,
>>> "systemFlags" : 0,
>>> "userFlags" : 0,
>>> "totalIndexSize" : 8176,
>>> "indexSizes" : {
>>> "time_1" : 8176
>>> },
>>> "capped" : true,
>>> "max" : 2147483647,
>>> "ok" : 1
>>> }

>>> Thank you,

>>> Thomas.

>>> Le vendredi 7 septembre 2012 19:18:35 UTC+2, Mathias Stearn a écrit :

>>>> Every database has a system.namespaces collection. Note that there is
>>>> no "systems" collection, the actual name of the collection is
>>>> "system.namespaces". Please try running db.system.namespaces.find()

>>>> On Thursday, September 6, 2012 3:55:03 AM UTC-4, Thomas wrote:

>>>>> I don't have any namespaces in the system collection ...

>>>>> Le mercredi 5 septembre 2012 23:03:35 UTC+2, Mathias Stearn a écrit :

>>>>>> What does db.system.namespaces.find() say?

>>>>>> On Wednesday, September 5, 2012 10:21:54 AM UTC-4, Thomas wrote:

>>>>>>> Hello,
>>>>>>> I have a big backup issue with mongo 2.2 under ubuntu 12.04.

>>>>>>> When I backup my whole database (mongodump on the db) it badly
>>>>>>> exports some of my collections (the metadata contains undefined values)
>>>>>>> Exemple:
>>>>>>> > db.testairq.stats()
>>>>>>> {
>>>>>>> "ns" : "test.testairq",
>>>>>>> "count" : 0,
>>>>>>> "size" : 0,
>>>>>>> "storageSize" : 5713248256,
>>>>>>> "numExtents" : 32,
>>>>>>> "nindexes" : 1,
>>>>>>> "lastExtentSize" : 961069056,
>>>>>>> "paddingFactor" : 1.5299999998863023,
>>>>>>> "systemFlags" : 0,
>>>>>>> "userFlags" : 0,
>>>>>>> "totalIndexSize" : 8176,
>>>>>>> "indexSizes" : {
>>>>>>> "_id.loc___id.time_1" : 8176
>>>>>>> },
>>>>>>> "ok" : 1
>>>>>>> }

>>>>>>> And the resulted metadata :
>>>>>>> {options : { "create" : "testairq", undefined, undefined, undefined,
>>>>>>> "autoIndexId" : false }, indexes:[{ "v" : 1, "key" : { "_id.loc" : "2d",
>>>>>>> "_id.time" : 1 }, "unique" : true, "ns" : "test.testairq", "name" :
>>>>>>> "_id.loc___id.time_1", "sparse" : true }]}

>>>>>>> mongorestore failed then to restore with this message
>>>>>>> Wed Sep  5 16:17:12 dump/test/testairq.bson
>>>>>>> Wed Sep  5 16:17:12 going into namespace [test.testairq]
>>>>>>> Wed Sep  5 16:17:12   Assertion failure b.empty()
>>>>>>> src/mongo/db/json.cpp 645
>>>>>>> 0xad2211 0x67fced 0xa7daa5 0x56aa20 0x569cd1 0x569cd1 0x56db4a
>>>>>>> 0xb3b078 0xb41712 0x553f92 0x7f3a3c43076d 0x553e09
>>>>>>>  mongorestore(_ZN5mongo15printStackTraceERSo+0x21) [0xad2211]
>>>>>>>  mongorestore(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0x67fced]
>>>>>>>  mongorestore(_ZN5mongo8fromjsonEPKcPi+0x585) [0xa7daa5]
>>>>>>>  mongorestore(_ZN7Restore9drillDownEN5boost11filesystem210basic_pathISsNS1_1 1path_traitsEEEbbb+0xff0)
>>>>>>> [0x56aa20]
>>>>>>>  mongorestore(_ZN7Restore9drillDownEN5boost11filesystem210basic_pathISsNS1_1 1path_traitsEEEbbb+0x2a1)
>>>>>>> [0x569cd1]
>>>>>>>  mongorestore(_ZN7Restore9drillDownEN5boost11filesystem210basic_pathISsNS1_1 1path_traitsEEEbbb+0x2a1)
>>>>>>> [0x569cd1]
>>>>>>>  mongorestore(_ZN7Restore5doRunEv+0xdda) [0x56db4a]
>>>>>>>  mongorestore(_ZN5mongo8BSONTool3runEv+0x448) [0xb3b078]
>>>>>>>  mongorestore(_ZN5mongo4Tool4mainEiPPc+0x1712) [0xb41712]
>>>>>>>  mongorestore(main+0x32) [0x553f92]
>>>>>>>  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)
>>>>>>> [0x7f3a3c43076d]
>>>>>>>  mongorestore(__gxx_personality_v0+0x421) [0x553e09]
>>>>>>> assertion: 0 assertion src/mongo/db/json.cpp:645

>>>>>>> Is this a known bug or do I have to open a new issue ?
>>>>>>> It's really a big issue since my backup system is totally based on
>>>>>>> this mechanism.

>>>>>>> Thank you,

>>>>>>> Thomas.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.