mongodb old records missing

222 views
Skip to first unread message

tob...@gmail.com

unread,
Aug 15, 2015, 6:33:16 PM8/15/15
to mongodb-user
I have a mongodb database containing a collection that has remained unchanged for a period of time -- the underlying .ns and .0, .1 etc files have not been modified for months. Up until a few weeks ago, I had no problem reading records from the collection. There were thousands of records in the collection.

However, today when I attempted to read the records, many of them appeared missing -- e.g. records that I expected to be there were not, although some of the records were available. Now, there appear to be only 250 records. =

I copied the database files and did a repair of the (copied) database, which ran fine apparently, but only the 250 records appeared, and cannot find the others.

I can see manually that the bSON binary files in the database directory associated with this database still contain the data that I want to access (by grepping the contents of the database files, I can see some unique text strings that are associated just with missing records -- of course they do, since the files haven't been modified for months).

I cannot think of anything that changed between now and the last time that I successfully was able to access the records, except that the DB process crashed once. It was brought back up immediately. Although, as I say, it doesn't appear that anything modified the files associated with the database where these records are.

What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan

Chris De Bruyne

unread,
Aug 17, 2015, 3:26:20 AM8/17/15
to mongodb-user
Can you give some more info?

Like which query are you doing to find the documents, what is the structure of the docs, are there indexes on this collection?

Ankit Kakkar

unread,
Aug 17, 2015, 12:47:08 PM8/17/15
to mongodb-user
Hello Tobjan,

Thanks for reaching out to us. To assist us in the investigation of this case, please provide us with following information:

1) Describe your MongoDB deployment (standalone, replica set, or sharded cluster)
2) What version of MongoDB are you running? (You can check it with db.version() in the mongo shell)
3) Which query did you use to find or count the documents? Can you please run that query with explain() and send us the output?
4) Output of db.collection.getIndexes() for the collection where documents appear to be missing.

Regards,
ankit

Asya Kamsky

unread,
Aug 17, 2015, 6:18:36 PM8/17/15
to mongodb-user
In addition to the information below, can you please also check if you are running with journaling disabled?

You did say:
> I cannot think of anything that changed between now and the last time that I successfully was able to access the records, except that the DB process crashed once. 
> It was brought back up immediately. Although, as I say, it doesn't appear that anything modified the files associated with the database where these records are.

If your mongod runs with journaling disabled (which is *NOT* the default) then any crash could cause data loss.

Asya


--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
 To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/6ca93621-854a-4bcc-87bc-26faef6b6de7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

tob...@gmail.com

unread,
Aug 18, 2015, 12:54:07 PM8/18/15
to mongodb-user
Hi Ankit --

Thanks so much for your reply -- it's really appreciated. 

Here are some answers to your questions:

1) it's a standalone deployment, with no replica set (I'm using journaling though)

2) version is 2.4.6

3) Here is the output of explain (done through pymongo):

{u'allPlans': [{u'cursor': u'BasicCursor',
   u'indexBounds': {},
   u'n': 0,
   u'nscanned': 279,
   u'nscannedObjects': 279}],
 u'cursor': u'BasicCursor',
 u'indexBounds': {},
 u'indexOnly': False,
 u'isMultiKey': False,
 u'millis': 176,
 u'n': 0,
 u'nChunkSkips': 0,
 u'nYields': 1,
 u'nscanned': 279,
 u'nscannedAllPlans': 279,
 u'nscannedObjects': 279,
 u'nscannedObjectsAllPlans': 279,
 u'scanAndOrder': False,
 u'server': myserver@myport}

(I think is saying 279 here because that is the number of records shown -- not 250, like I said before, that was a mistake.)

4) Here is the index information (also through pymongo):

{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
 u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1), (u'uploadDate', -1)],
  u'v': 1},
 u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
  
As you can see, this is from a ".files" collection used to manage a gridfs instance. 


Also I think I have some additional information.   I copied the database files to a new location and ran a full repair on the whole database.   IN the new repaired version, there were many fewer files .ns files.   As is as if some records were removed using e.g. gridfs.GridFS.remove ... so in the new (compacted) version the data was finally deleted.  Maybe its somehow possible that the records were removed from the gridfs collection by someone running a remove operation?   I don't see any evidence of such an operation in the logs but perhaps it's possible?     I've looked carefully at the logs and am not seeing anything obvious, but maybe dont' know what to look for. 

Of course, that I guess does not modify the underlying files on the original collection where the (hypothetical) remove would have been run.   If this indeed the case, is there some way to get the data back?  After all, I do have the original .ns files.  

Thanks!
Tob

Asya Kamsky

unread,
Aug 18, 2015, 1:53:13 PM8/18/15
to mongodb-user
Do you have the logs from around the time the database crashed?   When was that relative to when you noticed the records missing?

Since you had journaling on, and you were able to restart mongod without any errors, I would rule out the crash *causing* records to disappear, which leaves you with the possibility that they were deleted.   If they were deleted and you don't have any backups from *before* they were deleted, then there's not much that can be done to recover the data - sure, it's possible to write code to literally look through the deleted list, but if any data was inserted after the deletes then that space would be reused and the old data would be overwritten forever.   :(

Asya


--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com.

tob...@gmail.com

unread,
Aug 18, 2015, 1:56:37 PM8/18/15
to mongodb-user


On Tuesday, August 18, 2015 at 1:53:13 PM UTC-4, Asya Kamsky wrote:
Do you have the logs from around the time the database crashed?   When was that relative to when you noticed the records missing?


Yes I have all the logs going back for a year.   I noticed the records missing the day that I wrote the help question originally (several days ago).   The crash happened several weeks ago. 


Since you had journaling on, and you were able to restart mongod without any errors, I would rule out the crash *causing* records to disappear, which leaves you with the possibility that they were deleted.   If they were deleted and you don't have any backups from *before* they were deleted, then there's not much that can be done to recover the data - sure, it's possible to write code to literally look through the deleted list, but if any data was inserted after the deletes then that space would be reused and the old data would be overwritten forever.   :(

The backups are not from before unfortunately.   But wouldn't there be some record in the logs if there was a deleation? 

We haven't written any records since the deletes.   (The files haven't changed for months.)

Asya Kamsky

unread,
Aug 19, 2015, 2:53:46 PM8/19/15
to mongodb-user
The only CRUD operations that would be logged would be those that take longer than 100ms.

Might look something like this:

2015-07-14T12:10:34.559-0500 I WRITE    [conn4822] remove blackbox.bigcoll query: { _id: { $gte: 10000.0 } } ndeleted:90000 keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: { acquireCount: { r: 12331, w: 12331 } }, Database: { acquireCount: { w: 12331 } }, Collection: { acquireCount: { w: 12331 } } } 249168ms

Of course this was a huge mass remove of giant docs which is why it took so long.   If the number of documents deleted was small (like if they were deleted one at a time instead of via single command) then it's much less likely they would have taken longer than 100ms.

I would suggest searching through the logs with equivalent command to:

grep 'remove <dbname>' *.log.*

where you would replace <dbname> with the name of your actual DB and replace *.log.* with path to your log files.

Asya


To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com.

tob...@gmail.com

unread,
Aug 19, 2015, 6:14:04 PM8/19/15
to mongodb-user
Thanks for the suggestion.   The logs don't appear to show any removals on that database since Oct. 2014.    At least as far as these logs go, it doesn't look to me like a remove was done.    

Any suggestions for what to do now?   I can definitely see traces of the data I want to get at in the (unmodified) database files.  Is there no way to get at the data in them? 

Thanks!
T

Asya Kamsky

unread,
Aug 21, 2015, 9:49:36 AM8/21/15
to mongod...@googlegroups.com
Some of the records are in a free list and haven't been overwritten yet.  It may be possible to get them back. 

Since you're using version 2.4.x I believe that there is code that will walk through the DB files and dump out every document it finds there.  

Let me dig around for it - obviously there is (a) no guarantee that this will restore all of them and (b) did you say these were GridFS chunks - if so then you won't get back a valid file unless all chunks from that file are restored (along with corresponding files record). 

Asya

tob...@gmail.com

unread,
Aug 21, 2015, 11:33:02 AM8/21/15
to mongodb-user
Great thanks so much for the help!
T
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

Asya Kamsky

unread,
Aug 21, 2015, 2:02:27 PM8/21/15
to mongodb-user
I'm assuming that you start up mongod with the original files (ones you did not repair but saved when you first noticed that something was amiss):

Okay, first, it would be good to know which collections are missing documents:

You'd mentioned having 279 documents in files collection - how many are in chunks collection?   Do they "cross-verify"?

You can see what each has here: http://docs.mongodb.org/manual/reference/gridfs/

So you can tell for each file how many chunk documents there should be and for each chunk document you can tell which file it belongs to and in what order.

Some possible ways to verify this:
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}}, {$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).

If you have 279 files in your fs.files collection, then you should get back 279 documents from this aggregation, each corresponding to a "file" in gridFS.

How many files do you expect to have?

You can now run this command (very slow):

db.fs.files.validate(true)


and
db.fs.chunks.validate(true)

This will give output including fields described here: http://docs.mongodb.org/manual/reference/command/validate/#output

You are interested in the deleted records (http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount and the next field, deletedSize).
Basically these are the records that can be potentially recovered - can you provide output to above experiments so we don't waste time trying to recover data that's not still there?

Asya


--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com.

tob...@gmail.com

unread,
Aug 21, 2015, 9:30:56 PM8/21/15
to mongodb-user
Asya, 

Thanks for the help.  Ok, here are the answers to your questions: 

1) The aggregation of the chunk collection produces 279 documents.   I don't recall how many there were before the ones I want went missing.   However, I guess there more than 279.  I don't know exactly how many more, but I expect on the order of several hundred at least (but again, I'm not sure).    The collection does verify, in that all the chunks in the chunks collection are accounted for by a single file in the files collection.  I computed the expected number of chunks for each file by dividing length by chunk size for each file record; that number exactly equals the count of chunks for that file as produced by the aggregation in the chunk collection.

2) Ok, here's the info from the validate commands:

db.files.validate(true) produced: 

{
        "ns" : "mydb.myfs.files",
        "firstExtent" : "0:19000 ns:mydb.myfs.files",
        "lastExtent" : "7:67757000 ns:mydb.myfs.files",
        "extentCount" : 5,
            
            <snip "extents">
            
        "datasize" : 24075568,
        "nrecords" : 279,
        "lastExtentSize" : 12902400,
        "padding" : 1,

            <snip some other stuff>

        "objectsFound" : 279,
        "invalidObjects" : 0,
        "bytesWithHeaders" : 24080032,
        "bytesWithoutHeaders" : 24075568,
        "deletedCount" : 3,
        "deletedSize" : 12008944,
        "nIndexes" : 3,
        "keysPerIndex" : {
                "mydb.myfs.files.$_id_" : 279,
                "mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
                "mydb.myfs.files.$timestamp_1" : 279
        },
        "valid" : true,
        "errors" : [ ],
        "ok" : 1
}

db.chunks.validate(true) produced:

db.chunks.validate(true)
{
        "ns" : "mydb.myfs.chunks",
        "firstExtent" : "0:5000 ns:mydb.myfs.chunks",
        "lastExtent" : "28:2000 ns:mydb.myfs.chunks",
        "extentCount" : 43,
        
            <snip "extents">

        "datasize" : 49854830688,
        "nrecords" : 190427,
        "lastExtentSize" : 2146426864,
        "padding" : 1,
        "firstExtentDetails" : {
                "loc" : "0:5000",
                "xnext" : "0:162000",
                "xprev" : "null",
                "nsdiag" : "mydb.myfs.chunks",
                "size" : 8192,
                "firstRecord" : "0:50b0",
                "lastRecord" : "0:6f28"
        },
        "lastExtentDetails" : {
                "loc" : "28:2000",
                "xnext" : "null",
                "xprev" : "27:2000",
                "nsdiag" : "mydb.myfs.chunks",
                "size" : 2146426864,
                "firstRecord" : "28:20b0",
                "lastRecord" : "28:189c20b0"
        },
        "objectsFound" : 190427,
        "invalidObjects" : 0,
        "bytesWithHeaders" : 49857877520,
        "bytesWithoutHeaders" : 49854830688,
        "deletedCount" : 20,
        "deletedSize" : 1733626640,
        "nIndexes" : 2,
        "keysPerIndex" : {
                "mydb.myfs.chunks.$_id_" : 190427,
                "mydb.myfs.chunks.$files_id_1_n_1" : 190427
        },
        "valid" : true,
        "errors" : [ ],
        "ok" : 1
}

I'm not sure whether "deletedCount" of "3" in the files collection would make up for all the deleted records that I expect ... if I interpret it correctly.   It would also not make up for the huge compaction that occurred in the repaired version of the collection (which is much smaller on disk, but still contains 279 records). 

Any thoughts on what to do next?   Thanks again for your help!

T

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

Asya Kamsky

unread,
Aug 22, 2015, 7:00:54 PM8/22/15
to mongod...@googlegroups.com
This is somewhat surprising.  Are there other collections in this database?

You're correct, the deleted number *is* the number of deleted records and that's pretty much the upper bound on what you could conceivably recover out of these files.  

This is the original files, right?   I'm a bit surprised at relatively large number of extents in the files collection relative to number of deleted records.  Also normally there would be an array of deleted records as they are kept in "groups" - I guess that's for regular collections, in gridFS the sizes of records would be more similar...

Are you *sure* there are files missing?  I don't mean by count but by content... You mentioned you were about to find some strings matching something you expected to find in the DB but didn't?

How many dbname.ns files are there and what are their sizes?

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

tob...@gmail.com

unread,
Aug 24, 2015, 9:48:02 AM8/24/15
to mongodb-user
Thanks for the response -- sorry for the delay in reply (was away from computer). 


On Saturday, August 22, 2015 at 7:00:54 PM UTC-4, Asya Kamsky wrote:
This is somewhat surprising.  Are there other collections in this database?

Well, not really,  here is the result of ".collection_names()":

[u'system.indexes', u'myfs.chunks', u'myfs.files']



This is the original files, right?

Yes. 


 

Are you *sure* there are files missing?  I don't mean by count but by content... You mentioned you were about to find some strings matching something you expected to find in the DB but didn't?

Definitely 100% completely sure.    I can use "grep" on the binary files of this database and I see at least parts of the missing records, e.g. some keys that are unique and are exactly what the missing records would have.   Moreover, I (and several others) were happily using this collection as the source a stable restful API for months ... at least, until this problem occurred ... 
 

How many dbname.ns files are there and what are their sizes?

In the original collection there are 90 .ns files, most of which are the standard size "2146435072", but some of which (the early ones of course) are smaller. 

In the repaired collection there are 35 .ns files, same size pattern. 

Thanks again for all the help -- 

T

 

Asya

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

Asya Kamsky

unread,
Aug 24, 2015, 11:19:02 AM8/24/15
to mongodb-user
Okay, well, I guess it can't hurt so you should try to see if anything dumped out with something that reads the raw files ends up helping at all:

Check out this github repo:


This was mostly an exercise in dumping out on-disk format (*note* to anyone reading, this had never been updated since 2.4 so I'm only suggesting it specifically because OP is using 2.4 - using this on later versions without changes will simply give errors).

To build this you will need libbson version from when this was written which I believe is 1.0.   The executable you want to run is called mdbundo, here's its usage.

usage: mdbundo DBPATH DBNAME COLNAME

what it will do is walk through the DB files looking for deleted records and then try to restore them.   I can't remember for sure but I think it will send them to stdout so you can redirect the output of this to a file.    That file should have bson in them so you can use bsondump to view them (which is pointless for chunks but useful for "files") and you can run mongorestore on the output to load it into a new collection so you can then figure out if anything in there is valuable.

Obviously you run it twice on the same DB files, once for myfs.files, once for myfs.chunks.

Again, given the output to validate I don't really expect it to find anything more than a handful of records at most, possibly unusable (since for every "file" every chunk belonging to it must be present to have the full usable file back.

Good luck and let us know what you uncover,
Asya





Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/1e0efcf0-c0e3-4cd5-a9fd-7a21844fe44f%40googlegroups.com.

tob...@gmail.com

unread,
Aug 24, 2015, 6:43:51 PM8/24/15
to mongodb-user
Ok great -- thanks, will try this. 

In terms of getting the right version of libbson, I guess that getting a commit from 


here is the the way to go?   Which commit do you suggest I use? 

Thanks!
T

Asya

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

Asya Kamsky

unread,
Aug 25, 2015, 3:04:16 AM8/25/15
to mongodb-user
I'd suggest 1.0 or whichever version is tagged around the time of last
commit on mdb repo.

If you get the wrong version you'll find out during build.

Asya
>>>>> an email to mongodb-user...@googlegroups.com.
>>>>> To post to this group, send email to mongod...@googlegroups.com.
>>>>> Visit this group at http://groups.google.com/group/mongodb-user.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "mongodb-user"
>>> group.
>>>
>>> For other MongoDB technical support options, see:
>>> http://www.mongodb.org/about/support/.
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "mongodb-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to mongodb-user...@googlegroups.com.
>>> To post to this group, send email to mongod...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/mongodb-user.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/mongodb-user/1e0efcf0-c0e3-4cd5-a9fd-7a21844fe44f%40googlegroups.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/f19c3639-e220-4e27-a0bb-f1b808df823b%40googlegroups.com.

tob...@gmail.com

unread,
Aug 25, 2015, 11:51:18 AM8/25/15
to mongodb-user
Hi -- 

Having trouble compiling mdb.     I'm at commit e7d018cad20f2a6aa62aea5db088a8e7b05a190d of libbson.   Is that too early/late? 

To get the make to work at all, I included the following line in all the compilation steps:
     -I /usr/local/include/libbson-1.0

Is this right?   

Even so, here's the compilation error: 

/tmp/ccanf01B.o: In function `db_init':
/home/brejicz/mdb/mdb.c:177: undefined reference to `bson_strdup_printf'
/home/brejicz/mdb/mdb.c:179: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:182: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:189: undefined reference to `bson_strdup_printf'
/home/brejicz/mdb/mdb.c:191: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:195: undefined reference to `bson_realloc'
/home/brejicz/mdb/mdb.c:197: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:218: undefined reference to `bson_free'
/tmp/ccanf01B.o: In function `record_bson':
/home/brejicz/mdb/mdb.c:608: undefined reference to `bson_init_static'
/tmp/ccvrjSqo.o: In function `main':
/home/brejicz/mdb/mdbdump.c:70: undefined reference to `bson_as_json'
/home/brejicz/mdb/mdbdump.c:73: undefined reference to `bson_free'
collect2: ld returned 1 exit status
make: *** [mdbdump] Error 1


Sorry for the detailed questions, but any help would be great. 

Thanks!,
T

tob...@gmail.com

unread,
Aug 25, 2015, 12:03:37 PM8/25/15
to mongodb-user

I should add that the relevant libbson files with those symbols seem to be properly installed, e.g. running

    pkg-config --cflags --libs --libs libbson-1.0

returns

    -I/usr/local/include/libbson-1.0  -L/usr/local/lib -lbson-1.0

and inside /usr/local/include/libbson-1.0, running "grep -R bson_free ."

returns:

    ./bson.h: * Returns: A newly allocated string that should be freed with bson_free().
    ./bson-memory.h:void  bson_free           (void   *mem);
...

Asya Kamsky

unread,
Aug 30, 2015, 9:02:35 PM8/30/15
to mongodb-user
Here is what I know works.

Download 1.0.0 of libbson. You can get it from this url:
https://github.com/mongodb/libbson/archive/1.0.0.zip
Unzip it, cd into it, assuming you have all dependencies installed
(compiler, etc) run
./autoconf.sh
make
sudo make install

Then download mdb the same way from
https://github.com/chergert/mdb/archive/master.zip
unzip and cd into it
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
make mdbundo

Now you should have an executable called ./mdbundo
You may need to do this to run it:
export LD_LIBRARY_PATH=/usr/local/lib

./mdbundo --help
usage: mdbundo DBPATH DBNAME COLNAME

DBPATH is the path to the original files, DBNAME is myfs then you
would do this twice once for files and once for chunks.
If memory serves, the recovered bson will go to stderr so you want to
run it like

./mdbundo /path/to/my/dbfilesdir myfs files > myfs.files.bson
./mdbundo /path/to/my/dbfilesdir myfs chunks > myfs.chunks.bson

Let us know how it goes!

Asya
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/ea02d25c-7c17-47f1-a443-c80956fbc5cd%40googlegroups.com.

tob...@gmail.com

unread,
Aug 31, 2015, 9:33:00 AM8/31/15
to mongodb-user
Thanks -- helpful.    Still getting stuck on the compilation of mdbundo. 

I follow your exact steps (including getting libbson from the source you specified).   But at "make mdbundo" I get a compilation error (see below).   Any thoughts? 

Thanks!
T

cc -o mdbundo -Wall -Werror -O0 -ggdb -I/usr/local/include/libbson-1.0  -L/usr/local/lib -lbson-1.0   mdb.c mdb.h mdbundo.c
/tmp/cc1rVoR4.o: In function `db_init':
/home/brejicz/src/mdb-master/mdb.c:177: undefined reference to `bson_strdup_printf'
/home/brejicz/src/mdb-master/mdb.c:179: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:182: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:189: undefined reference to `bson_strdup_printf'
/home/brejicz/src/mdb-master/mdb.c:191: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:195: undefined reference to `bson_realloc'
/home/brejicz/src/mdb-master/mdb.c:197: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:218: undefined reference to `bson_free'
/tmp/cc1rVoR4.o: In function `record_bson':
/home/brejicz/src/mdb-master/mdb.c:608: undefined reference to `bson_init_static'
/tmp/ccvogrxm.o: In function `fixup_bson':
/home/brejicz/src/mdb-master/mdbundo.c:28: undefined reference to `bson_init_static'
/home/brejicz/src/mdb-master/mdbundo.c:32: undefined reference to `bson_iter_init'
/home/brejicz/src/mdb-master/mdbundo.c:36: undefined reference to `bson_iter_next'
/tmp/ccvogrxm.o: In function `get_bson_at_loc':
/home/brejicz/src/mdb-master/mdbundo.c:85: undefined reference to `bson_init_static'
/home/brejicz/src/mdb-master/mdbundo.c:89: undefined reference to `bson_validate'
/tmp/ccvogrxm.o: In function `mdbundo':
/home/brejicz/src/mdb-master/mdbundo.c:123: undefined reference to `bson_get_data'
collect2: ld returned 1 exit status
make: *** [mdbundo] Error 1

Asya Kamsky

unread,
Sep 1, 2015, 2:01:51 AM9/1/15
to mongodb-user
When I build mdbundo my compile looks like this, right below it is yours:

cc -o mdbundo -Wall -Werror -O0 -ggdb -I/usr/local/include/libbson-1.0
-L/usr/local/lib -lbson-1.0 mdb.c mdb.h mdbundo.c

cc -o mdbundo -Wall -Werror -O0 -ggdb -I/usr/local/include/libbson-1.0
-L/usr/local/lib -lbson-1.0 mdb.c mdb.h mdbundo.c

So that looks the same which means some sort of environment may not
have been set up the same way.
I assume you build libbson without errors, and exported PKG_CONFIG_PATH?

Is it possible you have a different version of libson installed from
your previous attempts?

Asya
> https://groups.google.com/d/msgid/mongodb-user/6f2898d9-6d00-4ff7-af8e-4d4abab4b523%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages