> Hi All,

> I've been confronted with a completely nonsensical issue this morning. On
> two of our MongoDB replica sets (running mongod 2.0.6) we have found a
> large number of databases with garbled names. Let me illustrate:

> > db.getMongo().getDBNames().length                                      

> 46025

> Our application code doesn't create 46,000+ databases. When I looked at
> the names of these DBs, I found the following:

>         "admin Y ",
>         "admin ZQ",
>         "admin Zy",
>         "admin Zﾐ",
>         "admin Zﾲ",
>         "admin Zﾸ",
>         "admin Zﾹ",
>         "admin Zￃ",
>         "admin [\u0004",
>         "admin [\u0006",
>         "admin [\u001e",
>         "admin [G",
>         "admin [e",
>         "admin [j",
>         "admin [}",
>         "admin [ﾏ",
>         "admin [ﾕ",
>         "admin [ﾝ",

> This is just a tiny excerpt. There are many, many of these DBs there.
> 46,000+ of them!

> Has anyone on this list seen anything like this before? Our application is
> written in Scala and doesn't do anything strange with strings. Furthermore,
> this seems to have "affected" (for lack of a better term) just the "admin"
> DB, which our application doesn't access in any way.

> The above printouts are coming from the replica set's master. On the slave
> I found a similar situation. When I ran a db.serverStatus() on the slave, I
> found the following potentially interesting tidbit:

>         "mem" : {
>                 "bits" : 64,
>                 "resident" : 5107,
>                 "virtual" : 100450,
>                 "supported" : true,
>                 "mapped" : 47242,
>                 "mappedWithJournal" : 94484,
>                 "note" : "virtual minus mapped is large. could indicate a
> memory leak"
>         },

> After a restart, the weirdly named DBs have gone and the "mem" section
> looks like this:

>         "mem" : {
>                 "bits" : 64,
>                 "resident" : 49,
>                 "virtual" : 94832,
>                 "supported" : true,
>                 "mapped" : 47242,
>                 "mappedWithJournal" : 94484
>         },

> Any ideas are greatly appreciated. This caused a panic in our monitoring
> software. The sysadmins are unhappy, and you all know what this means for
> me. *shudder*

> Cheers,
> -max

> what gui do you use to see data in your mongo? For example we use
>  RockMongo. For authorisation it requires to provide DB name. If you input
> wrong DB name - it will be created on server. So... don't you think that
> someone (hacker) tried automatically access to your data? :)

> On Monday, October 15, 2012 4:39:18 PM UTC-4, Sergey Ukolov wrote:

>> what gui do you use to see data in your mongo? For example we use
>>  RockMongo. For authorisation it requires to provide DB name. If you input
>> wrong DB name - it will be created on server. So... don't you think that
>> someone (hacker) tried automatically access to your data? :)

> I find this highly unlikely because our environment is strictly firewalled
> from the outside. It's also extremely difficult for me to believe that
> someone here would have made 46,000 of these databases, even if it were an
> honest mistake.

> Do you have the logs from right before this happened?

> On Tuesday, October 16, 2012 5:44:52 PM UTC-4, Shaun wrote:

>> Do you have the logs from right before this happened?

> Yes! Suddenly one fine day the log file began to get filled with thousands
> of messages like this one:

>  Fri Oct 12 21:31:07 [conn580285] Assertion: 10334:Invalid BSONObj size:
> -256 (0x00FFFFFF) first element: ^S: MinKey
> 0x5848a2 0x508f51 0x5df658 0x889079 0x88dcd9 0xaa0bc8 0x6389f7
> 0x7f7c18473efc 0x7f7c17a0d59d
>  /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x5848a2]
>  /usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x481) [0x508f51]
>  /usr/bin/mongod(_ZN5mongo12QueryMessageC1ERNS_9DbMessageE+0x348)
> [0x5df658]
>  /usr/bin/mongod() [0x889079]
>  /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseER KNS_11HostAndPortE+0x559)
> [0x88dcd9]
>  /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21Abst ractMessagingPortEPNS_9LastErrorE+0x78)
> [0xaa0bc8]
>  /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x6389f7]
>  /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f7c18473efc]
>  /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7c17a0d59d]

> Hi Max,

> Could you send the log for what happened right before the first of these
> messages appeared?  That way we can see more context on what it was trying
> to do when it got this error.

> Here's the log roll from just before the badness happened:

> On Tuesday, October 16, 2012 5:44:52 PM UTC-4, Shaun wrote:

>> Do you have the logs from right before this happened?

> Yes! Suddenly one fine day the log file began to get filled with thousands
> of messages like this one:

>  Fri Oct 12 21:31:07 [conn580285] Assertion: 10334:Invalid BSONObj size:
> -256 (0x00FFFFFF) first element: ^S: MinKey
> 0x5848a2 0x508f51 0x5df658 0x889079 0x88dcd9 0xaa0bc8 0x6389f7
> 0x7f7c18473efc 0x7f7c17a0d59d
>  /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x5848a2]
>  /usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x481) [0x508f51]
>  /usr/bin/mongod(_ZN5mongo12QueryMessageC1ERNS_9DbMessageE+0x348)
> [0x5df658]
>  /usr/bin/mongod() [0x889079]
>  /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseER KNS_11HostAndPortE+0x559)
> [0x88dcd9]
>  /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21Abst ractMessagingPortEPNS_9LastErrorE+0x78)
> [0xaa0bc8]
>  /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x6389f7]
>  /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f7c18473efc]
>  /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7c17a0d59d]

> this might just be symptomatic of something going wrong earlier in the run
> as QueryMessage is usely involved in read-only operations.

> aside : since this has been occurring, I recommend you run mongod with
> --objcheck in the future.

> if you can get something that reproduces happy to dive on it and analyze.

> On Monday, October 22, 2012 1:03:37 PM UTC-4, Dwight Merriman wrote:

>> this might just be symptomatic of something going wrong earlier in the
>> run as QueryMessage is usely involved in read-only operations.

>> aside : since this has been occurring, I recommend you run mongod with
>> --objcheck in the future.

>> if you can get something that reproduces happy to dive on it and analyze.

> Could this have been caused by JAVA-660? We've narrowed the source of
> badness down to our pre-release verification environment, which at the
> moment runs a much newer version of mongo-java-driver than our main
> production deployment. The bug states that mongod behavior is "undefined".
> I suppose that 46,000+ DBs with garbled names may be included in
> "undefined" :)