replSet initial sync exception 10356 invalid ns

[Luc Suryo]

Hello

Today our secondary server had a disk crash and so we had ro replace
the disk and start
the replication to this server from scratch, however Im seeing a lot
of this errors:

Mon Dec 20 09:17:22 [rs_sync] Assertion: 10356:invalid ns
0x540c7e 0x72d9c8 0x6d4900 0x6d61ad 0x6bab39 0x6bcb93 0x6b88b3
0x6b89b8 0x6b8c51 0x83a4b0 0x346f60673d 0x346eed3d1d
mongod(_ZN5mongo11msgassertedEiPKc+0x1de) [0x540c7e]
mongod(_ZN5mongo12userCreateNSEPKcNS_7BSONObjERSsbPb+0x5d8) [0x72d9c8]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0xed0) [0x6d4900]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x6d61ad]
mongod(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x619) [0x6bab39]
mongod(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x23) [0x6bcb93]
mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x73) [0x6b88b3]
mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x48) [0x6b89b8]
mongod(_ZN5mongo15startSyncThreadEv+0x111) [0x6b8c51]
mongod(thread_proxy+0x80) [0x83a4b0]
/lib64/libpthread.so.0 [0x346f60673d]
/lib64/libc.so.6(clone+0x6d) [0x346eed3d1d]
Mon Dec 20 09:17:22 [rs_sync] replSet initial sync exception 10356 invalid ns

info:
Mon Dec 20 09:06:42 MongoDB starting : pid=9211 port=27017
dbpath=/home/mongo/data/db 64-bit
Mon Dec 20 09:06:42 db version v1.6.5, pdfile version 4.5
Mon Dec 20 09:06:42 git version: 0eb017e9b2828155a67c5612183337b89e12e291
Mon Dec 20 09:06:42 sys info: Linux domU-12-31-39-06-79-A1
2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64
BOOST_LIB_VERSION=1_41

its a mongo 1.6.5 64 bits on Linux

Anyone has any idea what this means? Does it means the primary has
corrupted data?

thanks!

--
-ls

[Alvin Richards]

The assert is complaining that its trying to create an invalid
namespace.

Can you paste bin more of the log, I suspect that there may be
something earlier that may indicate the root cause.

-Alvin

[Nat]

Possible. Can you
- attach the log file from slave?
- try to list <dbname>.system.namespaces.find({}) from master

[Luc Suryo]

shoot, lost the old log files! So I only have the log file when we
started the server again...

is there any other file I could sent you?

[Luc Suryo]

Nat, see my other email I no longer have the log files.

[Luc Suryo]

Alvin

If this help, I can start from scratch, since mongod has been stopped
and all file deleted
So I can start mongod, and I will have the log file from the moment is
started, it should then
create the 'local' files and the arbiter will assign it as secondary
and it should try to sync
the data with the primary,

will this work ?? and then you want the full log file emailed to you?

thanks!

On Dec 20, 5:41 pm, Alvin Richards <al...@10gen.com> wrote:

[Nat]

I guess if you can reproduce the problem, yes :)

[Alvin Richards]

Yes, please pastebin the logs... or email be directly and I can open a
case to track.

-Alvin

[Luc Suryo]

Alvin

http://pastebin.com/qdVQK706

so the the error goes in loop over and over again, I did 2 iterations

what happen is the stream db file .ns, .0 and .1 get created, errors
they get deleted and the process try again and repeats it self over
and over again

thanks!!

[Luc Suryo]

see reply to Alvin
thanks!

[jrhernandez]

Hi Luc,

I had this same error happen to me just now, it was looping through
the error and retrying. I checked all the collections under each
database and noticed that there was a newline after I ran the "show
collections" command on one of my DB's. I then ran
"db.getCollectionNames();", and sure enough, one of the collection
names was an empty string. I then ran db[""].drop() and the slave was
then able to sync perfectly. Hope this helps!


Regards,

Jessie Hernandez

[Luc Suryo]

Thank you MUCH!

Im in the middle of a EC2 test, so as soon im done I will do the same
things and report back what I found!

[Luc Suryo]

Jessie,


We did had indeed a collection with an empty name! I deleted the
collection and restarted the secondary and it started to sync the
data. At this very moment it has already passed the db with the loop!

Again thanks, case close/solved.


10gen/Alvi: should this not be a bug? and I *think* is; if mongo does
not allow empty collection names then the problem could be in the PHP
driver? (we are using version 1.1.0), how do we report this ? thank!

On Dec 22, 11:59 am, jrhernandez <jrhernande...@gmail.com> wrote:

[Alvin Richards]

Reporting bugs is easy

1. http://jira.mongodb.org
2. Create an account if you do not have one
3. Click "Create a new issue"
4. Select "Core Server" from the Project drop down
5. Select "Bug" for Issue type
6. Fill in the rest of the details and submit!

-Alvin

[Kristina Chodorow]

I think the MongoDB creates the "" database as part of the dump or restore process... I can't remember exactly when.  The PHP driver doesn't let you create or use "" as a database, so file the bug with the Core Server project (as Alvin recommends). 

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Luc Suryo]

OK will do
thanks

On Dec 22, 4:10 pm, Kristina Chodorow <krist...@10gen.com> wrote:
> I think the MongoDB creates the "" database as part of the dump or restore
> process... I can't remember exactly when.  The PHP driver doesn't let you
> create or use "" as a database, so file the bug with the Core Server project
> (as Alvin recommends).
>
> On Wed, Dec 22, 2010 at 6:36 PM, Alvin Richards <al...@10gen.com> wrote:
> > Reporting bugs is easy
>

> > 1.http://jira.mongodb.org

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsu...@googlegroups.com>
> > .