NS File disappeared

[Kiri]

Hi people,

I stopped mongod using "kill -2" and when I start it again, the NS
file of my db has disappeared. I can see the db files in /data/db but
I cannot access them.

Any idea?

Thanks in advance
Jordi

[Alvin Richards]

Was dbpath set to be /data/db when the database was running?

-Alvin

[Kiri]

Yes, it points to /data/db

I can acces via mongo and hech the stats of a collection, and the
output is the following:

db.readings.BASIC.stats()
{
"ns" : "dexcellm_db.readings.BASIC",
"count" : 29085024,
"size" : 3993323232,
"avgObjSize" : 137.2982615383092,
"errmsg" : "exception: assertion db/pdfile.h:259",
"code" : 0,
"ok" : 0
}

"errmsg" : "exception: assertion db/pdfile.h:259", seems to be the
problem, but I don't know what it means or how to solve it...

[Eliot Horowitz]

Can you send an ls of the data directory and the end of log file when it was shut down?

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Kiri]

One thing I missed to say is that we had the DB without
authentication, we stopped it and we restarted it with flag --auth.
These are the outputs:

DATA DIRECTORY (NS file has disappeared:

ls -la
total 19037368
drwxr-xr-x 3 root root 4096 2011-07-21 17:00 .
drwxr-xr-x 28 root root 4096 2011-07-22 06:01 ..
-rw------- 1 root root 67108864 2011-07-21 16:12 admin.0
-rw------- 1 root root 134217728 2011-07-21 16:12 admin.1
-rw------- 1 root root 16777216 2011-07-21 16:12 admin.ns
-rw------- 1 root root 134217728 2011-07-21 16:03 dexcellm_db.1
-rw------- 1 root root 2146435072 2011-07-21 16:04 dexcellm_db.10
-rw------- 1 root root 2146435072 2011-07-21 16:05 dexcellm_db.11
-rw------- 1 root root 268435456 2011-07-21 16:05 dexcellm_db.2
-rw------- 1 root root 536870912 2011-07-21 16:05 dexcellm_db.3
-rw------- 1 root root 1073741824 2011-07-21 16:06 dexcellm_db.4
-rw------- 1 root root 2146435072 2011-07-21 16:06 dexcellm_db.5
-rw------- 1 root root 2146435072 2011-07-21 16:07 dexcellm_db.6
-rw------- 1 root root 2146435072 2011-07-21 16:08 dexcellm_db.7
-rw------- 1 root root 2146435072 2011-07-21 16:09 dexcellm_db.8
-rw------- 1 root root 2146435072 2011-07-21 16:10 dexcellm_db.9
drwxr-xr-x 2 root root 4096 2011-07-21 16:13 journal


STOPPING THE SERVER

Thu Jul 21 15:26:59 [initandlisten] connection accepted from
127.0.0.1:43886 #76601
Thu Jul 21 15:27:11 [dur] lsn set 5009177601
Thu Jul 21 15:28:03 [initandlisten] connection accepted from
127.0.0.1:43889 #76602
Thu Jul 21 15:28:11 [dur] lsn set 5009236572
Thu Jul 21 15:28:46 got kill or ctrl c or hup signal 2 (Interrupt),
will terminate after current cmd ends
Thu Jul 21 15:28:46 [interruptThread] now exiting
Thu Jul 21 15:28:46 dbexit:
Thu Jul 21 15:28:46 [interruptThread] shutdown: going to close
listening sockets...
Thu Jul 21 15:28:46 [interruptThread] closing listening socket: 6
Thu Jul 21 15:28:46 [interruptThread] closing listening socket: 7
Thu Jul 21 15:28:46 [interruptThread] closing listening socket: 8
Thu Jul 21 15:28:46 [interruptThread] closing listening socket: 9
Thu Jul 21 15:28:46 [interruptThread] removing socket file: /tmp/
mongodb-27017.sock
Thu Jul 21 15:28:46 [interruptThread] removing socket file: /tmp/
mongodb-28017.sock
Thu Jul 21 15:28:46 [interruptThread] shutdown: going to flush
diaglog...
Thu Jul 21 15:28:46 [interruptThread] shutdown: going to close
sockets...
Thu Jul 21 15:28:46 [interruptThread] shutdown: waiting for fs
preallocator...
Thu Jul 21 15:28:46 [interruptThread] shutdown: lock for final
commit...
Thu Jul 21 15:28:46 [interruptThread] shutdown: final commit...
Thu Jul 21 15:28:46 [interruptThread] shutdown: closing all files...
Thu Jul 21 15:28:47 closeAllFiles() finished
Thu Jul 21 15:28:47 [interruptThread] shutdown: journalCleanup...
Thu Jul 21 15:28:47 [interruptThread] removeJournalFiles
Thu Jul 21 15:28:47 [interruptThread] shutdown: removing fs lock...
Thu Jul 21 15:28:47 dbexit: really exiting now

STARTING SERVER
Thu Jul 21 15:29:24 [initandlisten] MongoDB starting : pid=4328
port=27017 dbpath=/data/db/ 64-bit
Thu Jul 21 15:29:24 [initandlisten] db version v1.8.1, pdfile version
4.5
Thu Jul 21 15:29:24 [initandlisten] git version:
a429cd4f535b2499cc4130b06ff7c26f41c00f04
Thu Jul 21 15:29:24 [initandlisten] build sys info: Linux bs-
linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28
EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Thu Jul 21 15:29:24 [initandlisten] journal dir=/data/db/journal
Thu Jul 21 15:29:24 [initandlisten] recover : no journal files
present, no recovery needed
Thu Jul 21 15:29:24 [initandlisten] waiting for connections on port
27017
Thu Jul 21 15:29:24 [websvr] web admin interface listening on port
28017
Thu Jul 21 15:29:57 [initandlisten] connection accepted from
127.0.0.1:43896 #1
Thu Jul 21 15:30:11 [initandlisten] connection accepted from
127.0.0.1:43898 #2
Thu Jul 21 15:30:11 [conn2] query admin.$cmd ntoreturn:1 command:
{ ismaster: 1 } reslen:87 167ms
Thu Jul 21 15:30:13 [initandlisten] connection accepted from
83.32.213.103:51477 #3
Thu Jul 21 15:30:14 [conn3] auth: couldn't find user kiri,
dexcellm_db.system.users
Thu Jul 21 15:30:14 [conn3] end connection 83.32.213.103:51477
Thu Jul 21 15:30:23 [dur] lsn set 59013
Thu Jul 21 15:30:27 [initandlisten] connection accepted from
83.32.213.103:54010 #4
Thu Jul 21 15:30:27 [initandlisten] connection accepted from
83.32.213.103:51478 #5
Thu Jul 21 15:30:27 [conn4] auth: couldn't find user kiri,
dexcellm_db.system.users
Thu Jul 21 15:30:27 [conn5] auth: couldn't find user kiri,
dexcellm_db.system.users
Thu Jul 21 15:30:27 [conn5] end connection 83.32.213.103:51478
Thu Jul 21 15:30:29 [conn1] end connection 127.0.0.1:43896
Thu Jul 21 15:30:33 [conn4] assertion 10057 unauthorized
db:dexcellm_db lock type:-1 client:83.32.213.103
ns:dexcellm_db.system.namespaces query:{}
Thu Jul 21 15:30:38 [conn4] end connection 83.32.213.103:54010
Thu Jul 21 15:30:52 [initandlisten] connection accepted from
83.32.213.103:54011 #6
Thu Jul 21 15:30:53 [conn6] auth: couldn't find user kiri,
dexcellm_db.system.users
Thu Jul 21 15:30:57 [initandlisten] connection accepted from
127.0.0.1:43900 #7
Thu Jul 21 15:31:15 [initandlisten] connection accepted from
127.0.0.1:43904 #8
Thu Jul 21 15:31:23 [dur] lsn set 118158
Thu Jul 21 15:31:26 [conn7] assertion 10057 unauthorized db:admin lock
type:-1 client:127.0.0.1 ns:admin.system.users query:{}
Thu Jul 21 15:31:36 [initandlisten] connection accepted from
127.0.0.1:53009 #9
Thu Jul 21 15:31:40 [conn6] assertion 10057 unauthorized
db:dexcellm_db lock type:-1 client:83.32.213.103
ns:dexcellm_db.system.namespaces query:{}
Thu Jul 21 15:31:42 [conn6] end connection 83.32.213.103:54011
Thu Jul 21 15:31:54 [conn7] assertion 10057 unauthorized
db:dexcellm_db lock type:-1 client:127.0.0.1
ns:dexcellm_db.system.users query:{}
Thu Jul 21 15:31:55 [initandlisten] connection accepted from
83.32.213.103:54018 #10
Thu Jul 21 15:31:55 [conn10] auth: couldn't find user kiri,
dexcellm_db.system.users
Thu Jul 21 15:32:03 [conn9] assertion 10057 unauthorized
db:dexcellm_db lock type:-1 client:127.0.0.1
ns:dexcellm_db.system.namespaces query:{}
Thu Jul 21 15:32:19 [initandlisten] connection accepted from
127.0.0.1:53018 #11
Thu Jul 21 15:32:23 [dur] lsn set 177381
Thu Jul 21 15:32:32 [conn9] end connection 127.0.0.1:53009
Thu Jul 21 15:32:59 [conn7] auth: couldn't find user kiri,
dexcellm_db.system.users
Thu Jul 21 15:33:06 [conn7] assertion 10057 unauthorized
db:dexcellm_db lock type:-1 client:127.0.0.1
ns:dexcellm_db.system.users query:{}
Thu Jul 21 15:33:08 got kill or ctrl c or hup signal 2 (Interrupt),
will terminate after current cmd ends
Thu Jul 21 15:33:08 [interruptThread] now exiting
Thu Jul 21 15:33:08 dbexit:
Thu Jul 21 15:33:08 [interruptThread] shutdown: going to close
listening sockets...
Thu Jul 21 15:33:08 [interruptThread] closing listening socket: 6
Thu Jul 21 15:33:08 [interruptThread] closing listening socket: 7
Thu Jul 21 15:33:08 [interruptThread] closing listening socket: 8
Thu Jul 21 15:33:08 [interruptThread] closing listening socket: 9
Thu Jul 21 15:33:08 [interruptThread] removing socket file: /tmp/
mongodb-27017.sock
Thu Jul 21 15:33:08 [interruptThread] removing socket file: /tmp/
mongodb-28017.sock
Thu Jul 21 15:33:08 [interruptThread] shutdown: going to flush
diaglog...
Thu Jul 21 15:33:08 [interruptThread] shutdown: going to close
sockets...
Thu Jul 21 15:33:08 [interruptThread] shutdown: waiting for fs
preallocator...
Thu Jul 21 15:33:08 [interruptThread] shutdown: lock for final
commit...
Thu Jul 21 15:33:08 [interruptThread] shutdown: final commit...
Thu Jul 21 15:33:08 [interruptThread] shutdown: closing all files...
Thu Jul 21 15:33:08 closeAllFiles() finished
Thu Jul 21 15:33:08 [interruptThread] shutdown: journalCleanup...
Thu Jul 21 15:33:08 [interruptThread] removeJournalFiles
Thu Jul 21 15:33:08 [interruptThread] shutdown: removing fs lock...
Thu Jul 21 15:33:08 dbexit: really exiting now
Thu Jul 21 15:33:16 [initandlisten] MongoDB starting : pid=4881
port=27017 dbpath=/data/db/ 64-bit

[Eliot Horowitz]

Can you run an fsck on the filesystem to make sure the filesystem is ok?

[Kiri]

I think I can't because the problem was in our production server. We
restored a backup and we lost "only" 1,5 hour of data...
I want to know what happened exactly in case it happens again or to
see if we did something wrong...

Do you see something strange in Log? When Mongodb removes de NS file??

> ...
>
> leer más »

[Laurent]

Your files are own by "root". If your mogod is not launched as root but as mongodb user for instance this could be explained by a repair launched as root that modified the files ownership. Always launch repairs using  the "normal" mongodb user account, e.g. :

sudo -u mongodb mongod -f /etc/mongodb.conf --repair

[Eliot Horowitz]

I would still run fsck.

Mongo never removes files.

I would check for file system corruption or disk errors asap.

--

[Kiri]

Ok, we will run FSCK ASAP.

But if we have all the database files but not the NS file, it is
possible to generate de NS File in some way?

> ...
>
> leer más »

[Eliot Horowitz]

No - without the .ns file, things aren't readable.

An old version could theoretically work if there weren't any collection changes.

> ...
>
> leer más »