Data loss of mongoDB
623 views
Skip to first unread message
JohnHenry
May 21, 2012, 3:16:59 AM5/21/12
to mongodb-user
Hi, all
We are using MongoDb
db version v2.0.2, pdfile version 4.5
It was set on single machine, using no "replica sets".
We use mongo as database of our system. But found that some times,
It will "drop some data" unexpectedly. We had make sure that there are
even no "remove" functions called in our code. (We are using C++
interface of mongo. DBClientConnection). but occasionally, some
records just missed.
Some other database, like mysql. If the data update function
failed, the old record will exist. But for mongo, (We are not sure
which operation cause this) the data just dropped. We can find nothing
usable on mongoDb log. So is anybody meet such case before?
Some body said the "kill -9" will cause data lost. But We never
use this command on mongod process. We some times use "kill -9" on our
bussiniess process which using mongoDB, But I don't think it will
cause mongo to lose data.
Best regards!
We are using MongoDb
db version v2.0.2, pdfile version 4.5
It was set on single machine, using no "replica sets".
We use mongo as database of our system. But found that some times,
It will "drop some data" unexpectedly. We had make sure that there are
even no "remove" functions called in our code. (We are using C++
interface of mongo. DBClientConnection). but occasionally, some
records just missed.
Some other database, like mysql. If the data update function
failed, the old record will exist. But for mongo, (We are not sure
which operation cause this) the data just dropped. We can find nothing
usable on mongoDb log. So is anybody meet such case before?
Some body said the "kill -9" will cause data lost. But We never
use this command on mongod process. We some times use "kill -9" on our
bussiniess process which using mongoDB, But I don't think it will
cause mongo to lose data.
Best regards!
Sam Millman
May 21, 2012, 3:22:41 AM5/21/12
to mongod...@googlegroups.com
So when you go to update a row if you get loss it loses the old record as well??
That doesn't really sound like a Mongo problem since most data loss is just insert or update unless Mongo failed partially during the update but then the record didn't miss it just died, though nothing useable in the Mongo logs is confusing too since Mongo will normally say if it encounters a problem.
kill -9 on anything will always effect anything it touches since it could be processing at the time of killing it ungracefully and that could cause it to partially write data and what not. Does it happen if you gracefully shut down your business process?
That doesn't really sound like a Mongo problem since most data loss is just insert or update unless Mongo failed partially during the update but then the record didn't miss it just died, though nothing useable in the Mongo logs is confusing too since Mongo will normally say if it encounters a problem.
kill -9 on anything will always effect anything it touches since it could be processing at the time of killing it ungracefully and that could cause it to partially write data and what not. Does it happen if you gracefully shut down your business process?
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
gregor
May 21, 2012, 3:48:21 AM5/21/12
to mongod...@googlegroups.com
This doesn't sound right - with 2.0.2 you run with journaling on by default unless you switch it off. The journal should protect you against most cases off data loss in the event of machine failure or a kill -9. The exception is where the journal hasn't been flushed to disk - this is done every 100ms so anything written more recently than that can be lost. But mongod should never remove data unless you specifically request it to. Can you give me an example of document that you expect to be there that isn't? You are not using sharding correct?
jm
May 21, 2012, 6:44:16 AM5/21/12
to mongodb-user
Just to make sure, are you sure the data "lost" ended up in mongo in
the first place?
If safe mode is not enabled or you dont use getLastError, it might
appear as if inserts were successfull when they were not because some
error. So the records would not be there when you later look for them.
No idea what could cause successfully stored documents to disappear
later. That's why I would suspect safe mode is off.
the first place?
If safe mode is not enabled or you dont use getLastError, it might
appear as if inserts were successfull when they were not because some
error. So the records would not be there when you later look for them.
No idea what could cause successfully stored documents to disappear
later. That's why I would suspect safe mode is off.
JohnHenry
May 22, 2012, 7:01:35 AM5/22/12
to mongodb-user
dbpath = /home/john/tmp/mongodb/data
logpath = /home/john/tmp/mongodb/log/mongodb.log
repairpath = /home/john/tmp/mongodb/repair
pidfilepath = /home/john/tmp/mongodb/mongodb.pid
directoryperdb = true
logappend = true
noauth = true
port = 33306
maxConns = 1024
fork = true
rest = true
quota = true
quotaFiles = 1024
nssize = 16
That is my configure about mongo. I did not find "safe-mode" like
options for mongod command. could you please tell me something about
that?
my operation on the collection, where data lose occurred, is just as
follows:
p_conn->update(s_player_table, BSON("UID"<<user_id), player_data,
true) //An upsert operation
BSONObj gu_data = pConn->findOne(s_player_table,
BSON("UID"<<user_id<<"PUID"<<puid)); //An query operation
May be the operation will be some what frequently, But never circle
like;
will that cause some problems?
Regards!
On 5月21日, 下午6时44分, jm <jtopp...@gmail.com> wrote:
> Just to make sure, are you sure the data "lost" ended up in mongo in
> the first place?
>
> If safe mode is not enabled or you dont use getLastError, it might
> appear as if inserts were successfull when they were not because some
> error. So the records would not be there when you later look for them.
>
> No idea what could cause successfully stored documents to disappear
> later. That's why I would suspect safe mode is off.
>
>
> - 显示引用的文字 -
logpath = /home/john/tmp/mongodb/log/mongodb.log
repairpath = /home/john/tmp/mongodb/repair
pidfilepath = /home/john/tmp/mongodb/mongodb.pid
directoryperdb = true
logappend = true
noauth = true
port = 33306
maxConns = 1024
fork = true
rest = true
quota = true
quotaFiles = 1024
nssize = 16
That is my configure about mongo. I did not find "safe-mode" like
options for mongod command. could you please tell me something about
that?
my operation on the collection, where data lose occurred, is just as
follows:
p_conn->update(s_player_table, BSON("UID"<<user_id), player_data,
true) //An upsert operation
BSONObj gu_data = pConn->findOne(s_player_table,
BSON("UID"<<user_id<<"PUID"<<puid)); //An query operation
May be the operation will be some what frequently, But never circle
like;
will that cause some problems?
Regards!
On 5月21日, 下午6时44分, jm <jtopp...@gmail.com> wrote:
> Just to make sure, are you sure the data "lost" ended up in mongo in
> the first place?
>
> If safe mode is not enabled or you dont use getLastError, it might
> appear as if inserts were successfull when they were not because some
> error. So the records would not be there when you later look for them.
>
> No idea what could cause successfully stored documents to disappear
> later. That's why I would suspect safe mode is off.
>
> On May 21, 10:16 am,JohnHenry<qhlonl...@163.com> wrote:
>
>
>
> > Hi, all
> > We are using MongoDb
> > db version v2.0.2, pdfile version 4.5
> > It was set on single machine, using no "replica sets".
> > We use mongo as database of our system. But found that some times,
> > It will "drop some data" unexpectedly. We had make sure that there are
> > even no "remove" functions called in our code. (We are using C++
> > interface of mongo. DBClientConnection). but occasionally, some
> > records just missed.
> > Some other database, like mysql. If the data update function
> > failed, the old record will exist. But for mongo, (We are not sure
> > which operation cause this) the data just dropped. We can find nothing
> > usable on mongoDb log. So is anybody meet such case before?
> > Some body said the "kill -9" will cause data lost. But We never
> > use this command on mongod process. We some times use "kill -9" on our
> > bussiniess process which using mongoDB, But I don't think it will
> > cause mongo to lose data.
>
> > Best regards!- 隐藏被引用文字 -
>
>
>
> > Hi, all
> > We are using MongoDb
> > db version v2.0.2, pdfile version 4.5
> > It was set on single machine, using no "replica sets".
> > We use mongo as database of our system. But found that some times,
> > It will "drop some data" unexpectedly. We had make sure that there are
> > even no "remove" functions called in our code. (We are using C++
> > interface of mongo. DBClientConnection). but occasionally, some
> > records just missed.
> > Some other database, like mysql. If the data update function
> > failed, the old record will exist. But for mongo, (We are not sure
> > which operation cause this) the data just dropped. We can find nothing
> > usable on mongoDb log. So is anybody meet such case before?
> > Some body said the "kill -9" will cause data lost. But We never
> > use this command on mongod process. We some times use "kill -9" on our
> > bussiniess process which using mongoDB, But I don't think it will
> > cause mongo to lose data.
>
>
> - 显示引用的文字 -
JohnHenry
May 22, 2012, 7:10:10 AM5/22/12
to mongodb-user
So what will happen when mongo update failed? will it throws any
exception?
kill -9 operation has no directive relationship with my problem.
Because even if my business process is running, the problem will
occure sometimes.
On 5月21日, 下午3时22分, Sam Millman <sam.mill...@gmail.com> wrote:
> So when you go to update a row if you get loss it loses the old record as
> well??
>
> That doesn't really sound like a Mongo problem since most data loss is just
> insert or update unless Mongo failed partially during the update but then
> the record didn't miss it just died, though nothing useable in the Mongo
> logs is confusing too since Mongo will normally say if it encounters a
> problem.
>
> kill -9 on anything will always effect anything it touches since it could
> be processing at the time of killing it ungracefully and that could cause
> it to partially write data and what not. Does it happen if you gracefully
> shut down your business process?
>
> > See also the IRC channel -- freenode.net#mongodb- 隐藏被引用文字 -
>
> - 显示引用的文字 -
exception?
kill -9 operation has no directive relationship with my problem.
Because even if my business process is running, the problem will
occure sometimes.
On 5月21日, 下午3时22分, Sam Millman <sam.mill...@gmail.com> wrote:
> So when you go to update a row if you get loss it loses the old record as
> well??
>
> That doesn't really sound like a Mongo problem since most data loss is just
> insert or update unless Mongo failed partially during the update but then
> the record didn't miss it just died, though nothing useable in the Mongo
> logs is confusing too since Mongo will normally say if it encounters a
> problem.
>
> kill -9 on anything will always effect anything it touches since it could
> be processing at the time of killing it ungracefully and that could cause
> it to partially write data and what not. Does it happen if you gracefully
> shut down your business process?
>
>
> - 显示引用的文字 -
gregor
May 22, 2012, 7:16:07 AM5/22/12
to mongod...@googlegroups.com
What driver are you using?
You can call getLastError http://www.mongodb.org/display/DOCS/getLastError+Command after upsert.
JohnHenry
May 23, 2012, 6:01:37 AM5/23/12
to mongodb-user
Thank you for help. I am using C++ driver.
> > > > mongodb-user...@googlegroups.com
> > > - 显示引用的文字 -- 隐藏被引用文字 -
>
> - 显示引用的文字 -
> > > - 显示引用的文字 -- 隐藏被引用文字 -
>
> - 显示引用的文字 -
gregor
May 24, 2012, 3:22:22 AM5/24/12
to mongod...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages