Server crashed, all data lost

63 views
Skip to first unread message

vitalybaev

unread,
Apr 26, 2012, 9:20:23 AM4/26/12
to mongod...@googlegroups.com
Hello, our server (shard) crashed . Have been stored about 30 millions objects. When it run back, we started repair and got "Invalid BSON size: 0";
Now there are NO any data files, only 2 files with 160 objects(!)

What we can to do, there are no backups...

Debian Linux 6.0
Mongodb 2.0.2

Eliot Horowitz

unread,
Apr 26, 2012, 9:51:07 AM4/26/12
to mongod...@googlegroups.com
Were you running with journalling off?
If it was on you shouldn't have been asked to repair?
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/PfY-d12F6XoJ.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

vitalybaev

unread,
Apr 26, 2012, 10:00:55 AM4/26/12
to mongod...@googlegroups.com
Journaling turned on
No, we shoudn't have been asked to repair

четверг, 26 апреля 2012 г., 17:51:07 UTC+4 пользователь Eliot Horowitz написал:
Were you running with journalling off?
If it was on you shouldn't have been asked to repair?

On Thu, Apr 26, 2012 at 9:20 AM, vitalybaev <d...@vitalybaev.ru> wrote:
> Hello, our server (shard) crashed . Have been stored about 30 millions
> objects. When it run back, we started repair and got "Invalid BSON size: 0";
> Now there are NO any data files, only 2 files with 160 objects(!)
>
> What we can to do, there are no backups...
>
> Debian Linux 6.0
> Mongodb 2.0.2
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/PfY-d12F6XoJ.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

Eliot Horowitz

unread,
Apr 26, 2012, 10:03:53 AM4/26/12
to mongod...@googlegroups.com
Can you send the full logs?

2012/4/26 vitalybaev <d...@vitalybaev.ru>:
> Journaling turned on
> No, we shoudn't have been asked to repair
>
> четверг, 26 апреля 2012 г., 17:51:07 UTC+4 пользователь Eliot Horowitz
> написал:
>>
>> Were you running with journalling off?
>> If it was on you shouldn't have been asked to repair?
>>
>> On Thu, Apr 26, 2012 at 9:20 AM, vitalybaev <d...@vitalybaev.ru> wrote:
>> > Hello, our server (shard) crashed . Have been stored about 30 millions
>> > objects. When it run back, we started repair and got "Invalid BSON size:
>> > 0";
>> > Now there are NO any data files, only 2 files with 160 objects(!)
>> >
>> > What we can to do, there are no backups...
>> >
>> > Debian Linux 6.0
>> > Mongodb 2.0.2
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/mongodb-user/-/PfY-d12F6XoJ.
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > mongodb-user...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/CTsQk5BofP4J.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.

vitalybaev

unread,
Apr 26, 2012, 10:08:27 AM4/26/12
to mongod...@googlegroups.com
Now we try to recovery files in readonly disk mode. We can send logs later
It looks like old resolved bug https://jira.mongodb.org/browse/SERVER-2067
And how logs can help us?

четверг, 26 апреля 2012 г., 18:03:53 UTC+4 пользователь Eliot Horowitz написал:
Can you send the full logs?

2012/4/26 vitalybaev <d...@vitalybaev.ru>:
> Journaling turned on
> No, we shoudn't have been asked to repair
>
> четверг, 26 апреля 2012 г., 17:51:07 UTC+4 пользователь Eliot Horowitz
> написал:
>>
>> Were you running with journalling off?
>> If it was on you shouldn't have been asked to repair?
>>
>> On Thu, Apr 26, 2012 at 9:20 AM, vitalybaev <d...@vitalybaev.ru> wrote:
>> > Hello, our server (shard) crashed . Have been stored about 30 millions
>> > objects. When it run back, we started repair and got "Invalid BSON size:
>> > 0";
>> > Now there are NO any data files, only 2 files with 160 objects(!)
>> >
>> > What we can to do, there are no backups...
>> >
>> > Debian Linux 6.0
>> > Mongodb 2.0.2
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/mongodb-user/-/PfY-d12F6XoJ.
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/CTsQk5BofP4J.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

Eliot Horowitz

unread,
Apr 26, 2012, 10:11:41 AM4/26/12
to mongod...@googlegroups.com
that bug isn't really terribly related.

logs can help understand what happened which could lead to a recovery process

2012/4/26 vitalybaev <d...@vitalybaev.ru>:
>> >> > mongodb-user...@googlegroups.com.
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/mongodb-user/-/CTsQk5BofP4J.
>> >
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > mongodb-user...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/dVY_qUoegYcJ.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.

vitalybaev

unread,
Apr 26, 2012, 11:43:50 AM4/26/12
to mongod...@googlegroups.com
Log is very big (about 20Gb)

Grep gets error

"grep: /var/log/mongodb/mongodb.log: Input/output error"


четверг, 26 апреля 2012 г., 18:11:41 UTC+4 пользователь Eliot Horowitz написал:
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/mongodb-user/-/CTsQk5BofP4J.
>> >
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/dVY_qUoegYcJ.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

Eliot Horowitz

unread,
Apr 26, 2012, 11:45:27 AM4/26/12
to mongod...@googlegroups.com
That message usually indicates a drive hardware error.
You should check the system logs as well.

2012/4/26 vitalybaev <d...@vitalybaev.ru>:
>> >> >> > mongodb-user...@googlegroups.com.
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To view this discussion on the web visit
>> >> > https://groups.google.com/d/msg/mongodb-user/-/CTsQk5BofP4J.
>> >> >
>> >> > To post to this group, send email to mongod...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user...@googlegroups.com.
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/mongodb-user/-/dVY_qUoegYcJ.
>> >
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > mongodb-user...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/EWNbFo7RINkJ.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.

vitalybaev

unread,
Apr 26, 2012, 11:57:35 AM4/26/12
to mongod...@googlegroups.com
Syslog contain many lines like this:

Apr 26 13:38:09 MongoShard6 kernel: [11664.726284] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390043
Apr 26 13:38:09 MongoShard6 kernel: [11664.726284] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390044
Apr 26 13:38:09 MongoShard6 kernel: [11664.726284] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390045
Apr 26 13:38:09 MongoShard6 kernel: [11664.726621] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390046
Apr 26 13:38:09 MongoShard6 kernel: [11664.726838] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390047
Apr 26 13:38:09 MongoShard6 kernel: [11664.727088] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390048
Apr 26 13:38:09 MongoShard6 kernel: [11664.727337] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390049
Apr 26 13:38:09 MongoShard6 kernel: [11664.727546] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390050
Apr 26 13:38:09 MongoShard6 kernel: [11664.728013] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390051
Apr 26 13:38:09 MongoShard6 kernel: [11664.728299] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390052
Apr 26 13:38:09 MongoShard6 kernel: [11664.728663] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390053
Apr 26 13:38:09 MongoShard6 kernel: [11664.729173] EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 12390054


четверг, 26 апреля 2012 г., 19:45:27 UTC+4 пользователь Eliot Horowitz написал:
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To view this discussion on the web visit
>> >> > https://groups.google.com/d/msg/mongodb-user/-/CTsQk5BofP4J.
>> >> >
>> >> > To post to this group, send email to mongod...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/mongodb-user/-/dVY_qUoegYcJ.
>> >
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/EWNbFo7RINkJ.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

Eliot Horowitz

unread,
Apr 26, 2012, 11:59:03 AM4/26/12
to mongod...@googlegroups.com
That looks like harddrive corruption....
Were you using replication? Any backups?

2012/4/26 vitalybaev <d...@vitalybaev.ru>:
>> >> >> >> > mongodb-user...@googlegroups.com.
>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To view this discussion on the web visit
>> >> >> > https://groups.google.com/d/msg/mongodb-user/-/CTsQk5BofP4J.
>> >> >> >
>> >> >> > To post to this group, send email to
>> >> >> > mongod...@googlegroups.com.
>> >> >> > To unsubscribe from this group, send email to
>> >> >> > mongodb-user...@googlegroups.com.
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To view this discussion on the web visit
>> >> > https://groups.google.com/d/msg/mongodb-user/-/dVY_qUoegYcJ.
>> >> >
>> >> > To post to this group, send email to mongod...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user...@googlegroups.com.
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/mongodb-user/-/EWNbFo7RINkJ.
>> >
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > mongodb-user...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/9nX5-69k16gJ.
>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages