So just how durable and reliable has Mongo become?

[Sammaye]

Hi everyone,

I recently read this whilst using my mongodb:
http://www.mikealrogers.com/2010/07/mongodb-performance-durability/

And it's got me thinking. He is basically saying that even with
replication on 3 or more servers I will still get data corruption and
the such.

I was hoping I could ask you guys exactly how reliable mongo db has
become.

I am asking since I wish to use it as a backbone technology within a
very large (multimillion user) intranet system for a huge global
company and it's related partners, but this blog post worries me and
make me think I need to use MySQL or something (which is what is
currently used and is....sorry for the swear....shit).

To get an idea of how heavy my intranet site is just think of facebook
or youtube (though they're not as heavy but gives you an idea of
traffic and load).

Thanks,

[Kyle Banker]

We don't support traditional single-server durability; MongoDB achieves durability through replication and judicious use of snapshotting.

At the moment, you can run a master with any number of slaves, placing one in a separate data center if needed. If a master server dies or has an unclean shutdown, you could see some corruption on that master node, but in that situation, you fail over to a slave. With replica sets (http://www.mongodb.org/display/DOCS/Replica+Sets), you have a set of N replicas, and the failover is handled automatically.

For ops that need the most durability, you can set a W value on writes to ensure that the write is replicated to at least N nodes. So really, the level of durability is configurable. With auto-sharding, which will be production-ready by the end of the month, you'll get automated failover across automatically managed sharded nodes, which may be what your system will require.

Here's more on our philosophy: http://blog.mongodb.org/post/381927266/what-about-durability

Yes, we look at the durability issue differently. But that doesn't mean that we have the wrong approach. I think that the biggest testament to the viability of our design is the number of working production MongoDB deployments. You can see many of them here:

http://www.mongodb.org/display/DOCS/Production+Deployments

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Sammaye]

Hi, thanks for your response. Yes I do not plan to use mongo single
server at all (my intranet site currently houses 25 SQL servers). But
I do need a db I can trust to always send information in and store it
in a manner that means only when there is a down (server fault, earth
quake etc) I or another administrator would require to do something.
I've just heard from many that it's only a matter of time until mongo
becomes corrupt even on a perfectly fine replication system.

What about reports of data not always reaching db?

Does mongo ensure that any data I send in will reach a db and a
collection?

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsubscribe@google groups.com>
> > .

[Kyle Banker]

Answers below:

On Fri, Jul 9, 2010 at 9:16 AM, Sammaye <sam.m...@gmail.com> wrote:

Hi, thanks for your response. Yes I do not plan to use mongo single
server at all (my intranet site currently houses 25 SQL servers). But
I do need a db I can trust to always send information in and store it
in a manner that means only when there is a down (server fault, earth
quake etc) I or another administrator would require to do something.
I've just heard from many that it's only a matter of time until mongo
becomes corrupt even on a perfectly fine replication system.

Not sure how to respond to that. Sounds like a pretty vague assertion. If it's just a matter of time before a given master server will

die unexpectedly, then yes, the master could see corruption on an unclean shutdown like that.

What about reports of data not always reaching db?

By default, writes from the drivers are 'fire-and-forget.' This is an intentional design decision. However, if you need durable writes that are assured to reach the db, use the safe mode implemented by all drivers.

 

Does mongo ensure that any data I send in will reach a db and a
collection?

See above.

 

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[Sam Millman]

"MongoDB, by default, doesn’t actually have a response for writes. You just write your data to the socket and assume it’s going to be available when you try to read it. Under concurrent load you can’t expect to reliably store stateful data like session information like this. It’s kind of like, in your webapp, if you were to spawn a thread to do some work and at the end set some global but you return a response to the client immediately. Where there wasn’t any load you would be fine because your thread is faster than the roundtrip time to the client. But under heavy load the operations in the thread could take too long and a subsequent request wouldn’t have access to that global because it wasn’t set yet. This is why you can’t store session data reliably in MongoDB without changing the default client option to return a response, because no matter how “fast” it is if you send a response to the client (or no response at all) before the data is available for read it’s useless.

To people who don’t live in databases all day it’s hard to explain just how odd this choice is. I don’t know of another database that even allows you to return a response before the write data is accessible much less not even have a response by default. This is kind of like using UDP for data that you care about getting somewhere, it’s theoretically faster but the importance of making sure your data gets somewhere and is accessible is almost always more important."

So if I use safe mode it would be like TCP instead of UDP when I transfer data to mongo (using the same metaphore as the author).

"Not sure how to respond to that. Sounds like a pretty vague assertion. If it's just a matter of time before a given master server will

die unexpectedly, then yes, the master could see corruption on an unclean shutdown like that."

It's just what I keep hearing. That mongo is good as front cache but for say the backend of a full site it's not really that reliable.

You see I want to use mongo, but I want to make sure that if I set it up right it would be just as durable and reliable as say MySQL.

--
Bow Chicka Bow Wow

[Kristina Chodorow]

I've just heard from many that it's only a matter of time until mongo
becomes corrupt even on a perfectly fine replication system.

By the by, it's not like data gradually corrupts itself or something.  Data gets corrupted by server crashes or hardware failures.

You see I want to use mongo, but I want to make sure that if I set it up right it would be just as durable and reliable as say MySQL.

It should be, but it is different.  Heck, it should be more reliable!  One of the points of MongoDB is to have practically zero database downtime: no replaying transaction logs and no manual failover (soon).

If you want to see how it's working out for real users, http://www.mongodb.org/display/DOCS/Production+Deployments has links a ton of blog posts, slideshows, and videos created by production users.

[Voltron]

It is a scary blog post, but read the comments from 10Gen below

[Kyle Banker]

Just to be clear, we use TCP to communicate with the server, not UDP.

As for all the hearsay, there's certainly a lot of it . There are also a lot of happy users, as Kristina points out. People say a lot of things. The only way to get the level of assurance you need is to investigate the system more deeply on your own and to do some prototyping. That's usually the best way to make a technology decision.

On Fri, Jul 9, 2010 at 9:44 AM, Sam Millman <smil...@nhbs.co.uk> wrote:

[Sam Millman]

"Just to be clear, we use TCP to communicate with the server, not UDP."

Yea I meant if I use safe mode it will wait for a reply from the db before it declares the record is set instead of the default write and forget. Sorry I was trying to be clever with my words...failed massively.

I have been using mongo for a few things now (personal) and even on one server setup it hasn't given me any problems. It's only recently I started to question.

Yea mongo is vetted by a lot of people and thats why I actually looked at it originally...though it looks like I would be the first to use it as a full replacement for everything in a huge server setup...

I did look at the 10Gen posts and thats why I'm torn between the two views...kyle is right I'm gonna need to build something like a chat system and throw it onto my network and see what happens.

[Kristina Chodorow]

Yea I meant if I use safe mode it will wait for a reply from the db before it declares the record is set instead of the default write and forget.

Yes.  You can even force an fsync before the db will return success or force it to replicate the operation to N nodes before declaring success.

On Fri, Jul 9, 2010 at 10:14 AM, Sam Millman <smil...@nhbs.co.uk> wrote:

I did get a tad of corruption on a single server setup (vmed dev server...I don't run multi in dev environment) but repairDB() took care of it nicely and I didnt loose any data which goes to contradict the blog post again.

[Sam Millman]

Which in turn takes out the unkown factor of knowing whether or not mongo really did write to db or not. 

I'm gonna test out fsync and see how it reacts on my servers :)

thanks for all your replies and being patient

[chriskessel]

Rather than start a new thread, I'll piggyback on this one.

How will I know when a DB gets corrupted? MySQL starts throwing
errors, but what will MongoDB do? What will I need to do to fix it?
These are questions I have to answer before we'd be allowed to put it
into production.

Also, already if I Control-C the MongoDB process, even if nothing is
going on, the next time I start it it'll say I need to run a repair
DB. In our production environment, there are no writes to the DB
currently (readonly MySQL, updates are to an offline DB that's swapped
in). If a MySQL process dies, we just restart it. I'm going to get
some serious flack if we have to run a repair on MongoDB just because
the process died or the system went down. We'd ideally like to make
updates to the DB in a batch once a day, so while it's read-only now,
it may not be in the future.

From a dev point of view, so far so good with MongoDB, but what's
really going to determine the final call is how it impacts operations.
Operations is done by a separate group in our company, who know
nothing about MongoDB (and probably never will for corporate political
reasons), so MongoDB needs to be completely brain-dead, idiot-proof to
maintain in production. We need an really low rate of run-time
problems and the "fix" to those problems needs to basically be restart
mongod, or at worst, copy in the offline backup DB and restart mongod.

If I set up a replication set or a master-slave setup, am I going to
be able to reach that level of reliability?

[Sam Millman]

Mongo will tell you you need to run a repairdb() function if corruption occurs (or at least has for me)

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Sam Millman]

I never get a problem when I control c mongo it closes gracefully

[sdotsen]

This is what I also want to know.

[Eliot Horowitz]

Corrupted means a lot of different things, and I think there is a lot
of confusion here....

If mongo gets shut down cleanly, there should be no corruption. If
there is, then that is a bug and you would see assertions in the log.
As far as I know - there are no unfixed cases of this happening, nor
should there be.

If mongo gets shutdown uncleanly (power failure, etc...) when you try
to restart it, it will say you have to run a repair first.

[Sam Millman]

that was in my post so I'll say yea

On 9 July 2010 15:57, Eliot Horowitz <elioth...@gmail.com> wrote:

What do you mean you got a tad of corruption?
Did your server get shut down uncleanly?

On Fri, Jul 9, 2010 at 10:14 AM, Sam Millman <smil...@nhbs.co.uk> wrote:
> I did get a tad of corruption on a single server setup (vmed dev server...I
> don't run multi in dev environment) but repairDB() took care of it nicely
> and I didnt loose any data which goes to contradict the blog post again.
>

[Eliot Horowitz]

To which question?

How did you determine it was corruption?
Was this after an unclean shutdown?

[Sam Millman]

How did you determine it was corruption?
Was this after an unclean shutdown?

I remember mongo saying there was corruption or something and asking me to repairDB()

and yea my server had an unclean shutdown.

that is a question: how does repair actually work...does it do the same as MySQL and just remove half inserted rows that have not been completed or does it do some other process?

[sdotsen]

Eliot,

For those who "CTRL-C" the process, should mongo ask you to repair the
DB? I run on Centos, i haven't looked into startup scripts, but I
heard from a friend who had to run repair on his DB. Server didn't
crash, he did a CTRL-C to kill the process in order to reboot the box.

On Jul 9, 11:01 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> To which question?
>
> How did you determine it was corruption?
> Was this after an unclean shutdown?
>

> On Fri, Jul 9, 2010 at 10:59 AM, Sam Millman <smill...@nhbs.co.uk> wrote:
> > that was in my post so I'll say yea

> > On 9 July 2010 15:57, Eliot Horowitz <eliothorow...@gmail.com> wrote:
>
> >> What do you mean you got a tad of corruption?
> >> Did your server get shut down uncleanly?
>

> >> On Fri, Jul 9, 2010 at 10:14 AM, Sam Millman <smill...@nhbs.co.uk> wrote:
> >> > I did get a tad of corruption on a single server setup (vmed dev
> >> > server...I
> >> > don't run multi in dev environment) but repairDB() took care of it
> >> > nicely
> >> > and I didnt loose any data which goes to contradict the blog post again.
>

> >> > On 9 July 2010 15:12, Sam Millman <smill...@nhbs.co.uk> wrote:
>
> >> >> "Just to be clear, we use TCP to communicate with the server, not UDP."
> >> >> Yea I meant if I use safe mode it will wait for a reply from the db
> >> >> before
> >> >> it declares the record is set instead of the default write and forget.
> >> >> Sorry
> >> >> I was trying to be clever with my words...failed massively.
> >> >> I have been using mongo for a few things now (personal) and even on one
> >> >> server setup it hasn't given me any problems. It's only recently I
> >> >> started
> >> >> to question.
> >> >> Yea mongo is vetted by a lot of people and thats why I actually looked
> >> >> at
> >> >> it originally...though it looks like I would be the first to use it as
> >> >> a
> >> >> full replacement for everything in a huge server setup...
> >> >> I did look at the 10Gen posts and thats why I'm torn between the two
> >> >> views...kyle is right I'm gonna need to build something like a chat
> >> >> system
> >> >> and throw it onto my network and see what happens.
>

> >> >> On 9 July 2010 15:04, Kyle Banker <k...@10gen.com> wrote:
>
> >> >>> Just to be clear, we use TCP to communicate with the server, not UDP.
> >> >>> As for all the hearsay, there's certainly a lot of it . There are also
> >> >>> a
> >> >>> lot of happy users, as Kristina points out. People say a lot of
> >> >>> things. The
> >> >>> only way to get the level of assurance you need is to investigate the
> >> >>> system
> >> >>> more deeply on your own and to do some prototyping. That's usually the
> >> >>> best
> >> >>> way to make a technology decision.

> >> >>> On Fri, Jul 9, 2010 at 9:44 AM, Sam Millman <smill...@nhbs.co.uk>

> >> >>>> On 9 July 2010 14:28, Kyle Banker <k...@10gen.com> wrote:
>
> >> >>>>> Answers below:
>

> >> >>>>> On Fri, Jul 9, 2010 at 9:16 AM, Sammaye <sam.mill...@gmail.com>

> ...
>
> read more »

[Kristina Chodorow]

Ctrl-C cleanly shuts down the database.  It is one of the recommended ways of shutting down the database and will not cause corruption or make a repair necessary.

--

[Richard Kreuter]

mongod treats SIGINT the same as SIGTERM: it attempts to shutdown
cleanly in this case.

sdotsen writes:
> Eliot,
>
> For those who "CTRL-C" the process, should mongo ask you to repair the
> DB? I run on Centos, i haven't looked into startup scripts, but I
> heard from a friend who had to run repair on his DB. Server didn't
> crash, he did a CTRL-C to kill the process in order to reboot the box.
>

> On Jul 9, 11:01=A0am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> > To which question?
> >
> > How did you determine it was corruption?
> > Was this after an unclean shutdown?
> >
> > On Fri, Jul 9, 2010 at 10:59 AM, Sam Millman <smill...@nhbs.co.uk> wrote:
> > > that was in my post so I'll say yea
> > > On 9 July 2010 15:57, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> >
> > >> What do you mean you got a tad of corruption?
> > >> Did your server get shut down uncleanly?
> >

> > >> On Fri, Jul 9, 2010 at 10:14 AM, Sam Millman <smill...@nhbs.co.uk> wro=
> te:

> > >> > I did get a tad of corruption on a single server setup (vmed dev
> > >> > server...I
> > >> > don't run multi in dev environment) but repairDB() took care of it
> > >> > nicely

> > >> > and I didnt loose any data which goes to contradict the blog post ag=
> ain.

> >
> > >> > On 9 July 2010 15:12, Sam Millman <smill...@nhbs.co.uk> wrote:
> >

> > >> >> "Just to be clear, we use TCP to communicate with the server, not U=

> DP."
> > >> >> Yea I meant if I use safe mode it will wait for a reply from the db
> > >> >> before

> > >> >> it declares the record is set instead of the default write and forg=
> et.

> > >> >> Sorry
> > >> >> I was trying to be clever with my words...failed massively.

> > >> >> I have been using mongo for a few things now (personal) and even on=

> one
> > >> >> server setup it hasn't given me any problems. It's only recently I
> > >> >> started
> > >> >> to question.

> > >> >> Yea mongo is vetted by a lot of people and thats why I actually loo=
> ked
> > >> >> at
> > >> >> it originally...though it looks like I would be the first to use it=

> as
> > >> >> a
> > >> >> full replacement for everything in a huge server setup...

> > >> >> I did look at the 10Gen posts and thats why I'm torn between the tw=
> o

> > >> >> views...kyle is right I'm gonna need to build something like a chat
> > >> >> system
> > >> >> and throw it onto my network and see what happens.
> >
> > >> >> On 9 July 2010 15:04, Kyle Banker <k...@10gen.com> wrote:
> >

> > >> >>> Just to be clear, we use TCP to communicate with the server, not U=
> DP.
> > >> >>> As for all the hearsay, there's certainly a lot of it . There are =

> also
> > >> >>> a
> > >> >>> lot of happy users, as Kristina points out. People say a lot of
> > >> >>> things. The

> > >> >>> only way to get the level of assurance you need is to investigate =
> the
> > >> >>> system
> > >> >>> more deeply on your own and to do some prototyping. That's usually=

> the
> > >> >>> best
> > >> >>> way to make a technology decision.
> > >> >>> On Fri, Jul 9, 2010 at 9:44 AM, Sam Millman <smill...@nhbs.co.uk>
> > >> >>> wrote:
> >

> > >> >>>> "MongoDB, by default,=A0doesn=92t actually have a response for wr=
> ites.
> > >> >>>> You
> > >> >>>> just write your data to the socket and assume it=92s going to be
> > >> >>>> available
> > >> >>>> when you try to read it. Under concurrent load you can=92t expect=
> to
> > >> >>>> reliably
> > >> >>>> store stateful data like session information like this. It=92s ki=
> nd of
> > >> >>>> like,
> > >> >>>> in your webapp, if you were to spawn a thread to do some work and=
> at
> > >> >>>> the end
> > >> >>>> set some global but you return a response to the client immediate=
> ly.
> > >> >>>> Where
> > >> >>>> there wasn=92t any load you would be fine because your thread is =
> faster
> > >> >>>> than
> > >> >>>> the roundtrip time to the client. But under heavy load the operat=
> ions
> > >> >>>> in the
> > >> >>>> thread could take too long and a subsequent request wouldn=92t ha=
> ve
> > >> >>>> access to
> > >> >>>> that global because it wasn=92t set yet. This is why you can=92t =
> store
> > >> >>>> session
> > >> >>>> data reliably in MongoDB without changing the default client opti=
> on
> > >> >>>> to
> > >> >>>> return a response, because no matter how =93fast=94 it is if you =
> send a
> > >> >>>> response
> > >> >>>> to the client (or no response at all) before the data is availabl=
> e
> > >> >>>> for read
> > >> >>>> it=92s useless.
> >
> > >> >>>> To people who don=92t live in databases all day it=92s hard to ex=
> plain
> > >> >>>> just
> > >> >>>> how odd this choice is. I don=92t know of another database that e=
> ven

> > >> >>>> allows
> > >> >>>> you to return a response before the write data is accessible much
> > >> >>>> less not

> > >> >>>> even have a response by default. This is kind of like using UDP f=
> or
> > >> >>>> data
> > >> >>>> that you care about getting somewhere, it=92s theoretically faste=
> r but
> > >> >>>> the
> > >> >>>> importance of making sure your data gets somewhere and is accessi=
> ble

> > >> >>>> is
> > >> >>>> almost always more important."
> >
> > >> >>>> So if I use safe mode it would be like TCP instead of UDP when I
> > >> >>>> transfer data to mongo (using the same metaphore as the author).
> >
> > >> >>>> "Not sure how to respond to that. Sounds like a pretty vague
> > >> >>>> assertion.
> > >> >>>> If it's just a matter of time before a given master server will
> > >> >>>> die unexpectedly, then yes, the master could see corruption on an
> > >> >>>> unclean shutdown like that."

> > >> >>>> It's just what I keep hearing. That mongo is good as front cache =

> but
> > >> >>>> for
> > >> >>>> say the backend of a full site it's not really that reliable.

> > >> >>>> You see I want to use mongo, but I want to make sure that if I se=

> t it
> > >> >>>> up
> > >> >>>> right it would be just as durable and reliable as say MySQL.
> >
> > >> >>>> On 9 July 2010 14:28, Kyle Banker <k...@10gen.com> wrote:
> >
> > >> >>>>> Answers below:
> >
> > >> >>>>> On Fri, Jul 9, 2010 at 9:16 AM, Sammaye <sam.mill...@gmail.com>
> > >> >>>>> wrote:
> >

> > >> >>>>>> Hi, thanks for your response. Yes I do not plan to use mongo si=
> ngle
> > >> >>>>>> server at all (my intranet site currently houses 25 SQL servers=
> ).
> > >> >>>>>> But
> > >> >>>>>> I do need a db I can trust to always send information in and st=
> ore

> > >> >>>>>> it
> > >> >>>>>> in a manner that means only when there is a down (server fault,
> > >> >>>>>> earth
> > >> >>>>>> quake etc) I or another administrator would require to do
> > >> >>>>>> something.
> > >> >>>>>> I've just heard from many that it's only a matter of time until
> > >> >>>>>> mongo
> > >> >>>>>> becomes corrupt even on a perfectly fine replication system.
> >
> > >> >>>>> Not sure how to respond to that. Sounds like a pretty vague
> > >> >>>>> assertion.
> > >> >>>>> If it's just a matter of time before a given master server will

> > >> >>>>> die unexpectedly, then yes, the master could see corruption on a=
> n

> > >> >>>>> unclean shutdown like that.
> >
> > >> >>>>>> What about reports of data not always reaching db?
> >

> > >> >>>>> By default, writes from the drivers are 'fire-and-forget.' This =

> is
> > >> >>>>> an
> > >> >>>>> intentional design decision. However, if you need durable writes
> > >> >>>>> that are
> > >> >>>>> assured to reach the db, use the safe mode implemented by all
> > >> >>>>> drivers.
> >
> > >> >>>>>> Does mongo ensure that any data I send in will reach a db and a
> > >> >>>>>> collection?
> >
> > >> >>>>> See above.
> >

> > >> >>>>>> On Jul 9, 2:08=A0pm, Kyle Banker <k...@10gen.com> wrote:
> > >> >>>>>> > We don't support traditional single-server durability; MongoD=
> B
> > >> >>>>>> > achieves
> > >> >>>>>> > durability through replication and judicious use of snapshott=
> ing.
> >
> > >> >>>>>> > At the moment, you can run a master with any number of slaves=
> ,
> > >> >>>>>> > placing one
> > >> >>>>>> > in a separate data center if needed. If a master server dies =
> or
> > >> >>>>>> > has
> > >> >>>>>> > an
> > >> >>>>>> > unclean shutdown, you could see some corruption on that maste=
> r

> > >> >>>>>> > node,
> > >> >>>>>> > but in
> > >> >>>>>> > that situation, you fail over to a slave. With replica sets

> > >> >>>>>> > (http://www.mongodb.org/display/DOCS/Replica+Sets), you have =

> a
> > >> >>>>>> > set of N
> > >> >>>>>> > replicas, and the failover is handled automatically.
> >

> > >> >>>>>> > For ops that need the most durability, you can set a W value =

> on
> > >> >>>>>> > writes to
> > >> >>>>>> > ensure that the write is replicated to at least N nodes. So
> > >> >>>>>> > really,
> > >> >>>>>> > the

> > >> >>>>>> > level of durability is configurable. With auto-sharding, whic=
> h
> > >> >>>>>> > will
> > >> >>>>>> > be
> > >> >>>>>> > production-ready by the end of the month, you'll get automate=
> d

> > >> >>>>>> > failover
> > >> >>>>>> > across automatically managed sharded nodes, which may be what
> > >> >>>>>> > your
> > >> >>>>>> > system
> > >> >>>>>> > will require.
> >
> > >> >>>>>> > Here's more on our
> >

> > >> >>>>>> > philosophy:http://blog.mongodb.org/post/381927266/what-about-=
> durability

> >
> > >> >>>>>> > Yes, we look at the durability issue differently. But that
> > >> >>>>>> > doesn't
> > >> >>>>>> > mean that

> > >> >>>>>> > we have the wrong approach. I think that the biggest testamen=

> t to
> > >> >>>>>> > the
> > >> >>>>>> > viability of our design is the number of working production
> > >> >>>>>> > MongoDB
> > >> >>>>>> > deployments. You can see many of them

> > >> >>>>>> > here:http://www.mongodb.org/display/DOCS/Production+Deploymen=
> ts
> >
> > >> >>>>>> > On Fri, Jul 9, 2010 at 3:34 AM, Sammaye <sam.mill...@gmail.co=

> m>
> > >> >>>>>> > wrote:
> > >> >>>>>> > > Hi everyone,
> >
> > >> >>>>>> > > I recently read this whilst using my mongodb:
> >

> > >> >>>>>> > > >http://www.mikealrogers.com/2010/07/mongodb-performance-du=
> rability/
> >
> > >> >>>>>> > > And it's got me thinking. He is basically saying that even =

> with
> > >> >>>>>> > > replication on 3 or more servers I will still get data
> > >> >>>>>> > > corruption
> > >> >>>>>> > > and
> > >> >>>>>> > > the such.
> >

> > >> >>>>>> > > I was hoping I could ask you guys exactly how reliable mong=

> o db
> > >> >>>>>> > > has
> > >> >>>>>> > > become.
> >
> > >> >>>>>> > > I am asking since I wish to use it as a backbone technology
> > >> >>>>>> > > within
> > >> >>>>>> > > a
> > >> >>>>>> > > very large (multimillion user) intranet system for a huge
> > >> >>>>>> > > global

> > >> >>>>>> > > company and it's related partners, but this blog post worri=
> es
> > >> >>>>>> > > me
> > >> >>>>>> > > and
> > >> >>>>>> > > make me think I need to use MySQL or something (which is wh=
> at

> > >> >>>>>> > > is
> > >> >>>>>> > > currently used and is....sorry for the swear....shit).
> >

> > >> >>>>>> > > To get an idea of how heavy my intranet site is just think =
> of
> > >> >>>>>> > > facebook
> > >> >>>>>> > > or youtube (though they're not as heavy but gives you an id=
> ea

> > >> >>>>>> > > of
> > >> >>>>>> > > traffic and load).
> >
> > >> >>>>>> > > Thanks,
> >
> > ...
> >

> > read more =BB
>
> --=20
> You received this message because you are subscribed to the Google Groups "=

> mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.

> To unsubscribe from this group, send email to mongodb-user+unsubscribe@goog=
> legroups.com.
> For more options, visit this group at http://groups.google.com/group/mongod=
> b-user?hl=3Den.
>

[Andreas Jung]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kristina Chodorow wrote:
> Ctrl-C cleanly shuts down the database. It is one of the recommended
> ways of shutting down the database and will not cause corruption or make
> a repair necessary.
>

I evaluated MongoDB over several weeks in various scenarios and have
never seen any corruption by using ctrl-c - and I used it very
frequently. Reports of data corruption with the latest 1.4.X versions of
MongoDB caused by using ctrl-c are a myth.

- -aj
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkw3PKYACgkQCJIWIbr9KYxFhwCfQWGxF6v4Q225hB00PonCBetG
aVcAnRZ8c5cCVEQrpewVnNpub3f/5YBs
=45et
-----END PGP SIGNATURE-----

[Sam Millman]

http://173.203.115.147/Untitled-2.png like so

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Andreas Jung]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This tells us what? (except showing a clean shutdown)?

- -aj

Sam Millman wrote:
> http://173.203.115.147/Untitled-2.png like so
>
> On 9 July 2010 16:13, Andreas Jung <li...@zopyx.com

> <mailto:li...@zopyx.com>> wrote:
>
> Kristina Chodorow wrote:
>> Ctrl-C cleanly shuts down the database. It is one of the recommended
>> ways of shutting down the database and will not cause corruption
> or make
>> a repair necessary.
>
>
> I evaluated MongoDB over several weeks in various scenarios and have
> never seen any corruption by using ctrl-c - and I used it very
> frequently. Reports of data corruption with the latest 1.4.X versions of
> MongoDB caused by using ctrl-c are a myth.
>

> -aj

- --

You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com

<mailto:mongod...@googlegroups.com>.

To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com

<mailto:mongodb-user%2Bunsu...@googlegroups.com>.

For more options, visit this group at
http://groups.google.com/group/mongodb-user?hl=en.

> --
> Bow Chicka Bow Wow

> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

- --
ZOPYX Limited | zopyx group
Charlottenstr. 37/1 | The full-service network for Zope & Plone
D-72070 T�bingen | Produce & Publish
www.zopyx.com | www.produce-and-publish.com
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkw3Pk4ACgkQCJIWIbr9KYyO2wCfZeFSgQuppNL9ov8T5OVBa0E8
Nc4AoNf3AH9VznAQyqMOGz2w2rAspsTd
=WY/n
-----END PGP SIGNATURE-----

[Richard Kreuter]

Sam,

In case there's some doubt, that's a clean shutdown you're looking at
there.

--
Richard

[Eliot Horowitz]

After a unclean shutdown, mongo will say it was not shutdown cleanly,
and ask you to do a repair.

This is absolutely not the same as corruption, this is mongo saying it
can't 100% verify what's going on, and to be paranoid, run a repair.

Terminology is very important these days...

[Sam Millman]

For those who "CTRL-C" the process, should mongo ask you to repair the
DB? I run on Centos, i haven't looked into startup scripts, but I
heard from a friend who had to run repair on his DB. Server didn't
crash, he did a CTRL-C to kill the process in order to reboot the box.

in response to that

On 9 July 2010 16:25, Sam Millman <smil...@nhbs.co.uk> wrote:

my screenshot shows the crtl-c command being initiated and what should happen

[Sam Millman]

Eliot,

Ah I see; OK I get ya now.

[Sam Millman]

Ok one more time, i posted in response to:

For those who "CTRL-C" the process, should mongo ask you to repair the
DB? I run on Centos, i haven't looked into startup scripts, but I
heard from a friend who had to run repair on his DB. Server didn't
crash, he did a CTRL-C to kill the process in order to reboot the box.

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Giorgio]

Another point of that blog post is that when corruption occurs it can
be harder to recover from it because mongodb doesn't use an append
only log.

Just to be clear. Will this be fixed in the 1.8 release?

What other changes that affect single server durability are coming?

On Jul 9, 11:30 am, Sam Millman <smill...@nhbs.co.uk> wrote:
> Ok one more time, i posted in response to:
>
> For those who "CTRL-C" the process, should mongo ask you to repair the
> DB? I run on Centos, i haven't looked into startup scripts, but I
> heard from a friend who had to run repair on his DB. Server didn't
> crash, he did a CTRL-C to kill the process in order to reboot the box.
>

> On 9 July 2010 16:22, Richard Kreuter <rich...@10gen.com> wrote:
>
>
>
>
>
> > Sam,
>
> > In case there's some doubt, that's a clean shutdown you're looking at
> > there.
>
> > --
> > Richard
>

> > Sam Millman <smill...@nhbs.co.uk> wrote:
>
> > >http://173.203.115.147/Untitled-2.pnglike so

>
> > > On 9 July 2010 16:13, Andreas Jung <li...@zopyx.com> wrote:
>
> > >     -----BEGIN PGP SIGNED MESSAGE-----
> > >     Hash: SHA1
>
> > >     Kristina Chodorow wrote:
> > >     > Ctrl-C cleanly shuts down the database.  It is one of the
> > recommended
> > >     > ways of shutting down the database and will not cause corruption or
> > make
> > >     > a repair necessary.
>
> > >     I evaluated MongoDB over several weeks in various scenarios and have
> > >     never seen any corruption by using ctrl-c  - and I used it very
> > >     frequently. Reports of data corruption with the latest 1.4.X versions
> > of
> > >     MongoDB caused by using ctrl-c are a myth.
>
> > >     - -aj
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongod...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsubscribe@google groups.com>
> > .

[Eliot Horowitz]

1.8 will have durability. This means an append log or something like that

Note this is not a bug, but a decision about what we think the best way to guarantee durability on a set of servers

Can see my blog post for more in depth notes.

> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[Giorgio]

Thanks. That was a bad choice of words on my part. I understand that
it was a design choice not a bug.

I'm always amazed at the responsiveness of the 10gen guys. Keep up the
great work.

On Jul 9, 11:48 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> 1.8 will have durability.  This means an append log or something like that
>
> Note this is not a bug, but a decision about what we think the best way to guarantee durability on a set of servers
>
> Can see my blog post for more in depth notes.
>

[chriskessel]

On Jul 9, 8:13 am, Andreas Jung <li...@zopyx.com> wrote:
>Reports of data corruption with the latest 1.4.X versions of
> MongoDB caused by using ctrl-c are a myth.

Maybe 1.4.x is immune, but I've had it a couple times with 1.5.3.
It's rare, but I've had it. Maybe something other than the Control-C
was responsible though.

[Eliot Horowitz]

If you've actually have issues with ctrl c and 1.5.3 please report. It's the unstable release but bug reports are vital to making sure 1.6 is stable.

[Sam Millman]

So 1.8 will contain durability, sweet :D. That should solve all my previous questions about mongo

On 9 July 2010 17:05, Eliot Horowitz <elioth...@gmail.com> wrote:

[chriskessel]

On Jul 9, 9:05 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> If you've actually have issues with ctrl c and 1.5.3 please report. It's the unstable release but bug reports are vital to making sure 1.6 is stable.

Will do. So far, I can't reproduce it, which leads me to believe it
probably wasn't the Control-C, but that the DB was going to need
repair on the next restart regardless based on some other cause.

[Kenny Gorman]

On Jul 9, 2010, at 8:24 AM, Eliot Horowitz wrote:

> After a unclean shutdown, mongo will say it was not shutdown cleanly,
> and ask you to do a repair.
>
> This is absolutely not the same as corruption, this is mongo saying it
> can't 100% verify what's going on, and to be paranoid, run a repair.

This is a very very very good point.

[Sam Millman]

Yea it should be stated somewhere clearly thats the case, in big bold writing so no one can miss it, like I did

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Kristina Chodorow]

I added a big warning box on this page: http://www.mongodb.org/display/DOCS/Starting+and+Stopping+Mongo

More obvious?

[Sam Millman]

Damn I must be blind should go get my eyes checked out

[Kenny Gorman]

All this hubub about durability is missing one key point: hardware fails too.

fsync() is just a first step in a whole process of events.  If you are on a SAN there are even more steps until your data is on disk.  Don't be fooled into thinking that just because something has the word 'durable' on it that everything is safe.  You still have to understand the stack underneath the DB and how it is setup.

PostgreSQL is highly durable.  Yet, I spent 2 weeks day and night fixing (really deleting) corrupt data when a hardware bug caused corruption to our data and PostgreSQL didn't report anything and kept going.  I mean *lots* of data.  It was horrible.

Typically if I had to choose between single server durability and N replication as durability I would choose the latter.

-kg

[Sam Millman]

Eliot wrote:

1.8 will have durability.  This means an append log or something like that

Note this is not a bug, but a decision about what we think the best way to guarantee durability on a set of servers

Can see my blog post for more in depth notes.

[Joe Bowman]

For the post that inspired this discussion, my response is here -
http://joerussbowman.tumblr.com/post/790280680/some-thoughts-on-the-mongodb-durability-issue

I have seen some issues with ctrl-c making me a run --repair-all on
the next startup, but I've never seen it be a problem. And this has
been on a laptop with not the most reliable drive.

[Riyad]

Ok this topic worked me into a tizzy -- I kept seeing things like
"fsync your stuff!" or "use your driver in safe mode!" and no where
except for 1 Ruby example in the original conversation did I see
someone actually say how to do that stuff...

So I wrote it up into a huge guide that I hope you guys find helpful:
http://www.thebuzzmedia.com/mongodb-single-server-data-durability-guide/

Any comments, corrections or feedback, please leave a comment there (I
don't get reply notifications for this thread). I really just want it
to be a handy resource for folks dealing with Mongo now that are
worried about durability (like I was) and it gives some tools to
combat it as effectively as they can *right now*, and eventually when
Replica Sets are done (1.6) and transaction log (1.8) then they have
many more options.


On Jul 9, 11:32 am, Joe Bowman <bowman.jos...@gmail.com> wrote:
> For the post that inspired this discussion, my response is here -http://joerussbowman.tumblr.com/post/790280680/some-thoughts-on-the-m...

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/09/2010 07:58 AM, Eliot Horowitz wrote:
> If mongo gets shutdown uncleanly (power failure, etc...) when you try
> to restart it, it will say you have to run a repair first.

Using your Ubuntu build it only says this once. ie if you try to start it a
second time then it will start just fine. The "saying" goes to the log file.

I did lose database contents on a virtual machine the other day as the user
didn't cleanly shut it down some of the time, and ran 'sudo start mongodb'
if it didn't appear to be running. Do that a few times, and suddenly all
the data disappears (well everything except for 4 out of 1.6 million
documents :-) This was a development environment, single unreplicated
server etc so I'm not bothered.

Or put in the form of a ticket:

http://jira.mongodb.org/browse/SERVER-1386

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkw3cN4ACgkQmOOfHg372QRc0ACfUh2NguqmdXkxLBGXpN77ilbE
dlQAoJnpbKaev4ij2NskQQki9peuFsMr
=UKhv
-----END PGP SIGNATURE-----

[sdotsen]

Nice, I was curious as well! Now I have to comb through the PHP
drivers page to see how it's done.

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/09/2010 10:43 AM, Kenny Gorman wrote:
> PostgreSQL is highly durable. Yet, I spent 2 weeks day and night fixing
> (really deleting) corrupt data when a hardware bug caused corruption to
> our data and PostgreSQL didn't report anything and kept going. I mean
> *lots* of data. It was horrible.

I assume that Postgres does not have a per record checksum of some sort so
there is effectively no possibility to detect externally corrupted data.

Does Mongo have any form of record checksum?

CouchDB did gain it a while back, mainly to help replication, but it does at
least mean there is a way of verifying contents just as with modern version
control systems like git and Mercurial.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkw3cjQACgkQmOOfHg372QTuZQCg4dp0y+3Ljm7yfpdUzEfWZy+o
i8IAoJNfw+Ye9eDkIuGaJSX999F+4b0N
=2YI+
-----END PGP SIGNATURE-----

[sdotsen]

Oh, I had the same exact issue with an instance running on my VM. I
run mongo inside a centos VM on my desktop for development purposes. A
couple of times my VM would crash or I totally forgot to pause it when
I shut down my desktop (the host). Upon boot up, I go run mongo and it
wouldn't start. I ran ps -ef to check and nothing. So I ran it a
second time and it worked. I didn't think much of it, but it didnt ask/
require me to run dbrepair or anything. Next time this happens I'll
definitely check the logs.

On Jul 9, 2:56 pm, Roger Binns <rog...@rogerbinns.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 07/09/2010 07:58 AM, Eliot Horowitz wrote:
>
> > If mongo gets shutdown uncleanly (power failure, etc...) when you try
> > to restart it, it will say you have to run a repair first.
>
> Using your Ubuntu build it only says this once.  ie if you try to start it a
> second time then it will start just fine.  The "saying" goes to the log file.
>
> I did lose database contents on a virtual machine the other day as the user
> didn't cleanly shut it down some of the time, and ran 'sudo start mongodb'
> if it didn't appear to be running.  Do that a few times, and suddenly all
> the data disappears (well everything except for 4 out of 1.6 million
> documents :-)  This was a development environment, single unreplicated
> server etc so I'm not bothered.
>
> Or put in the form of a ticket:
>
>  http://jira.mongodb.org/browse/SERVER-1386
>
> Roger
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)

> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/

[Riyad]

sdotsen,

I should probably add PHP and Ruby examples in there, the good news is
that the raw JSON commands that I'm sending as Java strings are the
only "important" part -- so however the PHP driver issues inserts/
deletes/etc. to the server, you just issue the command, like "{fsync:
1}", and that's it.

Hopefully not too hairy with PHP driver.

[sdotsen]

Gotcha, I'll have to look into it. Thanks.

[chriskessel]

On Jul 9, 11:56 am, Roger Binns <rog...@rogerbinns.com> wrote:
> I did lose database contents on a virtual machine the other day as the user
> didn't cleanly shut it down some of the time, and ran 'sudo start mongodb'
> if it didn't appear to be running.  Do that a few times, and suddenly all
> the data disappears (well everything except for 4 out of 1.6 million
> documents :-)  This was a development environment, single unreplicated
> server etc so I'm not bothered.

You weren't bothered? This would be a huge deal breaker for our
deployment. Frankly, I'm stunned you weren't bothered by losing your
entire database. Single server or not, losing the entire database and
not being aware of it until it's long gone is completely out of the
question for a production environment (at least for me).

We use virtualized machines and typically the operations folks start
and stop everything via sudo commands. Which, if I'm reading it right,
is exactly the situation you're describing. We have to have zero data
loss. Or, at a minimum, know exactly when data is lost so we can
restore from backup.

Maybe one of the 10gen folks can speak to this, because if what Frank
says is true, I can pretty much stop my POC now. :(

[Michael Dirolf]

It doesn't matter what command you're using to start mongod - you
should *always* be checking the log after starting it. I think Roger's
case for automatically doing a repair is probably a good idea - maybe
Eliot, et. al. have more thoughts on it though - should probably track
that case.

If mongod needs to be repaired it won't start - it will print a
message and error out when you try to start it. The correct response
when that happens is to not just try restarting again w/o checking the
logs and seeing why it didn't start.

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/09/2010 12:16 PM, chriskessel wrote:
> You weren't bothered? This would be a huge deal breaker for our
> deployment. Frankly, I'm stunned you weren't bothered by losing your
> entire database. Single server or not, losing the entire database and
> not being aware of it until it's long gone is completely out of the
> question for a production environment (at least for me).

It was a development environment not production. Repair was never run. It
was single server. The virtual machine was stopped a few times either by
the host being yanked (a laptop) or the virtual machine being stopped
(equivalent of a power pull). If you were trying to corrupt/lose data then
this is pretty much the recipe to follow. Why would I be bothered on
succeeding?

SQLite does include a nice section on how to corrupt your database. Perhaps
there should be something in a similar spirit on the Mongo site?

http://www.sqlite.org/lockingv3.html#how_to_corrupt

> We use virtualized machines and typically the operations folks start
> and stop everything via sudo commands. Which, if I'm reading it right,
> is exactly the situation you're describing.

Nope. Things were not stopped via commands (which would be clean) but
rather by abruptly terminating the virtual machine (equivalent of pulling
the power cable) or terminating the host (a laptop). As I said, this was a
development environment which is why things like that were done.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkw3d/gACgkQmOOfHg372QTlKACfZxHp/0o5FRi5QSjGVCQqfocP
NUMAnjAFfXMk+VrCv5AKX+T4RZJ2ATHW
=VO6f
-----END PGP SIGNATURE-----

[sdotsen]

Mike, I beg to differ. Taking mysql as an example, there were times
when mysql failed for some reason in production. I wasn't able to
start it for whatever reason but I got an error message right away on
the console. With mongo, I used the fork option to put the process in
the background. I don't always remember to check the log file and go
about coding until I realize my code doesn't work cause mongo didn't
start.

If I'm running the process in the foreground then yes it shows error
messages.

On Jul 9, 3:20 pm, Michael Dirolf <m...@10gen.com> wrote:
> It doesn't matter what command you're using to start mongod - you
> should *always* be checking the log after starting it. I think Roger's
> case for automatically doing a repair is probably a good idea - maybe
> Eliot, et. al. have more thoughts on it though - should probably track
> that case.
>
> If mongod needs to be repaired it won't start - it will print a
> message and error out when you try to start it. The correct response
> when that happens is to not just try restarting again w/o checking the
> logs and seeing why it didn't start.
>

[Michael Dirolf]

Yeah it's a bug that restarting clears the lock file (so restarting
twice causes the error message to stop being printed) - we're in the
process of fixing that.

Automatic repair is probably not a great idea, but failing to start no
matter how many times you try is. This behavior should be fixed soon.

[chriskessel]

On Jul 9, 12:20 pm, Michael Dirolf <m...@10gen.com> wrote:
> It doesn't matter what command you're using to start mongod - you
> should *always* be checking the log after starting it. I think Roger's
> case for automatically doing a repair is probably a good idea - maybe
> Eliot, et. al. have more thoughts on it though - should probably track
> that case.

Always checking the log simply isn't feasibly in some environments.
We're going to have probably 30 systems running mongod, we do rolling
restarts 1/week to accommodate certain maintenance windows. Don't ask
why, it doesn't matter as I couldn't change that process barring an
act of congress. The scripts that do this are automated and when they
fail they're basically manually run by operations people who have zero
knowledge of even what the product is. An automated --repair sounds
like it could certainly help in my environment.

It's only when a cluster---k happens they call the folks who built it
(me and others) to figure out how to get things moving again. To many
of those calls, and I might as well find a new job.

So, when I say MongoDB has to be reliable and ultra-trivial to fix,
I'm serious :). I have to convince C-level managers that this is going
to reduce operational costs, or at least not increase them, over a
proposed Oracle solution (yea, that'd include a 10gen contract).

[Richard Kreuter]

Roger,

You're describing a number of issues here, but the fact that starting
mongod twice broke the check for unclean shutdown is a bug, which is now
fixed. Thanks for pointing it out!

--
Richard

Roger Binns <rog...@rogerbinns.com> wrote:

> On 07/09/2010 07:58 AM, Eliot Horowitz wrote:
> > If mongo gets shutdown uncleanly (power failure, etc...) when you try
> > to restart it, it will say you have to run a repair first.
>
> Using your Ubuntu build it only says this once. ie if you try to start it a
> second time then it will start just fine. The "saying" goes to the log file.
>
> I did lose database contents on a virtual machine the other day as the user
> didn't cleanly shut it down some of the time, and ran 'sudo start mongodb'
> if it didn't appear to be running. Do that a few times, and suddenly all
> the data disappears (well everything except for 4 out of 1.6 million
> documents :-) This was a development environment, single unreplicated
> server etc so I'm not bothered.
>
> Or put in the form of a ticket:
>
> http://jira.mongodb.org/browse/SERVER-1386
>
> Roger
>

[Sam Millman]

Riyad, nice post most def will follow it.

[chriskessel]

On Jul 9, 12:26 pm, Roger Binns <rog...@rogerbinns.com> wrote:
> It was a development environment not production.  Repair was never run.  It
> was single server.  The virtual machine was stopped a few times either by
> the host being yanked (a laptop) or the virtual machine being stopped
> (equivalent of a power pull).

I still am stunned the escalating data corruption s acceptable even in
that situation. I can understand new writes not being seen, but
yanking cords shouldn't be corrupting existing data. It's not like the
bits on the physical hard drive are being twisted by the harsh
shutdown.

[Jason J. W. Williams]

An append-only log will be a huge improvement. Now if Mongo would play
nicely on a box shared with other processes my life would be complete.

-J

[Chuck Remes]

Chris,

you (and I) really have no information about this at all. Some file some where on some VM maybe got lost when things were yanked, pulled, etc. Is it reproducible? No one knows. No one is even sure what happened!

I wouldn't get worked up into a lather about this. It's entirely possible the person using that VM typed "del mongod.exe" or something similar. We. Just. Don't. Know.

I've been using mongod since version 1.2 and have pumped hundreds of gigs through most of the intervening versions. I haven't lost a single byte and I've had plenty of hardware failures in the interim.

See that? My anecdotal data trumps the earlier poster's. :)

If this were a problem, there would be tickets opened up for it. Check the archives for all the people asking about running mongo on EC2 and other virtualized platforms. If there were problems, they would be OBVIOUS.

cr

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/09/2010 01:17 PM, chriskessel wrote:
> I still am stunned the escalating data corruption s acceptable even in
> that situation.

There wasn't escalating data corruption. It was all there and then wasn't.
The data is almost entirely read-only used for our application, and is
generated from an external data source, with lots of copies all over the place.

The virtual machine was also configured to use the host I/O cache (this is a
virtualbox settting) so even if the guest OS called fsync it wouldn't
actually be on real disk platters. Memory settings for the guest (RAM and
swap) were also quite small so an OOM killer may also have been called.

As I said earlier, it was pretty much following the checklist on how to
corrupt your database with MongoDB. It was a development environment on a
crappy Windows laptop (guest Ubuntu). The user couldn't care less about
MongoDB or any of the other components - they just wanted to try some
features of my code.

Hopefully it will now be more obvious why the server has failed to start and
that doing a repair is easy to describe over the phone to a Windows user.

http://jira.mongodb.org/browse/SERVER-1386

It should also be very clear that the way you do things in a throwaway
development environment is *very* different than what you would do in
production.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkw3h4IACgkQmOOfHg372QT/GQCeOFbQuO5voAXttj229IqSYKw3
rNQAoIlsdC3hXXcvlvRHGhn1oFzI8VAY
=NpYN
-----END PGP SIGNATURE-----

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/09/2010 01:24 PM, Jason J. W. Williams wrote:
> An append-only log will be a huge improvement.

If done well. It was what drove me away from CouchDB where the entire data
storage format is an append only log. I was getting massive file sizes (eg
23GB to store 2GB of data, 43GB of "index" data). Dealing with that much
I/O plus the data being scattered amongst haystacks that size lead to very
poor performance.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkw3iOIACgkQmOOfHg372QTGJACcDMiOSLcUBz4fp2U1m7KtOh7V
wqcAn06c88cWYs0NZYFUebfCUBCzSxDP
=/VSk
-----END PGP SIGNATURE-----

[kevin]

i am running mongodb, via runit, so i cannot do CTRL-C.
i do
kill pid

is this OK to do? what is the recommended way?

On Fri, Jul 9, 2010 at 8:09 AM, Kristina Chodorow <kris...@10gen.com> wrote:

Ctrl-C cleanly shuts down the database.  It is one of the recommended ways of shutting down the database and will not cause corruption or make a repair necessary.

On Fri, Jul 9, 2010 at 11:07 AM, sdotsen <samna...@gmail.com> wrote:

Eliot,

For those who "CTRL-C" the process, should mongo ask you to repair the
DB? I run on Centos, i haven't looked into startup scripts, but I
heard from a friend who had to run repair on his DB. Server didn't
crash, he did a CTRL-C to kill the process in order to reboot the box.

On Jul 9, 11:01 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> To which question?
>
> How did you determine it was corruption?
> Was this after an unclean shutdown?
>

> On Fri, Jul 9, 2010 at 10:59 AM, Sam Millman <smill...@nhbs.co.uk> wrote:
> > that was in my post so I'll say yea

> > On 9 July 2010 15:57, Eliot Horowitz <eliothorow...@gmail.com> wrote:
>
> >> What do you mean you got a tad of corruption?
> >> Did your server get shut down uncleanly?
>

> >> On Fri, Jul 9, 2010 at 10:14 AM, Sam Millman <smill...@nhbs.co.uk> wrote:
> >> > I did get a tad of corruption on a single server setup (vmed dev
> >> > server...I
> >> > don't run multi in dev environment) but repairDB() took care of it
> >> > nicely
> >> > and I didnt loose any data which goes to contradict the blog post again.
>

> >> > On 9 July 2010 15:12, Sam Millman <smill...@nhbs.co.uk> wrote:
>
> >> >> "Just to be clear, we use TCP to communicate with the server, not UDP."
> >> >> Yea I meant if I use safe mode it will wait for a reply from the db
> >> >> before
> >> >> it declares the record is set instead of the default write and forget.
> >> >> Sorry
> >> >> I was trying to be clever with my words...failed massively.
> >> >> I have been using mongo for a few things now (personal) and even on one
> >> >> server setup it hasn't given me any problems. It's only recently I
> >> >> started
> >> >> to question.
> >> >> Yea mongo is vetted by a lot of people and thats why I actually looked
> >> >> at
> >> >> it originally...though it looks like I would be the first to use it as
> >> >> a
> >> >> full replacement for everything in a huge server setup...
> >> >> I did look at the 10Gen posts and thats why I'm torn between the two
> >> >> views...kyle is right I'm gonna need to build something like a chat
> >> >> system
> >> >> and throw it onto my network and see what happens.
>

> >> >> On 9 July 2010 15:04, Kyle Banker <k...@10gen.com> wrote:
>
> >> >>> Just to be clear, we use TCP to communicate with the server, not UDP.
> >> >>> As for all the hearsay, there's certainly a lot of it . There are also
> >> >>> a
> >> >>> lot of happy users, as Kristina points out. People say a lot of
> >> >>> things. The
> >> >>> only way to get the level of assurance you need is to investigate the
> >> >>> system
> >> >>> more deeply on your own and to do some prototyping. That's usually the
> >> >>> best
> >> >>> way to make a technology decision.

> >> >>> On Fri, Jul 9, 2010 at 9:44 AM, Sam Millman <smill...@nhbs.co.uk>

> >> >>> wrote:
>
> >> >>>> "MongoDB, by default, doesn’t actually have a response for writes.
> >> >>>> You
> >> >>>> just write your data to the socket and assume it’s going to be
> >> >>>> available
> >> >>>> when you try to read it. Under concurrent load you can’t expect to
> >> >>>> reliably
> >> >>>> store stateful data like session information like this. It’s kind of
> >> >>>> like,
> >> >>>> in your webapp, if you were to spawn a thread to do some work and at
> >> >>>> the end
> >> >>>> set some global but you return a response to the client immediately.
> >> >>>> Where
> >> >>>> there wasn’t any load you would be fine because your thread is faster
> >> >>>> than
> >> >>>> the roundtrip time to the client. But under heavy load the operations
> >> >>>> in the
> >> >>>> thread could take too long and a subsequent request wouldn’t have
> >> >>>> access to
> >> >>>> that global because it wasn’t set yet. This is why you can’t store
> >> >>>> session
> >> >>>> data reliably in MongoDB without changing the default client option
> >> >>>> to
> >> >>>> return a response, because no matter how “fast” it is if you send a
> >> >>>> response
> >> >>>> to the client (or no response at all) before the data is available
> >> >>>> for read
> >> >>>> it’s useless.
>
> >> >>>> To people who don’t live in databases all day it’s hard to explain
> >> >>>> just
> >> >>>> how odd this choice is. I don’t know of another database that even
> >> >>>> allows
> >> >>>> you to return a response before the write data is accessible much
> >> >>>> less not
> >> >>>> even have a response by default. This is kind of like using UDP for
> >> >>>> data
> >> >>>> that you care about getting somewhere, it’s theoretically faster but
> >> >>>> the
> >> >>>> importance of making sure your data gets somewhere and is accessible
> >> >>>> is
> >> >>>> almost always more important."
>
> >> >>>> So if I use safe mode it would be like TCP instead of UDP when I
> >> >>>> transfer data to mongo (using the same metaphore as the author).
>
> >> >>>> "Not sure how to respond to that. Sounds like a pretty vague
> >> >>>> assertion.
> >> >>>> If it's just a matter of time before a given master server will
> >> >>>> die unexpectedly, then yes, the master could see corruption on an
> >> >>>> unclean shutdown like that."
> >> >>>> It's just what I keep hearing. That mongo is good as front cache but
> >> >>>> for
> >> >>>> say the backend of a full site it's not really that reliable.
> >> >>>> You see I want to use mongo, but I want to make sure that if I set it
> >> >>>> up
> >> >>>> right it would be just as durable and reliable as say MySQL.
>

> >> >>>> On 9 July 2010 14:28, Kyle Banker <k...@10gen.com> wrote:
>
> >> >>>>> Answers below:
>

> >> >>>>> On Fri, Jul 9, 2010 at 9:16 AM, Sammaye <sam.mill...@gmail.com>

> >> >>>>> wrote:
>
> >> >>>>>> Hi, thanks for your response. Yes I do not plan to use mongo single
> >> >>>>>> server at all (my intranet site currently houses 25 SQL servers).
> >> >>>>>> But
> >> >>>>>> I do need a db I can trust to always send information in and store
> >> >>>>>> it
> >> >>>>>> in a manner that means only when there is a down (server fault,
> >> >>>>>> earth
> >> >>>>>> quake etc) I or another administrator would require to do
> >> >>>>>> something.
> >> >>>>>> I've just heard from many that it's only a matter of time until
> >> >>>>>> mongo
> >> >>>>>> becomes corrupt even on a perfectly fine replication system.
>
> >> >>>>> Not sure how to respond to that. Sounds like a pretty vague
> >> >>>>> assertion.
> >> >>>>> If it's just a matter of time before a given master server will
> >> >>>>> die unexpectedly, then yes, the master could see corruption on an
> >> >>>>> unclean shutdown like that.
>
> >> >>>>>> What about reports of data not always reaching db?
>
> >> >>>>> By default, writes from the drivers are 'fire-and-forget.' This is
> >> >>>>> an
> >> >>>>> intentional design decision. However, if you need durable writes
> >> >>>>> that are
> >> >>>>> assured to reach the db, use the safe mode implemented by all
> >> >>>>> drivers.
>
> >> >>>>>> Does mongo ensure that any data I send in will reach a db and a
> >> >>>>>> collection?
>
> >> >>>>> See above.
>
> >> >>>>>> On Jul 9, 2:08 pm, Kyle Banker <k...@10gen.com> wrote:
> >> >>>>>> > We don't support traditional single-server durability; MongoDB
> >> >>>>>> > achieves
> >> >>>>>> > durability through replication and judicious use of snapshotting.
>
> >> >>>>>> > At the moment, you can run a master with any number of slaves,
> >> >>>>>> > placing one
> >> >>>>>> > in a separate data center if needed. If a master server dies or
> >> >>>>>> > has
> >> >>>>>> > an
> >> >>>>>> > unclean shutdown, you could see some corruption on that master
> >> >>>>>> > node,
> >> >>>>>> > but in
> >> >>>>>> > that situation, you fail over to a slave. With replica sets
> >> >>>>>> > (http://www.mongodb.org/display/DOCS/Replica+Sets), you have a
> >> >>>>>> > set of N
> >> >>>>>> > replicas, and the failover is handled automatically.
>
> >> >>>>>> > For ops that need the most durability, you can set a W value on
> >> >>>>>> > writes to
> >> >>>>>> > ensure that the write is replicated to at least N nodes. So
> >> >>>>>> > really,
> >> >>>>>> > the
> >> >>>>>> > level of durability is configurable. With auto-sharding, which
> >> >>>>>> > will
> >> >>>>>> > be
> >> >>>>>> > production-ready by the end of the month, you'll get automated
> >> >>>>>> > failover
> >> >>>>>> > across automatically managed sharded nodes, which may be what
> >> >>>>>> > your
> >> >>>>>> > system
> >> >>>>>> > will require.
>
> >> >>>>>> > Here's more on our
>
> >> >>>>>> > philosophy:http://blog.mongodb.org/post/381927266/what-about-durability
>
> >> >>>>>> > Yes, we look at the durability issue differently. But that
> >> >>>>>> > doesn't
> >> >>>>>> > mean that
> >> >>>>>> > we have the wrong approach. I think that the biggest testament to
> >> >>>>>> > the
> >> >>>>>> > viability of our design is the number of working production
> >> >>>>>> > MongoDB
> >> >>>>>> > deployments. You can see many of them
> >> >>>>>> > here:http://www.mongodb.org/display/DOCS/Production+Deployments
>
> >> >>>>>> > On Fri, Jul 9, 2010 at 3:34 AM, Sammaye <sam.mill...@gmail.com>
> >> >>>>>> > wrote:
> >> >>>>>> > > Hi everyone,
>
> >> >>>>>> > > I recently read this whilst using my mongodb:
>
> >> >>>>>> > > >http://www.mikealrogers.com/2010/07/mongodb-performance-durability/
>
> >> >>>>>> > > And it's got me thinking. He is basically saying that even with
> >> >>>>>> > > replication on 3 or more servers I will still get data
> >> >>>>>> > > corruption
> >> >>>>>> > > and
> >> >>>>>> > > the such.
>
> >> >>>>>> > > I was hoping I could ask you guys exactly how reliable mongo db
> >> >>>>>> > > has
> >> >>>>>> > > become.
>
> >> >>>>>> > > I am asking since I wish to use it as a backbone technology
> >> >>>>>> > > within
> >> >>>>>> > > a
> >> >>>>>> > > very large (multimillion user) intranet system for a huge
> >> >>>>>> > > global
> >> >>>>>> > > company and it's related partners, but this blog post worries
> >> >>>>>> > > me
> >> >>>>>> > > and
> >> >>>>>> > > make me think I need to use MySQL or something (which is what
> >> >>>>>> > > is
> >> >>>>>> > > currently used and is....sorry for the swear....shit).
>
> >> >>>>>> > > To get an idea of how heavy my intranet site is just think of
> >> >>>>>> > > facebook
> >> >>>>>> > > or youtube (though they're not as heavy but gives you an idea
> >> >>>>>> > > of
> >> >>>>>> > > traffic and load).
>
> >> >>>>>> > > Thanks,
>

> ...
>
> read more »

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Michael Dirolf]

A kill is fine - will send SIGTERM. Just be sure to only send INT or TERM.

[kevin]

please do not compromise on the speed or performance of mongodb.

i havent had any problem with corruption or data loss or anything like that.

On Fri, Jul 9, 2010 at 8:48 AM, Eliot Horowitz <elioth...@gmail.com> wrote:

1.8 will have durability.  This means an append log or something like that

Note this is not a bug, but a decision about what we think the best way to guarantee durability on a set of servers

Can see my blog post for more in depth notes.

On Jul 9, 2010, at 8:40 AM, Giorgio <giorgio...@gmail.com> wrote:

> Another point of that blog post is that when corruption occurs it can
> be harder to recover from it because mongodb doesn't use an append
> only log.
>
> Just to be clear. Will this be fixed in the 1.8 release?
>
> What other changes that affect single server durability are coming?
>
>
> On Jul 9, 11:30 am, Sam Millman <smill...@nhbs.co.uk> wrote:
>> Ok one more time, i posted in response to:

>>
>> For those who "CTRL-C" the process, should mongo ask you to repair the
>> DB? I run on Centos, i haven't looked into startup scripts, but I
>> heard from a friend who had to run repair on his DB. Server didn't
>> crash, he did a CTRL-C to kill the process in order to reboot the box.
>>

>> On 9 July 2010 16:22, Richard Kreuter <rich...@10gen.com> wrote:
>>
>>
>>
>>
>>
>>> Sam,
>>
>>> In case there's some doubt, that's a clean shutdown you're looking at
>>> there.
>>
>>> --
>>> Richard
>>
>>> Sam Millman <smill...@nhbs.co.uk> wrote:
>>
>>>> http://173.203.115.147/Untitled-2.pnglike so

>>
>>>> On 9 July 2010 16:13, Andreas Jung <li...@zopyx.com> wrote:
>>
>>>>     -----BEGIN PGP SIGNED MESSAGE-----
>>>>     Hash: SHA1
>>

>>>>     Kristina Chodorow wrote:
>>>>     > Ctrl-C cleanly shuts down the database.  It is one of the
>>> recommended
>>>>     > ways of shutting down the database and will not cause corruption or
>>> make
>>>>     > a repair necessary.
>>

>>>>     I evaluated MongoDB over several weeks in various scenarios and have
>>>>     never seen any corruption by using ctrl-c  - and I used it very
>>>>     frequently. Reports of data corruption with the latest 1.4.X versions

>>> of
>>>>     MongoDB caused by using ctrl-c are a myth.
>>

>>>>     - -aj

>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "mongodb-user" group.
>>> To post to this group, send email to mongod...@googlegroups.com.
>>> To unsubscribe from this group, send email to

>>> mongodb-user...@googlegroups.com<mongodb-user%2Bunsubscribe@google groups.com>

>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --

>> Bow Chicka Bow Wow

[kevin]

http://www.thebuzzmedia.com/mongodb-single-server-data-durability-guide/#help
this says doing kill is bad. can you pl. confirm it is ok to kill. i am not familiar with different types of signals.
thank you!

[Michael Dirolf]

kill pid is fine
kill -9 pid is bad

the default for kill is SIGTERM, which is okay.

[Sam Millman]

mongo does still not truely support single server durability

Bow Chicka Bow Wow

[Tim Hawkins]

If you do the same thing to a mysql instance running  at high load, you will probably get the same problem, 

Typical problems are:

Mysql complaining of "crashed" tables. Requiring the table to be rebuilt

Corruption of the .Frm files in innodb.  Usually requiring a restore from backup. 

Both are know issues with sudden termination of mysql instances. Both require a repair action to be taken. 

Most relational DB systems dont handle sudden process termination well either.  

BTW with Mysql fatal exceptions are more common

than you believe, if the mysql daemon coredumps, then the outer mysqld_safe wrapper auto restarts it.  And most folks don't

notice until their queries start failing with crashed table errors. 

[Sam Millman]

Yea interesting idea that.

I mean everyone seems to associate MySQL with Maximum durability but when you really get down to it it's just as prone to any tech to failure and more so when you try and scale it as I just found out.

On 10 July 2010 13:13, Лоик <lame...@gmail.com> wrote:

Haven't had any problem with mongodb. It runs on a dedicated server as single instance.. (yet, i'm still in development so it doesn't matter much.) Never had any problem with it.) As pointed out, hardware failure will break things even with things like mysql. One of our server went down because it wasn't plugged to an UPS. And yeah, we lost a massive amount of data.

I don't believe single server durability makes much sense. If there is a hardware failure, there is almost no way to be 100% certain that there won't be datacorruption. On the other hand, having a complete cluster to shutdown non cleanly at the same time is realllly unlikely. And even if you cannot run mongo in a cluster, just like our mysql problem, what really saved us was the daily backups we made. I think something similar may be done with mongodumps if you are really paranoid.

cronjob

mongodump [what to dump] and where with a date
gzip everything
remove files that are too old

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.