--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Hi, thanks for your response. Yes I do not plan to use mongo single
server at all (my intranet site currently houses 25 SQL servers). But
I do need a db I can trust to always send information in and store it
in a manner that means only when there is a down (server fault, earth
quake etc) I or another administrator would require to do something.
I've just heard from many that it's only a matter of time until mongo
becomes corrupt even on a perfectly fine replication system.
What about reports of data not always reaching db?
Does mongo ensure that any data I send in will reach a db and a
collection?
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
"MongoDB, by default, doesn’t actually have a response for writes. You just write your data to the socket and assume it’s going to be available when you try to read it. Under concurrent load you can’t expect to reliably store stateful data like session information like this. It’s kind of like, in your webapp, if you were to spawn a thread to do some work and at the end set some global but you return a response to the client immediately. Where there wasn’t any load you would be fine because your thread is faster than the roundtrip time to the client. But under heavy load the operations in the thread could take too long and a subsequent request wouldn’t have access to that global because it wasn’t set yet. This is why you can’t store session data reliably in MongoDB without changing the default client option to return a response, because no matter how “fast” it is if you send a response to the client (or no response at all) before the data is available for read it’s useless.
To people who don’t live in databases all day it’s hard to explain just how odd this choice is. I don’t know of another database that even allows you to return a response before the write data is accessible much less not even have a response by default. This is kind of like using UDP for data that you care about getting somewhere, it’s theoretically faster but the importance of making sure your data gets somewhere and is accessible is almost always more important."
So if I use safe mode it would be like TCP instead of UDP when I transfer data to mongo (using the same metaphore as the author).
I've just heard from many that it's only a matter of time until mongo
becomes corrupt even on a perfectly fine replication system.
You see I want to use mongo, but I want to make sure that if I set it up right it would be just as durable and reliable as say MySQL.
Yea I meant if I use safe mode it will wait for a reply from the db before it declares the record is set instead of the default write and forget.
I did get a tad of corruption on a single server setup (vmed dev server...I don't run multi in dev environment) but repairDB() took care of it nicely and I didnt loose any data which goes to contradict the blog post again.
--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
If mongo gets shut down cleanly, there should be no corruption. If
there is, then that is a bug and you would see assertions in the log.
As far as I know - there are no unfixed cases of this happening, nor
should there be.
If mongo gets shutdown uncleanly (power failure, etc...) when you try
to restart it, it will say you have to run a repair first.
What do you mean you got a tad of corruption?
Did your server get shut down uncleanly?
On Fri, Jul 9, 2010 at 10:14 AM, Sam Millman <smil...@nhbs.co.uk> wrote:
> I did get a tad of corruption on a single server setup (vmed dev server...I
> don't run multi in dev environment) but repairDB() took care of it nicely
> and I didnt loose any data which goes to contradict the blog post again.
>
How did you determine it was corruption?
Was this after an unclean shutdown?
--
sdotsen writes:
> Eliot,
>
> For those who "CTRL-C" the process, should mongo ask you to repair the
> DB? I run on Centos, i haven't looked into startup scripts, but I
> heard from a friend who had to run repair on his DB. Server didn't
> crash, he did a CTRL-C to kill the process in order to reboot the box.
>
> On Jul 9, 11:01=A0am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> > To which question?
> >
> > How did you determine it was corruption?
> > Was this after an unclean shutdown?
> >
> > On Fri, Jul 9, 2010 at 10:59 AM, Sam Millman <smill...@nhbs.co.uk> wrote:
> > > that was in my post so I'll say yea
> > > On 9 July 2010 15:57, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> >
> > >> What do you mean you got a tad of corruption?
> > >> Did your server get shut down uncleanly?
> >
> > >> On Fri, Jul 9, 2010 at 10:14 AM, Sam Millman <smill...@nhbs.co.uk> wro=
> te:
> > >> > I did get a tad of corruption on a single server setup (vmed dev
> > >> > server...I
> > >> > don't run multi in dev environment) but repairDB() took care of it
> > >> > nicely
> > >> > and I didnt loose any data which goes to contradict the blog post ag=
> ain.
> >
> > >> > On 9 July 2010 15:12, Sam Millman <smill...@nhbs.co.uk> wrote:
> >
> > >> >> "Just to be clear, we use TCP to communicate with the server, not U=
> DP."
> > >> >> Yea I meant if I use safe mode it will wait for a reply from the db
> > >> >> before
> > >> >> it declares the record is set instead of the default write and forg=
> et.
> > >> >> Sorry
> > >> >> I was trying to be clever with my words...failed massively.
> > >> >> I have been using mongo for a few things now (personal) and even on=
> one
> > >> >> server setup it hasn't given me any problems. It's only recently I
> > >> >> started
> > >> >> to question.
> > >> >> Yea mongo is vetted by a lot of people and thats why I actually loo=
> ked
> > >> >> at
> > >> >> it originally...though it looks like I would be the first to use it=
> as
> > >> >> a
> > >> >> full replacement for everything in a huge server setup...
> > >> >> I did look at the 10Gen posts and thats why I'm torn between the tw=
> o
> > >> >> views...kyle is right I'm gonna need to build something like a chat
> > >> >> system
> > >> >> and throw it onto my network and see what happens.
> >
> > >> >> On 9 July 2010 15:04, Kyle Banker <k...@10gen.com> wrote:
> >
> > >> >>> Just to be clear, we use TCP to communicate with the server, not U=
> DP.
> > >> >>> As for all the hearsay, there's certainly a lot of it . There are =
> also
> > >> >>> a
> > >> >>> lot of happy users, as Kristina points out. People say a lot of
> > >> >>> things. The
> > >> >>> only way to get the level of assurance you need is to investigate =
> the
> > >> >>> system
> > >> >>> more deeply on your own and to do some prototyping. That's usually=
> the
> > >> >>> best
> > >> >>> way to make a technology decision.
> > >> >>> On Fri, Jul 9, 2010 at 9:44 AM, Sam Millman <smill...@nhbs.co.uk>
> > >> >>> wrote:
> >
> > >> >>>> "MongoDB, by default,=A0doesn=92t actually have a response for wr=
> ites.
> > >> >>>> You
> > >> >>>> just write your data to the socket and assume it=92s going to be
> > >> >>>> available
> > >> >>>> when you try to read it. Under concurrent load you can=92t expect=
> to
> > >> >>>> reliably
> > >> >>>> store stateful data like session information like this. It=92s ki=
> nd of
> > >> >>>> like,
> > >> >>>> in your webapp, if you were to spawn a thread to do some work and=
> at
> > >> >>>> the end
> > >> >>>> set some global but you return a response to the client immediate=
> ly.
> > >> >>>> Where
> > >> >>>> there wasn=92t any load you would be fine because your thread is =
> faster
> > >> >>>> than
> > >> >>>> the roundtrip time to the client. But under heavy load the operat=
> ions
> > >> >>>> in the
> > >> >>>> thread could take too long and a subsequent request wouldn=92t ha=
> ve
> > >> >>>> access to
> > >> >>>> that global because it wasn=92t set yet. This is why you can=92t =
> store
> > >> >>>> session
> > >> >>>> data reliably in MongoDB without changing the default client opti=
> on
> > >> >>>> to
> > >> >>>> return a response, because no matter how =93fast=94 it is if you =
> send a
> > >> >>>> response
> > >> >>>> to the client (or no response at all) before the data is availabl=
> e
> > >> >>>> for read
> > >> >>>> it=92s useless.
> >
> > >> >>>> To people who don=92t live in databases all day it=92s hard to ex=
> plain
> > >> >>>> just
> > >> >>>> how odd this choice is. I don=92t know of another database that e=
> ven
> > >> >>>> allows
> > >> >>>> you to return a response before the write data is accessible much
> > >> >>>> less not
> > >> >>>> even have a response by default. This is kind of like using UDP f=
> or
> > >> >>>> data
> > >> >>>> that you care about getting somewhere, it=92s theoretically faste=
> r but
> > >> >>>> the
> > >> >>>> importance of making sure your data gets somewhere and is accessi=
> ble
> > >> >>>> is
> > >> >>>> almost always more important."
> >
> > >> >>>> So if I use safe mode it would be like TCP instead of UDP when I
> > >> >>>> transfer data to mongo (using the same metaphore as the author).
> >
> > >> >>>> "Not sure how to respond to that. Sounds like a pretty vague
> > >> >>>> assertion.
> > >> >>>> If it's just a matter of time before a given master server will
> > >> >>>> die unexpectedly, then yes, the master could see corruption on an
> > >> >>>> unclean shutdown like that."
> > >> >>>> It's just what I keep hearing. That mongo is good as front cache =
> but
> > >> >>>> for
> > >> >>>> say the backend of a full site it's not really that reliable.
> > >> >>>> You see I want to use mongo, but I want to make sure that if I se=
> t it
> > >> >>>> up
> > >> >>>> right it would be just as durable and reliable as say MySQL.
> >
> > >> >>>> On 9 July 2010 14:28, Kyle Banker <k...@10gen.com> wrote:
> >
> > >> >>>>> Answers below:
> >
> > >> >>>>> On Fri, Jul 9, 2010 at 9:16 AM, Sammaye <sam.mill...@gmail.com>
> > >> >>>>> wrote:
> >
> > >> >>>>>> Hi, thanks for your response. Yes I do not plan to use mongo si=
> ngle
> > >> >>>>>> server at all (my intranet site currently houses 25 SQL servers=
> ).
> > >> >>>>>> But
> > >> >>>>>> I do need a db I can trust to always send information in and st=
> ore
> > >> >>>>>> it
> > >> >>>>>> in a manner that means only when there is a down (server fault,
> > >> >>>>>> earth
> > >> >>>>>> quake etc) I or another administrator would require to do
> > >> >>>>>> something.
> > >> >>>>>> I've just heard from many that it's only a matter of time until
> > >> >>>>>> mongo
> > >> >>>>>> becomes corrupt even on a perfectly fine replication system.
> >
> > >> >>>>> Not sure how to respond to that. Sounds like a pretty vague
> > >> >>>>> assertion.
> > >> >>>>> If it's just a matter of time before a given master server will
> > >> >>>>> die unexpectedly, then yes, the master could see corruption on a=
> n
> > >> >>>>> unclean shutdown like that.
> >
> > >> >>>>>> What about reports of data not always reaching db?
> >
> > >> >>>>> By default, writes from the drivers are 'fire-and-forget.' This =
> is
> > >> >>>>> an
> > >> >>>>> intentional design decision. However, if you need durable writes
> > >> >>>>> that are
> > >> >>>>> assured to reach the db, use the safe mode implemented by all
> > >> >>>>> drivers.
> >
> > >> >>>>>> Does mongo ensure that any data I send in will reach a db and a
> > >> >>>>>> collection?
> >
> > >> >>>>> See above.
> >
> > >> >>>>>> On Jul 9, 2:08=A0pm, Kyle Banker <k...@10gen.com> wrote:
> > >> >>>>>> > We don't support traditional single-server durability; MongoD=
> B
> > >> >>>>>> > achieves
> > >> >>>>>> > durability through replication and judicious use of snapshott=
> ing.
> >
> > >> >>>>>> > At the moment, you can run a master with any number of slaves=
> ,
> > >> >>>>>> > placing one
> > >> >>>>>> > in a separate data center if needed. If a master server dies =
> or
> > >> >>>>>> > has
> > >> >>>>>> > an
> > >> >>>>>> > unclean shutdown, you could see some corruption on that maste=
> r
> > >> >>>>>> > node,
> > >> >>>>>> > but in
> > >> >>>>>> > that situation, you fail over to a slave. With replica sets
> > >> >>>>>> > (http://www.mongodb.org/display/DOCS/Replica+Sets), you have =
> a
> > >> >>>>>> > set of N
> > >> >>>>>> > replicas, and the failover is handled automatically.
> >
> > >> >>>>>> > For ops that need the most durability, you can set a W value =
> on
> > >> >>>>>> > writes to
> > >> >>>>>> > ensure that the write is replicated to at least N nodes. So
> > >> >>>>>> > really,
> > >> >>>>>> > the
> > >> >>>>>> > level of durability is configurable. With auto-sharding, whic=
> h
> > >> >>>>>> > will
> > >> >>>>>> > be
> > >> >>>>>> > production-ready by the end of the month, you'll get automate=
> d
> > >> >>>>>> > failover
> > >> >>>>>> > across automatically managed sharded nodes, which may be what
> > >> >>>>>> > your
> > >> >>>>>> > system
> > >> >>>>>> > will require.
> >
> > >> >>>>>> > Here's more on our
> >
> > >> >>>>>> > philosophy:http://blog.mongodb.org/post/381927266/what-about-=
> durability
> >
> > >> >>>>>> > Yes, we look at the durability issue differently. But that
> > >> >>>>>> > doesn't
> > >> >>>>>> > mean that
> > >> >>>>>> > we have the wrong approach. I think that the biggest testamen=
> t to
> > >> >>>>>> > the
> > >> >>>>>> > viability of our design is the number of working production
> > >> >>>>>> > MongoDB
> > >> >>>>>> > deployments. You can see many of them
> > >> >>>>>> > here:http://www.mongodb.org/display/DOCS/Production+Deploymen=
> ts
> >
> > >> >>>>>> > On Fri, Jul 9, 2010 at 3:34 AM, Sammaye <sam.mill...@gmail.co=
> m>
> > >> >>>>>> > wrote:
> > >> >>>>>> > > Hi everyone,
> >
> > >> >>>>>> > > I recently read this whilst using my mongodb:
> >
> > >> >>>>>> > > >http://www.mikealrogers.com/2010/07/mongodb-performance-du=
> rability/
> >
> > >> >>>>>> > > And it's got me thinking. He is basically saying that even =
> with
> > >> >>>>>> > > replication on 3 or more servers I will still get data
> > >> >>>>>> > > corruption
> > >> >>>>>> > > and
> > >> >>>>>> > > the such.
> >
> > >> >>>>>> > > I was hoping I could ask you guys exactly how reliable mong=
> o db
> > >> >>>>>> > > has
> > >> >>>>>> > > become.
> >
> > >> >>>>>> > > I am asking since I wish to use it as a backbone technology
> > >> >>>>>> > > within
> > >> >>>>>> > > a
> > >> >>>>>> > > very large (multimillion user) intranet system for a huge
> > >> >>>>>> > > global
> > >> >>>>>> > > company and it's related partners, but this blog post worri=
> es
> > >> >>>>>> > > me
> > >> >>>>>> > > and
> > >> >>>>>> > > make me think I need to use MySQL or something (which is wh=
> at
> > >> >>>>>> > > is
> > >> >>>>>> > > currently used and is....sorry for the swear....shit).
> >
> > >> >>>>>> > > To get an idea of how heavy my intranet site is just think =
> of
> > >> >>>>>> > > facebook
> > >> >>>>>> > > or youtube (though they're not as heavy but gives you an id=
> ea
> > >> >>>>>> > > of
> > >> >>>>>> > > traffic and load).
> >
> > >> >>>>>> > > Thanks,
> >
> > ...
> >
> > read more =BB
>
> --=20
> You received this message because you are subscribed to the Google Groups "=
> mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user+unsubscribe@goog=
> legroups.com.
> For more options, visit this group at http://groups.google.com/group/mongod=
> b-user?hl=3Den.
>
Kristina Chodorow wrote:
> Ctrl-C cleanly shuts down the database. It is one of the recommended
> ways of shutting down the database and will not cause corruption or make
> a repair necessary.
>
I evaluated MongoDB over several weeks in various scenarios and have
never seen any corruption by using ctrl-c - and I used it very
frequently. Reports of data corruption with the latest 1.4.X versions of
MongoDB caused by using ctrl-c are a myth.
- -aj
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkw3PKYACgkQCJIWIbr9KYxFhwCfQWGxF6v4Q225hB00PonCBetG
aVcAnRZ8c5cCVEQrpewVnNpub3f/5YBs
=45et
-----END PGP SIGNATURE-----
--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
This tells us what? (except showing a clean shutdown)?
- -aj
Sam Millman wrote:
> http://173.203.115.147/Untitled-2.png like so
>
> On 9 July 2010 16:13, Andreas Jung <li...@zopyx.com
> <mailto:li...@zopyx.com>> wrote:
>
> Kristina Chodorow wrote:
>> Ctrl-C cleanly shuts down the database. It is one of the recommended
>> ways of shutting down the database and will not cause corruption
> or make
>> a repair necessary.
>
>
> I evaluated MongoDB over several weeks in various scenarios and have
> never seen any corruption by using ctrl-c - and I used it very
> frequently. Reports of data corruption with the latest 1.4.X versions of
> MongoDB caused by using ctrl-c are a myth.
>
> -aj
- --
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
<mailto:mongod...@googlegroups.com>.
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
<mailto:mongodb-user%2Bunsu...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/mongodb-user?hl=en.
> --
> Bow Chicka Bow Wow
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
- --
ZOPYX Limited | zopyx group
Charlottenstr. 37/1 | The full-service network for Zope & Plone
D-72070 T�bingen | Produce & Publish
www.zopyx.com | www.produce-and-publish.com
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkw3Pk4ACgkQCJIWIbr9KYyO2wCfZeFSgQuppNL9ov8T5OVBa0E8
Nc4AoNf3AH9VznAQyqMOGz2w2rAspsTd
=WY/n
-----END PGP SIGNATURE-----
In case there's some doubt, that's a clean shutdown you're looking at
there.
--
Richard
This is absolutely not the same as corruption, this is mongo saying it
can't 100% verify what's going on, and to be paranoid, run a repair.
Terminology is very important these days...
my screenshot shows the crtl-c command being initiated and what should happen
--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Note this is not a bug, but a decision about what we think the best way to guarantee durability on a set of servers
Can see my blog post for more in depth notes.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> After a unclean shutdown, mongo will say it was not shutdown cleanly,
> and ask you to do a repair.
>
> This is absolutely not the same as corruption, this is mongo saying it
> can't 100% verify what's going on, and to be paranoid, run a repair.
This is a very very very good point.
--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
On 07/09/2010 07:58 AM, Eliot Horowitz wrote:
> If mongo gets shutdown uncleanly (power failure, etc...) when you try
> to restart it, it will say you have to run a repair first.
Using your Ubuntu build it only says this once. ie if you try to start it a
second time then it will start just fine. The "saying" goes to the log file.
I did lose database contents on a virtual machine the other day as the user
didn't cleanly shut it down some of the time, and ran 'sudo start mongodb'
if it didn't appear to be running. Do that a few times, and suddenly all
the data disappears (well everything except for 4 out of 1.6 million
documents :-) This was a development environment, single unreplicated
server etc so I'm not bothered.
Or put in the form of a ticket:
http://jira.mongodb.org/browse/SERVER-1386
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkw3cN4ACgkQmOOfHg372QRc0ACfUh2NguqmdXkxLBGXpN77ilbE
dlQAoJnpbKaev4ij2NskQQki9peuFsMr
=UKhv
-----END PGP SIGNATURE-----
On 07/09/2010 10:43 AM, Kenny Gorman wrote:
> PostgreSQL is highly durable. Yet, I spent 2 weeks day and night fixing
> (really deleting) corrupt data when a hardware bug caused corruption to
> our data and PostgreSQL didn't report anything and kept going. I mean
> *lots* of data. It was horrible.
I assume that Postgres does not have a per record checksum of some sort so
there is effectively no possibility to detect externally corrupted data.
Does Mongo have any form of record checksum?
CouchDB did gain it a while back, mainly to help replication, but it does at
least mean there is a way of verifying contents just as with modern version
control systems like git and Mercurial.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkw3cjQACgkQmOOfHg372QTuZQCg4dp0y+3Ljm7yfpdUzEfWZy+o
i8IAoJNfw+Ye9eDkIuGaJSX999F+4b0N
=2YI+
-----END PGP SIGNATURE-----
If mongod needs to be repaired it won't start - it will print a
message and error out when you try to start it. The correct response
when that happens is to not just try restarting again w/o checking the
logs and seeing why it didn't start.
On 07/09/2010 12:16 PM, chriskessel wrote:
> You weren't bothered? This would be a huge deal breaker for our
> deployment. Frankly, I'm stunned you weren't bothered by losing your
> entire database. Single server or not, losing the entire database and
> not being aware of it until it's long gone is completely out of the
> question for a production environment (at least for me).
It was a development environment not production. Repair was never run. It
was single server. The virtual machine was stopped a few times either by
the host being yanked (a laptop) or the virtual machine being stopped
(equivalent of a power pull). If you were trying to corrupt/lose data then
this is pretty much the recipe to follow. Why would I be bothered on
succeeding?
SQLite does include a nice section on how to corrupt your database. Perhaps
there should be something in a similar spirit on the Mongo site?
http://www.sqlite.org/lockingv3.html#how_to_corrupt
> We use virtualized machines and typically the operations folks start
> and stop everything via sudo commands. Which, if I'm reading it right,
> is exactly the situation you're describing.
Nope. Things were not stopped via commands (which would be clean) but
rather by abruptly terminating the virtual machine (equivalent of pulling
the power cable) or terminating the host (a laptop). As I said, this was a
development environment which is why things like that were done.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkw3d/gACgkQmOOfHg372QTlKACfZxHp/0o5FRi5QSjGVCQqfocP
NUMAnjAFfXMk+VrCv5AKX+T4RZJ2ATHW
=VO6f
-----END PGP SIGNATURE-----
Automatic repair is probably not a great idea, but failing to start no
matter how many times you try is. This behavior should be fixed soon.
You're describing a number of issues here, but the fact that starting
mongod twice broke the check for unclean shutdown is a bug, which is now
fixed. Thanks for pointing it out!
--
Richard
Roger Binns <rog...@rogerbinns.com> wrote:
> On 07/09/2010 07:58 AM, Eliot Horowitz wrote:
> > If mongo gets shutdown uncleanly (power failure, etc...) when you try
> > to restart it, it will say you have to run a repair first.
>
> Using your Ubuntu build it only says this once. ie if you try to start it a
> second time then it will start just fine. The "saying" goes to the log file.
>
> I did lose database contents on a virtual machine the other day as the user
> didn't cleanly shut it down some of the time, and ran 'sudo start mongodb'
> if it didn't appear to be running. Do that a few times, and suddenly all
> the data disappears (well everything except for 4 out of 1.6 million
> documents :-) This was a development environment, single unreplicated
> server etc so I'm not bothered.
>
> Or put in the form of a ticket:
>
> http://jira.mongodb.org/browse/SERVER-1386
>
> Roger
>
-J
Chris,
you (and I) really have no information about this at all. Some file some where on some VM maybe got lost when things were yanked, pulled, etc. Is it reproducible? No one knows. No one is even sure what happened!
I wouldn't get worked up into a lather about this. It's entirely possible the person using that VM typed "del mongod.exe" or something similar. We. Just. Don't. Know.
I've been using mongod since version 1.2 and have pumped hundreds of gigs through most of the intervening versions. I haven't lost a single byte and I've had plenty of hardware failures in the interim.
See that? My anecdotal data trumps the earlier poster's. :)
If this were a problem, there would be tickets opened up for it. Check the archives for all the people asking about running mongo on EC2 and other virtualized platforms. If there were problems, they would be OBVIOUS.
cr
On 07/09/2010 01:17 PM, chriskessel wrote:
> I still am stunned the escalating data corruption s acceptable even in
> that situation.
There wasn't escalating data corruption. It was all there and then wasn't.
The data is almost entirely read-only used for our application, and is
generated from an external data source, with lots of copies all over the place.
The virtual machine was also configured to use the host I/O cache (this is a
virtualbox settting) so even if the guest OS called fsync it wouldn't
actually be on real disk platters. Memory settings for the guest (RAM and
swap) were also quite small so an OOM killer may also have been called.
As I said earlier, it was pretty much following the checklist on how to
corrupt your database with MongoDB. It was a development environment on a
crappy Windows laptop (guest Ubuntu). The user couldn't care less about
MongoDB or any of the other components - they just wanted to try some
features of my code.
Hopefully it will now be more obvious why the server has failed to start and
that doing a repair is easy to describe over the phone to a Windows user.
http://jira.mongodb.org/browse/SERVER-1386
It should also be very clear that the way you do things in a throwaway
development environment is *very* different than what you would do in
production.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkw3h4IACgkQmOOfHg372QT/GQCeOFbQuO5voAXttj229IqSYKw3
rNQAoIlsdC3hXXcvlvRHGhn1oFzI8VAY
=NpYN
-----END PGP SIGNATURE-----
On 07/09/2010 01:24 PM, Jason J. W. Williams wrote:
> An append-only log will be a huge improvement.
If done well. It was what drove me away from CouchDB where the entire data
storage format is an append only log. I was getting massive file sizes (eg
23GB to store 2GB of data, 43GB of "index" data). Dealing with that much
I/O plus the data being scattered amongst haystacks that size lead to very
poor performance.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkw3iOIACgkQmOOfHg372QTGJACcDMiOSLcUBz4fp2U1m7KtOh7V
wqcAn06c88cWYs0NZYFUebfCUBCzSxDP
=/VSk
-----END PGP SIGNATURE-----
Ctrl-C cleanly shuts down the database. It is one of the recommended ways of shutting down the database and will not cause corruption or make a repair necessary.
On Fri, Jul 9, 2010 at 11:07 AM, sdotsen <samna...@gmail.com> wrote:
Eliot,
For those who "CTRL-C" the process, should mongo ask you to repair the
DB? I run on Centos, i haven't looked into startup scripts, but I
heard from a friend who had to run repair on his DB. Server didn't
crash, he did a CTRL-C to kill the process in order to reboot the box.
On Jul 9, 11:01 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> To which question?
>
> How did you determine it was corruption?
> Was this after an unclean shutdown?
>
> On Fri, Jul 9, 2010 at 10:59 AM, Sam Millman <smill...@nhbs.co.uk> wrote:
> > that was in my post so I'll say yea
> > On 9 July 2010 15:57, Eliot Horowitz <eliothorow...@gmail.com> wrote:
>
> >> What do you mean you got a tad of corruption?
> >> Did your server get shut down uncleanly?
>
> >> On Fri, Jul 9, 2010 at 10:14 AM, Sam Millman <smill...@nhbs.co.uk> wrote:
> >> > I did get a tad of corruption on a single server setup (vmed dev
> >> > server...I
> >> > don't run multi in dev environment) but repairDB() took care of it
> >> > nicely
> >> > and I didnt loose any data which goes to contradict the blog post again.
>
> >> > On 9 July 2010 15:12, Sam Millman <smill...@nhbs.co.uk> wrote:
>
> >> >> "Just to be clear, we use TCP to communicate with the server, not UDP."
> >> >> Yea I meant if I use safe mode it will wait for a reply from the db
> >> >> before
> >> >> it declares the record is set instead of the default write and forget.
> >> >> Sorry
> >> >> I was trying to be clever with my words...failed massively.
> >> >> I have been using mongo for a few things now (personal) and even on one
> >> >> server setup it hasn't given me any problems. It's only recently I
> >> >> started
> >> >> to question.
> >> >> Yea mongo is vetted by a lot of people and thats why I actually looked
> >> >> at
> >> >> it originally...though it looks like I would be the first to use it as
> >> >> a
> >> >> full replacement for everything in a huge server setup...
> >> >> I did look at the 10Gen posts and thats why I'm torn between the two
> >> >> views...kyle is right I'm gonna need to build something like a chat
> >> >> system
> >> >> and throw it onto my network and see what happens.
>
> >> >> On 9 July 2010 15:04, Kyle Banker <k...@10gen.com> wrote:> >> >>> On Fri, Jul 9, 2010 at 9:44 AM, Sam Millman <smill...@nhbs.co.uk>
>
> >> >>> Just to be clear, we use TCP to communicate with the server, not UDP.
> >> >>> As for all the hearsay, there's certainly a lot of it . There are also
> >> >>> a
> >> >>> lot of happy users, as Kristina points out. People say a lot of
> >> >>> things. The
> >> >>> only way to get the level of assurance you need is to investigate the
> >> >>> system
> >> >>> more deeply on your own and to do some prototyping. That's usually the
> >> >>> best
> >> >>> way to make a technology decision.
> >> >>>>> On Fri, Jul 9, 2010 at 9:16 AM, Sammaye <sam.mill...@gmail.com>> >> >>> wrote:
>
> >> >>>> "MongoDB, by default, doesn’t actually have a response for writes.
> >> >>>> You
> >> >>>> just write your data to the socket and assume it’s going to be
> >> >>>> available
> >> >>>> when you try to read it. Under concurrent load you can’t expect to
> >> >>>> reliably
> >> >>>> store stateful data like session information like this. It’s kind of
> >> >>>> like,
> >> >>>> in your webapp, if you were to spawn a thread to do some work and at
> >> >>>> the end
> >> >>>> set some global but you return a response to the client immediately.
> >> >>>> Where
> >> >>>> there wasn’t any load you would be fine because your thread is faster
> >> >>>> than
> >> >>>> the roundtrip time to the client. But under heavy load the operations
> >> >>>> in the
> >> >>>> thread could take too long and a subsequent request wouldn’t have
> >> >>>> access to
> >> >>>> that global because it wasn’t set yet. This is why you can’t store
> >> >>>> session
> >> >>>> data reliably in MongoDB without changing the default client option
> >> >>>> to
> >> >>>> return a response, because no matter how “fast” it is if you send a
> >> >>>> response
> >> >>>> to the client (or no response at all) before the data is available
> >> >>>> for read
> >> >>>> it’s useless.
>
> >> >>>> To people who don’t live in databases all day it’s hard to explain
> >> >>>> just
> >> >>>> how odd this choice is. I don’t know of another database that even
> >> >>>> allows
> >> >>>> you to return a response before the write data is accessible much
> >> >>>> less not
> >> >>>> even have a response by default. This is kind of like using UDP for
> >> >>>> data
> >> >>>> that you care about getting somewhere, it’s theoretically faster but
> >> >>>> the
> >> >>>> importance of making sure your data gets somewhere and is accessible
> >> >>>> is
> >> >>>> almost always more important."
>
> >> >>>> So if I use safe mode it would be like TCP instead of UDP when I
> >> >>>> transfer data to mongo (using the same metaphore as the author).
>
> >> >>>> "Not sure how to respond to that. Sounds like a pretty vague
> >> >>>> assertion.
> >> >>>> If it's just a matter of time before a given master server will
> >> >>>> die unexpectedly, then yes, the master could see corruption on an
> >> >>>> unclean shutdown like that."
> >> >>>> It's just what I keep hearing. That mongo is good as front cache but
> >> >>>> for
> >> >>>> say the backend of a full site it's not really that reliable.
> >> >>>> You see I want to use mongo, but I want to make sure that if I set it
> >> >>>> up
> >> >>>> right it would be just as durable and reliable as say MySQL.
>
> ...> >> >>>>> wrote:
>
> >> >>>>>> Hi, thanks for your response. Yes I do not plan to use mongo single
> >> >>>>>> server at all (my intranet site currently houses 25 SQL servers).
> >> >>>>>> But
> >> >>>>>> I do need a db I can trust to always send information in and store
> >> >>>>>> it
> >> >>>>>> in a manner that means only when there is a down (server fault,
> >> >>>>>> earth
> >> >>>>>> quake etc) I or another administrator would require to do
> >> >>>>>> something.
> >> >>>>>> I've just heard from many that it's only a matter of time until
> >> >>>>>> mongo
> >> >>>>>> becomes corrupt even on a perfectly fine replication system.
>
> >> >>>>> Not sure how to respond to that. Sounds like a pretty vague
> >> >>>>> assertion.
> >> >>>>> If it's just a matter of time before a given master server will
> >> >>>>> die unexpectedly, then yes, the master could see corruption on an
> >> >>>>> unclean shutdown like that.
>
> >> >>>>>> What about reports of data not always reaching db?
>
> >> >>>>> By default, writes from the drivers are 'fire-and-forget.' This is
> >> >>>>> an
> >> >>>>> intentional design decision. However, if you need durable writes
> >> >>>>> that are
> >> >>>>> assured to reach the db, use the safe mode implemented by all
> >> >>>>> drivers.
>
> >> >>>>>> Does mongo ensure that any data I send in will reach a db and a
> >> >>>>>> collection?
>
> >> >>>>> See above.
>
> >> >>>>>> On Jul 9, 2:08 pm, Kyle Banker <k...@10gen.com> wrote:
> >> >>>>>> > We don't support traditional single-server durability; MongoDB
> >> >>>>>> > achieves
> >> >>>>>> > durability through replication and judicious use of snapshotting.
>
> >> >>>>>> > At the moment, you can run a master with any number of slaves,
> >> >>>>>> > placing one
> >> >>>>>> > in a separate data center if needed. If a master server dies or
> >> >>>>>> > has
> >> >>>>>> > an
> >> >>>>>> > unclean shutdown, you could see some corruption on that master
> >> >>>>>> > node,
> >> >>>>>> > but in
> >> >>>>>> > that situation, you fail over to a slave. With replica sets
> >> >>>>>> > (http://www.mongodb.org/display/DOCS/Replica+Sets), you have a
> >> >>>>>> > set of N
> >> >>>>>> > replicas, and the failover is handled automatically.
>
> >> >>>>>> > For ops that need the most durability, you can set a W value on
> >> >>>>>> > writes to
> >> >>>>>> > ensure that the write is replicated to at least N nodes. So
> >> >>>>>> > really,
> >> >>>>>> > the
> >> >>>>>> > level of durability is configurable. With auto-sharding, which
> >> >>>>>> > will
> >> >>>>>> > be
> >> >>>>>> > production-ready by the end of the month, you'll get automated
> >> >>>>>> > failover
> >> >>>>>> > across automatically managed sharded nodes, which may be what
> >> >>>>>> > your
> >> >>>>>> > system
> >> >>>>>> > will require.
>
> >> >>>>>> > Here's more on our
>
> >> >>>>>> > philosophy:http://blog.mongodb.org/post/381927266/what-about-durability
>
> >> >>>>>> > Yes, we look at the durability issue differently. But that
> >> >>>>>> > doesn't
> >> >>>>>> > mean that
> >> >>>>>> > we have the wrong approach. I think that the biggest testament to
> >> >>>>>> > the
> >> >>>>>> > viability of our design is the number of working production
> >> >>>>>> > MongoDB
> >> >>>>>> > deployments. You can see many of them
> >> >>>>>> > here:http://www.mongodb.org/display/DOCS/Production+Deployments
>
> >> >>>>>> > On Fri, Jul 9, 2010 at 3:34 AM, Sammaye <sam.mill...@gmail.com>
> >> >>>>>> > wrote:
> >> >>>>>> > > Hi everyone,
>
> >> >>>>>> > > I recently read this whilst using my mongodb:
>
> >> >>>>>> > > >http://www.mikealrogers.com/2010/07/mongodb-performance-durability/
>
> >> >>>>>> > > And it's got me thinking. He is basically saying that even with
> >> >>>>>> > > replication on 3 or more servers I will still get data
> >> >>>>>> > > corruption
> >> >>>>>> > > and
> >> >>>>>> > > the such.
>
> >> >>>>>> > > I was hoping I could ask you guys exactly how reliable mongo db
> >> >>>>>> > > has
> >> >>>>>> > > become.
>
> >> >>>>>> > > I am asking since I wish to use it as a backbone technology
> >> >>>>>> > > within
> >> >>>>>> > > a
> >> >>>>>> > > very large (multimillion user) intranet system for a huge
> >> >>>>>> > > global
> >> >>>>>> > > company and it's related partners, but this blog post worries
> >> >>>>>> > > me
> >> >>>>>> > > and
> >> >>>>>> > > make me think I need to use MySQL or something (which is what
> >> >>>>>> > > is
> >> >>>>>> > > currently used and is....sorry for the swear....shit).
>
> >> >>>>>> > > To get an idea of how heavy my intranet site is just think of
> >> >>>>>> > > or youtube (though they're not as heavy but gives you an idea
> >> >>>>>> > > of
> >> >>>>>> > > traffic and load).
>
> >> >>>>>> > > Thanks,
>
>
> read more »
--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
1.8 will have durability. This means an append log or something like that
Note this is not a bug, but a decision about what we think the best way to guarantee durability on a set of servers
Can see my blog post for more in depth notes.
On Jul 9, 2010, at 8:40 AM, Giorgio <giorgio...@gmail.com> wrote:
> Another point of that blog post is that when corruption occurs it can
> be harder to recover from it because mongodb doesn't use an append
> only log.
>
> Just to be clear. Will this be fixed in the 1.8 release?
>
> What other changes that affect single server durability are coming?
>
>
> On Jul 9, 11:30 am, Sam Millman <smill...@nhbs.co.uk> wrote:
>> Ok one more time, i posted in response to:
>>
>> For those who "CTRL-C" the process, should mongo ask you to repair the
>> DB? I run on Centos, i haven't looked into startup scripts, but I
>> heard from a friend who had to run repair on his DB. Server didn't
>> crash, he did a CTRL-C to kill the process in order to reboot the box.
>>
>> On 9 July 2010 16:22, Richard Kreuter <rich...@10gen.com> wrote:
>>
>>
>>
>>
>>
>>> Sam,
>>
>>> In case there's some doubt, that's a clean shutdown you're looking at
>>> there.
>>
>>> --
>>> Richard
>>
>>> Sam Millman <smill...@nhbs.co.uk> wrote:
>>
>>>> http://173.203.115.147/Untitled-2.pnglike so
>>
>>>> On 9 July 2010 16:13, Andreas Jung <li...@zopyx.com> wrote:
>>
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>
>>>> Kristina Chodorow wrote:
>>>> > Ctrl-C cleanly shuts down the database. It is one of the
>>> recommended
>>>> > ways of shutting down the database and will not cause corruption or
>>> make
>>>> > a repair necessary.
>>
>>>> I evaluated MongoDB over several weeks in various scenarios and have
>>>> never seen any corruption by using ctrl-c - and I used it very
>>>> frequently. Reports of data corruption with the latest 1.4.X versions
>>> of
>>>> MongoDB caused by using ctrl-c are a myth.
>>
>>>> - -aj
>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "mongodb-user" group.
>>> To post to this group, send email to mongod...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> mongodb-user...@googlegroups.com<mongodb-user%2Bunsubscribe@google groups.com>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> Bow Chicka Bow Wow
the default for kill is SIGTERM, which is okay.
Haven't had any problem with mongodb. It runs on a dedicated server as single instance.. (yet, i'm still in development so it doesn't matter much.) Never had any problem with it.) As pointed out, hardware failure will break things even with things like mysql. One of our server went down because it wasn't plugged to an UPS. And yeah, we lost a massive amount of data.
I don't believe single server durability makes much sense. If there is a hardware failure, there is almost no way to be 100% certain that there won't be datacorruption. On the other hand, having a complete cluster to shutdown non cleanly at the same time is realllly unlikely. And even if you cannot run mongo in a cluster, just like our mysql problem, what really saved us was the daily backups we made. I think something similar may be done with mongodumps if you are really paranoid.
cronjob
mongodump [what to dump] and where with a date
gzip everything
remove files that are too old
--You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.