disabling index creation per environment

[Karl Baum]

Our mongo instance is over 400 gigs and we have to be very careful
about creating indexes. It's nice to see our indexes declared within
our mongoid model, but frequently we run into problems within
production when our mongoid app starts up and automatically creates an
index which locks up our mongodb instance. It would be great if
instead of actually creating the index within production, mongoid
could just output a warning message to the console that an index is
missing. Then we could create it ourselves in the background at a
convenient time. Is this possible? Maybe with some monkey patching?

thx.

-karl

[Cyril Mougel]

And maybe add an option to define if we want or not creating index in
background ? Can be a good idea in your case or not ?

--
Cyril Mougel
http://blog.shingara.fr

[Karl Baum]

Building indexes in the background would definitely help as well.

[Kyle Banker]

I'm going to second Karl's request. Based on a number of incidents I've noticed in the wild, allowing people to define indexes within the model is a recipe for disaster. Index building is serious business; it should be thought of as a kind of database migration.

I think we should work toward a convention where indexes are defined in a special file outside of the model. Index creation can then be triggered manually by the developer.

In the meantime, I'd recommend just as a precaution that anyone with even a moderately-sized production deployment cease to define indexes on the model layer and to move to a more explicit, migration-like index creation practice.

[Adam Fields]

On Thu, Jul 01, 2010 at 08:57:26PM -0400, Karl Baum wrote:
> Building indexes in the background would definitely help as well.

But you can only build one index in the background per collection at a
time, and the mongoid one may not be the one you want (and if another
one is building, the mongoid one will still block). I'd much prefer to
have manual control over this.

--
- Adam
----------
If you liked this email, you might also like:
"Some iPad apps I like"
-- http://workstuff.tumblr.com/post/680301206
"In praise of the Sous Vide Supreme"
-- http://www.aquick.org/blog/2010/05/17/in-praise-of-the-sous-vide-supreme/
"Just Married!"
-- http://www.flickr.com/photos/fields/4721009476/
"fields: @VLAIC Some of those are gooseberries."
-- http://twitter.com/fields/statuses/17522175900
----------
** I design intricate-yet-elegant processes for user and machine problems.
** Custom development project broken? Contact me, I can help.
** Some of what I do: http://workstuff.tumblr.com/post/70505118/aboutworkstuff

[ http://www.adamfields.com/resume.html ].. Experience
[ http://www.morningside-analytics.com ] .. Latest Venture
[ http://www.confabb.com ] ................ Founder

[Kyle Banker]

Agreed. Background indexing doesn't solves.

[Jacques Crocker]

Agreed. I've created a ticket for this: http://github.com/durran/mongoid/issues/issue/168

Will try to get this fixed soon

[Kyle Banker]

Thanks, Jacques. I'd vote for deprecation rather than a configuration option. However, if it must be done via configuration, I think it should default to not creating the indexes automatically.  It seems like that'd be the only way to avoid the problems we've been seeing.

[Adam Fields]

On Thu, Jul 01, 2010 at 09:30:34PM -0400, Kyle Banker wrote:
> I'm going to second Karl's request. Based on a number of incidents I've
> noticed in the wild, allowing people to define indexes within the model is a
> recipe for disaster. Index building is serious business; it should be
> thought of as a kind of database migration.

[...]

Having the command in mongo be "ensure index" instead of "create
index" doesn't help with this perception. With the semantics of not
erroring out if the index already exists, you're encouraging people to
run this at init time to make sure the indexes are covered instead of
having a separate index creation process.

If I didn't know any better, I'd assume from the docs
http://www.mongodb.org/display/DOCS/Indexes that building an index
isn't going to hurt anything. It's only once you get to the
http://www.mongodb.org/display/DOCS/Indexing+Advice+and+FAQ page that
there's any discussion of the problems this might cause (and even
there, it doesn't even come out and say that index building blocks
queries).

Is there any reason why multiple indexes can't be built in parallel in
the background for the same collection? If we could do that, I think
it would make sense to just get rid of foreground index building
entirely and make all of the indexes builds backgrounded.