Re: [mongodb-user] Adaptive indexing

[Sam Millman]

Depends on your definition of adaptive indexing. Can you explain some more?

On 7 October 2012 04:44, deepthi shekar <deep...@gmail.com> wrote:

Can adaptive indexing help to improve the performance of inserts and updates in MongoDB? 

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

[deepthi shekar]

since adaptive indexes work well with big data, do you think it is a good idea to have it in a nosql db like mongo? Im planning to work on this as my masters project and need some inputs and comments from readers. 

I plan to use a genetic algorithm to create the index as a side effect of the query execution. I also looked into other methods of adaptive indexing like database cracking and adaptive merging. Which one would be better ?

[Sam Millman]

Can you provide an example of what you mean cos my idea of storing genetic material would not need "adaptive indexing" (still don't know exactly what you mean by it)?

It would be good to outline your problem fully and what kind of structure you are looking to store and how you imagine it to be stored.

--

[Andre de Frere]

Hi Deepthi,

To the best of my knowledge, there is no current plan to introduce adaptive or auto tuning indexing to MongoDB.

I would be very interested in any discussion on the subject that might come out of this post.

My reading on the subject has mostly been around column stores, rather than document stores.  I've not yet seen adaptive merging or database cracking done on a document basis.  I think a hurdle here might be the lack of strict schema - since database cracking would attempt to 'crack' the data structures on a per query basis, while there is no guarantee that the data structure will be consistent per document.  I have yet to do any experiment on this theory however.

From what I have read, adaptive merging tends to cost more but take fewer queries to converge on an optimal index, while database cracking takes longer to converge but is cheaper.  Initial queries in both schemes are as expensive as full table scans (for less or more queries depending on the scheme), you would need to measure this against the time it takes to generate a full index.

Using a genetic algorithm to create an index as as side effect of query execution sounds interesting.  Again I would be concerned that the document structure could change on a per document basis, which could lead to a difference in allele values and make comparisons harder between generations, but you could overcome that hurdle by inserting only documents with the same structure - or you may find that this is not a concern after testing.

Hope this helps,

André