Possible to disable dynamic index creation?

[Ken Dale]

Is it possible to disable dynamic index creation and throw an exception if a dynamic index must be created to fulfill a query? Is there some way to accomplish this with the configuration settings here: http://ravendb.net/docs/server/administration/configuration ?

[Matt Warren]

I don't know about a global option but if you explicitly specify an index name it will not use/create a dynamic index

session.Query<T>("index_name")

[Oren Eini (Ayende Rahien)]

Currently,no.

Can you explain why you want to do this?

[Chris Marisic]

Total control over indexes, i never want an index randomly created in production. I'd rather it fail, than compile a whole index.

[Oren Eini (Ayende Rahien)]

Can you create an issue for this?

It is a more complex feature than you think, it requires us to have hooks into query optimizer so that we can report errors.

[Chris Marisic]

http://issues.hibernatingrhinos.com/issue/RavenDB-416

[Kijana Woodard]

Wouldn't/shouldn't this be caught in QA? Run through all automated and manual tests. Dynamic index count should be zero.

If I didn't catch in QA, i'd rather get a warning, if anything, rather than blow up production. My wager would be that- if it was missed by QA, the dynamic index would be in something with lower impact which wouldn't cause too many problems.

[Oren Eini (Ayende Rahien)]

Kijana,

There are scenarios where you want to fail rather than create a new index.

I can't think of one off hand, but I am sure there are :-)

[Kijana Woodard]

:-)

Perhaps. But turning on a flag that means, "shut down production if this happens anywhere in the app" would give me more than a little pause. Lol.

[Kijana Woodard]

Hmmm. I suppose it would just affect the particular query/business function without the index, but that's a pretty nasty way to find out your QA process has holes. And you'd better have a pretty fast launch process to get the fix out there. Angry execs. Haha.

[Itamar Syn-Hershko]

In those cases, I'd suggest going the route Matt suggested - make queries to named indexes, never rely on the query optimizer

[Chris Marisic]

Unfortunately this statement does not seem to be sufficient Itamar. We analyzed our code base and do not have a single usage of a query call for a non-specified index. We have temp indexes that are created even when static ones include all of the fields that the temp indexes are created for. We have been able to produce this 100% consistently in our code base for indexes that have transform results, however we have not been able to reproduce this occurrence outside of our solution to submit a failing test. So far the only thing we've been able to come up with is to set Raven/TempIndexPromotionMinimumQueryCount to 1 so atleast the indexes don't disappear.

[Matt Warren]

That sound like a bug, my understanding was that the query optimiser is only used if you don't explicitly specify an index.

[Itamar Syn-Hershko]

Does this involve range queries? we had a bug that could have caused this that we fixed

What build are you using?

[Chris Marisic]

No range queries, stable 960, the indexes it creates are dead simple:

from doc in docs.Things
select new { Created = doc.Created }

Field: Created Storage: No Indexing Default Sort String

In all of our static indexes Created is in the map statement (Created is a DateTime) however it doesn't display in the fields list. We also tried numerous combinations of Store/Index/Sorting settings and it would happen regardless, however once we took away the TransformResults it would work as expected, but that isn't really an option as it's needed.

[Matt Warren]

There was a bug/issue raised a while ago where indexes were rebuit when all that had changed was the TransformResults (whereas it didn't need to be rebuilt).

Could this be related to that?

[Chris Marisic]

When tracing this through server logging we see it specifically queries the named index, then immediately says "creating new temp index" and then says querying the temp index.

[Matt Warren]

That's wierd, see the code here https://github.com/ayende/ravendb/blob/1.2/Raven.Database/Server/Responders/Index.cs#L184

If should use an existing index unless your indexes happen to start with "dynamic/"?

[Chris Marisic]

The index does not start with the name dynamic, it queries the static index and then decides it's going to create the temp index anyway.

[Oren Eini (Ayende Rahien)]

Chris,

TransformResults tells RavenDB to avoid using this index, because the output is different than what the user may expect.

Thinking about this, we can probably use this anyway, just skip the transform results.

[Chris Marisic]

I'm confused by this statement. We're using the TransformResults for the most standard reason so instead of showing:

Book           | Author
Harry Potter | authors/1
Twilight        | authors/2

That we can show a list that uses names not ids.

Book           | Author
Harry Potter | Witch lady
Twilight        | World's worst author

We are reading from the results of the index with the query using As<> because we only want the values from the index not full documents.

[Oren Eini (Ayende Rahien)]

Chris,

We just added three new features:

* Transform Results no longer prevent an index from being selected as the target of a dynamic query

* You can now ask RavenDB why it selected a particular index:

localhost:8080/indexes/dynamic/Companies?query=Name:Northwind&explain=true

* We have a new config option:

Raven/CreateTemporaryIndexesForAdHocQueriesIfNeeded

Which you can set to false

[Chris Marisic]

On Tuesday, July 10, 2012 4:35:40 PM UTC-4, Oren Eini wrote:

Chris,

We just added three new features:

localhost:8080/indexes/dynamic/Companies?query=Name:Northwind&explain=true

Does this only work on the /dynamic/ address? or will this work on any query request?

[Itamar Syn-Hershko]

Any query request that is issued without a named index

[Chris Marisic]

Well that's the problem, these requests ARE issued to a named index, yet they still end up on the dynamic.

[Itamar Syn-Hershko]

I may be wrong here, I just had a quick look at the changes Oren made.

Can you try the latest build - it should give a lot more details too, we can work off that

[Itamar Syn-Hershko]

Also, a Fiddler capture we can replay would be ideal

[Oren Eini (Ayende Rahien)]

There isn't a code path that allows this to happen

Can you send a failing test?

[Ken Dale]

I'm having trouble getting this to work using Build 2032 invoking the GET with fiddler. Am I missing something that the C# client does when a dynamic index must be created to fulfill the request?

[Oren Eini (Ayende Rahien)]

No, a GET request to:

http://localhost:8080/indexes/dynamic/Users?query=Name:ayende

Is all you need.

[Ken Dale]

Found the issue, need to use Build 2033. Didn't download the latest, oops.

Unfortunately what we want to analyze is not a call to a dynamic index and it seems explain=true does not work without dynamic. I modified the GET request a bit for it to work with dynamic and I'm seeing this:

[{"Index":"AgencyIndex","Reason":"The following field have aren't using default indexing: "}, ... more here ...

It doesn't indicate which field or fields "aren't using default indexing".

On Tuesday, July 17, 2012 4:31:28 AM UTC-4, Oren Eini wrote:

No, a GET request to:

http://localhost:8080/indexes/dynamic/Users?query=Name:ayende

Is all you need.

[Matt Warren]

> Unfortunately what we want to analyze is not a call to a dynamic index and it seems explain=true does not work without dynamic

That's expected, it'll only give you the explanation when you are using dynamic indexes. If you don't specify dynamic, it only uses the index name you give it, it doesn't try to find a best match.

> [{"Index":"AgencyIndex","Reason":"The following field have aren't using default indexing: "}, ... more here ...

> It doesn't indicate which field or fields "aren't using default indexing".

What does the index look like in the management studio? It'll tell you how the fields are indexed.

[Ken Dale]

With build 960 I'm running a Session.Query<MyType, MyTypeIndex>() with filtering sorting paging along with a .As<MyTypeIndex.IndexResult>() then ToList(); and a temp index appears in the management studio -- which is not supposed to be able to happen because the query is directed which index to use, correct?

[Oren Eini (Ayende Rahien)]

Can you create a failing test?

[Oren Eini (Ayende Rahien)]

Because dynamic searches requires default indexing.

You can't do an identical search on analyzed and not analyzed fields.

You can do Foo = doc.Foo (not analyzed) and then Foo_Analyzed = doc.Foo (analyzed)

On Wed, Jul 25, 2012 at 7:32 PM, Lance McDonald <lmcd...@healthdialog.com> wrote:

I've been struggling with what sounds like a similar issue.  I had a working search prototype using a single static index which now insists on generating a dynamic index for reasons I cannot yet understand.  If I specify the index name all is well, but I am trying to understand why the query optimizer isn't choosing my static index automatically, which is what leads me to this thread.

I have installed unstable build 2043 and when I run the query using the "explain" parameter I too see the reason "The following field have aren't using default indexing".  I guess my question is, what is this really telling me?  I know from my index definition that two of the three fields are using default indexing and one field is analyzed.  I do not understanding why this index is ignored in favor of a dynamic index given that my query references only the 3 fields contained in the index.

On Tuesday, July 17, 2012 9:09:23 AM UTC-4, Matt Warren wrote: