index with one document remains stale

[Wallace Turner]

using raven 888

I added a new document:

{

  "Username": "yyyy",

  "Password": "xxxx"

}

and a new index on that document:

from doc in docs.Users

select new { Username = doc.Username }

and yet 10 minutes after adding this the index is *still* stale and thus when querying the index from C# I get zero results returned:

var user = Session.Advanced.LuceneQuery<User>("Users/ByUsername").WhereEquals("Username", username).Statistics(out stats).FirstOrDefault();

//user is null

What am I doing wrong?

[Oren Eini (Ayende Rahien)]

How many documents do you have in the db?

All indexes have to go through all documents, in later builds, we have some optimizations around this, but that can still take some time on very large dbs.

Note that "stale" doesn't mean it won't return the right result.

[Wallace Turner]

How many documents do you have in the db? 

of type DataResult ~550,000

of type User, 1

A new user document is added to the system (ie User.Count goes from 0 to 1) and it took approx 15min to be returned by the query. I havent tested this but is this going to be the case for adding a second user? If yes, how is best to 'force index' on that result as the added user will need to be returned in a query so it can reliably be used upstream (eg for a login sequence)

Cheers.

[Oren Eini (Ayende Rahien)]

Once that new index is caught up, it should take a few ms to operate.

[Wallace Turner]

> Once that new index is caught up, it should take a few ms to operate. 

Yes, going from 1 to 2 is very quick, (didnt noticeable stay stale) however adding the index initially took a significant amount of time, if this is normal then i'll move on. note the new index was not related in any way to the existing 550,000 documents.

[Kijana Woodard]

I took from Oren's comment that your new index will go through all docs in the db regardless of the .net types involved.

[Oren Eini (Ayende Rahien)]

Yes, that is how it works

[Oren Eini (Ayende Rahien)]

We have a lot of optimizations around that, but in the end, to figure out what the type is we have to load the document, so we have to scan through all of them.

[John]

With the new release 2261, Can you please confirm if this is still the same were it scans all the documents to update the index
even though there are changes to 1 document field.

thanks
John

[Chris Marisic]

I think you're misunderstanding something.

When you update 1 field inside of a document, EACH INDEX must look at that document and determine "do i care?"

Updating 1 document does not cause anything other than that 1 document to be processed for indexing.

Now if your question is, does modifying 1 field in the INDEX require it to scan ALL DOCUMENTS, yes, yes it does. Because you've invalidated the entire index. However that will impact only 1 index and is unrelated to documents except that it will walk over the entire document store to rebuild that 1 index.