Version store out of memory

[Kijana Woodard]

I ran into "Version store out of memory (cleanup already attempted) error" just now: http://managedesent.codeplex.com/discussions/248471

I didn't increase MaxVerPages, I just hit q on the console....waited an agonizingly long time [5 minutes :-)].....and everything seems to be working again.

Is this something you guys would like a more complete write up about or is it a known issue and I should research MaxVerPages?

[Oren Eini (Ayende Rahien)]

Kijana,

RavenDB has a maximum space in which is handles transactions. That is the MaxVerPages.

This is used for both user & system transactions.

User transactions is modifying documents. System transactions are usually book keeping (light) and map/reduce, which tend to be more expensive. 

Getting this error usually means that ravendb run out of place to do that type of work.

It is pretty rare unless you are really hammering the server with work.

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Kijana Woodard]

Hmmm. No hammering. This is a dev integration environment. Indexes get created in every commit, but no changes around the time this happened. What I did which seemed to instigate this was run an update to done app level docs which are tiny. One of them is used in an index via LoadDocument. That index has several let statements which I'm not happy about. I'll save that story for another thread.

Storing a few tiny docs took a few hundred thousand ms, and then I saw the error.

I'm pretty sure I can't repro on demand. Haven't seen it since I restarted.

Running 2330.

Can post index and docs if this is interesting.

[Chris Marisic]

a few hundred thousand ms?

[Oren Eini (Ayende Rahien)]

That would force the ravendb server to basically re-index everything.

[Oren Eini (Ayende Rahien)]

And there is a cost associated with handling updates to documents being referenced by a lot of other stuff.

[Kijana Woodard]

Yes.

[Kijana Woodard]

My concern is that it appears that the entire db locked up. Another dev reached out to me with a YSOD basically saying the app couldn't talk to raven.

The server was running under console, would there be logs written to disk I could send?

[Kijana Woodard]

Found the logs. There was more going on than I thought. One thing that could be a contributing factor, for lack of requirements, there was a query using .RandomOrdering(seed). I've removed it from the main flow.

The witnessed behavior from the browser was that the request hung until it returned YSOD after the ~12 minutes shown in the console. Should it even be possible for a request to hang for that long?

[Oren Eini (Ayende Rahien)]

It is possible when you using LoadDocument and a MASSIVE number of documents / indexes are referencing an item that you updated.

[Kijana Woodard]

There are about 750k product documents. One item I was inserting is a list of vendors to exclude from search. I do a LoadDocument inside the product index and see if the vendor is in the list. I expected it to reindex (should be a rare event to mod this doc), but I didn't expect it to hang on SaveChanges.

I suppose I could re-engineer things so that I create an empty doc at vendors/excluded/{vendor key} and LoadDocument that. Would that be better? I'm guessing the index agent would be able to target just the affected products in that case.

[Oren Eini (Ayende Rahien)]

It would be able to just touch the changed stuff, yes.

[Kijana Woodard]

Is the other scenario something that should have a ticket or maybe a warning in the docs?

[Kijana Woodard]

Maybe this is something that is fixed in 2.5.

I modified an index with ~200k documents.

The modification was from

select new

{

  ...

}.Boost((float)trend);

to

select new

{

  ...

}.Boost((float)trend + 1);

Is it trying to do work while accepting the index put? 

I was expecting it to take the index definition quickly and then for it to be stale and lose search results for a while. Instead I got 502s and timeouts in my web app. I also noticed that the index disappeared after the first 502. That made queries that explicitly use the index crash.

Since this is dev, no big deal, but definitely am going to look at _not_ creating indexes on web app start. I was thinking of that as a long term todo, but maybe it's moving up the priority list. :-D

[Oren Eini (Ayende Rahien)]

No, in 2.5 we just did a lot of performance improvements, I think that this is it.

[Oren Eini (Ayende Rahien)]

Huh? What do you mean, this happened at index creation?

[Oren Eini (Ayende Rahien)]

Yes
http://issues.hibernatingrhinos.com/issue/RDoc-125

[Kijana Woodard]

Yes. I changed the loaded document so that one document wasn't loaded by all products as we discussed. I also moved index creation out of the app startup pipeline. I created a separate route that only creates the indexes. I redeployed and started the app which worked fine. Then I hit the index creation route and got these response times. That was bad, but not the worse. The worse aspect was that the old index was deleted very quickly. That meant that all pages that use that index got a ysod saying the index did not exist. Effectively, app down.

[Oren Eini (Ayende Rahien)]

Hm... any way for me to try to reproduce this?

[Kijana Woodard]

I'll work on putting something together.

[Kijana Woodard]

It seems to be related to LoadDocument.

3 files using Test Helpers v2360: https://gist.github.com/kijanawoodard/5638693

v1.cs most closely resembles my actual code. Results as a comment at top of file.

In v2.cs, I removed the excess let statements and built the score in the stats doc. The timings were actually worse, but I forgot to save them.

In v3.cs, I got rid of LoadDocument altogether. Obviously, I can get the same results, but updating the index is _much_ faster. 

I also tried doing a single LoadDocument inside the select instead of as a let, but it didn't help.

Thanks in advance.

[Oren Eini (Ayende Rahien)]

Looking at the code, the part that is probably most expensive here is DeleteIndex, in which we have to go through the following internal structures

 MappedResults,
 ScheduledReductions,
 ReducedResults,
 ReduceKeysCounts,
 ReduceKeysStatus,
 IndexedDocumentsReferences,

The process of updating an index is internally a Delete/Add command.

And we need to cleanup all of this data. In your case, this is probably: IndexedDocumentsReferences

I am not sure that there is something that we can really do here. We can't keep processing queries to the index while we are doing this, and we can't not do it.

[Chris Marisic]

Couldn't it go something like?

Update Index:

rename-existing foo_index_expired

create foo_index

delete food_index_expired?

[Oren Eini (Ayende Rahien)]

All of the index data is saved with the index name, it would be pretty much the same cost to do that and to do the actual deletion.

[Kijana Woodard]

Well this explains the behavior I was seeing. For the test I was creating 500k documents. In my actual environment I had ~2M docs. Now I have ~6.5M docs (testing the bulk import feature). The web app was timing out the requests. Then I was in a state where the index was deleted but not created again and seeing "index does not exist" errors in the web app. Eventually I was able to submit the create index request and have it return instantly as expected. I assume that was when it was completely done cleaning up.

I confirmed that adding an index with LoadDocument is fast even with 500k docs if the index is not already present (v4.cs: https://gist.github.com/kijanawoodard/5638693).

Why does LoadDocument slow down index delete so much? Without LoadDocument, replacing the index on 500k docs is sub second.

Giving up LoadDocument leaves me with two unpalatable choices. 

Originally, I wanted to patch the product document. I can't do that because the system imports the products from third parties and I'm using Bulk Insert. With Bulk Insert, I can overwrite, but not "update". The third parties can send changes to the products at any time and the product doc will be overwritten, losing the patched data.

I could give up Bulk Insert and Load each product while parsing the csv and Store/Update as needed. I figure this will slow down product import dramatically.

I could switch to a MultiMap index. The LoadDocument syntax is sooo much more succinct, I'd hate to have to resort to multimap when this is really stitching together two docs with a 1-1 relationship.

So my mind starts to look for other ways.

I could do some kind of Blue/Green index scheme where I have ProductIndex1 and ProductIndex2. Whenever I make a change to the index, I make to the one "not in production" and switch all my code to the proper version. This sounds terrible for even a single developer, and awful for a team.

My next thought is to give up strongly typed index names and go with a string for index name. Then I can apply ProductIndex2 and then change a config setting to update the index name for queries on product to "ProductIndex2" whenever I feel the index is sufficiently far along.

That's better, but not wonderful.

At this point, I'm thinking I could create a document with "Active Index Names" which my queries would use to get the index name string by a key (like "Product Index"). Then in the "create index" code, I would generate an index name  by doing something like "new ProductIndex().IndexName + DateTimeOffset.UtcNow.Ticks". When the index was created, I would update my "Active Index" document to use the new index name.

If I wanted to get fancy, I could have some kind of "AsOf" timestamp so I could say "start using the new index in one hour". If I wanted to get really really fancy, I could watch the indexing process and update it once it got to the current etag.

Now, could some or all of this fit inside RavenDB itself so that, effectively, replacing an index was Add/Delete instead of Delete/Add?

p.s.

Another piece to the puzzle on LoadDocument. I've noticed that both Bulk Insert and Indexing gets much much slower as more docs are added. With an empty db, I see bulk insert times of 1-2k/second. With ~6 million docs, it takes ~10-20 seconds for a single 512 doc batch. 

I'd rather that be faster, but it just churns away all the time and keeps up with the feed volume for the moment. The site doesn't slow down when querying - less than 5 seconds Glimpse time for a never before preformed text search across the 6 million docs. Once done once, less than 100 ms. This is with raven on an azure vm and free azure website hosting for the app.

[Oren Eini (Ayende Rahien)]

Kijana,

There is a lot of stuff there, answers inline.

On Fri, May 24, 2013 at 5:28 PM, Kijana Woodard <kijana....@gmail.com> wrote:

Well this explains the behavior I was seeing. For the test I was creating 500k documents. In my actual environment I had ~2M docs. Now I have ~6.5M docs (testing the bulk import feature). The web app was timing out the requests. Then I was in a state where the index was deleted but not created again and seeing "index does not exist" errors in the web app. Eventually I was able to submit the create index request and have it return instantly as expected. I assume that was when it was completely done cleaning up.

I confirmed that adding an index with LoadDocument is fast even with 500k docs if the index is not already present (v4.cs: https://gist.github.com/kijanawoodard/5638693).

Why does LoadDocument slow down index delete so much? Without LoadDocument, replacing the index on 500k docs is sub second.

When you call LoadDocument, you create an association from docs/1 to docs/2. So when docs/2 changes, docs/1 will be re-indexed.

On large data sets, that is a lot of associations that needs to be managed, which is why you are seeing this issue.

 

Giving up LoadDocument leaves me with two unpalatable choices. 

Originally, I wanted to patch the product document. I can't do that because the system imports the products from third parties and I'm using Bulk Insert. With Bulk Insert, I can overwrite, but not "update". The third parties can send changes to the products at any time and the product doc will be overwritten, losing the patched data.

I don't understand the choices that you present here.

 

I could give up Bulk Insert and Load each product while parsing the csv and Store/Update as needed. I figure this will slow down product import dramatically.

I could switch to a MultiMap index. The LoadDocument syntax is sooo much more succinct, I'd hate to have to resort to multimap when this is really stitching together two docs with a 1-1 relationship.

MultiMap wouldn't have the issue, because it doesn't need to deal with the associations, however, it does mean you'll probably have map/reduce there.

 

So my mind starts to look for other ways.

I could do some kind of Blue/Green index scheme where I have ProductIndex1 and ProductIndex2. Whenever I make a change to the index, I make to the one "not in production" and switch all my code to the proper version. This sounds terrible for even a single developer, and awful for a team.

My next thought is to give up strongly typed index names and go with a string for index name. Then I can apply ProductIndex2 and then change a config setting to update the index name for queries on product to "ProductIndex2" whenever I feel the index is sufficiently far along.

You can do that using conventions, no?

 

That's better, but not wonderful.

At this point, I'm thinking I could create a document with "Active Index Names" which my queries would use to get the index name string by a key (like "Product Index"). Then in the "create index" code, I would generate an index name  by doing something like "new ProductIndex().IndexName + DateTimeOffset.UtcNow.Ticks". When the index was created, I would update my "Active Index" document to use the new index name.

If I wanted to get fancy, I could have some kind of "AsOf" timestamp so I could say "start using the new index in one hour". If I wanted to get really really fancy, I could watch the indexing process and update it once it got to the current etag.

Now, could some or all of this fit inside RavenDB itself so that, effectively, replacing an index was Add/Delete instead of Delete/Add?

p.s.

Another piece to the puzzle on LoadDocument. I've noticed that both Bulk Insert and Indexing gets much much slower as more docs are added. With an empty db, I see bulk insert times of 1-2k/second. With ~6 million docs, it takes ~10-20 seconds for a single 512 doc batch. 

Depends on the options you do for bulk insert, if you told it to do updates & to update related indexes, that might mean it has to do a lot more work.

[Kijana Woodard]

I'll break this apart a bit.

"Depends on the options you do for bulk insert, if you told it to do updates & to update related indexes, that might mean it has to do a lot more work."

Here is my bulk insert setup:

using (var bulkInsert = _store.BulkInsert(options: new BulkInsertOptions() { CheckForUpdates = true }))

[Kijana Woodard]

"You can do that using conventions, no?"

Maybe, but that would require any developer modifying an index to remember to update the conventions and we'd better do it in lock step with _production_ not just what happens to be in source control.

[Oren Eini (Ayende Rahien)]

CheckForUpdatesis the costly thing.

It needs to check for the associated documents, and if they exist, update them all.
If you have a lot of associations, it is costly.

Please note, LoadDocument is a really nice feature, but it has a cost associated with it.

[Kijana Woodard]

"When you call LoadDocument, you create an association from docs/1 to docs/2. So when docs/2 changes, docs/1 will be re-indexed.
On large data sets, that is a lot of associations that needs to be managed, which is why you are seeing this issue."

Here's what's interesting - As in the tests, in my actual environment, there are  ~5 "stats" docs associated to to ~5 "product" docs out of 6 million. So there shouldn't be a lot of data here. I'm guessing using LoadDocument is setting up some kind of placeholders?

[Oren Eini (Ayende Rahien)]

Yeah, it does.

LoadDocument("does-not-exists") still need to setup things so when a does-not-exists document gets created, the doc will be indexed.

If you can check it ahead of time, that is _much_ better.

[Kijana Woodard]

"CheckForUpdatesis the costly thing.

It needs to check for the associated documents, and if they exist, update them all.
If you have a lot of associations, it is costly.

Please note, LoadDocument is a really nice feature, but it has a cost associated with it."

"Yeah, it does.

LoadDocument("does-not-exists") still need to setup things so when a does-not-exists document gets created, the doc will be indexed.

If you can check it ahead of time, that is _much_ better."

Ok. This makes sense. Maybe the "CheckForUpdates" is not what I want. What I really want is "allow overwrite an existing doc with the same id". So it would behave like session.Store("existing-id"). Then just let the normal indexing behavior happen, eventually. I know some changes were made for BulkInsert in unstable, maybe this is addressed there.

Now, the "products" are not referenced in a LoadDocument. The index for products is using LoadDocument<ProductStat>("the-stat-id"). The stats are _not_ being updated by bulk import (or at all in the tests). Same rules should apply?

[Oren Eini (Ayende Rahien)]

CheckForUpdates needs to do a check per document to see if there is a need to update or insert.
CheckReferencesInIndexes is the one I was thinking about.

[Kijana Woodard]

Ok. Then I'm a bit surprised Bulk Insert slows down over time. I would assume it's the same as doing session.Store without optimistic concurrency. Again, maybe 2.5 has improvements. It could also be disk speed. I plan on running some tests with millions of docs on my local with ssd to see what things look like.

What I'm worried about is that session.Store is going to slow down as well.

One thing I realized with Bulk Insert is that it's going to be ACID. When I leave the bulk insert using block, I can do a Load("just-bulk-inserted-id") and it will return the doc. In my particular case, I'm never going to do that. I really just want to get the docs in the server and I will discover them when they get indexed, but I realize this is not the general use case. 

It would be more of a political victory to have this conversation:

"Look how fast we inserted those docs". 

Why aren't they on the site? 

"They are are being indexed for full text search."

Oh. Ok. Great. And search is still fast even though we're importing. Cool!

I've considered importing into an embedded db on disk to skip indexing (wouldn't have the index there) plus remove the http overhead. Then I'd replicate to the master/master nodes the web site will use. Up to this point, I've considered it to be a micro-optimization but is it crazy or a reasonable choice? I then worry about whether I should replicate to both nodes or to one with Transitive.

[Kijana Woodard]

Let me back off of the statement "bulk insert slows down". I just remembered I added a feature that detects duplicates in the product feed and skips them. The slow down I observe might be from skipping tons of rows while parsing the feed csv. I will investigate further.

In 2.5, is there a way to do "Bulk Insert but the docs don't have to be available when the using block finishes"?

I think 2.5 is getting close, so it may be time for me to start investigating. I'd like to do some things with Stable Streaming as well. :-D

[Kijana Woodard]

The app was pretty much DOA as far as indexing was concerned. Any modification to the product index would result in site down.

I decided to implement a quick and dirty "Insert New Index then Delete Old Index" in my app code just so I can push code changes until a better answer comes along.

I ended up with this guy: https://gist.github.com/kijanawoodard/5647379

It creates a name based on GetIndexHash. It checks to see if the index needs an update. If so, PutIndex. Then delete the old index.

All in all, I'm happier because the web app does not YSOD. This also opens up the possibility of creating a new index version and moving to it _after_ I see that the index has progressed "far enough". That's still a bit tricky if code is using fields that were not indexed previously, but it's something.

The reason IndexNameMap is a separate class is so that I can register it as a singleton and use it for the entire app life. I didn't want to force everything that queries for products to make a minimum of two requests (one being for the index name to use). This is where I think a solution on the server side would excel.

IIndexNameProvider is for controllers to ask for the current index name.

Unfortunately, I think I need to _not_ delete the old index until the new one has made some progress. It _seems_ like the delete stops the new one from going to work. I read a comment is the source code about a single handler for indexes and wonder if that is what's happening.

I waited an hour and indexing hadn't moved forward one etag. Then I hit "q" in the console. 35 minutes later it still hadn't shut down, but, of course, requests from the web app were no longer being handled, so I shut the web app down. I also shut down the product importer app so that _nothing_ was possibly connecting to the db. 30 minutes after that, I was desperate for sleep and went to kill the process. Low and behold it stopped before I could kill it. It started up ok. But when I started the product importer, I got this: "Could open database named: xzyabc An operation was attempted on a nonexistent network connection". I have no idea what this means because the app and the db are on the same machine. Replication is turned on, but there are no other nodes. 

I shut it down with q again. This time it came down fast. I started it back up and the product importer and everything looked good. By the time I alt-tabbed to azure and started the web site and then refreshed raven studio, indexing had progressed 40k+ documents. 6.5M documents to go. Time for bed. :-)

Here is the last hour of "life" from the logs (times are utc):

2013-05-25 01:39:10.5907,Raven.Database.Server.HttpServer,Debug,,Error when using events transport,"System.AggregateException: One or more errors occurred. ---> System.Net.HttpListenerException: An operation was attempted on a nonexistent network connection

   at System.Net.HttpResponseStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)

   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.<>c__DisplayClass3.<WriteAsync>b__1(AsyncCallback callback, Object state) in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 116

   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)

   at System.Threading.Tasks.TaskFactory.FromAsync(Func`3 beginMethod, Action`1 endMethod, Object state, TaskCreationOptions creationOptions)

   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.WriteAsync(String data) in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 115

   --- End of inner exception stack trace ---

---> (Inner Exception #0) System.Net.HttpListenerException (0x80004005): An operation was attempted on a nonexistent network connection

   at System.Net.HttpResponseStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)

   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.<>c__DisplayClass3.<WriteAsync>b__1(AsyncCallback callback, Object state) in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 116

   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)

   at System.Threading.Tasks.TaskFactory.FromAsync(Func`3 beginMethod, Action`1 endMethod, Object state, TaskCreationOptions creationOptions)

   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.WriteAsync(String data) in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 115<---

"

2013-05-25 01:44:15.3122,Raven.Database.Indexing.WorkContext,Debug,<system>,"No work was found, workerWorkCounter: 0, for: ReducingExecuter, will wait for additional work",

2013-05-25 01:44:15.4685,Raven.Database.Indexing.WorkContext,Debug,<system>,"No work was found, workerWorkCounter: 0, for: IndexingExecuter, will wait for additional work",

2013-05-25 02:26:49.8784,Raven.Database.Server.HttpServer,Debug,,Error when using events transport,"System.AggregateException: One or more errors occurred. ---> System.Net.HttpListenerException: An operation was attempted on a nonexistent network connection

   at System.Net.HttpResponseStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)

   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.<>c__DisplayClass3.<WriteAsync>b__1(AsyncCallback callback, Object state) in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 116

   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)

   at System.Threading.Tasks.TaskFactory.FromAsync(Func`3 beginMethod, Action`1 endMethod, Object state, TaskCreationOptions creationOptions)

   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.WriteAsync(String data) in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 115

   --- End of inner exception stack trace ---

---> (Inner Exception #0) System.Net.HttpListenerException (0x80004005): An operation was attempted on a nonexistent network connection

   at System.Net.HttpResponseStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)

   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.<>c__DisplayClass3.<WriteAsync>b__1(AsyncCallback callback, Object state) in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 116

   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)

   at System.Threading.Tasks.TaskFactory.FromAsync(Func`3 beginMethod, Action`1 endMethod, Object state, TaskCreationOptions creationOptions)

   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.WriteAsync(String data) in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 115<---

"

2013-05-25 02:44:15.2584,Raven.Database.Indexing.WorkContext,Debug,<system>,"No work was found, workerWorkCounter: 0, for: ReducingExecuter, will wait for additional work",

2013-05-25 02:44:15.4043,Raven.Database.Indexing.WorkContext,Debug,<system>,"No work was found, workerWorkCounter: 0, for: IndexingExecuter, will wait for additional work",

Here's the error from trying to delete the old index:

Line 123:			if (!string.IsNullOrWhiteSpace(oldName))
Line 124:			{
Line 125:				_session.Advanced.DocumentStore.DatabaseCommands.DeleteIndex(oldName);
Line 126:			}
Line 127:		}

[WebException: The remote server returned an error: (500) Internal Server Error.]
   System.Net.HttpWebRequest.GetResponse() +6442408
   Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:332

[InvalidOperationException: Url: "/indexes/ProductIndex"

System.InvalidOperationException: Cannot modify indexes while indexing is in progress (already waited full minute). Try again later
   at Raven.Database.Storage.IndexDefinitionStorage.TryRemoveIndexContext() in c:\Builds\RavenDB-Stable\Raven.Database\Storage\IndexDefinitionStorage.cs:line 271
   at Raven.Database.DocumentDatabase.DeleteIndex(String name) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 1278
   at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx) in c:\Builds\RavenDB-Stable\Raven.Database\Server\HttpServer.cs:line 864
   at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx) in c:\Builds\RavenDB-Stable\Raven.Database\Server\HttpServer.cs:line 609
]
   Raven.Client.Connection.HttpJsonRequest.HandleErrors(WebException e) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:475
   

[Oren Eini (Ayende Rahien)]

Why would you want to go that way?
It seems like it would be very complex?

How fast Do you need it to be?

[Oren Eini (Ayende Rahien)]

What do you mean, not have to be available?

[Kijana Woodard]

Calling Load('id-just-bulk-loaded') can result in null. In other words, the server accepts a batch and returns with putting them in the db.

I'd like to check out 2.5 before thinking about this more. Currently I see about 500/s. I noticed with the tests on embedded that it was around 5k/s, but of course that's just memory.

[Kijana Woodard]

The story from last night continues.

9 hours later, all indexes are still stale. The product index is at 4.8M / 6.6M.

I think Monday, I have to consider designing around LoadDocument in the Map. The tests I ran for this also indicated that indexing speed without LoadDocument was 2-3x faster.  

Between not being able to modify an index without bringing down the app and slow index speeds, LoadDocument appears to be unusable on 2330 with millions of documents. 

I try to stay clear of unstable so I can go to production comfortably at a moments notice, but I might be doing some experiments there too to see if I should try to wait for 2.5 stable or redesign this portion of the app.

[Oren Eini (Ayende Rahien)]

Yes, that is because we do some internal batching.

It would appear soon, and the Dispose() won't return until this is completed.