I was going to update a customer system with build-888 today, but got a serious indexing issue. I've rebuilt all indexes, but there were still documents missing in the indexes.
In the log I see an OOME:
2012-04-10 18:46:33.1658| 5|WARN |Raven.Database.Indexing.AbstractIndexingExecuter |Failed to index documents for index: Product/Number|"System.OutOfMemoryException
The largest batch size I saw was 16.384 and I' missing about 14.000 docs from the index.
It looks like as if this OOME happens, it looses all the remaining documents from the current batch, which is a really bad thing.
(This happened with a 32bit application and plenty of RAM)
I tried with a 64bit RavenDB.Server.exe which indexed all documents but still uses all my physical memory until swapping kicks in and makes the system nearly unusable.
After setting a limit of MaxNumberOfItemsToIndexInSingleBatch = 128 everything works fine.
So I see two problems:
1. It is troubling that the OOME during the index process causes lots of docs to be missing from the index, with no way to recover and the only indication that something went wrong being a log entry. 2. The Auto-Batch-Size-Tuning still isn't safe by default
On Wed, Apr 11, 2012 at 3:00 AM, Justin A <jus...@adler.com.au> wrote: > I don't think I can help you Tobi .. but just to clarify what you're > saying ..
> *>>The largest batch size I saw was 16.384 and I' missing about 14.000 > >>docs from the index. *
> Is that 16 *thousand *and 384 ... and your missing around 14 *thousand * > docs?
> Just making sure I'm getting the Culture right, on your statement. 16K vs > 16 is a big difference :)
Tobi, A few interesting things. * Any exception during the indexing process should force a re-index, so I don't know why it didn't. * I can't think of any reason why in 64 bits, it would force paging. We are actively checking to make sure that this won't happen.
Is there any way you can reproduce this? Note that I am pretty sure that paging on 64 bits is another manifestation of the same problem.
On Tue, Apr 10, 2012 at 9:32 PM, Tobi <listacco...@e-tobi.net> wrote: > I was going to update a customer system with build-888 today, but got a > serious indexing issue. I've rebuilt all indexes, but there were still > documents missing in the indexes.
> In the log I see an OOME:
> 2012-04-10 18:46:33.1658| 5|WARN |Raven.Database.Indexing.**AbstractIndexingExecuter > |Failed to index documents for index: Product/Number|"System.** > OutOfMemoryException
> The largest batch size I saw was 16.384 and I' missing about 14.000 docs > from the index.
> It looks like as if this OOME happens, it looses all the remaining > documents from the current batch, which is a really bad thing.
> (This happened with a 32bit application and plenty of RAM)
> I tried with a 64bit RavenDB.Server.exe which indexed all documents but > still uses all my physical memory until swapping kicks in and makes the > system nearly unusable.
> After setting a limit of MaxNumberOfItemsToIndexInSingl**eBatch = 128 > everything works fine.
> So I see two problems:
> 1. It is troubling that the OOME during the index process causes lots > of docs to be missing from the index, with no way to recover and > the only indication that something went wrong being a log entry. > 2. The Auto-Batch-Size-Tuning still isn't safe by default
On 11.04.2012 08:12, Oren Eini (Ayende Rahien) wrote:
> * I can't think of any reason why in 64 bits, it would force paging. We > are actively checking to make sure that this won't happen.
I have only 8GB RAM installed, which isn't that much, but it still should be enough. The data might contain some large docs, but I haven't figured out yet how larger they are. How do you measure the document size?
> Is there any way you can reproduce this?
Yes, I can. I'll prepare some sample data today and e-mail you a download link.
On Wed, Apr 11, 2012 at 10:16 AM, Tobi <listacco...@e-tobi.net> wrote: > On 11.04.2012 08:12, Oren Eini (Ayende Rahien) wrote:
> > * I can't think of any reason why in 64 bits, it would force paging. We > > are actively checking to make sure that this won't happen.
> I have only 8GB RAM installed, which isn't that much, but it still should > be enough. The data might contain some large docs, but I haven't figured > out yet how larger they are. How do you measure the document size?
> > Is there any way you can reproduce this?
> Yes, I can. I'll prepare some sample data today and e-mail you a download > link.
