My dear old friend: The OutOfMemoryException

[Tobi]

I was going to update a customer system with build-888 today, but got a
serious indexing issue. I've rebuilt all indexes, but there were still
documents missing in the indexes.

In the log I see an OOME:

2012-04-10 18:46:33.1658| 5|WARN
|Raven.Database.Indexing.AbstractIndexingExecuter |Failed to index
documents for index: Product/Number|"System.OutOfMemoryException

The largest batch size I saw was 16.384 and I' missing about 14.000
docs from the index.

It looks like as if this OOME happens, it looses all the remaining
documents from the current batch, which is a really bad thing.

(This happened with a 32bit application and plenty of RAM)

I tried with a 64bit RavenDB.Server.exe which indexed all documents but
still uses all my physical memory until swapping kicks in and makes the
system nearly unusable.

After setting a limit of MaxNumberOfItemsToIndexInSingleBatch = 128
everything works fine.

So I see two problems:

1. It is troubling that the OOME during the index process causes lots
of docs to be missing from the index, with no way to recover and
the only indication that something went wrong being a log entry.
2. The Auto-Batch-Size-Tuning still isn't safe by default

Tobias

[Justin A]

I don't think I can help you Tobi .. but just to clarify what you're saying ..

>>The largest batch size I saw was 16.384 and I' missing about 14.000 
>>docs from the index. 

Is that 16 thousand and 384 ...  and your missing around 14 thousand docs?

Just making sure I'm getting the Culture right, on your statement. 16K vs 16 is a big difference :) 

[Itamar Syn-Hershko]

Yes, the groups is easier for us to follow

We try to track SO as well, but emails just pop on our desktop when we work hence the first-class support :)

[Itamar Syn-Hershko]

Wrong thread. Speaking of great support :)

Tobi, we will look at that tomorrow or after the holiday. I think it'll be best if we'll try to repro and go from there.

[Tobi]

On 11.04.2012 02:00, Justin A wrote:

> Is that 16 /thousand /and 384 ... and your missing around 14 /thousand /docs?

>
> Just making sure I'm getting the Culture right, on your statement. 16K vs
> 16 is a big difference :)

Yes, 16K batch size and around 14k docs missing from the index.

Tobias

[Oren Eini (Ayende Rahien)]

Tobi,

A few interesting things.

* Any exception during the indexing process should force a re-index, so I don't know why it didn't.

* I can't think of any reason why in 64 bits, it would force paging. We are actively checking to make sure that this won't happen.

Is there any way you can reproduce this? Note that I am pretty sure that paging on 64 bits is another manifestation of the same problem.

[Tobi]

On 11.04.2012 08:12, Oren Eini (Ayende Rahien) wrote:

> * I can't think of any reason why in 64 bits, it would force paging. We
> are actively checking to make sure that this won't happen.

I have only 8GB RAM installed, which isn't that much, but it still should
be enough. The data might contain some large docs, but I haven't figured
out yet how larger they are. How do you measure the document size?

> Is there any way you can reproduce this?

Yes, I can. I'll prepare some sample data today and e-mail you a download
link.

Tobias

[Tobi]

On 11.04.2012 02:08, Itamar Syn-Hershko wrote:

> Tobi, we will look at that tomorrow or after the holiday.
> I think it'll be best if we'll try to repro and go from there.

I'll prepare some sample data today.

[Oren Eini (Ayende Rahien)]

Document size is a bit hard to calculate, but you can go to the studio, and click on edit document, and it will tell you how big a doc is.

[Tobi]

On 11.04.2012 09:21, Tobi wrote:

>> Tobi, we will look at that tomorrow or after the holiday.
>> I think it'll be best if we'll try to repro and go from there.
>
> I'll prepare some sample data today.
>

Did you had a chance to reproduce this with the samples I sent you?

Tobias

[Itamar Syn-Hershko]

Sorry, not yet. Holiday here so things are moving slower than usual. Next week probably.

[Tobi]

On 12.04.2012 16:59, Itamar Syn-Hershko wrote:

> Sorry, not yet. Holiday here so things are moving slower than usual. Next
> week probably.

Ok, then: Happy Passover!

Tobias

[Itamar Syn-Hershko]

Thanks :)