High memory consumption after new index

[Matthias De Ridder]

Yesterday around 10 o'clock we added new indexes to our Raven database. A few hours later users started to complain about performance. We noticed the database being unavailable on several occasions. The memory consumption of the IIS thread went through the roof. After deleting the new indexes, the memory consumption dropped. 

We noticed a lot of exceptions in the log: 

Could not open index ReplacementOf/Invoice/Overview. Recovery operation failed, forcibly resetting index;"System.InvalidOperationException: Index directory 'D:\InfohosServices\PROD\Financieel\Tarfac\Databases\Databases\tarfac.149\Indexes\110' does not exists for 'ReplacementOf/Invoice/Overview' index.

Some additional information:

- There were two new indexes added: ReplacementOf/Invoice/Overview and ReplacementOf/InvoiceSample/Overview.

- The new indexes are no actual replacement indexes. They are just normal indexes with ReplacementOf/ in their name.

- There are 30000 items these indexes need to index.

- ReplacementOf/InvoiceSample/Overview had indexed all items, ReplacementOf/Invoice/Overview never indexed any items. In the studio the status said the index never ran, even after querying it.

We did the same on other production environments were 2 million or more items were indexed without any problem. Do you see something what might be causing this problem?

We are using Raven version 30000.

[Oren Eini (Ayende Rahien)]

The error happens during db start, but I'm seeing a LOT of startups in the log, did you restart the database during that time?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthias De Ridder]

No, we didn't. That's the reason of the complaints. It looks like the database is almost constantly being loaded.

Op vrijdag 23 september 2016 12:23:16 UTC+2 schreef Oren Eini:

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

It is hard to know just from the warn log, what happens when you set it to debug?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Matthias De Ridder]

Both indexes are now unstale, after which the high memory consumption dropped. Since it occured on only one of our production environments, we can't just try it again.

Op vrijdag 23 september 2016 13:17:49 UTC+2 schreef Oren Eini:

[Grisha Kotler]

Can you reproduce that in your development environment?

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Matthias De Ridder]

Unfortunatelly not. We would not have released these indexes if we would have seen it in our development environment. It only happened in one production environment.

Op maandag 26 september 2016 09:44:16 UTC+2 schreef Grisha Kotler:

[Grisha Kotler]

Do you have the same amount of documents as in production?

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Matthias De Ridder]

No, our test environment has more documents. We will restore a backup of the production environment and try to reproduce the problem in our test environment.

Op maandag 26 september 2016 10:07:21 UTC+2 schreef Grisha Kotler:

[Grisha Kotler]

How many documents in total do you have? How many are relevant for that index?

How does the index definition looks like?

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Matthias De Ridder]

The database has 561,540 documents. Depending on the index, 27,661 or 17,242 documents are relevant. The indexes have some stored fields (InvoiceNumber, TotalAmount, TotalAdvance, PatientAmount, InsuranceAmount). The indexes looks like this:

First one:

from invoice in docs.Invoices

select new {

invoice = invoice,

rejections = invoice.Lines.Select(l => l.Error).Where(r => r != null && r.IsRejection)

} into this0

select new {

this0 = this0,

autoCorrections = this0.invoice.Lines.Select(l0 => l0.Error).Where(r1 => r1 != null && !r1.IsRejection)

} into this1

select new {

RunId = this1.this0.invoice.InvoiceRun.RunId,

ShipmentNumber = this1.this0.invoice.InvoiceRun.ShipmentNumber,

InvoiceNumber = this1.this0.invoice.InvoiceNumber,

BillingNumber = this1.this0.invoice.PaymentPlan.BillingNumber,

TypeOfAdmission = this1.this0.invoice.PaymentPlan.TypeOfAdmission,

TotalAmount = this1.this0.invoice.TotalAmount,

TotalAdvance = this1.this0.invoice.TotalAdvance,

InsuranceAmount = this1.this0.invoice.InsuranceAmount,

PatientAmount = this1.this0.invoice.PatientAmount,

PreviousInvoiceNumber = this1.this0.invoice.PreviousInvoiceNumber,

HasPreviousInvoiceNumber = !String.IsNullOrEmpty(this1.this0.invoice.PreviousInvoiceNumber),

DossierNumber = this1.this0.invoice.AdmissionInfo == null ? (object)"" : (object)this1.this0.invoice.AdmissionInfo.DossierNumber,

PermitNumber = ((object)(this1.this0.invoice.PaymentPlan.PermitNumber ?? ((object)""))),

PatientNumber = this1.this0.invoice.Patient.PatientNumber,

LastName = this1.this0.invoice.Patient.LastName,

FirstName = this1.this0.invoice.Patient.FirstName,

InvoicePeriodEndDate = this1.this0.invoice.InvoiceRun.InvoicePeriodEndDate,

InsuranceNumber = this1.this0.invoice.Insurance.InsuranceNumber,

HasUnbilledCorrections = this1.this0.invoice.HasUnbilledCorrections,

NeedsProcessingForCredit = this1.this0.invoice.NeedsProcessingForCredit,

SettlementYear = this1.this0.invoice.SettlementInfo.Year,

SettlementMonth = this1.this0.invoice.SettlementInfo.Month,

HasModifications = Enumerable.Any(this1.this0.invoice.Lines, l2 => l2.Error != null || Enumerable.Any(l2.Corrections)),

HandledRejectionCodes = this1.this0.rejections.Where(r3 => r3.IsHandled).SelectMany(r4 => r4.Codes),

UnhandledRejectionCodes = this1.this0.rejections.Where(r5 => !r5.IsHandled).SelectMany(r6 => r6.Codes),

HasHandledRejections = Enumerable.Any(this1.this0.rejections, r7 => r7.IsHandled),

HasUnhandledRejections = Enumerable.Any(this1.this0.rejections, r8 => !r8.IsHandled),

HasHandledAutoCorrections = Enumerable.Any(this1.autoCorrections, r9 => r9.IsHandled),

HasUnhandledAutoCorrections = Enumerable.Any(this1.autoCorrections, r10 => !r10.IsHandled),

HasUnhandledLineErrors = Enumerable.Any(DynamicEnumerable.Concat(this1.this0.rejections, this1.autoCorrections), r11 => !r11.IsHandled)

}

Second one:

from invoiceSample in docs.InvoiceSamples

select new {

RunId = invoiceSample.Invoice.InvoiceRun.RunId, 

InvoiceNumber = invoiceSample.Invoice.InvoiceNumber,

TypeOfAdmission = invoiceSample.Invoice.PaymentPlan.TypeOfAdmission,

TotalAmount = invoiceSample.Invoice.TotalAmount,

TotalAdvance = invoiceSample.Invoice.TotalAdvance,

InsuranceAmount = invoiceSample.Invoice.InsuranceAmount,

PatientAmount = invoiceSample.Invoice.PatientAmount,

PreviousInvoiceNumber = invoiceSample.Invoice.PreviousInvoiceNumber,

HasPreviousInvoiceNumber = !String.IsNullOrEmpty(invoiceSample.Invoice.PreviousInvoiceNumber),

DossierNumber = invoiceSample.Invoice.AdmissionInfo == null ? (object)"" : (object)invoiceSample.Invoice.AdmissionInfo.DossierNumber,

    PermitNumber = ((object)(invoiceSample.Invoice.PaymentPlan.PermitNumber ?? ((object)""))),

BillingNumber = invoiceSample.Invoice.PaymentPlan.BillingNumber,

PatientNumber = invoiceSample.Invoice.Patient.PatientNumber,

LastName = invoiceSample.Invoice.Patient.LastName,

FirstName = invoiceSample.Invoice.Patient.FirstName,

InvoicePeriodEndDate = invoiceSample.Invoice.InvoiceRun.InvoicePeriodEndDate,

InsuranceNumber = invoiceSample.Invoice.Insurance.InsuranceNumber

}

[Grisha Kotler]

For the first index: how many "Lines" are there for each invoice?

Did you set Max Index Outputs per document?

This might happened due to the precomputed batch optimization. This was fixed in the latest 3.0 stable.

To verify, can you deploy the first index on the test environment, using the same documents that you have in production?

(Both on build 30000 and the latest stable)

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

--

[Matthias De Ridder]

The number of lines for an invoice can range between 0 to 5000 lines. We have set the MaxIndexOutputsPerDocument to -1.

[Grisha Kotler]

You're creating up to 5000 results for each document. You're creating more than 2.5 million results (using the lowest batch size: 512)

You can try lowering the Raven/MaxNumberOfItemsToIndexInSingleBatch (default is 131,072)

or limit the MaxIndexOutputsPerDocument.

You should consider remodeling the Invoice document.

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

On 27 September 2016 at 11:39, Matthias De Ridder <orcr...@gmail.com> wrote:

The number of lines for an invoice can range between 0 to 5000 lines. We have set the MaxIndexOutputsPerDocument to -1.

--

[Matthias De Ridder]

No, we don't create up to 5000 results for each document. Each document will occur only once in the index. Depending on the index, it contains 27,661 or 17,242 documents.

[Oren Eini (Ayende Rahien)]

What does your index looks like?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Tue, Sep 27, 2016 at 1:27 PM, Matthias De Ridder <orcr...@gmail.com> wrote:

No, we don't create up to 5000 results for each document. Each document will occur only once in the index. Depending on the index, it contains 27,661 or 17,242 documents.

--

[Grisha Kotler]

Each document might a large number of entries:

from invoice in docs.Invoices

select new {

invoice = invoice,

rejections = invoice.Lines.Select(l => l.Error).Where(r => r != null && r.IsRejection)

} into this0

select new {

this0 = this0,

autoCorrections = this0.invoice.Lines.Select(l0 => l0.Error).Where(r1 => r1 != null && !r1.IsRejection)

} into this1

select new 

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

On 27 September 2016 at 13:27, Matthias De Ridder <orcr...@gmail.com> wrote:

No, we don't create up to 5000 results for each document. Each document will occur only once in the index. Depending on the index, it contains 27,661 or 17,242 documents.

--

[Oren Eini (Ayende Rahien)]

So each of those might generate 5000 index entries

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[Matthias De Ridder]

I think you are focusing on the wrong thing: the index doesn't generate an entry for each line. The index has one entry for each invoice. The lines are only used to calculate some values.

Also, although invoices might have up to 5000 lines, the average is around 25 lines per invoice.

invoices =>

                from invoice in invoices

                let rejections = invoice.Lines.Select(l => l.Error).Where(r => r != null && r.IsRejection)

                let autoCorrections = invoice.Lines.Select(l => l.Error).Where(r => r != null && !r.IsRejection)

                select new

                {

                    invoice.InvoiceRun.RunId,

                    invoice.InvoiceRun.ShipmentNumber,

                    invoice.InvoiceNumber,

                    invoice.PaymentPlan.BillingNumber,

                    invoice.PaymentPlan.TypeOfAdmission,

                    invoice.TotalAmount,

                    invoice.TotalAdvance,

                    invoice.InsuranceAmount,

                    invoice.PatientAmount,

                    invoice.PreviousInvoiceNumber,

                    HasPreviousInvoiceNumber = !String.IsNullOrEmpty(invoice.PreviousInvoiceNumber),

                    DossierNumber = invoice.AdmissionInfo == null ? (object)"" : invoice.AdmissionInfo.DossierNumber,

                    PermitNumber = invoice.PaymentPlan.PermitNumber ?? (object)"",

                    invoice.Patient.PatientNumber,

                    invoice.Patient.LastName,

                    invoice.Patient.FirstName,

                    invoice.InvoiceRun.InvoicePeriodEndDate,

                    invoice.Insurance.InsuranceNumber,

                    invoice.HasUnbilledCorrections,

                    invoice.NeedsProcessingForCredit,

                    SettlementYear = invoice.SettlementInfo.Year,

                    SettlementMonth = invoice.SettlementInfo.Month,

                    HasModifications = invoice.Lines.Any(l => l.Error != null || l.Corrections.Any()),

                    HandledRejectionCodes = rejections.Where(r => r.IsHandled).SelectMany(r => r.Codes),

                    UnhandledRejectionCodes = rejections.Where(r => !r.IsHandled).SelectMany(r => r.Codes),

                    HasHandledRejections = rejections.Any(r => r.IsHandled),

                    HasUnhandledRejections = rejections.Any(r => !r.IsHandled),

                    HasHandledAutoCorrections = autoCorrections.Any(r => r.IsHandled),

                    HasUnhandledAutoCorrections = autoCorrections.Any(r => !r.IsHandled),

                    HasUnhandledLineErrors = rejections.Concat(autoCorrections).Any(r => !r.IsHandled)

                };