UnexpectedReduceTreePageException

[Wade Wright]

We have several map/reduce indexes that fail with the below error. However, not ALL of them fail with this, and sometimes if we rebuild the index it is fine and sometimes it errors again.   It doesn't fail with every document, in fact, it usually takes us a few thousand before we see the error. 

Unexpected exception occurred: Raven.Server.Documents.Indexes.MapReduce.UnexpectedReduceTreePageException: Encountered empty page which isn't a root. Page #1074 (count: 0) Leaf in '__raven/map-reduce/#reduce-tree-10289476376014822401' tree.

   at Raven.Server.Documents.Indexes.MapReduce.ReduceMapResultsBase`1.HandleTreeReduction(TransactionOperationContext indexContext, IndexingStatsScope stats, CancellationToken token, MapReduceResultsStore modifiedStore, LowLevelTransaction lowLevelTransaction, IndexWriteOperation writer, LazyStringValue reduceKeyHash, Table table) in C:\Builds\RavenDB-4.0-RC\src\Raven.Server\Documents\Indexes\MapReduce\ReduceMapResultsBase.cs:line 196

   at Raven.Server.Documents.Indexes.MapReduce.ReduceMapResultsBase`1.Execute(DocumentsOperationContext databaseContext, TransactionOperationContext indexContext, Lazy`1 writeOperation, IndexingStatsScope stats, CancellationToken token) in C:\Builds\RavenDB-4.0-RC\src\Raven.Server\Documents\Indexes\MapReduce\ReduceMapResultsBase.cs:line 104

   at Raven.Server.Documents.Indexes.Index.DoIndexingWork(IndexingStatsScope stats, CancellationToken cancellationToken) in C:\Builds\RavenDB-4.0-RC\src\Raven.Server\Documents\Indexes\Index.cs:line 1262

   at Raven.Server.Documents.Indexes.Index.ExecuteIndexing() in C:\Builds\RavenDB-4.0-RC\src\Raven.Server\Documents\Indexes\Index.cs:line 791

After several days of attempting to figure out a commonality, we have failed. 

Things tried so far:

1) Tried and failed to create a repeatable test of the conditions. 

2) Made sure we were on .NET Core 2.03 (We think that is the same as RC2)

3) Tried all kinds of weird places where nulls might be causing issue with no luck. 

We are simply not sure how to debug this any further, so looking for some help/guidance.  

--

W

[Oren Eini (Ayende Rahien)]

Do you have a non repeatable test here?

This typically happen when editing / modifying stuff that was already indexed to get something like this.

What does your index look like?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Wade Wright]

Here is the simplest of our indexes that are doing this...

    public class InstitutionalClaimsByBillTypeAndMatchingStatusIndex : AbstractIndexCreationTask<InstitutionalClaim, InstitutionalClaimsByBillTypeAndMatchingStatusIndex.Result>

    {

        public class Result

        {

            public string FacilityId { get; set; }

            public string BillType { get; set; }

            public string BillTypeDescription { get; set; }

            public string MatchStatus { get; set; }

            public int Count { get; set; }

        }

        public InstitutionalClaimsByBillTypeAndMatchingStatusIndex()

        {

            Map = institutionalClaims => from institutionalClaim in institutionalClaims

                                         select new

                                         {

                                             FacilityId = institutionalClaim.Facility.Id,

                                             institutionalClaim.BillType,

                                             institutionalClaim.BillTypeDescription,

                                             MatchStatus = institutionalClaim.MatchingStatus,

                                             Count = 1

                                         };

            Reduce = results => from result in results

                                group result by new { result.FacilityId, result.BillType, result.BillTypeDescription, MatchStatus = result.MatchStatus }

                                into g

                                let count = g.Sum(x => x.Count)

                                select new

                                {

                                    g.Key.FacilityId,

                                    g.Key.BillType,

                                    g.Key.BillTypeDescription,

                                    g.Key.MatchStatus,

                                    Count = count

                                };

        }

    }

For this project, we are importing thousands of claims from X12 format files for healthcare. A fairly simple parse the file, new up an InstitutionalClaim, map the data to the InstitutionalClaim and store it in Raven. During the same import process, but later, we attempt to match up the InstitutionalClaim to an Encounter that may already exist in our DB, and if so, we update the institutionalClaim.MatchingStatus to indicate that matched it up. 

--

W

[ar...@ayende.com]

Hi Wade,

I think the issue is related to specific indexing pattern in particular the order of documents that are processed by the index.

1) Does it apply only to indexes on InstitutionalClaims collection? How many docs is there?
2) Do you see it also after the initial import or does it happen only after some of InstitutionalClaim docs get updated?
3) You mentioned that you see it even after resetting the index. Are there any document updates meanwhile?
4) Can you send us the export of your database?

Regards,
Arek

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

W

[jhutz]

Hi Arek, I work with Wade on this project...

1) We get this exception for indexes using different document types, not just the InstitutionalClaim document.

2) It appears this only happens when the application is in the process of inserting and updating documents. It does look like it is happening some time after a document gets updated. The initial import does not appear to cause the issue.

3) Yes, there were updates running during the time the reset on the index was called. I am able to reset the index only after the update process has stopped. 

4)  The database in question has sensitive data. I will see if I can get you a copy with some test data that can be used to reproduce the issue.

Note: 

There are 1126 documents in the InstitutionalClaim collection when the error index is reported.

A different document set only has 300 records when the index error is reported.

Thanks,

Juls

[ar...@ayende.com]

Hi Juls,

getting the data with similar documents as your production would be great. The things like size of documents, in particular size of fields that that are indexed can be very important to reproduce it. The number of docs you have is quite low - does the error happen often to you?

Can you try to record all of the updates to the docs after the initial import until the error happen. That should create a consistent repro.

A.

[Arkadiusz Palinski]

Hi Juls,

getting the data with similar documents as your production would be great. The things like size of documents, in particular size of fields that that are indexed can be very important to reproduce it. The number of docs you have is quite low - does the error happen often to you?

Can you try to record all of the updates to the docs after the initial import until the error happen. That should create a consistent repro.

A.

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/jc_1ZxY-bXU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[jhutz]

A,

I am going to upload a test solution that I create that will recreate the scenario using a sample of the documents for which the index error occurs.  There appears to be several things that contribute to this scenario:

1. Having Multiple Databases - I am using 2 in the test project

2. Having Multiple Documents Types - I only have 2 in the test project (Claim Document and an Encounter Document)

3. Having Multiple Map Reduce Indexes - 2 in Database A (Claim Database) and 1 in Database B (Encounter Database)

4. Calling SaveChanges() multiple times

5. Size of the documents

6. The number of documents

The test project includes 2 tests... 

1) UnexpectedReduceTreePageExceptionTests

Expect_Index_Not_To_Fail_Using_Multiple_Sessions

Uses a scaled down version of our claim document

2) UnexpectedReduceTreePageExceptionTests_With_Inst_Claim

Expect_Index_Not_To_Fail_Using_Multiple_Sessions

Uses a the full version our claim document

Both tests fail with the MapReduce index error we described...

I've annotated the tests with comments... run the tests and you should see them fail, you may have a wait a few minutes to see the failure (using xunit test runner).

** Uploading solution in separate post **

[jhutz]

Sent test solution to the raven support e-mail under the same subject.

[ar...@ayende.com]

We haven't received anything so far.

[jhutz]

I sent it again from a different e-mail address.

[ar...@ayende.com]

Got it. Thanks.

[ar...@ayende.com]

The fix will be available in the next nightly. Thank you very much for providing the test. I was able to narrow down the problem using it and creating even smaller one which consistently fails immediately.

[jhutz]

Awesome. Thanks for the help.