Indexes keep rebuilding keeps rebuilding

[Brent Mills]

We have a very mysterious problem that is causing very serious issues on our production site. For the last few days the indexes seem to randomly empty themselves and then rebuild.  This means the site effectively goes down for a few minutes at a time.  Any idea what could cause such behaviour? (we are on 3.5)

[Brent Mills]

One other mysterious characteristic is that around the same time as this problem, the database doubled in size.  I initially thought it was due to a code change by one of our team which was essentially duplicating our main index, but even after reverting that (bad) code, the index rebuilding problem persists.

[Arkadiusz Palinski]

Is there anything in the studio - Alerts, Indexing Errors or in the logs?

You can make sure nobody deploys different index definition behind your back by locking the indexes - Locked (Error) so an attempt to change index definition will result in an exception.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brent Mills]

Definitely no different definitions being deployed. The whole db just randomly resets all indexes.  I was getting a lot of errors like this and  in many other auto indexes, although I don't use any of them. For the moment I have locked the 2 crucial indexes so at least the site is not going down every 20 minutes or so.

 

Index

Auto/CommunityDocuments/ByMembers_Id

Error

Index 'Auto/CommunityDocuments/ByMembers_Id' has already produced 16 map results for a source document 'CommunityDocuments/53f3vjbla8t', while the allowed max number of outputs is 15 per one document. Please verify this index definition and consider a re-design of your entities or index.

Timestamp

30 minutes ago (08/29/18, 7:29pm)

Index

Auto/CommunityDocuments/ByMembers_Id

Document

CommunityDocuments/53f3vjbla8t 

[Arkadiusz Palinski]

Does it mean that locking the index fixed the issue?

Regarding the auto index errors you might increase Raven/MaxSimpleIndexOutputsPerDocument setting (https://ravendb.net/docs/article-page/3.5/csharp/server/configuration/configuration-options) although if you are sure you don't need them you'll be better off by removing those indexes.

--

[Brent Mills]

Well locking the 2 indexes has stopped the catastrophe, but it appears that the others are still rebuilding themselves. When I set it to 'lock with error', then after a while the site went down with the error message 'can't modify fixturesindex'.  SO it seems something in the code is trying to change all the indexes definitions which is triggering the reset, but for the life of me I can't think how that might be possible.

 

[Oren Eini (Ayende Rahien)]

Check the full stack of the error?

On Thu, Aug 30, 2018 at 10:51 AM, Brent Mills <bgmi...@gmail.com> wrote:

Well locking the 2 indexes has stopped the catastrophe, but it appears that the others are still rebuilding themselves. When I set it to 'lock with error', then after a while the site went down with the error message 'can't modify fixturesindex'.  SO it seems something in the code is trying to change all the indexes definitions which is triggering the reset, but for the life of me I can't think how that might be possible.

 

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Oren Eini CEO   /   Hibernating Rhinos LTD Mobile:  +972-52-548-6969 Sales:  sa...@ravendb.net Skype:  ayenderahien Support:  sup...@ravendb.net

[Brent Mills]

Attached file.  It looks like it fails on our bootstrapper method in global.asax which should only be running on a new deployment.  I don't follow how that can be triggered without issuing a rebuild/deploy command. (we use appharbor).  

I guess this edging out of the realms of a raven issue and more of an appharbor issue.  However even if this is appharbor's fault, I thought the database would only rebuild an index if the definition has changed? ( they haven't.)

[Oren Eini (Ayende Rahien)]

We have seen this when you have two different instance of the apps, with different versions of hte indexes

On Thu, Aug 30, 2018 at 11:29 AM, Brent Mills <bgmi...@gmail.com> wrote:

Attached file.  It looks like it fails on our bootstrapper method in global.asax which should only be running on a new deployment.  I don't follow how that can be triggered without issuing a rebuild/deploy command. (we use appharbor).  

I guess this edging out of the realms of a raven issue and more of an appharbor issue.  However even if this is appharbor's fault, I thought the database would only rebuild an index if the definition has changed? ( they haven't.)

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brent Mills]

Thanks Oren, this was the problem. One of our developers had his local machine pointing to the live database and his build had a different index definition.  Was a rather stressful time for a while there but thankfully we have gotten to the bottom of it!

[Oren Eini (Ayende Rahien)]

http://issues.hibernatingrhinos.com/issue/RavenDB-11880

We created an issue to make it easy to track next time

On Fri, Aug 31, 2018 at 1:23 PM, Brent Mills <bgmi...@gmail.com> wrote:

Thanks Oren, this was the problem. One of our developers had his local machine pointing to the live database and his build had a different index definition.  Was a rather stressful time for a while there but thankfully we have gotten to the bottom of it!

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.