Production issue w/ build 2360 - Index results growing out of control

[Jason Dentler]

I have a multi-map/reduce index that has far too many results, which is throwing off the counts of the facet index it supports. This issue was not discovered in testing and has now made it out to production. It began with our upgrade this morning to build 2360.

We have 270 community documents. This index groups by community id. Therefore, we should not have more than 270 entries in this index. At the moment, we have 16,477 results. It's grown by 4000 in the time it's taken me to write this email. I don't know what to check. What else do you need to help diagnose this issue?

- Jason Dentler

using System;
using System.Linq;
using DWH.Website.ViewModels;
using Raven.Abstractions.Indexing;
using Raven.Client.Indexes;

namespace DWH.Website.Indices.Search
{
    public class TransformResult
    {
        public Community Community { get; set; }

        public FloorPlan FloorPlan { get; set; }

        public Showcase Showcase { get; set; }
    }
    public class MapResult : TransformResult
    {
        public string Id { get; set; }
        public string MarketId { get; set; }
        public string CityId { get; set; }
        public Community.Geolocations Geolocation { get; set; }
        public string[] SchoolDistricts { get; set; }
        public string[] Amenities { get; set; }

        public string Name { get; set; }
        public int MinPrice { get; set; }
        public int MaxPrice { get; set; }
        public int MinSquareFootage { get; set; }
        public int MaxSquareFootage { get; set; }
        public int MinBedrooms { get; set; }
        public int MinBathrooms { get; set; }
        public bool? MasterBedroomUpstairs { get; set; }
        public double MinStories { get; set; }
        public double MinGarages { get; set; }
        public bool InteractiveFloorPlan { get; set; }
        public bool VirtualTour { get; set; }
        public bool CloseOutCommunity { get; set; }

        public string CommunityName { get; set; }
        public string CommunityGrouping { get; set; }
        public string CommunityId { get; set; }
        public Community.CommunityTypes CommunityType { get; set; }

    }

    public class CommunitySearchIndex : AbstractMultiMapIndexCreationTask<MapResult>

    {
        public const string TheIndexName = "Communities/Search";

        public override string IndexName
        {
            get { return TheIndexName; }
        }


        public CommunitySearchIndex()
        {

            AddMap<FloorPlan>(floorPlans =>
                              from floorPlan in floorPlans
                              let community = LoadDocument<Community>(floorPlan.Community.Id)
                              where MetadataFor(community).Value<string>("Raven-Entity-Name") == "Communities"
                              let schoolDistricts = community.SchoolIds.Select(id => LoadDocument<School>(id).SchoolDistrictName).ToArray()
                              select new MapResult
                                  {
                                      Id = floorPlan.Id,
                                      MarketId = floorPlan.Community.City.Market.Id,
                                      CityId = floorPlan.Community.City.Id,
                                      Geolocation = floorPlan.Community.Geolocation,
                                      SchoolDistricts = schoolDistricts,
                                      Amenities = community.Amenities,
                                      MinPrice = floorPlan.BasePrice,
                                      MaxPrice = floorPlan.BasePrice,
                                      MinSquareFootage = floorPlan.SquareFootage.Maximum,
                                      MaxSquareFootage = floorPlan.SquareFootage.Minimum,
                                      MinBedrooms = floorPlan.Bedrooms.Maximum,
                                      MinBathrooms = floorPlan.FullBaths.Maximum,
                                      MasterBedroomUpstairs = floorPlan.MasterBedroomUpstairs,
                                      MinStories = floorPlan.Stories.Maximum,
                                      MinGarages = floorPlan.Garages.Maximum,
                                      InteractiveFloorPlan = floorPlan.InteractiveFloorPlan,
                                      VirtualTour = floorPlan.VirtualTour != null,
                                      CloseOutCommunity = community.CommunityStatus == Community.CommunityStatuses.CloseOut,
                                      Community = community,
                                      CommunityId = community.Id,
                                      CommunityName = community.Name,
                                      CommunityType = community.CommunityType
                                  }
                              );

            AddMap<Showcase>(showcases =>
                              from showcase in showcases
                              let community = LoadDocument<Community>(showcase.Community.Id)
                              where MetadataFor(community).Value<string>("Raven-Entity-Name") == "Communities"
                              let schoolDistricts = community.SchoolIds.Select(id => LoadDocument<School>(id).SchoolDistrictName).ToArray()
                              select new MapResult
                                  {
                                      Id = showcase.Id,
                                      MarketId = showcase.Community.City.Market.Id,
                                      CityId = showcase.Community.City.Id,
                                      Geolocation = showcase.Community.Geolocation,
                                      SchoolDistricts = schoolDistricts,
                                      Amenities = community.Amenities,
                                      MinPrice = showcase.Price,
                                      MaxPrice = showcase.Price,
                                      MinSquareFootage = showcase.SquareFootage,
                                      MaxSquareFootage = showcase.SquareFootage,
                                      MinBedrooms = showcase.Bedrooms,
                                      MinBathrooms = showcase.FullBaths,
                                      MasterBedroomUpstairs = showcase.MasterBedroomUpstairs,
                                      MinStories = showcase.Stories,
                                      MinGarages = showcase.Garages,
                                      InteractiveFloorPlan = showcase.InteractiveFloorPlan,
                                      VirtualTour = showcase.VirtualTour != null,
                                      CloseOutCommunity = community.CommunityStatus == Community.CommunityStatuses.CloseOut,
                                      Community = community,
                                      CommunityId = community.Id,
                                      CommunityName = community.Name,
                                      CommunityType = community.CommunityType
                                  }
                              );

            Reduce = results =>
                     from result in results
                     group result by result.CommunityId
                     into g
                     let firstResult = g.First()
                     select new 
                         {
                             Id = firstResult.CommunityId,
                             MarketId = firstResult.Community.City.Market.Id,
                             CityId = firstResult.Community.City.Id,
                             Geolocation = firstResult.Community.Geolocation,
                             SchoolDistricts = firstResult.SchoolDistricts,
                             Amenities = firstResult.Amenities,
                             MinPrice = g.Max(r => r.MinPrice),
                             MaxPrice = g.Min(r => r.MaxPrice),
                             MinSquareFootage = g.Max(r => r.MinSquareFootage),
                             MaxSquareFootage = g.Min(r => r.MaxSquareFootage),
                             MinBedrooms = g.Max(r => r.MinBedrooms),
                             MinBathrooms = g.Max(r => r.MinBathrooms),
                             MasterBedroomUpstairs = g.Select(r => r.MasterBedroomUpstairs),
                             MinStories = g.Max(r => r.MinStories),
                             MinGarages = g.Max(r => r.MinGarages),
                             InteractiveFloorPlan = g.Any(r => r.InteractiveFloorPlan),
                             VirtualTour = g.Any(r => r.VirtualTour),
                             CloseOutCommunity = firstResult.CloseOutCommunity,
                             Community = firstResult.Community,
                             CommunityId = firstResult.Community.Id,
                             CommunityName = firstResult.Community.Name,
                             CommunityType = firstResult.Community.CommunityType
                         };


            Index(r => r.Amenities, FieldIndexing.NotAnalyzed);
            Index(r => r.Geolocation, FieldIndexing.NotAnalyzed);
            Index(r => r.SchoolDistricts, FieldIndexing.NotAnalyzed);

        }



    }
}

[Kijana Woodard]

Wow. And the only change was the upgrade to 2360? 

I'm still on 2330 and was contemplating the upgrade (still dev only).

At a glance, I don't see anything odd in your m/r.

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jason Dentler]

Yep.

[Oren Eini (Ayende Rahien)]

That REALLY shouldn't be happening.

The only thing that I can think of is maybe you have different casing? But that doesn't make sense.
Can you send us a way to reproduce this?

[Jason Dentler]

I'll work on a specific failing test, but I'm a little foggy right now. It's 2 am here.

In the mean time, I can zip up the whole solution and send it over. Would that be helpful?

- J

[Oren Eini (Ayende Rahien)]

That would probably be too big

[Jason Dentler]

On session.Query<T>(string indexName, bool isMapReduce = false), what does isMapReduce? Is that new?

[Jason Dentler]

I meant "What does isMapReduce do?" 

[Oren Eini (Ayende Rahien)]

It determines whatever Id would be translated to __document_id

[Jason Dentler]

Here's a github repo with a single failing test.

https://github.com/jasondentler/RavenBugHunt/blob/master/DWH.Raven.Tests/FailingTest.cs

[Jason Dentler]

I really doubt it matters, but I did enable the Expiration bundle when I created the db.

[Oren Eini (Ayende Rahien)]

Okay, the problem is here:

 Reduce = results =>
          from result in results
          group result by result.CommunityId
              into g
              let firstResult = g.First()
              select new
              {
                  Id = firstResult.CommunityId,

                  Geolocation = firstResult.Community.Geolocation,
                  SchoolDistricts = firstResult.SchoolDistricts,

                  Community = firstResult.Community,
                  CommunityId = firstResult.Community.Id,
                  CommunityName = firstResult.Community.Name,
                  CommunityType = firstResult.Community.CommunityType
              };

Change that to:

                  CommunityId = firstResult.CommunityId,

And it works.

The reason for the problem is that you are storing the entire Community document in the reduce, and that is not something the we expect.
Moreover, we resolve Id to document id only on actual document, not on reduce results, so that we got null, there, and you had your issue.

[Oren Eini (Ayende Rahien)]

In general, drop the Community property from there, it is quite expensive and you can include it much more cheaply

[Jason Dentler]

Thanks Oren. We're testing this now. I'll let you know how it turns out.

- J

[Chris Marisic]

Is there any way the index could detect this scenario and error instead of returning incomprehensible results?

[Oren Eini (Ayende Rahien)]

I am not sure why it didn't handle it nicely. I _think_ that it has to do with the different way we represent null.
I'll be checking on this.

[Jason Dentler]

Oren,

When we need some professional-level support ASAP, how do we reach your team during the day (in the US)?

I think we have things under control now, but management is losing faith in your product, and I'd rather not use NHibernate or EF for the next 5 years because you sleep. :)

- J

[Oren Eini (Ayende Rahien)]

And, to be fair, from raising the issue to solution, it took about 18 hours

[Kijana Woodard]

@Jason, you mentioned that this issue wasn't caught in testing which means you did make an attempt to verify the upgrade before pushing to production.

I think we could all learn something here.

In hindsight, do you see any way you could have detected the problem in dev/qa before the prod upgrade?

[Jason Dentler]

@Oren - You're right. In fact, from the moment you first responded to the time it was resolved was less than 8 hours. This is the 2nd time I've bragged to my boss about how we could never get an equivalent response from Microsoft. Still, you are a small mostly-foreign company compared to a massive international corporation, so perception is not in your favor.

Every outage has to be explained. When I say free public Google Group support, the obvious next question will be about more reliable & immediate professional support options. I just need to have an answer ready, and you've given me one. It may or may not be worth it, but that's not my call.

@Kijana - We got backed in to a quick & dirty upgrade by problems with the old nightly build. It was solid for nearly 6 months - 2 of those in production, then started locking up at really inconvenient times.

We only tested the new version enough to prove it was functional. If we had the same problem, at least "the Internets" would be sympathetic because we were on the latest stable version. If the upgrade brought on different problems, they had to be minor in comparison. It was the right call.

It's tougher now when I say this little open-source database is the right tool for our next app while my great grand-boss still struggles to admit MS SQL Server is "better" than IBM DB2.

- J

[Kijana Woodard]

Jason, that sounds like a very reasonable approach to the situation. In my experience, when projects have problems, people search for something to blame. A "new db technology" is an obvious target.

 

I'm trying to figure out a set of metrics I could watch when upgrading / making code changes. This one was interesting, though I'm sure not so interesting for you at 3 am. :-D

[Jason Dentler]

We just had the same problem again on the latest stable build. The index was growing out of control, but when I checked the index definition, it was still using the correct CommunityId property in the reduce.

I recycled our website's app pool, which runs IndexCreation.CreateIndexes. After doing that, the index has the correct results again. We've turned on debug logging to troubleshoot the timeout issues, but the log is several GB, so I won't bother to upload unless you need it.

Ideas?

- J

[Oren Eini (Ayende Rahien)]

I am not sure, is there a way that something else (like a debug session) created the old index def?

[Jason Dentler]

That's why I checked the index def before recycling. It was still current.

[Jason Dentler]

It acts like every write to a floor plan or showcase is triggering the map to duplicate entries, and the reduce isn't running at all.

- J

[Oren Eini (Ayende Rahien)]

Any way we can pair on this on Skype?

[Jason Dentler]

Not Skype, but yes. I'm sure we can find something to bust through our firewall. Give me 2 hours to get in to the office.

It's intermittent.

[Jason Dentler]

Oren,

Shoot me a message when you can work on this. I'm in the office now. If you prefer, we can work on this tonight (tomorrow morning for you.), but I'll need to set a wake-up alarm.

- J

[Oren Eini (Ayende Rahien)]

I am here now

[Jason Dentler]

Getting settled & booting up now

[Oren Eini (Ayende Rahien)]

I am ayenderahien on skype

[Jason Dentler]

Thanks again for your help Oren. Since you had a look, the system has behaved perfectly without changing a thing. We're going to implement your suggestions one at a time, slowly over the next few weeks.

- J

[Jason Dentler]

Oren,

We did have a crash Thursday afternoon. Raven stopped responding to all requests. I was out of the office and the rest of the team didn't know to capture a dump of the process. I do have the debug log, if that would be helpful.

- J

[Oren Eini (Ayende Rahien)]

Please send it to us

Will analyze it first thing tomorrow 

On Sunday, June 9, 2013, Jason Dentler wrote:

Oren,

[Jason Dentler]

sent a link to the zipped file to aye...@ayende.com

Thanks!

[Oren Eini (Ayende Rahien)]

Jason,

can you ping me on skype?

[Jason Dentler]

Hi Oren,

Sorry I didn't get back to you. I'm going to hand this issue over to our web team's programming manager. His name is Matthew Roberts and he'll be following up with you directly.

- J