Building index dynamically inside of AbstractMultiMapIndexCreationTask<TReduceResult>

[ronnieoverby]

I'm trying to create an index at app startup using AbstractMultiMapIndexCreationTask<TReduceResult>.

However, the maps are not known at compile time. So, I'm trying to use some reflection to add the maps. Anyway here's my futile attempt: https://gist.github.com/2305762

public class DynamicIndex : AbstractMultiMapIndexCreationTask<DynamicIndex.ReduceResult>
    {
        public class ReduceResult
        {
            public string Value { get; set; }
            public int Count { get; set; }
        }

        public DynamicIndex()
        {
            IEnumerable<PropertyInfo> propertyInfos = GetThePropertyInfos();

            foreach (var property in propertyInfos)
            {
                AddMap<object>(content => content
                    .SelectMany(c => (IEnumerable<string>)c.GetPropertyValue(property.Name))
                    .Select(value => new ReduceResult
                    {
                        Value = value,
                        Count = 1
                    }));

            }

            Reduce = results => from r in results
                                group r by r.Value into g
                                select new ReduceResult
                                {
                                    Value = g.Key,
                                    Count = g.Sum(x => x.Count)
                                };
        }

        private IEnumerable<PropertyInfo> GetThePropertyInfos()
        {
            // implementation omitted
        }
    }

And here's the error I get at runtime:

System.InvalidOperationException was unhandled
Message=Url: "/indexes/CMSTags" System.InvalidOperationException: Source code:
using Raven.Abstractions;
using Raven.Database.Linq;
using System.Linq;
using System.Collections.Generic;
using System.Collections;
using System;
using Raven.Database.Linq.PrivateExtensions;
using Lucene.Net.Documents;
using Raven.Database.Indexing;
public class Index_CMSTags : AbstractViewGenerator
{
public Index_CMSTags()
{
this.ViewText = @"docs
.SelectMany(c => (c.GetPropertyValue(ScratchPad.CMSTags+__c__DisplayClass9.t.Name)))
.Select(x => new () {Tag = x, Count = 1})
results
.GroupBy(r => r.Tag)
.Select(g => new () {Tag = g.Key, Count = g.Sum(x => (System.Int32)(x.Count))}) ";
this.AddMapDefinition(docs => docs.SelectMany((Func<dynamic, IEnumerable<dynamic>>)(c => (IEnumerable<dynamic>)((c.GetPropertyValue(ScratchPad.CMSTags + __c__DisplayClass9.t.Name))))).Select((Func<dynamic, dynamic>)(x => new { Tag = x, Count = 1, __document_id = x.__document_id })));
this.ReduceDefinition = results => results.GroupBy((Func<dynamic, dynamic>)(r => r.Tag)).Select((Func<IGrouping<dynamic,dynamic>, dynamic>)(g => new { Tag = g.Key, Count = g.Sum((Func<dynamic, System.Int32>)(x => (System.Int32)(x.Count))) }));
this.GroupByExtraction = r => r.Tag;
this.AddField("Tag");
this.AddField("Count");
}
} z:\lib\ravendb\701\Server\Tenants\OverbyCmsScratchPad\IndexDefinitions\TemporaryIndexDefinitionsAsSource\jewld2ao.0.cs(22,135) : error CS0103: The name 'ScratchPad' does not exist in the current context
z:\lib\ravendb\701\Server\Tenants\OverbyCmsScratchPad\IndexDefinitions\TemporaryIndexDefinitionsAsSource\jewld2ao.0.cs(22,156) : error CS0103: The name '__c__DisplayClass9' does not exist in the current context at Raven.Database.Linq.QueryParsingUtils.Compile(String source, String name, String queryText, OrderedPartCollection`1 extensions, String basePath) in c:\Builds\RavenDB-Stable\Raven.Database\Linq\QueryParsingUtils.cs:line 253
at Raven.Database.Linq.DynamicViewCompiler.GenerateInstance() in c:\Builds\RavenDB-Stable\Raven.Database\Linq\DynamicViewCompiler.cs:line 491
at Raven.Database.Storage.IndexDefinitionStorage.AddAndCompileIndex(IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\IndexDefinitionStorage.cs:line 149
at Raven.Database.Storage.IndexDefinitionStorage.AddIndex(IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\IndexDefinitionStorage.cs:line 134
at Raven.Database.DocumentDatabase.PutIndex(String name, IndexDefinition definition) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 713
at Raven.Database.Server.Responders.Index.Put(IHttpContext context, String index) in c:\Builds\RavenDB-Stable\Raven.Database\Server\Responders\Index.cs:line 69
at Raven.Database.Server.Responders.Index.Respond(IHttpContext context) in c:\Builds\RavenDB-Stable\Raven.Database\Server\Responders\Index.cs:line 46
at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx) in c:\Builds\RavenDB-Stable\Raven.Database\Server\HttpServer.cs:line 534
at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx) in c:\Builds\RavenDB-Stable\Raven.Database\Server\HttpServer.cs:line 309 Source=Raven.Client.Lightweight
StackTrace:
at Raven.Client.Connection.HttpJsonRequest.HanldeErrors(WebException e) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line 410
at Raven.Client.Connection.HttpJsonRequest.ReadStringInternal(Func`1 getResponse) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line 279
at Raven.Client.Connection.HttpJsonRequest.ReadResponseString() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line 195
at Raven.Client.Connection.HttpJsonRequest.ReadResponseJson() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line 616
at Raven.Client.Connection.ServerClient.DirectPutIndex(String name, String operationUrl, Boolean overwrite, IndexDefinition definition) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:line 641
at Raven.Client.Connection.ServerClient.<>c__DisplayClass34.<PutIndex>b__33(String operationUrl) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:line 613
at Raven.Client.Connection.ServerClient.TryOperation[T](Func`2 operation, String operationUrl, Boolean avoidThrowing, T& result) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:line 201
at Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:line 171
at Raven.Client.Connection.ServerClient.PutIndex(String name, IndexDefinition definition, Boolean overwrite) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\ServerClient.cs:line 613
at Raven.Client.Indexes.AbstractIndexCreationTask.Execute(IDatabaseCommands databaseCommands, DocumentConvention documentConvention) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Indexes\AbstractIndexCreationTask.cs:line 114
at Raven.Client.Indexes.IndexCreation.CreateIndexes(ExportProvider catalogToGetnIndexingTasksFrom, IDatabaseCommands databaseCommands, DocumentConvention conventions) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Indexes\IndexCreation.cs:line 48
at Raven.Client.Indexes.IndexCreation.CreateIndexes(ExportProvider catalogToGetnIndexingTasksFrom, IDocumentStore documentStore) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Indexes\IndexCreation.cs:line 59
at Raven.Client.Indexes.IndexCreation.CreateIndexes(Assembly assemblyToScanForIndexingTasks, IDocumentStore documentStore) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Indexes\IndexCreation.cs:line 36
at ScratchPad.Program.SetupRaven() in Z:\Code\overby.cms\ScratchPad\Program.cs:line 157
at ScratchPad.Program.Main(String[] args) in Z:\Code\overby.cms\ScratchPad\Program.cs:line 25
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException:

I hope I've made it clear what I'm trying to do. Thanks for any help.

[Oren Eini (Ayende Rahien)]

We are parsing the expression to figure out how to build it, we don't actually execute this code.

If you want to dynamically build the maps, override the CreateIndexDefinition and generate them by directly calling the indexDef.Maps.Add(string);

[ronnieoverby]

Ok that makes sense, I knew the code was more of definition than something that would be executed, but didn't know where else to start. I'll try the override. Thanks!

[ronnieoverby]

Thanks, Oren, that worked! 

I'm surprised it worked though and here's why:

docs.BlogPost

.SelectMany(x => x.Tags, (x, t) => new () {Tag = t, Count = 1})

That's a map that got created. I wasn't expecting this to work because I used the type name as is and not pluralized. Why did it work?

On Wednesday, April 4, 2012 5:27:02 PM UTC-4, Oren Eini wrote:

We are parsing the expression to figure out how to build it, we don't actually execute this code.

If you want to dynamically build the maps, override the CreateIndexDefinition and generate them by directly calling the indexDef.Maps.Add(string);

[ronnieoverby]

Also, I'd like to supply those PropertyInfo objects to the AbstractIndexCreationTask through a constructor if possible. Is there a way to do this?

On Wednesday, April 4, 2012 5:27:02 PM UTC-4, Oren Eini wrote:

We are parsing the expression to figure out how to build it, we don't actually execute this code.

If you want to dynamically build the maps, override the CreateIndexDefinition and generate them by directly calling the indexDef.Maps.Add(string);

On Thu, Apr 5, 2012 at 12:24 AM, ronnieoverby worte:

[ronnieoverby]

Nevermind, I see that I need to create the IndexCreationTask my self and call execute on it.

[Oren Eini (Ayende Rahien)]

How do you do this?

[ronnieoverby]

[ronnieoverby]

Like this:

    public class DynamicIndex : AbstractIndexCreationTask

    {

        private PropertyInfo[] _propertyInfos;

        public DynamicIndex(PropertyInfo[] propertyInfos)

        {

            if (propertyInfos == null || propertyInfos.Length == 0)

                throw new ArgumentException("tagsProperties is null or empty.", "tagsProperties");

            _propertyInfos = propertyInfos;

        }

        public override IndexDefinition CreateIndexDefinition()

        {

            var maps = new HashSet<string>();

            foreach (var property in _propertyInfos)

            {

                maps.Add(string.Format(@"docs.{0}.SelectMany(x => x.{1}, (x, t) => new () {{Tag = t, Count = 1}})",

                    property.DeclaringType.Name,

                    property.Name));

            }

            return new IndexDefinition

            {

                Maps = maps,

                Reduce = "results.GroupBy(r => r.Tag).Select(g => new () {Tag = g.Key, Count = g.Sum(x => (System.Int32)(x.Count))})"

            };

[Oren Eini (Ayende Rahien)]

Hm, that shouldn't work unless you modified you conventions to not use the pluralization

[ronnieoverby]

Yes, that's why I was surprised. It does certainly work though. How do you guys do pluralization, so I can make it match up?

[Oren Eini (Ayende Rahien)]

that happens in FindTypeTagName in the conventions.

[ronnieoverby]

I used DocumentConvention.DefaultTypeTagName(property.DeclaringType) and that seems to be doing the trick. Thanks!