Query based on Values inside a Dictionary

[Benjamin]

What I want to do is to filter results based on the key and values in
a dictionary. What I have been able to do is to store the key as a
part of the value and store the values as an array instead of a
dictionary. I have simplified my Classes for ease of reading.

This is what works (Array Based):

public class MyArrayStructure
{
public IList<MyArrayValueCollection> DataValues { get; set; }
}
public class MyArrayValueCollection
{
public int Key { get; set; }
public double Value { get; set; }
}

ArrayIndex =
IndexDefinition<MyArrayStructure>
{
Map = structures =>
from structure in structures
from dv in structure.DataValues
select new {Key = dv.Key, Value = dv.Value }
}

session.Advanced.LuceneQuery<MyArrayStructure, ArrayIndex>()
.WhereEquals("Key", 3)
.AndAlso()
.WhereGreaterThanOrEqual("Value", "500000")

This does return all of my MyArrayStructures that has data with a
"Key" of 3 and "Value" greater than or equal to 500,000



This is what I am trying to accomplish (Dictionary Based):

public class MyDictionaryStructure
{
public IDictionary<int, MyDictionaryValueCollection> DataValues
{ get; set; }
}
public class MyDictionaryValueCollection
{
public double Value { get; set; }
}

// This obviously won't work, because in JSON there is no "key"
property, but it gets across the idea of what I'm trying to accomplish
DictionaryIndex =
IndexDefinition<MyDictionaryStructure>
{
Map = structures =>
from structure in structures
from dv in structure.DataValues
select new { Key = dv.Key, Value = dv.Value.Value }
}

session.Advanced.LuceneQuery<MyDictionaryStructure,
DictionaryIndex>()
.WhereEquals("Key", 3)
.AndAlso()
.WhereGreaterThanOrEqual("Value", "500000")

I do not want to mangle my Class (and it's usage in code) to use an
Array of values, when I clearly want to use the data as a dictionary.
My thought was to create some sort of sophisticated Lucene filter for
my Index, but I have not had any success.

I would really appreciate any pointers/advice.

Thank you,

Benjamin

[Ayende Rahien]

Here is how you do it:

[Fact]

public void CanIndexValuesForDictionary()

{

using (var store = NewDocumentStore())

{

using(var s = store.OpenSession())

{

s.Store(new User

        {

        Items = new Dictionary<string, string>

                {

                {"Color", "Red"}

                }

        });

s.SaveChanges();

}

using (var s = store.OpenSession())

{

var users = s.Advanced.LuceneQuery<User>()

.WhereEquals("Items.Color", "Red")

.ToArray();

Assert.NotEmpty(users);

[Benjamin]

Thank you for the response.  I'm running into a problem with the key is an int.  This occurs even if the dictionary is <string, string>.  To be clear what I want is a dictionary of <int, customObject>, but I believe the test below will fail.

[Fact]

public void CanIndexValuesForDictionary()

{

using (var store = NewDocumentStore())

{

using(var s = store.OpenSession())

{

s.Store(new User

        {

        Items = new Dictionary<string, string>

                {

                {"3", "Red"}

                }

        });

s.SaveChanges();

}

using (var s = store.OpenSession())

{

var users = s.Advanced.LuceneQuery<User>()

.WhereEquals("Items.3", "Red")

.ToArray();

Assert.NotEmpty(users);

}

}

}

I may be missing something obvious here, but the error message I get is:

Exception:

{

  "Url": "/indexes/dynamic/UserStrings?query=Items.3%253ARed&start=0&pageSize=128",

  "Error": "System.InvalidOperationException: Could not understand query: \r\n-- line 2 col 32: invalid Expr\r\n\r\n   at Raven.Database.Linq.QueryParsingUtils.GetVariableDeclarationForLinqQuery(String query, Boolean requiresSelectNewAnonymousType)\r\n   at Raven.Database.Linq.DynamicViewCompiler.TransformMapDefinitionFromLinqQuerySyntax(String& entityName)\r\n   at Raven.Database.Linq.DynamicViewCompiler.TransformMapDefinition(String& entityName)\r\n   at Raven.Database.Linq.DynamicViewCompiler.TransformQueryToClass()\r\n   at Raven.Database.Linq.DynamicViewCompiler.GenerateInstance()\r\n   at Raven.Database.Storage.IndexDefinitionStorage.AddAndCompileIndex(String name, IndexDefinition indexDefinition)\r\n   at Raven.Database.Storage.IndexDefinitionStorage.AddIndex(String name, IndexDefinition indexDefinition)\r\n   at Raven.Database.DocumentDatabase.PutIndex(String name, IndexDefinition definition)\r\n   at Raven.Database.Queries.DynamicQueryRunner.CreateIndex(DynamicQueryMapping map, String indexName)\r\n   at Raven.Database.Queries.DynamicQueryRunner.TouchTemporaryIndex(DynamicQueryMapping map, String temporaryIndexName, String permanentIndexName)\r\n   at Raven.Database.Queries.DynamicQueryRunner.FindDynamicIndexName(DynamicQueryMapping map)\r\n   at Raven.Database.Queries.DynamicQueryRunner.ExecuteDynamicQuery(String entityName, IndexQuery query)\r\n   at Raven.Database.Queries.DynamicQueryExtensions.ExecuteDynamicQuery(DocumentDatabase self, String entityName, IndexQuery indexQuery)\r\n   at Raven.Database.Server.Responders.Index.OnGet(IHttpContext context, String index)\r\n   at Raven.Database.Server.Responders.Index.Respond(IHttpContext context)\r\n   at Raven.Http.HttpServer.DispatchRequest(IHttpContext ctx)\r\n   at Raven.Http.HttpServer.HandleActualRequest(IHttpContext ctx)"

}

[Ayende Rahien]

That is because fields should start with _ or letter.

I fixed the issue.

This is how it will work:

var users = s.Advanced.LuceneQuery<User>()

    .WhereEquals("Items._3", "Red")

    .ToArray();

[Benjamin]

I see the build has already completed with the change.

Thank you for such a quick response and solution.

Cheers

[Benjamin]

Because of the modification Ayende made, I was able to extract the data that I wanted.  I am looking for a more elegant (more efficient) solution to my problem.  

I have come up with an arbitrary structure that should model my situation.  The reason that I use a Dictionary in the Recipe class is because the "Ingredients" that are attached to the Recipe, may contain over a hundred entries, but I only need to access one or two on a given request.

Given the following structure:

public class Ingredient

{

public int Id { get; set; }

public string Name { get; set; }

public int AmountInStock { get; set; }

}

public class IngredientInfo

{

public string UnitOfMeasure { get; set; }

public double Quantity { get; set; }

}

public class Recipe

{

public int Id { get; set; }

public string Name { get; set; }

public string Instructions { get; set; }

public IDictionary<int, IngredientInfo> Ingredients { get; set; }

}

I can construct a LuceneQuery:

session.Advanced.LuceneQuery<Recipe>()

.WhereEquals("Ingredients._" + key + ".Quantity", value.ToString())

.ToList();

In my use case, the key and quantity values are chosen by the User at runtime.

Thanks to the dynamic querying abilities it generates a dynamic index on the server used in returning the results from the above query:

from doc in docs.Recipes

select new { Ingredients_2Quantity = doc.Ingredients._2.Quantity }

This functions as you would expect, but since there may be hundreds of different Ingredients and I do not know which ingredients the user of the application will be querying on, it seems inefficient to create a dynamic index for each permutation of parameters the user may want.

What I have been attempting to do is generate a pre-computed Index _like_ the following:

from doc in docs.Recipes

from ingredient in doc.Ingredients

select new {id = ingredient.Key, quantity = ingredient.Value.Quantity}

I know Lucene doesn't know about dictionary's Keys or Values so my thought was that if I could enumerate over the "key" in JSON it would look similar to:

from doc in docs.Recipes

from ingredient in doc.Ingredients

from ingredientInfo in ingredient

select new {ingredientId = ingredient, quantity = ingredientInfo.Quantity}

I _am_ open to changing my document structure if a good alternative can be found.  That being said, in my specific use case, there may be hundreds of IngredientInfo Items in a Recipe.  That is why I want to use a dictionary instead of an array (for the runtime benefit of not having to enumerate over the collection every time i want to access an IngredientInfo item).

Does anyone have any thoughts on how I might accomplish this?

[Benjamin]

Also, it doesn't seem philosophically wrong (from a design standpoint) for me to modify the model a bit to add the key value to the item in the dictionary. In this case it would add the field "IngredientId" to the IngredientInfo class.

public class Ingredient

{

public int Id { get; set; }

public string Name { get; set; }

public int AmountInStock { get; set; }

}

public class IngredientInfo

{

public int IngredientId { get; set; }

public string UnitOfMeasure { get; set; }

public double Quantity { get; set; }

}

public class Recipe

{

public int Id { get; set; }

public string Name { get; set; }

public string Instructions { get; set; }

public IDictionary<int, IngredientInfo> Ingredients { get; set; }

}

This would only change the index to be something similar to: (Note: This does not work, because currently RavenDB does not seem to be able to enumerate over the values on the server)

from doc in docs.Recipes

from ingredient in doc.Ingredients

from ingredientInfo in ingredient

select new {ingredientId = ingredientInfo.IngredientId, quantity = ingredientInfo.Quantity}

The only thing this accomplishes is bypassing the need to reference the "key" from the dictionary directly.

My plan is to poke around in the internals of RavenDB today to see if I can find a way to make this work.  Any suggestions / help / guidance anyone might be able to offer, would be greatly appreciated.

Cheers

[Ayende Rahien]

Will be in the next build

[Ayende Rahien]

To be rather more exact, this will work:

[Fact]

public void CanIndexValuesForDictionaryAsPartOfDictionary()

{

using (var store = NewDocumentStore())

{

using (var s = store.OpenSession())

{

s.Store(new User

{

Items = new Dictionary<string, string>

                {

                {"Color", "Red"}

                }

});

s.SaveChanges();

}

using (var s = store.OpenSession())

{

var users = s.Advanced.LuceneQuery<User>()

.WhereEquals("Items,Key", "Color")

.AndAlso()

.WhereEquals("Items,Value", "Red")

.ToArray();

Assert.NotEmpty(users);

}

}

}

Which generates the following index:

from doc in docs.Users

from docItemsItem in doc.Items

select new { ItemsKey = docItemsItem.Key, ItemsValue = docItemsItem.Value }

[Benjamin]

This will be fantastic.

Thank you again Ayende for such quick and thorough responses and solutions.

[Benjamin]

I have submitted a pull request for my failing test, but I figured I would document it here so others could see.

Given the following structure:

public class UserWithIDictionary

{

public string Id { get; set; }

public IDictionary<string, NestedItem> NestedItems { get; set; }

}

The following test case will fail:

[Fact]

public void CanIndexNestedValuesForIDictionaryAsPartOfIDictionary()

{

using (var store = NewDocumentStore())

{

using (var s = store.OpenSession())

{

s.Store(new UserWithIDictionary

{

NestedItems = new Dictionary<string, NestedItem>

{

        { "Color", new NestedItem{ Name="Red" } }

}

});

s.SaveChanges();

}

using (var s = store.OpenSession())

{

var users = s.Advanced.LuceneQuery<UserWithIDictionary>()

.WhereEquals("NestedItems,Key", "Color")

.AndAlso()

.WhereEquals("NestedItems,Value.Name", "Red")

.ToArray();

Assert.NotEmpty(users);

}

}

}

[Ayende Rahien]

Fixed

[Benjamin]

Was this fixed by upgrading to a newer version of Json.Net or were there changes to RavenDB made? 

If it was a RavenDB change, I'd love to see the commit that fixed it (I didn't see anything specific when looking at the commit list on github).

I'm trying to understand how RavenDB works better so I can actually make contributions in the future.  I'd love to see the resolution.

The commit # would be fine.

Thank you.

[Ayende Rahien]

No, it was fixed in RavenDB.

The actual problem was fixed here:

https://github.com/ayende/ravendb/commit/d5e7ca08445c6bd0167eeca781b41155519a9bbc

Raven.Abstractions/Linq/DynamicJsonObject.cs

• View file @ d5e7ca0

...	...	@@ -24,6 +24,9 @@ namespace Raven.Database.Linq
24	24	{
25	25	foreach (var item in Inner)
26	26	{
	27	+                if(item.Key[0] == '$')
	28	+                    continue;
	29	+
27	30	yield return new KeyValuePair<string,object>(item.Key, TransformToValue(item.Value));
28	31	}
29	32	}

What happened was that the dictionary was serialized with a $type attribute, which obviously doesn't have a Name property.

That was causing the document indexing to fail.