Query collection inside document in complex index

[Everett Muniz]

Build 322

I have to query across documents so I'm using AbstractIndexCreationTask<JObject, DesignSearchResult> as the base for my index creation task.  I need to return a property from the items of a collection contained by one of the documents and I'm running into trouble doing that starting from JObject.

I originally tried using the linq methods supported by JObject but something falls apart in the translation from client to server.  On the server the object being used is Raven.Database.Linq.DynamicJsonObject.DynamicList and it doesn't have/support the Json linq methods.

The index code follows with comments inline.

    public class Designs_Search : AbstractIndexCreationTask<JObject, DesignSearchResult>

    {

        public Designs_Search()

        {

            Map = docs => docs

                .WhereEntityIs<JObject>("Forms", "Designs")

                .Select(item => new

                {

                    Form = item["@metadata"]["Raven-Entity-Name"].Equals("Forms") ? item : null,

                    Design = item["@metadata"]["Raven-Entity-Name"].Equals("Designs") ? item : null,

                })

                .Select(item => new

                {

                    Form = item.Form,

                    Design = item.Design,

                    FormId = item.Form != null

                        ? item.Form["Id"]

                        : item.Design["FormId"]

                })

                .Select(item => new

                {

                    FormId = item.FormId,

                    FormName = item.Form != null ? item.Form["Name"] : null,

                    DesignId = item.Design != null ? item.Design["Id"] : null,

                    DesignName = item.Design != null ? item.Design["Name"] : null,

                    DesignDescription = item.Design != null ? item.Design["Description"] : null,

                    DesignPublishedDescription = item.Design != null ? item.Design["PublishedDescription"] : null,                    

                    DesignBarcodeSymbologies = item.Design != null

                        ? item.Design["Shapes"]

                        : null,

                    /*

                     * Shapes is a list of items and I need the value of the Symbology  

                     * property from every item in the list that has one.

                     * 

                     * I've tried item.Design["Shapes"].Children()["Symbology"] - the 

                     * index is created but this throws an error when indexing occurs 

                     * because Raven.Database.Linq.DynamicJsonObject.DynamicList 

                     * doesn't know about the Children() method.

                     * 

                     * I've tried item.Design["Shapes"].Select(...) - the index isn't

                     * created due to an exception: "Cannot use a lambda expression as an argument 

                     * to a dynamically dispatched operation without first casting it to a 

                     * delegate or expression tree type"

                    */

                });

            Indexes.Add(x => x.FormId, FieldIndexing.Analyzed);

            Indexes.Add(x => x.FormName, FieldIndexing.Analyzed);

            Indexes.Add(x => x.DesignId, FieldIndexing.Analyzed);

            Indexes.Add(x => x.DesignName, FieldIndexing.Analyzed);

            Indexes.Add(x => x.DesignDescription, FieldIndexing.Analyzed);

            Indexes.Add(x => x.DesignPublishedDescription, FieldIndexing.Analyzed);

            Indexes.Add(x => x.DesignBarcodeSymbologies, FieldIndexing.Analyzed);

        }

    }

[Ayende Rahien]

This wouldn't work.

You can't create an index on JObject, it is an internal type that RavenDB uses.

What exactly are you trying to do? Not the code, the meaning?

[Everett Muniz]

:-) That's confusing because I already have an index defined this way and it works great.  It's the one you helped me with spanning Accounts, Users and Designs.  How should it be defined?

I'm searching for designs by characteristics of the shapes contained in the design and/or of the form associated with the design.  Forms and Designs are documents and a Design must refer to 1 form.

[Everett Muniz]

I should have said I'm trying to create an index to allow me to search for designs by characteristics of the shapes contained in them and/or of the form they're associated with.  Forms and Designs are documents and a Design must refer to 1 form.  Designs have a collection of shapes whose available properties differ.

[Everett Muniz]

I know I can use the non-generic AbstractIndexCreationTask and just write the LINQ queries for mapping/reducing as strings.  Is that the recommended approach in a situation like this?  I was trying to avoid that but if using the generic version of AbstractIndexCreationTask is just complicating things then I'll punt and use strings.

[Ayende Rahien]

What is your model?

What should the index look like eventually?

[Everett Muniz]

-- Model --

Design

{

    Name:"Cool Design",

    FormId:"forms/123",

    ColorSchemes: 

    [

        {

            Name:"Color Scheme 1", Entries:[{}]

        }

    ],

    Shapes: 

    [

        {

            Symbology:"3of9"

        }

    ]

}

Form

{

    Name:"test form"

}

-- Index --

{

    FormId:"",

    FormName:"",

    DesignId:"",

    DesignName:"",

    DesignBarcodeSymbology: {"", "", ""}

    DesignColorSchemeName: {"", "", ""}

[Ayende Rahien]

Um, no, I meant, what should the index definition looks like (the linq query), not the indexed result.

[Everett Muniz]

The following map and reduce queries seem to produce what I want.  The reduce was tough to get right.  The results in the index should equal the number of designs and each design should include a FormName and that's what I'm seeing.  Right now I'm storing the map and reduce seen below in text files that are included as resources and then the AbstractIndexCreationTask reads the resources and assigns the contents to the Map and Reduce properties of the IndexDefinition.  Is there a better way?

# Map #

from doc in docs.WhereEntityIs("Forms", "Designs")

let form = doc["@metadata"]["Raven-Entity-Name"] == "Forms" ? doc : null

let design = doc["@metadata"]["Raven-Entity-Name"] == "Designs" ? doc : null

select new 

{

    FormId = form != null ? form.Id : design.FormId,

    FormName = form != null ? form.Name : null,

    DesignId = design != null ? design.Id : null,

    DesignName = design != null ? design.Name : null,

    DesignLastSavedDate = design != null ? design.LastSavedDate : null,

    DesignDescription = design != null ? design.Description : null,

    DesignPublishedDescription = design != null ? design.PublishedDescription : null,

    DesignBarcodeSymbology = design != null ? design.Shapes.Select((Func<dynamic,dynamic>)(x => x.Symbology)) : null,

    DesignColorSchemeName = design != null ? design.ColorSchemes.Select((Func<dynamic,dynamic>)(x => x.Name)) : null,

    DesignOriginalColorSchemeName = design != null ? design.ColorSchemes.Select((Func<dynamic,dynamic>)(x => x.OriginalName)) : null

}

# Reduce #

from result in results

group result by result.FormId into g

let formId = g.Key

from design in g

where design.DesignId != null

select new 

{

    FormId = formId,

    FormName = g.Where(x => x.FormName != null).Select(x => x.FormName).FirstOrDefault(),

    design.DesignId,

    design.DesignName,

    design.DesignLastSavedDate,

    design.DesignDescription,

    design.DesignPublishedDescription,

    design.DesignBarcodeSymbology,

    design.DesignColorSchemeName,

    design.DesignOriginalColorSchemeName,

})

[Ayende Rahien]

You are basically doing a join between Forms & Design, right?

The answer is that right now there isn't a better option, but there will be in the future.

[Everett Muniz]

Yes, it's functionally a join on Form and Design.  Thanks.

[Dan]

Hello Ayende,

Do you have any idea on a time frame for this feature?

The situation we have is two different websites sharing a raven data
source. In the admin site, the sensible choices for aggregate roots
(docs) can be fairly small. On the public site however, the logical
choice of aggregate root for the read model is denormalised tuples of
the admin AR's. I was under the impression that I would be able to
build these tuples through creating indexes, but this doesn't seem to
be the case? It doesn't matter if the index takes a while to catch
up, I just want to be able to configure raven, upon insert/update to
update a persistent index in the background whenever it detects one of
its tuples has changed. Essentially I would love a TransformResults
that is executed on the server at insert time and stored. Is this
what's ultimately planned?

On May 2, 5:03 pm, Ayende Rahien <aye...@ayende.com> wrote:
> You are basically doing a join between Forms & Design, right?
> The answer is that right now there isn't a better option, but there will be
> in the future.
>

> >> On Sat, Apr 30, 2011 at 3:57 AM, Everett Muniz <everettmu...@gmail.com>wrote:
>
> >>> -- Model --
> >>> Design
> >>> {
> >>>     Name:"Cool Design",
> >>>     FormId:"forms/123",
> >>>     ColorSchemes:
> >>>     [
> >>>         {
> >>>             Name:"Color Scheme 1", Entries:[{}]
> >>>         }
> >>>     ],
> >>>     Shapes:
> >>>     [
> >>>         {
> >>>             Symbology:"3of9"
> >>>         }
> >>>     ]
> >>> }
>
> >>> Form
> >>> {
> >>>     Name:"test form"
> >>> }
>
> >>> -- Index --
> >>> {
> >>>     FormId:"",
> >>>     FormName:"",
> >>>     DesignId:"",
> >>>     DesignName:"",
> >>>     DesignBarcodeSymbology: {"", "", ""}
> >>>     DesignColorSchemeName: {"", "", ""}
> >>> }
>
> >>> On Fri, Apr 29, 2011 at 6:52 PM, Ayende Rahien <aye...@ayende.com>wrote:
>
> >>>> What is your model?
> >>>> What should the index look like eventually?
>

[Ayende Rahien]

Dan,

What you are describing is basically a put trigger, isn't it?

why can't you build the tuples via indexes?

[fschwiet]

Do I understand correctly that the admin creates Form objects and
Design objects separately?

What I don't understand is by what criteria they are joined when the
user looks at them together.

In your example, the Map step basically transform either a Form or
Design into an object with the same fields. (formId, formName,
designId, designName, ...)
So a form would become something like ("FormID1", "FormName1", null,
null, etc.) and a design object would become (null, null, "designID1",
"designName1", etc).

Then in the reduce, you are grouping on the form ID. This will
create a separate group for each form that exists, since each form has
a unique ID.
This grouping will have a single group for all design objects, since
they all have a form id of NULL.
So to me, it seems your reduce hasn't really joined any forms with
designs.

Did I miss something?

> > >>>>>>> I have toqueryacross documents so I'm

> > >>>>>>> using AbstractIndexCreationTask<JObject, DesignSearchResult> as the base for
> > >>>>>>> my index creation task.  I need to return a property from the items of a

> > >>>>>>>collectioncontained by one of the documents and I'm running into trouble

> ...
>
> read more »

[Everett Muniz]

Hi fschwiet 

good question, the critical line in the reduce is highlighted with ascii arrows :-).  We aggregate by formid but then we return reduce results by each item in the aggregate.  Since a design can only ever have 1 form we account for all designs this way and we get the benefit of form name in the index.  we use this for search but it could also be another way to get a denormalized reference (i say that by way of explanation not recommendation)

from result in results
group result by result.FormId into g
let formId = g.Key

-->>from design in g <<--

where design.DesignId != null
select new 
{
    FormId = formId,
    FormName = g.Where(x => x.FormName != null).Select(x => x.FormName).FirstOrDefault(),
    design.DesignId,
    design.DesignName,
    design.DesignLastSavedDate,
    design.DesignDescription,
    design.DesignPublishedDescription,
    design.DesignBarcodeSymbology,
    design.DesignColorSchemeName,
    design.DesignOriginalColorSchemeName,
})