RavenDB 4.0 - having trouble doing nested includes with load results

[Ciel]

In RavenDB 3.5, I could use a Transformer and call Include<T>() on the result of a Load<T> in order to include nested data, but I'm having a difficult time figuring out how to reproduce this behavior in RavenDB 4.0. Is it even possible?

But in RavenDB 4.0, I can't get to an Include<T> after I use RavenQuery.Load<T>. The goal is to optimize the request so that subsequent requests won't be as expensive.

[Ciel]

I did try just nesting the actual RavenQuery inside the Include, like this - but I got a strange error that it didn't know how to deal with that.

  System.InvalidOperationException : No idea how to deal with convert Load(n.Pages.get_Item("Endowments")) to a member expression

[Oren Eini (Ayende Rahien)]

Can you show the document directly?

It is probably something that will need to use a raw query.

Easiest to test this out in the studio

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ciel]

What do you mean by "can you show the document directly?" I'm afraid I do not understand. Is it possible to do LINQ expressions in a RawQuery like that, as well?

[Grisha Kotler]

How does your document looks like?

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

On 30 November 2017 at 11:45, Ciel <stacey.ci...@gmail.com> wrote:

What do you mean by "can you show the document directly?" I'm afraid I do not understand. Is it possible to do LINQ expressions in a RawQuery like that, as well?

--

[Ciel]

Oh, sorry. It looks like this. The Pages is a dictionary of List<string>.

{
 Id: string,
 Owner: string,
 Name: string,
 Pages: {
    Endowments: string[],
    Allocations: string[]
 }
}

[Ciel]

And then the loaded documents have a shape that looks like this;

{
 Id: string,
 Name: string,
 Chosen: [
    {
       Id: string,
       Name: string,
       // ...
    }
 ]
}

[Ciel]

My biggest problem is that I don't even know where to begin for the LINQ queries in RQL.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Can you do the following?

Create a sample database with the documents structure in 4.live-test.ravendb.net

And show the sample output you want?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Thu, Nov 30, 2017 at 5:01 PM, Ciel <stacey.ci...@gmail.com> wrote:

My biggest problem is that I don't even know where to begin for the LINQ queries in RQL.

On Thu, Nov 30, 2017, 7:27 AM Ciel <stacey.ci...@gmail.com> wrote:

And then the loaded documents have a shape that looks like this;

{
 Id: string,
 Name: string,
 Chosen: [
    {
       Id: string,
       Name: string,
       // ...
    }
 ]
}

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ciel]

Certainly.

On Thu, Nov 30, 2017, 10:42 AM Oren Eini (Ayende Rahien) <aye...@ayende.com> wrote:

Can you do the following?

Create a sample database with the documents structure in 4.live-test.ravendb.net

And show the sample output you want?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Thu, Nov 30, 2017 at 5:01 PM, Ciel <stacey.ci...@gmail.com> wrote:

My biggest problem is that I don't even know where to begin for the LINQ queries in RQL.

On Thu, Nov 30, 2017, 7:27 AM Ciel <stacey.ci...@gmail.com> wrote:

And then the loaded documents have a shape that looks like this;

{
 Id: string,
 Name: string,
 Chosen: [
    {
       Id: string,
       Name: string,
       // ...
    }
 ]
}

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Ciel]

Alright, I created the sample database and filled it with some random data. Essentially, I'm wanting to get something like this out of it by querying a single Notebook. (http://4.live-test.ravendb.net/studio/index.html#databases/documents?&database=stacey.cielia.lynn%40gmail.com)

{
  "Id": "characters/1/notebooks/1",
    "Name": "CHARACTER_A",
    "Endowments": [
        "endowments/66",
        "endowments/4",
        "endowments/16",
        "endowments/7",
        "endowments/45",
        "endowments/16",
        "endowments/32"
    ]
}

However, I want to use the Include feature on the Ids, since they'll have to be loaded in the near future for what is happening.

[Ciel]

Normally, the data in those documents isn't needed, and there's a lot of it, so I opted not to denormalize much of it. There will be other kinds of pages in the dictionary on the notebook, so I can't just save them as a big string array because there has to be a way to infer the specific type at some point.

--

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Ryan Heath]

I got something like this:

declare function Ids(e) { 

    ids = []; 

    for(var i = 0; i < e.length; i++)

    for(var j = 0; j < e[i].Chosen.length; j++)

    { 

        var c = e[i].Chosen[j];

        ids.push(c.Id); 

    }

    return ids; 

}

from notebooks as n

load n.Pages.Endowments as e[]

select n.Id, n.Name, Ids(e) as Endowments

I tried various syntax but using a javascript function was working for me.

// Ryan

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Oren Eini (Ayende Rahien)]

The query you'll want to use is here:

from Notebooks as n

load n.Pages.Endowments as e[]

select  n.Name, e.Chosen.Id as Endowments

include Endowments

I had to do a small fixup to get this working, so please wait for the build on Sunday for this.

In the meantime, you can use this:

declare function output(n) {

    var endowments = [];

    for(var i = 0; i< n.Pages.Endowments.length; i++) {

        var e = load(n.Pages.Endowments[i]);

        for(var j = 0; j < e.Chosen.length; j ++) {

            endowments.push(e.Chosen[j].Id);

            include(e.Chosen[j].Id);

        }

    }

    return {Name: n.Name, Endowments: endowments };

}

from Notebooks as n

select output(n)

You can invoke this using session.Advanced.RawQuery from the C# client.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ciel]

Thank you both so much! I had no idea you could use pure Javascript in RQL!

So can you have multiple selects? For example, there will be other collections I eventually want to include in the same way shortly, in the same query, but don't necessarily want to have them in the final model. The purpose of including them is just to make subsequent pieces of the code more efficient.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Yes, you can call `include()` on values that aren't in the select if you so wish

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ciel]

I'm a bit confused, is it not possible to load the Endowments and put them in the include without being part of the select? For the example I gave you, I had them in the final output, but realistically they wouldn't actually be part of the output model.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Ciel]

Oh wait, I think I'm starting to get it. So the select isn't the last part of the RQL ... so this would work?

from Notebooks as n

load n.Pages.Endowments as e[]
load n.Pages.Traits as t[]
load n.Pages.Commands as c[]

select n.Name

include e.Chosen.Id
include t.Chosen.Id
include c.Chosen.Id

[Ciel]

This is truly incredible. RQL is so powerful! I think I've got something working now - is there any way to save the RQL in the studio and just call it from code? I'm not really fond of these string literals. Thank you both again for all of your help.

[Oren Eini (Ayende Rahien)]

Yes, except that include & load are  single clause, you'll need:

from Notebooks as n

load n.Pages.Endowments as e[], n.Pages.Traits as t[], n.Pages.Commands as c[]

select n.Name

include e.Chosen.Id, t.Chosen.Id, c.Chosen.Id

There is another issue, and that is that the include operates on the shape of the output, not the shape of the query, so you'll want to include it in the output, or use a function

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[Oren Eini (Ayende Rahien)]

No, there is no such way. 

This is by design, because otherwise you'll run into issue that route (see, stored procedures and how they are handled).

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Sun, Dec 3, 2017 at 10:26 AM, Ciel <stacey.ci...@gmail.com> wrote:

This is truly incredible. RQL is so powerful! I think I've got something working now - is there any way to save the RQL in the studio and just call it from code? I'm not really fond of these string literals. Thank you both again for all of your help.

[Ciel]

"There is another issue, and that is that the include operates on the shape of the output, not the shape of the query, so you'll want to include it in the output, or use a function"

Forgive me, but I'm a little unclear on what this means. You said I can use include on things not in the output, but this sounds contradictory to that. Can I bother for an example of what said function might be like then?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

declare function foo()

{

    // you can call include on anything here.

}

include // only things that are in the output will be included.

Note that this make sense, because you need to know what to call Load on in the client side, otherwise you are just including documents for no reason

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Mon, Dec 4, 2017 at 10:43 PM, Ciel <stacey.ci...@gmail.com> wrote:

"There is another issue, and that is that the include operates on the shape of the output, not the shape of the query, so you'll want to include it in the output, or use a function"

Forgive me, but I'm a little unclear on what this means. You said I can use include on things not in the output, but this sounds contradictory to that. Can I bother for an example of what said function might be like then?

On Sun, Dec 3, 2017, 7:51 AM Oren Eini (Ayende Rahien) <aye...@ayende.com> wrote:

No, there is no such way. 

This is by design, because otherwise you'll run into issue that route (see, stored procedures and how they are handled).

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Sun, Dec 3, 2017 at 10:26 AM, Ciel <stacey.ci...@gmail.com> wrote:

This is truly incredible. RQL is so powerful! I think I've got something working now - is there any way to save the RQL in the studio and just call it from code? I'm not really fond of these string literals. Thank you both again for all of your help.

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ciel]

Hrnm, I'm having a tough time figuring out what it considers a valid script. I'm using this;

declare function includable(e) {
    load e as ids[]
    include ids
}
from Notebooks as n
includable(n.Pages.Endowments)
select n.Id

On Tue, Dec 5, 2017 at 1:39 AM Oren Eini (Ayende Rahien) <aye...@ayende.com> wrote:

declare function foo()

{

    // you can call include on anything here.

}

include // only things that are in the output will be included.

Note that this make sense, because you need to know what to call Load on in the client side, otherwise you are just including documents for no reason

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Mon, Dec 4, 2017 at 10:43 PM, Ciel <stacey.ci...@gmail.com> wrote:

"There is another issue, and that is that the include operates on the shape of the output, not the shape of the query, so you'll want to include it in the output, or use a function"

Forgive me, but I'm a little unclear on what this means. You said I can use include on things not in the output, but this sounds contradictory to that. Can I bother for an example of what said function might be like then?

On Sun, Dec 3, 2017, 7:51 AM Oren Eini (Ayende Rahien) <aye...@ayende.com> wrote:

No, there is no such way. 

This is by design, because otherwise you'll run into issue that route (see, stored procedures and how they are handled).

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Sun, Dec 3, 2017 at 10:26 AM, Ciel <stacey.ci...@gmail.com> wrote:

This is truly incredible. RQL is so powerful! I think I've got something working now - is there any way to save the RQL in the studio and just call it from code? I'm not really fond of these string literals. Thank you both again for all of your help.

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Inside the function, there is normal JS code, nothing fancy.

declare function includable(endowments) {
    var ids = load(endowments);

    for(var i = 0; i <ids.length; i++){

          include(ids.Chosen.Id);

    }

   return null;
}
from Notebooks as n
select {

   Id: n.Id,

  _: includeMore(n.Pages.Endowments)

}

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ciel]

Oh! I see - so inside the Javascript function, they're just normal functions and not LINQ-type statements. Thank you!

Is there a reason that Id is null when using the `select { }` syntax, opposed to `select n.Id`?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/8NlMEpLTnRs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

yeah, sorry, inside the {} it is id(n)

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.