Should that index / transform work?

[Alexander Zeitler]

Hi,

 

I have been playing around with indexing and tried this:

 

I created a document products/1

 

{

  "Name": "Awesome Book"

}

 

I created this Index (which doesn’t make much sense but I tried to understand the problem):

 

Map:

from product in docs.products

select new {

Id = product.Id,

Name = product.Name + " " + product.Id }

 

Transform:

from result in results

select new  {

                Name = result.Name,

                Id = result.Id

}

 

The Name property of the transformation is always “Awesome Book” and not “Awesome Book products/1”

 

When I hit the “Terms” button in RavenDb Studio Name is “awesome book products/1”

 

What am I doing wrong?

 

Alex

 

 

 

 

[Oren Eini (Ayende Rahien)]

When you are dealing with the values in Transform Results, you are getting the results of the query.

The output of the map function is there strictly for the purpose of searching, in most cases.

RavenDB searches the index, then uses the document id to load that, then send _that_ to the transfrom results.

[Kijana Woodard]

Let's see your query

[Kijana Woodard]

Oh yes. I think you would get the behavior you're expecting if you had a reduce step.

[Alexander Zeitler]

Thanks, this now works as expected:

 

Map:

from product in docs.products

select new {

Id = product.Id,

Name = product.Name }

 

Transform:

from result in results

select new  {

                Name = result.Name + " " + result.Id,

                Id = result.Id

}

 

Btw: “Fields” (that can be stored etc.) always refer to properties of the map function, correct?

 

Alex

[Oren Eini (Ayende Rahien)]

Or the reduce func, if it is there.

[Alexander Zeitler]

Thanks again.

 

Enhancing the transform function as follows (similar to http://ravendb.net/docs/client-api/querying/static-indexes/live-projections)  results in an exception when trying to store the index:

 

from result in results

let data = database.Load<Product>(result.Id)

where data != null

select new  {

                Name = result.Name + " " + data.Name,

                Id = result.Id

}

 

Exception:
c:\RavenDb\RavenDB.Server.1.2.2082-Unstable\tools\Databases\ProductTests\IndexDefinitions\TemporaryIndexDefinitionsAsSource\ougmjb2o.0.cs(33,15) : error CS0103: The name 'database' does not exist in the current context

c:\RavenDb\RavenDB.Server.1.2.2082-Unstable\tools\Databases\ProductTests\IndexDefinitions\TemporaryIndexDefinitionsAsSource\ougmjb2o.0.cs(33,29) : error CS0246: The type or namespace name 'Product' could not be found (are you missing a using directive or an assembly reference?)

 

I’m editing the index definition inside RavenDb studio – how can I invoke the database.Load<Product>(result.Id) call from there?

[Oren Eini (Ayende Rahien)]

Database.Load

If you write it directly in the studio, you need to use "Database", not "database"

[Alexander Zeitler]

Works, thanks.

[Alexander Zeitler]

I guess, I found a bug:

 

This transform definition gets saved inside Studio without any exceptions:

 

from result in results

let data = Database.Load("/1) <-- missing closing quotation mark

where data != null

select new  {

                Criteria = data.SpecificCriterionInstances,

                Id = result.Id

}

 

When executing it, it returns empty documents.

[Alexander Zeitler]

Can I also add a parameter here?

 

Transform:

from result in results

let data = Database.Load(param) <-- param possible here?

where data != null

select new  {

                Name  = data.Name,

                Id = result.Id

[Oren Eini (Ayende Rahien)]

It is a bug, we automatically fix this to be:
let data = Database.Load("/1)")

The fix is quite hard to deal with, so we probably won't try. 

[Oren Eini (Ayende Rahien)]

I am not sure that I follow.

A param as in an external value?

No, you can only use the values you can derive from the Results.

[Alexander Zeitler]

Yes, I was thinking about an external parameter WRT the model we have been talking a bit last week in Luzern after the UG meeting.

The requirements have increased a bit:

We have products (500000+). Each product has an arbitrary amount of criteria (500-1000) that have to match a schema each (format/unit).

Each criteria needs to hold the information about which roles (50+) are allowed to access it.

Each product has a manufacturer and and an arbitrary number of customers (50+).

There is an arbitrary number of criteria for each product that have specific values to each customer but the same definition (name/format/unit) and again hold the information about which roles are allowed to access it.

I need to query a list of products of the same product type they belong to and get all criteria the user has permissions to read (due to it's role membership).

I thought about indexes but I also thought about creating product documents having combined id's like product/1/customer/1.

I don't like the last one at the moment as it won't work if products e.g. also have suppliers that can add criteria values also with the same rule behavior.

Alex

[Oren Eini (Ayende Rahien)]

So, what you are trying to do:

- Query a list of products by the roles that the user is associated with:

sess.Query<Product>()

   .Where(x=>x.Roles.In(user.Roles))

Then you transform the values on the client side.

[Alexander Zeitler]

class Product {

  public List<Criterion> Criteria {get;set;}

}

 

class Criterion {

public List<Role> Roles {get;set;}

}

 

That’s what I’m trying to do.

[Oren Eini (Ayende Rahien)]

Okay, and what is the actual query that you want to run?

Show the linq to object query

[Alexander Zeitler]

I’have been playing around a bit and it looks promising with Linq to object but I’m not done yet.

Anyhow: I’m using let in the Linq queries to solve that problem.

According to http://ravendb.net/docs/client-api/querying/using-linq-to-query-ravendb, let is not supported in RavenDb yet.

 

Is this valid for 1.2 also?

Any workarounds?

[Alexander Zeitler]

Here is the gist for the linq to object impl.:

https://gist.github.com/3854809

 

When using it with RavenDb I get an exception for the let assignment (or it’s usage) stating that RavenDb does not allow computation inside the query and I should use an index.

How can I replace the let using an index?

[Oren Eini (Ayende Rahien)]

What you seems to be trying to do is to get some _parts of the document_ based on your query.

The problem is that with RavenDB, the queries are meant to find you _the document_, not just the ability to just get some values out of _a_ document.

In other words ,you can write a query that will give you documents matching the query, but not a query that will match parts of a document.

[Alexander Zeitler]

Thanks. Do you think I could solve it by making that parts documents themselves?

Alex

[Oren Eini (Ayende Rahien)]

Yes.

[Oren Eini (Ayende Rahien)]

Usually if you need just parts of a doc, it means it needs to be separated. 
However, be sure that you actually need them to be different. From your model, you are applying different roles for the entity, and that does NOT sounds like it should be in a different location.

[Alexander Zeitler]

Why should it not be on a different location?

[Oren Eini (Ayende Rahien)]

You basically have a list of users that can access a document, right?

I don't understand why you are trying to get just that list out of the document? 

[Alexander Zeitler]

Sorry, I don't understand what you mean exactly.

[Oren Eini (Ayende Rahien)]

You have a list of roles, inside the docs, right?

[Alexander Zeitler]

Right

[Oren Eini (Ayende Rahien)]

And you are trying to get a list of those roles out, right?

[Alexander Zeitler]

The primary goal is to get the product criteria when a user is it least a member of one role for that criteria.

[Oren Eini (Ayende Rahien)]

Why do you care about the criteria outside of the document?

[Alexander Zeitler]

I want to return a product document that only contains the criteria the user is allowed to see. 

Another scenario is creating documents (outside the database) which contain that criteria but not the other parts of the product document.

[Oren Eini (Ayende Rahien)]

Yes, I understand. But you query is actually trying to load just the criteria that you have.

That is why you had the let & select statements there.

[Alexander Zeitler]

You mean because of returning a new Product without the name, id etc. of the product but only containing the criteria? I skipped that for brevity but the complete code would cover that also.

[Alexander Zeitler]

Yes

[Oren Eini (Ayende Rahien)]

Okay, so you need all of the document, right?

In that case, why are you trying to project stuff out of the document? Why not return the whole thing?

var user2Results = from p in session.Query<Product>()

        where product.CustomerCriteria.Roles.Any(role => role.Users.Any(user => user.Name == "user 2"))

&& product.CustomerCriteria.Customer.Id == "customers/2"

&& product.CustomerCriteria.Criterion.Roles.Any(role => role.Users.Any(user => user.Name == "user 2"))

select p;

[Alexander Zeitler]

Security. The main client is a JavaScript client, so somebody could easily inspect the object and see data he's not allowed to see. 

There are also other external service clients that should not get access to all properties.

[Oren Eini (Ayende Rahien)]

If the client is JS, the user can also make requests directly to the server and get ALL the documents, no?

[Alexander Zeitler]

Client = client of our Web API, not database client.

[Oren Eini (Ayende Rahien)]

Okay, then what is the problem?

You can do the security filtering in your web api code.

[Alexander Zeitler]

Ok, that’s what I have thought about from the beginning but I thought if I could get the data already from the database in the required format without huge effort it would be better.

So, I’ll stick with the first approach.

 

Thanks,