Re: [RavenDB] Live projection limitations

[Ayende Rahien]

I am not really sure that I am following.

Live projection are running on the server side, and they aren't really dealing with types.

In fact, the Load<AnswerFoo> is converted to a Load<dynamic> (well, sort of) on the server side.

This should work:

TransformResults = (database, results) =>

  from result in results

  select new 

  {

    result.QuestionId,

    Answer = database.Load(result.AnswerId),

    Delta = result.Delta

  };

On Wed, Feb 23, 2011 at 2:40 PM, jalchr <jal...@gmail.com> wrote:

I think I'm streching the use of 'live projections' a little bit.
This will be a long post, so please read carefully. I have submitted a
pull request onto it. So you can test it on your side.

Back to the StackOverFlow types, I decided to have rich Domain objects
and light persistence objects, like this:

Rich Domain objects: hold references to other objects
====================
public class AnswerEntity
   {
       public string Id { get; set; }
       public string UserId { get; set; }
       public Question Question { get; set; }
       public string Content { get; set; }
   }

   public class AnswerVoteEntity
   {
       public string Id { get; set; }
       public string QuestionId { get; set; }
       public AnswerEntity Answer { get; set; }
       public int Delta { get; set; }
   }

Light persistence objects: used as denormalized references
==========================
   public class Question
   {
       public string Id { get; set; }
       public string UserId { get; set; }
       public string Title { get; set; }
       public string Content { get; set; }
   }
   public class Answer
   {
       public string Id { get; set; }
       public string UserId { get; set; }
       public string QuestionId { get; set; }
       public string Content { get; set; }
   }

   public class AnswerVote
   {
       public string QuestionId { get; set; }
       public string AnswerId { get; set; }
       public int Delta { get; set; }
   }


The main reason for this separation, is that I don't want to "store"
large object graphs in a single document.
So when I insert the rich objects, I simply *Map* them to their light
counterparts ... and store .



Where is the problem?
========================
The problem is in the limitations of the live-projection as it can
load only a "stored" type....

TransformResults = (database, results) =>
                              from result in results
                              // this won't work because
'AnswerEntity' is not the stored type, its 'Answer' type
                              let answer =
database.Load<AnswerEntity>(result.AnswerId)
                              // Should be like this
                              //let answer =database.Load<Answer,
Answers_ByAnswerEntity>(result.AnswerId)
                              //                 .As<AnswerEntity>();
                                                    select new //
AnswerVoteEntity
                                                       {
                                                           QuestionId
= result.QuestionId,
                                                           Answer =
answer,
                                                           Delta =
result.Delta,
                                                       };


Then in querying ... this will fail

 using (var session = documentStore.OpenSession())
           {
               var votes = session.Query<AnswerVote,
Votes_ByAnswerEntity>()
                   .Customize(x => x.WaitForNonStaleResultsAsOfNow())
                   .Where(x => x.AnswerId == answerId)
                   .As<AnswerVoteEntity>()
                   .ToArray();
               Assert.NotNull(votes);
               Assert.True(votes.Length == 2);
               Assert.NotNull(votes[0].Answer);
               Assert.IsType<AnswerEntity>(votes[0].Answer);

 ....
}

I know I can workaround this limitation by sending another query for
the AnswerEntity ... but the natural place is to have the live-
projection give me what I want.

Final thing, Is the live-projection running on the Server-side or the
Client-side ?

Much appreciated

[jalchr]

Just run the pull request, I sent to you. "It will show you the path".
Actually, I don't care if the server is knowing about the types or
not ... what matters is the end result, right !

There is also a bug in loading the object Id from live projection.

[jalchr]

[jalchr]

In short the missing piece is this line

let answer =database.Load<Answer,
Answers_ByAnswerEntity>(result.AnswerId)

.As<AnswerEntity>().SingleOrDefault();

The ability to Load/Query from a certain index... think about this as
Multi-level projections.

[Ayende Rahien]

I don't like this idea of multiple levels, it creates a LOT of additional complexity for us.

OTOH, you can certainly do something like:

TransformResults = (database, results) =>

                     from result in results

                     let answer = database.Load(result.AnswerId)

                     let question = database.Load(answer.QuestionId)

                     select new // AnswerVoteEntity

                        {

                            QuestionId = result.QuestionId,

                            Answer = new 

                                      {

                                          Id = result.Id,

                                          Question = question,

                                          Content = result.Content,

                                          UserId = result.UserId

                                      },

                            Delta = result.Delta,

                        };

[Ayende Rahien]

I modified the results from the database to include the metadata of the loaded objects.

But how to deserialize that to the projected entities is something that you'll have to deal with, at any rate, please note that those values are _not_ entities and aren't managed as such by RavenDB

[jalchr]

You proposition works, but I still can't see why the "Client" api has
so much power in querying and loading, while a server side querying is
so limited.
No, I don't know how much complex this is to you, but I would like to
delegate all queries to the server (if possible) to avoid the overhead
of network latency.

[jalchr]

Can you commit your changes, plz ...
I need the 'id' (meta-data) issue functioning

[Ayende Rahien]

Sorry, I am not really sure that I am following you?

Client API has so much power compared to server side?

That is because you are actually _querying_ there. On the server side, we want you to do as little as possible.

[Ayende Rahien]

Yes, just did :-)

[jalchr]

mmm, I updated my repository ... still this test is failing

object_id_should_not_be_null_after_loaded_from_transformation()

Did you have this working ?

[Ayende Rahien]

No, it shouldn't work.

As I mentioned, look at the output that this now generates.

The output contains the @metadata element include the @id

The problem is how to de-serialize that to the user

[jalchr]

Ayende,
I can't understand how hard it is to put the '@id' metadata value in
the 'id' property of a 'dynamic' object.
If "database.load" can load all properties of object *Foo*, why can't
it fill the 'Id' the same way? (I can't see any mysterious way to
deserialize it ... treat the 'id' as type dynamic)

I tried to get the meta-data from our little 'test':
There is Nothing about the Question.Id meta-data property ... which is
missing

session.Advanced.GetMetadataFor(answerInfo)

Output
=============
{
"Raven-Entity-Name": "Answers",
"Raven-Clr-Type": "LiveProjectionsBug.Answer, Raven.Tests",
"@id": "answer\\540"
}
base {Newtonsoft.Json.Linq.JContainer}: {
"Raven-Entity-Name": "Answers",
"Raven-Clr-Type": "LiveProjectionsBug.Answer, Raven.Tests",
"@id": "answer\\540"
}
Count: 3
PropertyChanged: null
PropertyChanging: null

System.Collections.Generic.ICollection<System.Collections.Generic.KeyValuePair<System.String,Newtonsoft.Json.Linq.JToken>>.IsReadOnly:
false

System.Collections.Generic.IDictionary<System.String,Newtonsoft.Json.Linq.JToken>.Keys:
'(session.Advanced.GetMetadataFor(answerInfo)).System.Collections.Generic.IDictionary<System.String,Newtonsoft.Json.Linq.JToken>.Keys'
threw an exception of type 'System.NotImplementedException'

System.Collections.Generic.IDictionary<System.String,Newtonsoft.Json.Linq.JToken>.Values:
'(session.Advanced.GetMetadataFor(answerInfo)).System.Collections.Generic.IDictionary<System.String,Newtonsoft.Json.Linq.JToken>.Values'
threw an exception of type 'System.NotImplementedException'
Type: Object

[jalchr]

I tracked this down to this line:

var deserializedResult =
(T)
theSession.Conventions.CreateSerializer().Deserialize(new
JTokenReader(result), typeof (T));

the json it is trying to deserialize is this:
{
"Id": "answer\\540",
"Question": {
"UserId": "user\\222",
"Title": "How to do this in RavenDb?",
"Content": "I'm trying to find how to model documents for better
DDD support.",
"@metadata": {
"Raven-Entity-Name": "Questions",
"Raven-Clr-Type": "LiveProjectionsBug.Question, Raven.Tests",
"@id": "question\\259",
"Last-Modified": "Fri, 25 Feb 2011 08:21:35 GMT",
"@etag": "00000000-0000-0100-0000-000000000003",
"Non-Authoritive-Information": false
}
},
"Content": "This is doable",
"UserId": "user\\222"

}

I don't know how you are handling the "includes" which I think is very
similar to the above.
Anyway, our little problem has 2 solutions:
1) either 'copy' the @id metadata to an Id property dynamically, the
the deserializer will populate it correctly.
2) Allow the deserializer to investigate special properties like
"@metadata" and read the id from it.

Thoughts ?

> ...
>
> read more »

[Ayende Rahien]

Inline

On Fri, Feb 25, 2011 at 9:51 AM, jalchr <jal...@gmail.com> wrote:

Ayende,
I can't understand how hard it is to put the '@id' metadata value in
the 'id' property of a 'dynamic' object.
If "database.load" can load all properties of object *Foo*, why can't
it fill the 'Id' the same way? (I can't see any mysterious way to
deserialize it ... treat the 'id' as type dynamic)

Because this is happening in a completely different places. database.Load happens on the server, it result is a json document.

The actual serialization is happening on the client.

Ids are external to the object, and RavenDB doesn't even require that an entity will have an Id property. 

[Ayende Rahien]

Jalchr,

The problem is that I am not sure how to support something like that with the current serializer.

I would be very happy to see sample code that would turn an object graph properly from this json.

2) All the deserializer to investigate special properties like

"@metadata" and read the id from it.

Thoughts ?

On Feb 25, 9:51 am, jalchr <jal...@gmail.com> wrote:

> ...
>
> read more »

[jalchr]

Ayende,
I think I solved this one ... before deserializing the object, I will
inspect the json for any "internal" metadata attribute, then add the
"Id" property to the owning object ...

The test in question is passing now ...

I can send you a Pull Request if you are okay with this solution ...
cause a converter will not solve this as converters are TYPE specific
and our case is generic.

Its up to the deserializer now to populate the "id" property if the
object has it or it will ignore it.

The method below is recursive, it will "Fix" any object graph
depth ...

private void HandleInternalMetadata(JObject result)
{
// Dive in recursively
if (result.Value<JObject>("@metadata") == null &&
result.ToString().Contains("@metadata"))
{
foreach (var property in
queryResult.Results[0].Properties())
{
if (property.ToString().Contains("@metadata"))
{

HandleInternalMetadata((JObject)property.Value);
}
}
}
// Implant a property with "id" value ... if not exists
if (result.Value<JObject>("@metadata") != null)
{
if (result.Value<JObject>("@metadata") != null &&
result.Property("Id") == null)
{
// add Id
var id =
result.Value<JObject>("@metadata").Value<string>("@id");
result.Add("Id", id);

> ...
>
> read more »

[jalchr]

Ayende,
Check the pull request I sent to you ... its range is updated

https://github.com/jalchr/ravendb/commit/70bfb30bf1b3b8b1797b35285d5feeca2e4d39be

Plz check

> ...
>
> read more »

[jalchr]

Have you looked into this ?

On Mar 8, 3:44 pm, jalchr <jal...@gmail.com> wrote:
> Ayende,

> Check the pull request I sent to you ... its range is updated
>

> https://github.com/jalchr/ravendb/commit/70bfb30bf1b3b8b1797b35285d5f...

> ...
>
> read more »

[jalchr]

?

> ...
>
> read more »

[Ayende Rahien]

Currently traveling, will try to look on this on Sunday

[Ayende Rahien]

Problem,

What happens when the Identifier property is named differently?

In addition to that, your code is very expensive in terms of how much work we have do to.

For example, theToString will allocate a lot of memory, especially for large documents.

[Chris Marisic]

On Thursday, March 10, 2011 3:41:15 PM UTC-5, Ayende Rahien wrote:

For example, theToString will allocate a lot of memory, especially for large documents.

I took a look at the commit over at https://github.com/jalchr/ravendb/commit/70bfb30bf1b3b8b1797b35285d5f  and I assume your comment is in regards to

 if (result.Value<JObject>("@metadata") == null && result.ToString().Contains("@metadata"))

Would it be possible to rewrite that ToString to read from result either using a TextReader or some kind of stream reader that you only need to buffer small parts of the document at any 1 time instead of buffering the entire doc when ToString happens?

[Ayende Rahien]

No, the way to handle that is by scanning through the JObject directly, not by reading it as text

[Chris Marisic]

Sounds like that part has a known solution. I've never worked with JObjects yet.

[Ayende Rahien]

It is basically a dictionary of dictionaries

[jalchr]

> Problem,
> What happens when the Identifier property is named differently?

Well, that is not a problem I think, because it will not be part of
the metadata, right ?
class foo
{
string FooId {get;set;}
string Name {get; set;}
}

The metadata, somehow is treating the "Id" property of an entity in a
special way and that is causing this issue. Correct me plz.

> In addition to that, your code is very expensive in terms of how much work
> we have do to.
> For example, theToString will allocate a lot of memory, especially for large
> documents.

We can improve this part ... I just put a *working" code.

[Ayende Rahien]

No, it isn't.

We are just pulling the value out based on the IdentityProperty convention, we aren't storing this.

[Ayende Rahien]

That said, I have a few ideas on how to fix this, I'll probably have something ready by Monday

[jalchr]

Okay, very well.
I think its very clear now.

[Ayende Rahien]

Fixed

[Chris Marisic]

Can someone sum up what exactly this fixes? Like what feature can be implemented now with this?

Does this let you use live projects to create aggregate objects from denormalized references? Such as having Blog / Comments as 2 distinct aggregate roots and returning a BlogViewModel to render for page views?

[Ayende Rahien]

Yes, you can do that now.

Basically, we now have:

Conventions.GetIdentityPropertyNameFromEntityName that let you know what the id name for a particular entity name is.

Then we scan the results from Raven in search of Metadata and embed the value in the Json once again.

This only happen for projections.

Please note, _even_ though we setup the ID property, those are _not_ entities(!), they aren't part of the session.