Re: [RavenDB] Converting a relation model to a document model

[Oren Eini (Ayende Rahien)]

You already have root aggregates, so start with those:  User, Badge, Level, Group, Topic, Tag, Comment, Comparison 

Then follow naturally from there. Once you have the aggregates, you are pretty much done.

On Sat, Aug 11, 2012 at 11:23 AM, Mircea Chirea <chirea...@gmail.com> wrote:

Hello everyone!

I have been working on a website in my spare time for the past few months and I have reached a point where I don't think SQL databases can cut it. I originally picked that because it's what I had any experience with and Entity Framework made it very easy to use, but now queries have become rather hard to do and worse when you need to take into account the time and effort it takes to get EF to generate a good query. My model is rather complex and I decided a month ago to migrate to MongoDB but their LINQ provider is not great and I had issues getting the model right for Mongo anyway. Since I've heard about Raven I've been impressed, especially by the integrated nature of Lucene, so I decided to try again, but since I am a newbie, I find this task to be daunting. Take a look for yourself: http://imgur.com/9355f

The two biggest issues I have at the moment are getting a specific comparison, including topics with specs, votes, tags, comments with replies and votes. I have about 20 queries, for this alone, loading everything (as well as doing some checks first); even with memcached it only goes so far. It is also a pain to extend, migrations being hard to do without losing data and migrating down is a royal pain.

I have identified the entities (or aggregate roots... all these terms) to be: User, Badge, Level, Group, Topic, Tag, Comment, Comparison. Everything else simply has a is-owned-by relationship, even though the Synonyms table is shared with Tags and Topics, they don't use the same rows anywhere. An Edit is just a different version of a topic's content, in a particular point in time; I don't think I can use Raven's versions features, as I need to be able to have edits require approval (new users for example); currently edits don't save tags, specs and synonyms, as it would be a royal pain to refactor the code and create the migrations necessary (thanks EF!). I have separated most of the database activity into command objects, so 95% of the rest of the code (still a few bits to move) is oblivious to the underlying database, which also made memcached relatively easy to add.

So, my question is, how should I rewrite the model into one suitable for RavenDB?

[Mircea Chirea]

Right. The thing is I am not sure how to set up any relationships between these aggregates. I could just add an property ID a la SQL, but RavenDB doesn't seem to have joins or anything similar AFAIK, so I would need to issue another query; this starts to add up just as quickly as on SQL, if not worse.

I also have a special case:

Comments and votes are not per-topic, they are per-topic in a single comparison. So X has 4 comments and Y has 2, in the X vs Y, but not in X vs Z or Y vs Z. How would I store that? Currently I have set the relations to be Comparison -> Comments and Comparison - Votes, with properties like CommentsForFirst in code, but this is pretty horrible and I have to always switch on whether the topic id is the first or the second one in a specific comparison, to load comments and votes. A mess.

[Oren Eini (Ayende Rahien)]

inline

On Sat, Aug 11, 2012 at 9:24 PM, Mircea Chirea <chirea...@gmail.com> wrote:

Right. The thing is I am not sure how to set up any relationships between these aggregates. I could just add an property ID a la SQL, but RavenDB doesn't seem to have joins or anything similar AFAIK, so I would need to issue another query; this starts to add up just as quickly as on SQL, if not worse.

You hold the id, and then you include it when needed.

No additional query required.

 

I also have a special case:

Comments and votes are not per-topic, they are per-topic in a single comparison. So X has 4 comments and Y has 2, in the X vs Y, but not in X vs Z or Y vs Z. How would I store that?

 

You put it in the topic, comments and votes are part of it.

[Mircea Chirea]

I can't put the comments and votes in the topic, as... uh let's take some examples: VW Passat vs Audi A4. The Passat has 50 votes and 20 comments, the Audi has 30 votes and 10 comments. But if you compare Passat to say BMW 335d, Passat doesn't have those votes and comments from Passat vs Audi. That's what I mean, comments and votes are local inside a comparison and each side has votes/comments there. How do I store those?

[Oren Eini (Ayende Rahien)]

Okay, then why not put them directly on the comparison?

[Mircea Chirea]

Well how would I store them? Where exactly to put them? Surely there is a better way than the hack I did on EF: CommentsForFirst, CommentsForSecond, etc.

[Oren Eini (Ayende Rahien)]

What do you want to have there?

[Mircea Chirea]

An array or list of votes (embedded) and a list of comment ids.

[Oren Eini (Ayende Rahien)]

And what prevents you from doing just that?

[Mircea Chirea]

Well how do I store them in the Comparison document?

[Oren Eini (Ayende Rahien)]

Show us the class structure as you would like it to be

[Mircea Chirea]

I've spent some time reading your blog, the docs and some SO posts and I learned about denormalized references and a few tricks. I've converted my current EF entities to Raven as best as I could:

EF:  http://codepaste.net/sh3kke

Raven:  http://codepaste.net/cnb9ib

There are some differences:

• Removed some navigation properties, like User.CommentVotes and User.TopicVotes, which no longer make sense.
• Removed the hacks around EF not supporting enums (excluding the beta 5.0 release that needs .NET 4.5)
• Removed data annotations, since most are EF specific; can Raven handle data validation or do I have to do it in my application?
• Removed constructors and use plain arrays, for readability.
• Left the BuildModel static functions as they might help you better understand the relations between entities.
• Moved the "Computed Properties" from topic into auto properties, which would logically mirror the last edit (thus removing the need for a query).
• Created Ref classes to deal with references, storing a little more data than just the Id.

Please look at both and check if it is a sane conversion and more importantly, makes sense as a Raven model. I'd also appreciate criticism on my EF entities, as I will continue using it for a while until I can fully port the application, so a few improvements wouldn't hurt. I know that the topic edits do not handle specifications and tags; this is a flaw in the current design, I've not fixed it in the Raven model to make the differences easier to see, but ideally the edit will also store specs and tags, having those mirrored in the topic document itself for ease of accessibility.

For Comparison, I wrote a custom reference class "Computed Properties" which stores the topic ID and the respective comments and votes - there would be the CommentsForFirst and CommentsForSecond in the EF model. Is that a good thing to do? I might also make the Comparison class be something like Pair<Topic> and implement IEnumerable, so I could more easily use it in the code; how does Raven handle that? I'm worried it might treat each Comparison as a JSON array, which would be bad.

Lastly, Comment has a circular reference; a comment can be a reply to another, so I need to track that. What is the best way to do that? I currently use a FK key to the same table and a ReplyTo column; if that is NULL, the comment is not a reply, if it has a value it's the ID of the ascensor comment, so it's basically a tree: a comment can have many replies and can be a reply to a single comment.

Quite a lot of questions, but what are you gonna do :| Being used to SQL and being taught that SQL is the swiff army knife make it hard to think straight how to model objects and their relations in a non-relational database (heck, it's hard as hell to do it in a relational - seems to me "relational" is not the proper term for SQL databases, really).

Thank you very much for your effort so far Oren Eini, I've yet to see someone so dedicated to helping users of their products. I have subscribed to your blog and have a crapload of good posts bookmarked to read later.

[Mircea Chirea]

Nevermind the part about making Comparison a pair. I think I'll just store the topics in a single array, like public EmbeddedTopic[] Topics { get; set; }, as that does what I want just fine.

[Oren Eini (Ayende Rahien)]

inline

On Sun, Aug 12, 2012 at 8:33 PM, Mircea Chirea <chirea...@gmail.com> wrote:

I've spent some time reading your blog, the docs and some SO posts and I learned about denormalized references and a few tricks. I've converted my current EF entities to Raven as best as I could:

EF:  http://codepaste.net/sh3kke

Raven:  http://codepaste.net/cnb9ib

There are some differences:

• Removed some navigation properties, like User.CommentVotes and User.TopicVotes, which no longer make sense.

I really don't like Comment.Replies

Is there a reason each comment is a separate document?

 

• Removed the hacks around EF not supporting enums (excluding the beta 5.0 release that needs .NET 4.5)
• Removed data annotations, since most are EF specific; can Raven handle data validation or do I have to do it in my application?

You can add a listener that will do that for you.

[Mircea Chirea]

On Monday, August 13, 2012 11:23:34 AM UTC+3, Oren Eini wrote:

inline

On Sun, Aug 12, 2012 at 8:33 PM, Mircea Chirea <chirea...@gmail.com> wrote:

I've spent some time reading your blog, the docs and some SO posts and I learned about denormalized references and a few tricks. I've converted my current EF entities to Raven as best as I could:

EF:  http://codepaste.net/sh3kke

Raven:  http://codepaste.net/cnb9ib

There are some differences:

• Removed some navigation properties, like User.CommentVotes and User.TopicVotes, which no longer make sense.

I really don't like Comment.Replies

Is there a reason each comment is a separate document?

Yes, they are loaded async. I also need their ID to delete them, see them on the user's page, notify users or replies, that sort of thing. 

[Mircea Chirea]

I've been doing some research on Oren Eini's blog, but I haven't found answers to my questions, especially the circular reference one with comments. I am not sure if my model fits well into RavenDB's design either, though it seems it might work.

[Kijana Woodard]

@Oren - I'm not crazy about comments with replies or comments as separate docs either.

However, here are the use cases I'm facing: 

Comments can be replied to creating a "mini conversation" within the comments section of a post.

Comments can be marked as spam by the system.

Comments can be marked as spam/inappropriate by users; if enough users do this, the comment is hidden/removed.

Comments can be redacted (removing offensive words).

Users earn points for comments.

My initial approach was this:

I'm inserting the comments as individual docs and using an index to retrieve them as a single doc by post id. 

https://github.com/kijanawoodard/Commenter/blob/master/Commenter.Documents/Comment.cs 

https://github.com/kijanawoodard/Commenter/blob/master/Comments.Web/Indexes/CommentIndex.cs

I'll sort them so that the replies are inline either on the server or in javascript.

Assumptions: 

Most comments are of similar to "Yes, this is cool!!!", in other words short.

Most posts will have less than 500 comments.

We will use cache headers / akamai to limit the number of requests to raven.

Anyone have any....comments? :-)

[Mircea Chirea]

Those assumptions don't hold in my usage scenario; comments are going to have a limit of 500-1000 characters and posts could have well over 500 comments too (but that's something to worry about in the future).

But yeah, those are my reasons as having comments separate from posts.

[Kijana Woodard]

@Mircea - sorry, I hijacked your post. I should have started a new thread. Those assumptions were for my situation, not yours.