Raven DB architecture

[Joey]

I have only recently started programming in a corporate environment, where a lot of these "patterns and practices" have become important. I am working on a experimental project of my own implementing RavenDB. The project deals with analysing tweets and detecting user groups from them. The project stores all the tweets in RavenDB. Now I am in confusion about the client architecture which will do the analysis of data stored in ravendb. 

I just started learning about the repository pattern and as I started going about implementing the typical Repository pattern that I use, I ran into some confusion in regards to data "context". This prompted me to search around where I came across several articles that suggest not using Repository Pattern AT ALL, with RavenDB. Many suggesting even using the document session from the controllers themselves. 

Here is one link in particular so you can see what Im talking about: http://novuscraft.com/blog/ravendb-and-the-repository-pattern 

Now Im sure that works fine, but how would one now go about writing unit tests for that? Have these people abandoned TDD principles entirely or is there something I am missing here? Assuming I intend to maintain unit tests with my application, would it also make sense to maintain my repository? Furthermore is it even necessary that I implement DbContext now, or is that just a remnant from entity framework? Or should I have my documentsession requests directly in my repository or elsewhere (whatever the case may be)?

Please pardon my ignorance here, some of these concepts are still a little fresh for me. My goal here is to abstract as much as possible because I intend at some point to write both to RavenDB for real-time application data. So implementing ravendb directly in my controller does not seem like an option. I find sometimes recently that I am more confused by which "patterns" are correct than anything, there is so much conflicting information this that I decided to ask here. I appreicate your opinions and advice. 

Please let me know a good architecture for this scenario.  How effective is MVVM architecture in this case ?

[Fitzchak Yitzchaki]

We do not abandoned TDD. The opposite it the true, if you have an issue we tell to create a failing test.

Basically, you are working in the application against the IDocumentStore, which is typically the DocumentDtore. And in you tests you are using the EmbeddedDocumentStore, which is a full DB running in memory, should you add to it some data, and test.

MVVM is a UI paradigm, which is not related to RavenDB which is a storage solution.

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[afif]

Hi Joey,

A good starting point would be to avoid having the same architecture for your queries and commands. The repository pattern is in general not advised as its a useless abstraction and 'often' leads you down the track of creating huge classes that have many different queries and commands thrown in the one place simply because they all belong to (for e.g) a customer. Its a gross violation of the Single Responsibility Principle. A DbContext is a hangover from the EF implementation and by no means is a good pattern to stick to, like for e.g. the most popular ORM, Hibernate doesn't use such a thing. Its principles are more in line with how Session management is done in RavenDB. 

Any ways to answer you question, my humble opinion would be you start simple. For queries, to keep them unit testable, some thing as simple as creating extension methods against the IDocumentSession that capture your query logic will suffice. Your app will get the session from a DocumentStore, where as your unit tests will get it from an EmbeddableDocumentStore, i.e. an in memory one RavenDB gives you for frictionless unit testing. Please look at the source code or Oren's blog, and see how he manages session scope for a web app. You only need to add to it by putting the queries behind extension methods and calling those extension methods from your controllers using the IDocumentSession that is injected to the respective Controller.

There is no 'architecture' really required here for querying because RavenDB has done that heavy lifting for you already. If you are thinking, in the future I would like to query from some where else, and so my queries should not tie to a particular database (i.e. RavenDB), then this is a pointless abstraction IMHO. Its not something you should ever architect for. Never can a decent sized application simply switch its querying database, simply because it was 'abstracted' away. Yes, if your queries are only doing key value look ups, that abstraction 'can be' justified and understood, in cases where you may change from Couchbase to Redis to RavenDB etc. But when your query layer is doing full fledged LINQ queries, no amount of abstraction will give you the ability to simply change the underlying implementations itself and not care. How often do you do this, its not worth the noise it creates.

As for commands, please refer to the Command object pattern. That gives you neat, unit testable classes for every command, where every class encapsulates its respective command logic. Again because the commands are against an IDocumentSession, its very easy to test it, by simply using an EmbeddableDocumentStore in your unit tests. When requirements evolve to say also save to Database 'Z' after you save to RavenDB, then you can very easily extend your commands to publish events, and have handlers that listen to these events and save to your desired Database 'Z'. Mind you, for a starting point, I am talking about in memory events here, which you can very easily hook up with some subscribe and publish interfaces and IOC sprinkled on top.

MVVM has nothing to do with your underlying architecture, its more a pattern to dictate interactions and abstractions there of, from where your query returns a 'model', then how does the 'view' and the data required for the view a.k.a the 'view model', all interact together.

Hope this helps.

Afif

[Chris Marisic]

I'm the opposite of many users here (and to what Ayende advises) I never expose RavenDB to my application.  RavenDB usage is always buried at the lowest level so I can do full test first development without touching anything related to a persistence store. I follow the rule of always defer the database and other similar dependency details.

Note that just because i said this doesn't mean have FooService : IFooService everywhere, interfaces are rarely needed, generally public virtual is all that's needed for testability. Also using the repository pattern is no excuse for letting feature/scope creep happen that leads to glaring SRP violations. Repositories are great places for removing boiler plate code such as handling mapping, user audit tracking, entity validation (as opposed to business validation) etc. Raven provides many hooks itself for these AOP-ish actions however I feel these are first class application knowledge and should not be buried into the persistence store configuration.

Aka I try to avoid everything magic at this point.

[Jahmai Lay]

For what it's worth I also abstract RavenDB.

Actually I abstracted our data layer(s) before we migrated to RavenDB and made migration very easy.

Our abstraction is pretty thin and talks domain language (RavenUserStorage : UserStorageBase { Add(User user); }).

I think there is merit for both arguments and it really depends on your team and project.

[Kijana Woodard]

For me, at some point, some code has to understand the inputs, the storage mechanism, and the outputs. IMO, you're better off "just doing it in the controller" or abstracting the whole feature. Instead of an abstraction of all the pieces of a persistence engine, abstract the functionality.

For instance, input is a request object and the output is a ViewModel. The code under test uses raven to convert the input into a view model. For testing, use an embedded db. When you swap out the db to a theoretical "something else", you swap out the entire feature. This also lets you have different features on different persistence technology easily. In addition, you get the full power of whatever persistence technology you're using without trying to force it through a generic repository interface.

Honestly, I think Chris and Jahmai are closer here than their answers imply.

--

[Mircea Chirea]

I'm going to play devil's advocate a bit. See inline replies.

On Wednesday, June 19, 2013 4:24:07 PM UTC+3, Chris Marisic wrote:

I'm the opposite of many users here (and to what Ayende advises) I never expose RavenDB to my application.  RavenDB usage is always buried at the lowest level so I can do full test first development without touching anything related to a persistence store. I follow the rule of always defer the database and other similar dependency details.

Why bury RavenDB? It's not that you can move to another system without one of:

• A lot of effort to rewrite EVERY single query
• Stupidly complex code to abstract away complex queries.

Besides there isn't really a direct competitor to RavenDB; maybe RethinkDB, but it's not mature and certainly not for .NET.

The only good reason (IMO) to bury RavenDB is if you must run on a whole bunch of different databases.

Note that just because i said this doesn't mean have FooService : IFooService everywhere, interfaces are rarely needed, generally public virtual is all that's needed for testability. Also using the repository pattern is no excuse for letting feature/scope creep happen that leads to glaring SRP violations. Repositories are great places for removing boiler plate code such as handling mapping, user audit tracking, entity validation (as opposed to business validation) etc. Raven provides many hooks itself for these AOP-ish actions however I feel these are first class application knowledge and should not be buried into the persistence store configuration.

Mapping can be very easily handled with AutoMapper. You also will have to map to view models, which you don't want your repository to be aware of. I added a few extension methods to IEnumerable/IQueryable/IList to be able to quickly map a collection without polluting the code at all. Here's all I need to do to get a list of categories, from Raven document to view model:

Raven.Query<Category, Categories_InUse_ByCity>()

     .OrderBy(c => c.Index)

     .Map<Category, CategoryItem>()

     .ToList();

Entity validation is handled by the view models; I don't do validation on the documents themselves, as it would be redundant. You could always perform it yourself in a few lines of code with attributes (like MVC models), or use something like FluentValidation. Hooking it into Raven is pretty easy with a listener.

Audit tracking is not something I'd want hidden, really. I prefer critical code to be visible, even if it's ugly. You can always sweep it under the bed with #region if you are so inclined, but you can still quickly see it.

Aka I try to avoid everything magic at this point.

Embedded Raven running in-memory is hardly magic.

[Iulian Margarintescu]

Some thoughts on the Repository Pattern: http://www.erata.net/programming/repository-pattern/

Iulian

[Kijana Woodard]

You have to consider the source of each piece of advice you get as well.

For me, having over-architected and prematurely optimized my share of projects, I tend to lean on the naive solution and refactor aggressively as the real requirements become clear taking into the capabilities and inclinations of the team.

I understand that breaking open an ActionResult to do assertions isn't completely pleasant. It can be done though and it's not _that_ bad.

What I've realized is not that I want more abstractions, but that I don't really care for asp.net mvc. I use it because most other developers are comfortable with it. Fubu mvc, Service Stack, Nancy, and even Web Api make "more sense" in my head because they allow a model in and a model out. Note that this one model in one model out concept is directly taken from the Fubu Way. I find it's a good pattern regardless of platform.

What I _really_ want is a way to hook a route to an arbitrary function and use a model binder to marshal the query string into the input model. The function returns the output model and conneg formats the result. Testing is now a breeze. Swapping out a Raven Flavored Function for a Mongo Flavored Function is a breeze. Having both is a breeze. A generic Get<T>() does none of that for me.

My own devil's advocate play:

"Mapping can be very easily handled with AutoMapper."

I've removed about as much of this code as I've written in projects. Now all the mapping code is "somewhere else", possibly even in another assembly. I'd either rather have the "flavored function" do this mapping or have the view model do this mapping. As models evolve, I've seen my fair share of bugs of "oops, I forgot to change whatever in the "auto" mapping code and the Mapping Assert didn't catch it and the "my gosh, if you remove auto mapper the performance doubles".

Hand mapping is very boring code to write, but it's also dead obvious what's going on 6 months later.

--

[Mircea Chirea]

I have to agree with you on the AutoMapper part. I'm starting to hit the limit of what AutoMapper was designed to do.

There's also the old field you forgot to change, or when refactoring you forget about the mappings and all hell breaks loose.

That said, for simpler tasks, it's great.

[clayton collie]

I'm more and more developing in this fashion, with features ("user stories" in the agile sense) constructed from Commands, Events, and Queries.

State mutation by commands, reads from queries, notifications by events, message based throughout.

I freely use RavenDB or whatever else in my handlers, and since the messages are the contracts, it gives me sufficient isolation if i need 

to change come infrastructure underneath.

[Ryan Heath]

On Wed, Jun 19, 2013 at 4:55 PM, Kijana Woodard <kijana....@gmail.com> wrote:

My own devil's advocate play:

"Mapping can be very easily handled with AutoMapper."

I've removed about as much of this code as I've written in projects. Now all the mapping code is "somewhere else", possibly even in another assembly. I'd either rather have the "flavored function" do this mapping or have the view model do this mapping. As models evolve, I've seen my fair share of bugs of "oops, I forgot to change whatever in the "auto" mapping code and the Mapping Assert didn't catch it and the "my gosh, if you remove auto mapper the performance doubles".

Hand mapping is very boring code to write, but it's also dead obvious what's going on 6 months later.

+100

// Ryan 

[Jahmai Lay]

It's a worth while discussion to have and doesn't need to end with hearts and minds changed :)

The basic problem having hard dependencies on things outside of your control and/or things that can change (not a judgement of RavenDB at all).

Each time you take a hard dependency on something, you have to ask yourself, what does it cost the product / project / my sanity if it fails to do its job, stops being supported, becomes too expensive, becomes redundant or is superseded.

It's not about rewriting those queries. You have to do that no matter where they are written. It's about rewriting all the code that sits above and around the queries.

If your software is a MVC web app, your DB access is in your controller. Happy days that's one file per page (probably) that needs to be reviewed and changed.

If you have a enterprise application where there are many moving parts and database access is spread over many parts, like web interface, service interfaces, reporting functions, background running tasks, integration with other running systems and so on, it could well be that your particular DB interfaces (e.g. IDocumentStore) are passed all around the place for things to do their job.

I've worked on projects like this before, except they weren't RavenDB - they were SQL Server. Code everywhere had ad-hoc selects and inserts to get the job done and without much further thought (often the developer learned just enough SQL to solve their problem and then move on).

When SQL Server started to miss performance expectations, the word goes out to get input from developers on how to solve the problem. Typically even prototyping can be expensive in dev time because the amount of code you have to change - and I'm talking about the bootstrapping level down to the query level - is excessive. When this happens, developers start using scary words like "rewrite", managers lose their hair, end of the world, cats and dogs living together, mass hysteria. At the end of the day you buy a bigger box, pay for the higher tier SQL license and call the problem solved and try to move on with your life.

In this particular situation, abstracting away your database and having the dependency injected during bootstrap can, but might not mitigate this scenario. You need to decide how likely this outcome is, how painful it will be to go through, estimate the cost of the abstraction, and decide whether or not it is worth doing.

I made the choice to abstract in our project based on my experiences, and based on our product and the kind of project it is and it is constantly evolving and is heavily reliant on database performance and other characteristics.

I prototyped our product with SQL Server, because back then NoSQL was some romantic myth I barely heard about, but performance of SQL was aweful and dealing with schemas was aweful.

Then I found MongoDB, fell in love, learned that it didn't believe in cross document transactions, and read how to "work around this in the application layer", and after crying "why god why?" I found Oren's blog post sharing my sentiment, and then discovered RavenDB and the rest is history.

Each change I thank myself for having used abstractions, each change was about 2 days work for about 15 different data access objects to get the basics, then tuning those as time went by.

We don't use a generic base IRepository<T>.

Each of our data access objects has it's own specialized interface (IUserStorage) with functions that the domain / business layer understands and speaks in.

The data access object interfaces did change a little, but the adaptation was so slight it barely registered on the pain meter.

My story isn't everyones story. My product isn't everyones product. YMMV.

And finally, I think it's fantastic that I might be able to try LevelDB by simply changing "Raven/StorageTypeName" ... maybe? :)

[Jahmai Lay]

Yes.

The decision doesn't have to be either using IDocumentStore directly, or adopting IRepository<T>, it can be something in the middle, or even a combination depending on what risk you are trying to mitigate.

[Kijana Woodard]

Yes. This.

[Kijana Woodard]

"And finally, I think it's fantastic that I might be able to try LevelDB by simply changing "Raven/StorageTypeName" ... maybe? :)"

Shows how much of an abstraction your already using. ;-)

I think we all agree that IRepository<T> has no hope. After that, it's just picking your abstraction depth.

"If you have a enterprise application where there are many moving parts and database access is spread over many parts, like web interface, service interfaces, reporting functions, background running tasks, integration with other running systems and so on, it could well be that your particular DB interfaces (e.g. IDocumentStore) are passed all around the place for things to do their job."

Personally, my experience with these apps is that the problem is that, regardless of all the layers and visio diagrams, what we really have is a big ball of mud. There is, probably, a single massive database that everything has to bow to. Optimizing is really hard because optimizing for one scenario degrades another. Having the app split by business capabilities gives you much more leverage. Now the Shipping functionality can be modified and upgraded without having to worry about the Billing functionality. 

Effectively smaller apps that interface explicitly through Events / (persistent) view models / client side-composition. But where and how to do that depends entirely on the business: SOA/DDD/etc.

IMO, document databases like Raven move you in the right direction because documents do not have hard relationships and indexes bring eventual consistency to the forefront. Separating Sales from Shipping seems attainable when you don't have to suddenly disable a bunch of FK constraints. :-D

--

[Oren Eini (Ayende Rahien)]

Every single time that I used something like auto mapper, we have had some bugs related to that that was hard to figure out.

--

[Mircea Chirea]

It's a worth while discussion to have and doesn't need to end with hearts and minds changed :)

The basic problem having hard dependencies on things outside of your control and/or things that can change (not a judgement of RavenDB at all).

Each time you take a hard dependency on something, you have to ask yourself, what does it cost the product / project / my sanity if it fails to do its job, stops being supported, becomes too expensive, becomes redundant or is superseded.

It's not about rewriting those queries. You have to do that no matter where they are written. It's about rewriting all the code that sits above and around the queries.

If your software is a MVC web app, your DB access is in your controller. Happy days that's one file per page (probably) that needs to be reviewed and changed.

If you have a enterprise application where there are many moving parts and database access is spread over many parts, like web interface, service interfaces, reporting functions, background running tasks, integration with other running systems and so on, it could well be that your particular DB interfaces (e.g. IDocumentStore) are passed all around the place for things to do their job.

Two words: command objects.

Basically an that you feed the session, some information and it knows how to execute the query and give you the results. That way you don't need to know about RavenDB in all the parts, but also don't stuff all your code into a single class like the repository pattern. Oren's blog uses them IIRC.

[Jeremy Holt]

I just use Mapper.AssertConfigurationIsValid() at the end of the mapping - it gives you a nice exception telling you which mappings you forgot

[Kijana Woodard]

I recommend using the assertion as well, but still expect issues. :-)

[Jahmai Lay]

Yes

[Mircea Chirea]

That won't catch everything. Things like renaming a property on the destination type but not the source and you let AutoMapper match them, rather than explicitly specify the mappings.