resilient and scalable java entity manager/server

[polymorph]

Hello Guys,

I am bootstrapping a startup and need to develop a prototype.

As part of this project I need to have developed a entity manager/server. 

The goal should be for this component to be able to manage a lot of java objects following the DCI/DDD idea.

Obviously this entity server should manage persistence of the objects and should be scalable/elastic and resilient.

Ideally I should be also looking to implement CQRS/ES.

As I will hire resource to develop this component I am looking for a framework that I can leverage to build this component.

I am not mistaken Qi4j could be a good candidate for this.

I am reading the documentation and part of the code, I have to say that is a very nice peace of software with an awesome architecture/design ! 

Congratulation for this design/architecture, for the quality of your code and thank you for sharing it with the community !

So to summarize I would really appreciate if you can take some time to answer to the following questions:  

- can you advise if Qi4j should be a good candidate for my project? if yes any advises to get the most of it?

- could you need some help/collaboration/participation on this project? if yes what should be the best way to help? what kind of people/skillset should I be looking for?

Thanks in advance for your time and looking forward to participate/leverage this great piece of software.

Cheers

[Jiri Jetmar]

Hi, 

well, many aspects to cover here and yes, Qi4j contains some pretty cool concepts 

that can help you to build a scalable & flexible domain models. I guess you 

already got the concepts behind the EntityStory and EntityIndex ? 

CQRS - we are using that pattern on top of the above mentioned Entity Store/Index.

"As I will hire resource to develop this component I am looking for a framework that I can leverage to build this component."

I did the same about one year ago. Just make sure you got the concepts behind Qi4j as the guys that you are hiring will keep ask you about this and that.. :-)

As a summary - how does your architecture look like ?  Maybe you want to post here some slides that describes your architectural building blocks ?

Cheers, 

jj

--
You received this message because you are subscribed to the Google Groups "qi4j-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qi4j-dev+u...@googlegroups.com.
To post to this group, send email to qi4j...@googlegroups.com.
Visit this group at http://groups.google.com/group/qi4j-dev.
For more options, visit https://groups.google.com/d/optout.

[polymorph]

Hi Juergen,

Thanks for your quick reply. I have to admit that I tried as much as possible to read (and understand) the documentation but there is still a lot of concepts I am missing :-S

Entity Story and Entity Index are part of this unclear (for the moment) concepts; I do not see documentation on those concepts and reading the code can help but not the easiest way to understand the design and its building concepts. Any advise to help to catch up on this will be more than welcome!

As shared I am currently prototyping and the architecture evolved even though the building blocks should be; a cloud infrastructure, MDSD, SOI, Entity Manager, Reactive UI backend, WebApp ...

The MDSD building block should manage the models and their evolution an while generate code for the other building blocks (mainly for the Entity Manager).

Hope that this clarify a bit :-)

Kind regards,

Arnaud

[Jiri Jetmar]

Hi Arnaud, 

EntityStory + Index - the basic motivation behind this is simple - Use the right tool for the Job. I;m sure you know situation where a repository, lets say a "Oracle" is used for many different aspects at the same time - like high throughput transaction processing, high scalability requirements and not to forgot these guys from the Reporting Department that are execution Queries with 20+ SQL-Joins to collect their data from that repository for the needed Reports. A single repository can not fulfill all this requirements at the same time. I know one can argue to use e.g. some replication pattern to move data from A to B and then use the B-Replica for e.g. Reports, etc.. But at the end this are somehow infrastructural "hacks".. 

Qi4j is distinguishing between a EntityStore that is responsible for the storage of Data and a EntityIndex that is responsible for Indexing and therefore for searching (or query) of that Data. The structure of that Data are expressed/defined using the Entity concept. That is very simple, flexible and pretty powerful. E.g. Qi4j does not need any OR-Mappings with all their "magic".

This EntityStore and Index are conceptual patterns, and Qi4j abstracts using that concept the underlying concrete repository technologies. 

We are using for example during the development a simple InMemoryStore and for production a Postgresql and RIAK. A Postgresql offers consistency and transactional support and therefore this EntityStory is used for building blocks like Accounting, Billing, etc. The NoSQL Repository RIAK is used for those building blocks that does require a strict consistency. 

EntityIndex is used by the Qi4j Query API (pretty cool thing) to query for something. The query itself is delegated to the underlying indexing technology - we are using ElasticSearch. In our case the Queries are part of the Domain Model. 

Pls note that during the development of the Domain Model you do not care a lot about the underlying repository technologies as is abstracted by Qi4j. I remember many times where my team members asked for the "configuration" of the Entity Store as they could not find it.. :) In fact this are just few lines (in some cases just 1x line of code) to define a EntityStore. 

I hope I could help a bit. Regarding the "missing" documentation. Yes, you are 100% right and we are trying to improve it in the near future. 

Cheers, 

Jiri

[Jiri Jetmar]

The NoSQL Repository RIAK is used for those building blocks that does require a strict consistency.

"does not"

[Paul Merlin]

Hey Arnaud,

One thing Qi4j tries to do is to help you postpone a lot of decisions. Like Jiri said, you code against abstractions (ie. EntityStore/Index) and this is only at assembly time that you decide what implementation to use. This allow the use of in-memory (or simple filesystem-based) stores and indexes, easy/quick/fast to setup, understand etc... while using more robust, different, implementations in production.

What helped me the most when starting with Qi4j was reading the SDK unit-tests code. They show you the what and how in a pretty rude way, compared to some fancy documentation, but they are actually runnable code so they're easy to play with. Plus, this will teach you that testing Qi4j applications is damn simple.

I second what Jiri said: you should have at least one team member that's Qi4j ~fluent in order to help the others bootstrap. This very mailing list is also a good way to get help.

With regards to needed skillsets I'd say that a good knowledge of the Java language and the JVM are the only mandatory ones. Everything else depend on your use cases.

Finally, as a community we always welcome contributions of any form. From discussion, documentation, libraries or extensions to core changes proposals.
 
/Paul, happy to see you embark


polymorph a écrit :

[Niclas Hedhman]

Arnaud,

Adding to all said above; 

EntityStore vs EntityIndex

Think "World Wide Web". Website is the EntityStore, Google is the EntityIndex. You probably also realize quite quickly how impossible it would be to run that on a "all-in-one database management system".

Qi4j allows multiple "Google" and "multiple website" just like it should be.

Team Composition

When I introduced Qi4j in my team at Morgan Stanley, the "regular Joe" next to me, picked up the core concepts and approaches very quickly, and in ~2 weeks he was as productive as he ever got. And he was happy(!!!) since he could stop worrying about "serialization", "networking" and all kinds of "cruft" being in the way.

So, I think all of us would probably recommend "One Qi4j-fluent" team member, who learns all the ins-and-outs, possibly a bit ahead of time. Maybe that is You. Maybe you take help from me (available at times) or Paul (available at times), or simply learn the hard way with help of mailing list.

Contribution back

Well, we are a small, but dedicated community. We don't stand on the barricades and scream out our message on how great Qi4j is. We use it, because we find that it provides raw value without the network effect of "many other users", which is required for projects like Hadoop or Ruby on Rails.

The obvious first "contributions" will be technical questions that documentation is completely lacking. So, instead of simply answering the question in mailing list, we should make it a habit to add the answer to docs at the same time.

Secondly, you will probably find bugs sooner or later. Reporting them, preferably with a reproducible testcase, will be tremendous help. We try to not let bug reports lay around very long.

Documentation is always welcome in every project. Here too... We write docs in plain text Asciidoc, which is then generated into static text locally.

Some libraries and extensions need a bit of care. Assuming "ownership" of those would be great.

Beyond that, things that have been discussed and pondered over the last half year or so is the advent of Java 8. I think we should start working on a Qi4j 3.0, which is solely based on Java 8, and we fully leverage closures (Qi4j has its own functional approach, also extended to some io operations), and ensure that we can be better at combining Java and Scala, in the same application (proof of concept exists).

Architecture and Design

It depends a lot on what you are trying to achieve, whether Qi4j is a good fit or not. If you are building a bond market prop-trading engine, you are probably better off without the execution overhead in Qi4j. 

If the model you try to build is well understood, and has strong performance criteria, then Qi4j is probably not the best tool.

OTOH, if the model is "unknown" and expected to change a lot, especially initially, then Qi4j excels. Even in the long run, Qi4j has built-in Data Migration support, so that old entities can be on-the-fly upgraded as part of normal execution.

IF you are not sure which storage solution you want to work with, Qi4j allows you to delay that decision until later. Your code is practically agnostic to storage solution, AND the query engine.

Primarily, Qi4j gives you the choice to delay many of the "hard production questions" to much later in the development cycle. It allows you to make drastic changes to your model, without too much re-write of boilerplate code that you forget to update. It frees you from many of the mundane infra-related tasks, and let you focus on your actual domain model.

Stumbling Blocks

A few things are quite common to get going.

1. Getting the application architecture (layers and modules) right. Both "too many" as well as "too few" layers can be found. Examples try to show, 4-5 layers should do... Configuration at the bottom because many entity stores need configuration. Infrastructure such as EntityStores and Indexing on top of that. Then the Domain layer where all your interesting code end up. You may have an "orchestration" or "services" layer, and you may have an "Interfaces" (web, rest, mail, smoke signals) layer at the top. Often, people have problem with proper Visibility and set it higher than it should be. Every time you increase from the default "module" you should ask if that is the right thing to do, or whether you should have a repository that is visible, and it deals with all the access within the module.

2. Thinking in Queries. Most of us come from a SQL world, where everything are queries. So we chuck everything into the store and later figure out how to retrieve it. Instead, imagine that there are no queries at all, and you need to solve the domain model "in memory" and only with Java objects. You would aggregate and reference things as they are in the domain model, and you will realize that there are actually much fewer cases when Query is really needed. Our "Volatility Inference Engine", which took option quotes from the markets and computed the implied volatility and constructed a so called "volatility surface" (3 dimension graph of strike price(x), expiry(y) date and value(z), had NO queries in it all.

3. If you are doing DDD, not enough focus on "The Aggregate" and understanding its implications. Read Vaughn Vernon's book "Implementing Domain-Driven Design", which has a good chapter on "Aggregate" (well, I wrote a big chunk of it...). The Aggregate in Qi4j is supported, but in an odd way. The @Aggregated annotation will indicate that the referenced entity will be removed when the parent is removed. Which is kind of "ok" but not complete, since Qi4j doesn't refuse the child to be referenced more than once. But, Aggregate can almost always be "composed" into a single entity. For non-Qi4j developers this is a very foreign concept. But effectively, all the containing entities are effectively collapsed into a single object in terms of storage, but can be used as the individual parts. Yummy.

Please keep us informed of your thinking and eventual path forward, even if it doesn't include Qi4j

Cheers

Niclas

--

You received this message because you are subscribed to the Google Groups "qi4j-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qi4j-dev+u...@googlegroups.com.
To post to this group, send email to qi4j...@googlegroups.com.
Visit this group at http://groups.google.com/group/qi4j-dev.
For more options, visit https://groups.google.com/d/optout.

--
Niclas Hedhman, Software Developer
河南南路555弄15号1901室。
http://www.qi4j.org - New Energy for Java

I live here; http://tinyurl.com/3xugrbk
I work here; http://tinyurl.com/6a2pl4j
I relax here; http://tinyurl.com/2cgsug