Support for pure nosql

[coderinabstract]

Question on future

What is the current thinking of supporting a no-sql backbone for all dimensions of Axon, eventStore, Saga persistance, Scheduleing event persistance, query db repository with a distributed nosql stack.

Seems like one platform end to end would really enable multinode footprint for no sql infrastructure and support availability. Also, load balancing multiple jvms with disteibuted bus and message queuing persitance for multi-node bus is a ?

Could Mongo provide full end to end capability where no sql jpa infrastructure is needed. What would be the alternative to Quartz? especially in multi-node environment?

Thoughts?

[Allard Buijze]

Hi,

Mongo support is going to be supported (meaning not in incubator anymore) from Axon 2. In fact, yesterday I've spent some time on building a Serializer that serializes any object to a Mongo BSON model, so that they can be stored as native Mongo documents (in fact as part of a document). Meanwhile, Jettro is busy implementing the SagaRepository using Mongo. It's already in the incubator of 1.3. For an example, check out the Axon Trader sample application: https://github.com/AxonFramework/Axon-trader

As for scalability related command and event busses, some will be included in 2.0 as well. I've already built the dynamically scalable distributed commandbus, that does automatic routing of commands for optimal performance. We'll also soon start with an AMQP-based event bus soon. It will allow you to share events among different nodes using an AMQP compatible Message Broker, like RabbitMQ.

An alternative for Quartz wouldn't really be needed. Quartz has pluggable storage, so you could use MongoDB to store the jobs for quartz. An Quartz is also capable of working in multi-node environments.

I can't really give a timeline for all this yet. It depends a bit on the amount of time I can spend on it. I'm working on a sponsoring deal that will allow me to speed up things a bit. I'm hoping to be ready this summer.

Cheers,

Allard

[coderinabstract]

Thanks Allard... lot of work in progress and exciting stuff...

One more question is how to manage Axon scale with traditional java web app jvm hosting webservices/api end points or web app site. 

Together or separate configuration options. Elaborating on those design techniques for scalability also is a interesting area.

e.g. traditional webapp compile could run axon within jvm, then how to scale horizontally with that topology.

Alternatively webapp scaling could be seperated from axon i.e. separate jvms and remote connectivity into different types of clusters one for webapps and one for the axon jvms and another one for persistance multinode backbone.

These are some important design paradigms to consider and solve for dimensions. Any best practice for these too would be key for broader developer education and adoption and any framework simplification would also help adoption curve.

Thanks again for all the exciting work in this area.

Cheers

[Roland Kofler]

I am curious to know your motivation for nosql event-stores.
To me a file is the perfect store for ARs.
The notion of a FileStream, the way you read it, the way you append
events, fits perfectly.
There are small concerns though about integrity across AR's. But the
problem did not arise to me yet.
I've started to use CQRS, thanks to axon, on two projects, until now I
really don't see the benefit of (no)sqldata-bases on the command side.
On the read-side I use a nice, productive, mongodb.
And there is really no need for a CQRS-framework that implements read-
model stuff.

just my 2 cents
ro

On Feb 19, 12:01 am, coderinabstract <coderinabstr...@gmail.com>
wrote:

[coderinabstract]

This is a philosophical question... I will try to be breif

My 2 cents.. The main motivation is for horizontal scale, redudancy and availaibility of the AR/SAGA/Scheduled Event data along with read only... and drive for consistency for persistance across cqrs stores and read only stores to simplify persistance management for all data in a multi node infrastructure... and simplify maintenance and management for availability/redundancy and scalability

[Allard Buijze]

Hi,

the file system event store may be a fast and simple one, but it doesn't provide any failover capabilities. It simply requires you to always send commands for one aggregate to the machine that hosts the event store for that partition. You could share the event store using an NFS, but you're probably shooting yourself in the foot there.

Nosql databases acknowledge the need for certain scalability aspects (see CAP theorem) very well. Mongo is one that fits the typical Event Store requirements (CA, no P). Having them will allow you to scale dynamically as well as provide an easy infrastructure for failover.

Cheers,

Allard

[Roland Kofler]

Thanks, I understand the CA(P) idea. Still I suspect there is a
simpler solution to reach consistency and availability than using a
database that is build for a different use-case. We have done this for
decades with objects in SQL, and now we should not repeat the same
mistake with Events in NoSQL. As a business leaning dev I am weak in
the technical details but consistency and availability of a file can't
be a harder problem than with a document database. Given the
prominence of files in unix systems I can not get rid of the
impression: this is a solved problem. Again as a layman: SAN or NAS
e.g. provide automatic replication.

This is just to evolve the idea, getting a clearer picture...

ro

On Feb 22, 8:47 am, Allard Buijze <bui...@gmail.com> wrote:
> Hi,
>
> the file system event store may be a fast and simple one, but it doesn't
> provide any failover capabilities. It simply requires you to always send
> commands for one aggregate to the machine that hosts the event store for
> that partition. You could share the event store using an NFS, but you're
> probably shooting yourself in the foot there.
>
> Nosql databases acknowledge the need for certain scalability aspects (see
> CAP theorem) very well. Mongo is one that fits the typical Event Store
> requirements (CA, no P). Having them will allow you to scale dynamically as
> well as provide an easy infrastructure for failover.
>
> Cheers,
>
> Allard
>

[Allard Buijze]

Hi Roland,

document databases like MongoDB are a much better match for storing Object than relational databases. Other than that, I don't really see a problem in using databases for storing application state. The problem is in the devs, that expect not to have to think about storage at all. ORM is a craft, not a tool.

It might be possible to use a NAS and a file-based event store to achieve scalability. However, I seriously doubt that it will be any easier than using a Database. Mongo solves much of the scalability, replication and sharding issues that you need to do yourself when using a NAS. I wouldn't go there....

Scalability goes beyond replication.

Cheers,

Allard

[coderinabstract]

I know its not quite summer yet... Any pointers on how to pilot multjvm webapp with embedded axon to work in distributed node environment? Everything works great in 1 JVM, need to learn to start figuring out multinode.. 

Sample use-case..

1) need single webapp like axon-trader on two tomcat or any other webserver jvm load balanced nodes

2) traffic hitting each webapp should work for full command bus and related autonomity on the same jvm  to manage transactions and complete all event handlers before commiting (axon magic)

3) removal of one  node from the webapp cluster should keep app running smoothly

4) scale to 3 or 4 nodes.

Now behind the scenes multinode cluster for mongo or mysql, AMQP cluster for message processing and pegging command bus for full eventhandler chain on one aggregate in one machine, while queuing subsequent incoming domain commands to same aggregate and keeping update and eventhandler integrity for the same aggregate, multinode redundancy at command bus level. Key here is blocking the second thread from returning untill all eventhandlers of 1st request from 1s user and 2nd request from 2nd user are done (this is key Axon feature and not conceivable in pure async event handler design).

Any ideas or pointers.... could  not quite figure it out in axon-trader... seems like things have been quiet for couple of months.

Thanks and Cheers...

R

On Saturday, February 18, 2012 9:30:11 AM UTC-5, Allard wrote:

Hi,

Mongo support is going to be supported (meaning not in incubator anymore) from Axon 2. In fact, yesterday I've spent some time on building a Serializer that serializes any object to a Mongo BSON model, so that they can be stored as native Mongo documents (in fact as part of a document). Meanwhile, Jettro is busy implementing the SagaRepository using Mongo. It's already in the incubator of 1.3. For an example, check out the Axon Trader sample application: https://github.com/AxonFramework/Axon-trader

As for scalability related command and event busses, some will be included in 2.0 as well. I've already built the dynamically scalable distributed commandbus, that does automatic routing of commands for optimal performance. We'll also soon start with an AMQP-based event bus soon. It will allow you to share events among different nodes using an AMQP compatible Message Broker, like RabbitMQ.

An alternative for Quartz wouldn't really be needed. Quartz has pluggable storage, so you could use MongoDB to store the jobs for quartz. An Quartz is also capable of working in multi-node environments.

I can't really give a timeline for all this yet. It depends a bit on the amount of time I can spend on it. I'm working on a sponsoring deal that will allow me to speed up things a bit. I'm hoping to be ready this summer.

Cheers,

Allard

[coderinabstract]

Reviewed disruptor and jgroups base distributed bus code... interesting stuff there... any priority on Spring Integrtion with rabbitmq multinode clustering based distributed bus or am I missing something in connecting the dots... thanks and Cheers

[Allard Buijze]

Hi,

Axon trader is still an single-JVM solution, but we will start migrating that to Axon 2.0 shortly. We can only start that when the Axon 2.0 API is completely final. The idea is to base the migration guide on the findings we have there.

The distributed command bus is as good as finished. It can use Jgroups to dynamically connect and distribute command across members. That means that when a member drops, its share of the load is distributed over the remaining members.

On the Event Bus side, I am working on AMQP connectors. At a client, where I am helping out with scaling their application, I noticed that using Spring AMQP provides a lot of features that deal with resilience, such as automatically reconnecting when a connection is lost. So the first solution will probably build on that. You can always use the Spring Integration connectors (that have been around since the early Axon days) to connect with other protocols.

The use case you describe can be easily achieved by setting up a distributed command bus on each Tomcat node, with a SimpleEventBus to dispatch events locally. The distributed Command Bus will ensure that commands are consistently routed across available members. The SimpleEventBus will ensure that Event Handling (and thus query database update) is done in  consistently and in the same transaction. You can use a single (clustered) database behind the machines. 

Note that the consistency requirement puts the bottleneck in the database. If you don't need your QueryDB to be absolutely consistent, you can use an asynchronous events bus, such as the ClusteringEventBus with an AMQP connector.

Hope this helps,

Cheers,

Allard

[coderinabstract]

Thanks for the quick reply... this definately helps... quick question i had was 

that. instead of using jgroups for reselience on command bus distributed design, would Spring AMQP provide the same capability? Couldn't you use the same stack where distribution logic of the command scaling to multiple nodes for the command handling with local event bus for transaction integritiy  or distributed event bus cluster (async) can be managed by same featues of AMQP infrastructure? Could that help simplify the infrastructure setup and maintenance of these cluster types?

Again, thanks for the brillliant work on this framework.. and looking forward to multinoding...

Cheers..

On Wednesday, April 25, 2012 4:30:13 AM UTC-4, Allard wrote:

Hi,

Axon trader is still an single-JVM solution, but we will start migrating that to Axon 2.0 shortly. We can only start that when the Axon 2.0 API is completely final. The idea is to base the migration guide on the findings we have there.

The distributed command bus is as good as finished. It can use Jgroups to dynamically connect and distribute command across members. That means that when a member drops, its share of the load is distributed over the remaining members.

On the Event Bus side, I am working on AMQP connectors. At a client, where I am helping out with scaling their application, I noticed that using Spring AMQP provides a lot of features that deal with resilience, such as automatically reconnecting when a connection is lost. So the first solution will probably build on that. You can always use the Spring Integration connectors (that have been around since the early Axon days) to connect with other protocols.

The use case you describe can be easily achieved by setting up a distributed command bus on each Tomcat node, with a SimpleEventBus to dispatch events locally. The distributed Command Bus will ensure that commands are consistently routed across available members. The SimpleEventBus will ensure that Event Handling (and thus query database update) is done in  consistently and in the same transaction. You can use a single (clustered) database behind the machines. 

Note that the consistency requirement puts the bottleneck in the database. If you don't need your QueryDB to be absolutely consistent, you can use an asynchronous events bus, such as the ClusteringEventBus with an AMQP connector.

Hope this helps,

Cheers,

Allard

[Sebastian Ganslandt]

+1 

I've had thoughts in the same line. The issue I believe is that there is some more coordination to be done with the commands, making sure that commands to the same AR always ends up on the same node as long as its alive and keeping track of who's alive and which ARs they are responsible right? If the command nodes are not fully in sync as to what state they have in memory that is…

What about using an MQ for the command messages and groups just for the coordination?

[coderinabstract]

Initial thoughts is to really figure out a way to make the infrastructure experience for scaling/resilience and monitoring all of this in AMQP infra and eliminate the jgroups maintenance and management/complexity variable, however the design challenge is to figure out how. Can a dynamic pub/sub and dynamic topic subscription type 1 to1 can solve in some combination for coordination challenge (query is AR active in a  multinode queue).. something like the quartz cluster design where it does not process the same event twice if it is being processed on one node is the kind coordination solution to strive for... maybe jgroups is not a big deal?

Another thought is could a persistant cache for active ARs(Where they are and what they are doing) be done in mongo/mysql or some shared cache and a isActive check done before starting new operation and once found route it to the listener queue for that node for incoming ARs (alos these queues could be multithreaded to boost async performance). This query check has performance implications, however don't know how jGroup coordination can avoid that as synching consistency is also a challenge unless not guaranteed with transaction in persistent area, for resilience and preferably in a cache to speed things up. if mongo multi nodes were on same machine they would probably get data from that proximity for speed... ?

Just initial thoughts..

Cheers and thanks

[Allard Buijze]

Hi guys,

I understand where the idea comes from. Some colleagues have build a solution using an AMQP broker with multiple nodes processing the commands. Their conclusion: don't!

The reason is that there is a major design aspect about AMQP, that doesn't make it suitable for consistent routing. The idea behind message brokers is that it abstracts sender and receiver away from eachother. The sender just sends and "assumes" that any client will receive the message. The problem is that AMQP will just place it in a queue.You cannot dispatch consistently from a queue to its recipients. If you don't care which nodes handles which command, a single queue for them to read from is your solution. If you want to consistently route commands, AMQP is not the design you're looking for.

"and eliminate the jgroups maintenance and management/complexity". What's so complex about JGroups? All you need is to configure a list of IP addresses or a gossip router and you're set. It's really simple. And the biggest advantage is that Axon will use JGroups to dynamically (and consistently) route commands to the members. This will ensure that commands for the same aggregate will always hit the same server. So it's perfectly safe to use a cache. The power in this solution comes from the lack of the need for a centralized registry of who does what. Such a registry would only be a single point of failure, or at least a severe bottleneck.

"What about using an MQ for the command messages and groups just for the coordination?"

I don't see how this makes the world any easier. JGroups is perfectly fine for point to point communication. Adding MQ to the mix will take everything to another level of complexity where, honestly, I don't think the tools will really benefit from eachother.

Cheers,

Allard

[coderinabstract]

Agree... Feels like if the jgroups behind the curtain implementation management is hidden away with just few config ip/addresses, then it should be ok, however the resiliance and scale of that will be key (bringing down or adding nodes on the fly and down node detection to facilitate bringing up new nodes without compromising date integrity and state).

Cheers...

R

On Thursday, April 26, 2012 2:43:59 AM UTC-4, Allard wrote:

Hi guys,

I understand where the idea comes from. Some colleagues have build a solution using an AMQP broker with multiple nodes processing the commands. Their conclusion: don't!

The reason is that there is a major design aspect about AMQP, that doesn't make it suitable for consistent routing. The idea behind message brokers is that it abstracts sender and receiver away from eachother. The sender just sends and "assumes" that any client will receive the message. The problem is that AMQP will just place it in a queue.You cannot dispatch consistently from a queue to its recipients. If you don't care which nodes handles which command, a single queue for them to read from is your solution. If you want to consistently route commands, AMQP is not the design you're looking for.

"and eliminate the jgroups maintenance and management/complexity". What's so complex about JGroups? All you need is to configure a list of IP addresses or a gossip router and you're set. It's really simple. And the biggest advantage is that Axon will use JGroups to dynamically (and consistently) route commands to the members. This will ensure that commands for the same aggregate will always hit the same server. So it's perfectly safe to use a cache. The power in this solution comes from the lack of the need for a centralized registry of who does what. Such a registry would only be a single point of failure, or at least a severe bottleneck.

"What about using an MQ for the command messages and groups just for the coordination?"

I don't see how this makes the world any easier. JGroups is perfectly fine for point to point communication. Adding MQ to the mix will take everything to another level of complexity where, honestly, I don't think the tools will really benefit from eachother.

Cheers,

Allard

[Allard Buijze]

JGroups comes with a pretty good default configuration for TCP, UDP and the like. All you need to configure is IP addresses. In Axon, I will provide easy access to some of the sensible default settings. Of course, advanced tuning will always require a bit more jgroups knowledge.