Handling multi-threaded writes to an Aggregate

[Simon P]

Hi Allard,

I have been using Axon for some time now to implement the Command Side of my application. In prototyping, the implementation seems to work well in processing commands that are issued by remote clients over JMS (Active MQ). 

Initially, I only needed a single thread to process messages from the internal JMS session queue - so aggregates are fetched, updated and persisted in the same thread that polls the queue. (I am using GenericJpaRepository with Optimistic Locking)

What it is clear is that as the number of message producers grow - I am going to need to employ multi-threading, which is causing lots of problems with writing to my aggregates. Pessimistic Locking slows the application down tremendously and Hibernate does not like many threads updating different attributes on my aggregate (part of your concurrent modification checking using versioning I assume). 

I know that one of the big questions to ask yourself when implementing Axon is whether the application would expect more queries that it would writes to the database. Initially, I thought this was the case - however it is clear now that my system will in fact be expected to perform many writes every second (this includes some pretty heavy business logic).

I want to carry on using Axon but I would like your thoughts as to how I might be able to get more out of the command side. The messages I am processing are real time location messages which get updated every 5 seconds or so. When I receive a position update, I update the aggregate in the repository, generate the state changed event and then an Event Handler will invoke logic between other 'moving' aggregates based on the state change.  As many as 1000 'movable' objects could update their position every second. 

Because I get an updated position so often, it makes me wonder whether I need a repository at all. A simple in memory map of my 'movable' objects with the latest position stored so that BL can be applied could be sufficient. 

Sorry to ramble on, if you have any thoughts of how best to employ Axon for this business case I would appreciate it. 

Simon

[Allard Buijze]

Hi Simon,

this is an interesting usecase. It is definitely possible to deal with a large number of commands. Do you dispatch your events with JMS as well, or do you process the events in the threads that dispatch the command? Having them processed on another thread will boost performance as well.

The high-performance processing of Commands is one of the focus points in Axon 2. For instance, there are the DisruptorCommandBus and the DistributedCommandsBus. The former uses the Disruptor (code.google.com/p/disruptor) to process Commands up to 20 times faster than the SimpleCommandBus can. The latter will allow commands to be distributed over multiple nodes.

But Axon 2 isn't finished yet, do it won't help you on the short term.

What you could do in your situation, is make sure that multiple threads process the commands, but make a clear separation about which thread will process commands for each aggregate. That will allow you to have multiple threads to process your requests, while removing the need for a locking mechanism. This probably means you need to put something in between JMS and your command bus to do this.

But on the other hand, processing 1000 commands per second should not really be a problem for the SimpleCommandBus either. Do you use a cache on your repository? If your objects live a long time, do you also have snapshotting in place? Also make sure your database has the right indexes (check the last section in the reference guide for some tips).

Hope this helps. If you need more information, don't hesitate to get in touch.

Cheers,

Allard

[Simon P]

Hi Allard,

Thanks for your response.

To be honest - I am also very surprised with the performance I am encountering. I am processing Events Asynchronously so the bottleneck is definitely between the JMS receiver and the Event Bus.

In short - I cant clear the JMS queue quick enough and the Event Processor is hardly loaded. I will know I have it right when the Event Processing is under load and the JMS queue is always close to empty.

I am using the Axon GenericJpaRepository, with Hibernate as the Jpa Implementation (Hypersonic DB). I also use Hibernate 2nd Level Caching implemented with EhCache. 

So, it should be fast, but i can't get the throughput I need. I have not looked at indexes yet - I will do that next. I would prefer to stay on the single thread if it all possible. 

I will let you know if I manage any improvement (or if I was just doing something daft)

Thanks again

[Simon P]

Hi Allard,

I have reading a lot about Hibernate 2nd level caching.

This is a particularly interesting post..

http://stackoverflow.com/questions/3997072/spring-jpa-hibernate-ehcache-poor-performance 

Seems like there is significant overhead involved using Spring Transactions with the Hibernate Cache (I am using the SpringTranactionalInterceptor bean) - performance problems could be down to this?

[Allard Buijze]

Hmm.. interesting. 

Generally, I use Event Sourcing in my own projects and use a Cache in the repository level (which just holds an object reference). And these use the SpringTransactionalInterceptor as well, with pretty good performance.

[Allard Buijze]

Something that comes to mind:

Spring's transaction manager (as any other transaction manager I know) requires a Connection to the backing database. If you retrieve objects from cache, that connection is not used. If you use connection pooling, you can get contention there. If you don't use connection pooling, you have the overhead of creating an unused connection.

Always wrap your connection pool with the LazyConnectionDataSourceProxy. That will ensure a connection is only opened when a real query is sent to the database.

Cheers,

Allard

[Simon P]

Hi Allard,

I have already implemented the  LazyConnectionDataSourceProxy - as suggested, with only a marginal performance improvement.

I am still at a very disappointing 500 Commands / Second. I am sure it is not the framework but some something at my persistence level. I will investigate more - maybe there is something I am doing in my spring-data source configuration that is slowing down the db writes. 

Thanks for your help so far, but I am now convinced its not Axon. I just mocked up a simple Command Bus Dispatcher and knocked out nearly 100,000 commands per second!

Simon

[Simon P]

Hi Allard,

I am looking at whether Event Sourcing (Cached) maybe a better solution. Importantly, I would not want to store events for the aggregate for the entire lifetime of the aggregate (many years) but simply store events from the last snapshot. The snapshot should be the state of the Aggregate at a particular point from which I can reconstruct the Aggregate with the DomainEventStream from the last snapshot should I need to.

Is this how snap-shotting is implemented in Axon? Is it possible to stipulate the maximum number of events before a snapshot is taken such that there is never more than this number of domain events associated to the aggregate? My use case is that my Aggregates live perpetually and will accrue many thousands of domain events every day. I don't want to store everything - I just want to be able to reproduce state in the event of failover.

Simon

[Allard Buijze]

Hi Simon,

normally, the event store is configured to keep all events, and store snapshots as temporary replacements. However, there is nothing wrong with customizing it to make sure snapshots are a permanent replacement for old events.

The snapshot triggering mechanism in Axon will help you with the timing of snapshot creation. You can set it to create a snapshot each -fill in the blank- events. I think default is 250 events.

Make sure you do a small PoC with default Axon components first, to find out whether it helps you with your performance issue. Otherwise you might be solving the problem in the wrong place.

Cheers,

Allard