Autopsy of Akka Persistence
Flávio W. Brasil
From: http://activate-framework.org/autopsy-akka-persistence/
Activate 1.0 was launched, people were talking about it and for my pleasant surprise Jonas Bonér, the Typesafe CTO, posted about it on twitter:
@jboner Activate Framework – durable STM with pluggable persistence: http://activate-framework.org/ #scala
Really nice! But then there is another post:
@jboner For the record: I don’t believe in durable STMs. We tried that in Akka a couple of years ago. Dropped it for a reason. Use Slick instead.
While I fully respect Jonas Bonér and the Akka team, I disagree with this. I had already read about the reasons why the Akka Persistence module was discontinued, and it does not look like an STM problem, but rather design pitfalls in this specific Akka module. This is the document produced by the Akka Team:
https://docs.google.com/document/pub?id=1c9McZsW_mXiiQRWD-trViWmllHUb0ujOrJ92f-AUWYo
The analysis focuses on two main points:
1. It is not possible to guarantee the Durable STM consistency due the absence of ACID transactions on NoSQL databases. (“No failure atomicity” and “No consistency”)
2. The Akka Persistence STM is not distributed, so it is not possible to use it with multiple virtual machines. (“No isolation” and “Lost updates”)
AUTOPSY
The second point is about an absent Akka Persistence feature, so it is not possible to do a study on it. Activate addresses this issue by providing a Coordinator to make the Durable STM distributed. More information here and here.
The first point is about problems people were running into when using Akka Persistence. So it is possible to do a study on what went wrong. To start, we should locate the old source code. The module was moved to the also discontinued akka-modules repository. This is the source code (v1.0 tag):
https://github.com/akka/akka-modules/tree/82e7dea3dd93f976a519a5615b95d52fd8e6c28b
Binaries:
http://repo.akka.io/releases/se/scalablesolutions/akka/akka-persistence-common/1.0/
Since Activate achieves a high level of consistency even when used with MongoDB (non ACID), perhaps Akka Persistence was doing something different than Activate that could produce inconsistencies.
To find this difference, we can implement a very simple atomic storage:
This storage supports only refs and uses a synchronized map of atomic integers to store them. Placement and retrieval of items in this storage are fully atomic. Since Akka Persistence’s problem was due to the lack of atomicity at NoSQL databases, this atomic storage should have no problems in the following test case.
TEST CASE
This test creates a ref with initial value 0 and run 50 threads in parallel, each one incrementing the value by 1 with a STM transaction. The ref’s final value should be 50:
AkkaPersistenceConsistencySpecs.scala
Values are verified to be not equal to the expected value (50). The console output:
1 2 3 4 | expectedCurrentValue=50akkaCurrentValue=16databaseCurrentValue=16multiverseCurrentValue=0 |
The akkaCurrentValue and databaseCurrentValue are the same, but differ from the expected value. It varies on each test execution. The multiverseCurrentValue is always zero.
Why does Akka Persistence produce inconsistencies, even using an atomic storage?
THE COMMIT FLOW PROBLEM
Akka Persistence uses the Multiverse STM. It listens to the transaction events:
1 2 3 4 5 6 7 8 | mtx.registerLifecycleListener(new TransactionLifecycleListener() { def notify(mtx: MultiverseTransaction, event: TransactionLifecycleEvent) = event match { case TransactionLifecycleEvent.PostCommit => tx.commitJta case TransactionLifecycleEvent.PreCommit => tx.commitPersistentState case TransactionLifecycleEvent.PostAbort => tx.abort case _ => {} }}) |
The persistent commit occurs when a PreCommit event is fired by Multiverse. Looking at its implementation we can see where this event is fired:
1 2 3 4 5 6 7 8 9 10 11 12 | public final void commit() { ... case Active: prepare(); ...public final void prepare() { ... case Active: try { notifyAll(TransactionLifecycleEvent.PreCommit); ... |
Before these lines there are only console log actions, so we conclude the PreCommit event is fired as the first thing on a transaction commit. This is the basic flow of a STM transaction commit:
1. Lock each transactional unit (refs) used by the transaction;
2. Validate all reads and writes, throw an exception to retry the transaction in case of inconsistency;
3. Commit the write operations values on the transactional units (refs) in memory;
4. Release locks.
As the PreCommit event occurs before this workflow, the Akka Persistence was storing values that could be invalid. If the transaction retries, the invalid value is already placed on the storage. This looks like a design error. It would be more reasonable to do it:
- After validating the transaction (2), as the storage should receive only valid items.
- Before releasing the locks (4). Otherwise, it is not possible to ensure that placement of items in the storage occurs in the same order as STM transaction commits.
Activate uses a different commit flow in order to solve this problem:
1. Lock each transactional unit (refs) used by the transaction;
2. Validate all reads and writes, throw an exception to retry the transaction if there is inconsistency;
3. Place items in the storage;
4. Commit the write operations values on the transactional units (refs) in memory;
5. Release locks.
THE DIRECT ACCESS PROBLEM
It is possible to see a Durable STM as a way to represent in memory the data that is in the storage. Each data representation can be called as a STM transactional unit. The data representation must obey two important restrictions:
1. Each data representation must be unique in-memory. If there is more than one transactional unit representing the same data, the STM logic to validate transactions can not detect a write/read conflict on the data.
2. All access to the data must be done through a transactional unit. If there is a direct read or write to the storage, again the STM validation logic can not detect reads and writes conflicts.
The first restriction was broken in the Akka Persistence 1.0-M1. The problem was solved by the StorageManager class in 1.0.
The second restriction remains broken. The Persistent* classes access the storage directly when reading values. The read is done through the transaction unit only after a write. The PersistentRef for example:
1 | def get: Option[T] = if (ref.isDefined) ref.opt else storage.getRefStorageFor(uuid) |
Ref is defined only after a write. All reads before the write are not tracked by the STM concurrency control. This is another Akka Persistence design problem.
Activate obeys these two restrictions. It uses soft references to guarantee that there is only one transactional unit for each storage data. It also guarantees that all reads and writes are made through a transactional unit. The database is only read in order to initialize the transactional unit (entity lazy initialization), or whenever the entity is reloaded by the Coordinator in case it’s been modified at another virtual machine.
CONCLUSION
The Akka Persistence inconsistency problems would be present even with full ACID storages. The same concurrency test that fails using Akka Persistence with the implemented atomic storage, passes using Activate with the non atomic storage MongoDB:
ActivateConsistencySpecs.scala
Activate adds a very high consistency level on top of Mongo, but eventually (particularly when using database eventual consistency with the coordinator) there should be inconsistencies like stale reads. It is an expected limitation of any non ACID database, Activate just minimizes it.
If your application can not have rare inconsistencies, just use Activate with ACID storages so you can have full consistency support. You can choose which modules assemble in your application. The available modules for 1.1 are Jdbc, Mongo and Prevayler. In 1.2 we will have a module to support graph databases too.
This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages. Even if the arguments presented in the Akka Team document were completely right, they should be a reason to avoid Durable STM with NoSQL databases, not to discredit Durable STMs at all.
anli
√iktor Ҡlang
The Akka Persistence inconsistency problems would be present even with full ACID storages. The same concurrency test that fails using Akka Persistence with the implemented atomic storage, passes using Activate with the non atomic storage MongoDB:
I haven't had time to analyze everything you had in there, so I might have misunderstood your solution, but here's my take on it:
Even with a store-first-commit-later approach you run in to the issue that if the commit is aborted the store needs to be reverted, which means either
A) Running the store in a transaction for the storage, and now you're in an N-phased commit area where the commit of the stm transaction also needs to commit the persistence transaction, and if the last one of them fails you're in trouble unless you have support for joining your STM transactions with your persistence transactions (XA whathaveyou)
B) The persistence backend doesn't even support transactions where now data is stored in the persistence backend prior to the STM transaction being confirmed as successful, in which case you do Queries etc in that backend you'll end up with false results or potential garbage (replication etc).
"Activate adds a very high consistency level on top of Mongo, but eventually (particularly when using database eventual consistency with the coordinator) there should be inconsistencies like stale reads. It is an expected limitation of any non ACID database, Activate just minimizes it."
Exactly, and the size of the window of badness will differ depending on which backend is used, which means that you'll never know what code to write because you do not know which guarantees are given since you do not know which backend might be used. I think we explained this quite thoroughly in our post mortem.
As for the "Coordinator" I couldn't find much in the terms of design but from what I gathered it is a Single-point-of-failure and a Single-point-of-contention which means that it will not scale, and since Akka is about scalability it's nothing we could use. (Akka Cluster currently runs on > 100 nodes with > 1000 nodes as the target).
"If your application can not have rare inconsistencies, just use Activate with ACID storages so you can have full consistency support. You can choose which modules assemble in your application. The available modules for 1.1 are Jdbc, Mongo and Prevayler. In 1.2 we will have a module to support graph databases too."
So how would you mix things that needs to be fast and can tolerate inconsistencies with things that have to be consistent? You run multiple backends? How do you scale out ACID when your coordinator is the bottleneck (and then the bottleneck of the XA even if you would partition Coordinators). And then there's the problem of the size of the window of inconsistency, how do you know how big that window will be in a production environment and how do you know how that matches up with the tolerance of the individual pieces of code.
This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages. Even if the arguments presented in the Akka Team document were completely right, they should be a reason to avoid Durable STM with NoSQL databases, not to discredit Durable STMs at all."
Well, that is _exactly_ the conclusion in the post mortem, I quote:
"To combine STM and distribution/storage, you need to have a solution where the approaches are compatible. E.g. if you look at Terracotta or Gigaspaces, they provide the basic operations to satisfy the guarantees needed for Distributed STM. Failure atomicity (transactions), various levels of locking, isolation etc."
As for Jonas' personal, general, opinion of Durable STMs, I tend to agree: The problem of the more general issue of distributed STM is that we have yet to see a scalable version of it, and to us it is of little value without scalability.
Cheers,
√
...
[Message clipped]
Viktor Klang
Director of Engineering
Flávio W. Brasil
On Wednesday, 5 de December de 2012 at 08:17, √iktor Ҡlang wrote:
Hi Flavio,sorry for the delayed reply, usually emails will take longer to be replied to the more text they contain (my time is my most precious resource)."CONCLUSIONThe Akka Persistence inconsistency problems would be present even with full ACID storages. The same concurrency test that fails using Akka Persistence with the implemented atomic storage, passes using Activate with the non atomic storage MongoDB:
ActivateConsistencySpecs.scala"
I haven't had time to analyze everything you had in there, so I might have misunderstood your solution, but here's my take on it:
Even with a store-first-commit-later approach you run in to the issue that if the commit is aborted the store needs to be reverted, which means either
A) Running the store in a transaction for the storage, and now you're in an N-phased commit area where the commit of the stm transaction also needs to commit the persistence transaction, and if the last one of them fails you're in trouble unless you have support for joining your STM transactions with your persistence transactions (XA whathaveyou)H
B) The persistence backend doesn't even support transactions where now data is stored in the persistence backend prior to the STM transaction being confirmed as successful, in which case you do Queries etc in that backend you'll end up with false results or potential garbage (replication etc).
"Activate adds a very high consistency level on top of Mongo, but eventually (particularly when using database eventual consistency with the coordinator) there should be inconsistencies like stale reads. It is an expected limitation of any non ACID database, Activate just minimizes it."
Exactly, and the size of the window of badness will differ depending on which backend is used, which means that you'll never know what code to write because you do not know which guarantees are given since you do not know which backend might be used. I think we explained this quite thoroughly in our post mortem.
As for the "Coordinator" I couldn't find much in the terms of design but from what I gathered it is a Single-point-of-failure and a Single-point-of-contention which means that it will not scale, and since Akka is about scalability it's nothing we could use. (Akka Cluster currently runs on > 100 nodes with > 1000 nodes as the target).
"If your application can not have rare inconsistencies, just use Activate with ACID storages so you can have full consistency support. You can choose which modules assemble in your application. The available modules for 1.1 are Jdbc, Mongo and Prevayler. In 1.2 we will have a module to support graph databases too."
So how would you mix things that needs to be fast and can tolerate inconsistencies with things that have to be consistent? You run multiple backends? How do you scale out ACID when your coordinator is the bottleneck (and then the bottleneck of the XA even if you would partition Coordinators). And then there's the problem of the size of the window of inconsistency, how do you know how big that window will be in a production environment and how do you know how that matches up with the tolerance of the individual pieces of code.
This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages. Even if the arguments presented in the Akka Team document were completely right, they should be a reason to avoid Durable STM with NoSQL databases, not to discredit Durable STMs at all."
Well, that is _exactly_ the conclusion in the post mortem, I quote:
"To combine STM and distribution/storage, you need to have a solution where the approaches are compatible. E.g. if you look at Terracotta or Gigaspaces, they provide the basic operations to satisfy the guarantees needed for Distributed STM. Failure atomicity (transactions), various levels of locking, isolation etc."
As for Jonas' personal, general, opinion of Durable STMs, I tend to agree: The problem of the more general issue of distributed STM is that we have yet to see a scalable version of it, and to us it is of little value without scalability.
√iktor Ҡlang
Hi Viktor,It is really nice to have your response. My considerations are above.On Wednesday, 5 de December de 2012 at 08:17, √iktor Ҡlang wrote:
Hi Flavio,sorry for the delayed reply, usually emails will take longer to be replied to the more text they contain (my time is my most precious resource)."CONCLUSIONThe Akka Persistence inconsistency problems would be present even with full ACID storages. The same concurrency test that fails using Akka Persistence with the implemented atomic storage, passes using Activate with the non atomic storage MongoDB:
ActivateConsistencySpecs.scala"
I haven't had time to analyze everything you had in there, so I might have misunderstood your solution, but here's my take on it:
Even with a store-first-commit-later approach you run in to the issue that if the commit is aborted the store needs to be reverted, which means either
It is not just a store-first-commit-later approach, the STM transaction is validated before store.
A) Running the store in a transaction for the storage, and now you're in an N-phased commit area where the commit of the stm transaction also needs to commit the persistence transaction, and if the last one of them fails you're in trouble unless you have support for joining your STM transactions with your persistence transactions (XA whathaveyou)H
Yes, if the database is ACID Activate joins the STM and database transactions using a two-phase commit. This is a part of the commit flow:1. Pre-commit the STM transaction (validate reads and writes)2. Pre-commit the database transaction (store data)3. Commit the database transaction4. Commit the STM transactionAs an optimization, the 2 and 3 steps runs as only one.If there is an error thrown by the database (2/3), the commit process is rollbacked successfully. It is not expected to have an error when committing the STM transaction in memory (4), since it is already pre committed. It will fail only if there is a memory corruption, in this case it is not possible to guarantee consistency with any approach.
If the database is non ACID, the steps 2 and 3 are substituted by the database update only. It is possible to have an error during the items placements resulting in a partial commit. It is rare to have an error at this time, since the database is schema less, but it is possible to happen.I'm using Activate in production with MongoDB with great performance for about 9 months and until now I detected only one episode of inconsistency. It wasn't a problem for the application purpose.
B) The persistence backend doesn't even support transactions where now data is stored in the persistence backend prior to the STM transaction being confirmed as successful, in which case you do Queries etc in that backend you'll end up with false results or potential garbage (replication etc).
Activate runs queries in-memory for new and dirty transactional units. They are consistent even without a database transaction running during the STM transaction. Remember that the Distributed STM also validates reads.
"Activate adds a very high consistency level on top of Mongo, but eventually (particularly when using database eventual consistency with the coordinator) there should be inconsistencies like stale reads. It is an expected limitation of any non ACID database, Activate just minimizes it."
Exactly, and the size of the window of badness will differ depending on which backend is used, which means that you'll never know what code to write because you do not know which guarantees are given since you do not know which backend might be used. I think we explained this quite thoroughly in our post mortem.
The choice of what backends support is done at assemble time, you _can_ choose what activate modules to put in your application. If you choose the ACID storages modules, there is full consistency support.Not know the level of consistency if you _choose_ a non ACID storage (for now there is only MongoDB) is an expected limitation. There are applications that can afford some inconsistency and that is why non ACID databases exist.
As for the "Coordinator" I couldn't find much in the terms of design but from what I gathered it is a Single-point-of-failure and a Single-point-of-contention which means that it will not scale, and since Akka is about scalability it's nothing we could use. (Akka Cluster currently runs on > 100 nodes with > 1000 nodes as the target).
I chose the current Coordinator implementation for the first version of it based on a real experience with a big platform for telecom business written in java that I work with. It uses a central service like the Coordinator to do optimistic locking and has great scalability. At this moment in one of our clients there are about 2200 online nodes using this central service.
There is also an item in the Activate roadmap to check alternatives to the Coordinator, like late consistency and optimistic offline lock."If your application can not have rare inconsistencies, just use Activate with ACID storages so you can have full consistency support. You can choose which modules assemble in your application. The available modules for 1.1 are Jdbc, Mongo and Prevayler. In 1.2 we will have a module to support graph databases too."
So how would you mix things that needs to be fast and can tolerate inconsistencies with things that have to be consistent? You run multiple backends? How do you scale out ACID when your coordinator is the bottleneck (and then the bottleneck of the XA even if you would partition Coordinators). And then there's the problem of the size of the window of inconsistency, how do you know how big that window will be in a production environment and how do you know how that matches up with the tolerance of the individual pieces of code.
I think that assume that the Coordinator certainly will be a bottleneck is an error, according to my experience.
The size of the window of inconsistency is zero if you choose only ACID databases and indeterminate if you use non ACID databases. Like I said, Activate just minimizes it.
This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages. Even if the arguments presented in the Akka Team document were completely right, they should be a reason to avoid Durable STM with NoSQL databases, not to discredit Durable STMs at all."
Well, that is _exactly_ the conclusion in the post mortem, I quote:
"To combine STM and distribution/storage, you need to have a solution where the approaches are compatible. E.g. if you look at Terracotta or Gigaspaces, they provide the basic operations to satisfy the guarantees needed for Distributed STM. Failure atomicity (transactions), various levels of locking, isolation etc."
The akka persistence post-mortem was based on a view obfuscated by the design problems that I explained in the article, thats why i mean by "This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages". So, considering the text context, is not the same conclusion.
As for Jonas' personal, general, opinion of Durable STMs, I tend to agree: The problem of the more general issue of distributed STM is that we have yet to see a scalable version of it, and to us it is of little value without scalability.
I am working on a persistence benchmark project with Klaus, the creator of Prevayler. The first results show a very interesting scenario where Activate has considerably better scalability when comparing to other persistence solutions. Optimistic locking with efficient memory usage provides a new form of having high transaction throughput and minimizing the conversation with the database. I expect to have the first version of the benchmark in January.
Flávio W. Brasil
On Sunday, 16 de December de 2012 at 08:47, √iktor Ҡlang wrote:
Hi Flávio!My answers inline below:On Sat, Dec 15, 2012 at 10:52 PM, Flávio W. Brasil <fwbr...@gmail.com> wrote:Hi Viktor,It is really nice to have your response. My considerations are above.On Wednesday, 5 de December de 2012 at 08:17, √iktor Ҡlang wrote:
Hi Flavio,sorry for the delayed reply, usually emails will take longer to be replied to the more text they contain (my time is my most precious resource)."CONCLUSIONThe Akka Persistence inconsistency problems would be present even with full ACID storages. The same concurrency test that fails using Akka Persistence with the implemented atomic storage, passes using Activate with the non atomic storage MongoDB:
ActivateConsistencySpecs.scala"
I haven't had time to analyze everything you had in there, so I might have misunderstood your solution, but here's my take on it:
Even with a store-first-commit-later approach you run in to the issue that if the commit is aborted the store needs to be reverted, which means either
It is not just a store-first-commit-later approach, the STM transaction is validated before store.How does it behave under contention?
A) Running the store in a transaction for the storage, and now you're in an N-phased commit area where the commit of the stm transaction also needs to commit the persistence transaction, and if the last one of them fails you're in trouble unless you have support for joining your STM transactions with your persistence transactions (XA whathaveyou)H
Yes, if the database is ACID Activate joins the STM and database transactions using a two-phase commit. This is a part of the commit flow:1. Pre-commit the STM transaction (validate reads and writes)2. Pre-commit the database transaction (store data)3. Commit the database transaction4. Commit the STM transactionAs an optimization, the 2 and 3 steps runs as only one.If there is an error thrown by the database (2/3), the commit process is rollbacked successfully. It is not expected to have an error when committing the STM transaction in memory (4), since it is already pre committed. It will fail only if there is a memory corruption, in this case it is not possible to guarantee consistency with any approach.So what about contention? Or is your STM blocking?If the database is non ACID, the steps 2 and 3 are substituted by the database update only. It is possible to have an error during the items placements resulting in a partial commit. It is rare to have an error at this time, since the database is schema less, but it is possible to happen.I'm using Activate in production with MongoDB with great performance for about 9 months and until now I detected only one episode of inconsistency. It wasn't a problem for the application purpose.For that specific application perhaps, but as a library/framework you have to design for a plethora of possible applications.
B) The persistence backend doesn't even support transactions where now data is stored in the persistence backend prior to the STM transaction being confirmed as successful, in which case you do Queries etc in that backend you'll end up with false results or potential garbage (replication etc).
Activate runs queries in-memory for new and dirty transactional units. They are consistent even without a database transaction running during the STM transaction. Remember that the Distributed STM also validates reads.And how does that behave under contention?"Activate adds a very high consistency level on top of Mongo, but eventually (particularly when using database eventual consistency with the coordinator) there should be inconsistencies like stale reads. It is an expected limitation of any non ACID database, Activate just minimizes it."
Exactly, and the size of the window of badness will differ depending on which backend is used, which means that you'll never know what code to write because you do not know which guarantees are given since you do not know which backend might be used. I think we explained this quite thoroughly in our post mortem.
The choice of what backends support is done at assemble time, you _can_ choose what activate modules to put in your application. If you choose the ACID storages modules, there is full consistency support.Not know the level of consistency if you _choose_ a non ACID storage (for now there is only MongoDB) is an expected limitation. There are applications that can afford some inconsistency and that is why non ACID databases exist.My point is still that some modules might require an ACID storage, and some a non-ACID storage (because of different requirements of performance or scalability).
As for the "Coordinator" I couldn't find much in the terms of design but from what I gathered it is a Single-point-of-failure and a Single-point-of-contention which means that it will not scale, and since Akka is about scalability it's nothing we could use. (Akka Cluster currently runs on > 100 nodes with > 1000 nodes as the target).
I chose the current Coordinator implementation for the first version of it based on a real experience with a big platform for telecom business written in java that I work with. It uses a central service like the Coordinator to do optimistic locking and has great scalability. At this moment in one of our clients there are about 2200 online nodes using this central service.How many TPS, how big transactions, how many of them span multiple nodes and with how much contention? What happens when Coordinator crashes?
There is also an item in the Activate roadmap to check alternatives to the Coordinator, like late consistency and optimistic offline lock."If your application can not have rare inconsistencies, just use Activate with ACID storages so you can have full consistency support. You can choose which modules assemble in your application. The available modules for 1.1 are Jdbc, Mongo and Prevayler. In 1.2 we will have a module to support graph databases too."
So how would you mix things that needs to be fast and can tolerate inconsistencies with things that have to be consistent? You run multiple backends? How do you scale out ACID when your coordinator is the bottleneck (and then the bottleneck of the XA even if you would partition Coordinators). And then there's the problem of the size of the window of inconsistency, how do you know how big that window will be in a production environment and how do you know how that matches up with the tolerance of the individual pieces of code.
I think that assume that the Coordinator certainly will be a bottleneck is an error, according to my experience.Then, according to your experience, what is the bottleneck? (According to my experience and a plethora of other STM users, scalability is the ultimate bottleneck once it gets contended)
The size of the window of inconsistency is zero if you choose only ACID databases and indeterminate if you use non ACID databases. Like I said, Activate just minimizes it.So, in your mind, you have seem to found some sweet-spot in the CAP theorem where you can get all three maxed?
This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages. Even if the arguments presented in the Akka Team document were completely right, they should be a reason to avoid Durable STM with NoSQL databases, not to discredit Durable STMs at all."
Well, that is _exactly_ the conclusion in the post mortem, I quote:
"To combine STM and distribution/storage, you need to have a solution where the approaches are compatible. E.g. if you look at Terracotta or Gigaspaces, they provide the basic operations to satisfy the guarantees needed for Distributed STM. Failure atomicity (transactions), various levels of locking, isolation etc."
The akka persistence post-mortem was based on a view obfuscated by the design problems that I explained in the article, thats why i mean by "This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages". So, considering the text context, is not the same conclusion.Ok. Then we'll have to agree to disagree.
√iktor Ҡlang
That doesn't answer the question.
>>>>
>>>> A) Running the store in a transaction for the storage, and now you're in an N-phased commit area where the commit of the stm transaction also needs to commit the persistence transaction, and if the last one of them fails you're in trouble unless you have support for joining your STM transactions with your persistence transactions (XA whathaveyou)H
>>>
>>> Yes, if the database is ACID Activate joins the STM and database transactions using a two-phase commit. This is a part of the commit flow:
>>>
>>> 1. Pre-commit the STM transaction (validate reads and writes)
>>> 2. Pre-commit the database transaction (store data)
>>> 3. Commit the database transaction
>>> 4. Commit the STM transaction
>>>
>>> As an optimization, the 2 and 3 steps runs as only one.
>>> If there is an error thrown by the database (2/3), the commit process is rollbacked successfully. It is not expected to have an error when committing the STM transaction in memory (4), since it is already pre committed. It will fail only if there is a memory corruption, in this case it is not possible to guarantee consistency with any approach.
>>
>>
>> So what about contention? Or is your STM blocking?
>>
>>>
>>>
>>> If the database is non ACID, the steps 2 and 3 are substituted by the database update only. It is possible to have an error during the items placements resulting in a partial commit. It is rare to have an error at this time, since the database is schema less, but it is possible to happen.
>>>
>>> I'm using Activate in production with MongoDB with great performance for about 9 months and until now I detected only one episode of inconsistency. It wasn't a problem for the application purpose.
>>
>>
>> For that specific application perhaps, but as a library/framework you have to design for a plethora of possible applications.
>
> That is the point! If you have an application that can tolerate rare inconsistencies, it is possible to use non-ACID databases, that normally has better performance. Again, that is why non-ACID databases exists. As far as I know, is not possible guarantee, with _any_ approach, total consistency using non-ACID storages. Let me know if there is some framework that provides such guarantee.
My point is that the code has to demarcate its consistency requirements, which makes composition hard, i.e. how do you handle transactions spanning multiple persistence contexts? N-phased?
> If the application can not have inconsistencies, use Activate with ACID storages. As Activate supports ACID and non-ACID storages, it can be used for a plethora of possible applications.
>>>>
>>>> B) The persistence backend doesn't even support transactions where now data is stored in the persistence backend prior to the STM transaction being confirmed as successful, in which case you do Queries etc in that backend you'll end up with false results or potential garbage (replication etc).
>>>
>>> Activate runs queries in-memory for new and dirty transactional units. They are consistent even without a database transaction running during the STM transaction. Remember that the Distributed STM also validates reads.
>>
>>
>> And how does that behave under contention?
>>
>>>>
>>>> "Activate adds a very high consistency level on top of Mongo, but eventually (particularly when using database eventual consistency with the coordinator) there should be inconsistencies like stale reads. It is an expected limitation of any non ACID database, Activate just minimizes it."
>>>>
>>>> Exactly, and the size of the window of badness will differ depending on which backend is used, which means that you'll never know what code to write because you do not know which guarantees are given since you do not know which backend might be used. I think we explained this quite thoroughly in our post mortem.
>>>
>>> The choice of what backends support is done at assemble time, you _can_ choose what activate modules to put in your application. If you choose the ACID storages modules, there is full consistency support.
>>> Not know the level of consistency if you _choose_ a non ACID storage (for now there is only MongoDB) is an expected limitation. There are applications that can afford some inconsistency and that is why non ACID databases exist.
>>
>>
>> My point is still that some modules might require an ACID storage, and some a non-ACID storage (because of different requirements of performance or scalability).
>
> It is possible to have multiple persistence contexts.
See previous comment.
>>>>
>>>> As for the "Coordinator" I couldn't find much in the terms of design but from what I gathered it is a Single-point-of-failure and a Single-point-of-contention which means that it will not scale, and since Akka is about scalability it's nothing we could use. (Akka Cluster currently runs on > 100 nodes with > 1000 nodes as the target).
>>>
>>> I chose the current Coordinator implementation for the first version of it based on a real experience with a big platform for telecom business written in java that I work with. It uses a central service like the Coordinator to do optimistic locking and has great scalability. At this moment in one of our clients there are about 2200 online nodes using this central service.
>>
>>
>> How many TPS, how big transactions, how many of them span multiple nodes and with how much contention? What happens when Coordinator crashes?
>>
>
> I don't have these statistics here. If the coordinator crashes, transactions can not commit.
That information is crucial as I am arguing against the _scalability_ of distributed stms.
>>>
>>> There is also an item in the Activate roadmap to check alternatives to the Coordinator, like late consistency and optimistic offline lock.
>>>>
>>>> "If your application can not have rare inconsistencies, just use Activate with ACID storages so you can have full consistency support. You can choose which modules assemble in your application. The available modules for 1.1 are Jdbc, Mongo and Prevayler. In 1.2 we will have a module to support graph databases too."
>>>>
>>>> So how would you mix things that needs to be fast and can tolerate inconsistencies with things that have to be consistent? You run multiple backends? How do you scale out ACID when your coordinator is the bottleneck (and then the bottleneck of the XA even if you would partition Coordinators). And then there's the problem of the size of the window of inconsistency, how do you know how big that window will be in a production environment and how do you know how that matches up with the tolerance of the individual pieces of code.
>>>
>>> I think that assume that the Coordinator certainly will be a bottleneck is an error, according to my experience.
>>
>>
>> Then, according to your experience, what is the bottleneck? (According to my experience and a plethora of other STM users, scalability is the ultimate bottleneck once it gets contended)
>
> In my experience, normally the bottleneck is the business code that runs inside the transaction.
In my stm experience the bottleneck is contention when composing nontrivial transactions.
>>>
>>> The size of the window of inconsistency is zero if you choose only ACID databases and indeterminate if you use non ACID databases. Like I said, Activate just minimizes it.
>>
>>
>> So, in your mind, you have seem to found some sweet-spot in the CAP theorem where you can get all three maxed?
>
> No. In memory, Activate has full consistency and availability and hasn't partition tolerance. Inconsistencies are produced by the conversation with storages that uses the CAP theorem differently and hasn't ACID support.
So in case of partition it just dies?
>>>>
>>>> This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages. Even if the arguments presented in the Akka Team document were completely right, they should be a reason to avoid Durable STM with NoSQL databases, not to discredit Durable STMs at all."
>>>>
>>>>
>>>> Well, that is _exactly_ the conclusion in the post mortem, I quote:
>>>>
>>>>
>>>> "To combine STM and distribution/storage, you need to have a solution where the approaches are compatible. E.g. if you look at Terracotta or Gigaspaces, they provide the basic operations to satisfy the guarantees needed for Distributed STM. Failure atomicity (transactions), various levels of locking, isolation etc."
>>>
>>> The akka persistence post-mortem was based on a view obfuscated by the design problems that I explained in the article, thats why i mean by "This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages". So, considering the text context, is not the same conclusion.
>>
>>
>> Ok. Then we'll have to agree to disagree.
>
> Do you disagree that the Akka Persistence has two fundamental design problems or that our conclusions aren't the same?
I disagree that your perceived Akka Persistence defects has anything to do with the general concept of a scalable distributed stm that uses backends with varying guarantees.
And keeping the entire database in memory is not a general solution.
Cheers,
V
>>>>> def <code style="margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;font-size:1em!important;font:inherit;vertical-align:baseline!important;border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;float:none!important;line-height:1.1em!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;text-align:left!important;width:auto!important;font-family:Consolas,Monaco,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-weight:normal!important;min-height:inherit!i
>
> ...
√iktor Ҡlang
Sorry if I sound negative, if you solve the scalability issue with distributed STMs I'll be the first to congratulate you. I am just not convinced yet.
Cheers,
V
Hi Flávio!My answers inline below:
Cheers,
√
...
mtx.registerLifecycleListener(newTransactionLifecycleListener() {defnotify(mtx<code style="margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;font-size:1em!important;font:inherit;vertical-align:baseline!important;border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;float:none!important;line-height:1.1em!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;text-align:left!important;width:auto!important;font-family:Consolas,Monaco,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-weight:bold!im
Flávio W. Brasil
>>>>
>>>> A) Running the store in a transaction for the storage, and now you're in an N-phased commit area where the commit of the stm transaction also needs to commit the persistence transaction, and if the last one of them fails you're in trouble unless you have support for joining your STM transactions with your persistence transactions (XA whathaveyou)H
>>>
>>> Yes, if the database is ACID Activate joins the STM and database transactions using a two-phase commit. This is a part of the commit flow:
>>>
>>> 1. Pre-commit the STM transaction (validate reads and writes)
>>> 2. Pre-commit the database transaction (store data)
>>> 3. Commit the database transaction
>>> 4. Commit the STM transaction
>>>
>>> As an optimization, the 2 and 3 steps runs as only one.
>>> If there is an error thrown by the database (2/3), the commit process is rollbacked successfully. It is not expected to have an error when committing the STM transaction in memory (4), since it is already pre committed. It will fail only if there is a memory corruption, in this case it is not possible to guarantee consistency with any approach.
>>
>>
>> So what about contention? Or is your STM blocking?
>>
>>>
>>>
>>> If the database is non ACID, the steps 2 and 3 are substituted by the database update only. It is possible to have an error during the items placements resulting in a partial commit. It is rare to have an error at this time, since the database is schema less, but it is possible to happen.
>>>
>>> I'm using Activate in production with MongoDB with great performance for about 9 months and until now I detected only one episode of inconsistency. It wasn't a problem for the application purpose.
>>
>>
>> For that specific application perhaps, but as a library/framework you have to design for a plethora of possible applications.
>
> That is the point! If you have an application that can tolerate rare inconsistencies, it is possible to use non-ACID databases, that normally has better performance. Again, that is why non-ACID databases exists. As far as I know, is not possible guarantee, with _any_ approach, total consistency using non-ACID storages. Let me know if there is some framework that provides such guarantee.My point is that the code has to demarcate its consistency requirements, which makes composition hard, i.e. how do you handle transactions spanning multiple persistence contexts? N-phased?
> If the application can not have inconsistencies, use Activate with ACID storages. As Activate supports ACID and non-ACID storages, it can be used for a plethora of possible applications.
>>>>
>>>> B) The persistence backend doesn't even support transactions where now data is stored in the persistence backend prior to the STM transaction being confirmed as successful, in which case you do Queries etc in that backend you'll end up with false results or potential garbage (replication etc).
>>>
>>> Activate runs queries in-memory for new and dirty transactional units. They are consistent even without a database transaction running during the STM transaction. Remember that the Distributed STM also validates reads.
>>
>>
>> And how does that behave under contention?
>>
>>>>
>>>> "Activate adds a very high consistency level on top of Mongo, but eventually (particularly when using database eventual consistency with the coordinator) there should be inconsistencies like stale reads. It is an expected limitation of any non ACID database, Activate just minimizes it."
>>>>
>>>> Exactly, and the size of the window of badness will differ depending on which backend is used, which means that you'll never know what code to write because you do not know which guarantees are given since you do not know which backend might be used. I think we explained this quite thoroughly in our post mortem.
>>>
>>> The choice of what backends support is done at assemble time, you _can_ choose what activate modules to put in your application. If you choose the ACID storages modules, there is full consistency support.
>>> Not know the level of consistency if you _choose_ a non ACID storage (for now there is only MongoDB) is an expected limitation. There are applications that can afford some inconsistency and that is why non ACID databases exist.
>>
>>
>> My point is still that some modules might require an ACID storage, and some a non-ACID storage (because of different requirements of performance or scalability).
>
> It is possible to have multiple persistence contexts.See previous comment.
>>>>
>>>> As for the "Coordinator" I couldn't find much in the terms of design but from what I gathered it is a Single-point-of-failure and a Single-point-of-contention which means that it will not scale, and since Akka is about scalability it's nothing we could use. (Akka Cluster currently runs on > 100 nodes with > 1000 nodes as the target).
>>>
>>> I chose the current Coordinator implementation for the first version of it based on a real experience with a big platform for telecom business written in java that I work with. It uses a central service like the Coordinator to do optimistic locking and has great scalability. At this moment in one of our clients there are about 2200 online nodes using this central service.
>>
>>
>> How many TPS, how big transactions, how many of them span multiple nodes and with how much contention? What happens when Coordinator crashes?
>>
>
> I don't have these statistics here. If the coordinator crashes, transactions can not commit.That information is crucial as I am arguing against the _scalability_ of distributed stms.
>>>
>>> There is also an item in the Activate roadmap to check alternatives to the Coordinator, like late consistency and optimistic offline lock.
>>>>
>>>> "If your application can not have rare inconsistencies, just use Activate with ACID storages so you can have full consistency support. You can choose which modules assemble in your application. The available modules for 1.1 are Jdbc, Mongo and Prevayler. In 1.2 we will have a module to support graph databases too."
>>>>
>>>> So how would you mix things that needs to be fast and can tolerate inconsistencies with things that have to be consistent? You run multiple backends? How do you scale out ACID when your coordinator is the bottleneck (and then the bottleneck of the XA even if you would partition Coordinators). And then there's the problem of the size of the window of inconsistency, how do you know how big that window will be in a production environment and how do you know how that matches up with the tolerance of the individual pieces of code.
>>>
>>> I think that assume that the Coordinator certainly will be a bottleneck is an error, according to my experience.
>>
>>
>> Then, according to your experience, what is the bottleneck? (According to my experience and a plethora of other STM users, scalability is the ultimate bottleneck once it gets contended)
>
> In my experience, normally the bottleneck is the business code that runs inside the transaction.In my stm experience the bottleneck is contention when composing nontrivial transactions.
>>>
>>> The size of the window of inconsistency is zero if you choose only ACID databases and indeterminate if you use non ACID databases. Like I said, Activate just minimizes it.
>>
>>
>> So, in your mind, you have seem to found some sweet-spot in the CAP theorem where you can get all three maxed?
>
> No. In memory, Activate has full consistency and availability and hasn't partition tolerance. Inconsistencies are produced by the conversation with storages that uses the CAP theorem differently and hasn't ACID support.So in case of partition it just dies?
>>>>
>>>> This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages. Even if the arguments presented in the Akka Team document were completely right, they should be a reason to avoid Durable STM with NoSQL databases, not to discredit Durable STMs at all."
>>>>
>>>>
>>>> Well, that is _exactly_ the conclusion in the post mortem, I quote:
>>>>
>>>>
>>>> "To combine STM and distribution/storage, you need to have a solution where the approaches are compatible. E.g. if you look at Terracotta or Gigaspaces, they provide the basic operations to satisfy the guarantees needed for Distributed STM. Failure atomicity (transactions), various levels of locking, isolation etc."
>>>
>>> The akka persistence post-mortem was based on a view obfuscated by the design problems that I explained in the article, thats why i mean by "This autopsy evidences that Akka Persistence has two problems that make its usage unfeasible with atomic or non atomic storages". So, considering the text context, is not the same conclusion.
>>
>>
>> Ok. Then we'll have to agree to disagree.
>
> Do you disagree that the Akka Persistence has two fundamental design problems or that our conclusions aren't the same?I disagree that your perceived Akka Persistence defects has anything to do with the general concept of a scalable distributed stm that uses backends with varying guarantees.
And keeping the entire database in memory is not a general solution.
Flávio W. Brasil
Flavio W. Brasil
Flavio W. Brasil
√iktor Ҡlang
Director of Engineering
Twitter: @viktorklang
