Extending Clojure's STM with external transactions

[Dave Griffith]

It often happens that I would like to have gaurantees about the
consistency of in-memory data structures and external resources. For
instance, in the last two large systems I've built, the system would
respond to an external message received via socket, do some complex
processing based on the contents of the message which modified it's
internal state, write a log of the message to a database and update
some other database tables so that the internal state of the system
was reconstructable in case of crash, and then send some other
messages to clients via socket. This all had to either work, or not
work, with full ACID properties for the system as a whole (both the
database and memory). This was an enormous pain, requiring in-memory
rollback functionality to be coded by hand for each and every possible
interaction.

It seems like it should be possible to extend the STM semantics of
Clojure to allow this sort of thing to Just Work. From the user point
of view, one would call dosync-external, rather than dosync, and pass
it an external (JTA?) transaction manager along with the operations to
be executed. If either the in-memory operations or the external
operations required rollback, then both would be, otherwise both would
be committed. If committed, any agent sends that the transaction had
sent would occur. Simple to use, and pretty much exactly what's
required for a high-reliability message processing engine. Literally
thousands of lines of code would disappear from each of my last two
projects.

Looking over the current implementation of Clojure, I think I've
convinced myself of the following facts.

1) This can't be done purely in Clojure code itself, as the necessary
hooks into the STM don't exist (nor, I would argue, should they).

2) This could be done pretty easily by hacking the Clojure runtime, at
a (obviously excessive) cost of inducing a hard dependency between the
core runtime and some particular transaction manager. The STM
implementation already has enough of a two-phase commit structure to
make it pretty easy to integrate with any given transaction management
system.

3) Creating some sort of plugin system so that arbitrary transaction
managers could be integrated with the Clojure STM runtime is doable,
but doesn't much fit with the current architecture.

4) If one were to do such a thing, it would be pretty easy to extend
the STM with features like timeouts and such.

Thoughts? Am I missing anything? Has this already been attempted in
some way I was unable to google for? My temptation is to try to
implement something as a quick spike and see how it works.

--Dave Griffith

[Rich Hickey]

It's definitely an interesting idea - I don't think anyone is working
on it. I don't have enough knowledge of JTA/XA to do it myself, but
could provide advice about integration with the STM.

Rich

[Razvan Ludvig]

Hi Dave. I think your proposal would be useful to have in Clojure, I
have thought about something similar since I read about the STM. But I
also think there are quite a few difficulties in implementing this in
a "sane" way. Actually, the more I think about it, the more it seems
that the goal of STM (to aid concurrent programming at a micro-level)
is not really compatible with that of traditional transactions (to
enable persistent and consistent changes to a shared resource, at an
application/system level). Here are some observations:

1. Normal transactions
A successful commit on an external transaction can be (and usually is)
a side-effect, so it breaks the assumption that transactional
operations are side-effect free and can be retried by the STM. Thus
Clojure would need to recognize this as a special case and treat it as
a "composite" transaction, which can not be retried entirely. I don't
know how STM is implemented in Clojure. Maybe if the external commit
can be delayed until the end of the transaction, when it is certain
that the in-memory operations succeeded, it could work. I think this
is the most promising way of implementing this.

2. XA transactions
If Clojure's STM works similar to an XA transaction, it would still be
difficult to delegate the coordination to a JTA transaction manager.
XA-capable resources MUST have stable storage for crash recovery,
which definitely isn't the case (and also wouldn't make sense) with
the STM. The STM guarantees ACI, external transactions are usually
ACID, so there's a conceptual mismatch between the two. If you think
about it, why would you want to have transactions between a transient
and a durable resource ? Also, XA resources must be prepared to handle
the case of an "heuristic commit", when all resources confirmed the
"prepare" stage, but not the final "commit". This situation does not
exist with pure STM, so one would have to be aware of the risk and
settle for less guarantees when using "composite" transactions.

Razvan.

[Dave Griffith]

> Maybe if the external commit
> can be delayed until the end of the transaction, when it is certain
> that the in-memory operations succeeded, it could work. I think this
> is the most promising way of implementing this.

This was my plan. As near as I can tell, the current STM
implementation commits by
1) locking all the written refs
2) checking if the writes of the transactions are non-conflicting. If
any conflict, retry the whole transaction
3) then writing to the refs
4) unlocking the refs

The idea is that between 3) and 4) you would attempt to commit any
external transactions. If that requires a rollback, rollback the in-
memory transaction as well. Similarly, if there are any conflicts in
2), rollback any external transactions.

> The STM guarantees ACI, external transactions are usually
> ACID, so there's a conceptual mismatch between the two. If you think
> about it, why would you want to have transactions between a transient
> and a durable resource ?

The idea is that there would be ACI gaurantees not just within the
STM, but also between the STM and the external resource. This doesn't
imply any D properties on the in-memory structures. The program would
have to manage that itself, say by reloading the necessary structure
from the external resource at start up for crash recovery.

>Also, XA resources must be prepared to handle
> the case of an "heuristic commit", when all resources confirmed the
> "prepare" stage, but not the final "commit". This situation does not
> exist with pure STM, so one would have to be aware of the risk and
> settle for less guarantees when using "composite" transactions.

Hurm. True. You would still have ACI within the in-memory
structures, but could have inconsistency between the in-memory
structures and the external resource.

[Dave Griffith]

Okay, hacking complete. I've got a patch that extends the Clojure STM
so that it will make appropriate callouts to an external transaction
manager that cause the STMs transaction to be atomic/consistent/
isolated with respect to an external transaction, using a "dosync-
external" macro. Note that this is not the same as XA support. The
Clojure STM is in charge of the transaction as a whole, not the
external transaction manager. (Among other things, this means that
transactions can not span multiple transaction managers. You'll get
an exception if you try.) The external transaction manager must
implement the clojure.lang.ITransactionManager interface, which means
that to use any given external transaction manager you will need to
implement a small adapter to that interface, and instantiate it
appropriately.

This code is highly experimental, and should not be used for
production purposes. Contents may have settled during shipping. All
models over 18.

Where should I send the patch?

--Dave Griffith

[Razvan Ludvig]

Cool stuff, Dave. I'm interested to see/test it, could you post it as
an attachment to this group or commit it to contrib ?

Cheers,
Razvan.

[Dave Griffith]

Since it requires changes to the Clojure runtime, it probably doesn't
make much sense to put it in contrib. I've posted it as an attachment
to the group.

--Dave Griffith