Andrew,
I'm glad the STM is working well for you.
beforeCommit runs just before the STM tries to commit the transaction, as if it is the last action in the body of the outermost atomic block.
1. No, the transaction might still fail during or after beforeCommit has run.
2. The most likely scenario for a failure after beforeCommit has executed is that some of the transaction's reads were stale, but it wasn't discovered until the final commit check. Also possible: a higher priority transaction has stolen write permission from the current transaction, a later callback caused a failure, or there is an ExternalDecider that decided not to commit.
3. Yes, beforeCommit can be retried.
4. Yes, if an exception is thrown from beforeCommit then the transaction will be rolled back and not retried. This is basically the same as if the exception was thrown from inside the atomic block itself (except that we don't try to catch control flow exceptions from in the beforeCommit stage).
whilePreparing handlers are closer to what you are asking for. They run after the STM is satisfied that commit is allowable (but still not "decided"), so only other handlers can cause rollback. If you are integrating with a transactional resource like a DB then this is the right time to call XAResource.prepare() (with the followup in an afterCommit handler).
The single moment at which the commit decision is made can be controlled by setting an ExternalDecider. If you are trying to integrate with a non-transactional resource that might fail, this is the only option. The primary flaw is that there can be only one. If you are okay with the non-composability then you could also integrate a DB here with a single-phase database commit (just JDBC .commit()).
whileCommitting handlers run after commit is guaranteed but before the transaction's writes have become visible to other threads. They don't have the ability to roll the transaction back, however.
Can you describe a bit more about what you want to run in the handler?
Thanks,
Nathan