`IllegalStateException` / lost transaction association in `org.wildfly.clustering` session batch handling (10.0.8.Final, WildFly 40)
Igors Patrejevs
Since the "share transaction context between concurrent requests for the same session" change, three code paths in org.wildfly.clustering can lose a batch's JTA transaction association on a shared thread and either throw IllegalStateException (crashing the request or the session-expiration reaper) or silently orphan a transaction. We hit this under load on WildFly 40.0.1.Final / org.wildfly.clustering 10.0.8.Final with Infinispan-backed distributable web sessions, traced it to three distinct call sites, and have working patches + reproductions for all three. Posting here for feedback before we consider carrying these as a local patch long-term.
EnvironmentWildFly 40.0.1.Final (WildFly Core 32.0.2.Final)
org.wildfly.clustering 10.0.8.Final
Infinispan 16.0.13, distributable web sessions, <distributable-sessions> cache with a JDBC passivation store
Session manager transaction mode BATCH (JTA via Narayana)
Symptom in production: intermittent HTTP 500 on ordinary requests (/, /login, /logout, static assets) under concurrent load, always tied to an IllegalStateException originating from the distributable-session batch machinery.
We did not see this on WildFly 39 with the same application and load profile. As far as we can tell, WildFly 39's session manager did not share a batch/transaction context across concurrent requests to the same session at all, so there was nothing to race — the sharing behavior (and therefore the exposure) appears to have been introduced together in one change.
Root causeBatch/SuspendedBatch exposes two ways to resume a suspended transaction on the current thread:
resume() — unsafe: if the calling thread already has its own transaction/batch associated, this throws IllegalStateException instead of doing anything about it.
resumeWithContext() — safe: auto-suspends whatever is already on the thread, resumes the target batch, and restores the previous one when the returned Context is closed.
Several places in the session-cache decorator layer call the unsafe form (or inherit a default that does), on threads where a foreign batch/transaction can legitimately already be active because of the concurrent-request sharing this library now does. We found three independent call sites that hit this:
Path 1 — CachedSessionManager's cache-removal listenerCachedSessionManager (org.wildfly.clustering.session.cache) registers a removal listener on its local session cache. When an entry is evicted/removed while a CompletionStage<CacheableSession<C>> is still pending or already completed, the listener does:
CacheableSession<C> session = stage.toCompletableFuture().join(); if (session != null) { session.resume(); // <-- unsafe Optional.ofNullable(session.get()).ifPresent(Session::close); }session.resume() (CachedSession.resume() → SuspendedBatch.resume()) throws if the thread that happens to run this removal callback already has an active batch — which is exactly the case when two concurrent requests for the same session id finish close together and the removal fires on a thread still servicing the other request. This is the dominant path we saw in production logs, and it's 100% deterministic to reproduce artificially (see below): resume a foreign suspended batch on a thread without closing it, then let any session's normal request-completion path (DistributableSession.requestDone() → ... → CachedSessionManager's removal listener) land on that same thread.
Observed stack (our reproduction sandbox, WildFly 40.0.1.Final, same shape as what we independently saw in production):
2026-07-03 21:00:37,026 ERROR [io.undertow.request] (default task-16) UT005023: Exception handling request to /batch-repro/logout: java.util.concurrent.CompletionException: java.lang.IllegalStateException: org.wildfly.clustering.cache.infinispan.batch.ThreadContextBatch$2@254b3ae5 at java.base/java.util.concurrent.CompletableFuture.wrapInCompletionException(CompletableFuture.java:323) at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:359) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:364) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:670) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2206) at org.infin...@16.0.13//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:112) at org.infin...@16.0.13//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:87) at org.infin...@16.0.13//org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:33) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2206) at org.infin...@16.0.13//org.infinispan.interceptors.impl.CacheLoaderInterceptor.finishLoadInContext(CacheLoaderInterceptor.java:357) at org.infin...@16.0.13//org.infinispan.interceptors.impl.CacheLoaderInterceptor.lambda$loadInContext$1(CacheLoaderInterceptor.java:340) ... (CompletableFuture/Infinispan interceptor chain elided) Caused by: java.lang.IllegalStateException: org.wildfly.clustering.cache.infinispan.batch.ThreadContextBatch$2@254b3ae5 at org.wildfly.clustering....@10.0.8.Final//org.wildfly.clustering.cache.infinispan.batch.ThreadContextBatch$2.resume(ThreadContextBatch.java:82) at org.wildfly.cluste...@10.0.8.Final//org.wildfly.clustering.session.cache.CachedSession.resume(CachedSession.java:47) at org.wildfly.cluste...@10.0.8.Final//org.wildfly.clustering.session.cache.CachedSessionManager$1.accept(CachedSessionManager.java:66) at org.wildfly.cluste...@10.0.8.Final//org.wildfly.clustering.session.cache.CachedSessionManager$1.accept(CachedSessionManager.java:60) at org.wildfly.clus...@10.0.8.Final//org.wildfly.clustering.server.cache.CacheStrategy$2$1.lambda$computeIfAbsent$0(CacheStrategy.java:123) at org.wildfly.clus...@10.0.8.Final//org.wildfly.clustering.server.cache.CacheStrategy$2$1.remove(CacheStrategy.java:155) at org.wildfly.clus...@10.0.8.Final//org.wildfly.clustering.server.cache.CacheStrategy$2$1.lambda$computeIfAbsent$1(CacheStrategy.java:132) at org.wildfly.cluste...@10.0.8.Final//org.wildfly.clustering.session.cache.CachedSession.close(CachedSession.java:42) at org.wildfly.clu...@10.0.8.Final//org.wildfly.clustering.function.UnaryOperator$6.apply(UnaryOperator.java:137) at org.wildfly.clu...@10.0.8.Final//org.wildfly.clustering.function.UnaryOperator$7.apply(UnaryOperator.java:156) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:667) ... 22 more Caused by: org.wildfly.clustering.cache.infinispan.batch.ContextualException: 2 ... 33 more Path 2 — CachedSession.resumeWithContext()CachedSession overrides resume() but not resumeWithContext(), so it inherits SuspendedBatch's default implementation — which just calls the unsafe resume() internally. Any caller that reasonably expects resumeWithContext() to be safe (because that's the whole point of the method) gets the same crash instead.
Path 3 — DefaultTransactionalBatch.suspend()org.wildfly.clustering.cache.infinispan.batch.DefaultTransactionalBatch#suspend():
Transaction suspendedTx = this.tm.suspend(); if (suspendedTx != this.tx) { throw new IllegalStateException(this.tx.toString()); }If, by the time this batch is suspended, the thread's actual transaction association has already diverged from what this batch object believes it owns (e.g. because something upstream already lost/changed the association — which is exactly what paths 1/2 can cause), this throws unconditionally and fatally. We saw this fire from WildFly's own SessionExpirationTask reaper thread, which is particularly bad: it doesn't just fail one request, it can abort the expiration-processing cycle for that thread.
Observed stack (our reproduction sandbox, WildFly 40.0.1.Final):
2026-07-03 18:42:13,969 WARN [org.wildfly.clustering.server.listener.ConsumerRegistry] (SessionExpirationTask - 3) TransactionImpl{xid=Xid{formatId=1, globalTransactionId=B248234BD07CB352E65EDBBE73654621000000000000168F,branchQualifier=B248234BD07CB352E65EDBBE73654621000000000000168F}, status=ACTIVE}: java.lang.IllegalStateException: TransactionImpl{xid=Xid{formatId=1, globalTransactionId=B248234BD07CB352E65EDBBE73654621000000000000168F,branchQualifier=B248234BD07CB352E65EDBBE73654621000000000000168F}, status=ACTIVE} at org.wildfly.clustering....@10.0.8.Final//org.wildfly.clustering.cache.infinispan.batch.DefaultTransactionalBatch.suspend(DefaultTransactionalBatch.java:149) at org.wildfly.clustering....@10.0.8.Final//org.wildfly.clustering.cache.infinispan.batch.DefaultTransactionalBatch.suspend(DefaultTransactionalBatch.java:29) at org.wildfly.clustering....@10.0.8.Final//org.wildfly.clustering.cache.infinispan.batch.ThreadContextBatch.suspend(ThreadContextBatch.java:64) at org.wildfly.clus...@10.0.8.Final//org.wildfly.clustering.cache.batch.Batch.suspendWithContext(Batch.java:57) at org.wildfly.clu...@10.0.8.Final//org.wildfly.clustering.context.ContextualExecutor$1.execute(ContextualExecutor.java:100) at org.wildfly.clu...@10.0.8.Final//org.wildfly.clustering.context.Contextualizer$2$4.accept(Contextualizer.java:163) at org.wildfly.clust...@40.0.1.Final//org.wildfly.clustering.web.undertow.session.UndertowSessionExpirationListener.accept(UndertowSessionExpirationListener.java:48) at org.wildfly.clust...@40.0.1.Final//org.wildfly.clustering.web.undertow.session.UndertowSessionExpirationListener.accept(UndertowSessionExpirationListener.java:26) at org.wildfly.clus...@10.0.8.Final//org.wildfly.clustering.server.listener.ConsumerRegistry$1.accept(ConsumerRegistry.java:52) at org.wildfly.clustering.se...@10.0.8.Final//org.wildfly.clustering.session.infinispan.embedded.SessionExpirationTask.test(SessionExpirationTask.java:60) at org.wildfly.clustering.se...@10.0.8.Final//org.wildfly.clustering.session.infinispan.embedded.SessionExpirationTask.test(SessionExpirationTask.java:27) at org.wildfly.clust...@10.0.8.Final//org.wildfly.clustering.server.local.scheduler.LocalSchedulerService.run(LocalSchedulerService.java:184) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:545) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:309) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) at org.wildfly.clu...@10.0.8.Final//org.wildfly.clustering.context.ContextualExecutor$1.execute(ContextualExecutor.java:94) at org.wildfly.clu...@10.0.8.Final//org.wildfly.clustering.context.Contextualizer$2$1.run(Contextualizer.java:133) at java.base/java.lang.Thread.run(Thread.java:1474)Note this particular occurrence was already caught and merely logged as a WARN by ConsumerRegistry at that call site — it doesn't always propagate to a hard failure, which is part of why frequency/impact varies by which caller happens to hit it. But the same throw, reached through a caller without an equivalent catch, is what aborts the expiration cycle for that thread.
Why this is hard to reproduce with a small isolated test caseWe want to be upfront about this: we don't have a minimal, dependency-free reproduction, and we don't think one is realistic for this class of bug. The crash requires:
A real Infinispan-backed distributable session manager wired up with JTA/BATCH transaction mode — not just a plain in-memory HttpSession.
Genuine thread contention: either two concurrent requests to the same session id completing close together on different Undertow worker threads, or WildFly's own session-expiration reaper landing on a thread another request's batch still occupies.
For the reaper path specifically, load high enough (or a worker pool small enough) that thread reuse under concurrency becomes likely within the batch's lifetime window.
None of that fits in a five-line JUnit test. What we do have, at increasing levels of realism:
A deterministic unit test proving the underlying API contract violation in isolation (no WildFly): SuspendedBatch.resume() throws when the calling thread already has an active batch; resumeWithContext() does not. This isolates the exact defect but doesn't exercise the real session-cache call sites.
A deterministic artificial repro against a live WildFly 40.0.1 instance: an endpoint that manually resumes a foreign suspended batch on the request thread without closing it, then lets a normal session request finish on that same thread. 3/3 reproductions, matches the path 1 production stack exactly.
A load-based repro against the same live instance with a real Oracle-backed session store, using a reduced Undertow worker pool (io-threads=2, task-max-threads=4) to raise the odds of thread reuse under concurrency, driving concurrent requests to the same session id plus session-expiry timing. This reproduces both path 1 and path 3 organically (no manual thread manipulation) at a rate of roughly 2 incidents per 60 seconds under our load profile.
Production logs: the actual stack traces that started this investigation match (2) and (3) exactly.
We're happy to share (1) and (2) (self-contained, no external dependencies beyond WildFly + servlet API) if useful; (3) depends on our specific application and an Oracle datasource, so it's less portable but we can describe the setup in detail if it helps confirm the mechanism.
Proposed fixesWe patched all three call sites. None of these change the transaction semantics for the non-racing case — they only change what happens when a foreign transaction is found where this code previously assumed there could only ever be none or its own.
1. CachedSessionManager's removal listener — resume safely and still close the batch (needed because resumeWithContext() only re-suspends on its own close(), it does not commit/finalize):
CacheableSession<C> session = stage.toCompletableFuture().join(); if (session != null) { try (Context<Batch> context = session.resumeWithContext()) { Optional.ofNullable(session.get()).ifPresent(Consumer.close()); context.get().close(); } }2. CachedSession.resumeWithContext() — delegate to the batch's own safe implementation instead of inheriting the unsafe default:
@Override public Context<Batch> resumeWithContext() { return this.batch.resumeWithContext(); }3. DefaultTransactionalBatch.suspend() — graceful degradation instead of a fatal throw. If a foreign transaction is found in place of this batch's own, this batch's association was already lost before suspend() was even called; we can't recover this batch's transaction, but we can avoid taking down the calling thread (and whoever owns the foreign transaction) with it:
Transaction suspendedTx = this.tm.suspend(); if (suspendedTx != this.tx) { if (suspendedTx != null) { this.tm.resume(suspendedTx); } LOGGER.log(System.Logger.Level.WARNING, "Batch {0} lost its transaction association to a foreign transaction {1}", this, suspendedTx); return this; }Full patched source for all three files, and a small script that resolves the original 10.0.8.Final artifacts from Maven, recompiles the patched sources against them, and splices the result back into the two affected jars, is available on request.
ValidationAgainst the load-based repro (reduced worker pool, same profile that reliably produced ~2 incidents/60s pre-patch):
630/630 requests succeeded, 0×HTTP 500, 0× Undertow request-processing errors.
The new WARNING in patch 3 fired 7 times over the run, exactly where the unpatched code would have thrown and killed the reaper thread.
The deterministic artificial repro for paths 1/2 (item 2 above): 3/3 "no crash" post-patch, versus 3/3 crash pre-patch.
Does this match your understanding of the intended contract between resume()/resumeWithContext() and the concurrent-session-sharing feature? We'd like feedback on whether the graceful-degradation approach in patch 3 is the right shape, or whether the maintainers would rather see the callers into suspend() fixed so a foreign transaction can never appear there in the first place — happy to help however is most useful.
Paul Ferraro
This premise is false. The batch used by the distributed session manager has no relation to the JTA transaction provided by Narayana. This seems to be an assumption based on code analysis alone.