TimerService error

[Stephen Sill II]

I have a wildfly 37 application that uses an injected TimerService that get uses the default infinspan persistence.

I have an @Scheduled task that runs every minute that checks the database for task definitions and compares those to the list of running timer tasks.

I don't frequently change these scheduled tasks so I've not seen this issue before now.

Essentially the database contains some tasks specifications that have a name and a cron schedule as well as whether or not the task is enabled.

Every minute the @Scheduled task wakes up, it asks the TimerService for all it's configured tasks.  It compares this in-memory task list against the specifications loaded from the db.  if it finds one not running it schedules one.  If it finds one running it compares the schedule against the one in the db.  If the one in the db is different it cancels the running task and schedules a new one with the updated schedule.  likewise if it finds a running task the specification stored in db is not enabled, it simply cancels that running task. 

The problem I'm having is that when it finds the case the cron schedule has changed, cancels the task and attempts to schedule a new one, it gets the following error.

[pod/logi-jlogi-7d56669b97-82459/logi-jlogi] 2025-10-24 20:06:00,070 ERROR [org.jboss.as.ejb3.timer] (TimerScheduler - 3) WFLYEJB0020: Error invoking timeout for timer: {component=jlogi-ear.jlogi-ejb.SchedulerServiceEJB, id=b8566faf-913f-4b20-8d4e-7c7f67156c63}: java.util.concurrent.ExecutionException: jakarta.ejb.EJBException: java.lang.IllegalStateException: TransactionImpl{xid=Xid{formatId=1, globalTransactionId=A20DB98AB7D98B64A030FB7DF9E84D4300000000000010BA,branchQualifier=A20DB98AB7D98B64A030FB7DF9E84D4300000000000010BA}, status=COMMITTED}

And then further down in the exception.

[pod/logi-jlogi-7d56669b97-82459/logi-jlogi] 2025-10-24 20:06:00,074 INFO  [org.jboss.as.ejb3.timer] (TimerScheduler - 3) WFLYEJB0021: Timer: {component=jlogi-ear.jlogi-ejb.SchedulerServiceEJB, id=b8566faf-913f-4b20-8d4e-7c7f67156c63} will be retried
[pod/logi-jlogi-7d56669b97-82459/logi-jlogi] 2025-10-24 20:06:00,080 WARN  [org.infinispan.transaction.impl.TransactionTable] (TimerScheduler - 3) ISPN000101: Failed synchronization registration: java.lang.IllegalStateException: Transaction is done. Cannot register any more synchronization

The problem comes in this snippet of code

                                        // running timer represents a non-disabled task specification
                                        // now need to see if the running timer spec matches the db one,
                                        // in particular the schedule
                                        if (!taskSpecFromTimerInfo.getCronExpression().equals(taskSpecFromDb.getCronExpression()))
                                        {
                                            // the schedule in the task spec loaded
                                            // from db doesn't match the one
                                            // from the running Timer so we'll
                                            // be canceling the existing one
                                            // and rescheduling with the new
                                            // schedule
                                            LOG.info(String.format("bootstrapTasks: Running timer found for existing task specification %s but schedule changed from \"%s\" to \"%s\", rescheduling",
                                                    taskSpecFromTimerInfo.getName(),
                                                    taskSpecFromTimerInfo.getCronExpression(),
                                                    taskSpecFromDb.getCronExpression()));
                                            if (oobTimer != null)
                                            {
                                                LOG.info(String.format("bootstrapTasks: Timer with id %s with old schedule for task specification %s has been cancelled",
                                                        oobTimer.getId(),
                                                        taskSpecFromTimerInfo.getName()));
                                                oobTimer.suspend();
                                            }
                                            t.cancel();

                                            // create new task info json from the db version of the
                                            // task specification
                                            String newTaskInfo = om.writeValueAsString(taskSpecFromDb);
                                            ScheduleExpression jobSched
                                                    = getScheduleExpressionFromCronExpression(taskSpecFromDb.getCronExpression());
                                            TimerConfig tc = new TimerConfig(newTaskInfo, true);
                                            Timer newt = timerSvc.createCalendarTimer(jobSched, tc); // EXCEPTION is thrown here
                                            if (newt instanceof OOBTimer newtOob)
                                            {
                                                LOG.info(String.format("bootstrapTasks: Timer with id %s with new schedule for task specification %s has been scheduled",
                                                        newtOob.getId(),
                                                        taskSpecFromTimerInfo.getName()));
                                            }

Any thoughts?

[Stephen Sill II]

This is all kicked off from an @Schedule annotation method that's defined as persistent so it will only run on one of the wildfly hosts in the HA cluster.

    /**
     * Runs periodically to the Timers that TimerService has to what's in
     * jlogi__task_specification table. It also synchronizes between that table
     * and TaskSpecificationDefaults which holds the authoritative list of task
     * specifications that should be present.
     *
     * @see TaskSpecificationDefaults
     */
    @Schedule(minute = "*", hour = "*", persistent = true)
    @Override
    public void bootstrapAndSynchronizeTasks()
    {