ERROR quartzjobs.ExecutionJob - Execution 2838784 finishExecution: caught exception: No row with the given identifier exists: [rundeck.Orchestrator#1215784]

[fred.ell...@gmail.com]

Hi all, 

We are running Rundeck OSS 5.1.2 on RHEL8  (MariaDB database) quite happily, but once in a great while, we see what can only be described as a "false failure" of an execution.  All steps executed successfully, yet Rundeck is reporting the job exec as "failed".  The service.log shows a nice ugly stack trace that starts as follows:

[2025-04-23T11:18:28,672] ERROR quartzjobs.ExecutionJob - Failed execution 2838784 : No row with the given identifier exists: [rundeck.Orchestrator#1215784]

...

...

[2025-04-23T11:18:28,665] ERROR quartzjobs.ExecutionJob - Execution 2838784 finishExecution: caught exception: No row with the given identifier exists: [rundeck.Orchestrator#1215784]
org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [rundeck.Orchestrator#1215784]
        at org.hibernate.boot.internal.StandardEntityNotFoundDelegate.handleEntityNotFound(StandardEntityNotFoundDelegate.java:28) ~[hibernate-core-5.6.15.Final.jar!/:5.6.15.Final]
        at org.hibernate.proxy.AbstractLazyInitializer.checkTargetState(AbstractLazyInitializer.java:298) ~[hibernate-core-5.6.15.Final.jar!/:5.6.15.Final]
        at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:187) ~[hibernate-core-5.6.15.Final.jar!/:5.6.15.Final]
        at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:322) ~[hibernate-core-5.6.15.Final.jar!/:5.6.15.Final]
        at org.hibernate.proxy.pojo.bytebuddy.ByteBuddyInterceptor.intercept(ByteBuddyInterceptor.java:45) ~[hibernate-core-5.6.15.Final.jar!/:5.6.15.Final]
        at org.hibernate.proxy.ProxyConfiguration$InterceptorDispatcher.intercept(ProxyConfiguration.java:95) ~[hibernate-core-5.6.15.Final.jar!/:5.6.15.Final]
        at rundeck.Orchestrator$HibernateProxy$8sEEglHQ.getMetaClass(Unknown Source) ~[classes!/:?]

I'm attaching the fill stack trace to this post.

$ java -version
openjdk version "11.0.18" 2023-01-17
OpenJDK Runtime Environment Temurin-11.0.18+10 (build 11.0.18+10)
OpenJDK 64-Bit Server VM Temurin-11.0.18+10 (build 11.0.18+10, mixed mode)

$ uname -a
Linux rp000052387 4.18.0-553.40.1.el8_10.x86_64 #1 SMP Thu Feb 6 21:20:51 EST 2025 x86_64 x86_64 x86_64 GNU/Linux

[rac...@rundeck.com]

Hi Fred!

Is that a random issue? Does it happen in a specific job or globally? Also, do you know if that Rudeck instance is upgraded or "fresh"?

Thanks for your feedback! I'm looking into it.

Regards.

[rac...@rundeck.com]

After some research, I believe your issue might be related to this one. Does your workflow involve notifications with an orchestrator?

Regards!

[rac...@rundeck.com]

Hi Fred,

Is the backend only used by that single Rundeck instance? This error usually happens when Hibernate tries to load an entity by ID, but the row no longer exists — it might have been deleted in another transaction or never created in the first place.

Regards.

[fred.ell...@gmail.com]

Hello, the job definition is using the "Random Subset" orchestrator, which is a very common pattern in most of our jobs.  This issue is not readily reproducible, and it is not exclusive to this particular job.  In addition, it is my understanding that a user edited the job and saved it while this specific execution was in progress, which should not be an issue.

To answer your question about instances, yes, only a single Rundeck OSS Server is connected to this DB cluster.  It should be connecting to the primary DB node only.  That being said, there is a possibility that the server attempted to connect to a non-primary node if the primary was momentarily not available (not common), which is data replicated near instantaneously using the Galera clustering module.

Is there a query I can run against each DB node to verify if the data row it is looking for is present?

Thanks.

[rac...@rundeck.com]

Hi Fred,

Rundeck doesn’t support clustered DB setups. This issue happens when using backend cluster mode with multiple DB nodes. Use a single DB node or update your DB config to avoid this behavior.

You're facing this issue. (Solution here)

Best regards.

[fred.ell...@gmail.com]

Thanks, that is understood.  We are primarily using it for replication purposes for disaster recovery.  Will ensure the JDBC string makes reference to only one node in the cluster.  If we run into this issue again, what query can I run to validate data is either present or truly missing for the error exception experienced?

[rac...@rundeck.com]

Hi Fred,

Regarding this:

In addition, it is my understanding that a user edited the job and saved it while this specific execution was in progress, which should not be an issue.

This would indeed be the source. It looks like a change was done while the execution was still running, and some code attempted to lazy-load the Orchestrator row reference from the ScheduledExecution record with a cached primary key value.

As a way to determine whether the correct data is present, you can run the following SQL query:

SELECT se.id AS scheduled_execution_id, se.orchestrator_id AS referenced_orchestrator_id, o.id AS actual_orchestrator_id FROM scheduled_execution se LEFT JOIN orchestrator o ON se.orchestrator_id = o.id;

This will help you check whether the valid Orchestrator ID is present in the ScheduledExecution records. Feel free to improve/modify it. But I am fairly sure the above-mentioned modification is the cause of the issue.

Regards!

​