Unable to create cachedir error on multiple servers deployment

[Cristina Ruscau]

Hello,

I'm using rundeck docker image version 5.8.0 to do a deployment on multiple servers.

Sometimes, but not all the time (rundeck says 50% of times) it fails doing the deployment on some of the servers.

This is the error 

Execution failed: 297 in project XXXX: [Workflow result: , step failures: {1=Dispatch failed on 6 nodes: [xxxxx: PluginFailed: Unable to create cachedir: /home/rundeck/var/cache/ScriptURLNodeStepExecutor + {dataContext=MultiDataContextImpl(map={}, base=null)} ,

[...], flow control: Continue, status: failed]

Any idea how to fix this? 

Thank you very much,

Cristina Ruscau

[rac...@rundeck.com]

Hi Cristina, can you post the job output in debug mode to take a look? (please hide or redact any sensitive data/info) Do you see any clue in the service.log? (container log output in your case).

Following this:

PluginFailed: Unable to create cachedir

It seems a container's filesystem permissions issue.

Regards.

​

[Cristina Ruscau]

Hello,

I only see this in the job log after enabling debug regarding this issue (let me know if you need the full output log)

Requesting URL: https://xxxxx
Failed executing node plugin [script-file-url] on node xxxxx: java.lang.RuntimeException: Unable to create cachedir: /home/rundeck/var/cache/ScriptURLNodeStepExecutor
at org.rundeck.plugin.scriptnodestep.ScriptURLNodeStepExecutor$Downloader.downloadURLToTempFile(ScriptURLNodeStepExecutor.groovy:167)
at org.rundeck.plugin.scriptnodestep.ScriptURLNodeStepExecutor.executeScriptURL(ScriptURLNodeStepExecutor.groovy:84)
at org.rundeck.plugin.scriptnodestep.ScriptURLNodeStepExecutor$executeScriptURL$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
at org.rundeck.plugin.scriptnodestep.ScriptFileNodeStepPlugin.executeNodeStep(ScriptFileNodeStepPlugin.groovy:115)
at com.dtolabs.rundeck.core.execution.workflow.steps.node.NodeStepPluginAdapter.executeNodeStep(NodeStepPluginAdapter.java:180)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:207)
at com.dtolabs.rundeck.core.execution.dispatch.ParallelNodeDispatcher$ExecNodeStepCallable.call(ParallelNodeDispatcher.java:225)
at com.dtolabs.rundeck.core.execution.dispatch.ParallelNodeDispatcher$ExecNodeStepCallable.call(ParallelNodeDispatcher.java:192)
at com.dtolabs.rundeck.core.cli.CallableWrapperTask.execute(CallableWrapperTask.java:52)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:99)
at org.apache.tools.ant.Task.perform(Task.java:350)
at org.apache.tools.ant.taskdefs.Parallel$TaskRunnable.run(Parallel.java:454)
at java.base/java.lang.Thread.run(Thread.java:829)
Failed: PluginFailed: Unable to create cachedir: /home/rundeck/var/cache/ScriptURLNodeStepExecutor
[workflow] finishExecuteNodeStep(xxxxx): NodeDispatch: PluginFailed: Unable to create cachedir: /home/rundeck/var/cache/ScriptURLNodeStepExecutor

And this is what I notice in the container logs:

 [2025-01-31T12:07:31,558] DEBUG controllers.ExecutionController - read stream event: DefaultLogEvent{, eventType=log, loglevel=VERBOSE, datetime=Fri Jan 31 12:04:29 UTC 2025, message='1: Workflow step finished, result: Dispatch failed on 1 nodes: [xxxxxx: PluginFailed: Unable to create cachedir: /home/rundeck/var/cache/ScriptURLNodeStepExecutor + {dataContext=MultiDataContextImpl(map={}, base=null)} ]', metadata={node=ec3e3eee-c3d9-4f6e-4386-f281, stepctx=1, step=1, user=ruscau}}
 [2025-01-31T12:07:31,558] DEBUG controllers.ExecutionController - read stream event: DefaultLogEvent{, eventType=log, loglevel=VERBOSE, datetime=Fri Jan 31 12:04:29 UTC 2025, message='[workflow] Finish step: 1,NodeDispatch', metadata={node=ec3e3eee-c3d9-4f6e-4386-f281, stepctx=1, step=1, user=ruscau}}
[2025-01-31T12:07:31,559] DEBUG controllers.ExecutionController - read stream event: DefaultLogEvent{, eventType=log, loglevel=VERBOSE, datetime=Fri Jan 31 12:04:29 UTC 2025, message='[wf:2e2e447e-991e-4877-a4c8-74790bf9898d] OperationFailed: operation completed, success? false: OperationCompleted(identity=[1]Stop All Appservers, stepNum=1, newState=DataState{state={step.1.result.failedNodes=xxxxxxxx, step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}, stepResultCapture=StepResultCapture{stepResult=Dispatch failed on 1 nodes: [xxxxxx: PluginFailed: Unable to create cachedir: /home/rundeck/var/cache/ScriptURLNodeStepExecutor + {dataContext=MultiDataContextImpl(map={}, base=null)} ], stepSuccess=false, statusString='null', controlBehavior=null, resultData=MultiDataContextImpl(map={}, base=null)}, success=false)', metadata={node=ec3e3eee-c3d9-4f6e-4386-f281, user=ruscau}}
[2025-01-31T12:07:31,559] DEBUG controllers.ExecutionController - read stream event: DefaultLogEvent{, eventType=log, loglevel=VERBOSE, datetime=Fri Jan 31 12:04:29 UTC 2025, message='[wf:2e2e447e-991e-4877-a4c8-74790bf9898d] WillProcessStateChange: state changes: [1]Stop All Appservers DataState{state={step.1.result.failedNodes=xxxxxxxx, step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}', metadata={node=ec3e3eee-c3d9-4f6e-4386-f281, user=ruscau}}
[2025-01-31T12:07:31,559] DEBUG controllers.ExecutionController - read stream event: DefaultLogEvent{, eventType=log, loglevel=VERBOSE, datetime=Fri Jan 31 12:04:29 UTC 2025, message='[wf:2e2e447e-991e-4877-a4c8-74790bf9898d] Update conditional state: {step.1.result.failedNodes=xxxxxxxx, step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}', metadata={node=ec3e3eee-c3d9-4f6e-4386-f281, user=ruscau}}
[2025-01-31T12:07:31,560] DEBUG controllers.ExecutionController - read stream event: DefaultLogEvent{, eventType=log, loglevel=VERBOSE, datetime=Fri Jan 31 12:04:29 UTC 2025, message='[wf:2e2e447e-991e-4877-a4c8-74790bf9898d] Update conditional state: {workflow.done=true, step.1.start=true, step.2.start=true, step.1.skip=true}', metadata={node=ec3e3eee-c3d9-4f6e-4386-f281, user=ruscau}}
[2025-01-31T12:07:31,564] DEBUG controllers.ExecutionController - read stream event: DefaultLogEvent{, eventType=log, loglevel=VERBOSE, datetime=Fri Jan 31 12:04:29 UTC 2025, message='[wf:2e2e447e-991e-4877-a4c8-74790bf9898d] DidProcessStateChange: applied state changes and rules (changed? true): [1]Stop All Appservers - StateLogger{state=DataState{state={workflow.done=true, step.any.state.failed=true, after.step.2=false, after.step.3=false, after.step.1=true, step.2.start=true, after.step.6=false, workflow.state=started, after.step.7=false, after.step.4=false, after.step.5=false, before.step.7=true, step.1.result.failedNodes=xxxxxxxx, before.step.3=true, before.step.4=true, before.step.5=true, before.step.6=true, step.1.state=failure, step.1.completed=true, before.step.1=false, step.1.start=true, before.step.2=true, workflow.id=2e2e447e-991e-4877-a4c8-74790bf9898d, workflow.keepgoing=false, step.1.skip=true}}}', metadata={node=ec3e3eee-c3d9-4f6e-4386-f281, user=ruscau}}
[2025-01-31T12:07:31,564] DEBUG controllers.ExecutionController - read stream event: DefaultLogEvent{, eventType=log, loglevel=VERBOSE, datetime=Fri Jan 31 12:04:29 UTC 2025, message='[wf:2e2e447e-991e-4877-a4c8-74790bf9898d] WorkflowEndState: Workflow end state reached.', metadata={node=ec3e3eee-c3d9-4f6e-4386-f281, user=ruscau}}

Let me know if I can provide more information.

Thank you,

Cristina Ruscau

[rac...@rundeck.com]

Thanks!

1. Can you share a job definition example to take a look at? (please, hide or change any sensitive info)
2. Also, can you share the default node executor/file copier config? (same, hide or change any sensitive info)
3. Could you double-check the space where the rundeck.war file is stored? Is the filesystem (in the container) entirely owned by the rundeck user? I suspect the filesystem was changed for some reason.

Greetings!

​

[rac...@rundeck.com]

Hi Cristina,

Pretty sure the problem is the root user ownership in the rundeck user space. The ideal scenario is that the rundeck user must rule their space (config files, plugins on the libext subdirectory, etc).

If you're using a custom Docker Image, my advice is to use the official Rundeck Docker image.

Regards.

On Wednesday, February 5, 2025 at 10:01:24 AM UTC-3 cristina...@gmail.com wrote:

Hello,

Thank you for your reply!

I attached the info, hope I got everything:

Job definition.

Default node executor config (from UI) 

rundeck.war ownership (I found it in /home/rundeck) - seems to be rundeck user

and filesystem ownership - seems to be root

If you don't mind after checking them I will remove the files if possible.

Thank you very much,

Cristina Ruscau

[Cristina Ruscau]

Hello,

Thank you for helping with this, I will look into it.

I will remove my previous message now.

Thank you very much,

Cristina Ruscau

[Cristina Ruscau]

Hello,

I tried to only use the rundeck user but I don't think it's possible as I need to do some chmods and create some folders to put my configs, which can only be done by the root user.

Just wanted to leave this here in case someone bumps into the same issue.

Let me know if you have any other ideas.

Thank you very much for your support,

Cristina Ruscau

[rac...@rundeck.com]

Hi Cristina,

If you need to run root-level tasks from the Rundeck user space, you can use the sudo local execution plugin.

That said, the error:

PluginFailed: Unable to create cachedir

happens because Rundeck is trying to create files in a root-owned filesystem. Mixing root privileges with a regular user setup isn’t ideal, as it can lead to permission issues.

I’d recommend using the official Rundeck Docker image as a cleaner and more reliable starting point.

Best regards!

[Cristina Ruscau]

Hello,

I am using the official rundeck docker image as you mentioned.

When building the image I do some actions using the root user - like placing the login credentials in the right folder and changing its permissions to be able to be accessed by the rundeck user.

Is the sudo local execution plugin meant for these kinds of operations, or just to be used when running certain jobs with that docker container that resulted from the image?

Thank you,

Cristina Ruscau

[rac...@rundeck.com]

Hi Cristina,

Is the sudo local execution plugin meant for these kinds of operations, or just to be used when running certain jobs with that docker container that resulted from the image?

The second. If you have some root stuff in the container and you need to run local jobs with the sudo context, that plugin can help you. Basically, it allows you to run commands in sudo mode.

Regards.

​

[Cristina Ruscau]

Hello,

Thank you, these details help a lot.

I do have another question though, is it possible to read the sudo password from a keystore? since that would be more secure for storing it.

Storing it on the rundeck filesystem it's something I would like to avoid.

Thank you very much,

Cristina Ruscau

[rac...@rundeck.com]

Hi, Cristina.

Yes, that is doable.

You can reference the sudo password directly from the sudo-password-storage-path attribute (in the node xml/yml definition).

e.g:

<?xml version="1.0" encoding="UTF-8"?> <project> <node name="ubuntu20" description="My Server" tags="" hostname="192.168.1.35" osArch="amd64" osFamily="unix" osName="Linux" osVersion="3.10.0-1160.el7.x86_64" username="utest" ssh-authentication="password" ssh-password-storage-path="keys/userpasswd" sudo-command-enabled="true" sudo-password-storage-path="keys/sudopasswd" /> </project>

Where keys/sudopassword refers to the sudo password stored on the Rundeck key storage.

Please take a look at this. There you can learn all available options.

Regards!

​