job execution folder

[Andrew Meyer]

Hello,

I'm having trouble remembering the job execution folder flag.

This is all I have in my framework.properties

framework.file-copy-destination-dir=/var/lib/rundeck/var/tmp

Right now it is filling up /tmp.

[rac...@rundeck.com]

Hi Andrew.

Are you referring to the execution logs? If so, you need to set the framework.logs.dir property, take a look.

The framework.file-copy-destination-dir property defines where the scripts are copied on target nodes (for script steps).

Now if the problem is the /tmp directory. Rundeck doesn’t store logs there.

Both properties are explained well here.

​

[Andrew Meyer]

rdeck.base=/var/lib/rundeck

framework.projects.dir=/var/lib/rundeck/projects
framework.etc.dir=/etc/rundeck
framework.var.dir=/var/lib/rundeck/var
framework.tmp.dir=/var/lib/rundeck/var/tmp
framework.logs.dir=/var/lib/rundeck/logs
framework.libext.dir=/var/lib/rundeck/libext

framework.file-copy-destination-dir=/var/lib/rundeck/var/tmp

I ran into this problem a while back I think.

[Andrew Meyer]

So i'm still struggling with this. 

I have the framework.file-copy-destintation-dir set but when the jobs go to pull code it fails with this error:

/bin/sh: /tmp/rundeck/0-338042-npsxrundkadm01-dispatch-script.tmp.sh: Permission denied

I changed the folder destination from /var/lib/rundeck/var/tmp to /tmp/rundeck.  Do I need to change the tmp.dir one as well?

[Andrew Meyer]

Is there a way to set this -

RUNDECK_TEMPDIR="${RUNDECK_TEMPDIR:-/tmp/rundeck}"

I'm not seeing it in the documentation.  I tried creating a line in my ~/.bashrc like this:

export RUNDECK_TMPDIR=/path/to/tmpdir

And then restarted rundeck.  But it didn't take.

[rac...@rundeck.com]

Hi Andrew,

RUNDECK_TEMPDIR is to define the instance temp directory. framework.file-copy-destination-dir to specify the remote node directory to the file copier (to execute remote scripts). Probably the /tmp/rundeck destination doesn’t exist on your remote nodes. Make sure to define a valid path with rights permissions to execute scripts.

Regarding this:

I have the framework.file-copy-destintation-dir set but when the jobs go to pull code it fails with this error:

Try changing framework.file-copy-destintation-dir by file-copy-destination-dir (destintation != destination).

Another way to try: define it at the node level by adding file-copy-destination-dir as an attribute in your resources.xml/yml file. Check this example.

Regards.

​

[Chris Gadd]

If the advice below doesn’t resolve things it may be worth checking you don’t have SELinux preventing execution of files in /tmp.

 

C2 General

From: rundeck...@googlegroups.com <rundeck...@googlegroups.com> On Behalf Of rac...@rundeck.com
Sent: Thursday, May 23, 2024 1:56 AM
To: rundeck-discuss <rundeck...@googlegroups.com>
Subject: [rundeck] Re: job execution folder

 

CYBER SECURITY WARNING: This email is from an external source - be careful of attachments and links. Please follow the Cyber Code and report suspicious emails.

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/fd4c29de-5222-4328-b806-7250b90ba27an%40googlegroups.com.

[Andrew Meyer]

That's what someone else on my team said.  I will did apply that in my dev env.

[Andrew Meyer]

Also, I do have SELinux in enforcing.  And we are not allowing exec on /tmp.  I have created a /tmp/rundeck LV.

Something else: I did run a semanage to change the context on /tmp/rundeck but it still failed.

[Andrew Meyer]

I also tried adding a bash variable into .bashrc for the rundeck user.

RUNDECK_TEMPDIR=/tmp/rundeck but it keeps going to /tmp.

[Andrew Meyer]

Hello,

I'm back at this.  I have solved this in my dev and test environments.  But preprod and eventually prod are giving me trouble.

Every time I move the execution folder I get the following error it seems:

[2024-05-30T13:32:05,746] ERROR boot.SpringApplication - Application run failed
io.micronaut.context.exceptions.NoSuchBeanException: No bean of type [io.micronaut.context.event.ApplicationEventPublisher<io.micronaut.context.event.StartupEvent>] exists. Make sure the bean is not disabled by bean requirements (enable trace logging for 'io.micronaut.context.condition' to check) and if the bean is enabled then ensure the class is declared a bean and annotation processing is enabled (for Java and Kotlin the 'micronaut-inject-java' dependency should be configured as an annotation processor).
        at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2805) ~[micronaut-inject-3.8.6.jar!/:3.8.6]
        at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1617) ~[micronaut-inject-3.8.6.jar!/:3.8.6]
        at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:867) ~[micronaut-inject-3.8.6.jar!/:3.8.6]
        at io.micronaut.context.BeanLocator.getBean(BeanLocator.java:94) ~[micronaut-inject-3.8.6.jar!/:3.8.6]
        at io.micronaut.context.DefaultBeanContext.publishEvent(DefaultBeanContext.java:1703) ~[micronaut-inject-3.8.6.jar!/:3.8.6]
        at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:356) ~[micronaut-inject-3.8.6.jar!/:3.8.6]
        at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:194) ~[micronaut-inject-3.8.6.jar!/:3.8.6]
        at grails.boot.GrailsApp.createApplicationContext(GrailsApp.groovy:184) ~[grails-core-5.3.2.jar!/:5.3.2]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:304) [spring-boot-2.7.12.jar!/:2.7.12]
        at grails.boot.GrailsApp.run(GrailsApp.groovy:99) [grails-core-5.3.2.jar!/:5.3.2]
        at grails.boot.GrailsApp.run(GrailsApp.groovy:485) [grails-core-5.3.2.jar!/:5.3.2]
        at grails.boot.GrailsApp.run(GrailsApp.groovy:472) [grails-core-5.3.2.jar!/:5.3.2]
        at grails.boot.GrailsApp$run.call(Unknown Source) [grails-core-5.3.2.jar!/:?]
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) [groovy-3.0.9.jar!/:3.0.9]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-3.0.9.jar!/:3.0.9]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148) [groovy-3.0.9.jar!/:3.0.9]
        at rundeckapp.Application.execRunApp(Application.groovy:92) [classes!/:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107) [groovy-3.0.9.jar!/:3.0.9]
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) [groovy-3.0.9.jar!/:3.0.9]
        at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.invoke(StaticMetaMethodSite.java:44) [groovy-3.0.9.jar!/:3.0.9]
        at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.callStatic(StaticMetaMethodSite.java:100) [groovy-3.0.9.jar!/:3.0.9]
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:55) [groovy-3.0.9.jar!/:3.0.9]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:217) [groovy-3.0.9.jar!/:3.0.9]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:222) [groovy-3.0.9.jar!/:3.0.9]
        at rundeckapp.Application.main(Application.groovy:81) [classes!/:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [rundeck-4.15.0-20230725.war:?]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) [rundeck-4.15.0-20230725.war:?]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [rundeck-4.15.0-20230725.war:?]
        at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:59) [rundeck-4.15.0-20230725.war:?]

I have created a Logical Volume.  Instead of pointing it at /tmp/rundeck I moved it to /opt/rundecktmp.  I have SELinux set to Enforcing and I have set the tmp_t label and made sure that the owner is rundeck:rundeck. 

The permissions however are 755. 

[ ~]$ sudo ls -laZ /opt/rundecktmp/
total 4
drwxr-xr-x. 13 rundeck rundeck system_u:object_r:tmp_t:s0 4096 May 30 12:23 .
drwxr-xr-x. 14 root    root    system_u:object_r:usr_t:s0  203 May 30 12:45 ..
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 22 09:57 groovy-generated-tmpdir-13387357941709781062
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 29 11:19 groovy-generated-tmpdir-14608914003640820172
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 29 11:50 groovy-generated-tmpdir-15943547309467861737
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 22 11:53 groovy-generated-tmpdir-16584203504391461298
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 22 13:13 groovy-generated-tmpdir-17063051297815282898
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 29 11:46 groovy-generated-tmpdir-17530029301580182929
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 22 10:24 groovy-generated-tmpdir-2235229635538424856
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 29 11:35 groovy-generated-tmpdir-335715627753505008
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 29 11:41 groovy-generated-tmpdir-3745077411678836139
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 21 13:14 groovy-generated-tmpdir-9147887477881964383
drwx------.  2 rundeck rundeck system_u:object_r:tmp_t:s0    6 May 22 11:49 groovy-generated-tmpdir-9979120999103904895
[ ~]$

I think this is my last hurdle.

Thanks!

[rac...@rundeck.com]

Hi Andrew,

Regarding this:

Every time I move the execution folder I get the following error it seems:

Which execution folder are you moving? and why do you need to move it? Could you elaborate?

Which Rundeck version are you using?

Thanks!

Regards.

[Andrew Meyer]

My apologies.

I am setting the temp folder for script execution.  We have noexec set on /tmp and /var/tmp in RHEL 8.

I am using Rundeck 4.15.x

This is primarily for script execution.

I changed this value in framework.properties - framework.tmp.dir=/opt/rundecktmp

Now rundeck won't start.

[rac...@rundeck.com]

Hi Andrew,

Weird. I couldn't reproduce the problem after changing that parameter in the framework.properties file in Rundeck 4.15 (with or without the correct permissions). Which Java version are you using?

Another possible root cause could be the war file integrity. Could you re-download the war file from the official website?

Greetings.

[Andrew Meyer]

I am using the RPM installation.

[rac...@rundeck.com]

What about the Java version?

Regards.

[Andrew Meyer]

Java 11.0.18

[rac...@rundeck.com]

Could you test using the most recent Rundeck version in a test environment? Some Micronaut framework improvements/upgrades have been implemented. I recommend testing using 5.2.0 due to this 5.3.0 issue.

Regards.

[Andrew Meyer]

At this time it is not possible to upgrade to that.  I would have to upgrade the other environments.  It is the plan to upgrade to newer versions.

[Andrew Meyer]

Is there another option?  All the other my dev and test work just fine....

[rac...@rundeck.com]

Hi Andrew,

It seems a hard-to-reproduce issue. Some approaches:

1. Compare the configuration files between workable/nonworkable environments (mainly rundeck-config.properties and framework.properties files).
2. Same with the launch script: /etc/rundeck/profile and /etc/sysconfig/rundeckd. Which parameters are you using to launch Rundeck?
3. What about the server resources? Is the rundeck instance running with enough resources?
4. I still suspect the war file integrity (yes, RPM / DEB variants use a WAR file “wrapped” in the packaging system). You can check (and compare with the regular war file) it in the following path: /var/lib/rundeck/bootstrap/

Greetings.

​

[Andrew Meyer]

ok, so the server is running 8cpu with 8GB RAM.

/var/lib/rundeck has 20G+ allocated to it.

I don't have the open files limit set.  I will try that.

I'm not sure it is war file integrity.  If I set it back to the defaults...it works.  This is just in preprod where it doesn't.  We do have anti-virus....maybe that could be causing it.  I have seen it do weird things in the past.

This is what I have in ALL of the environments of rundeck sysconfig:

RDECK_JVM_OPTS="-Xmx4096m -Xms1024m -server -Dserver.session.timeout=3600 -Dserver.servlet.session.timeout=3600 -Dserver.address=AN.INT.IP.ADDR -Dserver.port=4440 -Drundeck.config.name=/etc/rundeck/rundeck-config.properties -Djava.security.auth.login.config=/etc/rundeck/jaas-multiauth.conf -Drundeck.jaaslogin=true -Dloginmodule.conf.name=/etc/rundeck/jaas-multiauth.conf -Dloginmodule.name=multiauth -Dcom.dtolabs.rundeck.jetty.jaas.LEVEL=DEBUG -Dcom.dtolabs.rundeck.jetty.access.LEVEL=debug"

So the IP address will obviously change.

[rac...@rundeck.com]

Also, make sure to check the SELinux policies (or test without SELinux just to discard), probably the Java apps execution is limited/restricted.

Regards,

[Andrew Meyer]

So I turned off SELinux:

sudo setenforce 0

Changed the path for rundeck tmp files to:

/opt/rundecktmp

Restarted rundeck and got this error:

Session terminated, killing shell...[2024-06-03T13:13:46,538] INFO  rundeckapp.BootStrap - Rundeck Shutdown detected
 ...killed.
[2024-06-03T13:15:18,189] ERROR boot.SpringApplication - Application run failed

[Andrew Meyer]

After rebooting the server I changed permissions on /tmp/rundeck and that fixed it.

[Andrew Meyer]

So after changing the permissions again and making it 775 on /opt/rundecktmp it now works.  I know for a fact I tried this all last week minus the setenforce 0.

I appreciate all the help everyone has given me here.  1 more server to go.

[rac...@rundeck.com]

Great news Andrew! Good to know for future references.

Cheers.

[Andrew Meyer]

I take that back.  Someone ran a different job and it failed.  It is still trying to execute in /tmp

[workflow] Begin step: 1,NodeDispatch
1: Workflow step executing: ScriptFileItem{script=[829 chars]}
preparing for sequential execution on 1 nodes
Executing command on node: rundeck-preprod01, NodeEntryImpl{tags=[], attributes={nodename=rundeck-preprod01, hostname=rundeck-preprod01, osVersion=, osFamily=unix, osArch=amd64, description=Rundeck server node, osName=Linux}, project='null'}
[workflow] beginExecuteNodeStep(rundeck-preprod01): NodeDispatch: ScriptFileItem{script=[829 chars]}
using charset: null
Current OS is Linux
Adding reference: ant.PropertyHelper
Project base dir set to: /var/lib/rundeck
Setting environment variable: RD_JOB_ID=e755d78b-3e69-4ede-958b-8c42abfeca9c
Setting environment variable: RD_JOB_USERNAME=
Setting environment variable: RD_OPTION_CREATE_FILESYSTEMS=true

Setting environment variable: RD_JOB_URL=
Setting environment variable: RD_JOB_PROJECT=Chef
Setting environment variable: RD_JOB_NAME=Bootstrap - RHEL7
Setting environment variable: RD_NODE_OS_ARCH=amd64
Setting environment variable: RD_NODE_OS_VERSION=

Setting environment variable: RD_JOB_THREADCOUNT=1
Setting environment variable: RD_JOB_RETRYATTEMPT=0
Setting environment variable: RD_JOB_USER_NAME=
Setting environment variable: RD_JOB_LOGLEVEL=DEBUG
Setting environment variable: RD_NODE_OS_NAME=Linux
Setting environment variable: RD_JOB_SERVERUUID=52a96f1d-8006-499b-b15d-574403945307
Setting environment variable: RD_NODE_OS_FAMILY=unix
Setting environment variable: RD_JOB_EXECID=340646
Setting environment variable: RD_NODE_USERNAME=

Setting environment variable: RD_NODE_TAGS=
Setting environment variable: RD_JOB_RETRYPREVEXECID=0

Setting environment variable: RD_JOB_EXECUTIONTYPE=user
Setting environment variable: RD_JOB_WASRETRY=false
Setting environment variable: RD_JOB_SUCCESSONEMPTYNODEFILTER=false
Setting environment variable: RD_JOB_RETRYINITIALEXECID=0
Setting environment variable: RD_NODE_DESCRIPTION=Rundeck server node
Executing '/bin/sh' with arguments:
'-c'
'chmod +x /tmp/32-340646-rundeck-preprod01-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
Execute:Java13CommandLauncher: Executing '/bin/sh' with arguments:
'-c'
'chmod +x /tmp/32-340646-rundeck-preprod01-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
Setting project property: 1717534419449.node.rundeck-preprod01.LocalNodeExecutor.result -> 0
using charset: null
Current OS is Linux
Adding reference: ant.PropertyHelper
Project base dir set to: /var/lib/rundeck
Setting environment variable: RD_JOB_ID=
Setting environment variable: RD_JOB_USERNAME=
Setting environment variable: RD_OPTION_CREATE_FILESYSTEMS=true
Setting environment variable: RD_NODE_HOSTNAME=rundeck-preprod01
Setting environment variable: RD_JOB_URL=
Setting environment variable: RD_JOB_PROJECT=Chef
Setting environment variable: RD_JOB_NAME=Bootstrap - RHEL7
Setting environment variable: RD_NODE_OS_ARCH=amd64
Setting environment variable: RD_NODE_OS_VERSION=
Setting environment variable: RD_NODE_NAME=rundeck-preprod01
Setting environment variable: RD_JOB_THREADCOUNT=1
Setting environment variable: RD_JOB_RETRYATTEMPT=0
Setting environment variable: RD_JOB_USER_NAME=
Setting environment variable: RD_JOB_LOGLEVEL=DEBUG
Setting environment variable: RD_NODE_OS_NAME=Linux
Setting environment variable: RD_JOB_SERVERUUID=52a96f1d-8006-499b-b15d-574403945307
Setting environment variable: RD_NODE_OS_FAMILY=unix
Setting environment variable: RD_JOB_EXECID=340646
Setting environment variable: RD_NODE_USERNAME=

Setting environment variable: RD_NODE_TAGS=
Setting environment variable: RD_JOB_RETRYPREVEXECID=0

Setting environment variable: RD_JOB_EXECUTIONTYPE=user
Setting environment variable: RD_JOB_WASRETRY=false
Setting environment variable: RD_JOB_SUCCESSONEMPTYNODEFILTER=false
Setting environment variable: RD_JOB_RETRYINITIALEXECID=0
Setting environment variable: RD_NODE_DESCRIPTION=Rundeck server node
Executing '/bin/sh' with arguments:
'-c'
'/tmp/32-340646-rundeck-preprod01-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
Execute:Java13CommandLauncher: Executing '/bin/sh' with arguments:
'-c'
'/tmp/32-340646-rundeck-preprod01-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
/bin/sh: /tmp/32-340646-rundeck-preprod01-dispatch-script.tmp.sh: Permission denied
Setting project property: 1717534419488.node.rundeck-preprod01.LocalNodeExecutor.result -> 126
Result: 126
using charset: null
Current OS is Linux
Adding reference: ant.PropertyHelper
Project base dir set to: /var/lib/rundeck
Setting environment variable: RD_JOB_ID=e755d78b-3e69-4ede-958b-8c42abfeca9c
Setting environment variable: RD_JOB_USERNAME=
Setting environment variable: RD_OPTION_CREATE_FILESYSTEMS=true
Setting environment variable: RD_NODE_HOSTNAME=rundeck-preprod01

Setting environment variable: RD_JOB_PROJECT=Chef
Setting environment variable: RD_JOB_NAME=Bootstrap - RHEL7
Setting environment variable: RD_NODE_OS_ARCH=amd64

Setting environment variable: RD_NODE_NAME=rundeck-preprod01
Setting environment variable: RD_JOB_THREADCOUNT=1
Setting environment variable: RD_JOB_RETRYATTEMPT=0
Setting environment variable: RD_JOB_USER_NAME=
Setting environment variable: RD_JOB_LOGLEVEL=DEBUG
Setting environment variable: RD_NODE_OS_NAME=Linux

Setting environment variable: RD_NODE_OS_FAMILY=unix
Setting environment variable: RD_JOB_EXECID=340646
Setting environment variable: RD_NODE_USERNAME=

Setting environment variable: RD_NODE_TAGS=
Setting environment variable: RD_JOB_RETRYPREVEXECID=0

Setting environment variable: RD_JOB_EXECUTIONTYPE=user
Setting environment variable: RD_JOB_WASRETRY=false
Setting environment variable: RD_JOB_SUCCESSONEMPTYNODEFILTER=false
Setting environment variable: RD_JOB_RETRYINITIALEXECID=0
Setting environment variable: RD_NODE_DESCRIPTION=Rundeck server node
Executing '/bin/sh' with arguments:
'-c'
'rm -f /tmp/32-340646-rundeck-preprod01-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
Execute:Java13CommandLauncher: Executing '/bin/sh' with arguments:
'-c'
'rm -f /tmp/32-340646-rundeck-preprod01-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
Setting project property: 1717534419523.node.rundeck-preprod01.LocalNodeExecutor.result -> 0
Failed: NonZeroResultCode: Result code was 126
[workflow] finishExecuteNodeStep(rundeck-preprod01): NodeDispatch: NonZeroResultCode: Result code was 126
1: Workflow step finished, result: Dispatch failed on 1 nodes: [rundeck-preprod01: NonZeroResultCode: Result code was 126 + {dataContext=MultiDataContextImpl(map={ContextView(node:rundeck-preprod01)=BaseDataContext{{exec={exitCode=0}}}, ContextView(step:1, node:rundeck-preprod01)=BaseDataContext{{exec={exitCode=0}}}}, base=null)} ]
[workflow] Finish step: 1,NodeDispatch

~]$ sudo cat /etc/rundeck/framework.properties
# framework.properties -
#

# ----------------------------------------------------------------
# Rundeck server connection information
# ----------------------------------------------------------------

framework.server.name =
framework.server.hostname =
framework.server.port = 4440
framework.server.url = http://localhost

# ----------------------------------------------------------------
# Installation locations
# ----------------------------------------------------------------

rdeck.base=/var/lib/rundeck

framework.projects.dir=/var/lib/rundeck/projects
framework.etc.dir=/etc/rundeck
framework.var.dir=/var/lib/rundeck/var
#framework.tmp.dir=/var/lib/rundeck/var/tmp
framework.tmp.dir=/opt/rundecktmp
framework.logs.dir=/var/lib/rundeck/logs
framework.libext.dir=/var/lib/rundeck/libext

#framework.file-copy-destination-dir=/opt/rundecktmp
# ----------------------------------------------------------------
# SSH defaults for node executor and file copier
# ----------------------------------------------------------------

framework.ssh.keypath = /var/lib/rundeck/.ssh/id_rsa
framework.ssh.user = ansible

# ssh connection timeout after a specified number of milliseconds.
# "0" value means wait forever.
framework.ssh.timeout = 0
framework.ssh-connection-timeout = 0
framework.ssh-command-timeout = 0

[rac...@rundeck.com]

Hi,

Could you try adding the following config in your framework.properties file like in my previous post?

framework.file-copy-destination-dir = /opt/rundecktmp

Regards.

​

[Andrew Meyer]

I will re-add that in a bit.

[Andrew Meyer]

I added the line and restarted.  But then I had to go back and remount /tmp w/ noexec. 

SO FAR we are in business.

I'm not saying we are through the forest but on hopefully coming up on the clearing.

[Andrew Meyer]

I've got jobs that are working and some that are not with the setting change AND reboot.  More to come.

[Andrew Meyer]

How can I force change tmp folder in /etc/sysconfig/rundeckd ??

[Andrew Meyer]

On some of the jobs it is trying to use /tmp.

[Andrew Meyer]

So I put in RUNDECK_TEMPDIR=/opt/rundecktmp in the /etc/sysconfig/rundeckd and restarted.

So far it works.

[Andrew Meyer]

But on the other server it does not.

[Andrew Meyer]

It is failing on my dev server now.  Running RHEL 8 w/ rundeck 4.15.

fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/usr/bin/git ls-remote 'ssh:********@git.source.com:7999/project/reponame.git' -h refs/heads/feature/branch-name-1", "msg": "fatal: cannot exec '/tmp/tmp082wujia': Permission denied\nfatal: unable to fork", "rc": 128, "stderr": "fatal: cannot exec '/tmp/tmp082wujia': Permission denied\nfatal: unable to fork\n", "stderr_lines": ["fatal: cannot exec '/tmp/tmp082wujia': Permission denied", "fatal: unable to fork"], "stdout": "", "stdout_lines": []}

/etc/sysconfig/rundeckd

RDECK_JVM_OPTS="-Xmx4096m -Xms1024m -server -Dserver.session.timeout=3600 -Dserver.servlet.session.timeout=3600 -Dserver.address=INT.IP.ADD.RESS -Dserver.port=4440 -Drundeck.config.name=/etc/rundeck/rundeck-config.properties -Djava.security.auth.login.config=/etc/rundeck/jaas-multiauth.conf -Drundeck.jaaslogin=true -Dloginmodule.conf.name=/etc/rundeck/jaas-multiauth.conf -Dloginmodule.name=multiauth -Dcom.dtolabs.rundeck.jetty.jaas.LEVEL=DEBUG -Dcom.dtolabs.rundeck.jetty.access.LEVEL=debug -Djava.io.tmpdir=/opt/rundecktmp"
RUNDECK_TEMPDIR="/opt/rundecktmp"
TMPDIR="/opt/rundecktmp"

/etc/rundeck/framework.properties:

# ----------------------------------------------------------------
# Installation locations
# ----------------------------------------------------------------

rdeck.base=/var/lib/rundeck

framework.projects.dir=/var/lib/rundeck/projects
framework.etc.dir=/etc/rundeck
framework.var.dir=/var/lib/rundeck/var
#framework.tmp.dir=/var/lib/rundeck/var/tmp
framework.tmp.dir=/opt/rundecktmp
framework.logs.dir=/var/lib/rundeck/logs
framework.libext.dir=/var/lib/rundeck/libext

framework.file-copy-destination-dir=/opt/rundecktmp

/var/lib/rundeck/.bashrc:

export PATH="/opt/ansible/ansible-2.12.10/bin:$PATH"
export ANSIBLE_CONFIG=/var/lib/rundeck/.ansible.cfg
export GIT_SSH_VARIANT=ssh
export RUNDECK_TEMPDIR=/opt/rundecktmp
export TMPDIR=/opt/rundecktmp

permissions of /opt/rundecktmp

[@hostname~]$ sudo ls -lahZ /opt/rundecktmp/
total 4.0K
drwxrwxr-x. 17 rundeck rundeck system_u:object_r:tmp_t:s0        4.0K Jun  7 11:40 .
drwxr-xr-x. 11 root    root    system_u:object_r:usr_t:s0         148 Jun  4 14:22 ..
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 Jun  6 16:07 groovy-generated-tmpdir-10234964252776307793
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 Jun  7 11:08 groovy-generated-tmpdir-12822447091318686659
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 May 28 16:04 groovy-generated-tmpdir-13995122265224524224
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 May 29 14:30 groovy-generated-tmpdir-15328737389918278924
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 Jun  6 16:22 groovy-generated-tmpdir-16200827289502854748
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 May 28 15:50 groovy-generated-tmpdir-17050465896571372769
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 Jun  6 15:10 groovy-generated-tmpdir-18311453266395750182
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 May 28 15:57 groovy-generated-tmpdir-1881759577795277830
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 May 22 15:12 groovy-generated-tmpdir-1971256739355385651
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 Jun  6 14:45 groovy-generated-tmpdir-379684645541592491
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 Jun  6 16:03 groovy-generated-tmpdir-5994759476396835577
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 Jun  7 11:35 groovy-generated-tmpdir-6973323348134456135
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 Jun  6 14:42 groovy-generated-tmpdir-9087148578087713791
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 Jun  6 15:43 groovy-generated-tmpdir-9714144297590461046
drwx------.  2 rundeck rundeck system_u:object_r:initrc_tmp_t:s0    6 May 28 15:47 groovy-generated-tmpdir-9896707509378063533
[@hostname~]$

/etc/fstab:

/dev/mapper/vg_opt-lv_tmp_rundeck /opt/rundecktmp       xfs     defaults        0 0

What am I doing wrong?

[rac...@rundeck.com]

Hi Andrew, does that only happen in a specific job? (maybe using a specific workflow/node step plugin?)

Not sure if you're using the git plugin, but if so, it seems related to this.

Regards.

[Andrew Meyer]

I would say yes.  I have the git plugin configured....BUT the rundeck/ansible job is calling a bash script locally to pull down code for deployment.

[rac...@rundeck.com]

But looking at your output it seems that you're using Ansible steps. Are your playbooks / ansible.cfg configured to store the tmp data in your custom tmp path? This should help you. I'm pretty sure that is the root cause.

[Andrew Meyer]

So I just tried setting the remote_tmp and re-running the rundeck job.  It still failed.

I then tried using these:

#local_tmp      = /opt/runecktmp
system_tmpdirs = /opt/rundecktmp

not together.  Tried each option and it still failed.   Local tmp gave me python errrors.

But i'm STILL getting the same error about not being able to fork.

The rundeck user HAS 777 to /opt/rundecktmp now.

[rac...@rundeck.com]

Hi Andrew. Try this way. 

Greetings.

[Andrew Meyer]

I don't have that option since i'm not the app owner for the ansible role....