jobs failing on new instance

[Andrew Meyer]

I am wokring with my teams to test jobs in their respective projects.  The first one we are seeing errors:

I'm trying to figure out if this is a shell issue or not.

[workflow] Begin step: 2,NodeDispatch
2: Workflow step executing: ScriptFileItem{script=[297 chars]}
preparing for sequential execution on 1 nodes
Executing command on node: rundeck-test.example.com, NodeEntryImpl{tags=[], attributes={nodename=rundeck-test.example.com, hostname=rundeck-test.example.com, osVersion=4.18.0-513.11.1.el8_9.x86_64, osFamily=unix, osArch=amd64, description=Rundeck server node, osName=Linux}, project='null'}
[workflow] beginExecuteNodeStep(rundeck-test.example.com): NodeDispatch: ScriptFileItem{script=[297 chars]}
using charset: null
Current OS is Linux
Adding reference: ant.PropertyHelper
Project base dir set to: /
Setting environment variable: RD_JOB_ID=
Setting environment variable: RD_JOB_USERNAME=admin
Setting environment variable: RD_NODE_HOSTNAME=rundeck-test.example.com
Setting environment variable: RD_JOB_URL=
Setting environment variable: RD_JOB_PROJECT=TEST_CHECK
Setting environment variable: RD_JOB_NAME=
Setting environment variable: RD_NODE_OS_ARCH=amd64
Setting environment variable: RD_NODE_OS_VERSION=4.18.0-513.11.1.el8_9.x86_64
Setting environment variable: RD_NODE_NAME=rundeck-test.example.com
Setting environment variable: RD_JOB_THREADCOUNT=1
Setting environment variable: RD_JOB_RETRYATTEMPT=0
Setting environment variable: RD_JOB_USER_NAME=admin
Setting environment variable: RD_OPTION_APPSTACK=mw
Setting environment variable: RD_JOB_LOGLEVEL=DEBUG
Setting environment variable: RD_NODE_OS_NAME=Linux
Setting environment variable: RD_JOB_SERVERUUID=
Setting environment variable: RD_NODE_OS_FAMILY=unix
Setting environment variable: RD_JOB_EXECID=918725
Setting environment variable: RD_NODE_USERNAME=
Setting environment variable: RD_NODE_TAGS=
Setting environment variable: RD_JOB_RETRYPREVEXECID=0
Setting environment variable: RD_JOB_SERVERURL=
Setting environment variable: RD_JOB_EXECUTIONTYPE=user
Setting environment variable: RD_JOB_WASRETRY=false
Setting environment variable: RD_JOB_SUCCESSONEMPTYNODEFILTER=false
Setting environment variable: RD_JOB_RETRYINITIALEXECID=0
Setting environment variable: RD_NODE_DESCRIPTION=Rundeck server node
Executing '/bin/sh' with arguments:
'-c'
'chmod +x /opt/rundeck/0-918725-rundeck-test.example.com-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
Execute:Java13CommandLauncher: Executing '/bin/sh' with arguments:
'-c'
'chmod +x /opt/rundeck/0-918725-rundeck-test.example.com-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
Setting project property: 1708554490880.node.rundeck-test.example.com.LocalNodeExecutor.result -> 0
using charset: null
Current OS is Linux
Adding reference: ant.PropertyHelper
Project base dir set to: /
Setting environment variable: RD_JOB_ID=6be251fe-6d0e-4805-b84a-e8cc1a585ee1
Setting environment variable: RD_JOB_USERNAME=admin
Setting environment variable: RD_NODE_HOSTNAME=rundeck-test.example.com
Setting environment variable: RD_JOB_URL=
Setting environment variable: RD_JOB_PROJECT=TEST_CHECK
Setting environment variable: RD_JOB_NAME=
Setting environment variable: RD_NODE_OS_ARCH=amd64
Setting environment variable: RD_NODE_OS_VERSION=4.18.0-513.11.1.el8_9.x86_64
Setting environment variable: RD_NODE_NAME=rundeck-test.example.com
Setting environment variable: RD_JOB_THREADCOUNT=1
Setting environment variable: RD_JOB_RETRYATTEMPT=0
Setting environment variable: RD_JOB_USER_NAME=admin
Setting environment variable: RD_OPTION_APPSTACK=mw
Setting environment variable: RD_JOB_LOGLEVEL=DEBUG
Setting environment variable: RD_NODE_OS_NAME=Linux
Setting environment variable: RD_JOB_SERVERUUID=
Setting environment variable: RD_NODE_OS_FAMILY=unix
Setting environment variable: RD_JOB_EXECID=918725
Setting environment variable: RD_NODE_USERNAME=
Setting environment variable: RD_NODE_TAGS=
Setting environment variable: RD_JOB_RETRYPREVEXECID=0
Setting environment variable: RD_JOB_SERVERURL=
Setting environment variable: RD_JOB_EXECUTIONTYPE=user
Setting environment variable: RD_JOB_WASRETRY=false
Setting environment variable: RD_JOB_SUCCESSONEMPTYNODEFILTER=false
Setting environment variable: RD_JOB_RETRYINITIALEXECID=0
Setting environment variable: RD_NODE_DESCRIPTION=Rundeck server node
Executing '/bin/sh' with arguments:
'-c'
'/opt/rundeck/0-918725-rundeck-test.example.com-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
Execute:Java13CommandLauncher: Executing '/bin/sh' with arguments:
'-c'
'/opt/rundeck/0-918725-rundeck-test.example.com-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
ERROR! the role 'TEST_CHECK' was not found in /opt/ansible/playbooks/mw/roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/opt/ansible/playbooks/mw
The error appears to be in '/opt/ansible/playbooks/mw/mw_TEST_CHECK.yml': line 25, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
  roles:
    - { role: TEST_CHECK }
      ^ here
Setting project property: 1708554490919.node.rundeck-test.example.com.LocalNodeExecutor.result -> 1
Result: 1
using charset: null
Current OS is Linux
Adding reference: ant.PropertyHelper
Project base dir set to: /
Setting environment variable: RD_JOB_ID=
Setting environment variable: RD_JOB_USERNAME=admin
Setting environment variable: RD_NODE_HOSTNAME=rundeck-test.example.com
Setting environment variable: RD_JOB_URL=
Setting environment variable: RD_JOB_PROJECT=TEST_CHECK
Setting environment variable: RD_JOB_NAME=
Setting environment variable: RD_NODE_OS_ARCH=amd64
Setting environment variable: RD_NODE_OS_VERSION=4.18.0-513.11.1.el8_9.x86_64
Setting environment variable: RD_NODE_NAME=rundeck-test.example.com
Setting environment variable: RD_JOB_THREADCOUNT=1
Setting environment variable: RD_JOB_RETRYATTEMPT=0
Setting environment variable: RD_JOB_USER_NAME=admin
Setting environment variable: RD_OPTION_APPSTACK=mw
Setting environment variable: RD_JOB_LOGLEVEL=DEBUG
Setting environment variable: RD_NODE_OS_NAME=Linux
Setting environment variable: RD_JOB_SERVERUUID=
Setting environment variable: RD_NODE_OS_FAMILY=unix
Setting environment variable: RD_JOB_EXECID=918725
Setting environment variable: RD_NODE_USERNAME=
Setting environment variable: RD_NODE_TAGS=
Setting environment variable: RD_JOB_RETRYPREVEXECID=0
Setting environment variable: RD_JOB_SERVERURL=
Setting environment variable: RD_JOB_EXECUTIONTYPE=user
Setting environment variable: RD_JOB_WASRETRY=false
Setting environment variable: RD_JOB_SUCCESSONEMPTYNODEFILTER=false
Setting environment variable: RD_JOB_RETRYINITIALEXECID=0
Setting environment variable: RD_NODE_DESCRIPTION=Rundeck server node
Executing '/bin/sh' with arguments:
'-c'
'rm -f /opt/rundeck/0-918725-rundeck-test.example.com-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
Execute:Java13CommandLauncher: Executing '/bin/sh' with arguments:
'-c'
'rm -f /opt/rundeck/0-918725-rundeck-test.example.com-dispatch-script.tmp.sh'

The ' characters around the executable and arguments are
not part of the command.
Setting project property: 1708554491573.node.rundeck-test.example.com.LocalNodeExecutor.result -> 0
Failed: NonZeroResultCode: Result code was 1
[workflow] finishExecuteNodeStep(rundeck-test.example.com): NodeDispatch: NonZeroResultCode: Result code was 1
2: Workflow step finished, result: Dispatch failed on 1 nodes: [rundeck-test.example.com: NonZeroResultCode: Result code was 1 + {dataContext=MultiDataContextImpl(map={ContextView(step:2, node:rundeck-test.example.com)=BaseDataContext{{exec={exitCode=0}}}, ContextView(node:rundeck-test.example.com)=BaseDataContext{{exec={exitCode=0}}}}, base=null)} ]
[workflow] Finish step: 2,NodeDispatch

[rac...@rundeck.com]

Hi Andrew,

Due this:

ERROR! the role 'TEST_CHECK' was not found in /opt/ansible/playbooks/mw/roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/opt/ansible/playbooks/mw The error appears to be in '/opt/ansible/playbooks/mw/mw_TEST_CHECK.yml': line 25, column 7, but may

Your job can’t find the required role. Make sure to have the roles accessible to your project. Pretty sure that you need to follow this in your project (more info here).

Greetings.

​

[Andrew Meyer]

Thanks!  Fixed that issue.

Now i'm seeing this:

Result: 2

Failed: NonZeroResultCode: Result code was 2

That is the only error in the whole stack.

[rac...@rundeck.com]

Hello, Andrew.

Your step was finished with the exit code 2. Rundeck identifies non-zero exit codes as failures.  So, reviewing the output in debug mode would be beneficial in determining the root cause.

Please examine your step/workflow; by default, it returns 2 as code.

You can examine the outcome code with an error handler and, if necessary, ignore exit code 2. Check this.

Greetings.

[Andrew Meyer]

Here is the debug output of that execution:

[workflow] Begin step: 1,NodeDispatch
1: Workflow step executing: CommandItem{command=[1 words]}

preparing for sequential execution on 1 nodes
Executing command on node: rundeck-test.example.com, NodeEntryImpl{tags=[], attributes={nodename=rundeck-test.example.com, hostname=rundeck-test.example.com, osVersion=4.18.0-513.11.1.el8_9.x86_64, osFamily=unix, osArch=amd64, description=Rundeck server node, osName=Linux}, project='null'}

[workflow] beginExecuteNodeStep(rundeck-test.example.com): NodeDispatch: CommandItem{command=[1 words]}

using charset: null
Current OS is Linux
Adding reference: ant.PropertyHelper
Project base dir set to: /
Setting environment variable: RD_JOB_ID=
Setting environment variable: RD_JOB_USERNAME=admin
Setting environment variable: RD_NODE_HOSTNAME=rundeck-test.example.com
Setting environment variable: RD_JOB_URL=

Setting environment variable: RD_JOB_PROJECT=Ansible_Checkout
Setting environment variable: RD_JOB_NAME=Deploy_Ansible_Code_From_Stash

Setting environment variable: RD_NODE_OS_ARCH=amd64
Setting environment variable: RD_NODE_OS_VERSION=4.18.0-513.11.1.el8_9.x86_64
Setting environment variable: RD_NODE_NAME=rundeck-test.example.com
Setting environment variable: RD_JOB_THREADCOUNT=1
Setting environment variable: RD_JOB_RETRYATTEMPT=0
Setting environment variable: RD_JOB_USER_NAME=admin
Setting environment variable: RD_OPTION_APPSTACK=mw
Setting environment variable: RD_JOB_LOGLEVEL=DEBUG
Setting environment variable: RD_NODE_OS_NAME=Linux
Setting environment variable: RD_JOB_SERVERUUID=
Setting environment variable: RD_NODE_OS_FAMILY=unix

Setting environment variable: RD_JOB_EXECID=918748

Setting environment variable: RD_NODE_USERNAME=
Setting environment variable: RD_NODE_TAGS=
Setting environment variable: RD_JOB_RETRYPREVEXECID=0
Setting environment variable: RD_JOB_SERVERURL=
Setting environment variable: RD_JOB_EXECUTIONTYPE=user
Setting environment variable: RD_JOB_WASRETRY=false
Setting environment variable: RD_JOB_SUCCESSONEMPTYNODEFILTER=false
Setting environment variable: RD_JOB_RETRYINITIALEXECID=0
Setting environment variable: RD_NODE_DESCRIPTION=Rundeck server node
Executing '/bin/sh' with arguments:
'-c'

'/opt/ansible/bin/mw/mw_ansible_setup.sh'

The ' characters around the executable and arguments are
not part of the command.
Execute:Java13CommandLauncher: Executing '/bin/sh' with arguments:
'-c'

'/opt/ansible/bin/mw/mw_ansible_setup.sh'

The ' characters around the executable and arguments are
not part of the command.

[WARNING]: Unable to parse /opt/ansible/environments/local as an inventory
source
[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************
TASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [Get DEV environment based on hostname] ***********************************
skipping: [localhost]
TASK [Get DEV environment based on hostname] ***********************************
skipping: [localhost]
TASK [Get TEST environment based on hostname] **********************************
skipping: [localhost]
TASK [Get PREPROD environment based on hostname] *******************************
skipping: [localhost]
TASK [Get PROD environment based on hostname] **********************************
skipping: [localhost]
TASK [Pull ansible_checkout role for specific environment] *********************
skipping: [localhost]
TASK [Check that ansible_setup.sh script exists] *******************************
ok: [localhost]
TASK [Add ansible_setup.sh to crontab] *****************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "You (ansible) are not allowed to use this program (crontab)\nSee crontab(1) for more information\n"}
PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=6    rescued=0    ignored=0  
Setting project property: 1708615226029.node.rundeck-test.example.com.LocalNodeExecutor.result -> 2

Result: 2
Failed: NonZeroResultCode: Result code was 2

[workflow] finishExecuteNodeStep(rundeck-test.example.com): NodeDispatch: NonZeroResultCode: Result code was 2
1: Workflow step finished, result: Dispatch failed on 1 nodes: [rundeck-test.example.com: NonZeroResultCode: Result code was 2 + {dataContext=MultiDataContextImpl(map={ContextView(node:rundeck-test.example.com)=BaseDataContext{{exec={exitCode=2}}}, ContextView(step:1, node:rundeck-test.example.com)=BaseDataContext{{exec={exitCode=2}}}}, base=null)} ]
[workflow] Finish step: 1,NodeDispatch

[rac...@rundeck.com]

It seems a permissions issue:

fatal: [localhost]: FAILED! => {"changed": false, "msg": "You (ansible) are not allowed to use this program (crontab)\nSee crontab(1) for more information\n"}

Regards.

​