is Rundeck Ansible integration 100% broken now ???

[Jim Richard]

[ResourceModelSource: 2.source (com.batix.rundeck.plugins.AnsibleResourceModelSourceFactory), project: AnsibleJobs] 

java.lang.NullPointerException: null

[Jim Richard]

Rundeck seems to be in a weird loop where it keeps trying to use the very old Ansible plugin and no amount of restarting, deleting from libext and the libext/cache dir get get RUndeck to be "unconfused"

[Jim Richard]

or maybe it's that at RD 5.6.1, the "built in" plugin referenced here:

https://docs.rundeck.com/docs/learning/howto/using-ansible.html

is now installing an ancient version of the plugin 3.0.1, and if you try to manually override that, all your stuff will go side ways...

you'll get Null pointer exceptions etc etc

right ?

[Jim Richard]

sorry if I sound frustrated, but may I ask, what is the current state of the Rundeck/Ansible integration?

I had a job that would dynamically generate an Ansible inventory and then I would "Refresh Project Nodes" job step.

Sometime in the last week that started throwing a Null Pointer exception, the  "Refresh Project Nodes", but I didn't notice because it was returning the latest cached Nodes (because of the Null pointer).

After uninstalling/re-installing the plugin in various ways, I can't seem to get anything to work unless I let it/Rundeck install "ansible-integrations" auto-magically and...

that installs, at least according to the UI, version 3.0.1, which of course is very old.

Please help :)

[rac...@rundeck.com]

Hi,

Could you share more information about your issue? eg: the error 500 stacks on the service.log? Do you have a job definition example to replicate that error?

Out of the box, Rundeck 5.6.1 includes the Ansible plugin version 4.0.4. Could you try stopping the rundeck service, cleaning the libext directory, and then relaunching Rundeck? In that way, Rundeck can "regenerate" the plugins in the instance.

Regards.

[Jim Richard]

I tried removing all from the libext directory and restarting Rundeck and indeed "ansible-plugin-4.0.4.jar" is what ends up being background installed

But when I run my "generate inventory" job I get the following failure in my service log.

That job is simple, two steps.

1) run a shell script that pulls inventory from a dynamic source (FreeIPA) and drop that into an Ansible inventory file - this step is working as expected.

2) do the built in "Refresh Project Nodes" step - this step should run the playbook that is part of the Ansible plugin, the playbook that generates the Rundeck nodes

step two there is just the built in Work Flow step that has always existed.

[2024-10-15T03:10:41,549] ERROR resources.ExceptionCatchingResourceModelSource - [ResourceModelSource: 2.source (com.batix.rundeck.plugins.AnsibleResourceModelSourceFactory), project: AnsibleJobs] 

java.lang.NullPointerException: null

        at com.rundeck.plugins.ansible.plugin.AnsibleResourceModelSource.ansibleInventoryList(AnsibleResourceModelSource.java:717) ~[?:?]

        at com.rundeck.plugins.ansible.plugin.AnsibleResourceModelSource.getNodes(AnsibleResourceModelSource.java:403) ~[?:?]

        at com.dtolabs.rundeck.core.resources.ExceptionCatchingResourceModelSource.getNodes(ExceptionCatchingResourceModelSource.java:58) ~[rundeck-core-5.6.1-20241014.jar!/:?]

        at com.dtolabs.rundeck.core.resources.DelegateResourceModelSource.getNodes(DelegateResourceModelSource.java:35) ~[rundeck-core-5.6.1-20241014.jar!/:?]

        at com.dtolabs.rundeck.core.common.ProjectNodeSupport.getNodeSet(ProjectNodeSupport.java:137) ~[rundeck-core-5.6.1-20241014.jar!/:?]

        at com.dtolabs.rundeck.core.common.ProjectNodeSupport$ProjectNodesSource.getNodes(ProjectNodeSupport.java:358) ~[rundeck-core-5.6.1-20241014.jar!/:?]

        at com.dtolabs.rundeck.core.resources.ExceptionCatchingResourceModelSource.getNodes(ExceptionCatchingResourceModelSource.java:58) ~[rundeck-core-5.6.1-20241014.jar!/:?]

        at com.dtolabs.rundeck.core.resources.ResourceModelSource$getNodes.call(Unknown Source) ~[?:?]

        at rundeck.services.nodes.CachedProjectNodes.reloadNodeSet(CachedProjectNodes.groovy:44) ~[classes!/:?]

        at rundeck.services.nodes.CachedProjectNodes$reloadNodeSet.call(Unknown Source) ~[?:?]

        at rundeck.services.NodeService$_loadNodes_closure4.doCall(NodeService.groovy:295) ~[classes!/:?]

        at rundeck.services.NodeService$_loadNodes_closure4.doCall(NodeService.groovy) ~[classes!/:?]

        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]

        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]

        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]

        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]

        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107) ~[groovy-3.0.19.jar!/:3.0.19]

        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) ~[groovy-3.0.19.jar!/:3.0.19]

        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274) ~[groovy-3.0.19.jar!/:3.0.19]

        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030) ~[groovy-3.0.19.jar!/:3.0.19]

        at groovy.lang.Closure.call(Closure.java:427) ~[groovy-3.0.19.jar!/:3.0.19]

        at groovy.lang.Closure.call(Closure.java:406) ~[groovy-3.0.19.jar!/:3.0.19]

        at com.codahale.metrics.Timer.time(Timer.java:118) ~[metrics-core-4.2.22.jar!/:4.2.22]

        at org.grails.plugins.metricsweb.MetricService.withTimer(MetricService.groovy:61) ~[grails-metricsweb-5.6.1-20241014-plain.jar!/:?]

        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]

        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]

        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]

        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]

        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48) ~[groovy-3.0.19.jar!/:3.0.19]

        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrap.invoke(PogoMetaMethodSite.java:178) ~[groovy-3.0.19.jar!/:3.0.19]

        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:69) ~[groovy-3.0.19.jar!/:3.0.19]

        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:157) ~[groovy-3.0.19.jar!/:3.0.19]

        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:112) ~[groovy-3.0.19.jar!/:3.0.19]

        at rundeck.services.NodeService$_loadNodes_closure5.doCall(NodeService.groovy:302) ~[classes!/:?]

        at rundeck.services.NodeService$_loadNodes_closure5.doCall(NodeService.groovy) ~[classes!/:?]

        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]

        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]

        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]

        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]

        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107) ~[groovy-3.0.19.jar!/:3.0.19]

        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) ~[groovy-3.0.19.jar!/:3.0.19]

        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274) ~[groovy-3.0.19.jar!/:3.0.19]

        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030) ~[groovy-3.0.19.jar!/:3.0.19]

        at groovy.lang.Closure.call(Closure.java:427) ~[groovy-3.0.19.jar!/:3.0.19]

        at groovy.lang.Closure.call(Closure.java:406) ~[groovy-3.0.19.jar!/:3.0.19]

        at groovy.lang.Closure.run(Closure.java:498) ~[groovy-3.0.19.jar!/:3.0.19]

        at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:284) ~[spring-core-5.3.34.jar!/:5.3.34]

        at java.lang.Thread.run(Thread.java:829) ~[?:?]

[2024-10-15T03:10:41,556] WARN  resources.LoggingResourceModelSourceCache - [ResourceModelSource: 2.source (com.batix.rundeck.plugins.AnsibleResourceModelSourceFactory), project: AnsibleJobs] Returning cached model data

[Jim Richard]

what's even more confusing is these two screen shots from the same running instance

and on the file system we have "ansible-plugin-4.0.4.jar "

so I'm like, which is it?

4.0.4

4.0.5-SNAPSHOT

or 3.0.1

I'm not surprised that a failure is happening if the system is confused about which plugin version is there, I've been using the plugin since way back when it was new and I know lot's has changed over the years.

[rac...@rundeck.com]

Hi,

I couldn’t reproduce your issue but I found a way to accomplish your goal; it seems that the default model source cache is applied over the Refresh Project node step. So, let me share what worked for me on my end:

1- Go to the Ansible Model Source, click on the Configuration tab, and disable the “Use Asynchronous Cache” (I also decreased the “Cache Delay” time, I’ve tested with 2 instead of 30).

2- Redesing your workflow in the following way:

a. Create a child job with the action you want to dispatch to your Inventory.

- defaultTab: nodes description: '' executionEnabled: true id: 5a781bf8-08af-4f27-8590-7ffe4205bd48 loglevel: INFO name: ExecuteAction nodeFilterEditable: false nodefilters: dispatch: excludePrecedence: true keepgoing: false rankOrder: ascending successOnEmptyNodeFilter: false threadcount: '1' filter: .* nodesSelectedByDefault: true plugins: ExecutionLifecycle: {} scheduleEnabled: true sequence: commands: - autoSecureInput: 'false' fileExtension: .sh interpreterArgsQuoted: false passSecureInput: 'false' script: |- sleep 2 whoami scriptInterpreter: /bin/bash keepgoing: false strategy: node-first uuid: 5a781bf8-08af-4f27-8590-7ffe4205bd48

b. Create a parent job with three steps (this job must be configured to execute locally). The first one generates/copies the Ansible Inventory (I tested with a simple YAML inventory). The third step is a Job Reference Step to the first job.

- defaultTab: nodes description: '' executionEnabled: true id: 1e99c753-7149-4067-b06a-1a5f202d2ad9 loglevel: INFO name: ParentJob nodeFilterEditable: false plugins: ExecutionLifecycle: {} scheduleEnabled: true sequence: commands: - autoSecureInput: 'false' fileExtension: .sh interpreterArgsQuoted: false passSecureInput: 'false' script: |- cp /home/user/Downloads/ansible/config/hosts.yaml /home/user/Downloads/ echo "done" scriptInterpreter: /bin/bash - configuration: sleep: '6' description: refreshing project nodes.... nodeStep: false type: source-refresh-plugin - jobref: group: '' name: ExecuteAction nodeStep: 'true' uuid: 5a781bf8-08af-4f27-8590-7ffe4205bd48 keepgoing: false strategy: node-first uuid: 1e99c753-7149-4067-b06a-1a5f202d2ad9

3- Save and execute the Parent Job (and observe the service log behavior).

4- Check the output.

Some tips:

• As you see I added some sleep time in the child’s job, this is just for “sanity” You can extend/reduce this time.
• Regarding the plugins page, it needs an update for sure, but internally Rundeck executes the plugins stored in the libext directory.

Regards.

​

[Jim Richard]

thanks for looking into this.

in fact though, this is not related to a specific job, it's all about the Ansible plugin node generation process. The Null Pointer error gets triggered anytime you touch Project Settings > Edit Nodes

just for fun I made the error how up in the UI as well (not just in service.log) - I wasn't expecting this but sure enough...

so, say you have a Project with an Ansible node source configured.

now, manually remove the Ansible plugin, via the UI or from the filesystem

now go back to your project, edit nodes and you will see ... the exact same error that I was seeing in the service.log with the plugin or at least Rundeck saying that, the Ansible plugin is installed.

obviously this error in UI happened because I caused it, inadvertently actually, but still, this shows that the root of the problem seems to be that Rundeck is confused about which Ansible plugin may or may not be installed.

Why is this happening, why can't you reproduce, why can't I reproduce it either on another instance? \

I have some theories...

like it could be related to the fact that I tried the plugin blocklist per this advice/guidance:

https://groups.google.com/g/rundeck-discuss/c/RYaPgt2B8ds/m/dPRrzRH3AgAJ

and that worked but I recently decided to remove.

I removed the setting from rundeck-config.properties and removed the blocklist file, restarted Rundeck and ever since, this Rundeck instance with regards to the Ansible plugin, has been misbehaving.

one thing for sure, the issue is not related to a Job, it's at the Project level, the  Node source configuration

[rac...@rundeck.com]

Hi Jim,

I'm still digging into this. I have some questions.

1. Could you share an inventory example (the inventory generated from the first step)? Is it a YAML-based inventory? (Also, can you test with a "static" inventory?).

2. How is configured your Ansible Model Source? Can you share this? (please hide/change any sensitive data)

3. Which Java version are you using to run the Rundeck instance?

4. Which Ansible version are you using in your Integration?

5. What kind of Rundeck instance are you using? (RPM, DEB, WAR, etc).

Also, regarding this issue.

Can you double-check that the inventory file generated in the first step is well-located/well-formed and reachable by the Ansible model source? That could be the root cause of the error 400.

Thank you.

[Jim Richard]

I also created a GH issue: https://github.com/rundeck/rundeck/issues/9401

And thank you again for taking the time to dig into this issue, it's much appreciated :) 

let me clarify again, this 100% for sure has nothing to do with a specific "job"

a Null Pointer error will get silently (as in the user is not aware) thrown/shown in server.log and "cached model data" returned if you do anything on the Nodes/Source config for the project.

Now, I do have a job that generates the Ansible Inventory file, (it generates a static file that I point my Rundeck Ansible Node Source config at) and that file is yaml based.

I'm not sure if I wanna post that inventory file here. It's about 1500 lines and contains no facts, just hosts and host groups.

The project config is attached.

The ansible config file, inventory file and such all sit in the rundeck user's home dir and are owned by rundeck:rundeck

Rundeck version 5.61 on Rocky Linux 8, VMware VM, 16 vCPU's + 16 GB memory

Ansible version:

ansible 2.9.27

  config file = /home/rundeck/ansible.cfg

  configured module search path = [u'/home/rundeck/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']

  ansible python module location = /usr/lib/python2.7/site-packages/ansible

  executable location = /bin/ansible

  python version = 2.7.5 (default, Nov 14 2023, 16:14:06) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]

Rundeck version 5.61 on Rocky Linux 8, VMware VM, 16 vCPU's + 16 GB memory

Java version:

openjdk 11.0.23 2024-04-16 LTS

OpenJDK Runtime Environment (Red_Hat-11.0.23.0.9-2.el7_9) (build 11.0.23+9-LTS)

OpenJDK 64-Bit Server VM (Red_Hat-11.0.23.0.9-2.el7_9) (build 11.0.23+9-LTS, mixed mode, sharing)

[Jim Richard]

and it's an RPM install with a Postgres database running locally

postgres 13

[rac...@rundeck.com]

Hi Jim,

let me clarify again, this 100% for sure has nothing to do with a specific “job”

I know :) However, when I check the source code, the error occurs when the plugin validates the inventory (weird). I want to see if we have a problem with the Inventory’s format or the amount of data, like this issue.

So, could you try to publish an Inventory example to test?

Thank you for your feedback!

​

[Jim Richard]

understood, example inventory file attached, this is typical, size stays about like this