[ANN]: Bring your remote execution framework to 1.3 !

[Alex-SF]

Some exciting news, fellow rundeck users!

The next rundeck release includes, perhaps, the most significant
feature to date: a plugin framework letting you integrate the remote
execution framework of your choice. http://rundeck.org/1.3beta/RunDeck-Guide.html#rundeck-plugins
During the 1.3beta period, we hope you will try using the plugin
framework, as well as, develop plugins of your own!

Download the 1.3 beta from here: http://build.rundeck.org/job/release-1.3/

As you know, current versions of rundeck use SSH as the remote
execution mechanism (based on Jsch). SSH is a good low common
denominator to manage remote execution of command strings and script
files but may not be practical nor ideal for your use cases. For
example, you may be a windows user and find the installation of an SSH
server (via Cygwin maybe) undesirable and prefer a more windows
oriented method (eg, PsExec, WMI, etc). Others may not want to use SSH
at all preferring mechanisms like MCollective or Chef's Knife.
Finally, some may need to integrate to a custom XML-RPC based
mechanism.

With these needs in mind, rundeck has been refactored to support the
notion of "Services" that provide functionality for the different
steps necessary to execute workflows, jobs, and commands across
multiple nodes.

Each Service makes use of "Providers". Each Provider has a unique
"Provider Name" that is used to identify it, and most Services have
default Providers unless you specify different ones to use.

There are currently two types of Providers that can be used and
developed for rundeck:

1. Node Executor Providers - these define ways of executing a command
on a single Node (local or remote).
2. File Copier Providers - these define ways of copying files to a
single Node.

Support for multiple node dispatcher plugins is planned for a future
release.

Check out the documentation for more info on developing plugins:
http://rundeck.org/1.3beta/RunDeck-Guide.html#plugin-development

Questions that you can help us to address for the 1.3 release:

Workflow failures:
* How should nodeset failure be handled? should "failed nodes"
include only failed nodes at the top-level job (not include job
references)?
* Detailed error messages are printed when multiple steps or nodes
fail, but the formatting is not pretty. How should these be
displayed?

Plugins:
* enabled by default? (yes). framework.plugins.enabled=true/false
property can change it
* do we need a way to configure enabled/disabled plugins by name?
* location for plugins dir, currently $RDECK_BASE/libext (launcher)
or /var/lib/rundeck/libext (rpm)
* java plugins - do we need support for external lib jars?
* WINDOWS - needs your testing and likely needs updating for support

Workflow context tracking:
* the format displayed in GUI should likely be changed. right now it
is "#-type" to indicate step number and type of step, and jobrefs are
appended with ":#-type".

Download the 1.3beta: http://build.rundeck.org/job/release-1.3/
Documentation for 1.3beta: http://rundeck.org/1.3beta/

[Greg Schueler]

For prospective plugin developers, there are a few examples to look at in the release-1.3 branch:

* An example script plugin using the system "ssh" and "scp" commands in lieu of the java based mechanism: https://github.com/dtolabs/rundeck/tree/release-1.3/examples/ssh-script-plugin

* Two Java plugins that are currently bundled in the 1.3 build: https://github.com/dtolabs/rundeck/tree/release-1.3/plugins

stub-plugin: simply echoes expected command instead of running anything (could use for testing)

script-plugin: lets you define external script to execute via node attributes or project properties instead of built-in java ssh. 

[Anthony Shortland]

I've got in mind to writing a native Java OpenSSH plugin that calls /usr/bin/ssh and /usr/bin/scp directly for efficiency's sake, but as a first step I grabbed the example script plugin Greg refers to below and gave that a whirl!

• Cloned the Rundeck repository on GitHub (https://github.com/dtolabs/rundeck) to get something to work with:

[anthony@centos55 tmp]$ git clone git://github.com/dtolabs/rundeck.git

Initialized empty Git repository in /home/anthony/tmp/rundeck/.git/

remote: Counting objects: 10289, done.

remote: Compressing objects: 100% (3733/3733), done.

remote: Total 10289 (delta 5904), reused 9639 (delta 5441)

Receiving objects: 100% (10289/10289), 33.40 MiB | 2086 KiB/s, done.

Resolving deltas: 100% (5904/5904), done.

Checking out files: 100% (1185/1185), done.

• Switched to the release-1.3 branch of the cloned repository:

[anthony@centos55 rundeck]$ git checkout origin/release-1.3

Note: moving to "origin/release-1.3" which isn't a local branch

If you want to create a new branch from this checkout, you may do so

(now or later) by using -b with the checkout command again. Example:

  git checkout -b <new_branch_name>

HEAD is now at ba5d969... Expect call to frameworkService.projects in copy test [#243]

• Located the example code:

[anthony@centos55 rundeck]$ ls examples/ssh-script-plugin/

Makefile  README.md  ssh-example-1.0-plugin

• Built the plugin:

[anthony@centos55 ssh-script-plugin]$ make

zip -r ssh-example-1.0-plugin.zip ssh-example-1.0-plugin

  adding: ssh-example-1.0-plugin/ (stored 0%)

  adding: ssh-example-1.0-plugin/plugin.yaml (deflated 59%)

  adding: ssh-example-1.0-plugin/contents/ (stored 0%)

  adding: ssh-example-1.0-plugin/contents/ssh-copy.sh (deflated 58%)

  adding: ssh-example-1.0-plugin/contents/ssh-exec.sh (deflated 54%)

[anthony@centos55 ssh-script-plugin]$ ls

Makefile  README.md  ssh-example-1.0-plugin  ssh-example-1.0-plugin.zip

• At this point you can follow the Usage instructions in README.md to install the plugin: 

[rundeck@centos55 etc]$ ls /var/lib/rundeck/libext
rundeck-script-plugin-1.3.0.jar  rundeck-stub-plugin-1.3.0.jar  ssh-example-1.0-plugin.zip

• I configured a single node to use the plug-in as follows (could also be configured at the project or even framework level to apply to all nodes):

[rundeck@centos55 etc]$ fgrep node1 /var/rundeck/projects/Development/etc/resources.xml 
  <node name="node1" description="Rundeck server node" tags="" hostname="node1" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user1" node-executor="ssh-example" file-copier="ssh-example" ssh-keyfile="/home/rundeck/.ssh/rundeck.id_rsa" scp-dir="/var/tmp"/>

• Ran a couple of sample commands:

[rundeck@centos55 etc]$ dispatch -L -I node1 -- pwd

/home/user1

[rundeck@centos55 etc]$ dispatch -v -L -I node1 -- pwd

verbose: [ssh-example] execCommand started, command: pwd

verbose: [ssh-example] scriptargs: ${node.username} ${node.hostname} ${exec.command}, interpreter: /bin/bash

verbose: [ssh-example] executing: /bin/bash /var/lib/rundeck/libext/cache/ssh-example-1.0-plugin/ssh-exec.sh user1 node1 pwd

/home/user1

verbose: [ssh-example]: result code: 0, success: true

[rundeck@centos55 etc]$ cat /tmp/doit
#!/bin/bash

echo "running doit"
[rundeck@centos55 etc]$ dispatch -L -I node1 -s /tmp/doit
running doit
[rundeck@centos55 etc]$ dispatch -v -L -I node1 -s /tmp/doit
verbose: [ssh-example] executing: /bin/bash /var/lib/rundeck/libext/cache/ssh-example-1.0-plugin/ssh-copy.sh user1 node1 /var/lib/rundeck/var/dispatch1870916665261623183.tmp
verbose: [ssh-example]: result filepath: /var/tmp/dispatch1870916665261623183.tmp
verbose: [ssh-example] execCommand started, command: chmod +x /var/tmp/dispatch1870916665261623183.tmp
verbose: [ssh-example] scriptargs: ${node.username} ${node.hostname} ${exec.command}, interpreter: /bin/bash
verbose: [ssh-example] executing: /bin/bash /var/lib/rundeck/libext/cache/ssh-example-1.0-plugin/ssh-exec.sh user1 node1 chmod +x /var/tmp/dispatch1870916665261623183.tmp
verbose: [ssh-example]: result code: 0, success: true
verbose: [ssh-example] execCommand started, command: /var/tmp/dispatch1870916665261623183.tmp
verbose: [ssh-example] scriptargs: ${node.username} ${node.hostname} ${exec.command}, interpreter: /bin/bash
verbose: [ssh-example] executing: /bin/bash /var/lib/rundeck/libext/cache/ssh-example-1.0-plugin/ssh-exec.sh user1 node1 /var/tmp/dispatch1870916665261623183.tmp
running doit
verbose: [ssh-example]: result code: 0, success: true

Sweet!

Anthony.

[Alex-SF]

Very nice.

On Apr 27, 6:25 pm, Anthony Shortland <anthony.shortl...@me.com>
wrote:

> > * An example script plugin using the system "ssh" and "scp" commands in lieu of the java based mechanism:https://github.com/dtolabs/rundeck/tree/release-1.3/examples/ssh-scri...

>
> > * Two Java plugins that are currently bundled in the 1.3 build:https://github.com/dtolabs/rundeck/tree/release-1.3/plugins
>
> > stub-plugin: simply echoes expected command instead of running anything (could use for testing)
> > script-plugin: lets you define external script to execute via node attributes or project properties instead of built-in java ssh.
>
> > On Wed, Apr 20, 2011 at 2:30 PM, Alex-SF <aho...@users.sourceforge.net> wrote:
> > Some exciting news, fellow rundeck users!
>
> > The next rundeck release includes, perhaps, the most significant
> > feature to date: a plugin framework letting you integrate the remote

> > execution framework of your choice.http://rundeck.org/1.3beta/RunDeck-Guide.html#rundeck-plugins

[Noah Campbell]

Now we need the rsh for performance ;)

Nice work anthony...is it ready to include in RunDeck?

-Noah

[Anthony Shortland]

Not in the distribution, just in the examples ... I guess we can include a cleaned up version for 1.3.

[Alex-SF]

Maybe on this topic we should discuss how future plugins will be
shared. There's a number of approaches we could consider.

Could be as loosely as separate github projects (or other project
hosters) that are linked to from a Rundeck wiki page. On the other end
of the extreme, we could even envision a plugin publishing model like
Jenkins/Hudson uses.

Probably other ideas, too!

On Apr 28, 8:33 am, Anthony Shortland <anthony.shortl...@me.com>
wrote:

> >>> * An example script plugin using the system "ssh" and "scp" commands in lieu of the java based mechanism:https://github.com/dtolabs/rundeck/tree/release-1.3/examples/ssh-scri...

>
> >>> * Two Java plugins that are currently bundled in the 1.3 build:https://github.com/dtolabs/rundeck/tree/release-1.3/plugins
>
> >>> stub-plugin: simply echoes expected command instead of running anything (could use for testing)
> >>> script-plugin: lets you define external script to execute via node attributes or project properties instead of built-in java ssh.
>
> >>> On Wed, Apr 20, 2011 at 2:30 PM, Alex-SF <aho...@users.sourceforge.net> wrote:
> >>> Some exciting news, fellow rundeck users!
>
> >>> The next rundeck release includes, perhaps, the most significant
> >>> feature to date: a plugin framework letting you integrate the remote

> >>> execution framework of your choice.http://rundeck.org/1.3beta/RunDeck-Guide.html#rundeck-plugins

[Anthony Shortland]

I'd prefer a model that allowed ("certified") plug-ins to be automatically installed ... so the plug-in publishing model is where its at ... is there a standard framework for that or does each project roll its own?

[redhot]

Hi Alex,

I prefer the Hudson/Jenkins plugin model where it can be available from a central location. Additionally, some core plugins should be packaged with the RPM/war/launcher. Others can be installed on as needed basis. What constitutes core could be rundeck community driven.

-redhot