Configuring Rundeck and ControlTier as part of a "loosely-coupled" tool-chain

[Anthony Shortland]

If you've been following the Rundeck project (www.rundeck.org) since it was announced on this list (http://groups.google.com/group/controltier/browse_thread/thread/20eda1cc6066dc2f#) you'll have seen that it is quickly evolving a rich set of feature and function that goes well beyond the capabilities of it's origins as CtlCenter under ControlTier 3.6.0. 

Rundeck is fast approaching its 1.3 release (http://groups.google.com/group/rundeck-discuss/browse_thread/thread/fe673999bb184bc8) and is a clear winner as a web-based command dispatching console that provides the orchestration "glue" to tie together "loosely-coupled" tool-chains to support build and deployment automation solutions (e.g. we've seen plug-ins contributed from lots of other projects such as Jenkins, Puppet and Chef).

The fact is that though ControlTier appears to be a tightly integrated framework, at the source level all the key components that make up the install are developed and built  separately and the integrated packaging has only been a matter of convenience. 

There is nothing preventing a ControlTier user with an operational solution based on a Workbench resource model and a set of Ctl modules taking advantage of these latest innovations by carefully reconfiguring the installation to drop certain components and switch to the newer stuff incrementally. In other words there's a clean migration path to adopting the "loosely-coupled" tool-chain approach rather than continue to rely on all the tightly-coupled components included in the ControlTier install.

That said, the most obvious first step is to replace CtlCenter with Rundeck.

In this case, the general approach is to install both the Rundeck and ControlTier servers (side-by-side in the network, but not necessarily on the same system -  though there are some advantages to that) and install the ControlTier client on each managed node, registering them with a Workbench project as usual. 

For the purposes of this write-up I installed the ControlTier server alongside the Rundeck server on a host called "rundeck" and installed the ControlTier client under six different user accounts setting up the "Development" project both in Rundeck and ControlTier:

[rundeck@rundeck etc]$ dispatch -p Development -L -I 'node.*' -- ctl -p Development -m netutil -c listening -- -port 8080

[user1@node1 Development.netutil listening][WARN] true
[user2@node2 Development.netutil listening][WARN] true
[user3@node3 Development.netutil listening][WARN] true
[user4@node4 Development.netutil listening][WARN] true
[user5@node5 Development.netutil listening][WARN] true
[user6@node6 Development.netutil listening][WARN] true

[rundeck@rundeck etc]$ dispatch -p Development -L -I 'node.*' -- ctl-project -p Development -a install
"resource-install" command running for resource: aMyServiceInstance[MyService]
"resource-install" command running for resource: aMyServiceInstance[MyService]
"resource-install" command running for resource: aMyServiceInstance[MyService]
"resource-install" command running for resource: aMyServiceInstance[MyService]
"resource-install" command running for resource: aMyServiceInstance[MyService]
"resource-install" command running for resource: aMyServiceInstance[MyService]

[rundeck@rundeck etc]$ dispatch -p Development -L -I 'node.*' -- ctl -p Development -t MyService -r aMyServiceInstance -c id
[user1@node1 Development.MyService.aMyServiceInstance id][INFO] uid=505(user1) gid=505(user1) groups=104(ctier),505(user1) context=user_u:system_r:java_t
[user2@node2 Development.MyService.aMyServiceInstance id][INFO] uid=506(user2) gid=506(user2) groups=104(ctier),506(user2) context=user_u:system_r:java_t
[user3@node3 Development.MyService.aMyServiceInstance id][INFO] uid=507(user3) gid=507(user3) groups=104(ctier),507(user3) context=user_u:system_r:java_t
[user4@node4 Development.MyService.aMyServiceInstance id][INFO] uid=508(user4) gid=508(user4) groups=104(ctier),508(user4) context=user_u:system_r:java_t
[user5@node5 Development.MyService.aMyServiceInstance id][INFO] uid=509(user5) gid=509(user5) groups=104(ctier),509(user5) context=user_u:system_r:java_t
[user6@node6 Development.MyService.aMyServiceInstance id][INFO] uid=510(user6) gid=510(user6) groups=104(ctier),510(user6) context=user_u:system_r:java_t

... as you can see from these examples, Rundeck (dispatch) manages the orchestration of distributed command execution (works fine from the web UI too, of course), and ControlTier (ctl) manages the execution of modular automation on each client (both a static named command and a service instance based command in these examples).

There are a couple of things to note here:

Firstly, Rundeck is inherently a node-orientated orchestration tool (i.e. it runs commands on selections of nodes ... "-I 'node.*'" in this case), and so what we're seeing is an approach that differs from the ControlTier Site-orientated dispatch model. No ControlTier distributed orchestration is being used here at all, as all the ControlTier command executions are local to one or other client nodes. But note that while then there's no need to retain Site resources in the ControlTier model for command dispatching, they can still play a role in site-wide configuration management. A case in point is replacing the ControlTier Update workflow (Change-Dependencies followed by Deploy at the Site level) with a Rundeck job-level workflow that first dispatches the Change-Dependencies command to a centrally deployed Site resource (probably on the ControlTier server) and then dispatches the Deploy command to all the nodes running the affected Service(s).

Secondly, there's the need to synchronize the Rundeck and ControlTier resource models since orchestration will fail if the two tools have different views of the set of nodes and services to manage. 

Out of the box, Rundeck uses a variant of resource XML format to manage the node list it operates on:

[rundeck@rundeck etc]$ cat /var/rundeck/projects/Development/etc/resources.xml 

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE project PUBLIC "-//DTO Labs Inc.//DTD Resources Document 1.0//EN" "project.dtd">

<project>

  <node name="rundeck" type="Node" description="Rundeck server node" tags="" hostname="rundeck" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="rundeck" editUrl="" remoteUrl=""/>

  <node name="node1" type="Node" description="Rundeck server node" tags="" hostname="node1" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user1" editUrl="" remoteUrl=""/>

  <node name="node2" type="Node" description="Rundeck server node" tags="" hostname="node2" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user2" editUrl="" remoteUrl=""/>

  <node name="node3" type="Node" description="Rundeck server node" tags="" hostname="node3" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user3" editUrl="" remoteUrl=""/>

  <node name="node4" type="Node" description="Rundeck server node" tags="" hostname="node4" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user4" editUrl="" remoteUrl=""/>

  <node name="node5" type="Node" description="Rundeck server node" tags="" hostname="node5" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user5" editUrl="" remoteUrl=""/>

  <node name="node6" type="Node" description="Rundeck server node" tags="" hostname="node6" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user6" editUrl="" remoteUrl=""/>

</project>

... while ControlTier is driven by the Workbench resource model, of course.

Rather than manage these two separately, it turns out that we can take advantage of Rundeck's resource model provider feature (http://rundeck.org/docs/RunDeck-Guide.html#resource-model-provider) to allow the Workbench resource model to supply the node resources needed by Rundeck.

With this is mind I've added the "generate-rundeck-resources" command to Projectbuilder (https://sourceforge.net/tracker/?func=detail&atid=779850&aid=3291471&group_id=151079):

[anthony@rundeck ProjectBuilder]$ svn diff type.xml 

Index: type.xml

===================================================================

--- type.xml    (revision 2257)

+++ type.xml    (working copy)

@@ -38,6 +38,7 @@

       <attribute-default name="organizationURL" value="http://controltier.org"/>

       <attribute-default name="defaults" value="${modules.dir}/ProjectBuilder/templates/defaults.properties"/>

       <attribute-default name="templateDir" value="${modules.dir}/ProjectBuilder/templates"/>

+      <attribute-default name="rundeckResourcesFile" value="${env.JETTY_HOME}/controltier/resources.${context.project}.xml"/>

       <attribute-default name="packageExtension" value="jar"/>

       <attribute-default name="packageFilebase" value=".*-seed"/>

       <attribute-default name="packageType" value="jar"/>

@@ -2208,6 +2209,33 @@

         </opts>

        </command>

 

+

+       <command name="generate-rundeck-resources" description="Generates a Rundeck compatible resource XML file from the current project context" command-type="AntCommand" is-static="true">

+         <implementation>

+           <echo>Generating &quot;${opts.rundeckresourcesfile}&quot; ...</echo>

+           <echo append="false" file="${opts.rundeckresourcesfile}">&lt;?xml version="1.0" encoding="UTF-8"?&gt;${line.separator}</echo>

+           <echo append="true" file="${opts.rundeckresourcesfile}">&lt;!DOCTYPE project PUBLIC "-//DTO Labs Inc.//DTD Resources Document 1.0//EN" "project.dtd"&gt;${line.separator}</echo>

+           <echo append="true" file="${opts.rundeckresourcesfile}">${line.separator}</echo>

+           <echo append="true" file="${opts.rundeckresourcesfile}">&lt;project&gt;${line.separator}</echo>

+           <apply-macro>

+               <entityquery select="default.name,default.type,default.description,default.selectedResourceTags,default.osFamily,default.osName,default.osArch,default.osVersion,default.hostname,default.ctlUsername" depot="${context.depot}">

+                  <typerestriction typeName="Node" />

+               </entityquery>

+ 

+               <sequential>

+                  <propertyregex property="tags" override="true" input="@{default.selectedResourceTags}" regexp="\[(.*)\]" select="\1"/>

+                  <echo append="true" file="${opts.rundeckresourcesfile}">&lt;node name="@{default.name}" type="@{default.type}" description="@{default.description}" tags="${tags}" hostname="@{default.hostname}" osArch="@{default.osArch}" osFamily="@{default.osFamily}" osName="@{default.osName}" osVersion="@{default.osVersion}" username="@{default.ctlUsername}" editUrl="" remoteUrl=""/&gt;${line.separator}</echo>

+                </sequential>

+           </apply-macro>

+           <echo append="true" file="${opts.rundeckresourcesfile}">&lt;/project&gt;${line.separator}</echo>

+         </implementation>

+        <opts>

+          <opt parameter="basedir" description="base directory" required="false" property="opts.basedir" type="string" defaultproperty="entity.attribute.basedir"/>

+          <opt parameter="targetdir" description="dir containing build" required="false" property="opts.targetdir" type="string" defaultproperty="entity.attribute.targetdir"/>

+          <opt parameter="rundeckresourcesfile" description="file containing Rundeck resources" required="false" property="opts.rundeckresourcesfile" type="string" default="" defaultproperty="resource.attribute.rundeckResourcesFile"/>

+        </opts>

+       </command>

+

     <!--

       **

       ** load-library

@@ -2766,4 +2794,13 @@

       <attribute name="templateDir" type-property="settingValue"/>

     </attributes>

   </type>

+  <type role="concrete" uniqueInstances="true" name="ProjectBuilderRundeckResourcesFile"  order="Setting">

+    <description>File containing Rundeck resources</description>

+    <supertype>

+      <typereference name="ProjectBuilderSetting"/>

+    </supertype>

+    <attributes>

+      <attribute name="rundeckResourcesFile" type-property="settingValue"/>

+    </attributes>

+  </type>

 </types>

The heart of the implementation relies on a Workbench entity query to select all the node resources from the project to feed an apply macro which iterates across them to generate Rundeck-style resource XML to a file (not surprisingly there's a tidy 1:1 map between ControlTier node properties and Rundeck node attributes).

Note that the implementation includes the "rundeckResourcesFile" setting/attribute with a type level default attribute value of "${env.JETTY_HOME}/controltier/resources.${context.project}.xml". This is the file where the generated resources are saved. The default value takes advantage of the fact that the ControlTier server Jetty installation includes a "free" document root that can serve up static content.

Taking this approach allows generate-rundeck-resources to be run on the ControlTier server:

[ctier@rundeck src]$ ctl -p Development -t ProjectBuilder -r development -c generate-rundeck-resources

Generating "/opt/ctier/pkgs/jetty-6.1.21/controltier/resources.Development.xml" ...

[ctier@rundeck src]$ cat /opt/ctier/pkgs/jetty-6.1.21/controltier/resources.Development.xml

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE project PUBLIC "-//DTO Labs Inc.//DTD Resources Document 1.0//EN" "project.dtd">

<project>

<node name="node5" type="Node" description="ControlTier managed node." tags="" hostname="node5" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user5" editUrl="" remoteUrl=""/>

<node name="node4" type="Node" description="ControlTier managed node." tags="" hostname="node4" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user4" editUrl="" remoteUrl=""/>

<node name="node3" type="Node" description="ControlTier managed node." tags="" hostname="node3" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user3" editUrl="" remoteUrl=""/>

<node name="node2" type="Node" description="ControlTier managed node." tags="" hostname="node2" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user2" editUrl="" remoteUrl=""/>

<node name="node6" type="Node" description="ControlTier managed node." tags="" hostname="node6" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user6" editUrl="" remoteUrl=""/>

<node name="node1" type="Node" description="ControlTier managed node." tags="mytag, yourtag" hostname="node1" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user1" editUrl="" remoteUrl=""/>

<node name="rundeck" type="Node" description="ControlTier managed node." tags="" hostname="rundeck" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="ctier" editUrl="" remoteUrl=""/>

</project>

... to create a Rundeck resources file available over the web from the ControlTier server ("http://rundeck:8080/resources.Development.xml" in this case):

[rundeck@rundeck etc]$ curl  http://rundeck:8080/resources.Development.xml

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE project PUBLIC "-//DTO Labs Inc.//DTD Resources Document 1.0//EN" "project.dtd">

<project>

<node name="node5" type="Node" description="ControlTier managed node." tags="" hostname="node5" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user5" editUrl="" remoteUrl=""/>

<node name="node4" type="Node" description="ControlTier managed node." tags="" hostname="node4" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user4" editUrl="" remoteUrl=""/>

<node name="node3" type="Node" description="ControlTier managed node." tags="" hostname="node3" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user3" editUrl="" remoteUrl=""/>

<node name="node2" type="Node" description="ControlTier managed node." tags="" hostname="node2" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user2" editUrl="" remoteUrl=""/>

<node name="node6" type="Node" description="ControlTier managed node." tags="" hostname="node6" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user6" editUrl="" remoteUrl=""/>

<node name="node1" type="Node" description="ControlTier managed node." tags="mytag, yourtag" hostname="node1" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="user1" editUrl="" remoteUrl=""/>

<node name="rundeck" type="Node" description="ControlTier managed node." tags="" hostname="rundeck" osArch="amd64" osFamily="unix" osName="Linux" osVersion="2.6.18-194.el5" username="ctier" editUrl="" remoteUrl=""/>

</project>

With this in place the Rundeck project can be reconfigured to use the ControlTier resource provider URL as follows:

[rundeck@rundeck etc]$ pwd

/var/rundeck/projects/Development/etc

[rundeck@rundeck etc]$ diff project.properties project.properties.orig 

19d18

< project.resources.url = http://rundeck:8080/resources.Development.xml

Finally, the process for keeping Rundeck's view of the nodes synchronized with your ControlTier project is:

1. Maintain the ControlTier project's node list as usual (by installing the client on new systems and maintaining node tags, etc in resource XML or Workbench).
2. Any time the node list is changed run generate-rundeck-resources (can be a Rundeck job, of course).
3. Each time the resources are regenerated click on "Update Nodes for project Development" on the Rundeck "Run" page.

This last (manual!) step has been a point of discussion on the Rundeck mail list (http://groups.google.com/group/rundeck-discuss/browse_thread/thread/9974a109810daae0) and there are plans afoot to automate it (probably for the upcoming 1.3 release).

Now. If you happen to have co-deployed the ControlTier server and Rundeck on the same system then you can by-pass the URL based resource provider setup and update the Rundeck project's resource XML file directly:

• Make sure your ControlTier user is in the rundeck group (in order to gain write access to the Rundeck project directories):

[ctier@rundeck ~]$ id

uid=102(ctier) gid=104(ctier) groups=104(ctier),504(rundeck) context=user_u:system_r:unconfined_t

• Point generate-rundeck-resources directly at the target Rundeck project:

[ctier@rundeck ~]$ ctl -p Development -t ProjectBuilder -r development -c generate-rundeck-resources -- -rundeckresourcesfile /var/rundeck/projects/Development/etc/resources.xml

Generating "/var/rundeck/projects/Development/etc/resources.xml" ...

... simple as that!

Now that Rundeck is integrated with ControlTier in this fashion, there's no need to run CtlCenter. Here's how to cleanly eliminate CtlCenter from the ControlTier installation:

• Starting with a standard ControlTier 3.6.0 installation (an RPM install in this case).
• Stop the ControlTier server:

[root@rundeck opt]# service ctier stop

Stopping Jetty: OK

• Remove the CtlCenter web application from the ControlTier Jetty installation:

[ctier@rundeck ~]$ rm -rf $JETTY_HOME/webapps/ctlcenter

• Eliminate the CtlCenter context from the Jetty configuration:

[ctier@rundeck ~]$ diff  $JETTY_HOME/etc/jetty.xml $JETTY_HOME/etc/jetty.xml.orig
168a169,174
>     <New id="ctlcenter"  class="org.mortbay.jetty.webapp.WebAppContext">
>       <Arg><Ref id="Contexts"/></Arg>
>       <Arg><SystemProperty name="jetty.home" default="."/>/webapps/ctlcenter</Arg>
>       <Arg>/ctlcenter</Arg>
>       <Set name="defaultsDescriptor"><SystemProperty name="jetty.home" default="."/>/etc/webdefault.xml</Set>
>     </New>

• Remove the CtlCenter application directory:

[ctier@rundeck ~]$ rm -rf $CTIER_ROOT/ctlcenter

• Restart ControlTier:

[root@rundeck ~]# service ctier start

Without CtlCenter web application the common log appender needs to be disabled for each installed client:

[ctier@rundeck jetty-6.1.21]$ diff $CTL_BASE/etc/log4j.properties $CTL_BASE/etc/log4j.properties.orig

9c9

< #log4j.logger.com.controltier.log.common = INFO,CommonLog

---

> log4j.logger.com.controltier.log.common = INFO,CommonLog

In summary: with this setup you can manage all jobs with Rundeck (migrating CtlCenter job XML is straightforward) and take advantage of Rundeck's latest features while maintaining your ControlTier-based Workbench module-based automation solution.

An excellent point of evolution to a "loosely-coupled" tool-chain! 

QED.

Anthony.