I'm looking to install rundeck in AWS. I've looked at the rundeck doco, there isn't much there.

[Evelyn Dibben]

The diagram shows an ELB, 2 or more EC2 instances, MySQL RDS and S3 for log storage.  

Does anyone have any recommendations/tips/stories for what they did?

Initially we are going to use rundeck for job scheduling.  We are going to start off controlling about 20 servers running about  600 jobs. Any information regarding EC2 and RDS sizing would be appreciated.

Thanks in advance.

[Evelyn Dibben]

Well I have things running so I'll just document what I did:

2 EC2 servers t2.small

1 ELB

S3 bucket

MySQL RDS db.t2.small

Steps:

1. Install rundeck on each EC2 server

yum install java-1.6.0 
rpm -Uvh http://repo.rundeck.org/latest.rpm 
yum install rundeck 
service rundeckd start




2.  Install MySQL on each server and create the rundeck database.




#yum install mysql




# mysql -h myamazonhost.rds.amazonaws.com -u admin --password

Enter password:

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 6930

Server version: 5.6.22-log MySQL Community Server (GPL)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| innodb             |

| mysql              |

| performance_schema |

+--------------------+

4 rows in set (0.01 sec)

mysql> create database rundeck;

Query OK, 1 row affected (0.01 sec)

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| innodb             |

| mysql              |

| performance_schema |

| rundeck            |

+--------------------+

5 rows in set (0.00 sec)




mysql> grant all on rundeck.* to 'rundeck'@'%' identified by 'xxxxxxxx';

Query OK, 0 rows affected (0.01 sec)

I verified connectivity from both nodes using mysql command line.  I should revisit the grant, I think I just got lazy ;)

3.     Add the folllowing lines the /etc/rundeck/rundeck-config.properties




dataSource.url = jdbc:mysql://myamazonhost.rds.amazonaws.com/rundeck?autoReconnect=true

dataSource.username = rundeck

dataSource.password = xxxxxxxx

4.  Set up the S3 log store

Download the rundeck-s3-log-plugin-1.0.0.jar from

https://github.com/rundeck-plugins/rundeck-s3-log-plugin/releases\

And put the file in 

/var/lib/rundeck/libext

Add the following lines to the /etc/rundeck/framework.properties

rundeck.execution.logs.fileStoragePlugin=org.rundeck.amazon-s3

#region

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.region=myregion

#name of the bucket

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.bucket=mys3bucket

#path to store the logs

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.path=logs/${job.project}/${job.execid}.log

Note:  It isn't obvious that the S3 bucket is actually being used.  I created a test job and ran it and then verified the existence of the log on the S3 bucket.




4.  Set up Rundeck to run as a cluster:




Add the following line to /etc/rundeck/rundeck-config.properties. The first parameter sets rundeck up as a cluster and the second changes the banner on the GUI.  I made the banner different on each node so I could see which node I was attaching to.

rundeck.clusterMode.enabled=true

rundeck.gui.title=Job Scheduler (1)

Perform the following to set the uuid of each server to be unique.  (uuidgen is a unix utility that creates a unique identifier)

 UUID=$(uuidgen)

 echo rundeck.server.uuid=$UUID >>  /etc/rundeck/framework.properties




5.  Configure the ELB




I won't go into all the details of instances, availability zones and security groups.  




Port Configuration:

4440 (HTTP) forwarding to 4440 (HTTP) Stickiness: LBCookieStickinessPolicy, expirationPeriod='0'

Not clear from the above but the Stickiness was required.  I chose this option:  Enable Load Balancer Generated Cookie Stickiness

6.  Set the grails value to point back to the ELB in the /etc/rundeck/rundeck-config.properties

grails.serverURL=http://myelb.elb.amazonaws.com:4440

You will need to restart rundeck on each server to put all the changes in place.  

I haven't started any type of performance testing yet, but so far this configuration seems to work very well.

[Alex Honor]

Hi Evelyn,

Thanks for this great write up. Let us know if you enhance this configuration any more.

Alex

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex Honor

[SimplifyOps, Inc | a...@simplifyops.com ]

Be sure to comment and vote on Rundeck Feature Development!

[Evelyn Dibben]

I forgot to mention that I'm using rsync running out of cron on each server to keep things synchronized (If there is a better way or if anyone has a more complete list I'd be grateful):

*/5 * * * *  rsync -avhe ssh --exclude "*.swp" /var/rundeck/projects rundeck@SERVER02:/var/rundeck >/tmp/rsync.projects

*/5 * * * *  rsync -avhe ssh --exclude "*.swp" /var/lib/rundeck/var/storage  rundeck@SERVER02:/var/lib/rundeck/var/ >/tmp/rsync.storage

*/5 * * * *  rsync -avhe ssh --exclude "*.swp" /etc/rundeck/realm.properties  rundeck@SERVER02:/etc/rundeck >/tmp/rsync.realm.properties

*/5 * * * *  rsync -avhe ssh --exclude "*.swp" /etc/rundeck/apitoken.aclpolicy  rundeck@SERVER02:/etc/rundeck >/tmp/rsync.apitoken.aclpolicy

My next steps are to work on authorization (don't want to use the realm.properties) and to add some windows servers (so far I'm just working with linux servers).

On Thursday, January 22, 2015 at 9:02:29 AM UTC-7, Evelyn Dibben wrote:

[Alex Honor]

Hi Evelyn,

You can configure the Storage Facility to use the RDB (eg Amazon RDS). See http://rundeck.org/docs/plugins-user-guide/configuring.html#storage-plugins

Also, in 2.5.0 you will also be able to store project configurations in the RDB.

We'd like to add ACL policy storage in the RDB as a new roadmap item.

That would leave the instance configuration outside of RDB.

Thanks

--

You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Evelyn Dibben]

I am using the MySQL as the back end RDB but not all of the information is stored in the database (as of this version) so I resorted to using rsync.  

On Thursday, January 22, 2015 at 9:02:29 AM UTC-7, Evelyn Dibben wrote:

[Evelyn Dibben]

Regarding the S3 log configuration:

The line:

rundeck.execution.logs.fileStoragePlugin=org.rundeck.amazon-s3

Should be in 

rundeck-config.properties

Not framework.properties

On Thursday, January 22, 2015 at 9:02:29 AM UTC-7, Evelyn Dibben wrote:

[rajesh...@gmail.com]

I done the same as you mentioned.

1.Downloaded rundeck-s3-log-plugin-1.0.1.jar and placed it in /var/lib/rundeck/libext

2.Placed the below parameter in /etc/rundeck/rundeck-config.properties

rundeck.execution.logs.fileStoragePlugin=org.rundeck.amazon-s3

3.Mentioned the below parameters in /etc/rundeck/framework.properties

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.AWSAccessKeyId=XXXX

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.AWSSecretKey=XXXX

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.region=XXXX

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.bucket=s3://XXXX

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.path=logs/${job.project}/${job.execid}.log

No hope. Please let me know Evelyn Dibben am i doing some thing wrong.