How to create init-scripts and immediately use them in a service?

[Martijn]

Hi all,

Looking for advice.

We're about to deploy queue-workers, which are essentially a shell-script which calls the PHP binary which keeps running and polling the jobqueue for jobs. Each website that we deploy to a server will get a queue-worker. These workers (there will be multiple per server) need to be kept running and should be started on boot, etc.

Sound like an init-script for each worker would do the trick. My goal is to make it easy for developers to deploy new workers, i.e. they should be able to create a new init-script, pointing to the location of their shell-script, and be able to count on it getting deployed and started on the server.

In other words:

1. The developer writes a shell-script called queue.sh and deploys it in the root of the website's directory. The shell-script starts the php cli binary which keeps running until it's stopped or dies for some reason. 

2. They call a single define in our Puppet code that takes the path to queue.sh as a parameter and ensures the init-script is created from a template.

3. The server picks up the init-script and starts managing the service.

3. The init-script takes care of running the queue.sh script at boot and can stop/restart it when needed. 

4. Repeat for each website.

For the moment, we're on Ubuntu 14.04 so Upstart scripts would be a logical choice. In the future we'll move to Ubuntu 16.04, with systemd.

I created a define that we can add to a node manifest. Here's an example of how I imagine calling this define:

site::queueworker { 'website01': path => '/var/www/website01/queue.sh' }

This drops a new Upstart init-script in /etc/init/ based on a template:

/etc/init/queue-website01.conf

I'm unsure if/how I can immediately use this new Upstart service in a service, e.g.: service {'queue-website01': ensure => running, enable => true}. I mean, how do Puppet and the init system know about the new service?

Will that work, or am I going about this completely the wrong way? Should I be using supervised or something else instead?

Thanks for any advice you can give.

Martijn

[Bas van de Wiel]

Hi Martijn,

Intriguing construction but let's not go into the architecture of what
you're trying to accomplish too much.

Some questions.

1. How does the queue.sh script get deployed to its final location?
2. How does it get executed the first time around?

For the init system to pick up a new service, you'll need to register
it. After dropping a file in /etc/init, you need to run initctl
reload-configuration for Upstart to pick it up. A sequence like that is
usually easy to do in Puppet.

If you make your service resource require both the new file in /etc/init
and an exec-resource that kicks the initctl command, Puppet should then
have no problems managing your new service through upstart because the
environment will be properly prepared for it.

Best regards,
Bas

[Martijn]

Hi Bas,

1. The queue.sh script is deployed via Capistrano, as part of the source code of each website, just like many websites have a cron.sh script.

2. I'm not entirely sure what you mean. The first time the worker is executed is when it's started by the init-script. There's an AWS SQS queue into which jobs are posted by various web applications. The queue.sh script sets a umask and starts a php command that subscribes to the queue for jobs that it can handle. If it does, it processes the job using code from the website's framework, writes the result, acknowledges success to the queue and goes back to polling. The php command should keep run like a daemon, one for each website, restarting if it dies and starting and stopping with the system. I'm not sure yet how to deal with restarting it when the website code is updated though.

Good point about reloading Upstart. Could I put those Service, Exec and File resources all into the same Define, using the define's title to make each resource unique?

I do feel my design could be simplified but am not seeing how.

Thanks, Martijn

[Garrett Honeycutt]

Hi Martin,

Suggest that you package your software. This way the init script lives
next to the other shell scripts and software necessary and not separated
between repos. You can setup a pipeline tool like Jenkins to monitor
your repo and then create and deploy packages automatically with FPM[1],
which makes creating packages really simple. This would simplify your
code quite a bit such that your queueworker define is just managing a
package and a service.

[1] - https://github.com/jordansissel/fpm/

Best regards,
-g

--
Garrett Honeycutt
@learnpuppet
Puppet Training with LearnPuppet.com
Mobile: +1.206.414.8658

[Bas van de Wiel]

Hi Martijn,

Something like this might do it.

define site::queueworker {

file { 'worker-config':
..do your /etc/init/configfile here
notify => Exec['kick-upstart'],
}

exec { 'kick-upstart':
...do your upstart-reload here
require => File['worker-config'],
refreshonly => true,
before => Service['worker'],
notify => Service['worker'],
}

service { 'worker':
..do your service here
require => File['worker-config']
}

}

[Christopher Wood]

Lots of different ways, it might make for lighter defines if you used some tags and chaining. Or even use the puppet4 iteration, for some reason defines puzzle a number of folks at work here.

class upstart {
$upstartreload = '/sbin/initctl reload-configuration'
exec { $upstartreload:
refreshonly => true,
}
File <| tag == 'upstart' |> ~> Exec[$upstartreload]
Exec[$upstartreload] -> Service <| tag == 'upstart' |>
}

class queues (
Hash $workers,
)
{
$workers.each |$name, $details|
$initfile = "/etc/init/queue-${name}.conf"
$package = "queue-${name}-files"
file { $initfile:
ensure => $details['file_ensure'],
content => template('queues/init.conf.erb'),
tag => 'upstart',
}
package { $package:
ensure => $details['package_ensure'],
}
service { "queue-${name}":
ensure => $details['service_ensure'],
subscribe => [File[$initfile], Package[$package]],
tag => 'upstart',
}
}
}

If you had non-template'able requirements for the upstart config you'd probably tag the package.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4fdd4175-3425-0b0f-479c-83cf3f98c883%40area536.com.
> For more options, visit https://groups.google.com/d/optout.