logstash module to control service

281 views

Skip to first unread message

Tim Dunphy

unread,

Jan 11, 2014, 12:56:29 PM1/11/14

to puppet...@googlegroups.com

Hey all,

I've created a number of puppet modules that control services in the form of init scripts on the hosts it controls.

However I recently created a logstash init script and a puppet module to control it. It also should push out a config file.

And the logstash module cannot control the logstash init script, even tho the module was created in the same way as the other modules I've done. So my thinking is that there may be a problem with the init script.

Here's my class definition:

class logstash {

service { logstash:

ensure => running,

}

file { "/usr/local/logstash/indexer.conf":

notify => Service["logstash"],

owner => "root",

group => "root",

mode => 0440,

#require => File["/usr/local/logstash/logstash-1.3.2-flatjar.jar"],

source => "puppet:///modules/logstash/indexer.conf",

}

And my node definition looks like this:

node 'logs.mydomain.com' {

include sudo, logstash, puppet, apache, bacula-client

}

And this is the init script that puppet can't seem to control. It might be worth knowing that on it's own, the init script I'm about to show you does successfully stop start and restart (or refresh if you prefer) the logstash application.

#! /bin/bash

# /etc/rc.d/init.d/logstash

# Starts Logstash as a daemon

# chkconfig: 2345 20 80

# description: Starts Logstash as a daemon

# pidfile: /var/run/logstash-agent.pid

### BEGIN INIT INFO

# Provides: logstash

# Required-Start: $local_fs $remote_fs

# Required-Stop: $local_fs $remote_fs

# Default-Start: 2 3 4 5

# Default-Stop: S 0 1 6

# Short-Description: Logstash

# Description: Starts Logstash as a daemon.

# Author: christia...@sbri.org, modified by https://github.com/paul-at

### END INIT INFO

# Amount of memory for Java

JAVAMEM=256M

# Location of logstash files

LOCATION='/usr/local/logstash'

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

DESC="Logstash Daemon"

NAME=java

DAEMON=$(which java)

CONFIG_DIR='/etc/logstash/logstash.conf'

LOGFILE="/var/log/logstash/logstash-indexer.log"

PATTERNSPATH="/usr/local/logstash/patterns"

JARNAME='logstash.jar'

ARGS="-Xmx$JAVAMEM -Xms$JAVAMEM -jar ${JARNAME} agent --config ${CONFIG_DIR} --log ${LOGFILE}"

SCRIPTNAME=/etc/init.d/logstash

base=logstash

# Exit if the package is not installed

if [ ! -x "$DAEMON" ]; then

{

echo "Couldn't find $DAEMON"

exit 99

}

. /etc/init.d/functions

# Function that starts the daemon/service

do_start()

{

cd $LOCATION && \

($DAEMON $ARGS &) \

&& success || failure

}

# Function that stops the daemon/service

do_stop()

{

pid=$(ps auxwww | grep -e logstash -e indexer | grep -i -v -e grep -e screen -e vim -e forwarder -e start)

if checkpid $pid 2>&1; then

# TERM first, then KILL if not dead

kill -TERM $pid >/dev/null 2>&1

usleep 100000

if checkpid $pid && sleep 1 &&

checkpid $pid && sleep $delay &&

checkpid $pid ; then

kill -KILL $pid >/dev/null 2>&1

usleep 100000

checkpid $pid

RC=$?

[ "$RC" -eq 0 ] && failure $"$base shutdown" || success $"$base shutdown"

}

case "$1" in

start)

echo -n "Starting $DESC: "

do_start

touch /var/lock/subsys/$JARNAME

;;

stop)

echo -n "Stopping $DESC: "

do_stop

rm /var/lock/subsys/$JARNAME

;;

restart|reload)

echo -n "Restarting $DESC: "

do_stop

do_start

;;

status)

status -p $PID

;;

It's just that if I stop the service and do a manual puppet run, everything goes fine, but nothing at all happens with the logstash service.

Allow me to demonstrate:

Step 1, stop services and remove files that puppet is meant to control:

root@logs:~] #service httpd stop
Stopping httpd: [ OK ]
[root@logs:~] #service bacula-fd stop
Shutting down bacula-fd: [ OK ]
[root@logs:~] #service logstash stop
Stopping Logstash Daemon: Terminated
[root@logs:~] #rm /etc/sudoers

The manual puppet run:

[root@logs:~] #puppet agent --test --server puppet.mydomain.com
Info: Retrieving plugin
Info: Caching catalog for logs.jokefire.com
Info: Applying configuration version '1389442074'
Notice: /Stage[main]/Apache/Service[httpd]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Apache/Service[httpd]: Unscheduling refresh on Service[httpd]
Notice: /Stage[main]/Sudo/File[/etc/sudoers]/ensure: defined content as '{md5}4682d5ac1b693284dd615a4366a2e8ce'
Notice: /Stage[main]/Bacula-client/Service[bacula-fd]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Bacula-client/Service[bacula-fd]: Unscheduling refresh on Service[bacula-fd]
Notice: Finished catalog run in 1.18 seconds

And a grep for logstash (filtering out non-relevant items) turns up nothing

[root@logs:~] #ps -ef | grep logstash | grep -i -v -e grep -e screen -e forwarder

But puppet returns the file that I removed:

-r--r----- 1 root root 3469 Jan 11 07:50 /etc/sudoers

Starting the daemon usign the init script I show above:

[root@logs:~] #service logstash start
Starting Logstash Daemon: [ OK ]

And doing the same grep as before now shows a happy, running logstash:

[root@logs:~] #ps -ef | grep logstash | grep -i -v -e grep -e screen -e forwarder
root 19915 1 91 07:54 pts/0 00:00:26 /usr/bin/java -Xmx256M -Xms256M -jar logstash.jar agent --config /etc/logstash/logstash.conf --log /var/log/logstash/logstash-indexer.log

So what am I missing here? Any clues or ideas? I'd really appreciate the list's input on this one.

Thanks!
Tim

--
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

jcbollinger

unread,

Jan 13, 2014, 2:42:46 PM1/13/14

to puppet...@googlegroups.com

On Saturday, January 11, 2014 6:56:29 AM UTC-6, bluethundr wrote:

Hey all,

I've created a number of puppet modules that control services in the form of init scripts on the hosts it controls.

However I recently created a logstash init script and a puppet module to control it. It also should push out a config file.

And the logstash module cannot control the logstash init script, even tho the module was created in the same way as the other modules I've done. So my thinking is that there may be a problem with the init script.

That is usually where the issue resides in cases like this, but I don't immediately see anything wrong with the initscript you presented.

Allow me to demonstrate:

Step 1, stop services and remove files that puppet is meant to control:

root@logs:~] #service httpd stop
Stopping httpd: [ OK ]
[root@logs:~] #service bacula-fd stop
Shutting down bacula-fd: [ OK ]
[root@logs:~] #service logstash stop
Stopping Logstash Daemon: Terminated

It's a little suspicious that shutting down the Logstash daemon did not cause the standard "[ OK ]" response to be emitted. It looked like the script was using the standard daemon-control tools, but they should have emitted the standard response.

[root@logs:~] #rm /etc/sudoers

The manual puppet run:

[root@logs:~] #puppet agent --test --server puppet.mydomain.com

That would be a lot more informative if you also specified the --debug option to obtain additional logging.

Info: Retrieving plugin
Info: Caching catalog for logs.jokefire.com
Info: Applying configuration version '1389442074'
Notice: /Stage[main]/Apache/Service[httpd]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Apache/Service[httpd]: Unscheduling refresh on Service[httpd]
Notice: /Stage[main]/Sudo/File[/etc/sudoers]/ensure: defined content as '{md5}4682d5ac1b693284dd615a4366a2e8ce'
Notice: /Stage[main]/Bacula-client/Service[bacula-fd]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Bacula-client/Service[bacula-fd]: Unscheduling refresh on Service[bacula-fd]
Notice: Finished catalog run in 1.18 seconds

Because Service[logstash] does not appear in Puppet's output, either the catalog you are applying does not specify it, or Puppet thinks it is already in the target state. (Debug logging should differentiate between those alternatives.) If the latter, then there is probably an issue with the script's 'status' command. For best results, the script should comply with LSB: http://refspecs.linuxbase.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-generic/iniscrptact.html .

John

Jose Luis Ledesma

unread,

Jan 13, 2014, 4:57:37 PM1/13/14

to puppet...@googlegroups.com

It seems that the script when stops kills also himself

Tim Dunphy

unread,

Jan 14, 2014, 8:25:14 AM1/14/14

to puppet...@googlegroups.com

It seems that the script when stops kills also himself

Thanks, I'll see if I can correct that.

But in the meantime here's the debug output of a puppet run on the logs host in which the logstash service does not run:

[0;36mDebug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist [0m

[0;36mDebug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dsimport does not exist [0m

[0;36mDebug: Puppet::Type::User::ProviderPw: file pw does not exist [0m

[0;36mDebug: Puppet::Type::User::ProviderLdap: true value when expecting false [0m

[0;36mDebug: Using settings: adding file resource 'requestdir': 'File[/etc/puppet/ssl/certificate_requests]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :path=>"/etc/puppet/ssl/certificate_requests", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'client_datadir': 'File[/var/lib/puppet/client_data]{:loglevel=>:debug, :links=>:follow, :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/client_data", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'lastrunfile': 'File[/var/lib/puppet/state/last_run_summary.yaml]{:loglevel=>:debug, :links=>:follow, :backup=>false, :mode=>"644", :path=>"/var/lib/puppet/state/last_run_summary.yaml", :ensure=>:file}' [0m

[0;36mDebug: Using settings: adding file resource 'confdir': 'File[/etc/puppet]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/etc/puppet", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'privatedir': 'File[/etc/puppet/ssl/private]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :mode=>"750", :path=>"/etc/puppet/ssl/private", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'resourcefile': 'File[/var/lib/puppet/state/resources.txt]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"root", :mode=>"640", :path=>"/var/lib/puppet/state/resources.txt", :ensure=>:file}' [0m

[0;36mDebug: Using settings: adding file resource 'hostcrl': 'File[/etc/puppet/ssl/crl.pem]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :mode=>"644", :path=>"/etc/puppet/ssl/crl.pem", :ensure=>:file}' [0m

[0;36mDebug: Using settings: adding file resource 'certdir': 'File[/etc/puppet/ssl/certs]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :path=>"/etc/puppet/ssl/certs", :ensure=>:directory}' [0m

[0;36mDebug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl does not exist [0m

[0;36mDebug: Puppet::Type::Group::ProviderPw: file pw does not exist [0m

[0;36mDebug: Puppet::Type::Group::ProviderLdap: true value when expecting false [0m

[0;36mDebug: Using settings: adding file resource 'rundir': 'File[/var/run/puppet]{:loglevel=>:debug, :group=>"puppet", :links=>:follow, :backup=>false, :owner=>"puppet", :mode=>"755", :path=>"/var/run/puppet", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'hostprivkey': 'File[/etc/puppet/ssl/private_keys/logs.mydomain.com.pem]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :mode=>"600", :path=>"/etc/puppet/ssl/private_keys/logs.mydomain.com.pem", :ensure=>:file}' [0m

[0;36mDebug: Using settings: adding file resource 'publickeydir': 'File[/etc/puppet/ssl/public_keys]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :path=>"/etc/puppet/ssl/public_keys", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'clientyamldir': 'File[/var/lib/puppet/client_yaml]{:loglevel=>:debug, :links=>:follow, :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/client_yaml", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'statedir': 'File[/var/lib/puppet/state]{:loglevel=>:debug, :links=>:follow, :backup=>false, :mode=>"1755", :path=>"/var/lib/puppet/state", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'logdir': 'File[/var/log/puppet]{:loglevel=>:debug, :group=>"puppet", :links=>:follow, :backup=>false, :owner=>"puppet", :mode=>"750", :path=>"/var/log/puppet", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'localcacert': 'File[/etc/puppet/ssl/certs/ca.pem]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :mode=>"644", :path=>"/etc/puppet/ssl/certs/ca.pem", :ensure=>:file}' [0m

[0;36mDebug: Using settings: adding file resource 'privatekeydir': 'File[/etc/puppet/ssl/private_keys]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :mode=>"750", :path=>"/etc/puppet/ssl/private_keys", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'libdir': 'File[/var/lib/puppet/lib]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'classfile': 'File[/var/lib/puppet/classes.txt]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"root", :mode=>"640", :path=>"/var/lib/puppet/classes.txt", :ensure=>:file}' [0m

[0;36mDebug: Using settings: adding file resource 'lastrunreport': 'File[/var/lib/puppet/state/last_run_report.yaml]{:loglevel=>:debug, :links=>:follow, :backup=>false, :mode=>"640", :path=>"/var/lib/puppet/state/last_run_report.yaml", :ensure=>:file}' [0m

[0;36mDebug: Using settings: adding file resource 'clientbucketdir': 'File[/var/lib/puppet/clientbucket]{:loglevel=>:debug, :links=>:follow, :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/clientbucket", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'vardir': 'File[/var/lib/puppet]{:loglevel=>:debug, :group=>"puppet", :links=>:follow, :backup=>false, :owner=>"puppet", :path=>"/var/lib/puppet", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'pluginfactdest': 'File[/var/lib/puppet/facts.d]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/facts.d", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'graphdir': 'File[/var/lib/puppet/state/graphs]{:loglevel=>:debug, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/state/graphs", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'hostcert': 'File[/etc/puppet/ssl/certs/logs.mydomain.com.pem]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :mode=>"644", :path=>"/etc/puppet/ssl/certs/logs.mydomain.com.pem", :ensure=>:file}' [0m

[0;36mDebug: Using settings: adding file resource 'ssldir': 'File[/etc/puppet/ssl]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :mode=>"771", :path=>"/etc/puppet/ssl", :ensure=>:directory}' [0m

[0;36mDebug: Using settings: adding file resource 'statefile': 'File[/var/lib/puppet/state/state.yaml]{:loglevel=>:debug, :links=>:follow, :backup=>false, :mode=>"660", :path=>"/var/lib/puppet/state/state.yaml", :ensure=>:file}' [0m

[0;36mDebug: Using settings: adding file resource 'hostpubkey': 'File[/etc/puppet/ssl/public_keys/logs.mydomain.com.pem]{:loglevel=>:debug, :links=>:follow, :backup=>false, :owner=>"puppet", :mode=>"644", :path=>"/etc/puppet/ssl/public_keys/logs.mydomain.com.pem", :ensure=>:file}' [0m

[0;36mDebug: /File[/etc/puppet/ssl/public_keys/logs.mydomain.com.pem]: Autorequiring File[/etc/puppet/ssl/public_keys] [0m

[0;36mDebug: /File[/etc/puppet/ssl/certs/logs.mydomain.com.pem]: Autorequiring File[/etc/puppet/ssl/certs] [0m

[0;36mDebug: /File[/var/lib/puppet/state/last_run_summary.yaml]: Autorequiring File[/var/lib/puppet/state] [0m

[0;36mDebug: /File[/var/lib/puppet/state/last_run_report.yaml]: Autorequiring File[/var/lib/puppet/state] [0m

[0;36mDebug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet] [0m

[0;36mDebug: /File[/etc/puppet/ssl/public_keys]: Autorequiring File[/etc/puppet/ssl] [0m

[0;36mDebug: /File[/etc/puppet/ssl/certificate_requests]: Autorequiring File[/etc/puppet/ssl] [0m

[0;36mDebug: /File[/var/lib/puppet/state/state.yaml]: Autorequiring File[/var/lib/puppet/state] [0m

[0;36mDebug: /File[/etc/puppet/ssl/private_keys]: Autorequiring File[/etc/puppet/ssl] [0m

[0;36mDebug: /File[/etc/puppet/ssl/certs/ca.pem]: Autorequiring File[/etc/puppet/ssl/certs] [0m

[0;36mDebug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet] [0m

[0;36mDebug: /File[/var/lib/puppet/state/resources.txt]: Autorequiring File[/var/lib/puppet/state] [0m

[0;36mDebug: /File[/var/lib/puppet/state/graphs]: Autorequiring File[/var/lib/puppet/state] [0m

[0;36mDebug: /File[/etc/puppet/ssl/certs]: Autorequiring File[/etc/puppet/ssl] [0m

[0;36mDebug: /File[/var/lib/puppet/classes.txt]: Autorequiring File[/var/lib/puppet] [0m

[0;36mDebug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet] [0m

[0;36mDebug: /File[/etc/puppet/ssl/crl.pem]: Autorequiring File[/etc/puppet/ssl] [0m

[0;36mDebug: /File[/etc/puppet/ssl/private]: Autorequiring File[/etc/puppet/ssl] [0m

[0;36mDebug: /File[/var/lib/puppet/facts.d]: Autorequiring File[/var/lib/puppet] [0m

[0;36mDebug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet] [0m

[0;36mDebug: /File[/etc/puppet/ssl/private_keys/logs.mydomain.com.pem]: Autorequiring File[/etc/puppet/ssl/private_keys] [0m

[0;36mDebug: /File[/var/lib/puppet/client_data]: Autorequiring File[/var/lib/puppet] [0m

[0;36mDebug: /File[/etc/puppet/ssl]: Autorequiring File[/etc/puppet] [0m

[0;36mDebug: Finishing transaction 69968132564280 [0m