Moloch upstart scripts for RHEL 6 / CentOS 6

[Eric G]

I've spent several hours learning more about RHEL's weird implementation of Upstart than I ever wanted to know (surprise! all the Ubuntu examples don't work!) but I think I've finally gotten together Upstart scripts that reliably start Moloch at boot, and automagically respawn the Moloch capture/viewer processes if they stop. They also use dependencies so that things start in the correct order

First off, I had to add 'sleep 15' to /data/moloch/bin/run_viewer.sh and /data/moloch/bin/run_capture.sh so that they would wait for elasticsearch to make itself available.

Here's my /data/moloch/bin/run_capture.sh:

#!/bin/sh

# Add to /etc/inittab something like 

# m1:2345:respawn:/data/moloch/bin/run_capture.sh

TDIR=/data/moloch

cd ${TDIR}/bin

/bin/rm -f ${TDIR}/capture.log.old

/bin/mv ${TDIR}/logs/capture.log ${TDIR}/logs/capture.log.old

# wait for elasticsearch to show up

sleep 15

${TDIR}/bin/moloch-capture -c ${TDIR}/etc/config.ini > ${TDIR}/logs/capture.log 2>&1

Here's my /data/moloch/bin/run_viewer.sh:

#!/bin/sh

# Add to /etc/inittab something like 

# v1:2345:respawn:/data/moloch/bin/run_viewer.sh

TDIR=/data/moloch

cd ${TDIR}/viewer

/bin/rm -f ${TDIR}/logs/viewer.log.old

/bin/mv ${TDIR}/logs/viewer.log ${TDIR}/logs/viewer.log.old

export NODE_ENV=production 

# wait for elasticsearch to show up

sleep 15

${TDIR}/bin/node viewer.js -c ${TDIR}/etc/config.ini > ${TDIR}/logs/viewer.log 2>&1

And here are my Upstart scripts:

/etc/init/moloch-elasticsearch.conf

escription "Upstart configuration for ElasticSearch"

 

# When to start the service

start on started network

 

# When to stop the service

stop on runlevel [016]

 

# Automatically restart the process if crashed

respawn

respawn limit 10 5

 

# Configuration

env ELASTICSEARCH_HOME=/data/moloch/elasticsearch-0.90.1

env MOLOCH_HOME=/data/moloch

env ES_HOSTNAME=moloch

env ES_HEAP_SIZE=20000M

 

console output

# bump up the number of files limit to a ridiculous number

limit nofile 65535 65535

 

script

  exec /data/moloch/elasticsearch-0.90.1/bin/elasticsearch -f -Des.config=/data/moloch/etc/elasticsearch.yml

end script

 

pre-stop script

  exec curl -XPOST http://localhost:9200/_shutdown

end script

/etc/init/moloch-capture.conf

description "Upstart configuration for Capture"

 

# When to start the service

start on (started network and started moloch-elasticsearch)

# When to stop the service

stop on stopping moloch-elasticsearch

 

# Automatically restart the process if crashed

respawn

respawn limit 10 5

script

  exec /data/moloch/bin/run_capture.sh

end script

/etc/init/moloch-viewer.conf

description "Upstart configuration for Viewer"

# When to start the service

start on started moloch-elasticsearch and started network

# When to stop the service

stop on stopping moloch-elasticsearch

# Automatically restart the process if crashed

respawn

respawn limit 10 5

script

  exec /data/moloch/bin/run_viewer.sh

end script

[JA e]

Eric,

I tried to apply these to my RHEL 6.4 instance after customizing them to my setup and it hung at boot. From all indications they seem like they should be ok. 

It wouldn't boot past bringing the eth interfaces up unless i removed moloch-*.conf from /etc/init.

Very curious...any thoughts?

[Eric G]

Hmm I specifically tested making sure the scripts work when I rebooted my VM that i have Moloch running on... Strange

--
Eric

--
You received this message because you are subscribed to the Google Groups "Moloch Full Packet Capture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moloch-fpc+...@googlegroups.com.
To post to this group, send email to moloc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[JA e]

I had a non-upstart command in the files that was causing the hang at boot, my mistake. 

Now i get moloch-elasticsearch.conf to start, but it's not tracking the proper PID. Still learning about the joys of upstart... :)

[Eric G]

On Sep 18, 2013 11:11 PM, "JA e" <jae.wi...@gmail.com> wrote:
>
> I had a non-upstart command in the files that was causing the hang at boot, my mistake. 
>
> Now i get moloch-elasticsearch.conf to start, but it's not tracking the proper PID. Still learning about the joys of upstart... :)

JA I can throw up a fresh VM today and run easybutton, apply my upstart scripts and make sure they work ok. Maybe there's something I did on my original test VM that I forgot to document

--
Eric
http://www.linkedin.com/in/ericgearhart

[JA e]

We ended up ditching upstart for a cron'd script that checks every minute that things are running and restarts everything properly should ES die. We couldn't get upstart to properly follow forks. Seeing as upstart won't be in RHEL7 and we don't much care for systemd, this seems like the right way for us. 

Thanks!

[Eric G]

On Friday, September 20, 2013 5:13:12 PM UTC-4, JA e wrote:

We ended up ditching upstart for a cron'd script that checks every minute that things are running and restarts everything properly should ES die. We couldn't get upstart to properly follow forks. Seeing as upstart won't be in RHEL7 and we don't much care for systemd, this seems like the right way for us. 

 

Now that you mention it I think I now remember modifying ES's startup to tell it not to fork, or something along those lines. Upstart could then follow it and restart it appropriately.

--

Eric

http://www.linkedin.com/in/ericgearhart

[Andy]

For ES they now support deb and rpm files.  Anyone try them?  Not sure how you tell it to use a different yml and memory config.