Clustered Pi HA openHAB setup

[Jonathan Baginski]

Hello all, I am new to the openHAB project/scene. I've been working with openHAB and Insteon devices and now have a decent working setup. One thing I want to do is make my setup more robust. I built a setup to make openHAB highly available on a 3-node Raspberry Pi cluster. The project is still very much a work in progress. I plan to add cloud based features as well as some other integrations. 

I have made it far enough with this project to share with anyone interested. The cluster is functional, and services failover upon hardware failure. It does take a few minutes for everything to fully come up, but I suspect once I do some tuning, this can be reduced drastically. 

I wanted to share this now in hopes of getting some feedback and thoughts from others who may be interested in building something similar. I plan to run this with 3 office UPS's on 2 networks (wired and wireless). 

Given the low power consumption, I think the battery backup will work well. I may even mate a 4G LTE backup connection into the mix for out of band control. 

Currently, I am working to build out a bonded nic pair containing the ethernet and wlan ports of each pi. This will be accomplished via "ifenslave". 

Below is the "build sheet" i've been updating as I go along. I've also attached it as a txt file incase it doesn't render correctly. In addition to sharing here, I'm also going to post updates on my progress over on my site (www.patchedsoul.com) if you want the most current progress. 

I welcome thoughts, input, and idea's :)

- Jonathan

openHAB Clustered Pi Build REV. 1.5

@@@@@@@@@@@@@@@@@@@@@@@@@@

Do this on all nodes          

@@@@@@@@@@@@@@@@@@@@@@@@@@

################################################

# Configure your wifi adapter on the mgmt node #

################################################

wlan0 iface wlan0 inet static
address x.x.x.x
metmask xxx.xxx.xxx.x
gateway 172.16.x.xxx
wpa-ssid "my wireless ssid"
wpa-psk "my wireless password"

################################################

# Configure static ip’s for each node in cluster    

################################################

sudo nano /etc/network/interfaces


iface lo inet loopback
iface eth0 inet static




address xxx.xxx.xxx.xxx


netmask xxx.xxx.xxx.xxx


network xxx.xxx.xxx.xxx


broadcast xxx.xxx.xxx.xxx


gateway xxx.xxx.xxx.xxx

## now add DNS servers ##

sudo nano /etc/resolv.conf


nameserver 8.8.8.8
nameserver 8.8.4.4

## Restart networking to reflect changes ##

sudo ./etc/init.d/networking restart

###################################

#Change hostnames for each node   #

###################################

## edit the file /etc/hostname and change the name from the default (raspberrypi)to “clusnodeX” ##

sudo nano /etc/hostname

 

############################################

edit the file /etc/hosts and replace the original name 

raspberrypi with the new name hostname                 

and also the ip’s of other cluster nodes:              

###########################################

sudo nano /etc/hosts


172.16.x.xxx clusnode1
172.16.x.xxx clusnode2
172.16.x.xxx clusnode3

#######################

 Config SSH keys     

#######################

## Create dirs on each node: ##

mkdir ~/.ssh

## Run on clusmgmt node: ##

ssh-keygen

## Copy the files to each node now: ##

scp -p /home/pi/.ssh/id_rsa.pub clusnode1:/home/pi/.ssh/authorized_keys
scp -p /home/pi/.ssh/id_rsa.pub clusnode2:/home/pi/.ssh/authorized_keys
scp -p /home/pi/.ssh/id_rsa.pub clusnode3:/home/pi/.ssh/authorized_keys

###################################################

install Heartbeat and Pacemaker 

(below will install both Heartbeat and Pacemaker 

with all dependancies)     

###################################################

## makes sure your system is up to date ##

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install heartbeat gawk-doc lm-sensors snmp-mibs-downloader

##################################################

#  Then we need to create the HA config files                                                                 

##################################################

sudo nano /etc/ha.d/ha.cf






debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
udpport 694
bcast eth0
node clusnode1 clusnode2 clusnode3
ping 172.16.x.xxx
crm respawn

## Save the HA auth keys ##

sudo nano /etc/ha.d/authkeys


auth 1
1 crc

## Change the file permission for the authkeys file ##

sudo chmod 600 /etc/ha.d/authkeys

## Restart cluster services on each node ##

sudo service heartbeat restart

## Test changes ##

sudo crm status

###############################################

Configure GlusterFS shared storage for openHAB

###############################################

## Install GlusterFS server ##

sudo apt-get install glusterfs-server

## Create GlusterFS directory/brick location (DO NOT WORK INSIDE THIS DIR DIRECTLY or you'll corrupt your Gluster volume !!!)

sudo mkdir /srv/glusexport1

## probe for each gluster peer. Establish's nodes in gluster cluster. 

## Only needs to be run from first node, but probe all nodes in cluster)

sudo gluster peer probe 172.16.x.xxx

## Create Gluster policy. define shared storage strategy. 

## Include the primary node or one this is being run from.

sudo gluster volume create glusexport1 replica 3 172.16.x.xxx:/srv/glusexport1 172.16.x.xx1:/srv/glusexport1 172.16.x.xx2:/srv/glusexport1

## Now that vol is created, let's start it ##

sudo gluster volume start glusexport1

## Verify it was created correctly ##

sudo gluster volume info

## Create local mounts for gluster drive ##

sudo mkdir -p /opt/openhab
sudo mount -t glusterfs 172.16.x.xxx:glusexport1 /opt/openhab

## Make mount permanent by editing /etc/fstab ##

sudo nano /etc/fstab
172.16.x.xxx:glusexport1  /opt/openhab  glusterfs  defaults,_netdev  0  0

######################################################

 install openHAB and a few other things                                                                       

######################################################

## check java version ##

java -version

## Download and unzip openHAB distro to newly created directory ##

wget openhab-runtime-<version>.zip


sudo unzip openhab-runtime-<version>.zip -d /opt/openhab

## Copy the default config template to useable starting config/state ##

sudo cp /opt/openhab/configurations/openhab_default.cfg /opt/openhab/configurations/openhab.cfg

## set openhab to be owned by root ##

sudo chown -hR root:root /opt/openhab/

## Create LSB/Init.d script to be handled by Pacemaker/Heartbeat CRM ##

sudo nano /etc/init.d/openhab


#! /bin/sh
    ### BEGIN INIT INFO
    # Provides:          openhab
    # Required-Start:    $remote_fs $syslog
    # Required-Stop:     $remote_fs $syslog
    # Default-Start:     2 3 4 5
    # Default-Stop:      0 1 6
    # Short-Description: OpenHAB Daemon
    ### END INIT INFO


    # Author: Thomas Brettinger


    # Do NOT "set -e"


    # PATH should only include /usr/* if it runs after the mountnfs.sh script
    PATH=/sbin:/usr/sbin:/bin:/usr/bin


    DESC="Open Home Automation Bus Daemon"
    NAME=openhab
    DAEMON=/usr/bin/java
    PIDFILE=/var/run/$NAME.pid
    SCRIPTNAME=/etc/init.d/$NAME
    ECLIPSEHOME="/opt/openhab";
    HTTPPORT=8080
    HTTPSPORT=8443
    TELNETPORT=5555
    # be sure you are adopting the user to your local OH user
    RUN_AS=root


    # get path to equinox jar inside $eclipsehome folder
    cp=$(find $ECLIPSEHOME/server -name "org.eclipse.equinox.launcher_*.jar" | sort | tail -1);


    DAEMON_ARGS="-Dosgi.clean=true -Declipse.ignoreApp=true -Dosgi.noShutdown=true -Djetty.port=$HTTPPORT -Djetty.port.ssl=$HTTPSPORT -Djetty.home=$ECLIPSEHOME -Dlogback.configurationFile=$ECLIPSEHOME/configurations/logback.xml -Dfelix.fileinstall.dir=$ECLIPSEHOME/addons -Djava.library.path=$ECLIPSEHOME/lib -Djava.security.auth.login.config=$ECLIPSEHOME/etc/login.conf -Dorg.quartz.properties=$ECLIPSEHOME/etc/quartz.properties -Djava.awt.headless=true -jar $cp -console ${TELNETPORT}"


    # Exit if the package is not installed
    [ -x "$DAEMON" ] || exit 0


    # Read configuration variable file if it is present
    [ -r /etc/default/$NAME ] && . /etc/default/$NAME


    # Load the VERBOSE setting and other rcS variables
    . /lib/init/vars.sh


    # Define LSB log_* functions.
    # Depend on lsb-base (>= 3.2-14) to ensure that this file is present
    # and status_of_proc is working.
    . /lib/lsb/init-functions


    #
    # Function that starts the daemon/service
    #
    do_start()
    {
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        start-stop-daemon --start --quiet --make-pidfile --pidfile $PIDFILE --chuid $RUN_AS --chdir $ECLIPSEHOME --exec $DAEMON --test > /dev/null \
            || return 1
        start-stop-daemon --start --quiet --background --make-pidfile --pidfile $PIDFILE --chuid $RUN_AS --chdir $ECLIPSEHOME --exec $DAEMON -- $DAEMON_ARGS \
            || return 2
        # Add code here, if necessary, that waits for the process to be ready
        # to handle requests from services started subsequently which depend
        # on this one.  As a last resort, sleep for some time.
        return 0
    }


    #
    # Function that stops the daemon/service
    #
    do_stop()
    {
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        [ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
        return "$RETVAL"
    }


    #
    # Function that sends a SIGHUP to the daemon/service
    #
    do_reload() {
        #
        # If the daemon can reload its configuration without
        # restarting (for example, when it is sent a SIGHUP),
        # then implement that here.
        #
        do_stop
        sleep 1
        do_start
        return 0
    }


    case "$1" in
      start)
        log_daemon_msg "Starting $DESC"
        do_start
        case "$?" in
            0|1) log_end_msg 0 ;;
            2) log_end_msg 1 ;;
        esac
        ;;
      stop)
        log_daemon_msg "Stopping $DESC"
        do_stop
        case "$?" in
            0|1) log_end_msg 0 ;;
            2) log_end_msg 1 ;;
        esac
        ;;
      status)
           status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
           ;;
      #reload|force-reload)
        #
        # If do_reload() is not implemented then leave this commented out
        # and leave 'force-reload' as an alias for 'restart'.
        #
        #log_daemon_msg "Reloading $DESC" "$NAME"
        #do_reload
        #log_end_msg $?
        #;;
      restart|force-reload)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC"
        do_stop
        case "$?" in
          0|1)
            do_start
            case "$?" in
                0) log_end_msg 0 ;;
                1) log_end_msg 1 ;; # Old process is still running
                *) log_end_msg 1 ;; # Failed to start
            esac
            ;;
          *)
              # Failed to stop
            log_end_msg 1
            ;;
        esac
        ;;
      *)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
        exit 3
        ;;
    esac
    :

    

 

## Make init.d script executable ##

 

sudo chmod a+x /etc/init.d/openhab  

    

## Test starting and stopping openhab before pushing config into Cluster Resource Manager##

sudo /etc/init.d/openhab start
sudo /etc/init.d/openhab stop

## !!! DO NOT CONFIGURE openHAB TO START AT BOOT, CLUSTER SOFTWARE WILL HANDLE THIS !!! ##

#####################################

Configure cluster parameters

#####################################

sudo crm configure property stonith-enabled=false
sudo crm configure property expected-quorum-votes="2"
sudo crm configure property no-quorum-policy=ignore
sudo crm configure rsc_defaults resource-stickiness=100
sudo crm_attribute --type rsc_defaults --attr-name migration-threshold --attr-value 2
sudo crm_attribute --type rsc_defaults --attr-name failure-timeout --attr-value 30s
sudo crm configure primitive clusterIP1 ocf:IPaddr2 params ip=172.16.x.xxx cidr_netmask=24 nic=eth0 op monitor interval=30s
sudo crm configure primitive openhab lsb:openhab op monitor interval=20s

## Test changes ##

sudo crm status

######################################################

Create cluster Group for resources to ensure they migrate together

######################################################

sudo crm configure group HA-GROUP clusterIP1 openhab

## Show cluster status (will reflect new group after few mins) ##

root@clusnode1:/etc/ha.d/resource.d# crm_mon -1
============
Last updated: Tue Jul  7 10:21:29 2015
Last change: Tue Jul  7 10:21:14 2015 via cibadmin on clusnode1
Stack: Heartbeat
Current DC: clusnode1 (d45fed8b-8e47-4cdf-a580-3588c68b05bc) - partition with quorum
Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff
3 Nodes configured, 2 expected votes
2 Resources configured.
============


Online: [ clusnode2 clusnode3 clusnode1 ]


 Resource Group: HA-Group
     clusterIP1 (ocf::heartbeat:IPaddr2): Started clusnode1
     openhab (lsb:openhab): Started clusnode2
root@clusnode1:/etc/ha.d/resource.d# crm_mon -1
============
Last updated: Tue Jul  7 10:21:54 2015
Last change: Tue Jul  7 10:21:14 2015 via cibadmin on clusnode1
Stack: Heartbeat
Current DC: clusnode1 (d45fed8b-8e47-4cdf-a580-3588c68b05bc) - partition with quorum
Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff
3 Nodes configured, 2 expected votes
2 Resources configured.
============


Online: [ clusnode2 clusnode3 clusnode1 ]


 Resource Group: HA-Group
     clusterIP1 (ocf::heartbeat:IPaddr2): Started clusnode1
     openhab (lsb:openhab): Started clusnode1
root@clusnode1:/etc/ha.d/resource.d#

## That's about it 

## test failover and such. When you pull the plug on a node, 

## it takes a few mins for the services 

## to failover and come up fully on the next node

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Helpful Stuff

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

########################################################################################

  On the head node, create script in /usr/local/bin/worker.sh Use this to run commands on all nodes.         

########################################################################################

sudo nano /usr/local/bin/worker.sh


 !/bin/sh HOSTS="clusnode1 clusnode2 clusnode3 clusnode4" for HOSTNAME in $HOSTS; do echo executing command on

 

## set the script to be executable using sudo chmod a+x /usr/local/bin/worker.sh ##

sudo chmod a+x /usr/local/bin/worker.sh

## example to shutdown all workers run the following command ##

worker.sh sudo shutdown -hP now

[Christoph Wempe]

Wow!

This looks like a really great project!

But I am wondering if there is a reason you install openhab via wget and unzip instewad of using the apt-get repository described in the wiki.

[Jonathan Baginski]

Thanks! I originally started using the apt method of installation. It proved cumbersome and painful to try and symlink everything properly across all 3 nodes to the Gluster mounts. 

Using the zip method, everything is self-contained within /opt/openhab.

[Jason Brunk]

How does your update process look?

I have considered doing something similar using HAProxy 

The only issue I have come up with is I have a zwave stick on my raspberry pi and considered using something like this

http://vmwarebits.com/content/virtualhere-raspberry-pi-usb-server

and put my zwave on that, and then run a raspberry pi and a vm with my clustered openhab.  

I definitely like this and would be happy to helpout where I can.

...

[Jason Brunk]

PS.  what is that bottom board on your pi stack there?

On Sunday, July 12, 2015 at 12:12:36 AM UTC-4, Jonathan Baginski wrote:

...

[Jonathan Baginski]

Updating is stupid simple since it's just dropping in the new runtime. I sort of like the zip approach for this reason. I will most likely script a backup and update routine in the near future though for runtime, bindings, and such. I am going to be incorporating zwave into the mix as well. Between openhab/insteon/zwave, theres almost nothing on the market you can't integrate with :) 

I've noticed the z-wave/pi threads but haven't stopped to read them or see whats up yet. Figured it was good to get Insteon running correctly first, then add another protocol and layer of complexity. As to the bottom board, thats a Tenda TE-100 5 port switch that just happens to be almost the same size of the pi's. I printed the cluster parts on my 3D printer from here: http://www.thingiverse.com/thing:141953 and just made some modifications for the switch board. It may be a cluster interconnect or mgmt network going forward. Once I have time to sort out the NIC bonding, it might become useless. 

On Monday, July 13, 2015 at 9:54:56 AM UTC-4, Jason Brunk wrote:

PS.  what is that bottom board on your pi stack there?

  ... 

[Jason Brunk]

once i move my zwave stick off to a network usb i will have to try this.  I like the idea of some fault tolerance in my home automation system. 

--
You received this message because you are subscribed to a topic in the Google Groups "openhab" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openhab/39L6C4Du7OM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openhab+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.
Visit this group at http://groups.google.com/group/openhab.
To view this discussion on the web visit https://groups.google.com/d/msgid/openhab/d855055e-4627-49ab-bf65-9f9136d7fe73%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joe Barneson]

Great project -- definitely will follow your progress if you update this thread.

Is it imperative you run as Root?