Notes for 2011-11-21: puppetlabs-f5, automating external devices, high availability, load balancing, proxies, backups, AWS S3 filesystems
Igal Koshevoy
Nan Liu talked about his recent work on managing F5 load balancers
using Puppet. This relies on the new version of Puppet which has a
native concept of external devices that are seen as nodes. You can
find further information and code at
<https://github.com/puppetlabs/puppetlabs-f5>. If you plan to manage
external devices -- e.g. Cisco switch, firewall, UPS -- then you
should definitely review this, and share your code as open source so
that others can benefit from it and help you improve it.
Igal Koshevoy talked about implementing similar solutions for
configuration systems without a native concept of external nodes
(Chef, older versions of Puppet, etc). You'd implement a custom
resource (e.g. Puppet type/provider, Chef LWRP, etc) and use it on a
node responsible for managing the external device, letting it make
remote calls to ensure the external device is configured and update it
if needed. You can define rules for the external device using
attributes, databags or other data in a database; or define them in
code within a manifest/recipe, e.g.:
# Describe an F5 load balancer rule called "redirect_to_https":
f5_rule 'redirect_to_https' do
# This rule applies to an F5 with the given hostname:
hostname 'myloadbalancer'
# If the F5 gets an 'http' protocol request...:
when_protocol 'http'
# ...then it should redirect to the 'https' protocol
equivalent of the URL:
then_redirect 'https://$1'
end
DISCUSSIONS
We talked about using AWS effectively, so I'd like to repeat my
regular pitch to also visit our friends at the Portland AWS User Group
-- they have a narrower scope than pdxdevops but more niche expertise
in AWS. Their next meeting is Wednesday, November 30, 2011 from 6–8pm:
http://calagator.org/events/1250461545
We had questions about dealing with slow and unpredictable EBS storage
on AWS. One seemingly crazy approach is to create a software RAID
using multiple EBS volumes
<http://www.mysqlperformanceblog.com/2009/08/06/ec2ebs-single-and-raid-volumes-io-bencmark/>.
Another less crazy-sounding approach is to fast local filesystem
storage (which goes away when the node shuts down) and replicate its
data to other nodes (e.g. DRBD filesystem or database log shipping),
and also keep a slave copy of the data on EBS just in case. Thus if
just the master fails, a slave can take over using its fast local
filesystem copy. If all nodes fail, a new cluster can be rebuilt by
loading the data from EBS into local filesystem storage. If EBS is
corrupted, you hopefully have backups.
We also had a bunch of discussions about various tools, so rather than
go through the discussions, here are the tools:
* Linux Virtual Server (LVS): Highly scalable and highly available
clustering load balancer for any protocol using Linux. It's very
powerful and sophisticated, e.g. used to run Wikipedia, but is
complicated and you're better off using a simpler failover and
HTTP-only load balancer if you can -- http://www.linux-vs.org/
* Keepalived: Sophisticated monitoring and failover for LVS clusters
-- http://www.keepalived.org/
* Heartbeat: Sophisticated cluster infrastructure for communications
and messaging -- http://linux-ha.org/wiki/Heartbeat
* Pacemaker: Sophisticated cluster resource manager for use with
Heartbeat -- http://linux-ha.org/wiki/Pacemaker
* UCARP: Easy to configure daemon allowing a couple of hosts to own a
virtual IP address in order to provide automatic failover --
https://github.com/jedisct1/UCarp/blob/master/README
* DRBD: Replicate a Linux mountable filesystem between hosts over the
network, e.g. master fileserver replicates its data to a slave, which
can then take over fileserving if the master fails --
http://www.drbd.org/
* HAProxy: HTTP reverse proxy and load-balancer that's fast and
featureful -- http://haproxy.1wt.eu/
* Nginx: HTTP server, load-balancer and reverse proxy that's very fast
and powers some huge sites -- http://nginx.org/
* Varnish: Reverse HTTP proxy and load balancer that's fast,
featureful and growing in popularity -- https://www.varnish-cache.org/
* Squid: Forward and reverse HTTP proxy that's been stable for over a
decade, but may be showing its age -- http://www.squid-cache.org/
* Apache Traffic Server: Forward and reverse HTTP proxy that handles
400TB of data a day at Yahoo! -- http://trafficserver.apache.org/ &&
http://twit.tv/show/floss-weekly/179
* rdiff-backup: Backup files to or from computers via ssh by mirroring
them and storing very efficient diffs of their changes --
http://www.nongnu.org/rdiff-backup/
* duplicity: Backup files to AWS S3, Rackspace, SSH, etc in a very
efficient way that only transfers changes --
http://duplicity.nongnu.org/
* s3cmd: Command-line tools for interacting with AWS S3 --
http://s3tools.org/s3cmd
* s3fs: Mount an AWS S3 bucket as a regular, albeit limited,
filesystem for easier manipulation -- http://code.google.com/p/s3fs/
* s3ql: Create a custom, full-featured filesystem on top of AWS S3,
OpenStack Storage, or Google Storage -- http://code.google.com/p/s3ql/
http://code.google.com/p/s3ql/
OFF-TOPIC STUFF FROM AFTER THE MEETING
Igal's photos from the recent Occupy events:
* http://www.flickr.com/photos/igalko/collections/72157628021032707/
Aircraft detection before RADAR was invented required people to
carefully listen for faint engine noises using a variety of
ridiculous-looking contraptions:
* http://aviationhumor.net/aircraft-detection-before-radar/
* http://blog.cwam.org/2011/03/before-radar.html
* http://post-concrete.com/dharmadhatu/?p=58
Svalbard is a remote Norwegian island which has a collection of plant
seeds as backups in case of global catastrophe, seed banks play a
major role in a particularly dystopian book, and has the "world’s
largest commercial ground station with more than 31 state-of-the-art
multi-mission and customer dedicated antenna systems in C-, L-, S- and
X-band":
* http://www.michaeljohngrist.com/2009/03/global-seed-vault-svalbard/
* http://www.amazon.com/Windup-Girl-Paolo-Bacigalupi/dp/1597801577
* http://www.ksat.no/Products/Svalsat.htm
* http://goo.gl/vaZZu
Signaling using microwave tropospheric scatter, with post-apocalyptic
looking photos of huge abandoned antenna:
* http://www.midlandsheritage.co.uk/military/4546-raf-stenigot-acehigh-chainhome-louth-lincs.html
* http://en.wikipedia.org/wiki/RAF_Stenigot
* http://en.wikipedia.org/wiki/Tropospheric_scatter
* http://en.wikipedia.org/wiki/Echelon_(signals_intelligence)
Micronations, quirky independence movements and related amusements
* http://en.wikipedia.org/wiki/Cascadia_(independence_movement)
* http://en.wikipedia.org/wiki/Principality_of_Sealand
* http://en.wikipedia.org/wiki/Republic_of_Rose_Island
* http://en.wikipedia.org/wiki/San_Serriffe
-igal