Re: bundles, hosts, monitoring
Igal Koshevoy
makes conversations easier to read and the online archives more useful. :)
Tim Uckun wrote:
> I have been fooling around a bit here is what I have done so far....
>
Thanks for sharing the details of your work. I deliberately made
AutomateIt flexible enough so that people could use it in ways that I
hadn't foreseen. However, some approaches are better than others. So
while I have certain ways that I recommend using AutomateIt, the person
using it is the domain expert for what's best for their particular needs.
> I have created a ruby project which is checked into version control.
> This project contains recipe bundles (a directory like puppet or chef)
> which contain the recipe and all the templates needed to run that
> recipe.
I'm surprised that you implemented these "bundles" and am curious why.
AutomateIt includes an easy way to create "projects", these are
directory structures that help you organize multiple recipes and their
related files:
http://automateit.org/documentation/classes/AutomateIt/Project.html
> It also contains a directory called "hosts" which contains
> host bundles (host recipe and all the files). I have built up a pretty
> simple framework which runs these bundles after determining the host
> name. I don't use the tag mechanism and instead the host recipe has a
> series of method calls like
>
> has_tag 'blah'
> has_tag 'something_else'
>
I'm surprised that you implemented this host-centric system and am
curious why. Doesn't this make it hard to add new servers, share code
between them, and avoid duplicate code?
AutomateIt includes an easy to use system which automatically generates
tags based on the hostname. E.g., hostname "foo.bar.baz" will generate
tags "foo" and "foo.bar.baz" for you:
http://automateit.org/documentation/classes/AutomateIt/TagManager.html
AutomateIt makes it easy to structure your recipes based on roles within
your recipes. For example, you would have all hosts will execute the
project's "all.rb" recipe, which contains a series of lines like:
invoke 'postresql' if tagged?(:database_server)
The above invokes the project's "postgresql.rb" recipe only if this host
is tagged as a "database_server". This "postgresql.rb" recipe should
then be written as generically as possible, with all variables extracted
out into fields:
http://automateit.org/documentation/classes/AutomateIt/FieldManager.html
Using this role-centric approach, you can easily add new machines by
just assigning them tags and have them use the same recipes. Also, if
all machines using a project use a recipe, you don't need to add the
conditional. And if there's a higher-level role, such as
"application_server", then you can instead use a single high-level "if
tagged?(:application_server)" conditional around a bunch of related
recipes for installing Apache, FastCGI, WSGI, Passenger and so on.
If you haven't already, I recommend reading the recipes I wrote for
running opensourcebridge.org. The only down side is that because the
servers are identical, they only use of a single optional role called
"standby" to indicate that a server should redirect all traffic
elsewhere. Anyways, this project is a good way to see how I use AutomateIt:
http://github.com/igal/osbp_automateit
> I am thinking of incorporating facter into the process so I can send
> "facts" to zabbix as well. Zabbix then can raise alarms if the disk
> drive is getting low on space or if there is not enough RAM or
> something.
>
I strongly recommend monitoring anything you care about. I install
"monit" on all machines to alert me of problems with local resources,
like CPU, memory and disk. I also have a central monit server that
checks that network services are running (e.g., confirms that all
websites are responding sanely) and that critical hosts are online. I
then have another server that alerts me if the central monitoring server
goes offline.
-igal
Tim Uckun
>
> PS: Please create new email threads when starting new discussions. This
> makes conversations easier to read and the online archives more useful. :)
>
Sorry, I know better too.
> I'm surprised that you implemented these "bundles" and am curious why.
>
> AutomateIt includes an easy way to create "projects", these are
> directory structures that help you organize multiple recipes and their
> related files:
> http://automateit.org/documentation/classes/AutomateIt/Project.html
Yes I read extensively about them and tried at first to go that way.
In the end it didn't "feel right" to me.
>
>>
> I'm surprised that you implemented this host-centric system and am
> curious why. Doesn't this make it hard to add new servers, share code
> between them, and avoid duplicate code?
The primary reason is because I am coming from puppet and I am used to
working like that. The secondary reason is that I wanted all the
relevant files together in one directory which makes it easier for me
to edit them in groups. Finally I wanted to avoid having to set up
different subdirectories under dist for each host. For example each
host has a http.conf but different hosts have different setups. It
made more sense to me to put the httpd conf for that host in the
same directory as the recipe for that host. I suppose another
solution to that would be to set up a different project for each host
but I didn't want to do that.
As for sharing recipes it's very easy. In the recipe for the host you
say has_tag apache. The apache tag may set up a default httpd.conf and
the host recipe can override that with it's own http.conf
> AutomateIt includes an easy to use system which automatically generates
> tags based on the hostname. E.g., hostname "foo.bar.baz" will generate
> tags "foo" and "foo.bar.baz" for you:
> http://automateit.org/documentation/classes/AutomateIt/TagManager.html
Yes I saw that. Automated is tag centric, even the YAML file is tag
centric as in
tag
- host
- host
I wanted the other way around which feels more natural to me.
host
- tag
- tag
I also wanted to control the order of the tag processing and couldn't
figure out how to accomplish that. Finally I wanted a way to have a
recipe call a dependent recipe and doing it my way seemed like a more
natural way to accomplish that (for me).
>
> AutomateIt makes it easy to structure your recipes based on roles within
> your recipes. For example, you would have all hosts will execute the
> project's "all.rb" recipe, which contains a series of lines like:
> invoke 'postresql' if tagged?(:database_server)
I saw your examples. I will probably make use of that feature heavily
in my own recipes I think. For example
if tagged? (:ubuntu)
$http_conf_path="/etc/apache2/httpd.conf"
elsif ....
...
end
That kind of thing. I like that things are tagged automatically by
automateit. If I had one wish that would be that the tags are a hash
instead of an array so I can query tags[:hostname] or
tags[:ip_address]. Again this is probably me looking for something
like facter and I will probably use facter for that and maybe even
populate the tags array with facts from facter.
> Using this role-centric approach, you can easily add new machines by
> just assigning them tags and have them use the same recipes. Also, if
> all machines using a project use a recipe, you don't need to add the
> conditional. And if there's a higher-level role, such as
> "application_server", then you can instead use a single high-level "if
> tagged?(:application_server)" conditional around a bunch of related
> recipes for installing Apache, FastCGI, WSGI, Passenger and so on.
Maybe I haven't given this a proper chance. I'll look at it again. It
just seems upside down to me.
> If you haven't already, I recommend reading the recipes I wrote for
> running opensourcebridge.org. The only down side is that because the
> servers are identical, they only use of a single optional role called
I have looked at those. I was especially impressed by your use of the
"manager" objects. I think that's a brilliant approach.
> I strongly recommend monitoring anything you care about. I install
> "monit" on all machines to alert me of problems with local resources,
> like CPU, memory and disk. I also have a central monit server that
> checks that network services are running (e.g., confirms that all
> websites are responding sanely) and that critical hosts are online. I
> then have another server that alerts me if the central monitoring server
> goes offline.
>
I agree. I am fond of zabbix myself but anything will do. One of the
major reasons I am looking into automated is because it makes it
trivial to integrate with zabbix. It's very handy to have access to
the entire run cycle from beginning to end. Your approach of "it's
just ruby" wins IMHO.
Igal Koshevoy
>> I'm surprised that you implemented this host-centric system and am
>> curious why. Doesn't this make it hard to add new servers, share code
>> between them, and avoid duplicate code?
>>
>
> The primary reason is because I am coming from puppet and I am used to
> working like that.
self-documenting, lets me derive other roles (e.g., tag "apache_servers"
can be automatically added to all hosts that are "application-servers"
if you find this makes your recipes clearer), and the meta-data is
useful (e.g., I can programmatically get a list of all database servers).
Compare these code samples:
1. Using AutomateIt with role-based tags makes it very clear what this
grouping is:
if tagged?(:app_server)
invoke 'apache'
invoke 'php'
invoke 'passenger'
end
2. Using Puppet nodes provides no hint as to what this group does:
node "host1", "host2" {
include apache
include php
include passenger
}
3. Using AutomateIt with hostnames lets you emulate the Puppet node
approach:
if tagged?("host1 || host2")
invoke 'apache'
invoke 'php'
invoke 'passenger'
end
I realize that you could add the comment "Application servers" above #2
and #3, but you're losing clarity and functionality.
> The secondary reason is that I wanted all the
> relevant files together in one directory which makes it easier for me
> to edit them in groups.
How many unique files do you have per host that this was necessary? How
do you reuse common commands between hosts, e.g., you have to similarly
configured application servers -- which directory does the common
application server recipe live?
For example, in my projects, I have a "recipes" directory that contains
recipes like "postgresql.rb", "apache.rb" and "myapp.rb". If I want to
add another application server, I just add it to the tags.
I typically pile everything right into "dist". However, if this got
overwhelming, I could partition the files, e.g.,:
render dist+"apache/etc/apache2/httpd.conf",
"/etc/apache2/httpd.conf", ...
> Finally I wanted to avoid having to set up
> different subdirectories under dist for each host. For example each
> host has a http.conf but different hosts have different setups. It
> made more sense to me to put the httpd conf for that host in the
> same directory as the recipe for that host.
I solve this sort of problem by either:
1. Rendering one of many templates to a single target. E.g., rendering
either dist+"/etc/http.conf~load_balancer.erb" or
dist+"/etc/http.conf~application_server.erb" to "/etc/http.conf".
2. Using conditionals within the template. E.g., using "if tagged?"
inside the template to render specific portions.
3. Relying on an external tool to partition configurations. E.g.,
putting each site configuration into a separate file, copying/rendering
only specific ones to "/etc/my_apache_sites" and telling Apache to
include everything in this directory, or using the Debian
"sites-enabled" mechanism.
> I suppose another
> solution to that would be to set up a different project for each host
> but I didn't want to do that.
[...]
>> AutomateIt includes an easy to use system which automatically generates
>> tags based on the hostname. E.g., hostname "foo.bar.baz" will generate
>> tags "foo" and "foo.bar.baz" for you:
>> http://automateit.org/documentation/classes/AutomateIt/TagManager.html
>>
>
> Yes I saw that. Automated is tag centric, even the YAML file is tag
> centric as in
>
> tag
> - host
> - host
>
> I wanted the other way around which feels more natural to me.
>
> host
> - tag
> - tag
>
[...]
>> Using this role-centric approach, you can easily add new machines by
>> just assigning them tags and have them use the same recipes. Also, if
>> all machines using a project use a recipe, you don't need to add the
>> conditional. And if there's a higher-level role, such as
>> "application_server", then you can instead use a single high-level "if
>> tagged?(:application_server)" conditional around a bunch of related
>> recipes for installing Apache, FastCGI, WSGI, Passenger and so on.
>>
>
> Maybe I haven't given this a proper chance. I'll look at it again. It
> just seems upside down to me.
I wonder if we ended up with such different solutions because we were
trying to optimize for a different problem. In my case, I have many
servers but few roles, and thus make the roles as central and generic as
possible so that I can have many servers reuse them. Because roles are
so important to me, I use them as the top-level items in the tags and
put the servers beneath them. Figuring out what roles a server has isn't
a big deal because I keep these to a minimum and it's easy to just
search the file.
> I also wanted to control the order of the tag processing and couldn't
> figure out how to accomplish that.
Can you give an example of what you were trying to achieve?
> Finally I wanted a way to have a
> recipe call a dependent recipe and doing it my way seemed like a more
> natural way to accomplish that (for me).
>
How is this different than my approach of doing things like "invoke
'apache'" from the central "all.rb"?
> If I had one wish that would be that the tags are a hash
> instead of an array so I can query tags[:hostname] or
> tags[:ip_address]. Again this is probably me looking for something
> like facter and I will probably use facter for that and maybe even
> populate the tags array with facts from facter.
>
You can get things like hostnames and addresses through address_manager,
e.g.:
ai> address_manager.hostnames
=> ["automateit", "automateit.org", "automateit.pragmaticraft",
"automateit.pragmaticraft.com", ...
ai> address_manager.addresses
=> ["67.207.132.74", "fe80::4240:43ff:fecf:844a/64", "127.0.0.1", "::1/128"]
>> If you haven't already, I recommend reading the recipes I wrote for
>> running opensourcebridge.org. The only down side is that because the
>> servers are identical, they only use of a single optional role called
>>
>
> I have looked at those. I was especially impressed by your use of the
> "manager" objects. I think that's a brilliant approach.
>
Cool. I've been trying to figure out how to clean these up enough and
distribute them as a contrib that people can more easily include, maybe
as part of a more sophisticated plugin mechanism than the simple "lib"
directory.
> It's very handy to have access to
> the entire run cycle from beginning to end. Your approach of "it's
> just ruby" wins IMHO.
Great. It seems that you liked a lot of what Puppet was doing and had to
go pretty far out of your way to adapt AutomateIt to your needs. Why did
you pick AutomateIt?
-igal
Tim Uckun
> configured application servers -- which directory does the common
> application server recipe live?
In the directory with the recipe itself. For example if there is a
ubuntu recipe all the common files for ubuntu to into that directory.
I also wanted to group the files together for some recipes. For
example I have a custom daemon and I need to init.d file for it, the
daemon itself (a single ruby file in this case), a logrotate file for
it, and a yaml file for it. I wanted to keep these together.
>>
>> Maybe I haven't given this a proper chance. I'll look at it again. It
>> just seems upside down to me.
> I wonder if we ended up with such different solutions because we were
> trying to optimize for a different problem. In my case, I have many
> servers but few roles, and thus make the roles as central and generic as
> possible so that I can have many servers reuse them. Because roles are
> so important to me, I use them as the top-level items in the tags and
> put the servers beneath them. Figuring out what roles a server has isn't
> a big deal because I keep these to a minimum and it's easy to just
> search the file.
You may have a point there.
>
>> I also wanted to control the order of the tag processing and couldn't
>> figure out how to accomplish that.
> Can you give an example of what you were trying to achieve?
An example might be that custom daemon needs to be installed after
ruby is installed. I thought it might be neat to be able to put
"has_tag 'ruby'" in the custom daemon recipe.
>> If I had one wish that would be that the tags are a hash
> You can get things like hostnames and addresses through address_manager,
> e.g.:
>
> ai> address_manager.hostnames
> => ["automateit", "automateit.org", "automateit.pragmaticraft",
> "automateit.pragmaticraft.com", ...
> ai> address_manager.addresses
> => ["67.207.132.74", "fe80::4240:43ff:fecf:844a/64", "127.0.0.1", "::1/128"]
Ah that's good to know.
>>
> Cool. I've been trying to figure out how to clean these up enough and
> distribute them as a contrib that people can more easily include, maybe
> as part of a more sophisticated plugin mechanism than the simple "lib"
> directory.
>
> Great. It seems that you liked a lot of what Puppet was doing and had to
> go pretty far out of your way to adapt AutomateIt to your needs. Why did
> you pick AutomateIt?
>
My "framework" is very small actually. I was fooling around one
evening while watching a boring movie and it just fell out. It didn't
seem like work at all.
Also puppet didn't really do what I wanted namely the zabbix
integration and emailing me the errors. Puppet has the tagmail feature
but that only works if the client processes the recipe and sends back
a report. If the client is unable to process the recipe you get
nothing. In order to integrate with zabbix I had to write a custom
report and even that didn't go quite like I expected.
Puppet is very nice. It's got a ton of features, I like a lot of
things about it but I found out that it really requires more work than
you think and IMHO more work than it should. The time it took to
write my stupid simple framework was not really significant compared
to setting up puppet and making it do what I wanted.
I also fooled around a lot with chef which is also very nice but to me
automateit seems easy and straightforward and malleable.