saltstack reference implementations and best practices

[John Albietz]

At the meetup tonight at Twilio in SF, Sumbry mentioned that he felt that there's a unique window of opportunity with Saltstack right now to correct what has happened with other systems such as Chef/Puppet, where they seemed to have a lack of accepted best practices that eventually fragmented the community and hurt the eco-system.

Other people also expressed that it sounded interesting, so I'd just like to propose that interested people in the community get together and break into groups to build the components of a reference saltstack platform setup that contains well-tested and high-availability setups of components that are common to many of our devops setups.

Sometimes it's fun building everything from scratch, but with some systems I definitely would appreciate having a good baseline implementation to begin with.

Some ideas for system components are:

• logging and monitoring systems
• Continuous Integration & Continuous Deployment
• low-level system management
• security hardening and access controls
• app-level system management
• dev/test/prod environment management
  
• auto-scaling groups
• auto-healing groups.
• alerting systems
• openstack grizzly
• artifact repository
• hadoop/hbase cluster
  
• percona cluster
  
• mongodb cluster
  

As an initial step, we could try integrating some of the states repos that a lot of people have been creating and sharing on github. I've started something like this at my (now inappropriately named) saltmine repo: https://github.com/saltops/saltmine

Clearly I'll need to rename the repo and project now that there's the new salt mine product, so maybe we can start trying to combine efforts here. I'll also throw out the idea of possibly taking over the badly neglected salt-states repo and making this effort quasi-blessed by Thatch? https://github.com/saltstack/salt-states

Any thoughts?

- John

[Luminous Salt]

On 2013-06-12 03:11, John Albietz wrote:
> At the meetup tonight at Twilio in SF, Sumbry mentioned that he felt
> that there's a unique window of opportunity with Saltstack right now
> to correct what has happened with other systems such as Chef/Puppet,
> where they seemed to have a lack of accepted best practices that
> eventually fragmented the community and hurt the eco-system.

Having left puppet for salt, I completely agree - getting one
individual's puppet-mysql working with your/another's puppet-apache
turned out to be a useless waste of time. Furthermore, no one in #puppet
seemed to think such an effort to build a common stack was worth the
investment, citing: "everyone's got different needs" - so silly! They
even had hiera integration! (Doesn't pillar make this easier..?)

> Other people also expressed that it sounded interesting, so I'd just
> like to propose that interested people in the community get together
> and break into groups to build the components of a reference saltstack
> platform setup that contains well-tested and high-availability setups
> of components that are common to many of our devops setups.
>
> Sometimes it's fun building everything from scratch, but with some
> systems I definitely would appreciate having a good baseline
> implementation to begin with.

The vast majority of a new user's needs would be met by such a "common"
stack - the biggest challenge seems to be _users_ getting over their own
ideas on how a server _should_ be setup. In other words, if we all just
let our stacks agree on some basic standards,

Obviously, not everyone's needs would be taken care of, but that's
fine.. that's why we stuff details in pillar and why we have such an
extensible system.

>
> As an initial step, we could try integrating some of the states repos
> that a lot of people have been creating and sharing on github. I've
> started something like this at my (now inappropriately named) saltmine
> repo: https://github.com/saltops/saltmine

It seemed the salt-formulae repo was going to be this, at least in some
capacity. It was disheartening to see this fall by the wayside. I would
like to propose that we take over that repo and start merging and
hacking away (though maybe interested contributors should meet up first
and formulate some baseline agreements for how things will work?), or
spin up our own if saltstack wants to hold on to salt-formulae for some
reason (Thomas, thoughts?)


just my two cents,


Thanks for bringing this up John!

[Avi Marcus]

Many real-live code examples for best practices would be great!

And if it's well documented (explaining how it works, and perhaps another section on why this method was chosen), that would be great for documentation for the project.

 

-Avi

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Stevearc]

This is a great idea!

I would like to see it lean more towards design and architecture rather than a bunch of states. For the most part, states are very easy to write. Once I've seen a few examples of states, I can go off and write a new one with not much trouble in not much time. What I would like to see more of is topics like:

• What does a well-organized state tree look like on disk?
• What are some of the ways to use pillars? (It was really useful to see Ari's use of general pillars with values that he then overrides in certain environments)
• More best practices (I've seen more and more people defining their own grains and using those as machine roles. This should be documented and/or encouraged)
• How to do testing of a salt environment (states mostly, but modules and runners would be helpful too)
  

And on the subject of some of the things you suggested, it would also be great to see tutorials/howtos on how to integrate salt with a logging and/or monitoring system, how to integrate nicely with a CI server, etc.

[Mike Chesnut]

I think this is a fantastic idea as well.  I recently switched jobs to a Chef shop, and have come to realize that one of the really compelling reasons people have for choosing Chef (and committing to/sticking with it) is that there is a centralized community repository of cookbooks that is maintained by Opscode.  They have well-established practices in place for community members to submit new recipes, or to submit fixes for existing recipes.  (It has its own issue tracking system, and established conventions that must be followed for submitting pull requests.)

There have been a couple of attempts to do this with Salt, but all seem to have lost steam pretty quickly.  What I think they're all lacking is "official" support from SaltStack, Inc, in terms of oversight and quality control and being the central point of authority for maintaining it.  (There's probably an FTE required for this, really, so it'd be a non-trivial commitment.)

While Salt has a much lower barrier to entry than other tools (such as Chef), I think there's a significant number of people out there who are looking for an easier bootstrap process to get them started with actually using it to provision and configure their hosts (rather than just seeing how easy it is to get a master set up and running test.ping).

[Steven Arcangeli]

A central repository of states could be quite useful. I know one of the reasons why I haven't bothered open-sourcing any of mine is that most of them were pretty easy to write and are very specific to the OS that we're running. If there were instead some canonical state files with edits/additions encouraged, I would love to change my workflow to:

1) Check out canonical state

2) Does it work? Yay! If not, step 3

3) Extend state file until it works on my OS as well

4) Send changes back in a pull request

If we could get lots of people on this, we could probably easily cover the most common states needed on the most common platforms.

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/Xj8d9-O2j58/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.

[Corey Quinn]

This may be unpopular, but I'd want to see a community manager / curator for these.

Let's take adding users as a simple example.

You have a way to add users that's different from my way to add users that's different from Mike's way to add users…

The problem isn't "How do I add users," it's "how do I build states to add users that isn't going to explode on me when a new user comes aboard with an edge case?"  For instance, my old way of adding users fell to pieces when we hired a guy with multiple ssh keys. Not having to refactor that every time we hit a new use case would be ideal, and the only way I see to get there is to have someone whose responsibility it is to dictate best practices for such thing.

-- Corey

You received this message because you are subscribed to the Google Groups "Salt-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.

[Thomas S Hatch]

Ok, sorry I have been slow on this one, I have spent the last 3 hours researching how we can do this in a clean way, I am going to open up the saltstack formulas organization of github and start populating it.

Can you guys pint me to some of your repos real quick so I can use your stuff to start populating?

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

[Thomas S Hatch]

I have started btw, I only have stuff up for ntp and nano so far, and a little up for salt, more coming tonight:

https://github.com/saltstack-formulas

I will announce this more formally once I have filled out most of these, as usual, pull reqs welcome! Right now I am just trying to stub out the basics!

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

[Ben Hosmer]

Is this replacing the salt-states?

--

You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Sent from Gmail Mobile

[Thomas S Hatch]

Yes, this is going to deprecate salt states and formulae, we need to transfer the working states out of salt states.

We are also working on tools to allow people to easily download states from the new repos as well

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

[John Albietz]

Rad! 

Can we have subdirectories in each repo for different templating examples? Jinja, pydsl, mako, straight python maybe?

You know I love jinja, but I'm sure that if someone really likes pydsl or mako they'd be happy to convert the jinja example they look at into their favorite template lang and open a pull req.

Just sayin'. :-)

[Thomas S Hatch]

The standard repos should stick close to jinja and keep things consistent, I do agree though that we should have more examples in other renderers. Perhaps we should have repos called -pydsl or -mako

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

[John Albietz]

Sounds good to have separate repos for pydsl / mako. 

Sent from my iPhone

[Fernando Mota]

Seems like Saltstack Formulas is good and constantly updated, but I'm missing other thing: A index.

Something like http://gruntjs.com/plugins page, which should be easy to make with Github API (I'm just suggesting, anyway).

Or there is any index for this repository? (which is not GIthub's own page of the organization).

Thanks