Salt in offline environments (without Internet): bootstrap from clean OS, install system from single package using standalone minion

[uvs...@gmail.com]

I'm working on application of Salt in offline environment (without Internet).

The idea is to have a single package (tar/zip) to deploy any host of a system "in one click".

This package is all-in-one solution:

* It has all necessary Salt states and pillars files specific for system being deployed.

* It includes Salt bootstrap which installs and configures standalone Salt minion (non-service) to execute all required states according to the role assigned to this specific host within system.

* It contains all necessary resources (binaries: rpm, zip, etc.) which are required for both: Salt bootstrap and system deployment.

I've just completed successfully one cycle from clean OS, through bootstrap to Salt setup, through `state.highstate` to fully configured nodes according to their roles.

One immediate benefit is actually offline bootstrap alone.

I took a look at Salt bootstrap, but it uses Internet which violates one of my required use case:

https://github.com/saltstack/salt-bootstrap

The rest is pretty much a boring automation to configure Salt minion so that it can find all states, pillars, resources to successfully execute `salt-call --local state.highstate`.

It took a while and now I think that I might have just re-invented the wheel.

* Is there an open sourced version of similar solution in common use?

* Has anyone did something similar before?

* Any thoughts/problems/solutions to share in words or sources?

[Colton Myers]

> * Is there an open sourced version of similar solution in common use?

I don't think so. As you've probably realized, this kind of solution takes some serious effort, especially if you want it to work on multiple platforms. If you're willing to opensource your solution, that would be awesome!

> * Has anyone did something similar before?

We do something similar for certain enterprise customers, but it's currently fairly hacky and we haven't done it for the general use case yet.

--
Colton Myers
Platform Engineer, SaltStack
@basepi on Github/Twitter/IRC

[Ryan Lane]

Probably not, but it's pretty easy to roll your own, assuming you have pretty homogenous infrastructure (all Ubuntu, small number of versions).
 

* Has anyone did something similar before?

Lyft is doing part of this for all of its infrastructure. We have a deployment model where our deployment artifacts live in S3 and we pull the artifacts from there.

We generate salt artifacts by making virtualenvs for each distro version and store them in S3. We generate them in containers, so that we can generate them all from a single system and so that the location of the virtualenv can be the same on the generation system and the system that'll be using it.

We have a deployment system that's scheduled from the client that checks for new versions of code, fetches them, then applies them.

 

* Any thoughts/problems/solutions to share in words or sources?

Check out: http://ryandlane.com/blog/2014/08/26/saltstack-masterless-bootstrapping/ We're of course not doing fully offline, but offline is only another step past masterless.

The hardest part will be putting all the external resources together, but I'd imagine you've already got that step down.

- Ryan

[uvs...@gmail.com]

On Thursday, April 2, 2015 at 6:28:24 AM UTC+8, Ryan Lane wrote:

The hardest part will be putting all the external resources together, but I'd imagine you've already got that step down.

Yes. It's actually a lot about external resources.

And the solution to managing them should satisfy three cases:

(1) Be extendable for types of data you might need to access in the future with their own peculiarities: simple external files, or entire repositories like Git, SVN, YUM, or even Syncthing, ...

(2) Be able to switch from online mode (resources are on the network) to offline mode (all required resources are downloaded).

(3) Make it transparent your Salt states so that they access resources regardless of online or online mode. This requires an access layer abstraction which hides these details from Salt states.

Actually, the 3rd point is the key.

And it even extends beyond that to bootstrap script (before Salt is ready) - must use offline resources as well.

On Thursday, April 2, 2015 at 6:19:24 AM UTC+8, basepi wrote:

We do something similar for certain enterprise customers, but it's currently fairly hacky and we haven't done it for the general use case yet. 

I'm in the same state. What eventually comes out is a framework on top of Salt:

some mandatory keys in your pillars,

some mandatory way to write states (which access external resources),

some repository with offline external resources for bootstrap per each supporting platform. 

 

And framework always have these problems:

For those who already has substantial number of states written, It will requires rewrite them all.

For those who start from beginning, it requires learning how to use the framework (on top of learning Salt).

So, I'm not in a hurry to opensource (it's hard, and probably useless due to all these issues).

Wish

Then I start hoping that somewhere in distant release of Salt there could be a built-in support for this.

The idea is that all attempts to access external resource may go through a module which is able to map from online to offline locations depending on the mode.

This will remove a need for framework and become a pure Salt feature.

And I believe this is much more useful than just full offline.

Unavailability of some parts of resources (i.e. proprietary ones in company's networks) is likely common case for many.

And the need to map only some specific resources from online to offline depending on deployment environment can be pretty general.

Unfortunately, I don't see simple solution here.

[rahav jv]

Hi Alexey,

Did you get around building a salt minion package to work in a offline environment.

Thanks,

Rahav