Salt with docker or jails

[Ari Maniatis]

We currently have a server with 30 jails, each running a single application. For those who are unfamiliar with FreeBSD, jails are lightweight containers which contain an entire userland environment isolated from the host, but running on the same host kernel. IP addresses can be shared or different from the host. In many ways it is similar to the recent Docker fashion.

Right now we implemented this setup with 31 minions, one for each jail and one for the host itself. But this is causing some difficulty because the host needs to know about all the minions in order to properly set up the jail in the first place and to maintain certain shared resources. But minion-to-minion communication is troublesome and slow in salt (even though it is possible). So this concept of a minion "owning" and controlling a bunch of other minions doesn't seem to sit right with salt.

On the other hand, if I discard the jail minions and keep only the host minion, many of my states become complex. Instead of using standard pkg upgrade commands, I need to build custom states which are aware of the jails. Rather than being able to share standard system configuration states, all changes needs to be iterated over 30 times and wrapped in "jexec" commands to ensure the updates are propagated properly to all the jails.

This has all come to a head because we now want to configure a duplicate server, with a copy of every jail. So either we need to generate another 31 minions, or re-organise everything into just a single minion per master host.


How are other people handling the situation of multiple lightweight jails with salt?


Cheers
Ari

[Ted Strzalkowski]

I have not managed multiple jails with salt, however, we manage docker containers via salt.  We see the docker containers as ephemeral, though, so we wouldn't be doing a pkg.upgrade inside a container.  We'd roll a new image and deploy it via salt.  I doubt this helps you much.

On the other hand, in thinking about your 30 minions per host problem, it made me think each metal host could be a syndic[1] for the jailed minions residing there.  This would allow you to use master-minion communication (not slow), but still manage the host itself from your main salt master.  

1 - https://docs.saltstack.com/en/latest/topics/topology/syndic.html

 

[Ari Maniatis]

On 9/02/2016 10:59pm, Ted Strzalkowski wrote:

I have not managed multiple jails with salt, however, we manage docker containers via salt.  We see the docker containers as ephemeral, though, so we wouldn't be doing a pkg.upgrade inside a container.  We'd roll a new image and deploy it via salt.  I doubt this helps you much.

It is certainly one possible approach. Our jails are backed by ZFS, so snapshots make this process quite similar to docker. But I'm not sure it is solving the core problem since I'd still want salt or similar scripts to create the snapshot image in the first place, which is mostly the same problem as managing the jails themselves. Perhaps I then avoid all the minions and key management.

On the other hand, in thinking about your 30 minions per host problem, it made me think each metal host could be a syndic[1] for the jailed minions residing there.  This would allow you to use master-minion communication (not slow), but still manage the host itself from your main salt master.  

1 - https://docs.saltstack.com/en/latest/topics/topology/syndic.html

That's very interesting. It seems like a master-proxy concept which looks useful for management of the keys and connections, but doesn't solve the underlying structural issue of inter-minion communication, or minions which are 'owned' by other minions.

[Alex Leonhardt]

Actually, syndic should be solving that "inter-minion comms problem" if what you are trying is to control the "jail-minions" from the "host-minion" but keep the "host-minion" still managed by the salt-master. Does that not sound like what you're trying to do?

Alex

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Florian Ermisch]

Also the keys for minions connected to the syndic are accepted on the syndic.
Thus you can manage (and pre-accept) the jail-minions' keys quite easily.

Regards, Florian

> <javascript:_e(%7B%7D,'cvml','salt-users%2Bunsu...@googlegroups.com');>
> > .

[Ari Maniatis]

On Wednesday, 10 February 2016 16:59:22 UTC+11, Alex Leonhardt wrote:

Actually, syndic should be solving that "inter-minion comms problem" if what you are trying is to control the "jail-minions" from the "host-minion" but keep the "host-minion" still managed by the salt-master. Does that not sound like what you're trying to do?

Alex

OK, then I'm not quite understanding syndic yet. I've read the docs several times without more clarity that this:

jail-minions connect not to the real master but to a master running on the host machine. That in turn connected to a syndic daemon which connects to the real master. There is also another salt-minion running on the host machine.

Now the real master will gain visibility of all the jail-minions and be able to control them normally. But the salt master trusts that the syndic daemon has successfully authorised the jail-minions so that key management problem is easier (or just different).

But how does the salt-minion running on the host machine get access to the configuration and pillar data of all the jail-minions? If a make a new jail, I'll want to get the host-minion to create the jail and do certain setup things. But then initialise the jail-minion to finish the job of setting up the contents of the jail and application setup.

Am I missing something?

[Alex Leonhardt]

Without trying myself I'm not sure I can help any further. My thought was that the host-minion doesn't need to know all that, as the salt-master does and manages all the jail-minions, via the host-minions.

Maybe if you shared some more background info of what it is that one jail-minion needs from another (on the same host-minion) that might help shed some light.

On Thursday, 11 February 2016, Ari Maniatis <aristedes...@gmail.com> wrote:

On Wednesday, 10 February 2016 16:59:22 UTC+11, Alex Leonhardt wrote:

Actually, syndic should be solving that "inter-minion comms problem" if what you are trying is to control the "jail-minions" from the "host-minion" but keep the "host-minion" still managed by the salt-master. Does that not sound like what you're trying to do?

Alex

OK, then I'm not quite understanding syndic yet. I've read the docs several times without more clarity that this:

[snip]

But how does the salt-minion running on the host machine get access to the configuration and pillar data of all the jail-minions? If a make a new jail, I'll want to get the host-minion to create the jail and do certain setup things. But then initialise the jail-minion to finish the job of setting up the contents of the jail and application setup.

Am I missing something?

--

[Aristedes Maniatis]

The jail-minions don't need to know about each other, but the host-minion needs to know about all the jails. Firstly, it is responsible for creating the jail environments and secondly there are some shared resources like monitoring which need to be set up correctly on the host.

After some thought, perhaps the best approach is to make the jails more lightweight and disposable. If I can get each of them down to just a single identical disk image (except one config file which contains db passwords, etc) then I can more easily control them from outside the jails using just the host-minion. And no jail-minion.

The simplest salt configuration might be no salt at all!

Ari

You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/qr_pPGaTRlc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.