How to Orchestrate State, such as use IPs of new systems for config other systems
49 views
Skip to first unread message
Joaquin Menchaca
Apr 27, 2018, 9:31:24 PM4/27/18
to Puppet Users
Hello Puppeteers,
Relearning Puppet (converting Ansible, Chef knowledge), not sure best approach for this. How can I configure systems based on state of another system? Essentially, I would use a tool like Terraform to bring up systems. When their IP addresses are available, I would then use that information (their IP address) to configure other systems.
I have two simple use cases:
- local repository, and if up and available, want to configure all clients, e.g. java clients, to use this system (such as /etc/hosts with download.oracle.com)
- elastic search cluster where the elasticsearch.yml has ip addresses of all nodes within the cluster.
Any way or best practice in the puppet community for this? In Ansible, as it is orchestration, I can get state of systems with tags, use that to configure the system, and with Chef/Knife can configure an environment attributes (statically), or override it during initial bootstrap process.
Thomas Müller
Apr 30, 2018, 8:19:08 AM4/30/18
to Puppet Users
Am Samstag, 28. April 2018 03:31:24 UTC+2 schrieb Joaquin Menchaca:
Hello Puppeteers,Relearning Puppet (converting Ansible, Chef knowledge), not sure best approach for this. How can I configure systems based on state of another system? Essentially, I would use a tool like Terraform to bring up systems. When their IP addresses are available, I would then use that information (their IP address) to configure other systems.I have two simple use cases:
- local repository, and if up and available, want to configure all clients, e.g. java clients, to use this system (such as /etc/hosts with download.oracle.com)
- elastic search cluster where the elasticsearch.yml has ip addresses of all nodes within the cluster.
you could report to puppetdb and query it in your manifests (puppet pql) . but this introduces a circular dependency as you need to run puppet before having data in puppetdb - so this wouldn't configure your cluster in the first run. And if you have many severs which need ip adresses of the other servers then different puppet runs will produce different config files and maybe restart the service often. I personally would not do it.
Another way is to use the roles/profiles pattern with hiera integrated into the environment or module and add the data to hiera. But it requires that you know your ip addresses (or dns names) before the deployment and that you're able to release code to a puppet environment as you need it.
Or you could take a look at bolt/puppet tasks, or if you're on PE then there is the orchestrator (which i have not used).
It really depends on how the state of your puppet infrastructure is. :-)
- Thomas
Another way is to use the roles/profiles pattern with hiera integrated into the environment or module and add the data to hiera. But it requires that you know your ip addresses (or dns names) before the deployment and that you're able to release code to a puppet environment as you need it.
Or you could take a look at bolt/puppet tasks, or if you're on PE then there is the orchestrator (which i have not used).
It really depends on how the state of your puppet infrastructure is. :-)
- Thomas
Joaquin Menchaca
Apr 30, 2018, 12:29:52 PM4/30/18
to Puppet Users
For pure-cluster pattern, I have to deploy all at once first, then put the config on all the systems. For load balancer + web nodes, the eventual convergence with puppetdb (poor person's service discovery) would work, but clusters are all or nothing deal, eventual convergence would set off alarms.
I am looking into roles/profiles w/ hiera (now roles/profiles w/ hiera + control repo), and trying to read up on this (and usign r10k). It's a bit daunting starting out.
I was wondering about these possibilities (if they are possibilities): (1) custom or external facts, command line, or ruby class, (2) ENC to pass in values, which reads current cloud config (gcp or aws) or local vagrant settings for dev, which can pass these values to the class.
Bolt has me curious. I have dabbled around with it, running commands or a script to a remote node. Are there other ways I could use this? Could I use it to introduce custom facts, or inject values into puppetdb? hiera? The docs are not all that intuitive (had to step through gem source code to discern how to use ssh transport for my environment).
I am looking into roles/profiles w/ hiera (now roles/profiles w/ hiera + control repo), and trying to read up on this (and usign r10k). It's a bit daunting starting out.
I was wondering about these possibilities (if they are possibilities): (1) custom or external facts, command line, or ruby class, (2) ENC to pass in values, which reads current cloud config (gcp or aws) or local vagrant settings for dev, which can pass these values to the class.
Bolt has me curious. I have dabbled around with it, running commands or a script to a remote node. Are there other ways I could use this? Could I use it to introduce custom facts, or inject values into puppetdb? hiera? The docs are not all that intuitive (had to step through gem source code to discern how to use ssh transport for my environment).
Thomas Mueller
Apr 30, 2018, 3:58:09 PM4/30/18
to puppet...@googlegroups.com
On 04/30/2018 06:29 PM, Joaquin Menchaca wrote:
> For pure-cluster pattern, I have to deploy all at once first, then put
> the config on all the systems. For load balancer + web nodes, the
> eventual convergence with puppetdb (poor person's service discovery)
> would work, but clusters are all or nothing deal, eventual convergence
> would set off alarms.
>
> I am looking into roles/profiles w/ hiera (now roles/profiles w/ hiera
> + control repo), and trying to read up on this (and usign r10k). It's
> a bit daunting starting out.
>
> I was wondering about these possibilities (if they are possibilities):
> (1) custom or external facts, command line, or ruby class, (2) ENC to
> pass in values, which reads current cloud config (gcp or aws) or local
> vagrant settings for dev, which can pass these values to the class.
to the master - which might not be safe to trust for important decisions
as you might send credentials to the client based on that data.
An ENC could provide any data you like - but you'll need to create an
own one.
Another possibility might be http://jerakia.io/ by Craig Dunn. which was
created to integrate different data sources. But more data sources ==
more external dependencies.
But none of these will run your puppet agents in correct order.
>
> Bolt has me curious. I have dabbled around with it, running commands
> or a script to a remote node. Are there other ways I could use this?
> Could I use it to introduce custom facts, or inject values into
> puppetdb? hiera? The docs are not all that intuitive (had to step
> through gem source code to discern how to use ssh transport for my
> environment).
- Thomas
Andreas Zuber
Apr 30, 2018, 4:28:21 PM4/30/18
to puppet...@googlegroups.com
On 04/30/2018 06:29 PM, Joaquin Menchaca wrote:
>
>
> Bolt has me curious. I have dabbled around with it, running commands or
> a script to a remote node. Are there other ways I could use this?
> Could I use it to introduce custom facts, or inject values into
> puppetdb? hiera? The docs are not all that intuitive (had to step
> through gem source code to discern how to use ssh transport for my
> environment).
Have a look at https://choria.io/docs/playbooks/basics/ . Like Bolt it
> a script to a remote node. Are there other ways I could use this?
> Could I use it to introduce custom facts, or inject values into
> puppetdb? hiera? The docs are not all that intuitive (had to step
> through gem source code to discern how to use ssh transport for my
> environment).
can run puppet tasks, but you can also write whole plans in the puppet
language to orchestrate a cluster.
It is super easy to setup with the modules provided and is based on
mcollective which already comes with the puppet AIO package.
Regards
Andreas
Joaquin Menchaca
May 1, 2018, 2:04:14 AM5/1/18
to Puppet Users
Both Choria and Jerakia look awesome. I am so glad these awesome folks are still innovating. I will explore these after my pilot project. They look useful outside of Puppet, especially Jerakia, I can see using this across CAP (Chef-Ansible-Puppet).
Whatever happened to MCollective? That sort of went to Puppet and died. I remember the book I picked up at an PupetConf said the tool was support to be orchestrator for both Puppet and Chef.
For my main scope, I want to use minimum method for data injection, so maybe bolt or hiera. It's interesting that this is not something easy on Puppet, and that Puppet really doesn't have variables (really connstants), so the original design was data was embedded into the module. The parameters is fills the role of variables that can be overridden.
Wait, Dayyyuuuum.. Jerakia can Consul... WOW.
Whatever happened to MCollective? That sort of went to Puppet and died. I remember the book I picked up at an PupetConf said the tool was support to be orchestrator for both Puppet and Chef.
For my main scope, I want to use minimum method for data injection, so maybe bolt or hiera. It's interesting that this is not something easy on Puppet, and that Puppet really doesn't have variables (really connstants), so the original design was data was embedded into the module. The parameters is fills the role of variables that can be overridden.
Wait, Dayyyuuuum.. Jerakia can Consul... WOW.
R.I. Pienaar
May 1, 2018, 2:19:46 AM5/1/18
to puppet...@googlegroups.com
Both Choria and Jerakia look awesome. I am so glad these awesome folks are still innovating. I will explore these after my pilot project. They look useful outside of Puppet, especially Jerakia, I can see using this across CAP (Chef-Ansible-Puppet).
Whatever happened to MCollective? That sort of went to Puppet and died. I remember the book I picked up at an PupetConf said the tool was support to be orchestrator for both Puppet and Chef.
Yeah that’s roughly what happened. Those testing out the Puppet 6 nightly will notice mcollective is not in puppet agent anymore
Choria as the site suggests is both modernising mcollective to make it usable to today’s open source user and also redeveloping huge parts of it and adding new end user features. It will provide a trivially installable way to keep using it post the packaging change in puppet agent. So it’s not like it’s going away.
As for puppet/chef etc support. It does, and have always, support other CM or indeed no CM. Given that I had to make it usable on my own again I out of realities of available time had to focus on one CM tool to fix the installation story first
That’s not to say it does not support living in other ones it’s just not as turnkey easy
Someone in the community can totally write some cookbooks to install and deploy it la plugins and it will work
This will become easier soon as the new choria daemons are 1 binary but we will see how that turns out.
For my main scope, I want to use minimum method for data injection, so maybe bolt or hiera. It's interesting that this is not something easy on Puppet, and that Puppet really doesn't have variables (really connstants), so the original design was data was embedded into the module. The parameters is fills the role of variables that can be overridden.
Wait, Dayyyuuuum.. Jerakia can Consul... WOW.
On Monday, April 30, 2018 at 1:28:21 PM UTC-7, Andreas Zuber wrote:On 04/30/2018 06:29 PM, Joaquin Menchaca wrote:
>
> Bolt has me curious. I have dabbled around with it, running commands or
> a script to a remote node. Are there other ways I could use this?
> Could I use it to introduce custom facts, or inject values into
> puppetdb? hiera? The docs are not all that intuitive (had to step
> through gem source code to discern how to use ssh transport for my
> environment).
Have a look at https://choria.io/docs/playbooks/basics/ . Like Bolt it
can run puppet tasks, but you can also write whole plans in the puppet
language to orchestrate a cluster.
It is super easy to setup with the modules provided and is based on
mcollective which already comes with the puppet AIO package.
Regards
Andreas
--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f5610774-cdd4-4dc6-ab7d-31839f945943%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
John Gelnaw
May 1, 2018, 11:18:43 AM5/1/18
to Puppet Users
On Friday, April 27, 2018 at 9:31:24 PM UTC-4, Joaquin Menchaca wrote:
Hello Puppeteers,Relearning Puppet (converting Ansible, Chef knowledge), not sure best approach for this. How can I configure systems based on state of another system? Essentially, I would use a tool like Terraform to bring up systems. When their IP addresses are available, I would then use that information (their IP address) to configure other systems.I have two simple use cases:
- local repository, and if up and available, want to configure all clients, e.g. java clients, to use this system (such as /etc/hosts with download.oracle.com)
- elastic search cluster where the elasticsearch.yml has ip addresses of all nodes within the cluster.
An ENC that queries puppetdb would work. In theory, you could also query something like mcollective-- but as the devs say, that's in kind of a shaky state.
What *is* the status of choria? I've been using mcollective for a long time-- it's auto-deployed on all of my puppet nodes and in my environment "just works", so I've been reluctant to yank out a system that works and replace it with something that's under heavy development, but obviously, I can't stay on the mcollective path forever (btw - this is the first I've heard it's been dropped from Puppet 6, although I haven't been following puppet 6 development closely).
R.I.Pienaar
May 1, 2018, 12:35:53 PM5/1/18
to puppet...@googlegroups.com
On Tue, 1 May 2018, at 17:18, John Gelnaw wrote:
> On Friday, April 27, 2018 at 9:31:24 PM UTC-4, Joaquin Menchaca wrote:
> >
> > Hello Puppeteers,
> >
> > Relearning Puppet (converting Ansible, Chef knowledge), not sure best
> > approach for this. How can I configure systems based on state of another
> > system? Essentially, I would use a tool like Terraform to bring up
> > systems. When their IP addresses are available, I would then use that
> > information (their IP address) to configure other systems.
> >
> > I have two simple use cases:
> >
> > clients, e.g. java clients, to use this system (such as /etc/hosts with
> > download.oracle.com)
> > 2. elastic search cluster where the elasticsearch.yml has ip addresses
> > download.oracle.com)
> > of all nodes within the cluster.
> >
> >
> An ENC that queries puppetdb would work. In theory, you could also query
> something like mcollective-- but as the devs say, that's in kind of a shaky
> state.
>
> What *is* the status of choria? I've been using mcollective for a long
> time-- it's auto-deployed on all of my puppet nodes and in my environment
> "just works", so I've been reluctant to yank out a system that works and
> replace it with something that's under heavy development, but obviously, I
> can't stay on the mcollective path forever (btw - this is the first I've
> heard it's been dropped from Puppet 6, although I haven't been following
> puppet 6 development closely).
>
see the other mail I posted on this thread about choria, happy to answer questions.
> >
> >
> An ENC that queries puppetdb would work. In theory, you could also query
> something like mcollective-- but as the devs say, that's in kind of a shaky
> state.
>
> What *is* the status of choria? I've been using mcollective for a long
> time-- it's auto-deployed on all of my puppet nodes and in my environment
> "just works", so I've been reluctant to yank out a system that works and
> replace it with something that's under heavy development, but obviously, I
> can't stay on the mcollective path forever (btw - this is the first I've
> heard it's been dropped from Puppet 6, although I haven't been following
> puppet 6 development closely).
>
short version you'll be able to continue as you were.
I don't think there's been official at-length communications about this in Puppet 6 but see mails from Eric recently on this list mentioning it etc
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
--
R.I.Pienaar / www.devco.net / @ripienaar
Joaquin Menchaca
May 1, 2018, 3:34:57 PM5/1/18
to Puppet Users
That is tragic for MCollective (and other things at Puppet). I hope it can be maintained outside of Puppet umbrella (where things go to die), and have integration/migration docs/tools later.
I noticed that Salt Stack had striking similarities. Any thoughts on diffs MCollective vs. Salt Stack vs. Choria? I ask this not from marketing bs, but real get down to the nerd enthusiasts perspective.
R.I.Pienaar
May 1, 2018, 5:04:35 PM5/1/18
to puppet...@googlegroups.com
On Tue, 1 May 2018, at 21:34, Joaquin Menchaca wrote:
> That is tragic for MCollective (and other things at Puppet). I hope it can
> be maintained outside of Puppet umbrella (where things go to die), and have
> integration/migration docs/tools later.
Choria runs on ~ 200k systems for my and I am very happy with where its headed, users seem happy and we're getting some pretty good out of the box end user features which is something mco never had
I'll start focussing a lot more on things like self managing clusters and such - the interactive layer and playbooks and such will be there but I think as a discipline we need to move on from centralised orchestration and focus a lot more on self managing clusters, so a lot is going on there. Think k8s operators, habitat etc and there's a lot of research to do there still
Also am very keen on IoT and have some clients looking there, choria is fully embeddable in golang projects and runs a charm on things like raspberry pis etc and I have a in-progress protocols for super light weight stuff like arduino wall plugs and such - bringing cluster management and life cycle management to that etc. You can literally go 'mco rpc wallplug off' and it transparently does protocol translation onto super tiny devices with only a couple k of mem etc
anyway, we have a #choria channel on puppet slack happy to discuss there, I've been fleshing out the content on choria.io a bit too this week
> I noticed that Salt Stack had striking similarities. Any thoughts on diffs
> MCollective vs. Salt Stack vs. Choria? I ask this not from marketing bs,
> but real get down to the nerd enthusiasts perspective.
>
> On Monday, April 30, 2018 at 11:19:46 PM UTC-7, R.I. Pienaar wrote:
> >
> >
> > On 1 May 2018, at 08:04, Joaquin Menchaca <joaqu...@gmail.com
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/puppet-users/f5610774-cdd4-4dc6-ab7d-31839f945943%40googlegroups.com
> > <https://groups.google.com/d/msgid/puppet-users/f5610774-cdd4-4dc6-ab7d-31839f945943%40googlegroups.com?utm_medium=email&utm_source=footer>
> > https://groups.google.com/d/msgid/puppet-users/f5610774-cdd4-4dc6-ab7d-31839f945943%40googlegroups.com
> > .
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/361ecaa7-8122-492a-8347-faf117d930ca%40googlegroups.com.
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
Reply all
Reply to author
Forward
0 new messages