metaparam question

[R.I.Pienaar]

hello,

I would like to add a metaparameter - which I think is easy now via
Type.newmetaparam.

The thing that I can't seem to find any example of though is how to
make this metaparameter do something on the nodes for all providers
or all types.

Imagine there's a metaparam that might describe how to test a resource
works, something like:

service{"httpd": validate => "check_http --port 80 -H localhost"}

I'd then want to have some code that would be run on the agent nodes
for any resource that has this param set.

Does anyone have any hints or know of an example of this?

thanks

---
R.I.Pienaar

[Kylo Ginsberg]

On Wed, Feb 3, 2016 at 7:47 AM, R.I.Pienaar <r...@devco.net> wrote:

hello,

I would like to add a metaparameter - which I think is easy now via
Type.newmetaparam.

We haven't been thinking of metaparameters as a general purpose extension point. This came up once before that I know of, about a year ago, and there's a little discussion of this in https://tickets.puppetlabs.com/browse/PUP-4281. The conclusion we reached at the time was, more or less, to explore whether the desired change could be accomplished with a puppet function and/or a change to core puppet.

 

The thing that I can't seem to find any example of though is how to
make this metaparameter do something on the nodes for all providers
or all types.

Imagine there's a metaparam that might describe how to test a resource
works, something like:

   service{"httpd": validate => "check_http --port 80 -H localhost"}

I'd then want to have some code that would be run on the agent nodes
for any resource that has this param set.

I don't think something like this exists per se, but 'validate' might be one such example of something worth adding to core puppet. Fwiw, one resource-specific example added not too long ago is the file type's validate_cmd: 

https://docs.puppetlabs.com/puppet/latest/reference/type.html#file-attribute-validate_cmd.

Also, I'm guessing you're aware but another tool in the toolbox *might* include the postrun_command setting (which is *not* per-provider as you were thinking):

https://docs.puppetlabs.com/puppet/latest/reference/configuration.html#postruncommand

More broadly, there has been some discussion on a few threads and tickets about making the agent lifecycle provide some more useful hooks, so I'd love to hear more ideas about how to improve this.

Thanks!

Kylo

Does anyone have any hints or know of an example of this?

thanks

---
R.I.Pienaar

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/70254077.345165.1454514435921.JavaMail.zimbra%40devco.net.
For more options, visit https://groups.google.com/d/optout.

--

Kylo Ginsberg | ky...@puppetlabs.com | irc: kylo | twitter: @kylog

[Luke Kanies]

On Feb 4, 2016, at 3:35 PM, Kylo Ginsberg <ky...@puppetlabs.com> wrote:

On Wed, Feb 3, 2016 at 7:47 AM, R.I.Pienaar <r...@devco.net> wrote:

hello,

I would like to add a metaparameter - which I think is easy now via
Type.newmetaparam.

We haven't been thinking of metaparameters as a general purpose extension point. This came up once before that I know of, about a year ago, and there's a little discussion of this in https://tickets.puppetlabs.com/browse/PUP-4281. The conclusion we reached at the time was, more or less, to explore whether the desired change could be accomplished with a puppet function and/or a change to core puppet.

I don't remember my original logic for metaparams not being an extension point. Given the system's support for easily loading this kind of plugin, it should be easy to add.

The discussion on the ticket is pretty limited. We've got multiple use cases here, and I have seen use cases in the past brought up, but they were mostly on lists or in person, so I don't think they made it into the ticketing system.

RI - you up for a pull request to add the autoloading? Have you thought much about how this might go wrong?

The thing that I can't seem to find any example of though is how to
make this metaparameter do something on the nodes for all providers
or all types.

Imagine there's a metaparam that might describe how to test a resource
works, something like:

   service{"httpd": validate => "check_http --port 80 -H localhost"}

I'd then want to have some code that would be run on the agent nodes
for any resource that has this param set.

I don't think something like this exists per se, but 'validate' might be one such example of something worth adding to core puppet. Fwiw, one resource-specific example added not too long ago is the file type's validate_cmd: 

https://docs.puppetlabs.com/puppet/latest/reference/type.html#file-attribute-validate_cmd.

Also, I'm guessing you're aware but another tool in the toolbox *might* include the postrun_command setting (which is *not* per-provider as you were thinking):

https://docs.puppetlabs.com/puppet/latest/reference/configuration.html#postruncommand

Ryan and others on his team have discussed ways of using the new consume/produce system for doing service validation. I don't think the whole system is quite in place, but there are some good ideas getting close.

That doesn't settle the generalized plugin capability issue, though.

More broadly, there has been some discussion on a few threads and tickets about making the agent lifecycle provide some more useful hooks, so I'd love to hear more ideas about how to improve this.

Thanks!

Kylo

Does anyone have any hints or know of an example of this?

thanks

---
R.I.Pienaar

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/70254077.345165.1454514435921.JavaMail.zimbra%40devco.net.
For more options, visit https://groups.google.com/d/optout.

--

Kylo Ginsberg | ky...@puppetlabs.com | irc: kylo | twitter: @kylog

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALsUZFHniTYyYU0ThP-Jk37f6QXroJ0P0-K0PkbRPXMsdB6v0Q%40mail.gmail.com.

[R.I.Pienaar]

hello,

Thanks for the replies, think I'll reply to both in one go

----- Original Message -----
> From: "Luke Kanies" <lu...@puppetlabs.com>
> To: "puppet-dev" <puppe...@googlegroups.com>
> Sent: Friday, February 5, 2016 6:15:28 AM
> Subject: Re: [Puppet-dev] metaparam question

>> On Feb 4, 2016, at 3:35 PM, Kylo Ginsberg <ky...@puppetlabs.com> wrote:
>>
>>> On Wed, Feb 3, 2016 at 7:47 AM, R.I.Pienaar <r...@devco.net> wrote:
>>> hello,
>>>
>>> I would like to add a metaparameter - which I think is easy now via
>>> Type.newmetaparam.
>>
>> We haven't been thinking of metaparameters as a general purpose extension point.
>> This came up once before that I know of, about a year ago, and there's a little
>> discussion of this in https://tickets.puppetlabs.com/browse/PUP-4281. The
>> conclusion we reached at the time was, more or less, to explore whether the
>> desired change could be accomplished with a puppet function and/or a change to
>> core puppet.
>
> I don't remember my original logic for metaparams not being an extension point.
> Given the system's support for easily loading this kind of plugin, it should be
> easy to add.
>
> The discussion on the ticket is pretty limited. We've got multiple use cases
> here, and I have seen use cases in the past brought up, but they were mostly on
> lists or in person, so I don't think they made it into the ticketing system.
>
> RI - you up for a pull request to add the autoloading? Have you thought much
> about how this might go wrong?

Unfortunately my time is pretty limited and I'll prefer to see what I can do
with this information in the catalog rather than making a merge worthy patch
at present.

I'm happy to just patch my actual puppet code with no modules to get to that
point, I'll have a look at the validate_cmd in file to see if there is a way
in the provider base class to do it so all providers get it.

For me this being in every provider is a nice to have I guess - it would be
great to be able to improve puppets ability to check if something worked
in the broad sense, and it's going to apply to all resource types equally.

I could take the route that https://github.com/nanliu/puppet-transport does
and instead of making the meta param do something use it just to store this
information and then some other module do something with this information and
that would let me play with what I want, but it seems a small step to improve
puppet in general.

I've told many people this story so no doubt you remember the context Luke
but what I'd like to do is allow people to attach these checks to resources
and of course use them in catalog apply stage to get a deeper sense of 'done'
in the same way that the File validate_cmd does but later I want to consume
the catalog in other tools and extract the same - or rather hopefully speak
to some API and get this.

Imagine I had a bridge to Sensu where it loads the catalog and just checks
everything that has a validate attached for free with no configuration required,
this is actually really easy with sensu as it doesnt require you to declare
the checks upfront on the server and all you have to do is write JSON to a
socket for every check, it'll promiscuously take them and show them and do
alerts etc.

Once you have this information and can surface it additionally into the Puppet
eco system you can gather the state regularly like a monitoring system does
and it becomes quite interesting later down the line when perhaps you can
use this to influence the node assignment logic in the orchastrator so that
a Web app will be configured to talk to a known good DB server - or that
you can detect that the web app should be reconfigured once it's chosen
DB isn't good anymore. That's quite far down the line though, I think just
being able to do the monitoring part into sensu would already be a huge win
for many.

> https://groups.google.com/d/msgid/puppet-dev/FB2D2883-CE00-4D61-ACC2-0B59F807A4D5%40puppetlabs.com.

[Kylo Ginsberg]

I kept meaning to get back to this thread and it slipped my mind. Mea culpa on that.

I'm curious where you got with this approach? E.g. did you end up patching an additional metaparameter and were there any lessons worth sharing in that?

 

For me this being in every provider is a nice to have I guess - it would be
great to be able to improve puppets ability to check if something worked
in the broad sense, and it's going to apply to all resource types equally.

I could take the route that https://github.com/nanliu/puppet-transport does
and instead of making the meta param do something use it just to store this
information and then some other module do something with this information and
that would let me play with what I want, but it seems a small step to improve
puppet in general.

I'm intrigued by the idea of a generic 'validate' or 'health check'. It sounds like you're thinking of it as separate from the rest of the type/provider, i.e. an external command that could be run, but I also wonder (this is half-baked) if, rather than a separate metaparameter, it could use a standard provider method to do the validation. I.e. when puppet is applying the catalog it has to essentially ask the same question that a 'validate' method would to figure out if it needs to remediate, so can the same need be met with some refinement of the generic type/provider api?

(And btw, thanks for the pointer to puppet-transport. I looked at that long ago and had forgotten (or more likely didn't internalize) the use there of a do-nothing metaparameter to store connection information.)

I've told many people this story so no doubt you remember the context Luke
but what I'd like to do is allow people to attach these checks to resources
and of course use them in catalog apply stage to get a deeper sense of 'done'
in the same way that the File validate_cmd does but later I want to consume
the catalog in other tools and extract the same - or rather hopefully speak
to some API and get this.

Imagine I had a bridge to Sensu where it loads the catalog and just checks
everything that has a validate attached for free with no configuration required,
this is actually really easy with sensu as it doesnt require you to declare
the checks upfront on the server and all you have to do is write JSON to a
socket for every check, it'll promiscuously take them and show them and do
alerts etc.

Once you have this information and can surface it additionally into the Puppet
eco system you can gather the state regularly like a monitoring system does
and it becomes quite interesting later down the line when perhaps you can
use this to influence the node assignment logic in the orchastrator so that
a Web app will be configured to talk to a known good DB server - or that
you can detect that the web app should be reconfigured once it's chosen
DB isn't good anymore.  That's quite far down the line though, I think just
being able to do the monitoring part into sensu would already be a huge win
for many.

I haven't heard this story before but it's super interesting. Since you mention the orchestrator and you're thinking about monitoring/validation to gate or to assign connections between nodes, have you seen http://docs.puppetlabs.com/pe/latest/app_orchestration_availability_tests.html#using-availability-test? Availability tests as described there are just a start but they seem like they're in the neighborhood of what you're describing above. 

The added idea here of integrating such a thing, be it availability test or validate metaparameter, with a monitoring system could be really powerful. Wrt catalog format (or perhaps report format?) it would be interesting to explore whether we could adapt the format to facilitate such integrations. Currently both catalog format and report format are more serializations of internal data structures than they are designed as external integration points. I'd have to noodle on this some more, but it's definitely a course worth exploring.

Kylo

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/1273177970.271.1454659320271.JavaMail.zimbra%40devco.net.

For more options, visit https://groups.google.com/d/optout.

[R.I.Pienaar]

Unfortunately work is a bit busy right now so I've not had time to really give it
any screen time :(

>> For me this being in every provider is a nice to have I guess - it would be
>> great to be able to improve puppets ability to check if something worked
>> in the broad sense, and it's going to apply to all resource types equally.
>>
>
>> I could take the route that https://github.com/nanliu/puppet-transport
>> does
>> and instead of making the meta param do something use it just to store this
>> information and then some other module do something with this information
>> and
>> that would let me play with what I want, but it seems a small step to
>> improve
>> puppet in general.
>>
>
> I'm intrigued by the idea of a generic 'validate' or 'health check'. It
> sounds like you're thinking of it as separate from the rest of the
> type/provider, i.e. an external command that could be run, but I also
> wonder (this is half-baked) if, rather than a separate metaparameter, it
> could use a standard provider method to do the validation. I.e. when puppet
> is applying the catalog it has to essentially ask the same question that a
> 'validate' method would to figure out if it needs to remediate, so can the
> same need be met with some refinement of the generic type/provider api?

As a POC yes, I'd like it external - there's a ton of Nagios plugins out there
and they would magically all be useful without any extra work, with the time
I have minimising the yaks have value :)

But there are common patterns to these things, so for sure you could mark
up a module/resource with some additional properties or allow provider authors
to do better than we do today - but then they more or less can already code their
modules to do in depth health tests if they wanted. Probably the type/provider
API needs some love in general and if that was something you'd make front and
centre it would be great. I imagine puppet might get some built in 'checks'
like this, say a generic built in TCP checker that we can reference here instead
of calling out to a nagios plugin (which you then have to manage), and that would
be pretty neat and a good place to add pretty good deep introspection value
for module authors. Putting some nasty CLI call with a bunch of arguments
hard coded into a resource is not elegant at all - like nagios based check
validation would cause, that would not be my final destination but a reasonable
stop along the way.

The meta parameter though has value - I see this to a large extend as user
serviceable. I get a module from the forge that has some provider, for me
a 'responds well' check might be < 10ms for others it might be < 100ms and
for others it might not even be a time based expression.. And the parameters
that makes up what is good - here I used response time - are actually very varied
so no doubt some space for getting this in a magical way for now a meta param
seems a easy thing to reason about for now while I look at how this works and
how feasible it is.

Another approach might be to introduce a more formal observer pattern into the
system, I've toyed with this a few times and there's a interesting example over
at https://forge.puppetlabs.com/ryanuber/tell though it's not what I want but
as a thing to look at and think about it's quite interesting. My thoughts are
not really well formed around it, but I did built a network wide observer for
puppet events and could do pretty crazy things with it, not sure they were
good ideas though.

Yes, I saw those after Eric mentioned it to me at the contributor summit, they
are interesting for sure and along the lines of what I mean above about the generic
patterns in checks and module authors bringing their own built in ones. Though
you'll find few/no successful monitoring systems that told people to rewrite every
check from fresh when thousands of nagios ones exist. The nagios layer could well
be one of these checks though.

> The added idea here of integrating such a thing, be it availability test or
> validate metaparameter, with a monitoring system could be really powerful.

yes in as much as monitoring isn't what monitoring is about. It's about the data
thats the side effect of monitoring and what you can do with it. I think Puppet
is fairly uniquely situated to do things with that data very few others can do.

> Wrt catalog format (or perhaps report format?) it would be interesting to
> explore whether we could adapt the format to facilitate such integrations.
> Currently both catalog format and report format are more serializations of
> internal data structures than they are designed as external integration
> points. I'd have to noodle on this some more, but it's definitely a course
> worth exploring.

Yeah, I've done nasty little things to extract information from catalogs and
they all broke recently with 4 :P Public APIs around interacting with it
would be good. A library to do so would be fantastic

[Trevor Vaughan]

Hi all,

This is a very interesting conversation and I'm monitoring to see where it ends up.

You're going to want to have a central location for organizing this information between nodes since there are no guarantees that any two nodes can get bi-directional communications between each other.

Take, for instance, a highly regulated HIPAA environment. You may be able to send data one way between two managed environments but bi-directional communication would be unable to be processed. Therefore, you would need some level of controlled and shared trigger between your host catalogs. (OK, fine, I still want language-level semaphores).

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/454494855.237167.1456911800479.JavaMail.zimbra%40devco.net.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699

-- This account not approved for unencrypted proprietary information --

[R.I.Pienaar]

----- Original Message -----
> From: "Trevor Vaughan" <tvau...@onyxpoint.com>
> To: "puppet-dev" <puppe...@googlegroups.com>
> Sent: Wednesday, 2 March, 2016 11:50:27
> Subject: Re: [Puppet-dev] metaparam question

> Hi all,
>
> This is a very interesting conversation and I'm monitoring to see where it
> ends up.
>
> You're going to want to have a central location for organizing this
> information between nodes since there are no guarantees that any two nodes
> can get bi-directional communications between each other.
>
> Take, for instance, a highly regulated HIPAA environment. You may be able
> to send data one way between two managed environments but bi-directional
> communication would be unable to be processed. Therefore, you would need
> some level of controlled and shared trigger between your host catalogs.
> (OK, fine, I still want language-level semaphores).

quite, that's what the puppet orchastrator and node catalogs will do eventually.

In the mean time as a building block what I am proposing:

service{"your_app":
ensure => "running",
validate => "check_my_app -H localhost -p 80"
}

Might connect to a local web app that was configured with its DB details and it
might hit something like /status which might return:

{"database": "ok", "health": "ok", "metrics": {"requests": 1234, "avg_response_time": 0.2}}

and so you know the service up and happy and working much more than just a:

service you_app start

would know. At a deeper level additional per resource reporting could include
things like this additional status and using the nagios metric protocol you could
communicate the metrics requests and avg_response_time back to Puppet for surfacing
in the various consoles.

I'd as a side effect feed that into let say sensu as well so your other monitoring
will automagically become aware of this resoure and its status.

> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXjF%3DTh1o5vJBU9hW%3DHhmxNDcDf5CHAkc-TUM9HF4hxDA%40mail.gmail.com.

[Trevor Vaughan]

Now that I'm thinking more about it, I 100% want this.

There have been quite a few occasions where I didn't want to pass back an epic ton of fact data just to change a small amount of logic.

Being able to bookend my resources without a hacky exec chain would be great.

Propagating that check to other resources would be kind of neat too but that's probably easier done with class chaining anyway.

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/435304898.284658.1456916431327.JavaMail.zimbra%40devco.net.

For more options, visit https://groups.google.com/d/optout.

[Nan Liu]

+1 for having an API to access resource attributes in the catalog. This would solve the entire class of issues where we need configuration data for types/providers.

Here's an example:

https://github.com/puppetlabs/puppetlabs-splunk/pull/14/files

Without this information, the splunk config files must exist where the the path is hardcoded in the resource type.

Thanks,

Nan 

[Kylo Ginsberg]

On Wed, Mar 2, 2016 at 2:50 AM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

Hi all,

This is a very interesting conversation and I'm monitoring to see where it ends up.

I see what you did there ;) Thanks for the chuckle.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXjF%3DTh1o5vJBU9hW%3DHhmxNDcDf5CHAkc-TUM9HF4hxDA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[Kylo Ginsberg]

Okay fair enough, I wouldn't want the perfect to be the enemy of the good and so forth.

Realistically while the type/provider API definitely needs some real love, that will take some more serious time and thought, and meanwhile I wouldn't want to block a more modest enhancement.

This thread is making me think we should massage the ideas here into a small RFC around how puppet core could add a generic 'validate' metaparameter (or something). So ... while I hesitate to make any time commitments and the next month is crazy, I'll try to carve out some time to play with this and work up an RFC.

 

The meta parameter though has value - I see this to a large extend as user
serviceable.  I get a module from the forge that has some provider, for me
a 'responds well' check might be < 10ms for others it might be < 100ms and
for others it might not even be a time based expression..  And the parameters
that makes up what is good - here I used response time - are actually very varied
so no doubt some space for getting this in a magical way for now a meta param
seems a easy thing to reason about for now while I look at how this works and
how feasible it is.

Another approach might be to introduce a more formal observer pattern into the
system, I've toyed with this a few times and there's a interesting example over
at https://forge.puppetlabs.com/ryanuber/tell though it's not what I want but
as a thing to look at and think about it's quite interesting.  My thoughts are
not really well formed around it, but I did built a network wide observer for
puppet events and could do pretty crazy things with it, not sure they were
good ideas though.

That's a nice little module, thanks for that. And yeah, I started a spike a few months back for an 'event streamer' that would allow streaming events out of catalog application as they happened to some specified network sink. It foundered on availability of (my) time, but it still seems like a nifty idea :)

Agreed on monitoring being about data somewhat generically. And good point just above about leveraging the plethora of extant nagios checks. I wasn't thinking about all that prior art (well, "art" may be generous). That's an important consideration.

 

> Wrt catalog format (or perhaps report format?) it would be interesting to
> explore whether we could adapt the format to facilitate such integrations.
> Currently both catalog format and report format are more serializations of
> internal data structures than they are designed as external integration
> points. I'd have to noodle on this some more, but it's definitely a course
> worth exploring.

Yeah, I've done nasty little things to extract information from catalogs and
they all broke recently with 4 :P Public APIs around interacting with it
would be good.  A library to do so would be fantastic

+1 for having an API to access resource attributes in the catalog. This would solve the entire class of issues where we need configuration data for types/providers.

Here's an example:

https://github.com/puppetlabs/puppetlabs-splunk/pull/14/files

Without this information, the splunk config files must exist where the the path is hardcoded in the resource type.

Thanks for that example, Nan. I know some folks have spitballed ideas on how to specify a legit, supportable public catalog API, but my (at least) thinking on that is still super hand-wavy. That would make another great RFC but I feel less prepared to craft that one :)

World enough and time ....

Kylo

Thanks,

Nan 

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CACqVBqB5KfVe6na9SDRb_U8UJcfz_67-vFo4B%2BoOzxxDMhoXaQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Erik Dalén]

On Fri, 5 Feb 2016 at 00:35 Kylo Ginsberg <ky...@puppetlabs.com> wrote:

On Wed, Feb 3, 2016 at 7:47 AM, R.I.Pienaar <r...@devco.net> wrote:

hello,

I would like to add a metaparameter - which I think is easy now via
Type.newmetaparam.

We haven't been thinking of metaparameters as a general purpose extension point. This came up once before that I know of, about a year ago, and there's a little discussion of this in https://tickets.puppetlabs.com/browse/PUP-4281. The conclusion we reached at the time was, more or less, to explore whether the desired change could be accomplished with a puppet function and/or a change to core puppet.

 

The thing that I can't seem to find any example of though is how to
make this metaparameter do something on the nodes for all providers
or all types.

Imagine there's a metaparam that might describe how to test a resource
works, something like:

   service{"httpd": validate => "check_http --port 80 -H localhost"}

I'd then want to have some code that would be run on the agent nodes
for any resource that has this param set.

I don't think something like this exists per se, but 'validate' might be one such example of something worth adding to core puppet. Fwiw, one resource-specific example added not too long ago is the file type's validate_cmd: 

https://docs.puppetlabs.com/puppet/latest/reference/type.html#file-attribute-validate_cmd.

Just to clarify, validate_cmd works differently in that it validates the new contents before replacing the file.

I guess the generic validate metaparameter would validate the resource after it has been synced.

[R.I.Pienaar]

----- Original Message -----
> From: "Erik Dalén" <erik.gus...@gmail.com>
> To: "puppet-dev" <puppe...@googlegroups.com>
> Sent: Thursday, 3 March, 2016 09:43:30
> Subject: Re: [Puppet-dev] metaparam question

yes indeed, subtle but important. thanks

[Trevor Vaughan]

It really seems like we want the logical extension of the Exec checks on all resources with additional validation.

service { 'httpd':

  ensure => 'running',

  onlyif => '/usr/bin/probe_remote_mysql', # or 'unless'

  validate => "curl -k http://${::fqdn}/service_check.html",

  # Ok, magical asynchronous fact update here with an ACL (ponies and unicorns area....)

  fact_update => {

    'my_unique_name' => 'w00t',

    # ACLs would need to be processed server side. Let's not give the nodes too much power here...

    # This could, perhaps, be a server setting.

    'acl' => {

      'allow' => $::domain,

      'deny' => 'ALL'

    }

  }

  class { 'thing_that_needs_apache':

    onlyif => fact_fetch('my_unique_name', 'w00t') # fact_fetch(<name>, <expected value/regex/math/whatever>)

  }

With the async updating of facts, and individual ACL restrictions, you get some really neat capabilities in terms of fine grained process control.

Thinking about this further, you'll probably need caching proxies for the fact updates though.

Maybe Puppet 8 :-D.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/1062762368.96001.1456995522124.JavaMail.zimbra%40devco.net.

For more options, visit https://groups.google.com/d/optout.

--

[R.I.Pienaar]

Late to follow up on this but I had some time to work on this again,

service{"http":
ensure => "running",
post_validate => "/usr/lib64/nagios/plugins/check_http -H localhost"
}

This works with the code below and I included a basic thingy that tells
sensu about this

I had to hack around a bit more than simply adding a metaparam since I had
to wait for puppet to completely finish with the resource and then do a
final check after it's done

https://github.com/ripienaar/puppet/commit/a6c659cc3fb3555632bfe94df1411878a04fdc51

This is really nice, it both makes my Puppet runs a bit better and allows
me to get instant visibility of failures - and configurable notifications
based on sensu. It works with any kind of resource.

Ideally the sensu notifier would be something a module can provide and you
could register multiple event handlers but Henrik says the bindings system
isn't initialized on the agent side at present and I didn't have time to
hack around with that.

[David Lutterkort]

On Sun, May 1, 2016 at 11:33 AM, R.I.Pienaar <r...@devco.net> wrote:

Late to follow up on this but I had some time to work on this again,

No matter how late you are, there's always somebody later ;)

For the whole cross-node dependency business, we built a sqlhealthcheck provider; it's a provider for a Sql service/capability type, but has the advantage that it is not tied to the local db service, i.e. geared towards being used by both on the DB node itself and nodes that need to talk to the DB.

We did discuss extending this to make it easy to incorporate nagios checks, but that has suffered from ENOTIME so far.

David

[R.I.Pienaar]

----- Original Message -----
> From: "David Lutterkort" <lut...@puppet.com>
> To: "puppet-dev" <puppe...@googlegroups.com>

> Sent: Friday, 6 May, 2016 21:30:18
> Subject: Re: [Puppet-dev] metaparam question

> On Sun, May 1, 2016 at 11:33 AM, R.I.Pienaar <r...@devco.net> wrote:
>
>> Late to follow up on this but I had some time to work on this again,
>>
>
> No matter how late you are, there's always somebody later ;)
>
> For the whole cross-node dependency business, we built a sqlhealthcheck

> <https://github.com/puppetlabs/puppetlabs-sqlhealthcheck> provider; it's a

> provider for a Sql service/capability type, but has the advantage that it
> is not tied to the local db service, i.e. geared towards being used by both
> on the DB node itself and nodes that need to talk to the DB.
>
> We did discuss extending this to make it easy to incorporate nagios checks,
> but that has suffered from ENOTIME so far.

so my plan with this is 2 fold, it lets people make puppet's generate state check
a bit better for any resource quite cheaply

but also once this is in the catalog I can write a sensu/nagios/graphite/tool of
choice thing that loads the catalog and checks everything regularly and send off
results to whatever they like.

I think I'll have to figure out how to autorequire the plugin being called in
case Puppet manages it but I think this concept is pretty powerful - and indeed
should work with your cross node stuff too.

Those checks you have in AO are good but the barrier to entry is quite high on
making those I felt

[David Lutterkort]

On Fri, May 6, 2016 at 1:02 PM, R.I.Pienaar <r...@devco.net> wrote:

Those checks you have in AO are good but the barrier to entry is quite high on
making those I felt

Understood, and not trying to argue, but because I am curious:  what's the barrier to entry that keeps you from using that mechanism ? Not sure if you know, but you can use service resources without writing out full applications or an environment catalog, i.e. you can do stuff like

define db { ... }

Db produces Sql { .. }

define web { ... }

node db_node {

  db { one: export => Sql[one] }
}

node web_node {

  web { one: require => Sql[one] }
}

which does little more than putting that sql into web_node's catalog, which would then trigger running the check as a gate for applying web.

Sensu/Nagios can then fish the Sql out of both db_node's and web_node's catalog and do their own checking.

I am mostly curious about the barrier to entry to see if there is anything we can do to lower it.

David

[R.I.Pienaar]

----- Original Message -----
> From: "David Lutterkort" <lut...@puppet.com>
> To: "puppet-dev" <puppe...@googlegroups.com>
> Sent: Saturday, 7 May, 2016 01:32:45
> Subject: Re: [Puppet-dev] metaparam question

hmm, so that seems very verbose and how to implement the SQL check is also
quite a task.

The verbosity is because you're modelling relationships across nodes of course
I do not want to do that - these specific checks are not for cross node checks
and as in recent emails I will not use puppetserver as long as operability
features are a paid feature, so those are not what I am after.

My needs for checks are literally to augment the:

service{"x": ensure => running} this will do 'service x status' equiv, I want it
to do additional checks and I want those checks to adhere to existing "standards"
which might include metrics and such.

Do you have an example of using the new check framework for this purpose?

[Gareth Rushgrove]

First up, this is great. A few minor comments:

* You explicitly don't trigger the script if it's a noop run or the
resource failed. That obvious matches your specific usecase but I'm
wondering if it matches all?
* Similar vein, the name post_validate assumes the usage is to
'validate' the resource. Given the power of this I can imagine other
uses too, it might be nicer to have a more generic name?

Gareth

> This works with the code below and I included a basic thingy that tells
> sensu about this
>
> I had to hack around a bit more than simply adding a metaparam since I had
> to wait for puppet to completely finish with the resource and then do a
> final check after it's done
>
> https://github.com/ripienaar/puppet/commit/a6c659cc3fb3555632bfe94df1411878a04fdc51
>
> This is really nice, it both makes my Puppet runs a bit better and allows
> me to get instant visibility of failures - and configurable notifications
> based on sensu. It works with any kind of resource.
>
> Ideally the sensu notifier would be something a module can provide and you
> could register multiple event handlers but Henrik says the bindings system
> isn't initialized on the agent side at present and I didn't have time to
> hack around with that.
>

> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/157346444.508343.1462127599948.JavaMail.zimbra%40devco.net.

> For more options, visit https://groups.google.com/d/optout.



--

Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[R.I.Pienaar]

yeah, as this is just playing around with the idea that choice seemed
the right one without dwelling on it too much - I am not actually sure
what would be right.

Exec for example has all the unless/onlyif stuff and I dont think those
are ran during a noop either - I think it assumes a value?

Thinking about it, not running it during noop is the only thing thats
safe, people might not do read only things there - though they really
should. And if its the first run the prereq might not be there?

> * Similar vein, the name post_validate assumes the usage is to
> 'validate' the resource. Given the power of this I can imagine other
> uses too, it might be nicer to have a more generic name?

validate or verify it works yes, what other uses did you have in mind?

[Trevor Vaughan]

Skipping actions during a noop is the only correct solution.

Any time you check a cross-daemon/node state it is relatively likely that you're going to be changing something on the network and/or the local system.

In many cases, this will be completely innocuous, but in others, it could add messages to a queue, or some other nonsense.

I suppose that, if you wanted to get fancy, you could have yet another metaparameter which was an array that would let you specify which checks could be run in noop mode defaulting to an empty array.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/933229095.334955.1462798703903.JavaMail.zimbra%40devco.net.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

[Trevor Vaughan]

On a similar vein to this thread....many years ago (six?) I wanted to introduce a semaphore-based logic construct into the language itself.

I think, with Puppet 4, it might actually be possible!

Technically, as pointed out by one of the team here, we probably want Phasers, but I didn't know about that construct at the time :-D.

I was going to try to add support to Puppet Server itself but my Clojure-fu is weak.

The idea was basically:

if amisafe("key_name") > 4 {

  class { 'do puppet stuff': ... }

   

  letsbesafe('key_name', 0)

}

else {

  class { 'do other optional puppet stuff': ... }

}

I still think it would be interesting to have and, of course, it would require a central service to manage these things but it could theoretically be decentralized (consul, etcd, whatever...) and I think it fits in the general idea of this conversation.

Thanks,

Trevor

[Gareth Rushgrove]

My thinking here was that failures are not bad, but informational, in
the same way as failures of a unit test are not bad. The script run
could be passed the success and the noop and do with those as it will,
which avoids the issue of a messier interface.

>> * Similar vein, the name post_validate assumes the usage is to
>> 'validate' the resource. Given the power of this I can imagine other
>> uses too, it might be nicer to have a more generic name?
>
> validate or verify it works yes, what other uses did you have in mind?

I guess reporting or metrics or some sort of registration mechanism.
Their are other ways of doing those I realise, but as a light-weight
integration point this struck me as interesting. I'd probably stick
with validate or verify (ie. without the post_) though if it came to
that.

I could see modules shipping with a verify directory containing
scripts to run to confirm a working system too.

G

>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/933229095.334955.1462798703903.JavaMail.zimbra%40devco.net.

[R.I.Pienaar]

yeah, so I stuck the post in there because this is different from say
the existing validate_cmd option on the file resource that puts the file in a
temporary location, checks it and then if that pass it puts it in the final
destination. The 'post' is to communicate that this happens after puppet
had done all it will do to make the resource in the desired state. File also
have a validate_replacement that might be useful here.

Trying to make it clear a possible what this does, anyway, terrible at naming
things as always :)

Chef had a minitest integration that let you stick some validation logic in
a cookbook to be run post resources and that was very helpful so I've wanted
something like this since I first saw that probably 6 o 7 years ago.

The idea that we can run these things out of band by monitoring systems at
a completely different time cadence than puppet runs which are very heavy is
interesting because it takes us a lot closer to zero config monitoring state
than ever before and CM systems are quite uniquely positioned to do that,
be it cross node or single node or whatever.

With metrics and a feedback mechanism back to orchastration tools this is
can become a very powerful tool for doing dynamic node placement or auto
scaling etc.

I am not sure but I hope we can treat the catalog as a stable integration
point to puppet so that external query tools to extract this kind of
information can be stable, if not I suppose this would be worth discussing
too

[Thomas Müller]

Am Dienstag, 10. Mai 2016 10:52:55 UTC+2 schrieb Gareth Rushgrove:

>
> validate or verify it works yes, what other uses did you have in mind?

I guess reporting or metrics or some sort of registration mechanism.
Their are other ways of doing those I realise, but as a light-weight
integration point this struck me as interesting. I'd probably stick
with validate or verify (ie. without the post_) though if it came to
that.

I could see modules shipping with a verify directory containing
scripts to run to confirm a working system too.

we started deploy serverspec tests with every profile class. The tests are run by puppet.conf's postrun_command on dev systems - other systems run it daily by a cronjob. So we can get fast feedback from real sytems if configuration is as expected.

- Thomas

[Felix Frank]

Hi,

this is me assuming the role of the weird secluded neighbor who sometimes yells incoherent ramblings over the fence. Although this is pertinent:

For onlyif and friends, there is already https://tickets.puppetlabs.com/browse/PUP-1125, where John and I have already done some thoughtwork.

Now two years have passed, and I'm a little overwhelmed by the prospect of building yet another complex feature (intense fretting going on over here). Don't hold your breath I guess? But if anyone wants to grab it, I'll be glad to offer whatever limited guidance I can muster.

Cheers,
Felix

On 03/03/2016 02:24 PM, Trevor Vaughan wrote:

It really seems like we want the logical extension of the Exec checks on all resources with additional validation.

service { 'httpd':

  ensure => 'running',

  onlyif => '/usr/bin/probe_remote_mysql', # or 'unless'

  validate => "curl -k http://${::fqdn}/service_check.html",

  # Ok, magical asynchronous fact update here with an ACL (ponies and unicorns area....)

  fact_update => {

    'my_unique_name' => 'w00t',

    # ACLs would need to be processed server side. Let's not give the nodes too much power here...

    # This could, perhaps, be a server setting.

    'acl' => {

      'allow' => $::domain,

      'deny' => 'ALL'

    }

  }

  class { 'thing_that_needs_apache':

    onlyif => fact_fetch('my_unique_name', 'w00t') # fact_fetch(<name>, <expected value/regex/math/whatever>)

  }

With the async updating of facts, and individual ACL restrictions, you get some really neat capabilities in terms of fine grained process control.

Thinking about this further, you'll probably need caching proxies for the fact updates though.

Maybe Puppet 8 :-D.

Thanks,

Trevor

<snip>