Ephemeral resources

[Trevor Vaughan]

How difficult would it be to create a third type of resource which is an 'ephemeral resource' whose only purpose is data collection on a host to be used by some other collector?

These items would not be part of the catalog or added to the graph but would instead just hang around for reference during compilation.

This would fix the catalog explosion issue when you start doing exported resources based on large numbers of things and/or things like firewall rules and copious file_line resources.

Basically, a 'data' -> 'collector' pattern where you can optimize...well...everything into a MUCH smaller catalog that is sent to the client for processing.

Thanks,

Trevor

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[John Bollinger]

On Monday, April 16, 2018 at 10:38:55 AM UTC-5, Trevor Vaughan wrote:

How difficult would it be to create a third type of resource which is an 'ephemeral resource' whose only purpose is data collection on a host to be used by some other collector?

These items would not be part of the catalog or added to the graph but would instead just hang around for reference during compilation.

This would fix the catalog explosion issue when you start doing exported resources based on large numbers of things and/or things like firewall rules and copious file_line resources.

Basically, a 'data' -> 'collector' pattern where you can optimize...well...everything into a MUCH smaller catalog that is sent to the client for processing.

I can't speak to how difficult such a thing might be to implement, but inasmuch as I take the idea to be that the ephemeral resources could serve as input data for constructing a smaller number of ordinary resources, I suggest two additional approaches that could be considered:

 1. Provide a convenient mechanism for directing (ordinary) collected or declared resources to a different aggregate than the target node's catalog.  One could use such a mechanism to divert resources of any type away from the catalog, so no new flavor of resource would be required.  This supposes that the data associated with such resources would still be available during catalog building, to inform declaration of resources that do go into the catalog.

 2. If the need is for a repository of dynamic node data, and especially if those data are expected to often not correlate directly to target-node resources, then sit down and design that thing from the ground up.


John

[Trevor Vaughan]

In terms of #2, there's actually a ticket that I put in a while ago for a shared data cache.

But, I recently realized that the common pattern is to literally map every line in a file as a separate resource. We've had the same issue with firewall rules, etc... for quite some time. So, the most "user obvious" thing to do is to let everyone keep throwing resources everywhere but be able to mark them explicitly as being 'lightweight' or 'ephemeral' or somesuch.

This was really solidified for me when I started passing around hashes to be able to collapse my augeas command into a single run. I thought about it and realized that's a LOT of work to get around limitations in the abstraction realization.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/6d665f35-9a86-4f9a-a256-a4fc443811cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Henrik Lindberg]

On 16/04/18 17:38, Trevor Vaughan wrote:
> How difficult would it be to create a third type of resource which is an
> 'ephemeral resource' whose only purpose is data collection on a host to
> be used by some other collector?
>
> These items would not be part of the catalog or added to the graph but
> would instead just hang around for reference during compilation.
>
> This would fix the catalog explosion issue when you start doing exported
> resources based on large numbers of things and/or things like firewall
> rules and copious file_line resources.
>
> Basically, a 'data' -> 'collector' pattern where you can
> optimize...well...everything into a MUCH smaller catalog that is sent to
> the client for processing.
>

Sounds a bit like the existing virtual resources, but with a better
collection mechanism. Would not be too difficult to write a function
that takes a data type predicate to match against virtual resources
data type predicate) and then calling a lambda with each.

Virtual resources do not end up in the catalog unless they are realized.
With the function I imagined, you would select virtual resources and
then do whatever you want in the lambda.

The function should probably return an Iterator over the resources. That
can then be iterated with each, map, or reduce.

The issue then is when to call that function - you want it at the very
end which we do not have a mechanism for.

The virtual collector could be modified to accept a lambda - since
collection runs late it would be at the right time. This is a much
bigger change naturally as it changes the language.

- henrik


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Trevor Vaughan]

I thought there were dangers with Virtual Resource being accidentally realized sometimes?

We found a super-hacky way to call functions at the end of a compile which we use in https://github.com/simp/pupmod-simp-compliance_markup/blob/master/manifests/map.pp. It would be *really nice* if there were an actual 'hook in here, no resources will be processed after this point' hook.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/pb32pa%24d3q%241%40blaine.gmane.org.

For more options, visit https://groups.google.com/d/optout.

[Henrik Lindberg]

On 16/04/18 23:13, Trevor Vaughan wrote:
> I thought there were dangers with Virtual Resource being accidentally
> realized sometimes?
>

Not without users doing something bad (like realizing all) - puppet
itself does not realize virtual resources willy-nilly.

> We found a super-hacky way to call functions at the end of a compile
> which we use in
> https://github.com/simp/pupmod-simp-compliance_markup/blob/master/manifests/map.pp.
> It would be *really nice* if there were an actual 'hook in here, no
> resources will be processed after this point' hook.
>

What if the logic called "after all resources processed" creates new or
modifies existing resources? Also, should it run after
collection/overrides or before?

- henrik

> send an email to puppet-dev+...@googlegroups.com
> <mailto:puppet-dev%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-dev/pb32pa%24d3q%241%40blaine.gmane.org

> <https://groups.google.com/d/msgid/puppet-dev/pb32pa%24d3q%241%40blaine.gmane.org>.

>
> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
>
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699 x788
>
> -- This account not approved for unencrypted proprietary information --
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to puppet-dev+...@googlegroups.com
> <mailto:puppet-dev+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUUaCqmHhKjUWiFRyqFF3%2Bc5BpgLEHe-faYZxcyUutcGA%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUUaCqmHhKjUWiFRyqFF3%2Bc5BpgLEHe-faYZxcyUutcGA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.


--

[Trevor Vaughan]

In terms of the 'after all resources processed' I think this is one of those cases where you present the capability because there is need and caveat it with the fact that you may, or may not, have access to resources that have been modified after the fact.

In theory, with a hook like this, you could also have a registration system that allows the extension to know how many other extensions have been loaded and whether or not they need to modify the catalog.

In my case, we *do* modify the catalog but it's just to create a file on the client in the `vardir` directory under our own namespace. But we needed to be able to see all of the *regular* resources, as created, so that we could audit them and report on the state.

Something like the following might work:

post_process_register('do_my_work', :creates_resources => true, :removes_resources => false)

lib/functions/do_my_work.rb would then do whatever it needs to do in Ruby space, could check for other registered hooks, etc...

Even without the 'creates/removes' part, it would be nice to have an official hook with the warning of "here there be dragons" instead of having to depend on compiler ordering or monkey patching.

This is also a case where the ephemeral resources would come in handy since this post-processing function could use the data collected in them and/or be the place where they are actually removed from the catalog.

Hrm...thinking about this slightly more, a pre-registration hook might also be nice that could do things prior to catalog compilation. It's not as useful but it could be handy for auto-hooking r10k and/or refreshing state based on something changing without needing to resort to wrapper scripts.

Thanks,

Trevor

Trevor

    send an email to puppet-dev+unsubscribe@googlegroups.com

    <mailto:puppet-dev%2Bunsubscrib...@googlegroups.com>.

    To view this discussion on the web visit
    https://groups.google.com/d/msgid/puppet-dev/pb32pa%24d3q%241%40blaine.gmane.org
    <https://groups.google.com/d/msgid/puppet-dev/pb32pa%24d3q%241%40blaine.gmane.org>.

    For more options, visit https://groups.google.com/d/optout
    <https://groups.google.com/d/optout>.




--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com <mailto:puppet-dev+unsubscribe@googlegroups.com>.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUUaCqmHhKjUWiFRyqFF3%2Bc5BpgLEHe-faYZxcyUutcGA%40mail.gmail.com <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUUaCqmHhKjUWiFRyqFF3%2Bc5BpgLEHe-faYZxcyUutcGA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/pb42so%24oqc%241%40blaine.gmane.org.

For more options, visit https://groups.google.com/d/optout.

--