Ephemeral Resources

[Trevor Vaughan]

I was looking through the puppetlabs-concat module as well as some of my modules that have aggregation functionality and I realized that the catalog really shouldn't be sending the intermediary resources to the client.

While I'm not positive, the ability to mark resources as ephemeral *might* go a long way toward solving the 10k+ resources issue.

The idea is that only the final 'aggregation' resource would be present in the catalog and all other resources would be removed right after they are processed.

The main issue that I see with this is in debugging. However, I think this is pretty much the case with concat presently so I don't know that it changes much.

Thoughts?

Thanks,

Trevor

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[John Bollinger]

On Wednesday, December 7, 2016 at 7:54:44 AM UTC-6, Trevor Vaughan wrote:

I was looking through the puppetlabs-concat module as well as some of my modules that have aggregation functionality and I realized that the catalog really shouldn't be sending the intermediary resources to the client.

Back in the day, of course, concat fragments were assembled on the client, so they needed to be in the catalog.  Imagine my surprise when I discovered, just a few minutes ago, that Concat doesn't work like that any more.  With that being the case, I'm inclined to agree that the fragment resources should not be included in catalogs delivered to clients.  Any useful information they convey is redundant, so their net contribution is only noise and bloat.

While I'm not positive, the ability to mark resources as ephemeral *might* go a long way toward solving the 10k+ resources issue.

The idea is that only the final 'aggregation' resource would be present in the catalog and all other resources would be removed right after they are processed.

The main issue that I see with this is in debugging. However, I think this is pretty much the case with concat presently so I don't know that it changes much.

Thoughts?

I'm inclined to think that using resources to represent concat fragments is no longer appropriate at all.  The module uses resources because Puppet provides no other mechanism that can be bent to its purposes, but this is really pushing up against the boundary of what a resource is (/ should be / has been).  If Puppet needs to provide a repository for computed general node data accessible to the catalog compiler, then maybe it would be better to stop trying to shoehorn that data into resource-shaped cement shoes.

Don't get me wrong: I like the simplicity of having resources as a consistent unit of data exchange, and I think that model served well for a long time.  But for some time now people have been stretching (and sometimes passing) the borders of that model.  Rather than continuing to weaken the meaning of "resource", I would prefer to see Puppet's model for data exchange extended or even replaced so as to accommodate data that aren't packaged as resources in the first place.

To some extent this is a question of semantics.  After all, in what sense would or should an ephemeral resource actually be a resource in anything but name?  But really that's the point.  We name and categorize things to help us reason about them, program with and for them, etc..  The more we weaken our concepts, such as "resource", the less useful they become.


John

[Erik Dalén]

On Wed, 7 Dec 2016 at 14:54 Trevor Vaughan <tvau...@onyxpoint.com> wrote:

I was looking through the puppetlabs-concat module as well as some of my modules that have aggregation functionality and I realized that the catalog really shouldn't be sending the intermediary resources to the client.

While I'm not positive, the ability to mark resources as ephemeral *might* go a long way toward solving the 10k+ resources issue.

The idea is that only the final 'aggregation' resource would be present in the catalog and all other resources would be removed right after they are processed.

The main issue that I see with this is in debugging. However, I think this is pretty much the case with concat presently so I don't know that it changes much.

Thoughts?

Another issue would be resource relationships on ephemeral resources. They might not be that common on concat_fragment, but on anchor it is pretty common to use them.

Also I think some benchmarking how much they actually slow down things would be good. And checking if there is anything that can be optimized when applying resources that do nothing.

/Erik

[Trevor Vaughan]

So, in theory, ephemeral resources would only have relationships on items that need them for data and/or building purposes.

We already have this problem and, as more items start getting managed, we're going to run into more issues with large catalog compile and application death.

I guess they could be called shared_data objects or something like that. They would be declared the same way as resources but perhaps a metaparameter could be added that marks them as data.

I was kind of thinking:

fragment { 'part1':

  item_type => 'ephemerai' (or data, or something),

  ....

}

This would mark it to be placed in the system in a location that will never make it into the live catalog because it's irrelevant to the application of the catalog itself.

Actually, looking at the catalogs, there may be quite a bit of information that is irrelevant to the clients in general and could be split into a reporting/metadata bundle in case there is an error. If there's no issue, you don't need any of this information at all. If you run in debug mode (and you know that at instantation time, you need all of it).

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAAAzDLfUzXQ-ZxzhZF2xT3bPFpLUbGMKaX%2BGnWAVvFurkGqs7A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Ryan Whitehurst]

On Thu, Dec 8, 2016 at 7:28 AM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:
> So, in theory, ephemeral resources would only have relationships on items
> that need them for data and/or building purposes.
>
> We already have this problem and, as more items start getting managed, we're
> going to run into more issues with large catalog compile and application
> death.
>
> I guess they could be called shared_data objects or something like that.
> They would be declared the same way as resources but perhaps a metaparameter
> could be added that marks them as data.
>
> I was kind of thinking:
>
> fragment { 'part1':
> item_type => 'ephemerai' (or data, or something),
> ....
> }
>
> This would mark it to be placed in the system in a location that will never
> make it into the live catalog because it's irrelevant to the application of
> the catalog itself.
>
> Actually, looking at the catalogs, there may be quite a bit of information
> that is irrelevant to the clients in general and could be split into a
> reporting/metadata bundle in case there is an error. If there's no issue,
> you don't need any of this information at all. If you run in debug mode (and
> you know that at instantation time, you need all of it).

It sounds like what we're discussing here might be more effectively
enabled using something like the Builder pattern:

https://en.wikipedia.org/wiki/Builder_pattern

As I understand, the main use case we're discussing is essentially "I
want to construct a resource by building it up from component pieces".
Really, what we want here is to be able to define components or
properties of a resource separately, which the type can later use to
construct the actual resource that will end up in the catalog. This
might look something like:

concat { '/etc/foo.conf':
ensure => present,
# ...
}

Concat['/etc/foo.conf'].add_fragment({
content => '[foo]',
priority => 0,
}

Concat['/etc/foo.conf'].add_fragment({
content => 'bar = true',
priority => 20,
})

Concat['/etc/foo.conf'].add_fragment({
content => 'baz = "bat"',
priority => 10,

>> email to puppet-dev+...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-dev/CAAAzDLfUzXQ-ZxzhZF2xT3bPFpLUbGMKaX%2BGnWAVvFurkGqs7A%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699 x788
>
> -- This account not approved for unencrypted proprietary information --
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to puppet-dev+...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVxgMXwnD_rKxRVh5mjwUPkCx0TVg%2Brc4y%3DLVjEUR08WQ%40mail.gmail.com.

[Trevor Vaughan]

That would work, but I'm guessing that we're going to want to be able to access this information across resources.

We can presently do this in the catalog as long as you time it properly.

Also, doing anything that looks more "programmish" is going to scare new users.

But +1, that's basically the idea.

Thanks,

Trevor

>> email to puppet-dev+unsubscribe@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-dev/CAAAzDLfUzXQ-ZxzhZF2xT3bPFpLUbGMKaX%2BGnWAVvFurkGqs7A%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699 x788
>
> -- This account not approved for unencrypted proprietary information --
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to puppet-dev+unsubscribe@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVxgMXwnD_rKxRVh5mjwUPkCx0TVg%2Brc4y%3DLVjEUR08WQ%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAHTHiAE6WauMyaLY6V6sJhS%2ByhKNZBF-2Bc2HMNbk0SWpH_Oiw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Eric Sorenson]

On Dec 8, 2016, at 11:50 AM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

That would work, but I'm guessing that we're going to want to be able to access this information across resources.

We can presently do this in the catalog as long as you time it properly.

A little bit of a tangent to your original question but do you have an example of doing this? (I'm assuming you mean an apply-time type/provider accessing values of resources other than the one currently being applied)

Also, doing anything that looks more "programmish" is going to scare new users.

But +1, that's basically the idea.

Thanks,

Trevor

Eric Sorenson - eric.s...@puppet.com 
director of product, puppet ecosystem

[Trevor Vaughan]

I do mean that.

I have a lot of close examples, the most clear (if you want to call it that) being iptables_optimize in https://github.com/simp/pupmod-simp-iptables.

It's basically the same pattern as puppetlabs/concat.

For even more fun, I *may* be altering the catalog at run time sometimes using the initialization function in the provider to detect if the changed needs to happen. Not too often though, debugging is hell.

Once you dig into the Ruby, you can access the entire catalog and this is a heck of a tool if used carefully.

And, no, I don't recommend that anyone do this without extreme care.

I guess a more concrete example is when I was discussing micro-catalogs with Adrien (never did 100% figure that out) and I needed to dynamically generate a File resource and an incron job on the system based on the ordered substate of the catalog.

As long as I shoved that bit in the last stage to be applied, the entire catalog, and all values, were present and I could pull out the micro-catalog snippet.

For power users and/or developers this is, by far, the best thing about Puppet being a compiled language.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/0975926B-4018-487D-828B-5DDF728774E9%40puppet.com.

For more options, visit https://groups.google.com/d/optout.

[John Bollinger]

On Thursday, December 8, 2016 at 9:29:00 AM UTC-6, Trevor Vaughan wrote:

So, in theory, ephemeral resources would only have relationships on items that need them for data and/or building purposes.

Meaning that ephemeral resources / shared_data will not have relationships at all, then?  Because relationships have no significance during catalog building; they affect only catalog application.  If ephemeral resources are not to be delivered to clients, then relationships among them (as currently construed) cannot be meaningful.

Alternatively, if indeed a complete break is made between resources and shared_data, then perhaps there will be scope for relationships among shared_data objects to affect catalog building.

John

[Trevor Vaughan]

This makes sense but it *is* compile order dependent though. Therefore care has to be taken in ensuring that all shared_data are available prior to full resource realization in the catalog.

This is why the ordering relationship matters, not for any client side reason.

Basically, it would mean deferred realization of catalog resources until the end of catalog processing.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/51388e3d-9b1e-47c3-b233-78a2678782b4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ryan Whitehurst]

On Fri, Dec 9, 2016 at 5:45 PM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:
> This makes sense but it *is* compile order dependent though. Therefore care
> has to be taken in ensuring that all shared_data are available prior to full
> resource realization in the catalog.
>
> This is why the ordering relationship matters, not for any client side
> reason.
>
> Basically, it would mean deferred realization of catalog resources until the
> end of catalog processing.

You wouldn't want or need to use explicit resource ordering
relationships for that, though, since those mean something different.
Presumably, whatever shared_data/ephemeral resources/etc are available
would be determined by the type registering them in some way, so you
could automatically define these compile-time dependencies similar to
how, e.g., file resources automatically have a resource dependency on
the file resource for the directory they are contained in.

> Trevor
>
> On Fri, Dec 9, 2016 at 1:13 PM, John Bollinger <john.bo...@stjude.org>
> wrote:
>>
>>
>>
>> On Thursday, December 8, 2016 at 9:29:00 AM UTC-6, Trevor Vaughan wrote:
>>>
>>> So, in theory, ephemeral resources would only have relationships on items
>>> that need them for data and/or building purposes.
>>
>>
>>
>> Meaning that ephemeral resources / shared_data will not have relationships
>> at all, then? Because relationships have no significance during catalog
>> building; they affect only catalog application. If ephemeral resources are
>> not to be delivered to clients, then relationships among them (as currently
>> construed) cannot be meaningful.
>>
>> Alternatively, if indeed a complete break is made between resources and
>> shared_data, then perhaps there will be scope for relationships among
>> shared_data objects to affect catalog building.
>>
>>
>> John
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to puppet-dev+...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-dev/51388e3d-9b1e-47c3-b233-78a2678782b4%40googlegroups.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699 x788
>
> -- This account not approved for unencrypted proprietary information --
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to puppet-dev+...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoWL-TDxv0nS1zG%2BZhJv656AMckvP7Qt6sE6MBcS78ObRw%40mail.gmail.com.