Exported resource realized by resource collector, bug?

[Julio Guevara]

So I'm working with puppet 5.5.1 and I have encountered what I think is a bug.

So on my site.pp i have the following line for ordering my firewall rules:

site.pp

---------

if $::kernel == 'Linux' {

# Make sure every firewall rule not pre or post is created in the middle

Firewall <| tag != 'pre' and tag != 'post' |> {

before +> Class['profiles::fw_rules::post'],

require +> Class['profiles::fw_rules::pre'],

}

}

Then on another class I'm doing the following (This is a class for a postgresql client):

@@firewall { "222 tcp:5432 pgpool from ${::ipaddress}/32" :

action => 'accept',

source => "${::ipaddress}/32",

dport => '5432',

proto => 'tcp',

tag => [ $tag, 'postgresql_client' ],

}

On the postgresql servers I do:

# Grab all firewall rules created for this cluster

Firewall <<| tag == $tag and tag == 'postgresql_client' |>>

I would expect that the rule to only appear on the postgresql server, but what ends up happening is that the rule is realized on both the server and the client.

On the documentation for resource collector (https://puppet.com/docs/puppet/5.5/lang_collectors.html#exported-resource-collectors):

Collectors realize virtual resources, are used in chaining statements, and override resource attributes. 

on that very same page we have the following for exported resource collectors:

An exported resource collector uses a modified syntax that realizes exported resources and imports resources published by other nodes.

Have anybody else found a similar issue?

Thanks

Julio Guevara

[jcbollinger]

I do not recall encountering this issue myself, but I concur that I expect ordinary resource collectors to not collect exported resources, no matter what node declares them.  Although exported resources and exported resource collectors are both syntactically and semantically analogous to virtual resources and ordinary resource collectors, they are and have always been distinct.  The documentation presents no reason to think otherwise, and in fact generally supports that position, though I am unaware of any place where it states it in so many words.

Therefore, if indeed it is the case that an ordinary resource collector is collecting exported resources, then I concur that that constitutes a bug, and a fairly serious one at that.  I suggest you file a bug report.

Do note, however, that in the event that an exported resource is collected for a given target node via an exported resource collector, I do expect that for all intents and purposes it is then treated as an ordinary resource, and in particular, that it ordinary collectors will then affect it with respect to parameter overrides and chaining (in the scope of the current catalog building job).  The thing that ordinary collectors ought not to do with respect to exported resources is import them.

John

[Johan De Wit]

Hi,

check this ticket.  https://tickets.puppetlabs.com/browse/PUP-6723

It is hard to explain, but the 'and' a resource collector does niot behave like the 'boolean and' as we expect this.

Grts

Jo

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/0834fac6-15de-4b7f-9ca1-fbce11b13754%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Henrik Lindberg]

On 2018-09-14 10:10, Johan De Wit wrote:
> Hi,
>
> check this ticket. https://tickets.puppetlabs.com/browse/PUP-6723
>
>
> It is hard to explain, but the 'and' a resource collector does niot
> behave like the 'boolean and' as we expect this.
>

It is not AND that is different; it is the == and != query operators
that must be understood. The == acts as "in" but != is an *exact* match
operator just like in puppet.

So, to find those that have neither 'pre' nor 'post' you would want to
write the following, only you cannot because the NOT operator is not
supported !!!

<| not(tag == 'pre' or tag == 'post') |> # gives error for not()

The expression:

tag != 'pre' and tag != 'post'

works only if tag is not an array (which it almost always is)

Consider this:

@notify { 'a': tag => 'blue' }
@notify { 'b': tag => 'green'}
@notify { 'c': tag => ['blue', 'green'] }
@notify { 'd': tag => 'red' }
@notify { 'e': tag => ['red', 'blue'] }

Notify <| tag != 'blue' and tag != 'green' |> { }

It will realize c, d, and e.

c because the array is != 'blue' and is also != 'green'
d because 'red' != 'blue' and 'red != 'green'
e because again, the array is != to either 'blue' nor 'green'

If you think that you can compare using an array, like this:

<| tag != ['blue', 'green'] |>

then you will also be disappointed because that is also not supported
and will give you an error since array values are not supported in the
query.

The documentation actually points out that the behavior is undefined for
arrays and hashes. That is its way of saying "it is a mess". How it
actually works (since way back) is what I described above.

You need to find a different way of structuring your logic.
I doubt that virtual/exported resources and features around those will
receive much love as there has been talks about dropping both virtual
and exported resources all together (at the moment we have no better
replacement for the features they enable, so this will not happen over
night).

Worth noting is also that using tags extensively is not a very robust
solution since there is no way you can know if your tags are unique or
not or if you will find elements by chance that are tagged with
something from part of a name in some scope. If you must use tags,
use tags that are as unique as possible. (More than one user have been
bitten by this).

Sorry for being the bearer of bad news.
Best,

- henrik

>
> Grts
>
> Jo
>
>
> -----Original message-----
> *From:* jcbollinger <John.Bo...@stJude.org>
> *Sent:* Thursday 13th September 2018 15:50
> *To:* Puppet Users <puppet...@googlegroups.com>
> *Subject:* [Puppet Users] Re: Exported resource realized by resource

> <https://puppet.com/docs/puppet/5.5/lang_collectors.html#exported-resource-collectors>):

> Collectors realize virtual resources, are used in chaining
> statements, and override resource attributes.
>
> on that very same page we have the following for exported
> resource collectors:
> An exported resource collector uses a modified syntax that
> realizes exported resources

> <https://puppet.com/docs/puppet/5.5/lang_exported.html> and

> imports resources published by other nodes.
>
> Have anybody else found a similar issue?
>
>
> I do not recall encountering this issue myself, but I concur that I

> expect ordinary resource collectors to /not/ collect exported

> resources, no matter what node declares them.  Although exported
> resources and exported resource collectors are both syntactically

> and semantically /analogous/ to virtual resources and ordinary

> resource collectors, they are and have always been distinct.  The
> documentation presents no reason to think otherwise, and in fact
> generally supports that position, though I am unaware of any place
> where it states it in so many words.
>
> Therefore, if indeed it is the case that an ordinary resource
> collector is collecting exported resources, then I concur that that
> constitutes a bug, and a fairly serious one at that.  I suggest you
> file a bug report.
>
> Do note, however, that in the event that an exported resource is
> collected for a given target node via an exported resource
> collector, I do expect that for all intents and purposes it is then
> treated as an ordinary resource, and in particular, that it ordinary
> collectors will then affect it with respect to parameter overrides
> and chaining (in the scope of the current catalog building job).
> The thing that ordinary collectors ought not to do with respect to
> exported resources is import them.
>
>
> John
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/0834fac6-15de-4b7f-9ca1-fbce11b13754%40googlegroups.com

> <https://groups.google.com/d/msgid/puppet-users/0834fac6-15de-4b7f-9ca1-fbce11b13754%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/zarafa.5b9b6d09.5d47.268d68ba72245991%40zarafa7.open-future.be
> <https://groups.google.com/d/msgid/puppet-users/zarafa.5b9b6d09.5d47.268d68ba72245991%40zarafa7.open-future.be?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[jcbollinger]

On Friday, September 14, 2018 at 3:10:59 AM UTC-5, Johan De Wit wrote:

Hi,

check this ticket.  https://tickets.puppetlabs.com/browse/PUP-6723

It is hard to explain, but the 'and' a resource collector does niot behave like the 'boolean and' as we expect this.

I don't follow, Jo.  The issue described here is not about the search expression, but the fact that an ordinary resource collector imports exported resources declared by the node.  The presence or details of a search expression is not relevant -- under no circumstances should this happen.

If this behavior is in fact happening, then I'd guess that Puppet is failing to distinguish between (1) resources exported by the target node and not imported, and (2) resources exported by any arbitrary node and imported for the target node.  Alternatively, perhaps Puppet is more simply just failing to distinguish between exported and virtual resources declared by the target node.

John

[Henrik Lindberg]

On 2018-09-14 14:47, jcbollinger wrote:
>
>
> On Friday, September 14, 2018 at 3:10:59 AM UTC-5, Johan De Wit wrote:
>
> Hi,
>
> check this ticket. https://tickets.puppetlabs.com/browse/PUP-6723
> <https://tickets.puppetlabs.com/browse/PUP-6723>
>
>
> It is hard to explain, but the 'and' a resource collector does niot
> behave like the 'boolean and' as we expect this.
>
>
>
> I don't follow, Jo.  The issue described here is not about the search

> expression, but the fact that an /ordinary/ resource collector imports
> /exported/ resources declared by the node.  The presence or details of a

> search expression is not relevant -- under no circumstances should this
> happen.
>
> If this behavior is in fact happening, then I'd guess that Puppet is
> failing to distinguish between (1) resources exported by the target node
> and not imported, and (2) resources exported by any arbitrary node and
> imported for the target node.  Alternatively, perhaps Puppet is more
> simply just failing to distinguish between exported and virtual
> resources declared by the target node.
>

The <| |> operator finds ALL resources (regular, virtual and exported)
that match. It realizes those that are not already realized.

The <<| |>> operator finds ONLY exported resources.

The query must thus be written to take that into account.
And THAT is where it becomes important how the != and == operators
actually work.

(I am in no way defending the way it works, but it will be very
difficult to change without difficult to find silent backwards
incompatibilities).

- henrik

>
> John
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/a4c43871-cd4e-4e40-a23b-5fa44bfc0dfc%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/a4c43871-cd4e-4e40-a23b-5fa44bfc0dfc%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

[Julio Guevara]

The <| |> operator finds ALL resources (regular, virtual and exported) 
that match. It realizes those that are not already realized.

The <<| |>> operator finds ONLY exported resources. 

If that is the case, maybe documentation is miss-leading since it makes a clear distinction between the two types of collectors. I will see If I can come up with another solution for my code but with such limited set of logical operators it will be hard. I think It would be easier to just make a query with puppetdb_query and then import exported resources that match my criteria correctly. 

Thanks

You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/6lgqC1HmKQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/pngbmn%24lqt%241%40blaine.gmane.org.

[Henrik Lindberg]

On 2018-09-14 17:13, Julio Guevara wrote:
> /The <| |> operator finds ALL resources (regular, virtual and exported)

> that match. It realizes those that are not already realized.
>

> The <<| |>> operator finds ONLY exported resources. /
> /
> /

> If that is the case, maybe documentation is miss-leading since it makes
> a clear distinction between the two types of collectors.

Yes, it is as it does not say that it works on all kinds of resources
which allows the reader to think that it is linked to them being
virtual. Feel free to add a ticket for documentation in Puppet's Jira.

> I will see If I
> can come up with another solution for my code but with such limited set
> of logical operators it will be hard. I think It would be easier to just
> make a query with puppetdb_query and then import exported resources that
> match my criteria correctly.

That is what I recommend and what most users end up doing when their use
case is not super trivial and works with the limitations of the collectors.

Best,
- henrik

> <mailto:puppet-users%2Bunsu...@googlegroups.com>
> > <mailto:puppet-users...@googlegroups.com
> <mailto:puppet-users%2Bunsu...@googlegroups.com>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/puppet-users/a4c43871-cd4e-4e40-a23b-5fa44bfc0dfc%40googlegroups.com
>
> >
> <https://groups.google.com/d/msgid/puppet-users/a4c43871-cd4e-4e40-a23b-5fa44bfc0dfc%40googlegroups.com?utm_medium=email&utm_source=footer>.
> > For more options, visit https://groups.google.com/d/optout.
>
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/6lgqC1HmKQk/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users...@googlegroups.com

> <mailto:puppet-users%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/pngbmn%24lqt%241%40blaine.gmane.org.
> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CADzjYHEJ3VnvKiO8nB6kmDTe4JAaupm7VLyN0ePpb0DNhkJxeQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-users/CADzjYHEJ3VnvKiO8nB6kmDTe4JAaupm7VLyN0ePpb0DNhkJxeQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.