Package duplicate resource issue

Drew Blessing

unread,

Mar 7, 2014, 1:55:51 PM3/7/14

to puppe...@googlegroups.com

I want to start a conversation about the package resource type and a bug that goes back about 6 years. The current ticket is https://tickets.puppetlabs.com/browse/PUP-1073 and relates to really old tickets such as https://projects.puppetlabs.com/issues/973.

If I have gem 'x' and RPM 'x', I cannot have puppet manage both of those resources on the same box. This collision happens quite often in reality. For example, we are currently working on etcd and have both the etcd server RPM and the etcd gem. Puppet uses the etcd gem to interact with etcd's API. We should be able to manage both of these with puppet without a conflict. Unfortunately, the unique resource name does not key off of the package name AND the provider.

Has anyone dug into the package type/provider or the puppet code base enough to know what it might take to fix this? On the surface it seems relatively easy but as long as the issue has been around it may be more involved.

Trevor Vaughan

unread,

Mar 7, 2014, 4:12:44 PM3/7/14

to puppe...@googlegroups.com

If you could, please go +1 it in Jira for visibility.

I too would love to see this fixed.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/058beb1c-3438-4881-9bd7-4961cb752b4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

Adrien Thebo

unread,

Mar 7, 2014, 5:15:45 PM3/7/14

to puppe...@googlegroups.com

Long story short, allowing multiple resources to exist with the same title but different providers is problematic. I've commented on this at https://projects.puppetlabs.com/issues/1398#note-13 which explains why this isn't trivial to solve. This is a limitation that is pretty core to how Puppet works and affects more than just package resources, so while I would love to see it solved I think we need to determine a general solution to name collisions in resource titles. If anyone has insights on how to solve the general problem I'm all ears. :)

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXMgnXvTOwe62Yz2o80E%3D5SqytRMrRXSXLZORiV50tnDg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Adrien Thebo | Puppet Labs

Trevor Vaughan

unread,

Mar 8, 2014, 12:24:07 PM3/8/14

to puppe...@googlegroups.com

How does 'exec' do it?

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALVJ9S%2BaxPFO6ZC92OACep0-pWsQjKv3CJVOtyjTuBwmy0TsUA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Pedro Côrte-Real

unread,

Mar 8, 2014, 3:48:21 PM3/8/14

to puppe...@googlegroups.com

On Fri, Mar 7, 2014 at 10:15 PM, Adrien Thebo <adr...@puppetlabs.com> wrote:
> Long story short, allowing multiple resources to exist with the same title
> but different providers is problematic.

There's no reason to need to do that though. Package just needs to be
able to override the package name without changing $name as that needs
to be unique. So you should be able to do something like:

package { 'somepackage-in-apt': ensure => present, pkgname =>
'somepackage', provider => apt, } package { 'somepackage-in-gem':
ensure => absent, pkgname => 'somepackage', provider => gem, }

Since we've used $pkgname instead of $name this doesn't have the
uniqueness issue. I've looked around the code and this seems easy
enough to do. The Package providers just need to do "pkgname ||= name"
so the older stuff doesn't break.

Can anyone find any fault with this solution? I've commented on these
bug reports a lot of times and never gotten any answer to this. It
seems pretty amazing that this bug still exists after so many years.

Pedro

Trevor Vaughan

unread,

Mar 8, 2014, 9:39:35 PM3/8/14

to puppe...@googlegroups.com

In theory a composite namevar could be used if you just specify the provider.

For instance, on a Red Hat system:

package { 'foo': ensure => 'latest' } ==> namevar == foo:yum

And

package { 'foo': ensure => 'latest', provider => 'gem' } ==> namevar == foo:gem

That said, I never could get composite namevars to work this way. I always had to have a unique name which ended up in something silly like package { 'foo_rpm':} or package { 'foo_gem':} which, while it should work, is absolutely horrible to read.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALprHx8hEvtyhMkwU1rKbhPXuWZt8NgVdiEu657-krix6d2nUg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

David Schmitt

unread,

Mar 9, 2014, 6:29:03 AM3/9/14

to puppe...@googlegroups.com

On 09.03.2014 03:39, Trevor Vaughan wrote:
> In theory a composite namevar could be used if you just specify the
> provider.
>
> For instance, on a Red Hat system:
>
> package { 'foo': ensure => 'latest' } ==> namevar == foo:yum
>
> And
>
> package { 'foo': ensure => 'latest', provider => 'gem' } ==> namevar ==
> foo:gem
>
> That said, I never could get composite namevars to work this way. I
> always had to have a unique name which ended up in something silly like
> package { 'foo_rpm':} or package { 'foo_gem':} which, while it should
> work, is absolutely horrible to read.

That was my understanding how composite namevars should have worked from
the beginning. Sadly, this seems "too complex" or "not useful enough" to
be actually implemented. It would even help for multi-arch and
multi-versions installations: "foo:1.0:i386:yum" vs "foo:2.0:amd64:yum".

Regards, David

Trevor Vaughan

unread,

Mar 9, 2014, 4:10:17 PM3/9/14

to puppe...@googlegroups.com

Oh, no, this works perfectly. I just *hate* having to stuff all of that into the name. It makes the hash of options completely pointless.

I want the name/title to be arbitrary and the rest to "just work". Unfortunately, I haven't found the special sauce for this yet.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/531C426F.1000903%40dasz.at.

For more options, visit https://groups.google.com/d/optout.

David Schmitt

unread,

Mar 10, 2014, 4:53:28 AM3/10/14

to puppe...@googlegroups.com

On 2014-03-09 21:10, Trevor Vaughan wrote:
> Oh, no, this works perfectly.

Seems like I have some reading to do, or, see below.

> I just *hate* having to stuff all of
> that into the name. It makes the hash of options completely
> pointless.

> I want the name/title to be arbitrary and the rest to "just work".

> Unfortunately, I havent found the special sauce for this yet.

package {
'foo:yum':
name => 'foo',
provider => 'yum';

'foo:gem':
name => 'foo',
provider => 'gem';
}

gives me

Error: Could not retrieve catalog from remote server: Error 400 on
SERVER: Puppet::Parser::AST::Resource failed with error
ArgumentError:
Cannot alias Package[foo:gem] to ["foo"] at
/vagrant/vagrant/manifests/
site.pp:127; resource ["Package", "foo"] already declared at
/vagrant/
vagrant/manifests/site.pp:127 at
/vagrant/vagrant/manifests/site.pp:127
on node puppetmaster.example.com

This tells me that packages do not have composite namevars and that
titles
do not have uniquification abilities.

A composite namevar for packages could include the package name,
version, arch
and provider and still work (in the DSL) as usual:

package {
'pipedream':
name => 'foo',
version => '1.0.0',
arch => 'noarch'
provider => 'yum';
}

would have the title "pipedream", the (composite) namevar value
"foo:1.0.0:noarch:yum" and the properties and parameters as specified
above.
References to this resource could be of the form

* Package['pipedream'], Package['foo:1.0.0:noarch:yum']: specific
instance
* Package['foo:1.0.0']: equivalent to Package <| name == 'foo' and
version == '1.0.0' |>
* Package['foo']: equivalent to Package <| name == 'foo' |>

I have no idea whether composite namevars work like this, but it is how
I think
the problem could be solved.

Regards, David

> Trevor
>
> On Sun, Mar 9, 2014 at 6:29 AM, David Schmitt <da...@dasz.at [4]>

> wrote:
>
>> On 09.03.2014 03:39, Trevor Vaughan wrote:
>>
>>> In theory a composite namevar could be used if you just specify
>>> the
>>> provider.
>>>
>>> For instance, on a Red Hat system:
>>>
>>> package { foo: ensure => latest } ==> namevar == foo:yum
>>>
>>> And
>>>

>>> package { foo: ensure => latest, provider => gem } ==> namevar ==

>>> foo:gem
>>>
>>> That said, I never could get composite namevars to work this way.
>>> I
>>> always had to have a unique name which ended up in something
>>> silly like
>>> package { foo_rpm:} or package { foo_gem:} which, while it should
>>> work, is absolutely horrible to read.
>>
>> That was my understanding how composite namevars should have worked
>> from the beginning. Sadly, this seems "too complex" or "not useful
>> enough" to be actually implemented. It would even help for
>> multi-arch and multi-versions installations: "foo:1.0:i386:yum" vs
>> "foo:2.0:amd64:yum".
>>
>> Regards, David
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Puppet Developers" group.
>> To unsubscribe from this group and stop receiving emails from it,

>> send an email to puppet-dev+...@googlegroups.com [1].

>> To view this discussion on the web visit
>>
>
> https://groups.google.com/d/msgid/puppet-dev/531C426F.1000903%40dasz.at

>> [2].
>>
>> For more options, visit https://groups.google.com/d/optout [3].

>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699

> tvau...@onyxpoint.com [5]

>
> -- This account not approved for unencrypted proprietary information
> --
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to puppet-dev+...@googlegroups.com [6].

> To view this discussion on the web visit
>

> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUwQXSZ%2BTL%2BxMMMff%3DnGLbpdurM9roQRAETX7EQxGPmQQ%40mail.gmail.com
> [7].
> For more options, visit https://groups.google.com/d/optout [8].
>
>
> Links:
> ------
> [1] mailto:puppet-dev%2Bunsu...@googlegroups.com
> [2]
> https://groups.google.com/d/msgid/puppet-dev/531C426F.1000903%40dasz.at
> [3] https://groups.google.com/d/optout
> [4] mailto:da...@dasz.at
> [5] mailto:tvau...@onyxpoint.com
> [6] mailto:puppet-dev+...@googlegroups.com
> [7]
>
> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUwQXSZ%2BTL%2BxMMMff%3DnGLbpdurM9roQRAETX7EQxGPmQQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> [8] https://groups.google.com/d/optout

Trevor Vaughan

unread,

Mar 10, 2014, 9:03:30 AM3/10/14

to puppe...@googlegroups.com

The first version would work if packages were implemented with composite namevars (which they're not). However, it's ugly and shouldn't be necessary in this case.

The second version should work no matter what but causes confusion when trying to do any type of resource dependency chaining.

You'd end up with something like:

class foo {

$pkgname = 'bar'

$pkgrealname = 'baz'

package { $pkgname:

name => $pkgrealname,

etc...
}
}

Then, everything else would have to do requires => Package[$::foo::pkgname], even though that might just be 'apache' and makes everything hard to maintain and trace.

Also, you *still* need to be able to detect conflicts between Package resources of the same provider.

package { 'mysql': provider => 'yum' } should *not* conflict with package { 'mysql': provider => 'gem' }. However, it *should* conflict with package { 'mysql': provider => 'yum', ensure => 'latest' }.

This is actually pretty easy to do in the global validate statement of the type. However, you'll need to parse the collected yum resources at each resource declaration to ensure that you don't conflict with anything else in the stack. I suppose that it could keep track in a hash on the fly, but that might not be any more efficient depending on how the resource catalog searches are done (I haven't looked).

Trevor

send an email to puppet-dev+unsubscribe@googlegroups.com [1].

To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/531C426F.1000903%40dasz.at

[2].

For more options, visit https://groups.google.com/d/optout [3].

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699

tvau...@onyxpoint.com [5]

-- This account not approved for unencrypted proprietary information
--

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it,

send an email to puppet-dev+unsubscribe@googlegroups.com [6].

To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUwQXSZ%2BTL%2BxMMMff%3DnGLbpdurM9roQRAETX7EQxGPmQQ%40mail.gmail.com
[7].
For more options, visit https://groups.google.com/d/optout [8].

Links:
------
[1] mailto:puppet-dev%2Bunsu...@googlegroups.com
[2] https://groups.google.com/d/msgid/puppet-dev/531C426F.1000903%40dasz.at
[3] https://groups.google.com/d/optout
[4] mailto:da...@dasz.at
[5] mailto:tvau...@onyxpoint.com

[6] mailto:puppet-dev+unsubscribe@googlegroups.com

[7]

https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUwQXSZ%2BTL%2BxMMMff%3DnGLbpdurM9roQRAETX7EQxGPmQQ%40mail.gmail.com?utm_medium=email&utm_source=footer
[8] https://groups.google.com/d/optout

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/ae3fafce9cb7e3521c2437070c291e2e%40hosting.edv-bus.at.

For more options, visit https://groups.google.com/d/optout.

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

Drew Blessing

unread,

Mar 10, 2014, 9:09:16 AM3/10/14

to puppe...@googlegroups.com, pe...@pedrocr.net

I agree, it seems like this solution would be simple and effective. I am almost positive there are other types that behave this way. It breaks nothing and fixes everything, as far as I can see.

John Bollinger

unread,

Mar 10, 2014, 9:54:07 AM3/10/14

to puppe...@googlegroups.com

On Saturday, March 8, 2014 11:24:07 AM UTC-6, Trevor Vaughan wrote:

How does 'exec' do it?

Exec doesn't do it. What Exec does (or did -- I am having trouble finding mention of it in the current docs), is to allow the title / command to be duplicated without restriction, regardless of provider.

John

Trevor Vaughan

unread,

Mar 10, 2014, 10:17:30 AM3/10/14

to puppe...@googlegroups.com

That would technically also solve the problem. There would just have to be an additional validation step to make sure that the keys in the resource are unique.

I wonder if all resources could be changed to more of a DB-style key model where you could denote specific attributes as your unique key set and all would need to match prior to a resource being declared duplicate.

It seems that it would solve this issue and some others that I've seen with people creating custom types (though I can't find any examples right now).

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/b38dada5-bc4a-4fd9-a28b-80f2f1a1d6c6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Adrien Thebo

unread,

Mar 10, 2014, 11:07:42 AM3/10/14

to puppe...@googlegroups.com

My concern with this solution is that it's a one time shim for a single type. Granted, it may work and could solve this particular problem. However I think this is a flaw in the RAL that has a number of touch points that also need to be fixed. This might be me being too idealistic but I think that we can fix this issue and improve the entire RAL rather than trying to make individual cases work as expected.

--

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/7eb053d7-be1b-47b0-8a72-73a57d898612%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Drew Blessing

unread,

Mar 10, 2014, 11:15:37 AM3/10/14

to puppe...@googlegroups.com

Unless you can definitively say that making major changes in the RAL to address this issue is slated for the near future (say, late 3.x release or first 4.x release), I'd say the individual fix for package type is warranted in the meantime. Are there any other types that you think people are having major issues with at the moment? Maybe the "shim" could be released in 3.x series and work can proceed on the longterm fix in the 4.x series?

Trevor Vaughan

unread,

Mar 10, 2014, 2:28:49 PM3/10/14

to puppe...@googlegroups.com

It's just this one, as far as I know.

I would like to see the RAL updated, but I'd like this fix in Puppet 3 and the RAL update in Puppet 4/5/whatever.

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/74d181f4-60ed-40d3-bcb8-84cc3543d862%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Drew Blessing

unread,

Mar 10, 2014, 4:29:39 PM3/10/14

to puppe...@googlegroups.com

I think that sounds like a great way to approach the issue, Trevor. Adrien, what do you think?

FWIW, I just encountered this issue for the second time in a week. I suspect this trend will continue. We are finally getting comfortable with custom types/providers and want to build providers to interact with APIs. If we control the package then we can shift things around easily but in many cases we control neither the gem nor the rpm package.

Trevor Vaughan

unread,

Mar 10, 2014, 4:36:41 PM3/10/14

to puppe...@googlegroups.com

Well, I'm certainly a fan.

Now, we just need some upstream traction.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/15b633d6-9713-4d94-8210-b202447e02c9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

David Schmitt

unread,

Mar 11, 2014, 7:31:50 AM3/11/14

to puppe...@googlegroups.com

Talk about coincidence. Today I implemented this fragment of a custom
type:

def self.title_patterns
[
[ /^([-_\w\/]+)\.(\d+)$/ , [ [:parent_interface ],
[:subinterface_number] ] ],
[ /.*/, [ ] ]
]
end

newparam(:parent_interface) do
desc "The name of the parent interface, automatically parsed from
the title."
newvalues(/^[-_\w\/]+$/)
isnamevar
end

newparam(:subinterface_number) do
desc "The number of this subinterface, automatically parsed from
the title."
newvalues(/^\d+$/)
isnamevar
end

usage:

blah_subinterface {
"ethernet1/2.3": ...
}

leads to this resource_hash:

:parent_interface => 'ethernet1/2',
:subinterface_number => '3',

without any further intervention on my side.

Awesome sauce.

This can be used like this too:

blah_subinterface { "important title":
parent_interface => 'ethernet1/2',
subinterface_number => '3',
...
}

so, there seems nothing in the way of your use-case, no?

Regards, David

On 2014-03-09 21:10, Trevor Vaughan wrote:

> Oh, no, this works perfectly. I just *hate* having to stuff all of
> that into the name. It makes the hash of options completely
> pointless.
>
> I want the name/title to be arbitrary and the rest to "just work".

> Unfortunately, I havent found the special sauce for this yet.
>
> Trevor
>
> On Sun, Mar 9, 2014 at 6:29 AM, David Schmitt <da...@dasz.at [4]>

> wrote:
>
>> On 09.03.2014 03:39, Trevor Vaughan wrote:
>>
>>> In theory a composite namevar could be used if you just specify
>>> the
>>> provider.
>>>
>>> For instance, on a Red Hat system:
>>>
>>> package { foo: ensure => latest } ==> namevar == foo:yum
>>>
>>> And
>>>

>>> package { foo: ensure => latest, provider => gem } ==> namevar ==

>>> foo:gem
>>>
>>> That said, I never could get composite namevars to work this way.
>>> I
>>> always had to have a unique name which ended up in something
>>> silly like
>>> package { foo_rpm:} or package { foo_gem:} which, while it should
>>> work, is absolutely horrible to read.
>>
>> That was my understanding how composite namevars should have worked
>> from the beginning. Sadly, this seems "too complex" or "not useful
>> enough" to be actually implemented. It would even help for
>> multi-arch and multi-versions installations: "foo:1.0:i386:yum" vs
>> "foo:2.0:amd64:yum".
>>
>> Regards, David
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Puppet Developers" group.
>> To unsubscribe from this group and stop receiving emails from it,

>> send an email to puppet-dev+...@googlegroups.com [1].

>> To view this discussion on the web visit
>>
>
> https://groups.google.com/d/msgid/puppet-dev/531C426F.1000903%40dasz.at

>> [2].
>>
>> For more options, visit https://groups.google.com/d/optout [3].

>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699

> tvau...@onyxpoint.com [5]

>
> -- This account not approved for unencrypted proprietary information
> --
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to puppet-dev+...@googlegroups.com [6].

> To view this discussion on the web visit
>

> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUwQXSZ%2BTL%2BxMMMff%3DnGLbpdurM9roQRAETX7EQxGPmQQ%40mail.gmail.com
> [7].

> For more options, visit https://groups.google.com/d/optout [8].
>
>
> Links:
> ------
> [1] mailto:puppet-dev%2Bunsu...@googlegroups.com
> [2]
> https://groups.google.com/d/msgid/puppet-dev/531C426F.1000903%40dasz.at
> [3] https://groups.google.com/d/optout
> [4] mailto:da...@dasz.at
> [5] mailto:tvau...@onyxpoint.com

> [6] mailto:puppet-dev+...@googlegroups.com

Andy Parker

unread,

Mar 11, 2014, 1:29:52 PM3/11/14

to puppe...@googlegroups.com

On Mon, Mar 10, 2014 at 1:36 PM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

Well, I'm certainly a fan.

Personally, I would be ok with yet another hack in puppet 3 to handle this issue since it has been coming up so often and since I also don't know a clear timeline for getting new functionality in to address this specific issue in a better way. And yes, my idealism is cracking :/

An advantage of getting a specific fix in for this is that it would clearly point out where the RAL needs for flexibility than it has right now. I think we all have an idea of where it needs it in general, but some of the nitty gritty details would be nice.

Now, we just need some upstream traction.

A patch that achieves what is being talked about here just needs to be submitted :)

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoV0g7gj5d__c-24DRJVcryszZNbARLKBiBH83KrKs4LEg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2014, September 23-24 in San Francisco - http://bit.ly/pupconf14

Pedro Côrte-Real

unread,

Mar 11, 2014, 5:23:33 PM3/11/14

to puppe...@googlegroups.com

On Tue, Mar 11, 2014 at 5:29 PM, Andy Parker <an...@puppetlabs.com> wrote:
> Personally, I would be ok with yet another hack in puppet 3 to handle this
> issue since it has been coming up so often and since I also don't know a
> clear timeline for getting new functionality in to address this specific
> issue in a better way. And yes, my idealism is cracking :/

It's great to finally see traction on this. I still don't understand
why this is a hack though. This is what is broken:

package {"foo_deb":
name => foo,
provider => apt,
}
package {"foo_gem":
name => foo,
provider => gem,
}

But the only reason this doesn't work is that we used $name to
override the deb/gem name. This on the other hand would work fine:

exec {"foo_as_root":
command => "/bin/foo",
user => root,
}
exec {"foo_as_someuser":
command => "/bin/foo",
user => someuser,
}

Yet the only difference between the two cases is that we used $command
and not $name to override the default given by $title.

Package was designed assuming the meaningless tokens we pass to apt to
select debs have some relation to the meaningless tokens we pass to
gem to select gems. Most of the time this doesn't bite us because
names are reasonably unique. But then someone goes and uses "memcache"
for the server deb and the client gem. These are totally different
software packages that just happen to use the same token in two
different package systems. Using $title as the default meaningless
token is economical in terms of keystrokes but then just like with
Exec there needs to be a not unique $meaningless_token variable.

This is just a bug in Package, fixing it isn't a hack.

Pedro

Trevor Vaughan

unread,

Mar 11, 2014, 9:13:14 PM3/11/14

to puppe...@googlegroups.com

If possible, I would love to see this done without a composite namevar.

The issue is that you're going to start ending up with variables *everywhere* to figure out what you're actually installing.

If possible, I would like the same 'title' but that should be smoothly combined with the provider.

That said, if it ends up being a composite namevar, that's not the end of the world. We just need to use a delimiter that isn't used in package names. An @ maybe? Something wide makes it easy to read. The cases where I used a composite namevar I used a plus. So, mysql+rpm, which is quite easy to scan.

package { "mysql@rpm": ... }

Trevor

Pedro

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALprHx_tTEKX%3D-%2B3iZiDDiWOS0fp20fT91TV%3DpWpd92eRNtVSA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Drew Blessing

unread,

Mar 11, 2014, 9:32:31 PM3/11/14

to <puppet-dev@googlegroups.com>

I am pretty sure a composite namevar won't be necessary. It will just be a new param such as pkgname that, if not specified, would default to the resource title/name for backward compatibility.

Since we have the go ahead from Puppet, does anyone have the know-how to do this? I could try to hack on it but experience and time may be a factor. I'll gladly defer to someone with these qualities. :) Let's not let this die here!

You received this message because you are subscribed to a topic in the Google Groups "Puppet Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-dev/LatVZFUkwEM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVJxr7FwwqZPhKuZ4RuHihkt1pTD6c-hsXjys7N-7LTag%40mail.gmail.com.

David Schmitt

unread,

Mar 12, 2014, 5:12:28 AM3/12/14

to puppe...@googlegroups.com

On 2014-03-12 02:13, Trevor Vaughan wrote:
> If possible, I would love to see this done without a composite
> namevar.
>

> The issue is that youre going to start ending up with variables
> *everywhere* to figure out what youre actually installing.

I'm not fully convinced that this is actually true.

I've now finished the composite type I was talking about earlier, and
the missing corner stone was defining

def name
"#{parent_interface}.#{subinterface_number}"
end

on the type, so that the prefetched resources are matched up to the
resources from the catalog.

> If possible, I would like the same title but that should be smoothly
> combined with the provider.
>

> That said, if it ends up being a composite namevar, thats not the end
> of the world. We just need to use a delimiter that isnt used in

> package names. An @ maybe? Something wide makes it easy to read. The
> cases where I used a composite namevar I used a plus. So, mysql+rpm,
> which is quite easy to scan.
>
> package { "mysql@rpm": ... }

Without having it tested, it should be possible to specify multiple
patterns to match on titles without '@' to set only the package name.

So the following should still work:

package {
"mysql":;
"apache":
package_name => 'apache2-mpm-itk';
"nokogiri":
provider => gem;
"memcache":;
"memcache@gem":;
}

With the following correct references:

* using the title:
Package['mysql']
Package['apache']
Package['nokogiri']
Package['memcache']
Package['memcache@gem']
* using the actual name, which probably be very brittle as the
provider is often defaulted based on facts
Package['mysql@rpm']
Package['apache2-mpm-itk@deb']
Package['nokogiri@gem']
Package['memcache@rpm']
Package['memcache@gem']
* using queries; this is very explicit about the user's requirement,
may return multiple resources
Package<| title == 'mysql' |>
Package<| package_name == 'mysql' |>
Package<| package_name == 'mysql' and provider != 'gem' |>
Package<| package_name == 'memcache' |>

I think there is no (trivial) way around the fundamental problem that
"Package['memcache']" cannot refer to a single resource when the package
name is not unique.

I'm currently working through these issues (with a different type) in
the project I'm working on, so I'd be very thankful about any
enlightenment about problems you see with this approach.

Regards, David

> Trevor
>
> On Tue, Mar 11, 2014 at 5:23 PM, Pedro Côrte-Real <pe...@pedrocr.net
> [5]> wrote:
>
>> On Tue, Mar 11, 2014 at 5:29 PM, Andy Parker <an...@puppetlabs.com
>> [1]> wrote:
>> > Personally, I would be ok with yet another hack in puppet 3 to
>> handle this

>> > issue since it has been coming up so often and since I also dont

>> know a
>> > clear timeline for getting new functionality in to address this
>> specific
>> > issue in a better way. And yes, my idealism is cracking :/
>>

>> Its great to finally see traction on this. I still dont understand

>> why this is a hack though. This is what is broken:
>>
>> package {"foo_deb":
>> name => foo,
>> provider => apt,
>> }
>> package {"foo_gem":
>> name => foo,
>> provider => gem,
>> }
>>

>> But the only reason this doesnt work is that we used $name to

>> override the deb/gem name. This on the other hand would work fine:
>>
>> exec {"foo_as_root":
>> command => "/bin/foo",
>> user => root,
>> }
>> exec {"foo_as_someuser":
>> command => "/bin/foo",
>> user => someuser,
>> }
>>
>> Yet the only difference between the two cases is that we used
>> $command
>> and not $name to override the default given by $title.
>>
>> Package was designed assuming the meaningless tokens we pass to apt
>> to
>> select debs have some relation to the meaningless tokens we pass to

>> gem to select gems. Most of the time this doesnt bite us because

>> names are reasonably unique. But then someone goes and uses
>> "memcache"
>> for the server deb and the client gem. These are totally different
>> software packages that just happen to use the same token in two
>> different package systems. Using $title as the default meaningless
>> token is economical in terms of keystrokes but then just like with
>> Exec there needs to be a not unique $meaningless_token variable.
>>

>> This is just a bug in Package, fixing it isnt a hack.

>>
>> Pedro
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Puppet Developers" group.
>>
>> To unsubscribe from this group and stop receiving emails from it,

>> send an email to puppet-dev+...@googlegroups.com [2].

>> To view this discussion on the web visit
>>
>
> https://groups.google.com/d/msgid/puppet-dev/CALprHx_tTEKX%3D-%2B3iZiDDiWOS0fp20fT91TV%3DpWpd92eRNtVSA%40mail.gmail.com

>> [3].
>>
>> For more options, visit https://groups.google.com/d/optout [4].

>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699

> tvau...@onyxpoint.com [6]

>
> -- This account not approved for unencrypted proprietary information
> --
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to puppet-dev+...@googlegroups.com [7].

> To view this discussion on the web visit
>

> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVJxr7FwwqZPhKuZ4RuHihkt1pTD6c-hsXjys7N-7LTag%40mail.gmail.com
> [8].
> For more options, visit https://groups.google.com/d/optout [9].
>
>
> Links:
> ------
> [1] mailto:an...@puppetlabs.com
> [2] mailto:puppet-dev%2Bunsu...@googlegroups.com
> [3]
>
> https://groups.google.com/d/msgid/puppet-dev/CALprHx_tTEKX%3D-%2B3iZiDDiWOS0fp20fT91TV%3DpWpd92eRNtVSA%40mail.gmail.com
> [4] https://groups.google.com/d/optout
> [5] mailto:pe...@pedrocr.net
> [6] mailto:tvau...@onyxpoint.com
> [7] mailto:puppet-dev+...@googlegroups.com
> [8]
>
> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVJxr7FwwqZPhKuZ4RuHihkt1pTD6c-hsXjys7N-7LTag%40mail.gmail.com?utm_medium=email&utm_source=footer
> [9] https://groups.google.com/d/optout

Thomas Linkin

unread,

Mar 12, 2014, 11:59:33 AM3/12/14

to puppe...@googlegroups.com

I agree with Adrien. I think the solution will be something with a benefit for the entire RAL.

The thought I have is of a solution that preserves the original behavior of having a resource type auto-discover it’s preferred provider on a system. So in this case, we could allow one resource per catalog that can auto discover its provider. Then, any other resource will require an explicit provider to be added.

package { ‘foo’:

ensure => installed,

}

package { ‘foo’:

ensure => installed,

provider => gem,

}

The next question would be how to reference the resource and display the resource in the report. I figure at this point, just merge the provider name and the title. I think I saw someone use ‘@‘ earlier, thats fine. Maybe a tilde, or a double ‘@‘. Some form of delimiter, whatever, it’s only used for referring to a title+provider internally, it should have no bearing on the name of the resource used out on the agent.

Package[‘foo’]

- and -

Package[‘foo~gem’]

The final issue is what do we do when the auto-discovered provider overlaps with a resource that contains and explicit provider. For example, in the same catalog, a package resource managing ‘foo' that automatically finds its provider on RHEL, and a package resource managing ‘foo' that explicitly has ‘provider => yum’. These two entries would overlap and express potentially competing states for the same resource.

This is the part I’m not so sure about, since the provider qualification occurs out on the agent. If we were only confining on facts, I’d say use the facts on the master to confine, find overlaps, and fail the catalog; though that sounds like a lot of work to me, and confinements aren’t always done on just facts alone. Instead, we could resolve the automatic resource on the agent and then ignore it in the event it overlaps with a resource that has an explicit provider, but that feels messy. The only other action I can imagine is to fail both resources on the agent and make note in the report.

There is one idea I do have to prevent overlap, if a resource only has a single provider (for example, the host type), then we don’t allow explicit entries and automatic entries in the same catalog. This does appear to be something we can check at compilation time.

--

Tom Linkin

Professional Services Engineer

http://puppetlabs.com/

twitter: @trlinkin

Join us at PuppetConf 2014, September 23-24 in San Francisco - http://bit.ly/pupconf14

Register now and save $350!

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALVJ9SKe5JWSA%3D1syKLHownEOu87GkXRyr_fpw%3DSUuA4b2vWmw%40mail.gmail.com.

John Bollinger

unread,

Mar 12, 2014, 2:35:12 PM3/12/14

to puppe...@googlegroups.com

On Tuesday, March 11, 2014 4:23:33 PM UTC-5, Pedro Côrte-Real wrote:

On Tue, Mar 11, 2014 at 5:29 PM, Andy Parker <an...@puppetlabs.com> wrote:
> Personally, I would be ok with yet another hack in puppet 3 to handle this
> issue since it has been coming up so often and since I also don't know a
> clear timeline for getting new functionality in to address this specific
> issue in a better way. And yes, my idealism is cracking :/

It's great to finally see traction on this. I still don't understand
why this is a hack though. This is what is broken:

package {"foo_deb":
name => foo,
provider => apt,
}
package {"foo_gem":
name => foo,
provider => gem,
}

But the only reason this doesn't work is that we used $name to
override the deb/gem name. This on the other hand would work fine:

exec {"foo_as_root":
command => "/bin/foo",
user => root,
}
exec {"foo_as_someuser":
command => "/bin/foo",
user => someuser,
}

Yet the only difference between the two cases is that we used $command
and not $name to override the default given by $title.

No, that's not really a difference at all. 'command' is the namevar of the Exec type (it has no 'name' property), and 'name' is the namevar of Package. It is the fact that 'name' is the namevar of Package that makes the package version not work. That the Exec version does work is the result of a special-case exception for that resource type; it has nothing in particular to do with the name of the type's namevar.

Package was designed assuming the meaningless tokens we pass to apt to
select debs have some relation to the meaningless tokens we pass to
gem to select gems.

I think Package and most other built-in types were designed around the idea that only one provider would ever be used on any given target system. And indeed, that's a superior policy, at least when it comes to packages. That it works -- up to a point -- for multiple package providers is a secondary feature. Now I'm not saying that's an ignorable feature or one that shouldn't be improved, but on the other hand I do not favor creating additional special cases for particular resource types.

This is just a bug in Package, fixing it isn't a hack.

It is not a bug in Package, it is a natural consequence of Puppet's architecture. Resource providers cannot (reliably) be distinguishing characteristics of any resource type, because for the most part they are not assigned or known during catalog compilation.

Consider this:

package { 'foo_native':
name => 'foo',
}

package { 'foo_gem':
name => 'foo',
provider => 'gem',
}

Do those declarations refer to the same resource or not? What if the target machine is some screwball system for which the default package provider is 'gem'?

John

Pedro Côrte-Real

unread,

Mar 12, 2014, 4:15:35 PM3/12/14

to puppe...@googlegroups.com

On Wed, Mar 12, 2014 at 6:35 PM, John Bollinger
<john.bo...@stjude.org> wrote:
> No, that's not really a difference at all. 'command' is the namevar of the
> Exec type (it has no 'name' property), and 'name' is the namevar of Package.
> It is the fact that 'name' is the namevar of Package that makes the package
> version not work. That the Exec version does work is the result of a
> special-case exception for that resource type; it has nothing in particular
> to do with the name of the type's namevar.

My question is why does $command or $pkgname (in my example) have to
be a namevar? I assumed $command actually wasn't a namevar. I
understand the problem now though (see below).

> I think Package and most other built-in types were designed around the idea
> that only one provider would ever be used on any given target system. And
> indeed, that's a superior policy, at least when it comes to packages. That
> it works -- up to a point -- for multiple package providers is a secondary
> feature. Now I'm not saying that's an ignorable feature or one that
> shouldn't be improved, but on the other hand I do not favor creating
> additional special cases for particular resource types.

That may very well be the historical reason. It's a totally arbitrary
restriction though. Once you support multiple providers on the same
system saying that "foo in gem conflicts with foo in apt" doesn't make
much sense.

> It is not a bug in Package, it is a natural consequence of Puppet's
> architecture. Resource providers cannot (reliably) be distinguishing
> characteristics of any resource type, because for the most part they are not
> assigned or known during catalog compilation.
>
> Consider this:
>
> package { 'foo_native':
> name => 'foo',
> }
>
> package { 'foo_gem':
> name => 'foo',
> provider => 'gem',
> }
>
> Do those declarations refer to the same resource or not? What if the target
> machine is some screwball system for which the default package provider is
> 'gem'?

This makes more sense. I'd just assume the user knows what he's doing
because he has to anyway. Asking for "Package[foo] from whatever
package system the target has" is not a useful thing to do because
there is no standardization of names between package systems. You
already have to know what package system you are targeting to be able
to select the package name. Enforcing the namevar constraint in this
case just hurts the user.

But if you want to keep it pure make this case conflict and only allow
the case where both Package declarations have a provider declared, so
you know the provider at compilation, it's less flexible but much less
of an issue than the current solution.

Cheers,

Pedro

Trevor Vaughan

unread,

Mar 12, 2014, 6:23:51 PM3/12/14

to puppe...@googlegroups.com

I just realized that there's another way around this. It's not so glamorous as fixing the type, but it would work.

Just create new types that inherit the package type and hard code the provider.

package_yum

package_gem

etc....

This would probably be pretty quick work overall and, of course, eliminates all of the conflict issues.

But....it doesn't feel like a good solution.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALprHx8bXHLiwWaUq3iSxqYw78OmmxA6FcDdF5O1C8jKo9fFRQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Felix Frank

unread,

Mar 13, 2014, 7:37:58 AM3/13/14

to puppe...@googlegroups.com

On 03/12/2014 11:23 PM, Trevor Vaughan wrote:
>
> package_yum
> package_gem
> etc....
>
> This would probably be pretty quick work overall and, of course,
> eliminates all of the conflict issues.
>
> But....it doesn't feel like a good solution.

I agree that the implementation you sketched is messy, but the basic
idea has actually occured to me as well when I read John's latest reply.

The assumption that the regular old system will only ever use one
provider for a given type holds true for many use cases. Package is a
notable exception insofar as it can now be used to manage things that
are not OS packages (but also gems and what have you).

IMO the source of the issues here is that a gem is fundamentally a
different kind of package than the rpm/deb/... your OS uses for native
software. (So are CPAN modules, puppet modules etcpp.)

Creating new types that explicitly name the type of package they deal
with would handle this, yes (although I would vote that yum/apt/dpkg/rpm
and friends stay in the pure "package" domain if we should ever go that
route).

It would be nice if we had a system were the compiler could infer the
subtype from the value of the provider parameter. Because really, we are
only prone to hit conflicts when non-OS packages are requested via
providers like gem/cpan/pecl/pear/forge/whatever.

The generic enhancement of the Model would allow to auto-switch to
subtypes via arbitrary rules, where going by the provider value would
likely suffice for the package type.

Regards,
Felix

Trevor Vaughan

unread,

Mar 13, 2014, 8:06:28 AM3/13/14

to puppe...@googlegroups.com

I think I just realized what I actually want to be able to do when creating types in a generic sense.

newparam(:name) do

isnamevar
...

end

newparam(:provider) do

isnamevar
...

end

Now, in the catalog, what would happen is that *all* parameters that were named as namevars would be combined.

:title = "#{@resource[:name]}:#{@resource[:provider]}" (obviously looping over all parameters and pulling out all that are namevars and clumping them together in some rational order, possibly just alphabetical)

This is fundamentally what I would like to see in the end solution to the more generic problem. And I still think that this type of thing should be done since, whenever you use a composite namevar, you probably want this instead.

The only sticking point here is on relationship dependencies. How do you specify them?

* Package['name'] (that would pick up *all* of them or possibly just the system default. Anything other than the system default would need to be explicitly specified with one of the methods below.

* Package['name:provider'] (that would pick up the right one but relies on knowing the implementation)

* Package['name(provider => "yum")'] (better since it's more specific and not magic order dependent but still relies on knowing the implementation to a point)

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/53219896.6080206%40alumni.tu-berlin.de.

For more options, visit https://groups.google.com/d/optout.

Drew Blessing

unread,

Mar 13, 2014, 8:24:44 AM3/13/14

to puppe...@googlegroups.com

I am kind of interested in what you describe. I think for our environment,
keeping the package type as is and creating a new ³gem² type that inherits
from package would be just fine. If we only consider RHEL-based systems
for a moment, is there really a possible conflict scenario other than
yum/rpm and gems (where you wouldn¹t be insane for trying to install
multiple packages of the same name)?

It would still be nice to see some work go into the RAL for Puppet 4 and
have a permanent solution. However, would it be easier in Puppet3 if
someone just created a ³gem² module that had the inherited type/provider
in it? It would be trivial to do that.

Drew

>--
>You received this message because you are subscribed to a topic in the

>Google Groups "Puppet Developers" group.

>To unsubscribe from this topic, visit
>https://groups.google.com/d/topic/puppet-dev/LatVZFUkwEM/unsubscribe.
>To unsubscribe from this group and all its topics, send an email to

>puppet-dev+...@googlegroups.com.
>To view this discussion on the web visit

>https://groups.google.com/d/msgid/puppet-dev/53219896.6080206%40alumni.tu-
>berlin.de.

Trevor Vaughan

unread,

Mar 13, 2014, 9:02:22 AM3/13/14

to puppe...@googlegroups.com

Looking at the list of providers for file, on RHEL systems you'll probably only run into conflicts with *native* (yum, rpm, up2date), pip, and gem.

I also am not opposed to this. As far as I can tell, 'package' is the only type that doesn't follow the rules of the other types (manage one thing and manage it well). It rolls lots of different things under the same abstract header.

Frankly, if I have to specify provider => 'gem', that's not any more work than specifying gem { 'name': ... } and the latter is a lot clearer in terms of dependency maintenance.

So, +1 from me for just creating a bunch of new types.

Trevor

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CF470C6D.14D64%25drew.blessing%40buckle.com.

For more options, visit https://groups.google.com/d/optout.

John Bollinger

unread,

Mar 13, 2014, 9:23:24 AM3/13/14

to puppe...@googlegroups.com

On Thursday, March 13, 2014 8:02:22 AM UTC-5, Trevor Vaughan wrote:

Looking at the list of providers for file, on RHEL systems you'll probably only run into conflicts with *native* (yum, rpm, up2date), pip, and gem.

I also am not opposed to this. As far as I can tell, 'package' is the only type that doesn't follow the rules of the other types (manage one thing and manage it well). It rolls lots of different things under the same abstract header.

Exactly so. If there is a design flaw here, it is that the meaning of the Package type is fuzzy. It should represent software packages for the target node's native package management system.

Frankly, if I have to specify provider => 'gem', that's not any more work than specifying gem { 'name': ... } and the latter is a lot clearer in terms of dependency maintenance.

So, +1 from me for just creating a bunch of new types.

That would work for me. Alternatively, there could be just one new type, say SecondaryPackage, that rolled up gem, pip, fink, etc.. The trick here would be that the provider(s) must not be based on package type, so that the package type could be used as part of a composite name.

John

Trevor Vaughan

unread,

Mar 13, 2014, 9:50:19 AM3/13/14

to puppe...@googlegroups.com

SecondaryPackage wouldn't fix it if you wanted to install using pip and gem on the same system.

I do think that native packages are what Package should manage and everything else should be its own type/provider combo because they are, indeed, *something else*.

This gets hairy with RPM since Debian, etc... can all install and use the RPM command so I'm not sure quite how to handle that one. I suppose that native types could be created for everything but just call the appropriate backend if not called explicitly. That does let people use package{} and rpm{} on the same system though.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/67e90ed8-a8af-4e3a-a02b-f8f2e2ee035a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Felix Frank

unread,

Mar 13, 2014, 10:13:35 AM3/13/14

to puppe...@googlegroups.com

> On Thu, Mar 13, 2014 at 9:23 AM, John Bollinger
> <john.bo...@stjude.org <mailto:john.bo...@stjude.org>> wrote:
> > That would work for me. Alternatively, there could be just one new
> > type, say SecondaryPackage, that rolled up gem, pip, fink, etc.

On 03/13/2014 02:50 PM, Trevor Vaughan wrote:
> SecondaryPackage wouldn't fix it if you wanted to install using pip and
> gem on the same system.

Huge +1. John, would you have us dance this one again in a year or two? ;-)

> This gets hairy with RPM since Debian, etc... can
> all install and use the RPM command so I'm not sure quite how to handle
> that one.

Ugh, that is a much better point than I'd like to admit.

It could be handled by the "pick subtype based on provider value" rule I
sketched earlier, if we opt to *not* let dpkg, rpm and friends remain
un-subtyped.

It's not even that much of an outer edge case, because given a Debian
box on which you want/have to use this RPM from Corporate IT, this would
be a likely manifest:

package {
"puppet": provider => "dpkg", ensure => absent;
"puppet": provider => "rpm", ensure => present;
}

And yes, specifying that the rpm should require the dpkg would be hella
non-intuitive (require Package["puppet:dpkg"]? Dpkg_package["puppet"]?
Code readability anyone?) In that vein, it would indeed be helpful to
allow the alternative syntax of

dpkg_package { "puppet": } vs. rpm_package { "puppet": }

...but that will introduce new pitfalls for users, I believe. Full
vicious circle.

Felix

Trevor Vaughan

unread,

Mar 13, 2014, 12:53:08 PM3/13/14

to puppe...@googlegroups.com

Yeah, after all this I think break out all of the subtypes, make the suitable but default the native *package* statement to whatever it does now.

Then...admins be careful!

We can do some fanciness by mining the catalog when building the subtypes and making sure there is no conflicting native Package statement now that we have the fancy 'after everything' functions.

Hopefully that makes sense, running on fumes.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/5321BD0F.5090507%40alumni.tu-berlin.de.

For more options, visit https://groups.google.com/d/optout.

John Bollinger

unread,

Mar 14, 2014, 9:20:53 AM3/14/14

to puppe...@googlegroups.com

On Thursday, March 13, 2014 8:50:19 AM UTC-5, Trevor Vaughan wrote:

SecondaryPackage wouldn't fix it if you wanted to install using pip and gem on the same system.

I see I should have devoted more text to my last statement: "The trick here would be that the provider(s) must not be based on package type, so that the package type could be used as part of a composite name." If the type's name were a composite of type (gem, pip, etc.) and name within that type, then it very well could support different package types all in one resource type. I suppose the individual package types could be features. Whereas such an approach cannot work for Package, it would be eminently workable for a unified SecondaryPackage type.

Putting it all in one type might make it a bit easier to convert existing manifests, and it would give users a single place to look for support for this sort of thing. On the other hand, the provider(s) would have to support multiple (secondary) package types. It's a trade-off between what aspects must be complicated and what parts can be simple.

John

Andy Parker

unread,

Oct 6, 2014, 6:13:43 PM10/6/14

to puppe...@googlegroups.com

Sorry to resurrect an old thread, but this came to my attention again today.

I think there might be a much simpler solution to the entire thing. I noticed that all of the error messages that I've seen about this are about being unable to alias. What seems to be happening is that since the "name" parameter of a package resource is the namevar, the system is automatically creating an alias for the package resource using the name. That means that we have both a Package[title] reference and a Package[name] reference. The same thing occurred in the comment that was recently added to the ticket about the tidy type.

So here is my proposal: just remove the automatic aliasing. That means that the only way to reference a resource is via the title or an explicit alias. I tried this out on a VM by simply commenting out one line (https://github.com/puppetlabs/puppet/blob/master/lib/puppet/resource/catalog.rb#L90) and it seemed to work wonders.

Why not just go with that change? Am I missing something?

John

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/3c52ca61-15b1-48e7-a694-c3fafd70b11c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2014, September 20-24 in San Francisco - www.puppetconf.com

John Bollinger

unread,

Oct 7, 2014, 3:27:37 PM10/7/14

to puppe...@googlegroups.com

On Monday, October 6, 2014 5:13:43 PM UTC-5, Andy Parker wrote:

Sorry to resurrect an old thread, but this came to my attention again today.

On Fri, Mar 14, 2014 at 6:20 AM, John Bollinger <john.bo...@stjude.org> wrote:

On Thursday, March 13, 2014 8:50:19 AM UTC-5, Trevor Vaughan wrote:
SecondaryPackage wouldn't fix it if you wanted to install using pip and gem on the same system.

I see I should have devoted more text to my last statement: "The trick here would be that the provider(s) must not be based on package type, so that the package type could be used as part of a composite name." If the type's name were a composite of type (gem, pip, etc.) and name within that type, then it very well could support different package types all in one resource type. I suppose the individual package types could be features. Whereas such an approach cannot work for Package, it would be eminently workable for a unified SecondaryPackage type.

Putting it all in one type might make it a bit easier to convert existing manifests, and it would give users a single place to look for support for this sort of thing. On the other hand, the provider(s) would have to support multiple (secondary) package types. It's a trade-off between what aspects must be complicated and what parts can be simple.

I think there might be a much simpler solution to the entire thing. I noticed that all of the error messages that I've seen about this are about being unable to alias. What seems to be happening is that since the "name" parameter of a package resource is the namevar, the system is automatically creating an alias for the package resource using the name. That means that we have both a Package[title] reference and a Package[name] reference. The same thing occurred in the comment that was recently added to the ticket about the tidy type.

So here is my proposal: just remove the automatic aliasing. That means that the only way to reference a resource is via the title or an explicit alias. I tried this out on a VM by simply commenting out one line (https://github.com/puppetlabs/puppet/blob/master/lib/puppet/resource/catalog.rb#L90) and it seemed to work wonders.

Why not just go with that change? Am I missing something?

The main issue is name (not necessarily title or alias) collisions. If you want to manage an RPM named 'ntp' and also a gem named 'ntp' on the same system then you are hosed. The two resources have the same unique identifier ('ntp') which you pretty much have to use to map correctly to the physical resource. If you give up on requiring uniqueness of resources' natural unique identifiers then you throw in the towel altogether on avoiding duplicate resources.

The two names in my example actually live in different namespaces, but those namespaces map to resource providers, which traditionally are not specified in manifests nor known during catalog building, except for packages that are expected or known in advance to be supported by a "secondary" package provider. If this is a problem that's going to be solved then I still like the SecondaryPackage resource type for that task, or perhaps even separate Gem, etc. types.

John

Trevor Vaughan

unread,

Oct 7, 2014, 3:39:13 PM10/7/14

to puppe...@googlegroups.com

I'm still a fan of separate types.

Just put a nail in the coffin once and for all since this is really the only type which tries to get fancy.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/90cc269a-13c4-4736-99a8-c3c81bd94d1e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Daniele Sluijters

unread,

Oct 7, 2014, 10:10:11 PM10/7/14

to puppe...@googlegroups.com

Separate types undoes basically what "package" now tries to make as transparent as possible. I guess a case could be made to have "package" only deal with the OS native package provider and have separate types and providers for "secondary" packages like gem, wheel/egg/whatever etc.

Trevor Vaughan

unread,

Oct 8, 2014, 6:49:09 AM10/8/14

to puppe...@googlegroups.com

That's the fundamental issue though.

Package isn't a package. Package is one of many distinct things.

This is like the following:

user { 'foo': provider => 'passwd' }

user { 'foo': provider => 'apache' }

They're simply two different things so they would be user (i.e. native user) and apache_user (i.e. apache user).

We have:

package { 'foo': provider => 'native (yum, rpm, deb, whatever)' }

package { 'foo': provider => 'gem' }

These are NOT the same thing. One is a native package and one is "something else" and I believe that they should just be separate because they're "native" and "not native".

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/a9e37f89-3cc3-4c3e-a989-116b26317da5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Trevor Vaughan

unread,

Oct 8, 2014, 7:23:56 AM10/8/14

to puppe...@googlegroups.com

Sorry for the double post but I just thought of this.

1) The ldap provider probably needs to be removed from 'user'. Yes, they are users but they're not native, they could be one of many things.

2) If we do want to keep gem as a package provider, you'll need something like the following:

service { 'my_service': require => Package['mysql']{ :provider => 'gem' } }

This way, you end up with the correct resource relationship instead of being bound to the mysql RPM and the mysql Gem (or worse, neither because Puppet can't figure it out).

I am not a fan of this as I think it decreases readability. I would much rather have:

service { 'my_service': require => Package_Gem['mysql'] }

I made this package_gem just to keep things alphabetical for human code parsing.

I definitely understand the impulse behind wrapping all items behind a simple front but I think that it decreases readability and makes maintenance more difficult.

For instance, things that I can think of off the top of my head:

Types of Packages: http://en.wikipedia.org/wiki/List_of_software_package_management_systems

Types of Users: <the usual>, htaccess, htdigest, tomcat, ldap(posix), ldap(other), weblogic, mysql, postgres, dovecot, postfix, (pretty much you name it....)

Personally, I don't want all of these smashed into the same high level interface.

Trevor

Felix Frank

unread,

Oct 8, 2014, 7:55:19 AM10/8/14

to puppe...@googlegroups.com

On 10/08/2014 01:23 PM, Trevor Vaughan wrote:
>
> service { 'my_service': require => Package['mysql']{ :provider => 'gem' } }

How about require => Package<| title == 'mysql' and provider == 'gem' |>

Not that it matters - having two Package[mysql] resources will *never*
be a thing.

If I understand Andy correctly, we might have something like

package { 'mysql-gem': package_name => mysql, provider => gem }

...which is awful I guess. Anyway, relationship targets will not be
jeopardized (that I can see).

> Types of Users: <the usual>, htaccess, htdigest, tomcat, ldap(posix),
> ldap(other), weblogic, mysql, postgres, dovecot, postfix, (pretty much
> you name it....)
>
> Personally, I don't want all of these smashed into the same high level
> interface.

Hmm, I think I agree (right now...let's see about next week ;-)

But we *do* need a subtyping system then. You wouldn't want to implement
each of those as their own independent thing, would you?

Cheers,
Felix

Trevor Vaughan

unread,

Oct 8, 2014, 9:24:01 AM10/8/14

to puppe...@googlegroups.com

So, I would *love* to see proper inheritance for Types.

Currently, I can inherit Providers and do classic OO stuff but I can't do that with Types and it drives me a bit nuts.

If you go with the descriptive namevar, you'll need to make the separator something that won't show up in a package name. So probably one of '+/,|' etc...

The bright side of this is that you don't need to specify the provider either since it can be auto-determined from the namevar.

package { 'mysql+gem': ensure => 'foo' } ==> Name is known to be mysql and provider is known to be gem from splitting on '+'.

You can find an example of this here: https://github.com/onyxpoint/pupmod-concat/blob/master/lib/puppet/type/concat_fragment.rb

But, I still would prefer just inheriting types.

Puppet::Type.newtype(:package_gem, :parent => Puppet::Type::Package)

This would mean that I would obtain all methods, properties, and parameters from the parent Type and be able to override/extend as usual.

Providers would not be inherited, only the Type API.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/54352621.7050200%40alumni.tu-berlin.de.

For more options, visit https://groups.google.com/d/optout.

Felix Frank

unread,

Oct 8, 2014, 9:34:03 AM10/8/14

to puppe...@googlegroups.com

On 10/08/2014 03:23 PM, Trevor Vaughan wrote:
>
> If you go with the descriptive namevar, you'll need to make the
> separator something that won't show up in a package name. So probably
> one of '+/,|' etc...

That's not what I'm saying. I'm saying let the namevar no longer be the
package name at all, except when no explicit package name is given. I
thought that was what Andy was suggesting.

Cheers,
Felix

Trevor Vaughan

unread,

Oct 8, 2014, 9:36:31 AM10/8/14

to puppe...@googlegroups.com

It may be but, if we're already going to munge about with the namevar to make it unique, why not just use it without having to type more?

Trevor

Cheers,
Felix

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/54353D47.8040108%40alumni.tu-berlin.de.

For more options, visit https://groups.google.com/d/optout.

Message has been deleted

Drew Blessing

unread,

Oct 8, 2014, 9:42:00 AM10/8/14

to puppe...@googlegroups.com

On Tuesday, October 7, 2014 9:10:11 PM UTC-5, Daniele Sluijters wrote:

Separate types undoes basically what "package" now tries to make as transparent as possible. I guess a case could be made to have "package" only deal with the OS native package provider and have separate types and providers for "secondary" packages like gem, wheel/egg/whatever etc.

Reposting with quote for clarity -

This is the best idea I've heard. Certain puppet types intend to provide an abstraction. This abstraction only works for native package providers in this case because anything else requires the user to make an informed decision. I think having all other package providers be separate types is just fine - package_gem or gem or whatever route you want to go is fine.

It seems this might be the only sane and agreeable solution to a problem people have fought for years. Keep the current providers in the package type (but deprecate) and create the new providers. We have a great opportunity to do this with Puppet 4 coming up.

Andy Parker

unread,

Oct 8, 2014, 2:43:09 PM10/8/14

to puppe...@googlegroups.com

That is very close to what I was suggesting. My proposal is to allow manifest authors to break the tie between the namevar and the title when they need to. The current behavior is that the title *and* the namevar must always be unique within the set resources of the same type. What this means is that absolutely everything has to conform to the strict model of global uniqueness, which is exactly the problem here. By allowing the user to split this it allows an "escape hatch" whereby an author can decide how to manage the resources more directly.

My proposal allows

package { 'mysql-gem': name => mysql, provider => gem, ensure => installed }

package { 'mysql': ensure => installed }

Such statements would have previously collided because both would have created a reference of Package[mysql]. My proposal is that the Package[mysql-gem] no longer gets aliased as Package[mysql], which means that the collision never occurs.

John is right that this approach is a little bit like throwing in the towel. It does drop some of the uniqueness constraints that have been part of puppet for a while, but only in the case where an author has decided that they don't want those constraints, specifically in cases where the author has determined that the title should be different from the namevar. It does open up some unwanted catalogs, specifically things like:

package { 'remove-mysql': name => mysql, ensure => absent }

package { mysql: ensure => installed }

Is protecting against that absolutely needed?

## How does exec do this?

Exec is an interesting type. It doesn't seem to play by the rules of the other types. For example you can do:

exec { "one": command => "/bin/echo hi" }

exec { "two": command => "/bin/echo hi" }

This works in spite of the fact that "command" is the namevar for exec (if you leave out command, it defaults to the title). How does it do this?

The answer is that you can mark a type as "not isomorphic" (the default is that they are "isomorphic"). The only thing that this seems to control is to stop creating that pesky alias that gets in the way! There is a bit of documentation about this at https://github.com/puppetlabs/puppet/blob/master/lib/puppet/type.rb#L58-L61

Another solution is to mark package as not isomorphic as well. Based on other comments, user is also not isomorphic. Neither is tidy. Maybe others as well. Doing this would achieve the same thing as my original suggestion.

John Bollinger

unread,

Oct 8, 2014, 2:51:32 PM10/8/14

to puppe...@googlegroups.com

On Wednesday, October 8, 2014 6:55:19 AM UTC-5, Felix Frank wrote:

On 10/08/2014 01:23 PM, Trevor Vaughan wrote:
>
> service { 'my_service': require => Package['mysql']{ :provider => 'gem' } }

How about require => Package<| title == 'mysql' and provider == 'gem' |>

I filed a feature request years ago to support exactly that (Collections as rvalues). I didn't think it was ever implemented as such, but the chain operators cover more or less the same ground.

Not that it matters - having two Package[mysql] resources will *never*
be a thing.

If I understand Andy correctly, we might have something like

package { 'mysql-gem': package_name => mysql, provider => gem }

...which is awful I guess. Anyway, relationship targets will not be
jeopardized (that I can see).

It is more than awful. It either overloads the package title in dangerous ways, or else it deeply undermines Puppet's protections against duplicate resources. Consider, what is the meaning of this:

package { 'mysql-gem': package_name => 'mysql', provider => 'yum' }
?

Does it duplicate any or all of these resources?

package { 'mysql-yum': package_name => 'mysql', provider => 'yum' }
package { 'mysql-rpm': package_name => 'mysql', provider => 'rpm' }
package { 'mysql-gem': package_name => 'mysql' }
?

Do the the above Package['mysql-yum'] and Package['mysql-rpm'] conflict with each other? (They should.)

I can come up with bunches of similar issues.

John

Charlie Sharpsteen

unread,

Oct 8, 2014, 4:34:29 PM10/8/14

to puppe...@googlegroups.com

On Wednesday, October 8, 2014 11:51:32 AM UTC-7, John Bollinger wrote:

On Wednesday, October 8, 2014 6:55:19 AM UTC-5, Felix Frank wrote:

If I understand Andy correctly, we might have something like

package { 'mysql-gem': package_name => mysql, provider => gem }

...which is awful I guess. Anyway, relationship targets will not be
jeopardized (that I can see).

It is more than awful. It either overloads the package title in dangerous ways, or else it deeply undermines Puppet's protections against duplicate resources. Consider, what is the meaning of this:

package { 'mysql-gem': package_name => 'mysql', provider => 'yum' }
?

Does it duplicate any or all of these resources?

package { 'mysql-yum': package_name => 'mysql', provider => 'yum' }
package { 'mysql-rpm': package_name => 'mysql', provider => 'rpm' }
package { 'mysql-gem': package_name => 'mysql' }
?

Under Andy's proposal, the declaration about would duplicate the last declaration in the list, based on a title collision.

Do the the above Package['mysql-yum'] and Package['mysql-rpm'] conflict with each other? (They should.)

I can come up with bunches of similar issues.

However, this does require an act of intention on the part of the user to set "name" to a value that is different from "title". Is there a major risk of accidentally creating duplicate resources? Would the resulting catalog fail in a way that obscured the source of the problem?

To me, this proposal seams like the most pragmatic way to alleviate the problem without a major retool of how the Package type works. At the moment, I'm not convinced that opening up the possibility of accidental misuse outweighs the current issues surrounding the workarounds people have to use in order to install a package and a gem that happen to share the same name.

John Bollinger

unread,

Oct 8, 2014, 4:39:31 PM10/8/14

to puppe...@googlegroups.com

On Wednesday, October 8, 2014 1:43:09 PM UTC-5, Andy Parker wrote:

My proposal allows

package { 'mysql-gem': name => mysql, provider => gem, ensure => installed }
package { 'mysql': ensure => installed }

Such statements would have previously collided because both would have created a reference of Package[mysql]. My proposal is that the Package[mysql-gem] no longer gets aliased as Package[mysql], which means that the collision never occurs.

John is right that this approach is a little bit like throwing in the towel. It does drop some of the uniqueness constraints that have been part of puppet for a while, but only in the case where an author has decided that they don't want those constraints, specifically in cases where the author has determined that the title should be different from the namevar. It does open up some unwanted catalogs, specifically things like:

package { 'remove-mysql': name => mysql, ensure => absent }
package { mysql: ensure => installed }

Is protecting against that absolutely needed?

Are third-party packages desirable? Seriously, if I am to accept third-party packages without extensive code review (which I might not do anyway, I grant), then there's no way I I can accept that the module author, at his sole discretion, can slip in an unwanted duplicate resource. Yes, I do want to be protected against the same physical resource being managed via multiple distinct logical Resources. I like that.

## How does exec do this?

Exec is an interesting type. It doesn't seem to play by the rules of the other types. For example you can do:

exec { "one": command => "/bin/echo hi" }
exec { "two": command => "/bin/echo hi" }

This works in spite of the fact that "command" is the namevar for exec (if you leave out command, it defaults to the title). How does it do this?

The answer is that you can mark a type as "not isomorphic" (the default is that they are "isomorphic"). The only thing that this seems to control is to stop creating that pesky alias that gets in the way! There is a bit of documentation about this at https://github.com/puppetlabs/puppet/blob/master/lib/puppet/type.rb#L58-L61

Do not confuse the interface with the implementation. Execs are "not isomorphic" because different instances with the same command do not manage the same physical resource. That support for that status is implemented so simply is not relevant to the issue at all.

Another solution is to mark package as not isomorphic as well. Based on other comments, user is also not isomorphic. Neither is tidy. Maybe others as well. Doing this would achieve the same thing as my original suggestion.

That drives to the crux of the issue: packages and users are partially isomorphic but partially not. Package resources for certain providers (e.g. yum, apt) certainly are isomorphic when grouped by provider. Moreover, yum- and rpm-provided packages together form a single isomorphic group, so strict grouping by provider doesn't work. On the other hand, at least some providers (gem, pip) could divide isomorphic groups even more finely.

Users are an even thornier issue. Inasmuch as user information may be recorded in various places and forms, each with its own provider, (system) Users can be construed as non-isomorphic. From the perspective of modeling requirements on the system, however, only one User resource for each username matters, and I need to be able confidently to model the properties of that user. Having to guess which one it is is not appealing.

Other Topics covered so far
-------------------------------------

There have been a lot of other proposals floated. They either don't address the issue, or they don't solve it in any fashion that is complete enough based on various cases I've seen.

## Introduce a new pkgname parameter

I see this solution as equivalent to what I proposed above.

Agreed.

## Primary/Secondary packages

The proposals to have "primary" and "secondary" package types and split the providers seems like a losing battle to me. There just isn't a clear distinction. For instance RPM is on a lot of different OSes, but isn't the primary packaging system (SLES?). Often users want to manage both ruby gems and python pips on the same node, but both are considered secondary package management tools.

What package type is "primary" is OS- and possibly system-dependent, but that's nothing new. There is a clear distinction, else it never could have worked for so long to use Package resources without specifying providers explicitly.

As proposed, the SecondaryPackage type can accommodate instances with the same (packaging-type-relative) package name but different packaging types by using a provider so that it doesn't map to packaging type ( == package namespace).

## Drop Package, have a type per provider

The proposal for separate types entirely is interesting (and also throws in the towel). If I'm understanding it correctly we would stop trying to have a single type with multiple providers and instead have a single type has a single provider.

I had suggested the alternative of doing that for "secondary" packages. I'm uncertain about anyone else, but my own idea never involved dropping the Package type, which would still require multiple providers (but maybe not as many as it has now).

This would allow for the some more flexibility, but also makes writing reusable modules much more of a pain. You could no longer write:

package { 'apache': ensure => installed }

As I said, I did not contemplate dropping the package type. It serves a useful purpose.

However, in practice I'm not sure that that has really worked out too well. Just take a look at what it really takes to install apache: https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/package.pp .

Even with all the mess involved in determining the correct properties of any given Package, the Package resource still serves a useful role. It's a good and well-known abstraction for real software packages, it serves as a place to seat commonly-needed relationships, it generates events in a standardized way, .... It's even pretty easy to use for people who aren't concerned with supporting a great diversity of environments (which I think you would find is a substantial fraction of users).

## Providers are subtypes of the Package type

A variation of the last proposal is to allow types to be subtypes. What exactly that would mean isn't entirely clear.

Not clear to me, either.

I think it is that rpm_package would inherit all of the parameters/properties of package, which is great! It allows rpm specific parameters to be on an RPM type. However, how to references work? Does this work:

rpm_package { 'rpm-apache': name => 'apache', ensure => installed }

file { '/etc/apache/httpd.conf': require => Package[apache] }

I had not supposed that allowing subtyping would involve supporting polymorphic resource references or collectors. In part, that's because I agree that result would not be substantially different than what we have now.

Thoughts
---------------

I looked into removing the constraint on unique namevars after noticing that all examples of users trying to get around this problem involved them setting a unique title and a conflicting namevar. That indicated that the mental model of most users is that the titles need to be unique and the explicit namevar value is the escape hatch. We can make that expectation work by either dropping the aliasing for the namevar universally or by marking certain types as not-isomorphic.

Your observations agree with mine, but consider an alternative interpretation: users on the whole don't understand the distinction between title and namevar, nor especially the significance of the namevar as distinguished from the significance of the title. It doesn't help that the most common bona fide scenario for differing title and namevar seems to be on Execs, which are excused from following some of the rules. I don't think many users have a sufficiently-developed mental model of Puppet to see namevars as an escape hatch, or as anything else, really. I suspect many don't even recognize the that the concept exists. (Too many people simply don't read docs.)

The essence of this problem is that people are trying to describe completely valid configurations on a node, but the constraints imposed by puppet's modeling system does not allow these configurations.

Yes.

A classic problem of type systems. My proposal allows some invalid configurations in addition to the valid configurations. I haven't seen any proposal yet that allows the valid configurations the people want as well as disallowing the invalid configurations.

The SecondaryPackage approach can do that, particularly if it is developed and maintained in conjunction with Package, with the proviso that under certain unlikely circumstances the conflict might not be recognized until catalog application.

Since this problem has gone on for so long, I'm not sure if we *will* find such a solution and so think we should just move ahead with either dropping the aliasing completely or making package non-isomorphic. I'd like to get this fixed in puppet 4.0.0. So as a way of driving to a conclusion: what would making package non-isomorphic break?

It would make Puppet unable to diagnose genuinely conflicting Package declarations, possibly resulting in unwanted or unstable system configurations, resource flapping, and even operational failure of managed systems. Probably most relevant failure scenarios would involve conflicting 'ensure' parameters. Some would likely involve the prevalent antipattern of using defined() to try to work around package collisions, either directly (the defined() check is no longer effective at all) or indirectly (the defined() check triggers on resources it was not intended to catch).

The most likely vector for introducing such failures into a Puppet environment is the same one that is responsible now for introducing most resource collisions: installation or update of third-party packages. It is unlikely to be clear to users, however, that they need to look out for (near-)silent package-related failures after working on their module collections, nor are they likely to be patient about such problems when they are discovered. This could be a significant setback for the module ecosystem.

John

Felix Frank

unread,

Oct 9, 2014, 10:12:41 AM10/9/14

to puppe...@googlegroups.com

On 10/08/2014 10:34 PM, Charlie Sharpsteen wrote:
> On Wednesday, October 8, 2014 11:51:32 AM UTC-7, John Bollinger wrote:
>
>
>
> On Wednesday, October 8, 2014 6:55:19 AM UTC-5, Felix Frank wrote:
>
> package { 'mysql-gem': package_name => mysql, provider => gem }
>
> ...which is awful I guess. Anyway, relationship targets will not be
> jeopardized (that I can see).
>
>
>
> It is more than awful. It either overloads the package title in
> dangerous ways, or else it deeply undermines Puppet's protections
> against duplicate resources. Consider, what is the meaning of this:
>
> package { 'mysql-gem': package_name => 'mysql', provider => 'yum' }
> ?

Sorry, I should have been more clear. The resource title is supposed to
be arbitrary here. These manage the same resource:

package { 'mysql-gem': package_name => 'mysql', provider => 'gem' }
package { 'mysql-foo': package_name => 'mysql', provider => 'gem' }
package { 'apache': package_name => 'mysql', provider => 'gem' }

> Does it duplicate any or all of these resources?
>
> package { 'mysql-yum': package_name => 'mysql', provider => 'yum' }
> package { 'mysql-rpm': package_name => 'mysql', provider => 'rpm' }
> package { 'mysql-gem': package_name => 'mysql' }
> ?

Only titles clash, so these three could share one catalog. If yum is
your default provider, then the first and third do manage the same resource.

> To me, this proposal seams like the most pragmatic way to alleviate the
> problem without a major retool of how the Package type works. At the
> moment, I'm not convinced that opening up the possibility of accidental
> misuse outweighs the current issues surrounding the workarounds people
> have to use in order to install a package and a gem that happen to share
> the same name.

Thanks for this summary Charlie, it mirrors my feeling quite exactly.

If we open Pandora's box, users will have ample new opportunity to shoot
their own feet. I don't think that there can be a solution that prevents
abuse in the form of conflicting resources, but we do allow a use case
that we know is problematic for several if not many users.

So in response to Andy's request for a pick, I feel that making packages
non-isomorphic and allow namevar != title would be a fair compromise.

package { 'mysql-foo': name => 'mysql', provider => 'gem' }

Yes this might get abused by Forge modules. Nothing we can do about
that, as far as I can tell.

Cheers,
Felix

John Bollinger

unread,

Oct 9, 2014, 6:10:55 PM10/9/14

to puppe...@googlegroups.com

On Thursday, October 9, 2014 9:12:41 AM UTC-5, Felix Frank wrote:

On 10/08/2014 10:34 PM, Charlie Sharpsteen wrote:
> On Wednesday, October 8, 2014 11:51:32 AM UTC-7, John Bollinger wrote:
>
>
>
> On Wednesday, October 8, 2014 6:55:19 AM UTC-5, Felix Frank wrote:
>
> package { 'mysql-gem': package_name => mysql, provider => gem }
>
> ...which is awful I guess. Anyway, relationship targets will not be
> jeopardized (that I can see).
>
>
>
> It is more than awful. It either overloads the package title in
> dangerous ways, or else it deeply undermines Puppet's protections
> against duplicate resources. Consider, what is the meaning of this:
>
> package { 'mysql-gem': package_name => 'mysql', provider => 'yum' }
> ?

Sorry, I should have been more clear. The resource title is supposed to
be arbitrary here. These manage the same resource:

package { 'mysql-gem': package_name => 'mysql', provider => 'gem' }
package { 'mysql-foo': package_name => 'mysql', provider => 'gem' }
package { 'apache': package_name => 'mysql', provider => 'gem' }

> Does it duplicate any or all of these resources?
>
> package { 'mysql-yum': package_name => 'mysql', provider => 'yum' }
> package { 'mysql-rpm': package_name => 'mysql', provider => 'rpm' }
> package { 'mysql-gem': package_name => 'mysql' }
> ?

Only titles clash, so these three could share one catalog. If yum is
your default provider, then the first and third do manage the same resource.

In fact, then, if yum is the default provider then all three manage the same resource.

> To me, this proposal seams like the most pragmatic way to alleviate the
> problem without a major retool of how the Package type works. At the
> moment, I'm not convinced that opening up the possibility of accidental
> misuse outweighs the current issues surrounding the workarounds people
> have to use in order to install a package and a gem that happen to share
> the same name.

Thanks for this summary Charlie, it mirrors my feeling quite exactly.

That depends heavily on how you characterize the problem. To me, the clash between packages of different types is primarily a facet of the more fundamental problem that Ruby gems, Python modules, and similar self-contained, distributable nuggets of software simply are not "packages" in the same way that DEBs are on a Debian machine or RPMs are on a RedHat-family machine.

The Package type does not need any retooling. Instead, a means needs to be provided for people to avoid overloading it with responsibility for managing things it was not designed to handle. I don't even care if all the current Package providers are retained (who knows, maybe some day there will be a RubyOS with gems as the native package type), but where gem (or <insert-name-here>) is not the native packaging type, such modules should not be managed via Package resources.

If we open Pandora's box, users will have ample new opportunity to shoot
their own feet. I don't think that there can be a solution that prevents
abuse in the form of conflicting resources, but we do allow a use case
that we know is problematic for several if not many users.

If making packages non-isomorphic were the only viable way to serve that use case then I would find that argument more persuasive. It isn't.

So in response to Andy's request for a pick, I feel that making packages
non-isomorphic and allow namevar != title would be a fair compromise.

package { 'mysql-foo': name => 'mysql', provider => 'gem' }

Yes this might get abused by Forge modules. Nothing we can do about
that, as far as I can tell.

I'm not so much worried about abuse as about well-intentioned and seemingly reasonable use that mixes badly with other well-intentioned and seemingly reasonable use. Hypothetical examples:

(1)
Module A declares
    package { 'foo-gem': name => 'foo', ensure => '1.0', provider => 'gem' }
Module B declares
    package { 'gem-foo': name => 'foo', ensure => '2.0', provider => 'gem' }
Result is that either A or B breaks.

(2)
Module A declares
    package { 'web-server': name => 'httpd-server', ensure => '2.0.12' }
Module B declares
    package { 'httpd-server': ensure => '2.4.0' }
Again, either A or B breaks.

One of Puppet's major features is that it avoids damaging managed systems systems by being conservative about what configuration specifications it is willing to accept. That trait is far more valuable to me than an ability to use specifically the Package type to manage gems etc..

John

Andy Parker

unread,

Oct 9, 2014, 7:36:49 PM10/9/14

to puppe...@googlegroups.com

I may have misunderstood your proposal.

If we open Pandora's box, users will have ample new opportunity to shoot
their own feet. I don't think that there can be a solution that prevents
abuse in the form of conflicting resources, but we do allow a use case
that we know is problematic for several if not many users.

If making packages non-isomorphic were the only viable way to serve that use case then I would find that argument more persuasive. It isn't.

So in response to Andy's request for a pick, I feel that making packages
non-isomorphic and allow namevar != title would be a fair compromise.

package { 'mysql-foo': name => 'mysql', provider => 'gem' }

Yes this might get abused by Forge modules. Nothing we can do about
that, as far as I can tell.

I'm not so much worried about abuse as about well-intentioned and seemingly reasonable use that mixes badly with other well-intentioned and seemingly reasonable use. Hypothetical examples:

(1)
Module A declares
    package { 'foo-gem': name => 'foo', ensure => '1.0', provider => 'gem' }
Module B declares
    package { 'gem-foo': name => 'foo', ensure => '2.0', provider => 'gem' }
Result is that either A or B breaks.

(2)
Module A declares
    package { 'web-server': name => 'httpd-server', ensure => '2.0.12' }
Module B declares
    package { 'httpd-server': ensure => '2.4.0' }
Again, either A or B breaks.

One of Puppet's major features is that it avoids damaging managed systems systems by being conservative about what configuration specifications it is willing to accept. That trait is far more valuable to me than an ability to use specifically the Package type to manage gems etc..

I may not be understanding your proposal. Can you explain how SecondaryPackage solves this problem? The way I understand it it just pushes the problem down the road a little:

package { 'mysql': ensure => installed }

secondary_package { 'mysql': ensure => installed, provider => gem }

secondary_package { 'mysql': ensure => installed, provider => pip }

That would conflict for the secondary_package resources. We could try not unifying the different providers under a single name:

package { 'mysql': ensure => installed }

gem { 'mysql': ensure => installed }

pip { 'mysql': ensure => installed }

That might solve the problem. Are there cases where it won't work? I suppose with this scheme we have the ability to make package specific types. The pip type could support virtualenv, for example. Would pip and gem still be available as providers for the package type? If they are available, then we have problems like this:

package { 'mysql': ensure => '2.0.1', provider => gem }

gem { 'mysql': ensure => '2.4.0' }

If they aren't available, then we've probably broken a lot of manifests. Would doing this split result in difficulties making references to installed software resources (I'm avoiding using the term packages)? One example where it would be difficult is something that installed a package via yum on RedHat boxes and gem everywhere else.

There is also the question about what constitutes a native packaging system and what doesn't.

I think this proposal might have some promise, but it seems like it opens up a lot of arbitrary decisions and unravels into a lot of different problems. How deep does the rabbit hole go?

John, could you succinctly outline how you think how we could go about doing this and what drawbacks and advantages it might have?

John

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/4cd4be7c-c0ce-4700-b0b3-8dafe7bfdf16%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan

unread,

Oct 9, 2014, 8:28:45 PM10/9/14

to puppe...@googlegroups.com

I'm still a big fan of this for clarity and ease of use:

That would conflict for the secondary_package resources. We could try not unifying the different providers under a single name:
package { 'mysql': ensure => installed }
gem { 'mysql': ensure => installed }
pip { 'mysql': ensure => installed }

To me, this allows for the correct semantics across the board and is blatantly/painfully clear.

At some point, when Type inheritance works, the code could be combined and solidified.

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANhgQXs-pcUaDsfkK0ic8GQ82rFaPfQ0x8smVFLZxQEwebDreg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

John Bollinger

unread,

Oct 10, 2014, 9:26:05 AM10/10/14

to puppe...@googlegroups.com

Hmm. The primary objective was to solve precisely that problem, which it aimed to accomplish by making the package type a standard parameter and part of a compound resource name. Probably multiple package types would then be served by SecondaryPackage providers; if different providers are needed for different platforms then the supported package types might map to features. Examples:

package { 'mysql': ensure => 'installed' }

secondary_package { 'mysql': type => 'gem', ensure => 'installed' }
secondary_package { 'mysql': type => 'pip', ensure => 'installed', provider => 'for_demo_purposes_provider' }
secondary_package { 'mysql': type => 'perl', ensure => 'installed', provider => 'for_demo_purposes_provider' }

I now recognize that a title collision problem would make this less useful than I had first thought, but it is still better than making Package non-isomorphic, and Package still doesn't need any retooling. The same applies to managing secondary packages via their own types as Trevor favors: Package does not need any retooling with that approach, especially not making it non-isomorphic. I'd be fine with that. I greatly favor anything else over making Package non-isomorphic.

John

Drew Blessing

unread,

Oct 10, 2014, 9:31:30 AM10/10/14

to puppe...@googlegroups.com

I greatly favor anything else over making Package non-isomorphic.

I completely appreciate what you're saying. However, I don't think this was the intent of the provider abstraction. It was intended to make the OS-specific provider transparent, right? The different types of packages we're talking about really don't fit that model. To have a package and secondary_package type seems more confusing than it should be, where the solution proposed by Daniele and Andy seems straightforward IMO.

John Bollinger

unread,

Oct 13, 2014, 9:39:12 AM10/13/14

to puppe...@googlegroups.com

On Friday, October 10, 2014 8:31:30 AM UTC-5, Drew Blessing wrote:

I greatly favor anything else over making Package non-isomorphic.

I completely appreciate what you're saying. However, I don't think this was the intent of the provider abstraction. It was intended to make the OS-specific provider transparent, right? The different types of packages we're talking about really don't fit that model.

I'm not sure I follow. It sounds like you are saying that using explicitly-specified providers to select among different packaging types is not consistent with the intent of providers. If so, then I completely agree. I am arguing that weakening the Package abstraction to make it more convenient to abuse its providers in that way is the worst possible solution to the name clash problem. Worse even than doing nothing about it at all.

That is why the Secondary_package proposal does not rely on selecting package type via provider, but instead makes the package type a separate, ordinary parameter.

To have a package and secondary_package type seems more confusing than it should be, where the solution proposed by Daniele and Andy seems straightforward IMO.

Let's not get hung up on names, if that's what's bothering you. My essential thesis is that the Package type should represent packages for the OS's native packaging system as effectively as possible, including supporting Puppet's conventional constraints against multiple declaration. If that does not afford adequate support for alternative kinds of packages then the desired supported should be implemented via one or more alternative resource types. Whether such an alternative type is called "Secondary_package" or "Module" or "Software", or whether there are several ("Gem", "Pip", "Cpan", ...) is very much a secondary concern to me.

John

Drew Blessing

unread,

Oct 13, 2014, 9:56:42 AM10/13/14

to puppe...@googlegroups.com

Whether such an alternative type is called "Secondary_package" or "Module" or "Software", or whether there are several ("Gem", "Pip", "Cpan", ...) is very much a secondary concern to me.

John

Perfect. I understand what you're saying and I agree.

Charlie Sharpsteen

unread,

Oct 13, 2014, 7:13:53 PM10/13/14

to puppe...@googlegroups.com

So, to recap, the issue with making packages non-isomorphic is that the title becomes the only method to enforce uniqueness of resources.

We may be able to solve the problem of name clashes and retain stronger guarantees by switching the Package type to use a composite namevar instead of dropping isomorphism.

I took a crack at this over the weekend and the required changes turned out to be very simple. A work in progress patch can be found here:

https://gist.github.com/Sharpie/3b2b12d9b3ef2cea6837

This change resolves example #1 by using the combination of [name, provider] to enforce uniqueness, instead of the current [name]. The following clash would no longer be possible:

# Composite key: ['foo', 'gem']

package { 'foo-gem': name => 'foo', ensure => '1.0', provider => 'gem' }

# Same composite key: ['foo', 'gem']

package { 'gem-foo': name => 'foo', ensure => '2.0', provider => 'gem' }

Example #2 is also resolved:

# Composite key: ['httpd-server', nil]

package { 'httpd-server': ensure => '2.4.0' }

# Same composite key: ['httpd-server', nil]

package { 'web-server': name => 'httpd-server', ensure => '2.0.12' }

The possibility for conflict still exists between providers that happen to manage the same pool of packages and between implicit and explicit use of the default provider. For example, the following will result in competing resources on RedHat:

# Composite key: ['httpd-server', nil]

package { 'httpd-server': ensure => installed }

# Composite key: ['httpd-server', 'yum']

package { 'httpd': ensure => absent, name => 'httpd-server', provider => 'yum' }

# Composite key: ['httpd-server', 'rpm']

package { 'webserver': ensure => '2.4.0', name => 'httpd-server', provider => 'rpm' }

So, a composite key does not provide an airtight guarantee of uniqueness but is better than dropping isomorphism. We may be able to improve this situation by turning missing composite key values into smart defaults when the agent prepares a catalog for application.

Thoughts on using a composite namevar as an alternative to dropping isomorphism?

Erik Dalén

unread,

Oct 14, 2014, 3:42:12 AM10/14/14

to Puppet Developers

Seems good to me. But tbh I was okay with just dropping isomorphism on packages as well.

--
Erik Dalén

Trevor Vaughan

unread,

Oct 14, 2014, 8:13:43 AM10/14/14

to puppe...@googlegroups.com

I think that dropping isomorphism would be clearer to end users.

It's not all that much more maintenance either really.

--

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAAAzDLdBexTyBmvzq30vdwAo884xseU0rauZzpMcOVzNaO65mg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Andy Parker

unread,

Oct 14, 2014, 5:18:56 PM10/14/14

to puppe...@googlegroups.com

On Tue, Oct 14, 2014 at 5:13 AM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

I think that dropping isomorphism would be clearer to end users.

Maybe, maybe not. I don't think that they'll often see a difference between the two.

Since dropping constraints is easier than adding constraints, I think we'll go with Charlie's suggestion. If it turns out that it still doesn't work in enough cases, then we can either make the package type non-isomorphic or start heading down the path of individual types.

Yay! A thread that has reached a conclusion! :D

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoWTLtRMk1MHsLPHdKSFGBQomhdjbrohJg6%3DsETT_OdiZA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Luke Kanies

unread,

Oct 15, 2014, 1:24:57 AM10/15/14

to puppe...@googlegroups.com

I’m very opposed to dropping the isomorphism, especially for such a critical type. I’d rather do the necessary work to make it a composite namevar. So many other assumptions about the world start to break down if you let that slip that it’s hard to predict what the consequences will be.

--

http://puppetlabs.com/ | http://about.me/lak | @puppetmasterd

John Bollinger

unread,

Oct 15, 2014, 9:39:22 AM10/15/14

to puppe...@googlegroups.com

On Monday, October 13, 2014 6:13:53 PM UTC-5, Charlie Sharpsteen wrote:

On Thursday, October 9, 2014 3:10:55 PM UTC-7, John Bollinger wrote:

On Thursday, October 9, 2014 9:12:41 AM UTC-5, Felix Frank wrote:
So in response to Andy's request for a pick, I feel that making packages
non-isomorphic and allow namevar != title would be a fair compromise.

package { 'mysql-foo': name => 'mysql', provider => 'gem' }

Yes this might get abused by Forge modules. Nothing we can do about
that, as far as I can tell.

I'm not so much worried about abuse as about well-intentioned and seemingly reasonable use that mixes badly with other well-intentioned and seemingly reasonable use. Hypothetical examples:

(1)
Module A declares
    package { 'foo-gem': name => 'foo', ensure => '1.0', provider => 'gem' }
Module B declares
    package { 'gem-foo': name => 'foo', ensure => '2.0', provider => 'gem' }
Result is that either A or B breaks.

(2)
Module A declares
    package { 'web-server': name => 'httpd-server', ensure => '2.0.12' }
Module B declares
    package { 'httpd-server': ensure => '2.4.0' }
Again, either A or B breaks.

One of Puppet's major features is that it avoids damaging managed systems systems by being conservative about what configuration specifications it is willing to accept. That trait is far more valuable to me than an ability to use specifically the Package type to manage gems etc..

John

So, to recap, the issue with making packages non-isomorphic is that the title becomes the only method to enforce uniqueness of resources.

I'd put it a bit differently: the issue with making packages non-isomorphic is that puppet then cannot enforce uniqueness of resources. Resource titles are arbitrary.

We may be able to solve the problem of name clashes and retain stronger guarantees by switching the Package type to use a composite namevar instead of dropping isomorphism.

I like that better than making Package non-isomorphic. In fact, a composite namevar is an essential component of my proposed Secondary_package resource type, for pretty much the same purpose Charlie describes.

I did not propose doing this for Package itself primarily because it doesn't completely prevent duplicate Package resources, as Charlie himself observes (with exactly the example I would have offered myself).

I reiterate that the root problem here is that gems, pips, Perl modules, etc. are not "packages" in the same sense that packages of the OS's native type are. Weakening the Package type to better support them at the expense of continued strong support for the native package type is an unwelcome trade off. The composite namevar approach offers less of a trade off in that dimension than does making Package non-isomorphic, but still a trade-off.

Drawing a clear distinction between primary and secondary packages via resource types provides an opportunity for more advantages than just solving the package name issue. A separate type or types for secondary packages can provide features that don't make sense for primary packages. For example, they could provide for distinguishing between gems of the same (gem-)name installed into different Ruby installations.

John

Reply all

Reply to author

Forward

Package duplicate resource issue - PUP-1073

Drew Blessing

Trevor Vaughan

Adrien Thebo

Trevor Vaughan

Pedro Côrte-Real

Trevor Vaughan

David Schmitt

Trevor Vaughan

David Schmitt

Trevor Vaughan

Drew Blessing

John Bollinger

Trevor Vaughan

Adrien Thebo

Drew Blessing

Trevor Vaughan

Drew Blessing

Trevor Vaughan

David Schmitt

Andy Parker

Pedro Côrte-Real

Trevor Vaughan

Drew Blessing

David Schmitt

Thomas Linkin

John Bollinger

Pedro Côrte-Real

Trevor Vaughan

Felix Frank

Trevor Vaughan

Drew Blessing

Trevor Vaughan

John Bollinger

Trevor Vaughan

Felix Frank

Trevor Vaughan

John Bollinger

Andy Parker

John Bollinger

Trevor Vaughan

Daniele Sluijters

Trevor Vaughan

Trevor Vaughan

Felix Frank

Trevor Vaughan

Felix Frank

Trevor Vaughan

Drew Blessing

Andy Parker

John Bollinger

Charlie Sharpsteen

John Bollinger

Felix Frank

John Bollinger

Andy Parker

Trevor Vaughan

John Bollinger

Drew Blessing

John Bollinger

Drew Blessing

Charlie Sharpsteen

Erik Dalén

Trevor Vaughan

Andy Parker

Luke Kanies

John Bollinger