Jira (PUP-7197) make type aliases from modules available on the agent

13 views
Skip to first unread message

David Schmitt (JIRA)

unread,
Feb 9, 2017, 4:33:03 AM2/9/17
to puppe...@googlegroups.com
David Schmitt created an issue
 
Puppet / Story PUP-7197
make type aliases from modules available on the agent
Issue Type: Story Story
Assignee: Unassigned
Created: 2017/02/09 1:32 AM
Priority: Normal Normal
Reporter: David Schmitt

As a provider author,
I want to have access to type aliases in code running on the agent
So that I can avoid duplication and drift in my code

Currently DSL data type aliases are only available with direct access to the installed module. For both the netconf, and new Resource API efforts, it is necessary to transport these definitions to agents (presumably via pluginsync) so that their infrastructure code can rely on the definitions.

CC Rick Sherman Thomas Honey Henrik Lindberg Thomas Hallgren

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Thomas Honey (JIRA)

unread,
Feb 9, 2017, 4:36:02 AM2/9/17
to puppe...@googlegroups.com

John Duarte (JIRA)

unread,
Feb 22, 2017, 11:49:02 AM2/22/17
to puppe...@googlegroups.com
John Duarte updated an issue
 
Change By: John Duarte
Team: Puppet Developer Experience

John Duarte (JIRA)

unread,
May 17, 2017, 2:59:06 PM5/17/17
to puppe...@googlegroups.com

Bert Hajee (JIRA)

unread,
Aug 9, 2019, 5:02:03 AM8/9/19
to puppe...@googlegroups.com
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Josh Cooper (JIRA)

unread,
Aug 9, 2019, 2:16:03 PM8/9/19
to puppe...@googlegroups.com
Josh Cooper commented on Story PUP-7197

David Schmitt could you provide more detail around how having access to type aliases on the agent "avoid duplication and drift in my code"? How would the provider running on the agent use the type aliases?

Bert Hajee (JIRA)

unread,
Aug 10, 2019, 3:34:03 AM8/10/19
to puppe...@googlegroups.com
Bert Hajee commented on Story PUP-7197

I'll try and describe my issue Josh Cooper.

The resource API allows you to specify a data type for the parameters and properties of a custom type and provider. The resource API takes care of typing the values returned from the provider and checks the values from the manifest against the specified type.

On a regular puppetserver this works with both the basic types but also with type aliases. A problem, however, arises when using

$ puppet resource my_custom_type

Another use case is when I use bolt to execute a task that uses (parts of) puppet to execute on a remote system. Again it cannot find the specified data types and fails.

On an agent system. The type alias is not available there and so the provider fails with a message it cannot find the data type. The workaround we use now is to only use basic data types and copy the value from the type alias. 

This workaround causes duplication of the type alias throughout the custom type code.

Hope this helps.

 

Regards

Corey Osman (JIRA)

unread,
Aug 13, 2019, 3:16:03 PM8/13/19
to puppe...@googlegroups.com
Corey Osman commented on Story PUP-7197

If I want to use the type aliases without the resource API so I can run code like this.  Without the datatype on the agent this will not work.

 

newparam :path do
    # includes windows and unix paths
    desc 'Absolute path to repository'
    isnamevar
    validate do |value|
      datatype =  Puppet::Pops::Types::TypeParser.singleton.parse("Stdlib::AbsolutePath")
      raise ArgumentError, "Path must be absolute: #{path}" unless datatype.instance?(value)
    end
end

Josh Cooper (JIRA)

unread,
Aug 13, 2019, 6:31:03 PM8/13/19
to puppe...@googlegroups.com
Josh Cooper commented on Story PUP-7197

Thanks Bert Hajee and Corey Osman. So I'm specifically interested in why datatypes need to be available on the agent, which implies that they need to be pluginsync'ed. Some comments below:

A problem, however, arises when using

$ puppet resource my_custom_type

That should print the current state of the system as "retrieved" by the provider. When would data types be needed to generate that?

when I use bolt to execute a task

I would expect the type aliases to be copied along with the custom type and providers. I assume bolt doesn't know to copy them like it does with lib, facts.d, etc?

I want to use the type aliases without the resource API

Why validate the data type in the provider, if it's already been validated at compile time?

Corey Osman (JIRA)

unread,
Aug 13, 2019, 8:35:04 PM8/13/19
to puppe...@googlegroups.com
Corey Osman commented on Story PUP-7197

"Why validate the data type in the provider, if it's already been validated at compile time?" 

Correct, it would already be compiled and not required on majority of cases, But if puppet resource was used and someone like me was doing

stupid stuff like this in the type code that puppet resource <type> would fail.

Puppet::Pops::Types::TypeParser.singleton.parse("Stdlib::AbsolutePath").instance?(value)

 

This is an extremely rare case as nobody is doing this and it would only fail on puppet resource. I was just wanting to shoehorn a method to validate input without the awesome resource_api.

Take my request with a grain of salt.

Bert Hajee (JIRA)

unread,
Aug 14, 2019, 3:09:03 AM8/14/19
to puppe...@googlegroups.com
Bert Hajee commented on Story PUP-7197

Why validate the data type in the provider, if it's already been validated at compile time?

The type is. not only checked when the catalog is compiled, but the type is also used to validate the value returned by the provider. 

I would expect the type aliases to be copied along with the custom type and providers. I assume bolt doesn't know to copy them like it does with libfacts.dfilestemplates, etc?

 

This happens when you use bolt and agentless puppet. But you can also execute pp files as tasks using the local installation of puppet agent. Then it will compile the manifest local on the agent.

 

Henrik Lindberg (JIRA)

unread,
Aug 14, 2019, 4:54:03 AM8/14/19
to puppe...@googlegroups.com

The data types are very convenient not only for validating the input - a resource type may for example read some system state and take different action depending on the obtained value. If there already exists a defined data type (say with some complicated regex pattern) you would want to reuse that rather than having to make a duplicate or code the check in Ruby.

To me, this is a very reasonable feature to add - question is more if the price (modifying plugin sync) makes it worth the trouble.

Alexander Fisher (JIRA)

unread,
Feb 18, 2020, 11:15:07 AM2/18/20
to puppe...@googlegroups.com

This is also needed so that functions that use type aliases can be 'Deferred'.

Alexander Fisher (JIRA)

unread,
Feb 18, 2020, 11:30:08 AM2/18/20
to puppe...@googlegroups.com

The lack of the pluginsync had an especially nasty side-effect in the code I was working on.

Deferred('vault_lookup::lookup',['a/valid/path', 'https://a.valid.url'])

ends up calling the wrong dispatch. (It should have matched lookup_without_key but ends up matching lookup_with_key instead).

Puppet::Functions.create_function(:'vault_lookup::lookup') do
  dispatch :lookup_without_key do
    param 'String', :path
    optional_param 'Stdlib::HTTPUrl', :vault_url
  end
 
  dispatch :lookup_with_key do
    param 'String', :path
    param 'String', :key
    optional_param 'Stdlib::HTTPUrl', :vault_url
  end
 
...

Henrik Lindberg (JIRA)

unread,
Feb 18, 2020, 3:32:04 PM2/18/20
to puppe...@googlegroups.com

In case you wonder why that happens Alexander Fisher (or anyone else for that matter) is that by definition in the specification that any mentioned data type is a Resource (type) reference unless there is a specific data type available for that name. It would have been reasonable for the type reference to simply not resolve and an error to be raised for the `Stdlib::HTTPUrl` data type.

David Schmitt (JIRA)

unread,
Feb 19, 2020, 5:02:05 AM2/19/20
to puppe...@googlegroups.com
David Schmitt commented on Story PUP-7197

Josh Cooper how much effort would it be to add the types module directory to pluginsync and how much more load would it put on the puppetserver?

I've done a quick survey of supported and voxpupuli modules:

  • accounts, apache, apt, ntp and stdlib have 62 type aliases together (more than half of that in stdlib)
  • voxpupuli has 32 (out of 134) modules with aggregate 209 aliases

looking over the results, I've also noticed that the files are not properly namespaced (i.e. Stdlib::Base64 is in puppetlabs-stdlib/types/base64.pp (instead of puppetlabs-stdlib/types/stdlib/base64.pp) so a naive pluginsync would lead to potential problems, if another module also has a base64.pp type alias. Maybe a stupid idea, but would it be possible/feasible to generate a PCore representation of all those aliases in a single file that gets sent to the agent instead?

Henrik Lindberg (JIRA)

unread,
Feb 19, 2020, 6:48:05 AM2/19/20
to puppe...@googlegroups.com

I looked into syncing more things - pluginsync builds a Puppet catalog that gets applied on the agent so it is a matter of adding more resources to that catalog that plugin sync builds.

It would be doable to create type metadata for a module and sync that, but you would still need to plugin sync that information and would then need to add a loader on the agent side to make use of that information. One possibility would be to let `puppet generate types` generate this information.

Since PCore can serialize types and type aliases it could possibly also include them in the catalog - but then the question is which types to include.

Best is to do it with pluginsync, and even better would be to redesign puppet sync.

Lee Lowder (Jira)

unread,
Apr 28, 2020, 1:24:03 PM4/28/20
to puppe...@googlegroups.com
Lee Lowder commented on Story PUP-7197

Another area where having type aliases (and puppet language functions) plugin sync are in the use of deferred functions and deferred templates.

Say I have an epp template that takes two parameters - a String for the password, and an Stdlib::Absolutepath for some config file or a binary.

If I use Deferred on the template so I can have the password come out of vault or some other secret management tool, then because type aliases are not pluginsnced, when the template is rendered on the agent side, I get an error about unknown type Stdlib::Absolutepath.

 

The same sort of thing happens if I try to deferr a puppet language based function - something that the docs have been pushing people very hard to start using over the last few years and versions.

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Apr 28, 2020, 7:45:04 PM4/28/20
to puppe...@googlegroups.com
Josh Cooper commented on Story PUP-7197

Adding a new mount for sync'ing types isn't hard, but I think the way stdlib defines the types is a problem, since they end up as lib/types/httpurl.pp instead of lib/types/stdlib/httpurl.pp and the agent-side cache loader expects to find them in the latter, since they're specified as param 'Stdlib::HTTPUrl', :url in puppet code. If stdlib properly namespaced the types, like it does with deferred functions, e.g. lib/puppet/functions/<module>/myfunction.pp, then it wouldn't be hard to sync and load types on the agent.

In $codedir/environments/production/module/mymodule/lib/puppet/functions/mymodule/myupcase.rb

Puppet::Functions.create_function(:'mymodule::myupcase') do
  dispatch :from_url do
    param 'Stdlib::HTTPUrl', :url
  end
 
  def from_url(url)
    url
  end
end

In $codedir/environments/production/module/mymodule/manifests/init.pp

class mymodule {
  $d = Deferred("mymodule::myupcase", ["http://puppet.com"])
 
  notify { example :
    message => $d
  }
}

In $codedir/environments/production/manifests/site.pp

class { 'mymodule': }

Then using https://github.com/joshcooper/puppet/commit/ff2d146a88adb03a69431e182f04cbdc6ebc87ca and moving httpurl.rb into a stdlib directory, puppet can evaluate a deferred function and load types in the process:

$ bx puppet agent -t --server localhost
...
Notice: /File[/Users/josh/.puppetlabs/opt/puppet/cache/lib/types]/ensure: removed
Info: Retrieving locales
Info: Retrieving types
Notice: /File[/Users/josh/.puppetlabs/opt/puppet/cache/lib/types]/ensure: created
...
Notice: http://puppet.com
Notice: /Stage[main]/Mymodule/Notify[example]/message: defined 'message' as 'http://puppet.com'

Note there is a problem with plugins and types syncing to the same directory (they get deleted and redownloaded each time). We could sync them to different directories, but the agent's cache loader would need to load from multiple places, or we could create an agent-loader specifically for types.

David Schmitt (Jira)

unread,
Apr 29, 2020, 10:45:04 AM4/29/20
to puppe...@googlegroups.com
David Schmitt commented on Story PUP-7197

Josh Cooper your comment about the stdlib types not being in the "correct" directory seems to raise another issue: as the stdlib types are laid out currently works with puppet apply:

david@zion:~/git/puppetlabs-stdlib (master)$ ls spec/fixtures/modules/stdlib/types/
absolutepath.pp  compat       filesource.pp  httpsurl.pp  mac.pp          port               unixpath.pp
base32.pp        ensure       fqdn.pp        httpurl.pp   objectstore     port.pp            windowspath.pp
base64.pp        filemode.pp  host.pp        ip           objectstore.pp  syslogfacility.pp  yes_no.pp
david@zion:~/git/puppetlabs-stdlib (master)$ pdk bundle exec puppet apply --modulepath spec/fixtures/modules --verbose -e 'notice("https://foo" =~ Stdlib::HTTPSUrl)'
pdk (INFO): Using Ruby 2.5.7
pdk (INFO): Using Puppet 6.13.0
Info: Loading facts
Info: Loading facts
Notice: Scope(Class[main]): true
Notice: Compiled catalog for zion in environment production in 0.05 seconds
Info: Applying configuration version '1588171313'
Notice: Applied catalog in 0.02 seconds
david@zion:~/git/puppetlabs-stdlib (master)$ 

whereas moving all types to a stdlib subfolder would not:

david@zion:~/git/puppetlabs-stdlib (master)$ mkdir types/stdlib
david@zion:~/git/puppetlabs-stdlib (master)$ mv types/* types/stdlib/
mv: cannot move 'types/stdlib' to a subdirectory of itself, 'types/stdlib/stdlib'
david@zion:~/git/puppetlabs-stdlib (master)$ ls spec/fixtures/modules/stdlib/types/
stdlib
david@zion:~/git/puppetlabs-stdlib (master)$ ls spec/fixtures/modules/stdlib/types/stdlib/
absolutepath.pp  compat       filesource.pp  httpsurl.pp  mac.pp          port               unixpath.pp
base32.pp        ensure       fqdn.pp        httpurl.pp   objectstore     port.pp            windowspath.pp
base64.pp        filemode.pp  host.pp        ip           objectstore.pp  syslogfacility.pp  yes_no.pp
david@zion:~/git/puppetlabs-stdlib (master)$ pdk bundle exec puppet apply --modulepath spec/fixtures/modules --verbose -e 'notice("https://foo" =~ Stdlib::HTTPSUrl)'
pdk (INFO): Using Ruby 2.5.7
pdk (INFO): Using Puppet 6.13.0
Info: Loading facts
Info: Loading facts
Error: Evaluation Error: Resource type not found: Stdlib::HTTPSUrl (line: 1, column: 25) on node zion
david@zion:~/git/puppetlabs-stdlib (master)$ 

not sure what's happening there.

Josh Cooper (Jira)

unread,
Apr 29, 2020, 1:26:03 PM4/29/20
to puppe...@googlegroups.com
Josh Cooper commented on Story PUP-7197

Ah, it's because puppet apply loads the types using a different loader, which has visibility to all of the modules in the load path for the environment and doesn't require them to be namespaced in the same way. But types on the agent-side have the same issue as 4x functions called via deferred like https://github.com/voxpupuli/puppet-vault_lookup/blob/master/lib/puppet/functions/vault_lookup/lookup.rb — the function needs to be in a directory named after the function.

It may be possible to preserve the module namespace for the types mount and create a new agent-side loader that doesn't have the same limitation as the cache loader. I'll take a look.

Josh Cooper (Jira)

unread,
Jun 8, 2020, 2:30:04 PM6/8/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Puppet / New Feature PUP-7197
Change By: Josh Cooper
Issue Type: Story New Feature

Aristarkh Zagorodnikov (Jira)

unread,
Nov 13, 2020, 7:58:03 AM11/13/20
to puppe...@googlegroups.com
Aristarkh Zagorodnikov commented on New Feature PUP-7197
 
Re: make type aliases from modules available on the agent

Very much looking forward to this since we would like to use deferred inline templates without losing custom types.

Maggie Dreyer (Jira)

unread,
Apr 28, 2021, 2:35:04 PM4/28/21
to puppe...@googlegroups.com
Maggie Dreyer updated an issue
 
Change By: Maggie Dreyer
Epic Link: PUP- 8587 11032
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Maggie Dreyer (Jira)

unread,
Apr 28, 2021, 2:35:07 PM4/28/21
to puppe...@googlegroups.com

David McTavish (Jira)

unread,
Dec 6, 2021, 12:22:02 PM12/6/21
to puppe...@googlegroups.com

David McTavish (Jira)

unread,
Jan 12, 2022, 7:52:01 PM1/12/22
to puppe...@googlegroups.com
David McTavish updated an issue
Change By: David McTavish
Labels: final_triage nice-to-have
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

David McTavish (Jira)

unread,
Jan 12, 2022, 7:52:02 PM1/12/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Jan 12, 2022, 10:48:02 PM1/12/22
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-7197
 
Re: make type aliases from modules available on the agent

I think this is still very important, though pluginsyncing one file at a time is not performant, so much so that PE has implemented a bulk pluginsync workaround, see https://tickets.puppetlabs.com/browse/PUP-8600?focusedCommentId=796724&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-796724. I think we need to fix pluginsync, at which point this ticket would be easy to fix.

Josh Cooper (Jira)

unread,
Feb 22, 2022, 12:30:02 PM2/22/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Feb 22, 2022, 2:46:02 PM2/22/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Labels: final_triage nice-to-have

Josh Cooper (Jira)

unread,
Jun 7, 2022, 12:15:03 PM6/7/22
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages