Jira (PUP-9707) Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins

[Nick Walker (JIRA)]

Nick Walker created an issue

Puppet / PUP-9707 Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins Issue Type: Improvement Assignee: Unassigned Created: 2019/05/15 10:24 AM Priority: Normal Reporter: Nick Walker Background In PE we added the ability for the agent to bulk download plugins before installing the agent via the frictionless installation script. On the master, a tarball of the plugins is created and placed in a directory for jetty to serve so the frictionless agent install script can pull it down. Suggested Improvement The instead of only doing this via the installation script we could move the logic into the puppet-agent so that the agent could detect that no plugins are present and instead of doing a one by one download the bulk pluginsync should be performed. Out of scope My suggestion would be that this functionality is added first with an assumption that the tarball will already be present on the master and it can be put there by a separate process other than puppetserver itself making the tarball. Then in the future we can improve puppetserver to make the tarball if we want to. SERVER-2539 Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Nick Walker (JIRA)]

Nick Walker updated an issue

Puppet / PUP-9707 Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins

Change By: Nick Walker h1.  Background

In PE we added the ability for the agent to bulk download plugins before installing the agent via the frictionless installation script.   On the master, a tarball of the plugins is created and placed in a directory for jetty to serve so the frictionless agent install script can pull it down.

h1.  Suggested Improvement

The instead of only doing this via the installation script we could move the logic into the puppet-agent so that the agent could detect that no plugins are present and instead of doing a one by one download the bulk pluginsync should be performed.

h1.  Out of scope

My suggestion would be that this functionality is added first with an assumption that the tarball will already be present on the master and it can be put there by a separate process other than puppetserver itself making the tarball.  Then in the future we can improve puppetserver to make the tarball if we want to.  SERVER-2539

h1.  Impact In order to show the impact of using the current bulk_pluginsync implementation we install the puppetlabs-windows module and then perform pluginsync via puppet or via bulk_pluginsync.   When we look at the results we can see a puppet agent run without bulk_pluginsync takes around 90 seconds and with bulk_pluginsync it takes about 14 seconds.  The download and extraction of the pluginsync tarball takes under 1 second and a reasonable part of the agent run after bulk_pluginsync seems to be validating the plugins.   Puppet-agent ( one file at a time ) ``` {code} /etc/puppetlabs/code/environments/production/modules └─┬ puppetlabs-windows (v6.1.0)   ├── puppet-download_file (v3.2.0)   ├── puppet-windows_env (v3.2.0)   ├── puppet-windowsfeature (v3.2.2)   ├── puppetlabs-acl (v2.1.0)   ├─┬ puppetlabs-chocolatey (v3.3.0)   │ ├── puppetlabs-powershell (v2.3.0)   │ ├── puppetlabs-registry (v2.1.0)   │ └── puppetlabs-stdlib (v5.2.0)   ├─┬ puppetlabs-dsc (v1.9.2)   │ └── puppetlabs-reboot (v2.1.2)   ├── puppetlabs-iis (v4.5.1)   └── puppetlabs-wsus_client (v1.1.0) [root@master201910-centos ~]# [root@master201910-centos ~]# [root@master201910-centos ~]# [root@master201910-centos ~]# rm -rf /opt/puppetlabs/puppet/cache/ facts.d/       lib/           locales/       preview/       puppet-module/ state/ [root@master201910-centos ~]# rm -rf /opt/puppetlabs/puppet/cache/ [root@master201910-centos ~]# time puppet agent -t Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/aio_agent_build.rb]/ensure: defined content as '{md5}cdcc1ff07bc245c66cc1d46be56b3af5' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/aio_agent_version.rb]/ensure: defined content as '{md5}d05c8cbf788f47d33efd46a935dda61e' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/choco_install_path.rb]/ensure: defined content as '{md5}36ece697e936cc950f8946ca2ac11729' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/choco_temp_dir.rb]/ensure: defined content as '{md5}299356b5cbc6f0aed554294a69824c00' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/chocolateyversion.rb]/ensure: defined content as '{md5}941fd5e3ab6f1d54a5886d803d19bf22' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/facter_dot_d.rb]/ensure: defined content as '{md5}9f65108ecd7b8c5ce9c90ab4cd9d45be' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/iis_version.rb]/ensure: defined content as '{md5}8642e756efacbdbcca8d5ccc93026101' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/inventory.rb]/ensure: defined content as '{md5}e6c76fe99ac2a64fac1b168f36d47235' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/package_provider.rb]/ensure: defined content as '{md5}12d57e65f9b90c13e7808c347fbdc0b0' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_build.rb]/ensure: defined content as '{md5}ee54c728457b32d6622c3985448918fa' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_concat_basedir.rb]/ensure: defined content as '{md5}0ccd3500f29b9dd346a45a61268c7c18' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_postgresql_info.rb]/ensure: defined content as '{md5}d65daa8950859bc7557b064203356565' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_razor_server_version.rb]/ensure: defined content as '{md5}ec91d8b92e03d5f952c789308d26dcd0' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_server_version.rb]/ensure: defined content as '{md5}17c2795fe8a56b731ae0fc81ba147e6a' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/pe_version.rb]/ensure: defined content as '{md5}245bb8ea121e8a55e1a38668cc480530' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/platform_symlink_writable.rb]/ensure: defined content as '{md5}fc1e2766ff9994fa5df95cdc14b9bcd2' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/platform_tag.rb]/ensure: defined content as '{md5}2f3b09a0f0b539117456e311ad28f190' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/powershell_version.rb]/ensure: defined content as '{md5}50bab11a2d7c70d7cbbae618f060ceb8' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/puppet_files_dir_present.rb]/ensure: defined content as '{md5}3900e124be2f377638dd1522079856bf' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/puppet_settings.rb]/ensure: defined content as '{md5}d86bacf2b962a4744b701868cda31729' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/root_home.rb]/ensure: defined content as '{md5}b384092f0470b9e2177edaf546de2ec1' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/service_provider.rb]/ensure: defined content as '{md5}66cc42526eae631e306b397391f1f01c' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/should_install_shiro_ini.rb]/ensure: defined content as '{md5}a932ec299c2a33762e2713b57be9facd' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/staging_http_get.rb]/ensure: defined content as '{md5}2c27beb47923ce3acda673703f395e68' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/uses_win32console.rb]/ensure: defined content as '{md5}074d6dbc1aa6699acc1662a333a7eb3c' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/util]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/util/puppet_settings.rb]/ensure: defined content as '{md5}f69007bcc02031cd772fa7f2dffa3c0c' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/windows.rb]/ensure: defined content as '{md5}d8880f6f32905f040f3355e2a40cf088' Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/windows_env.rb]/ensure: defined content as '{md5}d1d3a97851e655e4fc9c6a4945e0e670' Notice: /File[/opt/puppetlabs/puppet/cache/lib/hiera]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/hiera/backend]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/hiera/backend/hocon_backend.rb]/ensure: defined content as '{md5}47ff267a77a66b85dd98258c6e08a127' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/application]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/application/enterprise.rb]/ensure: defined content as '{md5}b0e6f1b94578dde29a3af39fe7652fb0' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/face]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/face/enterprise]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/face/enterprise.rb]/ensure: defined content as '{md5}c447263b49afebada8a4706654925a74' ... Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/powershell_manager.rb]/ensure: defined content as '{md5}e4194779ea96a79651ce0f82ed5180b5' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/powershell_version.rb]/ensure: defined content as '{md5}519ebd8f8b718c1afb2ce42f75b85c2a' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/hash.rb]/ensure: defined content as '{md5}1fb84a18a918aae2e76a3225aa0a2dbd' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/name.rb]/ensure: defined content as '{md5}473f314d46724c31b1f496b59031f64a' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/path.rb]/ensure: defined content as '{md5}fd3e53ba75d36e095d051ffded98c133' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/positive_integer.rb]/ensure: defined content as '{md5}9317877654cf840b4d10e912f1d20285' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/read_only.rb]/ensure: defined content as '{md5}297c977e0339cb123040998917400c70' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/string.rb]/ensure: defined content as '{md5}bf1cc4e3fc4cd8cd3159dd9c7bc8bf54' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/iis/property/timeformat.rb]/ensure: defined content as '{md5}5d16de4de8c0c77bdfbed2f0c3f63f90' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/config]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/config.rb]/ensure: defined content as '{md5}95b89bf089e175f17ee9047ca4f0fee6' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/config/modify.rb]/ensure: defined content as '{md5}c5d7317c5a67dadac2f7770f32633591' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/config/services.rb]/ensure: defined content as '{md5}aa0c7cd67e156bdb9fd8ce1560c17f03' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/defaults.rb]/ensure: defined content as '{md5}137d7250ef9872d5adb2f79a7767aa97' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/hiera_adapter.rb]/ensure: defined content as '{md5}c914281cb82008ad28b7bf81d0909a9a' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/meep_scope.rb]/ensure: defined content as '{md5}a6af7e4ca161715f37ec9213bf23745a' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/scope.rb]/ensure: defined content as '{md5}90beadb6836522b94e30dc717834dcb5' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/meep/validate.rb]/ensure: defined content as '{md5}ec3acfa929bd8d0001c43276da9be870' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell/compatible_powershell_version.rb]/ensure: defined content as '{md5}d85ac07f9b23964804f400737c846fb8' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell/powershell_manager.rb]/ensure: defined content as '{md5}1d6efc75a7fb8f95b02307a74e135fd3' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell/powershell_version.rb]/ensure: defined content as '{md5}0c8298e41ff0216f1f6ad32d2a5ce010' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell_manager.rb]/ensure: defined content as '{md5}71fc45bf2d401b718561080ac43bb0d2' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/powershell_version.rb]/ensure: defined content as '{md5}662d10a58e5e4512f3e59bbd72bf4732' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/registry.rb]/ensure: defined content as '{md5}1132a6f66c4e7d549ac90f9f58bd73e2' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script/v1]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script/v1/puppet-enterprise-support.ps1]/ensure: defined content as '{md5}167b5cbb2af5909322e474425d404fa5' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script/v1/puppet-enterprise-support.sh]/ensure: defined content as '{md5}5d3869be504e11fcfd44124a68ab1991' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script/v3]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/support_script/v3/puppet-enterprise-support.rb]/ensure: defined content as '{md5}4376e0c821978eb58b8070fcd27959dd' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/dsc]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/dsc/init_ps.ps1]/ensure: defined content as '{md5}ddaf94c38f20a293e040aed56d755edb' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/iis]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/iis/init_ps.ps1]/ensure: defined content as '{md5}afa7397ca63c05a6fd35425191cc2d90' Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/powershell]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/templates/powershell/init_ps.ps1]/ensure: defined content as '{md5}065293e35b7d035f5c54a26146106fec' Notice: /File[/opt/puppetlabs/puppet/cache/lib/shared]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/lib/shared/aio_build.rb]/ensure: defined content as '{md5}0b71fe86c349f3f41401313c0ac03a71' Notice: /File[/opt/puppetlabs/puppet/cache/lib/shared/pe_build.rb]/ensure: defined content as '{md5}4f4652af20c4f0391b9ca2976940a710' Notice: /File[/opt/puppetlabs/puppet/cache/lib/shared/pe_server_version.rb]/ensure: defined content as '{md5}f3d3fc8776512ae73d3293c97b8f3dfe' Info: Retrieving locales Notice: /File[/opt/puppetlabs/puppet/cache/locales/ja]/ensure: created Notice: /File[/opt/puppetlabs/puppet/cache/locales/ja/puppetlabs-stdlib.po]/ensure: defined content as '{md5}805e5d893d2025ad57da8ec0614a6753' Info: Loading facts Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::pe_metric_curl_cron_jobs for master201910-centos.puppetdebug.vlan (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 21, column: 5) on node master201910-centos.puppetdebug.vlan Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run real 1m44.931s user 0m25.433s sys 0m5.240s ... real 1m30.611s user 0m24.526s sys 0m5.231s ... real 1m29.392s user 0m24.535s sys 0m5.022s ``` {code} bulk_pluginsync ( download a tarball and extract it) ``` {code} [root@master201910-centos ~]# time ./bulk_psync.sh bulk downloading plugins extracting plugins Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::pe_metric_curl_cron_jobs for master201910-centos.puppetdebug.vlan (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 21, column: 5) on node master201910-centos.puppetdebug.vlan Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run real 0m13.542s user 0m7.250s sys 0m0.647s ... real 0m14.502s user 0m7.206s sys 0m0.767s ... real 0m13.055s user 0m6.994s sys 0m0.574s ``` {code}

Add Comment

[Nick Walker (JIRA)]

Nick Walker updated an issue

Puppet / PUP-9707 Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins Change By: Nick Walker

h1.  Background In PE we added the ability for the agent to bulk download plugins before installing the agent via the frictionless installation script.   On the master, a tarball of the plugins is created and placed in a directory for jetty to serve so the frictionless agent install script can pull it down.   h1.  Suggested Improvement The instead of only doing this via the installation script we could move the logic into the puppet-agent so that the agent could detect that no plugins are present and instead of doing a one by one download the bulk pluginsync should be performed.   h1.  Out of scope My suggestion would be that this functionality is added first with an assumption that the tarball will already be present on the master and it can be put there by a separate process other than puppetserver itself making the tarball.  Then in the future we can improve puppetserver to make the tarball if we want to.  SERVER-2539

h1.  Impact In order to show the impact of using the current bulk_pluginsync implementation we install the puppetlabs-windows module and then perform pluginsync via puppet or via bulk_pluginsync.   When we look at the results we can see a puppet agent run without bulk_pluginsync takes around 90 seconds and with bulk_pluginsync it takes about 14 seconds.  The download and extraction of the pluginsync tarball takes under 1 second and a reasonable part of the agent run after bulk_pluginsync seems to be validating the plugins.   Puppet-agent ( one file at a time ) ```

bulk_pluginsync ( download a tarball and extract it) ```

[root@master201910-centos ~]# time ./bulk_psync.sh bulk downloading plugins extracting plugins Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::pe_metric_curl_cron_jobs for master201910-centos.puppetdebug.vlan (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 21, column: 5) on node master201910-centos.puppetdebug.vlan Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run real 0m13.542s user 0m7.250s sys 0m0.647s ... real 0m14.502s user 0m7.206s sys 0m0.767s ... real 0m13.055s user 0m6.994s sys 0m0.574s ```

Add Comment

[Jorie Tappa (JIRA)]

Jorie Tappa updated an issue

Puppet / PUP-9707 Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins

Change By: Jorie Tappa Team: Coremunity

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-9707 Add ability for puppet-agent to detect an empty pluginsync directory and bulk download plugins

Change By: Josh Cooper Epic Link: PUP-11032

Add Comment

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)