Jira (PUP-11452) puppetserver_gem doesn't install gems when they are loaded by facter

14 views
Skip to first unread message

Jarret Lavallee (Jira)

unread,
Feb 7, 2022, 10:32:02 AM2/7/22
to puppe...@googlegroups.com
Jarret Lavallee created an issue
 
Puppet / Bug PUP-11452
puppetserver_gem doesn't install gems when they are loaded by facter
Issue Type: Bug Bug
Affects Versions: PUP 7.13.1
Assignee: Unassigned
Components: Types and Providers
Created: 2022/02/07 7:31 AM
Priority: Normal Normal
Reporter: Jarret Lavallee

The puppetserver_gem package provider fails to list the correct gem list in puppetserver when there is a gem installed with `puppet_gem` and added by a `require` during the agent run or apply. In this scenario, the gem is listed as already installed by the puppetserver_gem provider when it is not installed in the puppetserver gem space.

Logs
The gem is only installed into the puppet gem space

# puppetserver gem list aws-sdk-s3
 
*** LOCAL GEMS ***
 
# /opt/puppetlabs/puppet/bin/gem list aws-sdk-s3
 
*** LOCAL GEMS ***
 
aws-sdk-s3 (1.112.0)

Running the agent or apply shows the gem is prefetched in the puppetserver_gem provider

# puppet apply install.pp --debug |& grep aws-sdk-s3
Debug: /Package[aws-sdk-s3]: Provider puppetserver_gem does not support features targetable; not managing attribute command
Debug: /Package[aws-sdk-s3]: Provider puppetserver_gem does not support features install_only; not managing attribute install_only
Debug: /Package[aws-sdk-s3]: Provider puppetserver_gem does not support features virtual_packages; not managing attribute allow_virtual
Debug: Puppet::Type::Package::ProviderPuppetserver_gem: Prefetched instance: aws-sdk-s3 via command: default

Reproduction

  1. On a 7.x Puppetserver - PE 2021.4.0 in this example
  2. Install the gem using the `puppet_gem` provider: puppet resource package aws-sdk-s3 ensure=installed provider=puppet_gem
  3. Add the following fact into the environment and ensure it is downloaded with puppet plugin download

Facter.add(:testing) do
  setcode do
    require 'aws-sdk-s3'
    Aws::S3::GEM_VERSION
  end
end

  1. Apply the following manifest to observe that the puppetserver_gem package is not installed

     package{'aws-sdk-s3':
      ensure => installed,
      provider => puppetserver_gem
    }
    

    Potential Fix
    This looks to be an issue due to the puppetserver_gem not forking the gem list command, so the loaded gems are listed in these lines.

If I replace those lines with one that uses the /opt/puppetlabs/puppet/bin/gem command, the issue no longer appears. For example:

output = execute_gem_command(Puppet::Type::Package::ProviderPuppet_gem.provider_command, ["list","--local"], gem_env)
gem_list = output.lines.reject { |gem| gem =~ / \(default\: / }

There is a significant difference in the number of gems listed between the two outputs.

# The current implementation
> sio_out.string.lines.reject { |gem| gem =~ / \(default\: / }.length
=> 56
# Shelling out to `gem`
> output.lines.reject { |gem| gem =~ / \(default\: / }.length
=> 25

This aligns with the `puppetserver gem list` and `gem list` commands.

# puppetserver gem list |grep -v default | wc -l
25
# /opt/puppetlabs/puppet/bin/gem list |grep -v default | wc -l
56

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

Austin Boyd (Jira)

unread,
Feb 7, 2022, 10:32:03 AM2/7/22
to puppe...@googlegroups.com

Jarret Lavallee (Jira)

unread,
Feb 7, 2022, 10:36:02 AM2/7/22
to puppe...@googlegroups.com
Jarret Lavallee updated an issue
Change By: Jarret Lavallee
Zendesk Ticket Count: 1
Zendesk Ticket IDs: 47197

Jarret Lavallee (Jira)

unread,
Feb 7, 2022, 10:36:02 AM2/7/22
to puppe...@googlegroups.com
Jarret Lavallee updated an issue
The {{puppetserver_gem}} package provider fails to list the correct gem list in puppetserver when there is a gem installed with `puppet_gem` and added by a `require` during the agent run or apply. In this scenario, the gem is listed as already installed by the {{puppetserver_gem}} provider when it is not installed in the puppetserver gem space.

*Logs*

The gem is only installed into the puppet gem space
{code}# puppetserver gem list aws-sdk-s3


*** LOCAL GEMS ***

# /opt/puppetlabs/puppet/bin/gem list aws-sdk-s3

*** LOCAL GEMS ***

aws-sdk-s3 (1.112.0)
{code}

Running the agent or apply shows the gem is prefetched in the {{puppetserver_gem}} provider
{code}# puppet apply install.pp --debug |& grep aws-sdk-s3

Debug: /Package[aws-sdk-s3]: Provider puppetserver_gem does not support features targetable; not managing attribute command
Debug: /Package[aws-sdk-s3]: Provider puppetserver_gem does not support features install_only; not managing attribute install_only
Debug: /Package[aws-sdk-s3]: Provider puppetserver_gem does not support features virtual_packages; not managing attribute allow_virtual
Debug: Puppet::Type::Package::ProviderPuppetserver_gem: Prefetched instance: aws-sdk-s3 via command: default
{code}
*Reproduction*
1) On a 7.x Puppetserver - PE 2021.4.0 in this example
2) Install the gem using the `puppet_gem` provider: {{puppet resource package aws-sdk-s3 ensure=installed provider=puppet_gem}}
3) Add the following fact into the environment and ensure it is downloaded with {{puppet plugin download}}

{code}Facter.add(:testing) do

  setcode do
    require 'aws-sdk-s3'
    Aws::S3::GEM_VERSION
  end
end
{code}

4) Apply the following manifest to observe that the {{puppetserver_gem}} package is not installed

{code} package{'aws-sdk-s3':

  ensure => installed,
  provider => puppetserver_gem
}
{code}


*Potential Fix*
This looks to be an issue due to the {{puppetserver_gem}} not forking the gem list command, so the [loaded gems are listed in these lines|https://github.com/puppetlabs/puppet/blob/main/lib/puppet/provider/package/puppetserver_gem.rb#L162-L170].


If I replace those lines with one that uses the {{/opt/puppetlabs/puppet/bin/gem}} command, the issue no longer appears. For example:
{code}output = execute_gem_command(Puppet::Type::Package::ProviderPuppet_gem.provider_command, ["list","--local"], gem_env)

gem_list = output.lines.reject { |gem| gem =~ / \(default\: / }
{code}

There is a significant difference in the number of gems listed between the two outputs.
{code}# The current implementation

> sio_out.string.lines.reject { |gem| gem =~ / \(default\: / }.length
=> 56
# Shelling out to `gem`
> output.lines.reject { |gem| gem =~ / \(default\: / }.length
=> 25
{code}

This aligns with the `puppetserver gem list` and `gem list` commands.
{code}# puppetserver /opt/puppetlabs/puppet/bin/ gem list |grep -v default | wc - l
25
l56 # /opt/puppetlabs/puppet/bin/ puppetserver gem list |grep -v default | wc -l
56 25
{code}

Jarret Lavallee (Jira)

unread,
Feb 7, 2022, 10:36:02 AM2/7/22
to puppe...@googlegroups.com
Jarret Lavallee updated an issue
The {{puppetserver_gem}} package provider fails to list the correct gem list in puppetserver when there is a gem installed with `puppet_gem` and added by a `require` during the agent run or apply. In this scenario, the gem is listed as already installed by the {{puppetserver_gem}} provider when it is not installed in the puppetserver gem space.

*Logs*
The gem is only installed into the puppet gem space
{code}# puppetserver gem list aws-sdk-s3

*** LOCAL GEMS ***

# /opt/puppetlabs/puppet/bin/gem list aws-sdk-s3

*** LOCAL GEMS ***

aws-sdk-s3 (1.112.0)
{code}
Running the agent or apply shows the gem is prefetched in the {{puppetserver_gem}} provider
{code}# puppet apply install.pp --debug |& grep aws-sdk-s3
Debug: /Package[aws-sdk-s3]: Provider puppetserver_gem does not support features targetable; not managing attribute command
Debug: /Package[aws-sdk-s3]: Provider puppetserver_gem does not support features install_only; not managing attribute install_only
Debug: /Package[aws-sdk-s3]: Provider puppetserver_gem does not support features virtual_packages; not managing attribute allow_virtual
Debug: Puppet::Type::Package::ProviderPuppetserver_gem: Prefetched instance: aws-sdk-s3 via command: default
{code}
*Reproduction*
# 1) On a 7.x Puppetserver - PE 2021.4.0 in this example
# 2) Install the gem using the `puppet_gem` provider: {{puppet resource package aws-sdk-s3 ensure=installed provider=puppet_gem}}
# 3) Add the following fact into the environment and ensure it is downloaded with {{puppet plugin download}}


{code}Facter.add(:testing) do
  setcode do
    require 'aws-sdk-s3'
    Aws::S3::GEM_VERSION
  end
end
{code}
#
4)
Apply the following manifest to observe that the {{puppetserver_gem}} package is not installed

{code} package{'aws-sdk-s3':
  ensure => installed,
  provider => puppetserver_gem
}
{code}


*Potential Fix*
This looks to be an issue due to the {{puppetserver_gem}} not forking the gem list command, so the [loaded gems are listed in these lines|https://github.com/puppetlabs/puppet/blob/main/lib/puppet/provider/package/puppetserver_gem.rb#L162-L170].

If I replace those lines with one that uses the {{/opt/puppetlabs/puppet/bin/gem}} command, the issue no longer appears. For example:
{code}output = execute_gem_command(Puppet::Type::Package::ProviderPuppet_gem.provider_command, ["list","--local"], gem_env)
gem_list = output.lines.reject { |gem| gem =~ / \(default\: / }
{code}
There is a significant difference in the number of gems listed between the two outputs.
{code}# The current implementation
> sio_out.string.lines.reject { |gem| gem =~ / \(default\: / }.length
=> 56
# Shelling out to `gem`
> output.lines.reject { |gem| gem =~ / \(default\: / }.length
=> 25
{code}
This aligns with the `puppetserver gem list` and `gem list` commands.
{code}# puppetserver gem list |grep -v default | wc -l
25
# /opt/puppetlabs/puppet/bin/gem list |grep -v default | wc -l
56
{code}

Josh Cooper (Jira)

unread,
Feb 8, 2022, 5:30:02 PM2/8/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11452
 
Re: puppetserver_gem doesn't install gems when they are loaded by facter

Thanks for digging into this issue jarret.lavallee I agree that it's better to shell out than try to list gems "in process" as the puppetserver_gem provider is currently doing. Could you submit your changes (https://github.com/jarretlavallee/puppetlabs-puppetserver_gem/commit/327c7c90f40874d3c7d5f31d8708cf12366cca63) as a PR to puppet#6.x? We can help with the tests if needed.

Jarret Lavallee (Jira)

unread,
Feb 9, 2022, 10:27:01 AM2/9/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Apr 6, 2022, 8:54:01 PM4/6/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Apr 6, 2022, 9:08:01 PM4/6/22
to puppe...@googlegroups.com

Lisa Ross (Jira)

unread,
Apr 7, 2022, 11:15:02 AM4/7/22
to puppe...@googlegroups.com
Lisa Ross updated an issue
Change By: Lisa Ross
Fix Version/s: PUP 7.17.0
Fix Version/s: PUP 6.28.0

Nirupama Mantha (Jira)

unread,
Apr 13, 2022, 11:45:01 AM4/13/22
to puppe...@googlegroups.com

Christopher Thorn (Jira)

unread,
Apr 15, 2022, 12:56:01 PM4/15/22
to puppe...@googlegroups.com

Christopher Thorn (Jira)

unread,
Apr 15, 2022, 12:56:02 PM4/15/22
to puppe...@googlegroups.com
Christopher Thorn updated an issue
Change By: Christopher Thorn
Sprint: Phoenix DevInfraOps 2022- 05 04 - 11 27

Josh Cooper (Jira)

unread,
Apr 19, 2022, 11:39:03 AM4/19/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: DevInfraOps Phoenix 2022-04-27

Josh Cooper (Jira)

unread,
Apr 26, 2022, 10:09:02 PM4/26/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes: Bug Fix
Release Notes Summary: Fixes a bug that prevented the "puppetserver_gem" provider from managing gems that were first loaded by facter.

Parker Leach (Jira)

unread,
May 23, 2022, 12:47:02 PM5/23/22
to puppe...@googlegroups.com
Parker Leach updated an issue
Change By: Parker Leach
Labels: docs_reviewed jira_escalated
Reply all
Reply to author
Forward
0 new messages