Jira (PUP-11184) Autoloader is confused by Windows 8.3 paths

27 views
Skip to first unread message

Josh Cooper (Jira)

unread,
Jul 22, 2021, 7:56:03 PM7/22/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Puppet / Bug PUP-11184
Autoloader is confused by Windows 8.3 paths
Change By: Josh Cooper
*Puppet Version: 7.9.0*

The autoloader can load the same file twice if the ruby load path contains Windows 8.3 style paths. The file {{lib/puppet/file_system/uniquefile.rb}} creates a class whose parent is an anonymous class. If the file is loaded twice, then it will fail with: {{superclass mismatch for class Uniquefile (TypeError)}}.

To reproduce, install puppet-agent on Windows:
{noformat}C:\> cmd /c start /w msiexec /qn /i http://builds.puppetlabs.lan/puppet-agent/7.9.0/artifacts/windows/puppet-agent-7.9.0-x64.msi
C:\> net stop puppet
C:\> cd C:\PROGRA~1\PUPPET~1\Puppet\puppet\bin
{noformat}
create "puppettest.rb" containing:
{code:ruby}require 'puppet'
Puppet.initialize_settings
puts "RUBYLIBDIR #{RbConfig::CONFIG['rubylibdir']}"
puts "Loaded Puppet"
pp $LOADED_FEATURES.grep(/uniquefile/).first
Puppet::Type.type(:exec)
{code}
And execute the script:
{noformat}C:\PROGRA~1\PUPPET~1\Puppet\puppet\bin>ruby puppettest.rb
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/facter/util/file_helper.rb:9: warning: already initialized constant Class::DEBUG_MESSAGE
C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/facter/util/file_helper.rb:9: warning: previous definition of DEBUG_MESSAGE was here
...
Hundreds of already initialized constant errors
...
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/file_system/windows.rb:123: warning: already initialized constant Puppet::FileSystem::Windows::LOCK_VIOLATION
C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/file_system/windows.rb:123: warning: previous definition of LOCK_VIOLATION was here
Traceback (most recent call last):
        33: from puppettest.rb:6:in `<main>'
        32: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:154:in `type'
        31: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/concurrent/lock.rb:10:in `synchronize'
        30: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:171:in `block in type'
        29: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:182:in `load'
        28: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:78:in `load_file'
        27: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:78:in `load'
        26: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:1:in `<top (required)>'
        25: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:2:in `<module:Puppet>'
        24: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:73:in `newtype'
        23: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/concurrent/lock.rb:10:in `synchronize'
        22: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:126:in `block in newtype'
        21: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:196:in `loadall'
        20: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:91:in `loadall'
        19: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:91:in `each'
        18: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:93:in `block in loadall'
        17: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:78:in `load_file'
        16: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:78:in `load'
        15: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec/posix.rb:1:in `<top (required)>'
        14: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec/posix.rb:1:in `require_relative'
        13: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec.rb:1:in `<top (required)>'
        12: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec.rb:1:in `require_relative'
        11: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider.rb:37:in `<top (required)>'
        10: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider.rb:43:in `<class:Provider>'
         9: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider.rb:43:in `require_relative'
         8: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/confiner.rb:1:in `<top (required)>'
         7: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/confiner.rb:1:in `require_relative'
         6: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/confine_collection.rb:3:in `<top (required)>'
         5: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/confine_collection.rb:3:in `require_relative'
         4: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/confine.rb:3:in `<top (required)>'
         3: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/confine.rb:3:in `require_relative'
         2: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:11:in `<top (required)>'
         1: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:11:in `require_relative'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/file_system/uniquefile.rb:17:in `<top (required)>': superclass mismatch for class Uniquefile (TypeError)
{noformat}

The problem stems from a behavior change in Dir.glob in Ruby 2.1 combined with using relative_require. See PUP-6557 and [https://bugs.ruby-lang.org/issues/10819].

When puppet is initially loaded, all LOADED_FEATURES are added using 8.3 paths:

{noformat}
C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/file_system/uniquefile.rb
{noformat}

When we call `Puppet::Type.type(:exec)`, it uses the autoloader to load the exec type and all of its providers. The autoloader uses `Dir.glob` to find them. Ruby 2.1 changed the behavior of `Dir.glob` to always return the long path name on Windows. So then the autoloader tries to load:

{noformat}
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/file_system/uniquefile.rb
{noformat}

Also note in the stack trace above how the path changes from 8.3 to long paths:

{noformat}
17: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:78:in `load_file'
16: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:78:in `load'
15: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec/posix.rb:1:in `<top (required)>'
...
1: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:11:in `require_relative'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/file_system/uniquefile.rb:17:in `<top (required)>': Could not autoload puppet/type/exec: Could not autoload ../../../../../../../Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec/posix: superclass mismatch for class Uniquefile (Puppet::Error)
{noformat}
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Josh Cooper (Jira)

unread,
Jul 22, 2021, 7:56:04 PM7/22/21
to puppe...@googlegroups.com
Josh Cooper created an issue
Issue Type: Bug Bug
Assignee: Unassigned
Created: 2021/07/22 4:54 PM
Priority: Normal Normal
Reporter: Josh Cooper

Puppet Version: 7.9.0

The autoloader can load the same file twice if the ruby load path contains Windows 8.3 style paths. The file lib/puppet/file_system/uniquefile.rb creates a class whose parent is an anonymous class. If the file is loaded twice, then it will fail with: superclass mismatch for class Uniquefile (TypeError).

To reproduce, install puppet-agent on Windows:

C:\> net stop puppet
C:\> cd C:\PROGRA~1\PUPPET~1\Puppet\puppet\bin

create "puppettest.rb" containing:

require 'puppet'
Puppet.initialize_settings
puts "RUBYLIBDIR #{RbConfig::CONFIG['rubylibdir']}"
puts "Loaded Puppet"
pp $LOADED_FEATURES.grep(/uniquefile/).first
Puppet::Type.type(:exec)

And execute the script:

The problem stems from a behavior change in Dir.glob in Ruby 2.1 combined with using relative_require. See PUP-6557 and https://bugs.ruby-lang.org/issues/10819.

When puppet is initially loaded, all LOADED_FEATURES are added using 8.3 paths:

C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/file_system/uniquefile.rb

When we call `Puppet::Type.type(:exec)`, it uses the autoloader to load the exec type and all of its providers. The autoloader uses `Dir.glob` to find them. Ruby 2.1 changed the behavior of `Dir.glob` to always return the long path name on Windows. So then the autoloader tries to load:

C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/file_system/uniquefile.rb

Also note in the stack trace above how the path changes from 8.3 to long paths:

17: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:78:in `load_file'
 16: from C:/PROGRA~1/PUPPET~1/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:78:in `load'
 15: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec/posix.rb:1:in `<top (required)>'
 ...
 1: from C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:11:in `require_relative'
 C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/file_system/uniquefile.rb:17:in `<top (required)>': Could not autoload puppet/type/exec: Could not autoload ../../../../../../../Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec/posix: superclass mismatch for class Uniquefile (Puppet::Error)

Ciprian Badescu (Jira)

unread,
Jul 27, 2021, 10:33:02 AM7/27/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Sep 14, 2021, 12:40:03 AM9/14/21
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Unassigned
Change By: Josh Cooper
Assignee: Beth Glenfield

Josh Cooper (Jira)

unread,
Sep 17, 2021, 5:46:02 PM9/17/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11184
 
Re: Autoloader is confused by Windows 8.3 paths

This issue is a result of the switch to require_relative and can be reproduced in puppet 7.6.0 (before the ruby patch). It seems to be ruby bug with short filename and relative_require:

C:\>md abcdefghijk
C:\>cd abcdef~1
C:\abcdef~1>type *rb
 
a.rb
puts "dir a #{__dir__}"
require_relative 'b'
puts load('C:\abcdefghijk\b.rb')
 
b.rb
puts "dir b #{__dir__}"
require_relative 'c'
 
c.rb
puts "dir c #{__dir__}"
require 'tempfile'
class C < DelegateClass(File); end
 
C:\ABCDEF~1>ruby a.rb
dir a C:/ABCDEF~1
dir b C:/ABCDEF~1
dir c C:/ABCDEF~1
dir c C:/abcdefghijk
Traceback (most recent call last):
        2: from a.rb:3:in `<main>'
        1: from a.rb:3:in `load'
C:/abcdefghijk/c.rb:3:in `<top (required)>': superclass mismatch for class C (TypeError)

This fails because c.rb is loaded first using its shortname, then an attempt is made it load it again using its long name.

Puppet runs into this, because the autoloader uses Dir.glob to find candidate files to load, and ruby always returns globbed children using long paths:

      dir = Pathname.new(File.expand_path(dir))
      Dir.glob(File.join(dir, path, "*.rb")).collect do |file|
        Pathname.new(file).relative_path_from(dir).to_s
      end

Interestingly, Pathname#relative_path_from gets confused with file is a long path and dir is a short path:

 ../../../../../../../Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec/posix

It's possible to normalize that using File.expand_path:

irb(main):001:0> File.expand_path("../../../../../../../Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec/posix")
=> "C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/provider/exec/posix"

But I think the least surprising thing would be to resolve the long path name for the base dir so that the globbed children have the same prefix. Some of this was done in PUP-6557, but the autoloader wasn't touched. It didn't need to be because we weren't using require_relative yet.

Josh Cooper (Jira)

unread,
Sep 17, 2021, 7:25:02 PM9/17/21
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Josh Cooper
 
Change By: Josh Cooper
Assignee: Josh Cooper

Josh Cooper (Jira)

unread,
Sep 17, 2021, 7:26:02 PM9/17/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Sep 17, 2021, 7:26:02 PM9/17/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Coremunity Kanban

Josh Cooper (Jira)

unread,
Sep 24, 2021, 5:00:02 PM9/24/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 7.12.0
Fix Version/s: PUP 6.25.0

Josh Cooper (Jira)

unread,
Sep 24, 2021, 5:02:02 PM9/24/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes: Bug Fix
Release Notes Summary: Fixes a regression that prevented puppet from running if the current working directory was a short Windows path (8.3)

Josh Cooper (Jira)

unread,
Sep 27, 2021, 12:55:03 PM9/27/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.25.0

Josh Cooper (Jira)

unread,
Oct 1, 2021, 11:48:02 AM10/1/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11184
 
Re: Autoloader is confused by Windows 8.3 paths

Merged to main in 90ec8bd2c2

Josh Cooper (Jira)

unread,
Oct 4, 2021, 10:16:01 AM10/4/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11184

Passed CI in ba1e8af0717c6116f77d8665db115887d30c2ff1

Claire Cadman (Jira)

unread,
Oct 5, 2021, 8:05:02 AM10/5/21
to puppe...@googlegroups.com
Claire Cadman updated an issue
 
Change By: Claire Cadman
Labels: doc-reviewed

Josh Cooper (Jira)

unread,
Oct 8, 2021, 7:20:02 PM10/8/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Affects Version/s: PUP 7.9.0
Reply all
Reply to author
Forward
0 new messages