Jira (PUP-10924) `module install` can't unpack modules with long paths because of minitar bug

32 views
Skip to first unread message

David Schmitt (Jira)

unread,
Feb 18, 2021, 4:01:04 AM2/18/21
to puppe...@googlegroups.com
David Schmitt created an issue
 
Puppet / Bug PUP-10924
`module install` can't unpack modules with long paths because of minitar bug
Issue Type: Bug Bug
Affects Versions: PUP 7.3.0, PUP 6.21.1
Assignee: Unassigned
Created: 2021/02/18 1:00 AM
Priority: Normal Normal
Reporter: David Schmitt

Puppet Version: 6.21.1 but because of minitar 0.9 origins, likely also relevant to all other recent releases.
OS Name/Version: xenial

Trying to install modules with puppet module install when the module contains deeply nested paths fails with a minitar error.

See https://github.com/puppetlabs/puppet-modulebuilder/pull/37 for the puppet-modulebuilder code changes to trigger this; See https://travis-ci.com/github/puppetlabs/puppet-modulebuilder/jobs/484161960#L423-L430 for the test results.

Desired Behavior:

Puppet should be able to install all valid modules.

Actual Behavior:

puppet module install fails with 

Failed to install the module using Puppet. Exit code 1: Notice: Preparing to install into /tmp/d20210218-7106-kfle20 ...
 
Error: "/1234567890" is not a valid octal string
 
Error: Try 'puppet help module install' for usage

Josh Cooper would it make sense to move the unpack code also to puppet-modulebuilder to ease iterating and testing these connected code-paths? This gem is already used by voxpupuli/puppet-blacksmith and was slated to be used by the PDK (where this code was lifted from).
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Feb 18, 2021, 9:19:03 PM2/18/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10924
 
Re: `module install` can't unpack modules with long paths because of minitar bug

Shouldn't we submit a fix to upstream minitar? It's been downloaded 25M times so seems like we shouldn't reinvent the wheel.

Mihai Buzgau (Jira)

unread,
Feb 23, 2021, 10:23:03 AM2/23/21
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
 
Change By: Mihai Buzgau
Team: Night's Watch

Josh Cooper (Jira)

unread,
Feb 25, 2021, 1:47:02 AM2/25/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10924
 
Re: `module install` can't unpack modules with long paths because of minitar bug

Rubygems has the same issue https://github.com/rubygems/rubygems/issues/1376

I think my main concern is any puppet module packed in a new way and has file paths less than 256, should be compatible with existing PMT. But if a module has long paths then it would be ok to pack using one of the tar variants that has a higher limit.

Also we still need to fix the pluginsync issue with LFN

Michael Lombardi (Jira)

unread,
Apr 16, 2021, 11:31:04 AM4/16/21
to puppe...@googlegroups.com
Michael Lombardi updated an issue
 
Change By: Michael Lombardi
Priority: Normal High
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Michael Lombardi (Jira)

unread,
Apr 16, 2021, 11:32:04 AM4/16/21
to puppe...@googlegroups.com
Michael Lombardi commented on Bug PUP-10924
 
Re: `module install` can't unpack modules with long paths because of minitar bug

This problem has become a blocker for the Puppet.Dsc work as long paths in puppetized modules brings things to a state where the modules are buildable and publishable but not installable.

https://github.com/puppetlabs/Puppet.Dsc/issues/144

Beth Glenfield (Jira)

unread,
Apr 27, 2021, 8:41:01 AM4/27/21
to puppe...@googlegroups.com

Is there any relation between this bug and https://tickets.puppetlabs.com/browse/MODULES-3135?

Mihai Buzgau (Jira)

unread,
Apr 27, 2021, 11:03:02 AM4/27/21
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
 
Change By: Mihai Buzgau
Sprint: NW-2021-05-19

Ciprian Badescu (Jira)

unread,
Apr 28, 2021, 5:47:03 AM4/28/21
to puppe...@googlegroups.com
Ciprian Badescu updated an issue
Change By: Ciprian Badescu
Sprint: NW-2021-05-19 Triaged tickets

Ben Ford (Jira)

unread,
Jul 13, 2021, 2:14:02 PM7/13/21
to puppe...@googlegroups.com
Ben Ford commented on Bug PUP-10924
 
Re: `module install` can't unpack modules with long paths because of minitar bug

fwiw, I'm unable to replicate this locally (macOS). I removed the path length checks from the PDK and used it to build modules which I then installed with puppet module install. Both the build and the install succeeded until I exceeded the OS/filesystem maximum length, at which point I received an Errno::ENAMETOOLONG as I'd expect.

Josh Cooper (Jira)

unread,
Jul 14, 2021, 7:22:03 PM7/14/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10924

Ben Ford We only vendor the minitar gem on Windows and Solaris, so you'll need to be one one of those platforms to reproduce. On other platforms we rely on the native tar command, see https://github.com/puppetlabs/puppet/blob/d6038363b13cd989d2e7cb35fc9d5c605d31b786/lib/puppet/module_tool/tar.rb#L8-L17

Ben Ford (Jira)

unread,
Jul 15, 2021, 6:19:02 PM7/15/21
to puppe...@googlegroups.com
Ben Ford commented on Bug PUP-10924

I cannot reproduce the test failure indicated by https://travis-ci.com/github/puppetlabs/puppet-modulebuilder/jobs/484161960#L423-L430 on a Xenial machine.

I removed the validate_ustar_path check from the PDK, then used the code from the unit test to make the deeply nested empty file in a module. I used my modified PDK to build the module package and then puppet module install to install it. Everything worked as expected.

Heston Snodgrass (Jira)

unread,
Jul 26, 2021, 4:57:03 PM7/26/21
to puppe...@googlegroups.com

Ben FordI've seen this issue when using Litmus against Windows test hosts. It will unpack the module, but it will not unpack any files or directories in the module that are part of paths longer than 255 chars.

Ciprian Badescu (Jira)

unread,
Sep 7, 2021, 4:17:03 AM9/7/21
to puppe...@googlegroups.com
Ciprian Badescu updated an issue
Change By: Ciprian Badescu
Sprint: Triaged tickets

Ciprian Badescu (Jira)

unread,
Sep 7, 2021, 4:17:03 AM9/7/21
to puppe...@googlegroups.com
Ciprian Badescu assigned an issue to Ciprian Badescu
Change By: Ciprian Badescu
Assignee: Ciprian Badescu

Josh Cooper (Jira)

unread,
Dec 2, 2021, 12:23:01 AM12/2/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10924
 
Re: `module install` can't unpack modules with long paths because of minitar bug

Heston Snodgrass or anyone else following this ticket, can you try using a recent puppet-agent MSI (6.25 or 7.12) and install it with the ENABLE_LONG_PATHS=TRUE parameter, see https://puppet.com/docs/puppet/6/install_agents.html#msi_properties, reboot and verify this is no longer an issue?

Josh Cooper (Jira)

unread,
Mar 31, 2022, 4:43:01 PM3/31/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10924

I verified this is no longer an issue when specifying ENABLE_LONG_PATHS=true and rebooting. I created a module whose fact relatively requires a file whose path exceeds MAX_PATH:

Facter.add(:long) do
 
  setcode do
 
    require_relative '../01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789/abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij'
 
    true
 
  end
 
end

And the helper code contains:

C:\ProgramData\PuppetLabs>type "C:\ProgramData\PuppetLabs\code\environments\production\modules\longfilenames\lib\01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789\abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij.rb"
 
puts 'in helper'

The relative path to the file on disk is 402 chars which exceeds MAXPATH. I can install the module (using 7.15.0):

C:\ProgramData\PuppetLabs>puppet module install c:\cygwin64\home\Administrator\josh-longfilenames-0.1.0.tar.gz
 
Notice: Preparing to install into C:/ProgramData/PuppetLabs/code/environments/production/modules ...
 
Notice: Downloading from https://forgeapi.puppet.com ...
 
Notice: Installing -- do not interrupt ...
 
C:/ProgramData/PuppetLabs/code/environments/production/modules
 
└── josh-longfilenames (v0.1.0)

And the fact can load the helper code:

C:\ProgramData\PuppetLabs>puppet apply -e "notice($facts['long'])" --modulepath c:\ProgramData\PuppetLabs\code\environments\production\modules
 
in helper
 
Notice: Scope(Class[main]): true

So I'm going to close this as a dup of PA-3843
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages