Jira (PUP-9813) Performance of unpacking modules is extremely slow

[Chris Suszynski (JIRA)]

Chris Suszynski created an issue

Puppet / PUP-9813 Performance of unpacking modules is extremely slow Issue Type: Bug Affects Versions: PUP 6.6.0, PUP 5.5.14, PUP 4.10.12 Assignee: Unassigned Components: Modules Created: 2019/07/03 3:53 PM Priority: Normal Reporter: Chris Suszynski Original Estimate: 2 days Remaining Estimate: 2 days Puppet Version: any Puppet Server Version: any OS Name/Version: any Installing modules from Puppet Forge is extremelly slow. I have found a culprit. It is a Puppet::ModuleTool::Tar::Mini implementation. The unpack is 100x times slower then executing a {{tar xzvf }}system command. Steps to reproduce: Install a module from Puppet Forge and measure a time: time puppet module install puppetlabs-stdlib --version 4.25.1 Then execute: time bash -c 'curl https://forge.puppet.com/v3/files/puppetlabs-stdlib-4.25.1.tar.gz -o puppetlabs-stdlib-4.25.1.tar.gz && tar xzvf puppetlabs-stdlib-4.25.1.tar.gz' Desired Behavior: Extraction of downloaded modules should be about the same performance as system tar xzvf command. It might be at most 2x times slower then system command, but preferably should be about the same. Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Chris Suszynski (JIRA)]

Chris Suszynski updated an issue

Puppet / PUP-9813 Performance of unpacking modules is extremely slow

Change By: Chris Suszynski *Puppet Version: any* *Puppet Server Version: any* *OS Name/Version: any*

Installing modules from Puppet Forge is extremelly slow. I have found a culprit. It is a {{Puppet::ModuleTool::Tar::Mini}} implementation. The unpack is 100x times slower then executing a

\ {{tar xzvf }}system command. *Steps to reproduce:*

Install a module from Puppet Forge and measure a time: {{time puppet module install puppetlabs-stdlib --version 4.25.1}} Then execute: {{time bash -c 'curl [https://forge.puppet.com/v3/files/puppetlabs-stdlib-4.25.1.tar.gz] -o puppetlabs-stdlib-4.25.1.tar.gz && tar xzvf puppetlabs-stdlib-4.25.1.tar.gz'}}

*Desired Behavior:*

Extraction of downloaded modules should be about the same performance as system tar xzvf command. It might be at most 2x times slower then system command, but preferably should be about the same.

Temporal solution: To install modules faster i propose to switch to Gnu implementation as a first choice. Al least until Minitar can be much faster.

Add Comment

[Chris Suszynski (JIRA)]

Chris Suszynski updated an issue

Puppet / PUP-9813 Performance of unpacking modules is extremely slow Change By: Chris Suszynski

*Puppet Version: any* *Puppet Server Version: any* *OS Name/Version: any*

Installing modules from Puppet Forge is extremelly extremely slow. I have found a culprit. It is a

{{Puppet::ModuleTool::Tar::Mini}} implementation.

The unpack is * 100x times * slower then executing a \ {{tar xzvf}} system command . !

*Steps to reproduce:* Install a module from Puppet Forge and measure a time: {{time puppet module install puppetlabs-stdlib --version 4.25.1}} Then execute: {{time bash -c 'curl [https://forge.puppet.com/v3/files/puppetlabs-stdlib-4.25.1.tar.gz] -o puppetlabs-stdlib-4.25.1.tar.gz && tar xzvf puppetlabs-stdlib-4.25.1.tar.gz'}} *Desired Behavior:* Extraction of downloaded modules should be about the same performance as system tar xzvf command. It might be at most 2x times slower then system command, but preferably should be about the same.

* Temporal solution: *

To install modules faster i propose to switch to Gnu implementation as a first choice. Al least until Minitar can be much faster.

Add Comment

[Jorie Tappa (JIRA)]

Jorie Tappa updated an issue

Puppet / PUP-9813 Performance of unpacking modules is extremely slow

Change By: Jorie Tappa Team: Coremunity

Add Comment

[Jorie Tappa (JIRA)]

Jorie Tappa commented on PUP-9813

Re: Performance of unpacking modules is extremely slow This could introduce a regression on Solaris if implemented as-is, so the PR might need updating to account for that.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-9813

Re: Performance of unpacking modules is extremely slow

Added comments on the PR: https://github.com/puppetlabs/puppet/pull/7602#issuecomment-511561502

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-9813

Re: Performance of unpacking modules is extremely slow

Thanks for raising this issue Chris Suszynski. We submitted a patch to minitar, and it's been released. Now we need to update puppet's minitar usage, so we don't fsync every extracted file and directory. We're tracking that work in PUP-10013, so I'm going to close this ticket and related PR.

Add Comment