Jira (PUP-8338) Puppet apply doesn't stream file content

13 views
Skip to first unread message

Josh Cooper (JIRA)

unread,
Jan 9, 2018, 7:13:02 PM1/9/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Puppet / Bug PUP-8338
Puppet apply doesn't stream file content
Change By: Josh Cooper
Summary: Puppet apply doesn't stream  remote  file content
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Atlassian logo

Josh Cooper (JIRA)

unread,
Jan 9, 2018, 7:19:02 PM1/9/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
*Puppet Version:* Puppet 4 (at least) and up
*Puppet Server Version:* n/a
*OS Name/Version:* all

When running {{puppet apply}} with a file resource  whose  and  source parameter  is {{http(s)}} or {{puppet}} , then puppet will load the entire file into memory, instead of streaming it.  Note the {{puppet}} case only happens if a host isn't specified, as that is interpreted to mean "use the local fileserver".

This was noticed while researching why the bug described in PUP-7482 only affected {{puppet agent}} but not {{puppet apply}}.

*Desired Behavior:*

When managing files with a {{source}} parameter, puppet apply should always stream file content from all of the following sources

{{<absolute path>}} local file
{{file://<URI-encoded path>}} local file
{{puppet:///modules/mymodule}} local file
{{http?://host:port/path/to/file}} remote file

*Actual Behavior:*

{{puppet apply}} does not stream file content from http-based sources. See https://github.com/puppetlabs/puppet/blob/5.3.3/lib/puppet/type/file/source.rb#L274-L276.

At first glance it appears the {{Puppet::Type::File::ParameterSource#content}} method can be deleted entirely, and should always rely on {{ chunk_file_from_source chunk_file_from_disk }}  which will invoke the  or  {{ terminus_selector.rb chunk_file_from_source }}  to determine the correct source .

Rob Braden (JIRA)

unread,
Feb 3, 2020, 1:03:04 PM2/3/20
to puppe...@googlegroups.com
Rob Braden updated an issue
Change By: Rob Braden
Sprint: Coremunity Hopper
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Josh Cooper (JIRA)

unread,
Feb 10, 2020, 12:22:05 PM2/10/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.14.0

Josh Cooper (JIRA)

unread,
Feb 11, 2020, 12:57:04 AM2/11/20
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Josh Cooper
Change By: Josh Cooper
Assignee: Josh Cooper

Josh Cooper (JIRA)

unread,
Feb 11, 2020, 12:57:04 AM2/11/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Coremunity Hopper Platform Core KANBAN

Melissa Stone (JIRA)

unread,
Feb 21, 2020, 11:22:03 PM2/21/20
to puppe...@googlegroups.com

Melissa Stone (JIRA)

unread,
Feb 24, 2020, 12:42:05 PM2/24/20
to puppe...@googlegroups.com
Melissa Stone commented on Bug PUP-8338

This has passed ci as a part of puppet-agent 6.13.0.51.gf3fde108

Melissa Stone (JIRA)

unread,
Feb 24, 2020, 12:43:05 PM2/24/20
to puppe...@googlegroups.com
Melissa Stone commented on Bug PUP-8338

Josh Cooper could you add release notes to this ticket?

Josh Cooper (JIRA)

unread,
Feb 24, 2020, 12:58:04 PM2/24/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes Summary: This changes "puppet apply" to retrieve content for file resources with a "source" parameter in the same way that "puppet agent" does. Previously, "puppet apply" read the entire file contents into memory and cached it for the duration of the run. It also did not follow HTTP redirects or handle HTTP errors in the same way, such as HTTP service unavailable (503). Now "puppet apply" uses the same code path as "puppet agent" to stream file content from "puppet://" and "http(s)://" file sources. As a result, memory  footprint should be reduced when using "puppet apply".
Release Notes: Bug Fix
Reply all
Reply to author
Forward
0 new messages