Jira (PUP-6380) HTTP file sources fail for GET-only URIs

23 views
Skip to first unread message

Matt Casper (JIRA)

unread,
Jun 2, 2016, 2:22:03 PM6/2/16
to puppe...@googlegroups.com
Matt Casper created an issue
 
Puppet / Bug PUP-6380
HTTP file sources fail for GET-only URIs
Issue Type: Bug Bug
Affects Versions: PUP 4.5.1, PUP 4.5.0
Assignee: Unassigned
Created: 2016/06/02 11:21 AM
Priority: Normal Normal
Reporter: Matt Casper

URIs that only succeed on a GET cannot currently be used as file sources, as when retrieving file metadata, we use a HEAD request. The most notable of these URI types are Amazon S3 presigned URIs. These URIs' signatures include the HTTP request in them, which is usually GET for downloading. This causes them to return a 403 when responding to a HEAD request.

Pull request with a fix is here: https://github.com/puppetlabs/puppet/pull/5002
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9)
Atlassian logo

Henrik Lindberg (JIRA)

unread,
Jun 2, 2016, 4:27:03 PM6/2/16
to puppe...@googlegroups.com

Kylo Ginsberg (JIRA)

unread,
Jun 2, 2016, 6:06:05 PM6/2/16
to puppe...@googlegroups.com
Kylo Ginsberg updated an issue
 
Change By: Kylo Ginsberg
Scrum Team: Client Platform

Kylo Ginsberg (JIRA)

unread,
Jun 2, 2016, 6:09:21 PM6/2/16
to puppe...@googlegroups.com
Kylo Ginsberg commented on Bug PUP-6380
 
Re: HTTP file sources fail for GET-only URIs

Matt Casper our jira workflow has confusing names, but Ready for Merge means there's a pull request so I bumped the ticket to that state (Ready for Review is the final state before Resolved).

Matt Casper (JIRA)

unread,
Jun 2, 2016, 6:34:56 PM6/2/16
to puppe...@googlegroups.com
Matt Casper commented on Bug PUP-6380

Kylo Ginsberg Got it, thanks! I read that in the contributing guide, but didn't know if it was a typo or not. Noted!

Sean McDonald (JIRA)

unread,
May 16, 2017, 8:03:02 PM5/16/17
to puppe...@googlegroups.com
Sean McDonald updated an issue
 
Change By: Sean McDonald
Team: Agent
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Sean McDonald (JIRA)

unread,
May 16, 2017, 8:04:03 PM5/16/17
to puppe...@googlegroups.com
Sean McDonald assigned an issue to Matt Casper
Change By: Sean McDonald
Assignee: Matt Casper

Sean McDonald (JIRA)

unread,
May 16, 2017, 8:04:04 PM5/16/17
to puppe...@googlegroups.com
Sean McDonald updated an issue
Change By: Sean McDonald
Labels: triaged

Sean McDonald (JIRA)

unread,
May 16, 2017, 8:04:05 PM5/16/17
to puppe...@googlegroups.com
Sean McDonald commented on Bug PUP-6380
 
Re: HTTP file sources fail for GET-only URIs

Comments have been made on the PR

Jacob Helwig (JIRA)

unread,
Dec 13, 2017, 6:48:06 PM12/13/17
to puppe...@googlegroups.com
Jacob Helwig updated an issue
 
Change By: Jacob Helwig
Sub-team: Coremunity
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Atlassian logo

Jacob Helwig (JIRA)

unread,
Dec 13, 2017, 6:52:03 PM12/13/17
to puppe...@googlegroups.com
Jacob Helwig updated an issue
Change By: Jacob Helwig
Sprint: Platform Core Grooming

Josh Cooper (JIRA)

unread,
Sep 26, 2018, 8:54:15 PM9/26/18
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Platform Core Grooming
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Josh Cooper (Jira)

unread,
Apr 24, 2020, 8:40:04 PM4/24/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-6380
 
Re: HTTP file sources fail for GET-only URIs

puppet makes a head request to retrieve file metadata, but amazon presigned signatures don't allow HEAD requests. This can be simulated using curl:

$ curl -sv -ILO 'https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64'
 
*   Trying 192.30.255.112...
 
...
 
> HEAD /XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64 HTTP/1.1
 
> Host: github.com
 
> User-Agent: curl/7.54.0
 
> Accept: */*
 
>
 
< HTTP/1.1 302 Found
 
< date: Thu, 23 Apr 2020 18:27:35 GMT
 
< content-type: text/html; charset=utf-8
 
< server: GitHub.com
 
< status: 302 Found
 
< vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
 
< location: https://github-production-release-asset-2e65be.s3.amazonaws.com/68653753/d2ea368e-ed47-11e7-98f2-0c05ce59f7f9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200423T182735Z&X-Amz-Expires=300&X-Amz-Signature=7848a201a364caf83be84b32f0170c7ffd35066732d92d49d278454b025db4bc&X-Amz-SignedHeaders=host&actor_id=0&repo_id=68653753&response-content-disposition=attachment%3B%20filename%3Ddpp.aarch64&response-content-type=application%2Foctet-stream
 
< cache-control: no-cache
 
< strict-transport-security: max-age=31536000; includeSubdomains; preload
 
< x-frame-options: deny
 
< x-content-type-options: nosniff
 
< x-xss-protection: 1; mode=block
 
< expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
 
< content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com
 
< Set-Cookie: _gh_sess=X36lfIvZNxx7p5M%2F7rDbYDM9kYbjtJl72u%2FriXv1Z8sg9u0Rv9TjGo2zUsr6NNDwwmN1oAtFXhgpzOawDtov24j0cJ5buktyX%2B3OO5K10lI0OolJKkbqYD0axqs4vZQJM1I8M00BNSi%2B57RnFrWYlU4T1X556jpvaqWO%2FWCFMu8GHN9dg4sERe8RSa2pyz8Z969cbi1StBXWsjKACYmYU3clCZfL9dnoaLJOoF5wwL81I3ifT9Q9Kyj%2Fd4p5IkAzRj6Nm77A4Z9FL1YaeULuAw%3D%3D--HMpmlt%2FTH5vlXFWu--BDHdc%2BYZCX80UqipdsAUTw%3D%3D; Path=/; HttpOnly; Secure
 
< Set-Cookie: _octo=GH1.1.724492882.1587666465; Path=/; Domain=github.com; Expires=Fri, 23 Apr 2021 18:27:45 GMT; Secure
 
< Set-Cookie: logged_in=no; Path=/; Domain=github.com; Expires=Fri, 23 Apr 2021 18:27:45 GMT; HttpOnly; Secure
 
< Content-Length: 622
 
< X-GitHub-Request-Id: C563:214F:1289F6:1936D5:5EA1DE21
 
<
 
* Connection #0 to host github.com left intact
 
* Issue another request to this URL: 'https://github-production-release-asset-2e65be.s3.amazonaws.com/68653753/d2ea368e-ed47-11e7-98f2-0c05ce59f7f9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200423T182735Z&X-Amz-Expires=300&X-Amz-Signature=7848a201a364caf83be84b32f0170c7ffd35066732d92d49d278454b025db4bc&X-Amz-SignedHeaders=host&actor_id=0&repo_id=68653753&response-content-disposition=attachment%3B%20filename%3Ddpp.aarch64&response-content-type=application%2Foctet-stream'
 
*   Trying 52.216.12.20...
 
* TCP_NODELAY set
 
* Connected to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.12.20) port 443 (#1)
 
..
 
> HEAD /68653753/d2ea368e-ed47-11e7-98f2-0c05ce59f7f9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200423T182735Z&X-Amz-Expires=300&X-Amz-Signature=7848a201a364caf83be84b32f0170c7ffd35066732d92d49d278454b025db4bc&X-Amz-SignedHeaders=host&actor_id=0&repo_id=68653753&response-content-disposition=attachment%3B%20filename%3Ddpp.aarch64&response-content-type=application%2Foctet-stream HTTP/1.1
 
> Host: github-production-release-asset-2e65be.s3.amazonaws.com
 
> User-Agent: curl/7.54.0
 
> Accept: */*
 
>
 
< HTTP/1.1 403 Forbidden
 
< x-amz-request-id: 4E96C691C4F92E13
 
< x-amz-id-2: ARs0ZS1e80dSW03h8VFI5ppN0NTo8eKboV4BRaLL5xNbc1aHltnph36j3nc/yoaZ2iKsQRwctaY=
 
< Content-Type: application/xml
 
< Transfer-Encoding: chunked
 
< Date: Thu, 23 Apr 2020 18:27:45 GMT
 
< Server: AmazonS3
 
<
 
* Connection #1 to host github-production-release-asset-2e65be.s3.amazonaws.com left intact

One suggested workaround is to make a GET request with range 0-0:

curl -sv -LO -r 0-0 'https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64'
 
*   Trying 192.30.255.112...
 
* TCP_NODELAY set
 
* Connected to github.com (192.30.255.112) port 443 (#0)
 
...
 
> GET /XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64 HTTP/1.1
 
> Host: github.com
 
> Range: bytes=0-0
 
> User-Agent: curl/7.54.0
 
> Accept: */*
 
>
 
< HTTP/1.1 302 Found
 
< date: Thu, 23 Apr 2020 18:29:57 GMT
 
< content-type: text/html; charset=utf-8
 
< server: GitHub.com
 
< status: 302 Found
 
< vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
 
< location: https://github-production-release-asset-2e65be.s3.amazonaws.com/68653753/d2ea368e-ed47-11e7-98f2-0c05ce59f7f9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200423T182957Z&X-Amz-Expires=300&X-Amz-Signature=05ae8acbe66103e58cdf2caf02662a5a9bbf9444413ce5d1b5123d36b7373ec8&X-Amz-SignedHeaders=host&actor_id=0&repo_id=68653753&response-content-disposition=attachment%3B%20filename%3Ddpp.aarch64&response-content-type=application%2Foctet-stream
 
< cache-control: no-cache
 
< strict-transport-security: max-age=31536000; includeSubdomains; preload
 
< x-frame-options: deny
 
< x-content-type-options: nosniff
 
< x-xss-protection: 1; mode=block
 
< expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
 
< content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com
 
< Set-Cookie: _gh_sess=BRn%2FZoXFwSxfFI9bJAxZWWJMuL78uGcioy0kUd8TKQ%2BQvOeYNhTtQ%2F4wr8Sa5ZhyLmf%2BACFNfBUd%2BvyC4bPDcBsheQ7mRUPs%2FWrmO3yfnwtNPjW0ViJYXpgDCo%2BVgd%2FvuULQKcw0nJFmcvvySjZkrbub%2BJgJocRuB25LRk9FA0EQ65O832lISYrKJ6eAbeitVPvVEaJ5SJ5LQr0v7svTgnQ2v5MjKc9PvGEr9%2FLDMeeQogEsnnpV0GUBQ7Y5%2FRsmuNvG3NuODsaN8bvGG4C59w%3D%3D--18EGVNpAej7Mr0uN--EN6DYyDwXWNpb0Tz012LGw%3D%3D; Path=/; HttpOnly; Secure
 
< Set-Cookie: _octo=GH1.1.221206937.1587666603; Path=/; Domain=github.com; Expires=Fri, 23 Apr 2021 18:30:03 GMT; Secure
 
< Set-Cookie: logged_in=no; Path=/; Domain=github.com; Expires=Fri, 23 Apr 2021 18:30:03 GMT; HttpOnly; Secure
 
< Content-Length: 622
 
< X-GitHub-Request-Id: C585:1692:4A347:65418:5EA1DEAB
 
<
 
* Ignoring the response-body
 
{ [127 bytes data]
 
* Connection #0 to host github.com left intact
 
* Issue another request to this URL: 'https://github-production-release-asset-2e65be.s3.amazonaws.com/68653753/d2ea368e-ed47-11e7-98f2-0c05ce59f7f9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200423T182957Z&X-Amz-Expires=300&X-Amz-Signature=05ae8acbe66103e58cdf2caf02662a5a9bbf9444413ce5d1b5123d36b7373ec8&X-Amz-SignedHeaders=host&actor_id=0&repo_id=68653753&response-content-disposition=attachment%3B%20filename%3Ddpp.aarch64&response-content-type=application%2Foctet-stream'
 
*   Trying 52.216.80.176...
 
* TCP_NODELAY set
 
* Connected to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.80.176) port 443 (#1)
 
...
 
> GET /68653753/d2ea368e-ed47-11e7-98f2-0c05ce59f7f9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200423T182957Z&X-Amz-Expires=300&X-Amz-Signature=05ae8acbe66103e58cdf2caf02662a5a9bbf9444413ce5d1b5123d36b7373ec8&X-Amz-SignedHeaders=host&actor_id=0&repo_id=68653753&response-content-disposition=attachment%3B%20filename%3Ddpp.aarch64&response-content-type=application%2Foctet-stream HTTP/1.1
 
> Host: github-production-release-asset-2e65be.s3.amazonaws.com
 
> Range: bytes=0-0
 
> User-Agent: curl/7.54.0
 
> Accept: */*
 
>
 
< HTTP/1.1 206 Partial Content
 
< x-amz-id-2: LOxUNyDKUmCeFaYQ/kE9Y1b7ENGa+YkQSlhRWB+XfOrKy6gnM5wKLAyAWKlEMEJAxofdHYuTdUA=
 
< x-amz-request-id: A2D71D4AFEB124D4
 
< Date: Thu, 23 Apr 2020 18:30:04 GMT
 
< Last-Modified: Sat, 30 Dec 2017 08:57:21 GMT
 
< ETag: "df9a97e06f7e6b406044029f382147ae"
 
< Content-Disposition: attachment; filename=dpp.aarch64
 
< Accept-Ranges: bytes
 
< Content-Range: bytes 0-0/9567530
 
< Content-Type: application/octet-stream
 
< Content-Length: 1
 
< Server: AmazonS3
 
<
 
[1 bytes data]

The second get request will return 206 for a partial content and the Content-Range header includes the total length.
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
May 20, 2020, 4:19:03 PM5/20/20
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Unassigned
 
Change By: Josh Cooper
Assignee: Matt Casper

Josh Cooper (Jira)

unread,
Jun 12, 2020, 9:03:03 PM6/12/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Coremunity Hopper

Josh Cooper (Jira)

unread,
Jun 16, 2020, 2:18:03 AM6/16/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.17.0

Josh Cooper (Jira)

unread,
Jun 16, 2020, 2:18:03 AM6/16/20
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Josh Cooper
Change By: Josh Cooper
Assignee: Josh Cooper

Josh Cooper (Jira)

unread,
Jun 16, 2020, 2:21:08 AM6/16/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes: Enhancement
Release Notes Summary: Puppet can now retrieve file content from Amazon AWS and github releases, e.g. source => "https://github.com/path/to/released/artifact".

Josh Cooper (Jira)

unread,
Jun 16, 2020, 2:22:03 AM6/16/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Coremunity Hopper Platform Core KANBAN

Josh Cooper (Jira)

unread,
Jun 18, 2020, 1:59:04 PM6/18/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-6380

Josh Cooper (Jira)

unread,
Jun 18, 2020, 2:53:02 PM6/18/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-6380

Passed CI in 61efdfd23e

Josh Cooper (Jira)

unread,
Jun 18, 2020, 2:55:04 PM6/18/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
URIs that only succeed on a GET cannot currently be used as file sources, as when retrieving file metadata, we use a HEAD request. The most notable of these URI types are Amazon S3 presigned URIs. These URIs ' contain a signatures include with the HTTP request method in them, which is usually GET for downloading. This causes them to return a 403 when responding to a HEAD request. If that occurs puppet will fallback to making a partial GET request using the Range header, so that it can retrieve the file checksum and determine if the local file needs to be updated, but do so without retrieving any file content.

Pull request with a fix is here: https://github.com/puppetlabs/puppet/pull/5002

Claire Cadman (Jira)

unread,
Jul 7, 2020, 10:00:04 AM7/7/20
to puppe...@googlegroups.com
Claire Cadman updated an issue
Change By: Claire Cadman
Labels: doc_reviewed
Reply all
Reply to author
Forward
0 new messages