Jira (PUP-4080) support S3 URL as the file type 'source'

3 views
Skip to first unread message

Peter Foley (JIRA)

unread,
Mar 4, 2015, 6:05:23 AM3/4/15
to puppe...@googlegroups.com
Peter Foley created an issue
 
Puppet / New Feature PUP-4080
support S3 URL as the file type 'source'
Issue Type: New Feature New Feature
Assignee: Kylo Ginsberg
Components: Community, Types and Providers
Created: 2015/03/04 3:05 AM
Priority: Normal Normal
Reporter: Peter Foley

It would be nice to use AWS S3 resources as a source for the Puppet file type.

Something like:

file { 
    "/tmp/example.txt":
        source => 's3://foo-bucket/bar.file',
	region => 'ap-southeast-2',
}

This will allow puppet users who wish to manage sensitive files in an Amazon S3 bucket and use IAM roles to manage access to the files. It would be useful for both users of ec2 and computing infrastructure external to Amazon.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.3.10#6340-sha1:7ea293a)
Atlassian logo

Felix Frank (JIRA)

unread,
Mar 4, 2015, 9:30:29 AM3/4/15
to puppe...@googlegroups.com
Felix Frank commented on New Feature PUP-4080
 
Re: support S3 URL as the file type 'source'

I'm interested in concrete use cases for this.

The idea of moving files to blob storage was discussed as a possible workaround for some issues that inspired PUP-2526 (the debate around which has never been resolved).

I'm not really keen on bundling the AWS SDK with Puppet. But if we find the feature to be sufficiently desirable, then the ends probably justify the means.

Peter Foley (JIRA)

unread,
Mar 5, 2015, 5:57:22 AM3/5/15
to puppe...@googlegroups.com
Peter Foley commented on New Feature PUP-4080

The reason I decided to look at how feasible it was to add S3 support to the puppet file type was to help with managing sensitive information and how to get it onto the puppet master.

Essentially I want to have a secure s3 bucket that only my puppet masters can pull files from, this bucket would contain private keys to decode hiera data, private keys to connect to private repos on github, bitbucket and/or a private git server and other sensitive data the puppet master needs. I would than manage access to the bucket via AWS IAM roles.

To bootstrap a new LAN I would use the puppetlabs-aws module to create a number of ec2 instances (one of which would be a puppet master) for the puppet master ec2 instance I would give it the IAM role to grant it access to the puppetmaster s3 bucket. Once the ec2 instance is created I would run a bootstrap.pp file that contained:

bootstrap.pp

# .... other bootstrap stuff ....
 
package { 'aws-sdk-core':
       provider => gem,
       before => File['/root/.ssh/bitbucket-deploy-key.openssh', '/root/.ssh/github-deploy-key.openssh', '/root/.ssh/hiera-decryption-key.openssh']
}
 
file { '/root/.ssh/bitbucket-deploy-key.openssh':
        source => 's3://puppetmaster-IAM-locked-down-bucket/bitbucket-deploy-key.openssh',
        mode   => 0600,
	region  => 'ap-southeast-2',
}
 
file { '/root/.ssh/github-deploy-key.openssh':
        source => 's3://puppetmaster-IAM-locked-down-bucket/github-deploy-key.openssh',
        mode   => 0600,
	region  => 'ap-southeast-2',
}
 
file { '/root/.ssh/hiera-decryption-key.openssh':
        source => 's3://puppetmaster-IAM-locked-down-bucket/hiera-decryption-key.openssh',
        mode   => 0600,
	region  => 'ap-southeast-2',
}
 
$str = "HOST bitbucket.org
             StrictHostKeyChecking no
             IdentityFile /root/.ssh/bitbucket-deploy-key.openssh
           HOST github.com
             StrictHostKeyChecking no
             IdentityFile /root/.ssh/github-deploy-key.openssh
           "
 
file { '/root.ssh/config':
        source => 's3://puppetmaster-IAM-locked-down-bucket/root-ssh-config',
        mode   => 0600,
	region  => 'ap-southeast-2',
}
 
# Rest of the code to pull down the encrypted hieradata repo and other configuration information to bootstrap the puppet environment

If I ever need to change the sensitive keys I can just replace them in the s3 bucket and puppet will do the rest for me on the next puppet run.

As the information will be pulled from within the same AWS region (hopefully in the same data centre) it is unlikely that it can be MITM'ed or tampered with in transit (yes I know I have to originally put the private keys into the s3 bucket ...).

This obviously would work for puppet clients as well for applications where you need to seed the application with various files that you don't need to store on the puppet master.

Peter Foley (JIRA)

unread,
Mar 5, 2015, 6:03:24 AM3/5/15
to puppe...@googlegroups.com
Peter Foley commented on New Feature PUP-4080

I wanted to add this comment separately so it did not get lost in the above response.

I agree, I don't think the aws-sdk-core should be bundled with puppet. I am not advocating this, I envisage that this would work similar to how the package file type has a number of user choosable providers. I think it makes sense that the puppet file type abstraction should allow people to extend it to add additional file sources whether they are http(s) (like you are adding), s3, Google Cloud Storage, dropbox, ...

It would be upto the user to ensure that the correct client is installed to connect to the source (just like you need to have the gems package to use the gems provider).

One of the things I like about puppet is that it abstracts away a lot of the complexity behind the puppet syntax. I don't think it makes sense to replicate the whole puppet file type everytime someone wants to add a new file source.

If there was someway to allow a puppet module to inject this capability into the file type I think that could be a good compromise, I originally try to work out if I could add this capability to the puppetlabs-aws module but I am new to ruby programming and could not work out how to do it.

Peter Foley (JIRA)

unread,
Mar 17, 2015, 7:01:26 AM3/17/15
to puppe...@googlegroups.com
Peter Foley commented on New Feature PUP-4080

Melissa Stone I just wanted to check what the next step in the workflow is from a "ready for merge" status?

From the triage videos I had the impression the preference was to move this into a module (puppetlabs-aws) and more dialog needed to occur on the puppet-dev mailing list to flesh out how it could be flexible enough to work for other sources. If the consensus now is to move this into the puppet core (at least temporarily until after deciding how to give puppet modules superpowers to inject new sources into the file type) my PR code I submitted was sufficient to demonstrate how it would work.

Just off the top of my head I still need to add:

  • spec and acceptance tests
  • update the user facing documentation to highlight the need for users of this source will need to self install the aws-sdk-core (ruby gem)
  • fix up the gemfile to move the aws-sdk-core into a development group.
  • (I am sure there were some other suggestions from the triage videos that I have missed (I will have to re-watch)).

If it is still ultimately the intention of moving this functionality into a module is there any benefit (or precedence) adding a comment that this capability will move into the puppetlabs-aws in the future? Assuming this is something that Gareth Rushgrove or Ethan Brown would accept into the module?

Felix Frank (JIRA)

unread,
Mar 17, 2015, 7:08:23 AM3/17/15
to puppe...@googlegroups.com
Felix Frank assigned an issue to Eric Sorenson
 
Change By: Felix Frank
Assignee: Eric Sorenson

Felix Frank (JIRA)

unread,
Mar 17, 2015, 7:10:29 AM3/17/15
to puppe...@googlegroups.com
Felix Frank commented on New Feature PUP-4080
 
Re: support S3 URL as the file type 'source'

Handing this to Eric Sorenson since this needs a decision. Could perhaps be delegated to Ethan Brown or someone else.

Melissa Stone (JIRA)

unread,
Mar 17, 2015, 11:59:23 AM3/17/15
to puppe...@googlegroups.com
Melissa Stone commented on New Feature PUP-4080

Peter Foley the reason I moved this ticket to ready for merge is that it is a more appropriate state than open. Ready for merge does not mean we are ready to push the button right at this moment, but that we are deciding what is the best way forward. In this case, still discussing what needs to be added to your PR and what should be done with it. However, I believe Felix is correct in that 'Needs Information' is a better state for this ticket.

Ethan Brown (JIRA)

unread,
Mar 17, 2015, 1:49:30 PM3/17/15
to puppe...@googlegroups.com
Ethan Brown commented on New Feature PUP-4080

Yes, we're not ready to merge this, and there is ongoing discussion internally on how to address this in a more generic way that is easily pluggable by 3rd parties.

We definitely want this functionality, but it's a matter of determining the appropriate integration point. This might be something that we want to tackle as an architectural change inside Puppet core to accommodate protocol handlers in modules. As soon as there's anything to report on that front, I will let you know - we're in a bit of a holding pattern at the moment.

Thanks again for the contribution and all the time you've put into this.

Branan Riley (JIRA)

unread,
Apr 1, 2015, 1:47:23 PM4/1/15
to puppe...@googlegroups.com
Branan Riley commented on New Feature PUP-4080

Ethan Brown I filed PUP-4368 to allow us to track the idea of loading file_serving terminii from modules.

This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
Atlassian logo

Peter Foley (JIRA)

unread,
Apr 2, 2015, 9:24:10 AM4/2/15
to puppe...@googlegroups.com
Peter Foley commented on New Feature PUP-4080

Branan Riley Melissa Stone pretty funny on this weeks triage video and I would never ever hold you to delivering a feature that you may or may not have committed to in the video

I don't mind if you decide to close the PR, I recognise that under the current policy it will not be committed into puppet core.

Although I will say that Yes I do still want this capability just as long as there is a way to inject an additional source provider via a module.

I currently have the ugly hack that I opened a PR against the puppetlabs-aws module which at least meets my short term requirements.

Jeff Adams (JIRA)

unread,
May 12, 2017, 4:32:05 PM5/12/17
to puppe...@googlegroups.com
Jeff Adams commented on New Feature PUP-4080

This would help us out a lot as well. We're masterless, so we're storing secrets in S3.

Use case is to specify the file resource source via hiera, using local puppet urls for local development/testing (secrets that aren't really secret and for systems that may not have access to S3), s3 urls for ci/uat/production (for the real secrets, and systems that can access S3)

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

John Duarte (JIRA)

unread,
May 17, 2017, 12:57:03 PM5/17/17
to puppe...@googlegroups.com
John Duarte updated an issue
 
Change By: John Duarte
Labels: triaged

John Duarte (JIRA)

unread,
May 17, 2017, 12:58:04 PM5/17/17
to puppe...@googlegroups.com
John Duarte updated an issue
Change By: John Duarte
Labels: needs_decision triaged

Ethan Brown (JIRA)

unread,
May 17, 2017, 12:59:04 PM5/17/17
to puppe...@googlegroups.com

Moses Mendoza (JIRA)

unread,
May 18, 2017, 2:06:03 PM5/18/17
to puppe...@googlegroups.com
Moses Mendoza updated an issue
Change By: Moses Mendoza
Labels: needs_decision  triaged

Louis Mayorga (JIRA)

unread,
Jul 3, 2018, 10:39:03 AM7/3/18
to puppe...@googlegroups.com
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Ryan Hall (JIRA)

unread,
Dec 12, 2018, 1:40:03 PM12/12/18
to puppe...@googlegroups.com

Dirk Heinrichs (JIRA)

unread,
Aug 2, 2019, 8:48:06 AM8/2/19
to puppe...@googlegroups.com

>4 years have passed and Puppet still can't do it. What a pity...

Anyway, meanwhile, there's the AWS Ruby SDK version 3, which essentially splits the monolithic gem into lots of smaller ones. Just install the S3 one and you'll get everything needed to support s3:// URIs.

That would make writing classes which pull files from either source much easier, as one could simply define the base URI in hiera instead of coding corner cases using "aws s3 cp ..." in exec's into classes which need to get files from S3 sometimes.

Dirk Heinrichs (JIRA)

unread,
Aug 2, 2019, 8:48:06 AM8/2/19
to puppe...@googlegroups.com

Oh, forgot to mention: Salt can...

Dirk Heinrichs (Jira)

unread,
Jun 7, 2023, 9:27:02 AM6/7/23
to puppe...@googlegroups.com

Hmmm, still not... sad.png

This message was sent by Atlassian Jira (v8.20.21#820021-sha1:38274c8)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages