Extending the File Type to allow setting an (Amazon Web Services) s3:/// URI

[Peter Foley]

Hi,

I am looking for some guidance on how easy it might be to add a new source type to the File puppet type in the puppet code base.

Essentially it would work something like:

node 'aws-ec2-instance' {
  file { "/etc/puppet/secure/keys/private_key.pkcs7.pem":
    owner  => puppet,
    group  => puppet,
    mode   => 0400,
    source => "s3://myorg-puppetmaster/hiera-eyaml.key"
  }
}

(I assume that this would run locally on the agent "aws-ec2-instance' and not the puppetmaster.  So the agent could leverage the credentials provided by the IAM role).

Using the AWS SDK for Ruby it seems like it would be straight forward to add to puppet.  However I am new to the puppet code base (and have limited Ruby programming experience).

I have been trying to find areas of the code base I think I would either need to modify or add files to.  Here is a list of files I believe I would need to extend:

https://github.com/puppetlabs/puppet/blob/master/lib/puppet/file_serving/terminus_selector.rb

• Add the new s3 terminus

https://github.com/puppetlabs/puppet/blob/master/lib/puppet/indirector/request.rb

• Update set_uri_key to detect the new s3 URI scheme
• Make changes to do_request?

https://github.com/puppetlabs/puppet/tree/master/lib/puppet/indirector

• Add a new s3.rb file describing how to interact with the s3 service

Is there other areas I would need to modify?

Is this something that would ultimately make it into the application?

I have looked at https://docs.puppetlabs.com/guides/install_puppet/from_source.html is there any other good references (blog posts) which would assist with developing puppet.

Thanks,

Peter

[Felix Frank]

Hi,

I'm working on a similar feature request in order to support http://
URLs. It's nigh complete:

https://github.com/puppetlabs/puppet/pull/3372/files

The code points you identified sound about right.

On the other hand, you would have a much easier time building your own
simple type in a module. Have you looked into available modules such as

https://forge.puppetlabs.com/branan/s3file

Having the support in the core file type is mainly useful if you expect
to require regular checksum based probes for upstream changes etc.

HTH,
Felix

On 01/22/2015 01:26 PM, Peter Foley wrote:
> Hi,
>
> I am looking for some guidance on how easy it might be to add a new
> source type to the File puppet type in the puppet code base.
>
> Essentially it would work something like:
>
> |
> node 'aws-ec2-instance'{
> file {"/etc/puppet/secure/keys/private_key.pkcs7.pem":
> owner =>puppet,
> group =>puppet,
> mode =>0400,

> *source **=>"s3://myorg-puppetmaster/hiera-eyaml.key"*

> }
> }
> |
>
> (I assume that this would run locally on the agent "aws-ec2-instance'
> and not the puppetmaster. So the agent could leverage the credentials
> provided by the IAM role).
>
> Using the AWS SDK for Ruby it seems like it would be straight forward to
> add to puppet. However I am new to the puppet code base (and have
> limited Ruby programming experience).
>
> I have been trying to find areas of the code base I think I would either
> need to modify or add files to. Here is a list of files I believe I
> would need to extend:
>
> https://github.com/puppetlabs/puppet/blob/master/lib/puppet/file_serving/terminus_selector.rb
>

> * Add the new s3 terminus
>
> https://github.com/puppetlabs/puppet/blob/master/lib/puppet/indirector/request.rb
>
> * Update set_uri_key to detect the new s3 URI scheme
> * Make changes to do_request?
>
>
> https://github.com/puppetlabs/puppet/tree/master/lib/puppet/indirector
>
> * Add a new s3.rb file describing how to interact with the s3 service

[Peter Foley]

Hi Felix,

Thanks for the pointer to your pull request.  I now have working code to pull a file out of an s3 bucket modeled on your code, instructions to try it are included in the github PR.  I still need to write documentation and spec tests.

https://tickets.puppetlabs.com/browse/PUP-4080

https://github.com/puppetlabs/puppet/pull/3675

I thought about your suggestion to include it in a separate module.  Although the idea does have merit I felt that it was a cleaner abstraction to just extend the puppet file resource type.  It would be easier to teach the file type how to pull down another source type than build a new type and replicate all the platform specific stuff to manage permission, file bucket, ...

Happy for you or anyone else to provide comments on the code.

Thanks,

Peter

[Luke Kanies]

FWIW - I love that you’re adding this, people have been asking for this kind of thing for years, and I agree it’s much cleaner and better to have it in the core type.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/54C0F09F.7030100%40alumni.tu-berlin.de.
> For more options, visit https://groups.google.com/d/optout.


—
http://puppetlabs.com/ | http://about.me/lak | @puppetmasterd