Re: [Puppet-dev] file concat library

[James Polley]

What benefits does this offer over https://github.com/ripienaar/puppet-concat ?

On Tue, Dec 18, 2012 at 3:00 PM, Richard Pijnenburg <richard.p...@gmail.com> wrote:

Hi all,

I got a native file concat library which i believe could be very useful.
Current implementations of file concat's work with temporary files and finds and can cause issues.

the library can be found at https://github.com/electrical/puppet-lib-file_concat

Im pretty sure it will need some work to make it clean and lean but its a good beginning i think.

Wonder what you all think of this.

Cheers.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-dev/-/SHMnxjGYBD4J.
To post to this group, send email to puppe...@googlegroups.com.
To unsubscribe from this group, send email to puppet-dev+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.

[Richard Pijnenburg]

The main benefit is that it doesn't work with temp files and an, in my opinion, ugly exec construct.

Other benefit is that because it doesn't work with temp files it won't corrupt the concating of files what fails at times, and you will be able to see exactly what changes with the puppet runs ( this is not possible with the exec thing )

Also its much faster because it doesn't write temp files and sorts/read/puts them, this one directly pulls it out of the inventory.

Op dinsdag 18 december 2012 05:13:50 UTC+1 schreef James Polley het volgende:

[Dominic Cleal]

Hi Richard,

On 18/12/12 04:00, Richard Pijnenburg wrote:
> Hi all,
>
> I got a native file concat library which i believe could be very useful.
> Current implementations of file concat's work with temporary files and
> finds and can cause issues.
>
> the library can be found at
> https://github.com/electrical/puppet-lib-file_concat
>
> Im pretty sure it will need some work to make it clean and lean but its
> a good beginning i think.
>
> Wonder what you all think of this.

Not meaning to rain on your parade, but there's another native concat
module which works pretty well:

https://github.com/onyxpoint/pupmod-concat

As you say, there are good benefits to this over an exec based
implementation, such as --noop support. Unfortunately there were some
issues stopping it being merged in core:

http://projects.puppetlabs.com/issues/7688

--
Dominic Cleal
Red Hat Engineering

[R.I.Pienaar]

----- Original Message -----
> From: "Richard Pijnenburg" <richard.p...@gmail.com>
> To: puppe...@googlegroups.com
> Sent: Tuesday, December 18, 2012 7:59:53 AM
> Subject: Re: [Puppet-dev] file concat library
>
> The main benefit is that it doesn't work with temp files and an, in
> my opinion, ugly exec construct.

I agree the exec is hacky, a native one would be great.

> Other benefit is that because it doesn't work with temp files it
> won't corrupt the concating of files what fails at times, and you

can you explain how this corruption happens? I've not had even a single
report of such corruption with the concat library

> will be able to see exactly what changes with the puppet runs ( this
> is not possible with the exec thing )

actually it is - the concat gives you full diffs of each snipper and the
final file it is managing on any change, do you have an example of what
you mean here?

Here's an example of the concat module restoring a line to the motd
that got removed, full diffs and standard filebucket behavior remains
in place and fully functional:

/Stage[main]/Motd/Concat[/etc/motd]/File[/etc/motd]/content:
--- /etc/motd 2012-12-18 09:18:53.817195258 +0000
+++ /tmp/puppet-file20121218-14618-1wl2lyi-0 2012-12-18 09:19:26.127176896 +0000
@@ -7,5 +7,6 @@
- ip6tables
- iptables
- mongodb
+ - pptp server


Info: FileBucket adding {md5}e2e7f25ac68ac9f74073804cc57748dd
Info: /Stage[main]/Motd/Concat[/etc/motd]/File[/etc/motd]: Filebucketed /etc/motd to puppet with sum e2e7f25ac68ac9f74073804cc57748dd
/Stage[main]/Motd/Concat[/etc/motd]/File[/etc/motd]/content: content changed '{md5}e2e7f25ac68ac9f74073804cc57748dd' to '{md5}ac6612feae066bbdf274848cf6186954'

> Also its much faster because it doesn't write temp files and
> sorts/read/puts them, this one directly pulls it out of the
> inventory.

the writing isnt the problem so much as all the forking to do the concating
and validation if any change has to happen if those bits gets done natively
it would be a huge speedup

[Erik Dalén]

On Tuesday 18 December 2012 at 10:20, R.I.Pienaar wrote:

>
>
> ----- Original Message -----
> > From: "Richard Pijnenburg" <richard.p...@gmail.com (mailto:richard.p...@gmail.com)>
> > To: puppe...@googlegroups.com (mailto:puppe...@googlegroups.com)
> > Sent: Tuesday, December 18, 2012 7:59:53 AM
> > Subject: Re: [Puppet-dev] file concat library
> >

> > will be able to see exactly what changes with the puppet runs ( this
> > is not possible with the exec thing )
>
>
>
> actually it is - the concat gives you full diffs of each snipper and the
> final file it is managing on any change, do you have an example of what
> you mean here?
>
> Here's an example of the concat module restoring a line to the motd
> that got removed, full diffs and standard filebucket behavior remains
> in place and fully functional:
>
> /Stage[main]/Motd/Concat[/etc/motd]/File[/etc/motd]/content:
> --- /etc/motd 2012-12-18 09:18:53.817195258 +0000
> +++ /tmp/puppet-file20121218-14618-1wl2lyi-0 2012-12-18 09:19:26.127176896 +0000
> @@ -7,5 +7,6 @@
> - ip6tables
> - iptables
> - mongodb
> + - pptp server
>
>
> Info: FileBucket adding {md5}e2e7f25ac68ac9f74073804cc57748dd
> Info: /Stage[main]/Motd/Concat[/etc/motd]/File[/etc/motd]: Filebucketed /etc/motd to puppet with sum e2e7f25ac68ac9f74073804cc57748dd
> /Stage[main]/Motd/Concat[/etc/motd]/File[/etc/motd]/content: content changed '{md5}e2e7f25ac68ac9f74073804cc57748dd' to '{md5}ac6612feae066bbdf274848cf6186954'
>

This doesn't work properly on a --noop run though, Can't see the changes to the final file as the temp files won't be written and exec not executed. If the --noop run is the first run for that concat you will even get a resource failure when it tries to read the local file the exec should have produced.

[Erik Dalén]

One benefit of ripieenar's non native one though is that it creates a file resource for the destination file, so if you try to create another file resource for that file you get a proper duplicate resource declaration error. Is there any way to implement that protection with a native resource like this?


--
Erik Dalén

On Tuesday 18 December 2012 at 05:00, Richard Pijnenburg wrote:

> Hi all,
>
> I got a native file concat library which i believe could be very useful.
> Current implementations of file concat's work with temporary files and finds and can cause issues.
>
> the library can be found at https://github.com/electrical/puppet-lib-file_concat
>
> Im pretty sure it will need some work to make it clean and lean but its a good beginning i think.
>
> Wonder what you all think of this.
>

> Cheers.
> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/puppet-dev/-/SHMnxjGYBD4J.

> To post to this group, send email to puppe...@googlegroups.com (mailto:puppe...@googlegroups.com).
> To unsubscribe from this group, send email to puppet-dev+...@googlegroups.com (mailto:puppet-dev+...@googlegroups.com).

[R.I.Pienaar]

yeah, that's known, noop totally messes with concat in ways that's
hard to figure out - unfortunately unavoidable.

to be clear, I'd totally prefer a native concat type over the concat
module :)

[R.I.Pienaar]

----- Original Message -----
> From: "Erik Dalén" <erik.gus...@gmail.com>
> To: puppe...@googlegroups.com
> Sent: Tuesday, December 18, 2012 11:54:33 AM
> Subject: Re: [Puppet-dev] file concat library
>

> One benefit of ripieenar's non native one though is that it creates a
> file resource for the destination file, so if you try to create
> another file resource for that file you get a proper duplicate
> resource declaration error. Is there any way to implement that
> protection with a native resource like this?

it also allows you to include non managed files into the final
result.

often you want users to edit some part of a file, lets say a motd
like:

Machine overview

- Machine notes
- Machine notes

Stupid security notice

you might want to give the machine owner the right to edit machine
notes while the systems team must be sure hte security notice is
always present, the concat module lets you do that.

concat::fragment{"motd notes": ensure => /etc/machine_notes}

this is something I used to use a lot and I've not seen it in the
native concat modules.

Apart from that at the very least the file writer in the provider
of this one must write to a temp file and move that rather than
directly overwrite the destination, LOTS can go wrong with that
approach

[Richard Pijnenburg]

Hi,

The 'corruption' we've seen is that at times it doesn't properly pick up the files to concat from.

I must admit that this wasn't with your concat module but the one we used at work. ( R.I. Pienaar; eCG should ring a bell i think? )

Op dinsdag 18 december 2012 10:20:22 UTC+1 schreef R.I. Pienaar het volgende:

----- Original Message -----
> From: "Richard Pijnenburg" <richard.p...@gmail.com>
> To: puppe...@googlegroups.com
> Sent: Tuesday, December 18, 2012 7:59:53 AM
> Subject: Re: [Puppet-dev] file concat library
>
> The main benefit is that it doesn't work with temp files and an, in
> my opinion, ugly exec construct.

I agree the exec is hacky, a native one would be great.

> Other benefit is that because it doesn't work with temp files it
> won't corrupt the concating of files what fails at times, and you

can you explain how this corruption happens? I've not had even a single
report of such corruption with the concat library

> Erik Dalén: One benefit of ripieenar's non native one though is that it creates a file resource for the destination file, so if you try to create another file resource for that file you get a proper duplicate resource declaration error. Is there any way to implement that 

> protection with a native resource like this? 

Natively it uses the file resource code and should give a duplicate resource deceleration when wanting to write the same file.

[R.I.Pienaar]

----- Original Message -----
> From: "Richard Pijnenburg" <richard.p...@gmail.com>
> To: puppe...@googlegroups.com
> Sent: Tuesday, December 18, 2012 2:52:51 PM
> Subject: Re: [Puppet-dev] file concat library
>

> Hi,
>
> The 'corruption' we've seen is that at times it doesn't properly pick
> up the files to concat from.
> I must admit that this wasn't with your concat module but the one we
> used at work. ( R.I. Pienaar; eCG should ring a bell i think? )

OK :) Def not something I've ever seen or had reported about mine :)