File header including timestamp

[j4m3s]

I have noticed that if I create a hosts file entry using the host {} type, a header is added to teh top of the file showing when it was last updated.  I'm wondering if it is possible something similar in a template?  Calling <%= Time.now().to_s() %> inside the template (obviously) causes the file to be updated each time puppet runs - which isn't what I'm looking for. 

I have tried to find the host type on github (in the puppet source) but couldn't track it down.  Is it possible to reproduce the functionality in my own template headers?

Thanks, James.

[Matthew Burgess]

There's a thread on this topic at
https://groups.google.com/forum/?fromgroups#!search/puppet$20cron$20job$20class/puppet-users/PMTVEpXw8I0/Wsckx5euwRgJ
- in short, copy and paste code from Puppet's crontab.rb.

Hope this helps,

Matt.

[j4m3s]

Thank you very much Matt.  My searches hadn't turned that up - I was searching on keywords relating to templates I think.

Thanks again, James.

[jcbollinger]

On Wednesday, November 7, 2012 8:17:50 AM UTC-6, j4m3s wrote:

Thank you very much Matt.  My searches hadn't turned that up - I was searching on keywords relating to templates I think.

Thanks again, James.

Do note that the reason that approach can work for cron jobs and hosts entries is that the relevant resources model individual parts of the file (one job or one host) rather than the file as a whole.  I imagine you could patch something together that would treat ordinary files as made up of parts, but it will probably need to be somewhat specific to the target file format.  I personally can't see doing that unless it were particularly important to have a content timestamp in the file.  I certainly don't see doing it routinely for every managed file.


John

[James Fellows]

Thanks John. Yes I realised that from going through the crontab.rb
and parsedfile.rb scripts.

You are absolutely correct that there is nothing that could easily be
lifted and used on top of puppet.

I believe that, in order to make this work, a change would be required
to one or more of the core template funtions in puppet - possibly to
add a "flag" to say to include the last-updated header (or call a
separate templateWithHeader function) , and to later ignore it when
parseing the files to check for changes. The fact that the comment
marker would be different in some cases does complicate matters - but
not insurmountably. It could default to being determined based on the
file extension (the part before the .erb) and the function could take
a parameter that overwrode it if required. It's a common process when
auto-generating source file headers for copyright in maven projects
for example (where there are some java files, some sql files, some xml
files etc).

I'm actually a Java guy by background so am loving the power of puppet
but struggling someone to make anything more than the most basic
puppet functions. That said, if I get the chance I would love to code
or contribute to an enhancement like this, if it would be valued by
the community.

For me, the benefit of the header is being able to immediately tell
whether the file has been modified by someone or something since the
last puppet run (by comparing the header to the last modified
timestamp on the file). It's very helpful when fault-finding. It
also gives a nice straightforward view (from within the file itself)
of the last time it changed - again helping to narrow down the source
of problems.

Would others find it useful? If so I could raise a feature request
for it to open a formal discussion, and (one day, maybe!) fork and
contribute to the code.

James

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/XN7EX_wBe2EJ.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.

[jcbollinger]

On Thursday, November 8, 2012 3:49:12 AM UTC-6, j4m3s wrote:

Thanks John.  Yes I realised that from going through the crontab.rb
and parsedfile.rb scripts.

You are absolutely correct that there is nothing that could easily be
lifted and used on top of puppet.

I believe that, in order to make this work, a change would be required
to one or more of the core template funtions in puppet - possibly to
add a "flag" to say to include the last-updated header (or call a
separate templateWithHeader function) , and to later ignore it when
parseing the files to check for changes.

Nope.  There is no way the issue could be solved by modifying the template functions, because they have nothing to do with checking file content for changes.  I outlined already what would be needed: a file with a content-timestamp header would need to be treated as at least two parts (a composite resource -- header and body).  It might be possible to build something like that on top of the Concat module, which is all about building files from pieces.  Even if not, it's at that level (resource type) where the problem would need to be addressed.

 

 The fact that the comment
marker would be different in some cases does complicate matters - but
not insurmountably.  It could default to being determined based on the
file extension (the part before the .erb) and the function could take
a parameter that overwrode it if required.  It's a common process when
auto-generating source file headers for copyright in maven projects
for example (where there are some java files, some sql files, some xml
files etc).

The problem is not with generating or formatting the header, it's with ignoring the header when the file content is checked to see whether it's in sync.  The template functions have nothing whatever to do with that.

 

I'm actually a Java guy by background so am loving the power of puppet
but struggling someone to make anything more than the most basic
puppet functions.  That said, if I get the chance I would love to code
or contribute to an enhancement like this, if it would be valued by
the community.

I wouldn't personally use it.  For me,

1. It only matters that the content is in sync, not when it was generated.
   
2. If I wanted to know when the content was generated, then I would be content to rely on the file's modification timestamp as recorded by the file system.
3. The extra complexity involved in managing a content timestamp would produce extra work for the master and extra space for bugs.
   

Indeed, if I were looking for something along these lines, it would more likely be a template version or timestamp than a content timestamp, and a template timestamp doesn't need any special support.

For me, the benefit of the header is being able to immediately tell
whether the file has been modified by someone or something since the
last puppet run (by comparing the header to the last modified
timestamp on the file).  It's very helpful when fault-finding.  It
also gives a nice straightforward view (from within the file itself)
of the last time it changed - again helping to narrow down the source
of problems.

I understand.  I just don't have concerns along those lines with the files I am currently managing on any of my hosts.  I also think you may be overlooking some of the implications of the Puppet agent keeping managed files in sync -- unlike once- or rarely-generated files, files with Puppet-managed content can be re-generated up to once every Puppet run (with new content timestamps, if any).  Manual changes will be clobbered on that schedule, which is once every 30 minutes by default.  As a result, admins will quickly learn not to make manual changes to managed files, and if they do make such changes then you can always tease out the information you want (with more difficulty) from the Puppet log.


John

[Jakov Sosic]

On 11/08/2012 10:49 AM, James Fellows wrote:

> For me, the benefit of the header is being able to immediately tell
> whether the file has been modified by someone or something since the
> last puppet run (by comparing the header to the last modified
> timestamp on the file).

You can already do that with puppet noop run:

# puppet agent -t --noop

which won't modify system but only announce what changes will be applied
on the next "real" agent run. If you find your file in the list of
resources that puppet wants to (re)appy, you can be certain that someone
has messed around that resource. When it was done can be seen by
filesystem timestamps.

If, on the other hand, puppet was the last who modified that file and
you want to know when, again you can use the filesystem timestamps.

Now, the only corner case in which I can see potential use of that kind
of header is if someone changed manifests on your master, and puppet now
wants to modify the file - which was last modified also by puppet. But
if you use some kind of VCS for keeping track of your manifests, you
could know even that.

> Would others find it useful? If so I could raise a feature request
> for it to open a formal discussion, and (one day, maybe!) fork and
> contribute to the code.

That kind of header IMHO only brings one thing to the table - and that
is fancy look :) Other than that I find it pretty much useless.

But then again I don't mind useless as long as it doesn't do performance
penalties. But this does impose additional strain on both the master
(catalog compilation time increase) and the agents (run time increase).
And management software should be as non-intrusive as possible. I want
puppet, zabbix, ossec and other management related agents to consume as
little resources as possible...


--
Jakov Sosic
www.srce.unizg.hr