Find string and replace line

[Al @ Lab42]

A question, to save time:
has someone already written a reusable custom type that finds a string
in a file the replace its entire line with a specified line?

Something like the replace that David has documented here
http://puppet.reductivelabs.com/trac/puppet/wiki/Recipes/SimpleText
but with the difference that instead of a normal string substitution
there's a whole line substituition.

If based on the replace type in the above link, it should work somehow
like this:

replaceline {
postfixadmin_configdb_host:
file => $operatingsystem ?{
default => "/var/www/postfixadmin/config.inc.php",
},
pattern => "database_host",
replacement => "$CONF['database_host'] = '$postfix_mysqlhost'";
}


The replace type on the wiki is this:
define replace($file, $pattern, $replacement) {
$pattern_no_slashes = slash_escape($pattern)
$replacement_no_slashes = slash_escape($replacement)

exec { "/usr/bin/perl -pi -e 's/$pattern_no_slashes/
$replacement_no_slashes/' '$file'":
onlyif => "/usr/bin/perl -ne 'BEGIN { \$ret = 1; } \$ret = 0 if /
$pattern_no_slashes/ && ! /$replacement_no_slashes/ ; END { exit \
$ret; }' '$file'",
}
}

Maybe for someone who knows perl better than my zero, the modification
is quite simple.

Thanks for any suggestion

[Adam Jacob]

On Sat, May 10, 2008 at 1:08 AM, Al @ Lab42 <lab4...@gmail.com> wrote:
> If based on the replace type in the above link, it should work somehow
> like this:
>
> replaceline {
> postfixadmin_configdb_host:
> file => $operatingsystem ?{
> default => "/var/www/postfixadmin/config.inc.php",
> },
> pattern => "database_host",
> replacement => "$CONF['database_host'] = '$postfix_mysqlhost'";
> }
>
>
> The replace type on the wiki is this:
> define replace($file, $pattern, $replacement) {
> $pattern_no_slashes = slash_escape($pattern)
> $replacement_no_slashes = slash_escape($replacement)
>
> exec { "/usr/bin/perl -pi -e 's/$pattern_no_slashes/
> $replacement_no_slashes/' '$file'":
> onlyif => "/usr/bin/perl -ne 'BEGIN { \$ret = 1; } \$ret = 0 if /
> $pattern_no_slashes/ && ! /$replacement_no_slashes/ ; END { exit \
> $ret; }' '$file'",
> }
> }

define replace_line($file, $pattern, $replacement) {

$pattern_no_slashes = slash_escape($pattern)
$replacement_no_slashes = slash_escape($replacement)

exec { "/usr/bin/perl -pi -e 's/^.*$pattern_no_slashes.*$/

$replacement_no_slashes/' '$file'":
onlyif => "/usr/bin/perl -ne 'BEGIN { \$ret = 1; } \$ret = 0 if /
$pattern_no_slashes/ && ! /$replacement_no_slashes/ ; END { exit \
$ret; }' '$file'",
}
}

Give that a shot.

Adam

--
HJK Solutions - We Launch Startups - http://www.hjksolutions.com
Adam Jacob, Senior Partner
T: (206) 508-4759 E: ad...@hjksolutions.com

[Al @ Lab42]

Yeah that works almost fine. I was doing the same with sed but after
all is just a regexp.
It's ok for normal cases but I'm just struggling with some special
chars (in the example I'm working on there are $ ' [ ; ... ) and I'm
finding erratic results according to how I escape chars.
I'm trying to find a coherent and robust solution (i find such a
function really vital for many activities (typically the configuration
of some parameters on files over which you have no real control and no
real guarantee of future ccompatibility, such as config files of php
web apps).
The best way would be to be able to define as $pattern and
$replacement exactly the expected string, without any escapes excluded
\" but this would require probably some tweaks on the ruby wrapper
used in the function (always courtesy of David Schmitt):
# escape slashes in a String
module Puppet::Parser::Functions
newfunction(:slash_escape, :type => :rvalue) do |args|
args[0].gsub(/\//, '\\/')
end
end

still I wonder if it's a good idea to write a more general
regexp_escape ruby wrapper that correctly escapes all the characters
that may break a regexp line.

Working on it, if anyone has another ready and quick shortcut, it's
always welcomed.

Thanks again Adam.

Regards,
al

On 10 Mag, 19:16, "Adam Jacob" <a...@hjksolutions.com> wrote:

> HJK Solutions - We Launch Startups -http://www.hjksolutions.com
> Adam Jacob, Senior Partner
> T:(206) 508-4759E: a...@hjksolutions.com

[Thomas Bellman]

Al @ Lab42 wrote:

> A question, to save time:
> has someone already written a reusable custom type that finds a string
> in a file the replace its entire line with a specified line?

I have written a few extension types to Puppet for doing various
manipulations of lines in text files. I have planned to clean
them up and post them, but haven't had the time to do so yet.
Basically, they are:

- delete_lines, for deleting lines matching a regexp
- ensure_lines, for inserting and replacing a line
- regexp_replace_lines, for replacing (parts of) lines
- replace_sections, for replacing entire sections of lines

I can probably put them up as they are on a website sometime
tomorrow, if you want, but they really need some cleanup, both
in terms of their interface (names, parameters), and internals
(I only know a little Ruby, and too little of the internal
interfaces in Puppet).

/Thomas Bellman

[Al @ Lab42]

Thomas,
any working sample, even if not cleaned up, is welcomed.
I managed to have a working type using Adam's code: it works with
plain strings but there are some things to fix with special chars
like $ and ' ( I must ascii encode them and the command that checks if
they are already there doesn't work well).
Since I'm busy with other configurations I've left this part in s not
fully cleaned state, but I can reconsider any other way to obtain the
same result.

I think that a well tested, with standard naming conventions, set of
functions / types for similar activities (string substitution, lines
editing and so ono) could be very interesting for the community and a
good brick to build a community common modules infrastructure.

So, post what you want/can, I'll try it.

Best regards,
al

[Thomas Bellman]

Al @ Lab42 wrote:

> Thomas,
> any working sample, even if not cleaned up, is welcomed.
> I managed to have a working type using Adam's code: it works with
> plain strings but there are some things to fix with special chars
> like $ and ' ( I must ascii encode them and the command that checks if
> they are already there doesn't work well).
> Since I'm busy with other configurations I've left this part in s not
> fully cleaned state, but I can reconsider any other way to obtain the
> same result.
>
> I think that a well tested, with standard naming conventions, set of
> functions / types for similar activities (string substitution, lines
> editing and so ono) could be very interesting for the community and a
> good brick to build a community common modules infrastructure.

OK, here goes. I am attaching a snapshot of what we have to this
message as a gzip:ed tar archive. There is some documentation of
the types and functions inline in the source code. I have also
written a couple of definitions using these types, that work on a
somewhat higher level, that I'm also enclosing.

The types might seem to be a bit overgeneralized, with their
group and section parameters, but that is because I want to use
them as building blocks in definitions that work on structured
text files.

If you have any questions on how to use them, feel free to ask.
And if you have suggestions for improvement, both of the code and
of the design, I am very interrested.

/Thomas Bellman

[Al @ Lab42]

Thomas,
thank you, really.
The set of functions you provided seems quite interesting, I'll surely
examine them better.
My incumbet necessity (suppose quite common and easily faced by
others) is to replace in a file's (typically a php/perl/whatever
config file) single lines with basic configuration parameters.
I'm confident you can manage this with your functions (a fast sample
would be appreciated).

For the moment, starting from zero ruby knowledge, not great affinity
with regexps and some cuts of puppet code found around, if have done
something like this:

An elementary ruby external function, regexp_escape.rb, that escapes
regexps metachars AND single quotes:

# escapes regexp metachars in a String
module Puppet::Parser::Functions
newfunction(:regexp_escape, :type => :rvalue,
:doc => "Escapes RegExp metacharacters") do |vals|
vals[0] = Regexp.escape(vals[0])
vals[0].gsub(/[']/, '\\\\\'')
# args[0].gsub(%r{/}, '')
## [ \ ^ $ . | ? * + ( )
end
end



A custom define, replaceline, that uses regexp_escape, that works well
in MANY cases BUT NOT ALL (some nasty patterns, notes inside):

# Finds lines beginning with $pattern and substitutes entire line with
$replacement

define replaceline($file, $pattern, $replacement) {
$pattern_escaped = regexp_escape($pattern)
$replacement_escaped = regexp_escape($replacement)
exec {

# OK with pattern: define(DB_USER
# NOP with pattern: my($db_user)
## sed -i -e "s/^.*my(\$db_user).*$/my\(\$db_user\)\ =\ \'mailwatch\';
\ \#\ Modified\ by\ Puppet/" /usr/lib/MailScanner/MailScanner/
CustomFunctions/MailWatch.pm WORKS
# OK with pattern: $CONF['database_user']
"sed -i -e \"s/^.*$pattern.*$/$replacement_escaped/\" $file":

# BREAKS with pattern: define(DB_USER
# NOP with pattern: my($db_user)
# OK with pattern: $CONF['database_user']
## "sed -i -e \"s/^.*$pattern_escaped.*$/$replacement_escaped/\"
$file":

# OK with pattern: define(DB_USER
# NOP with pattern: my($db_user)
## ( grep -F "my(\$db_host) " /usr/lib/MailScanner/MailScanner/
CustomFunctions/MailWatch.pm WORKS )
# OK with pattern: $CONF['database_user']
unless => "grep -F \"$replacement\" $file",

}
}


various puppetism for different php and perl configuration files, such
as:

# Mailwatch config file configuration

$mailwatchconf = $operatingsystem ?{
default => "/usr/lib/MailScanner/MailScanner/
CustomFunctions/MailWatch.pm",
}

replaceline {
mailwatch_configdb_host:
file => $mailwatchconf,
pattern => "my(\$db_host)",
replacement => "my(\$db_host) =
'$mailwatch_mysqlhost'; # Modified by Puppet";

mailwatch_configdb_dbname:
file => $mailwatchconf,
pattern => "my(\$db_name)",
replacement => "my(\$db_name) =
'$mailwatch_mysqldbname'; # Modified by Puppet";
### replacement => "\x2[\x27database_name\x27] =
\x27$postfix_mysqldbname\x27; # Modified by Puppet";

mailwatch_configdb_user:
file => $mailwatchconf,
pattern => "my(\$db_user)",
replacement => "my(\$db_user) =
'$mailwatch_mysqluser'; # Modified by Puppet";

mailwatch_configdb_password:
file => $mailwatchconf,
pattern => "my(\$db_pass)",
replacement => "my(\$db_pass) =
'$mailwatch_mysqlpassword'; # Modified by Puppet";
}


--
In the comments of replaceline there are the exceptions I'm finding.
Escaping the right chars has been quite a pain, in order to obtain the
wanted result for insidious strings like: "define(DB_USER" or
"my($db_user)" or "$CONF['database_user']"
My last problem is escaping properly and ONLY the $ when inside ( ).
I suppose I'm going to write another simple ruby function for this.

Anyway.
Whatever my personal problems with puppet, I've found myself wasting
too much time on things that have been probably already faced and
solved.

Puppet code and modules sharing and standardization inside the
community is really a necessity, for everybody, to save time already
spent by others.

I'm convinced that we should start with few simple steps in order to
define a Common Public Community Puppet Module Repositoy.

I can imagine basic elements which should not have incompatibility
problems:
- custom facts with possibly standard convention for variables names
- custom, well tested, functions for typical operations on files
- standard, cross operating system, and extremely basic modules for
applications

With standard basic modules I mean really basic (and easily extendable
with standard naming convention) modules that should just install the
program, manage the services and ensures the presence of the main
configuration file, being natively platfrom indipendent for these
types (see example later).

All the rest is left to own tastes, skills and needs:
- submodules to handle different distros peculiarities
- submodules to configure the application in whatever way (according
to anyone taste, since I've realized that every solution can be a good
solution if it fits the local needs).
So that if I decide to configure Apache with single files, templates,
building blocks of texts with whatever custom function, method and
preferred variables names, I can do it.
Starting from the basic apache module and possibily using the public
community alternative modules of different sources (ex: apache::nsc,
apache::davids, apache::lab42 or whatever public repository)

Upon these basic-common-shared and rather static modules (they should
not change too much over time) everyone, I think, can easily adapt and
extend his set of configurations as preferred.

An example of such a basic class could be something like (the use of
"custom" $update variable, which I find useful to easily defin if a
node should be automatically updated or not, can be discussed, as
everything in this proposal):

class apache {

package { apache:
name => $operatingsystem ? {
debian => "apache2",
default => "httpd",
},
ensure => $update ?{
true => "latest",
false => "present",
default => "present",
},
}

service { apache:
name => $operatingsystem ? {
debian => "apache2",
# community efforts would be to gather here
all alternatives
default => "httpd",
},
ensure => running,
enable => true,
pattern => $operatingsystem ? {
debian => "/usr/sbin/apache2",
default => "/usr/sbin/httpd",
},
hasrestart => true,
hasstatus => true,
require => Package["apache"],
subscribe => File["httpd.conf"],
}

file {
"httpd.conf":
mode => 644, owner => root, group => root,
require => Package[apache],
backup => local,
ensure => present,
path => $operatingsystem ?{
default => "/etc/httpd/conf/
httpd.conf",
},
}
}

Incidentally a standard structure for a basic module can be easily
generated automatically by scripts or tools that can inherit packages/
service names and file pathnames from existing sources. Moreover it
can be easily managed with a extendable web frontend that would gift
the point and click masses with glimpses of the power of puppet.

But, hey, were we talking about something else?
Sorry for Over Posting, somehow hour is late, here.

All the best,
al

[Thomas Bellman]

Al @ Lab42 wrote:

> For the moment, starting from zero ruby knowledge, not great affinity
> with regexps and some cuts of puppet code found around, if have done
> something like this:

[snipped for brevity]

Feeding patterns to sed via a shell escape is non-trivial. You
need to quote things *both* for sed and for the shell. However,
your function regexp_escape() quotes things for Ruby regexps, and
a little bit for shell. Since sed and Ruby does not use the same
syntax for regexps, things will break. For instance, Ruby uses
"(" and ")" for grouping, while sed uses "\(" and "\)".

I'm fairly adept at using shell, sed, and regexps in general (in
lots of different programs, all having their own syntaxes and
quirks), and I seriously recommend against trying to do this kind
of thing with shell escapes; it's painful getting it right.

If I understand your examples right, this is how to do them with
my types:

define fixedstring_replace_lines($file, $pattern, $replacement)
{
$pe = regexp_quote($pattern)
$pe2 = ".*${pe}.*"
regexp_replace_lines {
"fixed--$title--$file--$pattern":
file => $file,
pattern => $pe2,
replacement => $replacement;
}
}

fixedstring_replace_lines {

mailwatch_configdb_host:
file => $mailwatchconf,
pattern => "my(\$db_host)",

replacement => "my(\$db_host) = '$mailwatch_mysqlhost'; #Puppet!";
}

This, however, assumes that the Puppet variable $mailwatch_mysqlhost
does not contain any characters that have special meaning within
Perl strings, or within Ruby replacement strings (thus, avoid
apostrophes and backslashes there).

Another way would be:

ensure_line {
mailwatch_configdb_host:
file => $mailwatchconf,
line => "my(\$db_host) = '$mailwatch_mysqlhost'; #Puppet!",
pattern => ".*my\\(\$db_host\\)";
}

This has the feature that if there is no line 'my($db_host)' in
the $mailwatchconf file, one will be added. regexp_replace_lines
won't do that. There are parameters that control where the line
is added.

Also, the line parameter to ensure_line is a fixed string, and
will not be interpreted for Ruby replacement escapes (see
http://www.ruby-doc.org/core/classes/String.html#M000831); you
will of course still need to watch out for Perl metacharacters.

I could have used regexp_quote() here also, and I would consider
making a special define for this:

define mailwatch_setting($value)
{
$l = sprintf("my(\$%s) = '%s'; #Puppet was here!",
$name, $value)
$p = regexp_quote("my(\$$name)")

ensure_line {
"mailwatch--$name":
file => $mailwatchconf,
line => $l,
pattern => ".*$p.*";
}
}

mailwatch_setting {
db_host: value => $mailwatch_mysqlhost;
db_name: value => $mailwatch_mysqldbname;
db_user: value => $mailwatch_mysqluser;
db_pass: value => $mailwatch_mysqlpassword;
}

If I wanted to be extra safe, I would create a custom function
'perl_string_quote()' that performed the proper quoting of
$value in the mailwatch_setting definition.

I hope these examples are of some help. (I haven't tested them,
so I hope I haven't made some foolish mistakes.)

/Thomas Bellman

[Al @ Lab42]

On May 15, 1:25 pm, Thomas Bellman <bell...@nsc.liu.se> wrote:
> For instance, Ruby uses
> "(" and ")" for grouping, while sed uses "\(" and "\)".

This definitively explains some of the odd behaviours I've found. :-I

> If I understand your examples right, this is how to do them with
> my types:
>
> define fixedstring_replace_lines($file, $pattern, $replacement)
> {
> $pe = regexp_quote($pattern)
> $pe2 = ".*${pe}.*"
> regexp_replace_lines {
> "fixed--$title--$file--$pattern":
> file => $file,
> pattern => $pe2,
> replacement => $replacement;
> }
> }
>
> fixedstring_replace_lines {
> mailwatch_configdb_host:
> file => $mailwatchconf,
> pattern => "my(\$db_host)",
> replacement => "my(\$db_host) = '$mailwatch_mysqlhost'; #Puppet!";
> }
>
> This, however, assumes that the Puppet variable $mailwatch_mysqlhost
> does not contain any characters that have special meaning within
> Perl strings, or within Ruby replacement strings (thus, avoid
> apostrophes and backslashes there).

Erm sorry for ignorance, apostrophe is a single quote? If so, I fear I
have to use them in some cases.

I wonder, and I ask, if you think it could be a good idea to ascii
encode all the pattern matching string that should be used, withouth
being interpretred by the sed/perl substitution command.
Something like $pe =ascii_encode($pattern)
I've no idea on how to make this ascii_encode function in ruby, I
think it should not be difficult and probably there's already an handy
class.

> Another way would be:
>
> ensure_line {
> mailwatch_configdb_host:
> file => $mailwatchconf,
> line => "my(\$db_host) = '$mailwatch_mysqlhost'; #Puppet!",
> pattern => ".*my\\(\$db_host\\)";
> }
>
> This has the feature that if there is no line 'my($db_host)' in
> the $mailwatchconf file, one will be added. regexp_replace_lines
> won't do that. There are parameters that control where the line
> is added.
>
> Also, the line parameter to ensure_line is a fixed string, and

> will not be interpreted for Ruby replacement escapes (seehttp://www.ruby-doc.org/core/classes/String.html#M000831);you

> will of course still need to watch out for Perl metacharacters.
>
> I could have used regexp_quote() here also, and I would consider
> making a special define for this:
>
> define mailwatch_setting($value)
> {
> $l = sprintf("my(\$%s) = '%s'; #Puppet was here!",
> $name, $value)
> $p = regexp_quote("my(\$$name)")
>
> ensure_line {
> "mailwatch--$name":
> file => $mailwatchconf,
> line => $l,
> pattern => ".*$p.*";
> }
> }
>
> mailwatch_setting {
> db_host: value => $mailwatch_mysqlhost;
> db_name: value => $mailwatch_mysqldbname;
> db_user: value => $mailwatch_mysqluser;
> db_pass: value => $mailwatch_mysqlpassword;
> }
>
> If I wanted to be extra safe, I would create a custom function
> 'perl_string_quote()' that performed the proper quoting of
> $value in the mailwatch_setting definition.
>
> I hope these examples are of some help. (I haven't tested them,
> so I hope I haven't made some foolish mistakes.)

These examples are really useful and I really thank you for the nice
effort in giving explanations.
I need to know something, before feeling free to use your code: I'm
going (actually I've already started in http://live.lab42.it/puppetinfrastructure/
) to release most of my puppet configurations and modules (where there
are not customers legitimate constraints) with a Creative Commons
Attribution Licence.
So before starting to use any of the code you provided I prefer to
have your explicit permission.
Of course I would leave any copyright reference and credits in your
code.
I think the best solution in the mid term is to have a common public
repository of shared code (erm, have I've already talked about
that ? :-)

Best regards,
Al

[Thomas Bellman]

Al @ Lab42 wrote:

>> This, however, assumes that the Puppet variable $mailwatch_mysqlhost
>> does not contain any characters that have special meaning within
>> Perl strings, or within Ruby replacement strings (thus, avoid
>> apostrophes and backslashes there).

> Erm sorry for ignorance, apostrophe is a single quote?

Yes: the ' character (ASCII code 0x27).

> If so, I fear I
> have to use them in some cases.

Then you need to quote them properly for Perl. (At least I *assume*
that it is Perl that is going to interpret the MailWatch.pm file;
the syntax looks like Perl, and the .pm suffix is often used by
Perl libraries.) Note that the problem here is that if you get a
line saying

my($foo) = 'foo'bar';

in a Perl script, then Perl will complain when trying to execute
it:

$ cat test.pl
#! /usr/bin/perl
my($foo) = 'foo'bar';
print "$foo\n";

$ perl test.pl
Bad name after bar' at test.pl line 2.

My Puppet types will gladly create such a line:

$ cat test.pp

node default {
# Note that the current implementation of ensure_line implicitly
# anchors the pattern regexp to the start of the line, but *not*
# to the end of the line.
ensure_line {
test1:
file => "/tmp/syntaxerror.pl",
line => "my(\$foo) = 'foo'bar';",
pattern => regexp_quote("my(\$foo)");
test2:
file => "/tmp/syntaxerror.pl",
line => 'my($gazonk) = \'gazonk\'del\';',
pattern => regexp_quote('my($gazonk)');
}
}

$ >/tmp/syntaxerror.pl
$ RUBYLIB=~/puppetlib/ruby puppet --libdir=$HOME/puppetlib test.pp
notice: //Node[default]/Ensure_line[test1]/line: line changed '' to 'my($foo) = 'foo'bar';'
notice: //Node[default]/Ensure_line[test2]/line: line changed '' to 'my($gazonk) = 'gazonk'del';'

$ cat /tmp/syntaxerror.pl
my($foo) = 'foo'bar';
my($gazonk) = 'gazonk'del';

> I wonder, and I ask, if you think it could be a good idea to ascii
> encode all the pattern matching string that should be used, withouth
> being interpretred by the sed/perl substitution command.
> Something like $pe =ascii_encode($pattern)
> I've no idea on how to make this ascii_encode function in ruby, I
> think it should not be difficult and probably there's already an handy
> class.

I think you are confused. In my opinion, it is a really bad idea
to shell out to sed in order to edit files from Puppet. Don't go
that way. The quoting problems are much too painful to make it
worth it; write native Puppet types instead (or use mine :-). My
types do *not* use sed, and not shell either, and thus avoids
much of the quoting pain. Not having to fork and exec a shell
and a sed process gives better performance also.

You may still need to quote the pattern parameter (and other
regexp parameters), depending on how you want them interpreted,
but my regexp_quote() function can do that for you.

If you absolutely insist on doing it the foolish way (calling sed
via the exec type), then I'm not going to help; you will have to
build your own gun to shoot yourself in the foot. :-) (And the
name 'ascii_encode()' is a rather bad name too; most regexps are
already in ASCII. ascii_encode() sounds more like something to
encode things in Base64, uuencode, Quoted unPrintable, or UTF-7.)

> So before starting to use any of the code you provided I prefer to
> have your explicit permission.

Absolutely! Go ahead!

> I think the best solution in the mid term is to have a common public
> repository of shared code (erm, have I've already talked about
> that ? :-)

Even better would be to get resource types like these into core
Puppet. I think editing of text files is important enough to
warrant that. Yes, types managing higher-level resources are
better than these low-level text editing operations, but it is
unrealistic to have Puppet natively support all the thousands of
strange config file formats that already exist, and the hundreds
that will be created just in the near future. Thus I believe it
is important to have these kinds of powerful low-level operations
in Puppet, and have them work reliably (unlike the recipes using
perl, grep and/or sed that people use today).

I would be glad to help in both specifying and implementing them
(but note that I have only rudimentary knowledge about Ruby and
about Puppet internals).

/Thomas Bellman

[Nigel Kersten]

On Fri, May 16, 2008 at 3:00 AM, Thomas Bellman <bel...@nsc.liu.se> wrote:

> Even better would be to get resource types like these into core
> Puppet. I think editing of text files is important enough to
> warrant that.

We have a native FileLine type we've been working on, and as soon
we're happy we've ironed out any bugs, we're planning to release it,
which should be relatively soon.

usage is like:

fileLine { "networkmanager":
path => "/etc/default/NetworkManager",
text => "exit 0",
ensure => present,
}

fileLine { "open_file_limit_hard":
path => "/etc/security/limits.conf",
text => "* hard nofile $open_file_limit",
match => '^\*\s+hard\s+nofile',
ensure => present,
replace => true,
regexp => true,
}

I tend to think of virtual types via definitions as a good way to
scaffold a new type if it can be done with a cluster of execs, but the
longer term goal is to always move to native types in Ruby.

--
Nigel Kersten
Systems Administrator
MacOps

[Thomas Bellman]

Nigel Kersten wrote:

> I tend to think of virtual types via definitions as a good way to
> scaffold a new type if it can be done with a cluster of execs, but the
> longer term goal is to always move to native types in Ruby.

Yeah, it can be good for prototyping new types. With the text
manipulations I wanted to do, however, it quickly became much too
painful getting the quoting right. So I started learning Ruby and
wrote them as native types instead. That way I could also make
the types much more powerful than I would have ever contemplated
doing with exec.

> We have a native FileLine type we've been working on, and as soon
> we're happy we've ironed out any bugs, we're planning to release it,
> which should be relatively soon.

I'm interrested!

> fileLine { "networkmanager":
> path => "/etc/default/NetworkManager",
> text => "exit 0",
> ensure => present,
> }
>
> fileLine { "open_file_limit_hard":
> path => "/etc/security/limits.conf",
> text => "* hard nofile $open_file_limit",
> match => '^\*\s+hard\s+nofile',
> ensure => present,
> replace => true,
> regexp => true,
> }

I assume this means to replace an existing line that matches the
'match' parameter? Does it add the line if nothing matches? What
does the match parameter mean if you say 'replace=>false'?

Do you have ways of telling fileLine *where* to put the line if it
needs to add the line? Or of only affecting matching lines at some
positions? I have found it absolutely essential to do that in some
cases. Think of managing parameters in xinetd config files; those
*must* be within the { and } of the required service. Or managing
parameters in /etc/ssh/ssh_config, which is also structured.

The types I have written always use regexps for patterns, so if you
want a constant string you need to quote it with a regexp_quote()
function (which I have also written). Having a flag parameter that
decides how to interpret another parameter doesn't feel very clean
to me.

/Thomas Bellman

[Nigel Kersten]

On Fri, May 16, 2008 at 8:08 AM, Thomas Bellman <bel...@nsc.liu.se> wrote:

> The types I have written always use regexps for patterns, so if you
> want a constant string you need to quote it with a regexp_quote()
> function (which I have also written). Having a flag parameter that
> decides how to interpret another parameter doesn't feel very clean
> to me.

No, it's rather dirty, but we didn't want to have to specify
everything as a regexp either.

Even more ideal I think is to Do The Right Thing and model specific
configuration file types as resources if you're manipulating them
regularly. We're not there yet though.

[Thomas Bellman]

Nigel Kersten wrote:

> Even more ideal I think is to Do The Right Thing and model specific
> configuration file types as resources if you're manipulating them
> regularly. We're not there yet though.

Absolutely agreed. But there will *always* be Yet Another Config
File Format that someone invents, making these low-level operations
a necessity.

/Thomas Bellman

[ADNET Ghislain]

> Absolutely agreed. But there will *always* be Yet Another Config
> File Format that someone invents, making these low-level operations
> a necessity.
>
>
> /Thomas Bellman
>
>

is there a way to create parser in ruby that will enable us to build
native type easely ?

can tools like treetop be used ?

http://www.rubyinside.com/treetop-powerful-but-easy-ruby-parser-library-701.html

anyone used this ?

--
Cordialement,
Ghislain

[David Lutterkort]

On Fri, 2008-05-16 at 17:36 +0200, ADNET Ghislain wrote:
> is there a way to create parser in ruby that will enable us to build
> native type easely ?

Shameless plug for Augeas[1] (which has Ruby bindings) In a nutshell,
Augeas gives you a language to describe config file formats; the
language is targetted at transforming files in that format into a tree,
and transforming the (changed) tree back into a file.

I've been talking about adding a native 'augeas' type for puppet, but
that has obviously not happened yet. Hopefully RSN ;) The other avenue
of integrating Augeas with Puppet is to use the Augeas API inside a
native Puppet type. That should be considerably more straightforward.

David

[1] http://augeas.net/

[ADNET Ghislain]

David Lutterkort a écrit :

> On Fri, 2008-05-16 at 17:36 +0200, ADNET Ghislain wrote:
>
>> is there a way to create parser in ruby that will enable us to build
>> native type easely ?
>>
>
> Shameless plug for Augeas[1] (which has Ruby bindings) In a nutshell,
> Augeas gives you a language to describe config file formats; the
> language is targetted at transforming files in that format into a tree,
> and transforming the (changed) tree back into a file.
>
>

Anybody gone through the pain of making a lense for httpd.conf ? :)

Is there a repository of augeas config files for various daemon (i found
the one in the augeas tgz) ? Hum perhaps this is not the best place to
ask for this...

--
Cordialement,
Ghislain

[Jean-Baptiste Quenot]

Interesting. This raises the question of how easy it is to write
custom Puppet types? Did you follow the "trial and error" method to
write those file manipulation types? How did you learn the various
subtleties of the Puppet internals? Do you find it easy to write
Puppet extensions? I for one would be hoping for a more
object-oriented API, instead of the numerous Ruby symbols or inline
function calls, for example.

I notice that you're using "File.new", that's a bit cheating in the
sense that one cannot use puppet:// protocol. I'm not criticizing
your work at all, but I'm trying to understand what we could do to
improve the Puppet API. I tried hard to extend the file type for
achieving proper file concatenation with support for puppet://
protocol, but failed miserably and reverted to using Git hooks instead
of trying to do it with Puppet.

So your "Puppet Power User" feedback would be greatly appreciated.
--
Jean-Baptiste Quenot
http://jbq.caraldi.com/

[Thomas Bellman]

Jean-Baptiste Quenot wrote:

(Digging through the headers, this seems to be a reply to my
message where I posted my types. It would be helpful if you had
quoted some of my message to give some context to the readers.)

> Interesting. This raises the question of how easy it is to write
> custom Puppet types? Did you follow the "trial and error" method to
> write those file manipulation types? How did you learn the various
> subtleties of the Puppet internals?

I read the wiki pages about creating custom types, I looked at
the source for the built-in types in Puppet, and I did a bunch of
trial and error.

> Do you find it easy to write Puppet extensions?

Not overly difficult. Much of my time was spent trying to
understand Ruby, since I hadn't used it at all before that. I
do have fairly extensive experience with programming in several
other languages, though, so I am used to looking at the source
for (to me unknown) programs to debug or extend them, which
probably helps. I suspect, though, that my code is not very
"rubyesque".

> I for one would be hoping for a more
> object-oriented API, instead of the numerous Ruby symbols or inline
> function calls, for example.

I'm not quite sure what you mean here. Much of the code in my
types deals with the actual processing and editing of the files.
In those parts, it does not need any support infrastructure from
Puppet, just ordinary Ruby file and text operations.

> I notice that you're using "File.new", that's a bit cheating in the
> sense that one cannot use puppet:// protocol.

Indeed, my types can only edit existing files on the client.
That's their entire purpose. I don't see how puppet:// support
could be used for this, but if you have any suggestions for what
parameters on what types should support puppet://, and what the
meaning should be, please tell. (I doubt that I would implement
it myself, since I don't use puppetd or puppetmasterd, just plain
'puppet', but maybe someone else could.)

> I'm not criticizing your work at all,

Coward! ;-) Please *do* criticize my work! (Just remember that
constructive and specific criticism is better than destructive
and vague criticism.)

> but I'm trying to understand what we could do to
> improve the Puppet API. I tried hard to extend the file type for
> achieving proper file concatenation with support for puppet://
> protocol, but failed miserably and reverted to using Git hooks instead
> of trying to do it with Puppet.

> So your "Puppet Power User" feedback would be greatly appreciated.

The biggest problem with the internal Puppet API, imho, is that
it is poorly documented. For instance, what values should the
sync() method return? Actually, the very existence of the sync()
method is undocumented. I had to look at existing types, and at
the code that calls the types, to find some of the information I
needed to write my types. Or at least the information I *think*
I needed; I'm not sure I did things the "kosher" way. If someone
who knows how things are supposed to be can tell me how I should
have done some things, I can fix my code, and perhaps help in
improving the documentation.

(This is actually a general problem with Puppet: there are many
"howtos" and recipes, but little reference documentation.)

/Thomas Bellman

[James Turnbull]

Thomas Bellman wrote:
> The biggest problem with the internal Puppet API, imho, is that
> it is poorly documented. For instance, what values should the
> sync() method return? Actually, the very existence of the sync()
> method is undocumented. I had to look at existing types, and at
> the code that calls the types, to find some of the information I
> needed to write my types. Or at least the information I *think*
> I needed; I'm not sure I did things the "kosher" way. If someone
> who knows how things are supposed to be can tell me how I should
> have done some things, I can fix my code, and perhaps help in
> improving the documentation.
>

Jump right in - as far as I am concerned and I suspect Luke also anyone
who has written their own type/provider is more than adequately equipped
to add to the doco. How about documenting sync() as a start? Or indeed
anything else you fancy. I am happy to help edit and expand. Indeed -
consolidating and rationalising the existing type/plugin/provider
documentation alone would be a valuable contribution. Currently it's
spread across 5-6 pages in differing levels of detail and depth.

> (This is actually a general problem with Puppet: there are many
> "howtos" and recipes, but little reference documentation.)
>

Well excluding the whole book and all... Oh you mean API docs - *looks
at Luke* :)

Regards

James Turnbull

--
James Turnbull (ja...@lovedthanlost.net)

Author of:
* Pulling Strings with Puppet
(http://www.amazon.com/gp/product/1590599780/)
* Pro Nagios 2.0
(http://www.amazon.com/gp/product/1590596099/)
* Hardening Linux
(http://www.amazon.com/gp/product/1590594444/)

[Jean-Baptiste Quenot]

2008/5/20 Thomas Bellman <bel...@nsc.liu.se>:

>
> > but I'm trying to understand what we could do to
> > improve the Puppet API. I tried hard to extend the file type for
> > achieving proper file concatenation with support for puppet://
> > protocol, but failed miserably and reverted to using Git hooks instead
> > of trying to do it with Puppet.
>
> > So your "Puppet Power User" feedback would be greatly appreciated.
>
> The biggest problem with the internal Puppet API, imho, is that
> it is poorly documented. For instance, what values should the
> sync() method return? Actually, the very existence of the sync()
> method is undocumented. I had to look at existing types, and at
> the code that calls the types, to find some of the information I
> needed to write my types. Or at least the information I *think*

> I needed [...] (This is actually a general problem with Puppet: there are many

> "howtos" and recipes, but little reference documentation.)

OK, I don't feel alone anymore. Indeed the lack of API docs (and IMHO
the lack of a true OO-API) prevents to efficiently extend Puppet. And
the fact that there is no example for a custom type in puppet is a
real sign (I mean
/usr/share/doc/puppet/examples/code/modules/sample-module on my
system). So in many cases, people revert to using the Exec type of
course, with funny Shell, Perl or any other
scripting-language-specific constructs.

Thanks for your feedback,

[Luke Kanies]

I agree that the internal API for the RAL is both ugly and
undocumented. I'm already making plans on fixing it, but it probably
won't get much done on it until the REST work is complete.

I think I had docs for this stuff about 2 years ago, but they kind of
died since no one seemed to care then.

I'd be glad to provide specific help on any of the internal APIs,
especially to the extent that whomever I'm helping also documents
them. Just ask for that help on puppet-dev, as I'm more likely to
follow every thread there these days (since there are so few).

--
Never try to tell everything you know. It may take too short a time.
--Norman Ford
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

[David Schmitt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 20 May 2008, Jean-Baptiste Quenot wrote:
> OK, I don't feel alone anymore. Indeed the lack of API docs (and IMHO
> the lack of a true OO-API) prevents to efficiently extend Puppet. And
> the fact that there is no example for a custom type in puppet is a
> real sign (I mean
> /usr/share/doc/puppet/examples/code/modules/sample-module on my
> system).

Hmm, all types and providers are - in a sense - "example" types. Also there is

http://reductivelabs.com/trac/puppet/wiki/CreatingCustomTypes

http://reductivelabs.com/trac/puppet/wiki/PracticalTypes

or the types at http://git.black.co.at/?p=module-mysql

The wrestling I had to do until I had the latter module, forces me to
acknowledge that the documentation is sub-par relatively to the other docs,
but it's not like there is nothing at all.

> So in many cases, people revert to using the Exec type of
> course, with funny Shell, Perl or any other
> scripting-language-specific constructs.

which is fine enough for simple things, prototyping and getting to grips with
the mental models that are needed for doing a ruby type anyways.

Regards, DavidS

- --
The primary freedom of open source is not the freedom from cost, but the free-
dom to shape software to do what you want. This freedom is /never/ exercised
without cost, but is available /at all/ only by accepting the very different
costs associated with open source, costs not in money, but in time and effort.
- -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIMvMy/Pp1N6Uzh0URAm2ZAJ4op14lWXM66hlE+06srqu7CHgerQCfYOW2
sNm5pOQ2A6vjLBZ4Kig0S80=
=vTG+
-----END PGP SIGNATURE-----

[David Lutterkort]

On Tue, 2008-05-20 at 09:46 +0200, ADNET Ghislain wrote:
> Anybody gone through the pain of making a lense for httpd.conf ? :)

That seems to keep sitting firmly on the TODO list ;)

> Is there a repository of augeas config files for various daemon (i found
> the one in the augeas tgz) ?

They are all in the lenses/ dir in the tarball - besides httpd.conf,
what other daemons do you need ? I'd be happy to help you write them.

> Hum perhaps this is not the best place to ask for this...

Maybe the Augeas list[1] is better for this. (Or #augeas on freenode)

David

[1] https://www.redhat.com/mailman/listinfo/augeas-devel

[ADNET Ghislain]

>> Anybody gone through the pain of making a lense for httpd.conf ? :)
>>
>
> That seems to keep sitting firmly on the TODO list ;)
>
>

eheheh

>> Is there a repository of augeas config files for various daemon (i found
>> the one in the augeas tgz) ?
>>
>
> They are all in the lenses/ dir in the tarball - besides httpd.conf,
> what other daemons do you need ? I'd be happy to help you write them.
>
>

vsftpd
proftpd
dovecot
spamd
clamav
hosts.allow
comes to mind

but i first get to try augeas, perhaps i will find time this week end :)
next is to link it to a puppet type...that will be hard for me

--
Cordialement,
Ghislain

[Larry Ludwig]

Hi Thomas and all.. I've tried installing the libraries in the .tgz
file and seem to have an issue to get it working. I assume everything
is installed correctly. I do not see any puppet or puppetmaster
errors. In the .tgz file I see and install these files in these
folders (per a centos 4.6 setup):


puppet/parser/functions -> /usr/lib/ruby/site_ruby/1.8/puppet/parser/
functions/
puppet/type -> /etc/puppet/modules/custom/plugins/puppet/type/ (which
then gets synced up on each puppetd node to /var/lib/puppet/lib/
plugins/puppet/type/)
ruby/nsc_utils.rb -> /usr/lib/ruby/site_ruby/1.8/nsc_utils.rb (I
assume this is the correct location?)

I created a define to configure sshd_config file. I have the notify
there to see if it's actually running this define.


define sshd_config($value, $service="") {
notify {"hello":
message => "they there",
}
ensure_line {"sshd_config--${title}":
file => "/etc/ssh/sshd_config",
line => "${title} ${value}",
pattern => "^(#)?${title}.*$",
notify => $service ? { "" => [], default =>
Service[$service] };
}
}

I then call this in a class:

# make sure SSH 2 only
sshd_config {"Protocol":
value => "2",
service => "sshd",
}
}


What appears in the the log file:

Jun 10 10:24:46 empoweri00 puppetd[6225]: Starting Puppet client
version 0.24.4
Jun 10 10:24:55 empoweri00 puppetd[6225]: Starting catalog run
Jun 10 10:25:09 empoweri00 puppetd[6225]: they there
Jun 10 10:25:12 empoweri00 puppetd[6225]: Finished catalog run in
17.70 seconds

so it's running but not making any changes to the file? Is there
something I"m missing?

-L

--
Larry Ludwig
Empowering Media
1-866-792-0489 x600
Managed and Unmanaged Xen VPSes
http://www.hostcube.com/

[Evan Hisey]

>
> What appears in the the log file:
>
> Jun 10 10:24:46 empoweri00 puppetd[6225]: Starting Puppet client
> version 0.24.4
> Jun 10 10:24:55 empoweri00 puppetd[6225]: Starting catalog run
> Jun 10 10:25:09 empoweri00 puppetd[6225]: they there
> Jun 10 10:25:12 empoweri00 puppetd[6225]: Finished catalog run in
> 17.70 seconds
>
> so it's running but not making any changes to the file? Is there
> something I"m missing?
>

Have you made sure there is something in the file to change? I know,
stupid question, but it has caused me much head scratching before.
Puppet won't make any changes if the file is already correct.

Evan

[Thomas Bellman]

Larry Ludwig wrote:

> Hi Thomas and all.. I've tried installing the libraries in the .tgz
> file and seem to have an issue to get it working. I assume everything
> is installed correctly. I do not see any puppet or puppetmaster
> errors. In the .tgz file I see and install these files in these
> folders (per a centos 4.6 setup):

> puppet/parser/functions -> /usr/lib/ruby/site_ruby/1.8/puppet/parser/
> functions/
> puppet/type -> /etc/puppet/modules/custom/plugins/puppet/type/ (which
> then gets synced up on each puppetd node to /var/lib/puppet/lib/
> plugins/puppet/type/)

If you say so. I have never used the syncing things in Puppet.
In fact, I have never used puppetd or puppetmasterd at all; I
only use plain 'puppet', and distribute my manifests, functions,
and types using NFS.

> ruby/nsc_utils.rb -> /usr/lib/ruby/site_ruby/1.8/nsc_utils.rb (I
> assume this is the correct location?)

Yes, that ought to work, I think. Myself, I set the environment
variable RUBYLIB to point to the directory where nsc_utils.rb
live. That should work for puppetd and puppetmasterd also.

In my setup, I have a subdirectory named 'Lib' under where my
manifests are, and have the directories from the tarball in it.
I.e, I have the following structure:

.
Lib/
Lib/ruby/
Lib/puppet/
Lib/puppet/parser/
Lib/puppet/parser/functions/
Lib/puppet/type/
site.pp

I then run puppet with:

# RUBYLIB="`pwd`/Lib/ruby" puppet --libdir="`pwd`/Lib" site.pp

> I created a define to configure sshd_config file. I have the notify
> there to see if it's actually running this define.
>
>
> define sshd_config($value, $service="") {
> notify {"hello":
> message => "they there",
> }
> ensure_line {"sshd_config--${title}":
> file => "/etc/ssh/sshd_config",
> line => "${title} ${value}",
> pattern => "^(#)?${title}.*$",
> notify => $service ? { "" => [], default =>
> Service[$service] };
> }
> }
>
> I then call this in a class:
>
> # make sure SSH 2 only
> sshd_config {"Protocol":
> value => "2",
> service => "sshd",
> }
> }

This looks correct.

> What appears in the the log file:
>
> Jun 10 10:24:46 empoweri00 puppetd[6225]: Starting Puppet client
> version 0.24.4
> Jun 10 10:24:55 empoweri00 puppetd[6225]: Starting catalog run
> Jun 10 10:25:09 empoweri00 puppetd[6225]: they there
> Jun 10 10:25:12 empoweri00 puppetd[6225]: Finished catalog run in
> 17.70 seconds
>
> so it's running but not making any changes to the file? Is there
> something I"m missing?

Are you sure the file doesn't already contain the correct line?
Have you tried a smaller example, something like:

node default {
ensure_line {
"null-test":
file => "/dev/null",
line => "Protocol 2",
pattern => "^(#)?Protocol.*$",
}
}

This gives me the message

notice: //Node[default]/Ensure_line[null-test]/line: line changed '' to 'Protocol 2'

when I try it using 'puppet'. Then change /dev/null to some
existing but empty file and try it again.

/Thomas Bellman

[Larry Ludwig]

No it does not.

>
> node default {
> ensure_line {
> "null-test":
> file => "/dev/null",
> line => "Protocol 2",
> pattern => "^(#)?Protocol.*$",
> }
> }
>
> This gives me the message
>
> notice: //Node[default]/Ensure_line[null-test]/line: line changed '' to 'Protocol 2'
>

No same result with /dev/null

Jun 10 12:43:30 devcentos39 puppetd[12617]: Starting Puppet client
version 0.24.4
Jun 10 12:43:36 devcentos39 puppetd[12617]: (/File[/var/lib/puppet/lib/
plugins/puppet/type]/checksum) checksum changed '{mtime}Mon Jun 09
19:38:42 -0400 2008' to '{mtime}Tue Jun 10 11:40:53 -0400 2008'
Jun 10 12:43:47 devcentos39 puppetd[12617]: Starting catalog run
Jun 10 12:43:55 devcentos39 puppetd[12617]: they there
Jun 10 12:44:13 devcentos39 puppetd[12617]: Finished catalog run in
25.85 seconds

Actually I think nsc_utils.rb has to be synced also since it must run
with the .rb files on the node?

I added this also and it doesn't work. the libdir is ok? I think:

[root@devcentos39 log]# puppetd --genconfig | grep "libdir ="
libdir = /var/lib/puppet/lib

[Larry Ludwig]

Even odder. It found the typo when I put the ensure_line directly in
the node and follows your example from above.

Jun 10 12:57:01 devcentos39 puppetd[14302]: Starting Puppet client
version 0.24.4
Jun 10 12:57:09 devcentos39 puppetd[14302]: Could not retrieve
catalog: Invalid parameter 'patttern' for type 'Ensure_line' at /etc/
puppet/manifests/servers/devcentos39.pp:7 on node
devcentos39.pvt.empoweringmedia.net
Jun 10 12:57:11 devcentos39 puppetd[14302]: Starting catalog run
Jun 10 12:57:32 devcentos39 puppetd[14302]: Finished catalog run in
20.64 seconds
Jun 10 12:57:45 devcentos39 puppetd[14302]: Caught TERM; shutting down
Jun 10 12:57:45 devcentos39 puppetd[14302]: Shutting down
Jun 10 12:57:46 devcentos39 puppetd[14530]: Reopening log files
Jun 10 12:57:46 devcentos39 puppetd[14530]: Starting Puppet client
version 0.24.4
Jun 10 12:57:59 devcentos39 puppetd[14530]: Starting catalog run
Jun 10 12:58:23 devcentos39 puppetd[14530]: Finished catalog run in
23.77 seconds

then when I fixed the typo and ran it again it appears it doesn't even
run it, yet does not complain about it in anyway.

[Larry Ludwig]

It also appears in the localconfig.yaml file:

- !ruby/object:Puppet::TransObject
name: null-test
type: ensure_line
tags:

So it sees it but doesn't execute?? What gives?