Working on "line" module

[Art Zemon]

Folks,

I am writing my "line" module, which will extend the semantics of managing one file within a filesystem to managing one line within one file within a filesystem. I have a few goals in mind here:

1. Simple and declarative. This is not about performing arbitrary edits on a file. This is about declaring that a line must be present (or absent).
2. Idempotent. The file will not be edited unless a change is required.
3. Intuitive. Humans should be able to use it. You don't need to be a sed or perl or regex guru to get the job done.

Would you please read the following and then answer these questions: Is the name "line" intuitive or confusing and, if confusing, what would you suggest instead? Is the parameter name "regexp" intuitive or confusing and, if confusing, what would you suggest instead?

# line module - assures that a line exists in a file
#
# Much as the file module assures that a file exists in the filesystem, this module
# extends the same semantics to one line within one file. You specify the filename,
# the regular expression which "identifies" the line (much as the filename identifies
# the file), and the value that the line must have (much as you specify the contents
# of the file). Just as the file module allows an administrator to control individual
# files in the filesystem while leaving the remaining files uncontrolled, the line
# modules allows an administrator to control individual lines in a file while leaving
# the remaining lines uncontrolled.
#
# Parameters:
#  dest: Absolute path of the file on the filesystem
#  regexp: Regular expression which matches the line in the file (if the line is present).
#          Note - You should write a regular expression which matches exactly 0 or 1 line.
#  value: The exact value of the line which should exist in the file (not a regular expression)
#  state: Either "exists" or "absent"
#
# Examples:
#  dest=/etc/hosts regexp="^127.0.0.1 " value="127.0.0.1 localhost localhost.localdomain" state=present
#  Assures that /etc/hosts has a correct line for localhost. It would, for instance, fix a line which
#  read only "127.0.0.1 localhost", adding "localhost.localdomain" to the end of the line.
#
#  dest=/etc/apache2/httpd.conf regexp="^SSLCipherSuite " value="SSLCipherSuite ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM" state=present
#  Assures that a defined set of ciphers is enabled in Apache.

Thanks,
    -- Art Z.

-- 
http://CheerfulCurmudgeon.com/

"Be kind, for everyone you meet is fighting a hard battle."  Plato

[Cosmin Luță]

Hi,

I've also written a prototype for something a similar module and I'm glad to see you're also working on one, as I don't have enough time to maintain mine.

On Monday, May 14, 2012 5:52:32 PM UTC+3, Art Zemon wrote:

Folks,

I am writing my "line" module, which will extend the semantics of managing one file within a filesystem to managing one line within one file within a filesystem. I have a few goals in mind here:

1. Simple and declarative. This is not about performing arbitrary edits on a file. This is about declaring that a line must be present (or absent).
2. Idempotent. The file will not be edited unless a change is required.
3. Intuitive. Humans should be able to use it. You don't need to be a sed or perl or regex guru to get the job done.

Would you please read the following and then answer these questions: Is the name "line" intuitive or confusing and, if confusing, what would you suggest instead? Is the parameter name "regexp" intuitive or confusing and, if confusing, what would you suggest instead?

"line" - mine is called "fileline".. :)  think it's more meaningful if you call it "file-something" to suggest it's operating on files

"regexp" - I like this one


In my fileline ( https://github.com/lcosmin/ansible/blob/devel/library/fileline - please excuse the ugly code, i needed something for this job and fast ) I can specify multiple "present" or "absent" lines by repeating the parameter on the command line; maybe you could allow the same since it'd be more efficient if the user needs to work on multiple lines (calls the module only once instead of multiple times).



 

[Dag Wieers]

On Mon, 14 May 2012, Art Zemon wrote:

> Would you please read the following and then answer these questions: Is
> the name "line" intuitive or confusing and, if confusing, what would you
> suggest instead?

Could it be useful to have a separate namespace for modules that only
check for items/compliancy ? So that the name makes this more obvious ?

I also envision a module that would do the same for XML, YAML or INI
files. And personally would need the ability to manipulate these files as
well (even though it is not a best-practice, by some standards).

--
-- dag wieers, d...@wieers.com, http://dag.wieers.com/
-- dagit linux solutions, in...@dagit.net, http://dagit.net/

[Any errors in spelling, tact or fact are transmission errors]

[Art Zemon]

Cosmin: Thank you for the pointer to your github repo. I will certainly
take a look at your code. I like your fileline name, too.

Dag: Your mention of INI files gave me an idea for another parameter:
insert_after. This would allow you to specify in which section of the
INI file the line must be placed.

Managing XML and YAML files is certainly in the same vein as what I am
doing with single files. I want to keep the syntax (and code) simple,
though. Perhaps those need to be distinct modules?

[Brad Olson]

Art,

I've wanted something like this at times, but backed off after reading
Michael's comments about Augeas (a massage every type of config file
you can think of utility), that it's not the way to manage systems.

He'll be the one to decide if it's core, of course, but I would avoid
trying to make something to hack every file type there is. My personal
opinion.

Usually you'll want to template instead.

Similar capability in fabric (http://docs.fabfile.org/en/1.4.2/api/
contrib/files.html) has made my life easier sometimes.

If I were to write it, I think version one would allow:

- dest|file: filename
- regexp|match: a RE. If it matches the line, that's the line we want
to deal with
- state:
absent, if <match> matches a line, delete it
present, if <match> matches a line and the line doesn't equal
<line>, make it equal <line>
commented, if <match> matches a line, make sure it's commented
out
uncommented, if <match> matches a line, make sure it's not
commented out
- comment: # or default comment character
- line: what the line should be exactly (only for state=present.

Version two would add:

- context: RE, the first match found is the context within which the
above logic is applied. Easy to do with Python's re.sub( find,
replace=function()...

[John Kleint]

While I agree that editing configs this way is a slippery slope, I can see that some people need it.  I'm sure you've already thought of this, but if you want idempotence, you have to make sure the replacement value matches the regexp, and fail and complain when it doesn't.

-John

[Brad Olson]

Yes, very good check to have.

[Art Zemon]

On 05/15/2012 09:02 AM, Brad Olson wrote:
> I've wanted something like this at times, but backed off after reading
> Michael's comments about Augeas (a massage every type of config file
> you can think of utility), that it's not the way to manage systems.
>
> He'll be the one to decide if it's core, of course, but I would avoid
> trying to make something to hack every file type there is. My personal
> opinion.

Brad,

Michael and I disagree on whether this module is "good" or not. I have
no expectation that it will become part of the ansible core. I am simply
availing myself of the added brainpower of a whole group of smart people
via this discussion. And I will make my code freely available.

BTW, I think Michael and I disagree about the appropriateness of this
module because we have different points of view. Michael sees this as
inappropriately editing the contents of files. (I hope that I did not
misrepresent your opinion, Michael.) I see it as extending configuration
management down from files within the filesystem to lines within files.
This probably means that Michael and I have different problems to solve.

[Michael DeHaan]

BTW, I think Michael and I disagree about the appropriateness of this

module because we have different points of view. Michael sees this as

inappropriately editing the contents of files. (I hope that I did not

misrepresent your opinion, Michael.) I see it as extending configuration

management down from files within the filesystem to lines within files.

This probably means that Michael and I have different problems to solve.

I see any view in which the central server does not declare exactly what the server contains and is imprecise as being subject to failure,

and it limits the ability to remediate failure or push out changes from the central server. 

Plus, it's a huge pain to do it that way when you can instead push out templates.

[Brad Olson]

Michael & Art,

> I see any view in which the central server does not declare exactly what the server contains and is imprecise as being subject to failure,
> and it limits the ability to remediate failure or push out changes from the central server.
>
> Plus, it's a huge pain to do it that way when you can instead push out templates.

I appreciate Michael's stern instinct to keep things simple and
robust. It's the number one reason I think Ansible has a good chance
at succeeding in this config management niche where other good starts
have lost steam. I tend to agree that editing lines is Not The Way
when it comes to managing multiple systems well.

I can also appreciate Art's desire, though, especially when I have to
kick off idiosyncratic development and test servers, especially on
distros that roll out changes more aggressively (e.g. I don't want to
second guess new distro policy on sshd.conf, I just want to make sure
key pairs are required). This is also the case when I want to use
ansible instead of ansible-playbook.

The first thing I like about ansible is it recognizes the need for
(excruciatingly) simple, transparent, consistent, disciplined
configuration ('ansible-playbook'). But the second thing I love about
it is a recognition that sysadmins have to do staged, single-shot
alterations from time to time ('ansible'), and it does them both
without having to get as big as puppet+mcollective or puppet+fabric
(not bad-mouthing those--I rely on them often, but they're fat).

Michael, I'm wondering if you would be open to someone else
maintaining a repo for useful modules that should never make core, but
are worth collecting? It would be a bit like ansible-contrib was, but
1) you wouldn't have to hassle maintaining it, 2) it would have looser
guidelines (though still expect some basic documentation and the
ansible license statement) for inclusion. I'm willing to start that. I
wouldn't want it to detract from core ansible development.

--Brad

[Michael DeHaan]

> Michael, I'm wondering if you would be open to someone else
> maintaining a repo for useful modules that should never make core, but
> are worth collecting? It would be a bit like ansible-contrib was, but
> 1) you wouldn't have to hassle maintaining it, 2) it would have looser
> guidelines (though still expect some basic documentation and the
> ansible license statement) for inclusion. I'm willing to start that. I
> wouldn't want it to detract from core ansible development.
>
> --Brad

+100.

Some one else maintaining contrib would be great for me. We can
still keep it in the ansible repo and I can grant permissions. Start
something and I'll fork it (so it shows up on the main ansible repo
list) and give whoever needs it access.

I kind of care that playbooks and modules included look presentable,
as they are the first thing people will see when deciding whether to
use the tool... but then, I don't have the bandwidth for that. Also
other CM tools don't really review them either and it seems to be
working out ok for them.

Great idea.

--Michael

[Brad Olson]

Michael,

>
> I kind of care that playbooks and modules included look presentable,
> as they are the first thing people will see when deciding whether to
> use the tool... but then, I don't have the bandwidth for that.   Also
> other CM tools don't really review them either and it seems to be
> working out ok for them.
>
> Great idea.
>
> --Michael

The next few days are very full for me, but I'll try sketch out some
guidelines, structure, and workflow ideas to shoot by you hopefully by
Monday. --Brad