Formats and inline files

[Nevermind...]

Hi!

Blueprint is a really great tool, it's a shame it's not more popular.

Storing configuration in a simple format like json and tracking changes with git is a nice idea - after you have reverse engineered a server, packages and services inside blueprint file can easily be manually edited (if needed) and changes can be easily tracked with git diff; but there is an issue with inline files, e.g.:

"/etc/ssh/sshd_config": {

"content": "# Package generated configuration file\n# See the sshd_config(5) manpage for details\n\n# What ports, IPs and protocols we listen for\nPort 55553\n# Use these options to restrict which interfaces/protocols sshd will bind to\n#ListenAddress ::\n#ListenAddress 0.0.0.0\nProtocol 2\n# HostKeys for protocol version 2\nHostKey /etc/ssh/ssh_host_rsa_key\nHostKey /etc/ssh/ssh_host_dsa_key\n#Privilege Separation is turned on for security\nUsePrivilegeSeparation yes\n\n# Lifetime and size of ephemeral version 1 server key\nKeyRegenerationInterval 3600\nServerKeyBits 768\n\n# Logging\nSyslogFacility AUTH\nLogLevel INFO\n\n# Authentication:\nLoginGraceTime 120\nPermitRootLogin yes\nStrictModes yes\n\nRSAAuthentication yes\nPubkeyAuthentication yes\n#AuthorizedKeysFile\t%h/.ssh/authorized_keys\n\n# Don't read the user's ~/.rhosts and ~/.shosts files\nIgnoreRhosts yes\n# For this to work you will also need host keys in /etc/ssh_known_hosts\nRhostsRSAAuthentication no\n# similar for protocol version 2\nHostbasedAuthentication no\n# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication\n#IgnoreUserKnownHosts yes\n\n# To enable empty passwords, change to yes (NOT RECOMMENDED)\nPermitEmptyPasswords no\n\n# Change to yes to enable challenge-response passwords (beware issues with\n# some PAM modules and threads)\nChallengeResponseAuthentication no\n\n# Change to no to disable tunnelled clear text passwords\n#PasswordAuthentication yes\n\n# Kerberos options\n#KerberosAuthentication no\n#KerberosGetAFSToken no\n#KerberosOrLocalPasswd yes\n#KerberosTicketCleanup yes\n\n# GSSAPI options\n#GSSAPIAuthentication no\n#GSSAPICleanupCredentials yes\n\nX11Forwarding yes\nX11DisplayOffset 10\nPrintMotd no\nPrintLastLog yes\nTCPKeepAlive yes\n#UseLogin no\n\n#MaxStartups 10:30:60\n#Banner /etc/issue.net\n\n# Allow client to pass locale environment variables\nAcceptEnv LANG LC_*\n\nSubsystem sftp /usr/lib/openssh/sftp-server\n\n# Set this to 'yes' to enable PAM authentication, account processing,\n# and session processing. If this is enabled, PAM authentication will\n# be allowed through the ChallengeResponseAuthentication and\n# PasswordAuthentication.  Depending on your PAM configuration,\n# PAM authentication via ChallengeResponseAuthentication may bypass\n# the setting of \"PermitRootLogin without-password\".\n# If you just want the PAM account and session checks to run without\n# PAM authentication, then enable this but set PasswordAuthentication\n# and ChallengeResponseAuthentication to 'no'.\nUsePAM yes\n", 

"encoding": "plain", 

"group": "root", 

"mode": "100644", 

"owner": "root"

},

It's very hard to, for example, manually replace value in single line in content of this file or see changes with "git diff".

Problem is that json doesn't support heredoc-like syntax (or even un-enscaped new lines inside string), which is fine for machines, but not so much for humans (at least for this type of content).

Here are two ideas:

- externally stored files

This will solve above mentioned problem, and problems with different encodings, but you lose compact form.

- YAML

YAML supports block scalars, which can solve this problem, and whole things is IMHO easier on eyes. Consider:

{

  "files": {

    "/etc/apache2/sites-enabled/000-default": {

      "content": "../sites-available/default", 

      "encoding": "plain", 

      "group": "root", 

      "mode": "120777", 

      "owner": "root"

    }, 

    "/etc/apt/sources.list": {

      "content": "deb http://ftp.debian.org/debian squeeze main contrib non-free\ndeb http://security.debian.org squeeze/updates main contrib non-free\ndeb http://packages.devstructure.com/ squeeze main\n\n", 

      "encoding": "plain", 

      "group": "root", 

      "mode": "100644", 

      "owner": "root"

    }

  }

}

vs:

---

files:

  /etc/apache2/sites-enabled/000-default:

    content: ../sites-available/default

    encoding: plain

    group: root

    mode: 120777

    owner: root

  /etc/apt/sources.list:

    content: |+

      deb http://ftp.debian.org/debian squeeze main contrib non-free

      deb http://security.debian.org squeeze/updates main contrib non-free

      deb http://packages.devstructure.com/ squeeze main

    encoding: plain

    group: root

    mode: 100644

    owner: root

This two ideas can be combined together, so you can, for example, externally store files larger thax x bytes and non-utf8 files, and everything else inline in YAML.

That would be trivial to edit/create, and after you apply changes and do reverse engineering, you'll get the same result.

What do you think?

[Richard Crowley]

On Fri, Feb 1, 2013 at 12:19 PM, Nevermind... <neverm...@gmail.com> wrote:
> Hi!
> Blueprint is a really great tool, it's a shame it's not more popular.

Thanks!

I think the single-file form is a huge win for Blueprint (among other
things, it enabled the use of the blueprint(5) file format in AWS
CloudFormation). Which brings us to your YAML suggestion.

My usual kneejerk reaction to someone suggesting YAML is to point out
that it's impossible to tell if it's been truncated. That's a low
risk in this case, however.

The next concern is in adding a dependency that's not a part of the
Python 2.6 standard library. This can be addressed: I'm not opposed
to distributing a YAML implementation with Blueprint.

>
> What do you think?

One further proposal: does the workflow that's frustrating you get
better with a blueprint-diff-file(1) tool that works like git-diff(1)
to display the differences in a file's contents in two different
blueprints?

Thanks for the input,

Richard

[Nevermind...]

On Saturday, 2 February 2013 02:47:14 UTC+1, Richard Crowley wrote:

I think the single-file form is a huge win for Blueprint (among other 
things, it enabled the use of the blueprint(5) file format in AWS
CloudFormation).  Which brings us to your YAML suggestion.

 I agree, it has many advantages.

 

My usual kneejerk reaction to someone suggesting YAML is to point out
that it's impossible to tell if it's been truncated.  That's a low
risk in this case, however.

The next concern is in adding a dependency that's not a part of the
Python 2.6 standard library.  This can be addressed: I'm not opposed
to distributing a YAML implementation with Blueprint.

Yes, I agree, truncation can be a problem, but here is a very low risk.

I don't think that PyYAML is a big dependency problem, it's available in most of distribution repos.

btw. Ansible, a lightweight python configuration management tool, which is becoming popular, it also uses YAML. It would be nice to have an export ability from blueprint (which may also use this library). 

One further proposal: does the workflow that's frustrating you get
better with a blueprint-diff-file(1) tool that works like git-diff(1)
to display the differences in a file's contents in two different
blueprints?

Sure, you can write "wrapper" scripts around git-diff, around editors (decode json file contents to a temporary location, open in editor, encode result, remove tmp file) and everything else you might need, but that complicates things, you depend on this scripts and you lose ability to use just standard tools.

I'm not sure what exactly project goals are, but I will try to explain my point of view.

Blueprint works great as it is, if you use it as reverse engineering tool for a server.

But, if you take the opposite approach - now I have a blueprint "source file", which describes target server state.

I can use git to version this "source file" (which blueprint already does), which I can then checkout on my development machine (maybe even integrate as a subtree/submodule in my project - which will need this server configuration).

Then, for example, just want to set PHP memory_limit variable to 1GB in php.ini in blueprint source inside my repo, push that to couple of servers and they will automatically restart deamons. I don't want to connect to a server, do change there, do reverse engineering and pull changes in my repo (though, that process may work better when you need to explore different options, edit multiple files, etc.).

Blueprint already can do all that, just it's hard to edit/track its "source files". 

I could use exported chef/puppet files instead of blueprint json files, track/edit them in a separate repo, but this creates another (unnecessary) layer and this tools are bloated, slow and have huge dependencies.

Ansible exporter could solve some of this problems, but blueprint still has a big advantage - it can be converted to anything else when needed, including shell script.

I think that can have huge potential in a configuration management. In a simple scenario - pacakges list, files(+templates) and services definitions are all you need. For a more complex scenario, that can be extended with some kind of external callbacks to scripts which can be written in any language. 

[Richard Crowley]

> --
> You received this message because you are subscribed to the Google Groups
> "blueprint-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to blueprint-use...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

I believe I misinterpreted your goals initially. Apologies.

Let me build on your example of setting memory_limit to 1GB in
php.ini, I see two paths that do not end in pain. The first is the
conf.d pattern which would allow your blueprint to leave
/etc/php5/php.ini alone and set memory_limit in
/etc/php5/conf.d/memory_limit.ini. The second is to manage the entire
contents of /etc/php5/php.ini.

blueprint-apply(1) would not be reliable or even deterministic were
patching a line into (or out of) a file part of its algorithm.
Cfengine, Puppet, and Chef have all arrived at this same conclusion
semi-independently.

I hope one of these two alternatives suits your purposes well. If
not, please write back and I'll do what I can to help.

Thanks,

Richard

[Nevermind...]

On Saturday, February 2, 2013 7:23:06 PM UTC+1, Richard Crowley wrote:

I believe I misinterpreted your goals initially.  Apologies.

Let me build on your example of setting memory_limit to 1GB in
php.ini, I see two paths that do not end in pain.  The first is the
conf.d pattern which would allow your blueprint to leave
/etc/php5/php.ini alone and set memory_limit in
/etc/php5/conf.d/memory_limit.ini.  The second is to manage the entire
contents of /etc/php5/php.ini.

blueprint-apply(1) would not be reliable or even deterministic were
patching a line into (or out of) a file part of its algorithm.
Cfengine, Puppet, and Chef have all arrived at this same conclusion
semi-independently.

I hope one of these two alternatives suits your purposes well.  If
not, please write back and I'll do what I can to help.

 

Well, I'm not sure if I've chosen the best example, let me try to clarify.

Let's say I have a PHP project in a git repo, and decided to put blueprint json file inside (e.g subtree read from blueprint repo from original server), so every developer can easily create own server in a virtual machine for testing.

Now, after some time of development, one developer need to change some server configuration - php.ini's memory_limit in this case. 

Entire server configuration is already inside project repo, so he only needs to open blueprint.json, scroll down to php.ini file and change memory_limit line (except we have a json encoded file content, so it doesn't look pretty :)).

(Yes, if target server is debian-based, he could also add a new file to /etc/php5/conf.d/ which will override original php.ini settings, but that's not the issue here.)

All developers can now run a whole new, updated blueprint inside their virtual machines. Same blueprint will be also applied to original server and everything is up-to-date. All changes are idempotent.

I know that patching can be very tricky, and that's what I'm avoiding here.