Wood/trees/hiera discrimination problem
31 views
Skip to first unread message
John Hawkes-Reed
Sep 21, 2012, 7:04:09 AM9/21/12
to puppet...@googlegroups.com
Hello.
I think I'm being an idiot (not for the first time), but given the following hiera paragraph:
directorymap:
www.blah.com:
- source: dodgy-uploaded-jpegs
target: www.blah.com/wp-content/uploads
- source: disturbing-home-videos
target: www.blah.com/wp-content/filth
www.nice-site.co.uk:
- source: kitten-pics
- target: www.nice-site.co.uk/site-content/user-data
What's the least-worst way of extracting that, such that puppet pseudo-code along the lines of
'ln -s ${site}/${source} ${site}{$target}' would work?
There is likely a Better Way To Do It...
--
John Hawkes-Reed
Systems Administrator. Future Publishing. x 2526
--
Future Publishing Limited (registered company number 2008885) is a wholly owned subsidiary of Future plc (registered company number 3757874), both of which are incorporated in England and Wales and share the same registered address at Beauford Court, 30 Monmouth Street, Bath BA1 2BW.
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error please reply to this email and then delete it. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Future.
The recipient should check this email and any attachments for the presence of viruses. Future accepts no liability for any damage caused by any virus transmitted by this email.
Future may regularly and randomly monitor outgoing and incoming emails and other telecommunications on its email and telecommunications systems. By replying to this email you give your consent to such monitoring.
*****
Save resources: think before you print.
I think I'm being an idiot (not for the first time), but given the following hiera paragraph:
directorymap:
www.blah.com:
- source: dodgy-uploaded-jpegs
target: www.blah.com/wp-content/uploads
- source: disturbing-home-videos
target: www.blah.com/wp-content/filth
www.nice-site.co.uk:
- source: kitten-pics
- target: www.nice-site.co.uk/site-content/user-data
What's the least-worst way of extracting that, such that puppet pseudo-code along the lines of
'ln -s ${site}/${source} ${site}{$target}' would work?
There is likely a Better Way To Do It...
--
John Hawkes-Reed
Systems Administrator. Future Publishing. x 2526
--
Future Publishing Limited (registered company number 2008885) is a wholly owned subsidiary of Future plc (registered company number 3757874), both of which are incorporated in England and Wales and share the same registered address at Beauford Court, 30 Monmouth Street, Bath BA1 2BW.
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error please reply to this email and then delete it. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Future.
The recipient should check this email and any attachments for the presence of viruses. Future accepts no liability for any damage caused by any virus transmitted by this email.
Future may regularly and randomly monitor outgoing and incoming emails and other telecommunications on its email and telecommunications systems. By replying to this email you give your consent to such monitoring.
*****
Save resources: think before you print.
jcbollinger
Sep 21, 2012, 10:42:19 AM9/21/12
to puppet...@googlegroups.com
On Friday, September 21, 2012 6:04:37 AM UTC-5, John Hawkes-Reed wrote:
Hello.
I think I'm being an idiot (not for the first time), but given the following hiera paragraph:
directorymap:
www.blah.com:
- source: dodgy-uploaded-jpegs
target: www.blah.com/wp-content/uploads
- source: disturbing-home-videos
target: www.blah.com/wp-content/filth
www.nice-site.co.uk:
- source: kitten-pics
- target: www.nice-site.co.uk/site-content/user-data
What's the least-worst way of extracting that, such that puppet pseudo-code along the lines of
'ln -s ${site}/${source} ${site}{$target}' would work?
So, it looks like that would expand to commands such as
ln -s www.nice-site.co.uk/kitten-pics www.nice-site.co.ukwww.nice-site.co.uk/site-content/user-data
I suspect the duplication of the site on the target side is not what you want, but it follows from your data. Nevertheless, I guess your real question is about how to make Puppet digest your nested data structure.
(Note also that your YAML data structure under "nice-site" is inconsistent with its cousin under "blah"; it appears that the structure under "blah" is what you intended. Not so interested in kitten pics? :-) )
There is likely a Better Way To Do It...
In fact, although they appear to make sense from a data design point of view, the arrays at the second nested level of your data structure are going to cause you trouble in Puppet. Puppet does understand arrays, but not iteration, so you would probably need to resort to Ruby DSL to do anything useful with them. If you change them to hashes, however, then you may have an easier time. I.e.:
directorymap:
www.blah.com:
www.blah.com0:
source: dodgy-uploaded-jpegs
target: www.blah.com/wp-content/uploads
www.blah.com1:
www.nice-site.co.uk0:
source: kitten-pics
target: www.nice-site.co.uk/site-content/user-data
That duplicates data even more, however. You could improve it further (IMO) by flattening it a bit and pushing down the site info, such as:
directorymap:
artificial-id-0:
source: dodgy-uploaded-jpegs
site: www.blah.com
target: wp-content/uploads
artificial-id-1:
source: disturbing-home-videos
site: www.blah.com
target: wp-content/filth
artificial-id-2:
source: kitten-pics
site: www.nice-site.co.uk
target: site-content/user-data
Note that now the data structure is a hash whose keys are suitable identifiers for resources you want to manage, and whose values are hashes of the desired properties of those resources. That's exactly what you need for the create_resources() function.
To use create_resources(), you also need a resource type to accept those parameters, and since there is no built-in type that does what you want, your best bet would be a defined type:
define content::link($source, $site, target) {
file { "${content::base_directory}/${site}/${target}":
ensure => link,
target => "${site}/${source}"
}
}
Note that variable $content::base_directory must be defined elsewhere (in a separate class "content", in this case). This is essential because the base directory is not included in your data, yet is needed to correctly identify the link's full path.
With those pieces in place, you can tie it all together as simply as this:
create_resources('content::link', hiera('directorymap'))
If you don't like the revised form of your data that I suggested, or for future problems, there are at least a few other tools you should have in your toolbox:
1) A Puppet function to extract an array of keys from a hash. You can write your own if you like (it's easy), but there is one ready-made in Puppetlabs's useful 'stdlib' module (https://github.com/puppetlabs/puppetlabs-stdlib).
2) Defined types. Learn them, live them, love them. And especially, know that inside a defined type body, you can refer to instances' given titles via the automatic $name and/or $title variables.
3) Multiple resource declaration syntax. If the title of a resource declaration is an array -- either a literal one, or a variable with an array value -- then it means one resource for each element of the array, with the corresponding element as resource title. For instance,
file { [ '/tmp/foo', '/tmp/bar' ]: ensure => present }
is equivalent to
file { '/tmp/foo': ensure => present }
file { '/tmp/bar': ensure => present }
That can be combined in useful ways with (1) and (2). In particular, you can extract an array of hash keys, declare instances of a defined type with those titles, and in the body of the defined type use the title to extract and work with one element of the original hash. So, if you wanted to avoid the create_resources() function (but use the same data structure) then you could do something like this instead:
$keys = keys($content::directories)
content_link2 { $keys: }
define content::link2() {
$source = $content::directories[$name]['source']
$site = $content::directories[$name]['site']
$target = $content::directories[$name]['target']
file { "${content::base_directory}/${site}/${target}":
ensure => link,
target => "${site}/${source}"
}
}
John
source: kitten-pics
target: www.nice-site.co.uk/site-content/user-data
That duplicates data even more, however. You could improve it further (IMO) by flattening it a bit and pushing down the site info, such as:
directorymap:
artificial-id-0:
source: dodgy-uploaded-jpegs
site: www.blah.com
target: wp-content/uploads
artificial-id-1:
source: disturbing-home-videos
site: www.blah.com
target: wp-content/filth
artificial-id-2:
source: kitten-pics
site: www.nice-site.co.uk
target: site-content/user-data
Note that now the data structure is a hash whose keys are suitable identifiers for resources you want to manage, and whose values are hashes of the desired properties of those resources. That's exactly what you need for the create_resources() function.
To use create_resources(), you also need a resource type to accept those parameters, and since there is no built-in type that does what you want, your best bet would be a defined type:
define content::link($source, $site, target) {
file { "${content::base_directory}/${site}/${target}":
ensure => link,
target => "${site}/${source}"
}
}
Note that variable $content::base_directory must be defined elsewhere (in a separate class "content", in this case). This is essential because the base directory is not included in your data, yet is needed to correctly identify the link's full path.
With those pieces in place, you can tie it all together as simply as this:
create_resources('content::link', hiera('directorymap'))
If you don't like the revised form of your data that I suggested, or for future problems, there are at least a few other tools you should have in your toolbox:
1) A Puppet function to extract an array of keys from a hash. You can write your own if you like (it's easy), but there is one ready-made in Puppetlabs's useful 'stdlib' module (https://github.com/puppetlabs/puppetlabs-stdlib).
2) Defined types. Learn them, live them, love them. And especially, know that inside a defined type body, you can refer to instances' given titles via the automatic $name and/or $title variables.
3) Multiple resource declaration syntax. If the title of a resource declaration is an array -- either a literal one, or a variable with an array value -- then it means one resource for each element of the array, with the corresponding element as resource title. For instance,
file { [ '/tmp/foo', '/tmp/bar' ]: ensure => present }
is equivalent to
file { '/tmp/foo': ensure => present }
file { '/tmp/bar': ensure => present }
That can be combined in useful ways with (1) and (2). In particular, you can extract an array of hash keys, declare instances of a defined type with those titles, and in the body of the defined type use the title to extract and work with one element of the original hash. So, if you wanted to avoid the create_resources() function (but use the same data structure) then you could do something like this instead:
$keys = keys($content::directories)
content_link2 { $keys: }
define content::link2() {
$source = $content::directories[$name]['source']
$site = $content::directories[$name]['site']
$target = $content::directories[$name]['target']
file { "${content::base_directory}/${site}/${target}":
ensure => link,
target => "${site}/${source}"
}
}
John
John Hawkes-Reed
Sep 21, 2012, 12:25:21 PM9/21/12
to puppet...@googlegroups.com
On 21 Sep 2012, at 15:42, jcbollinger wrote:
>
>
> On Friday, September 21, 2012 6:04:37 AM UTC-5, John Hawkes-Reed wrote:
> Hello.
>
> I think I'm being an idiot (not for the first time), but given the following hiera paragraph:
>
> directorymap:
> www.blah.com:
> - source: dodgy-uploaded-jpegs
> target: www.blah.com/wp-content/uploads
> - source: disturbing-home-videos
> target: www.blah.com/wp-content/filth
> www.nice-site.co.uk:
> - source: kitten-pics
> - target: www.nice-site.co.uk/site-content/user-data
>
> What's the least-worst way of extracting that, such that puppet pseudo-code along the lines of
>
> 'ln -s ${site}/${source} ${site}{$target}' would work?
>
>
> So, it looks like that would expand to commands such as
>
> ln -s www.nice-site.co.uk/kitten-pics www.nice-site.co.ukwww.nice-site.co.uk/site-content/user-data
>
> I suspect the duplication of the site on the target side is not what you want, but it follows from your data. Nevertheless, I guess your real question is about how to make Puppet digest your nested data structure.
>
> (Note also that your YAML data structure under "nice-site" is inconsistent with its cousin under "blah"; it appears that the structure under "blah" is what you intended. Not so interested in kitten pics? :-) )
As a team, we're in the middle of re-factoring a lot of old (in Puppet terms) code, which means Barberising the modules, lots of Jenkins runs with puppet-lint... And now discovering that we can throw away circa half the cruft if we use hiera/create_resources right and spend some time looking carefully at the design of the yaml in the hiera tree.
Reply all
Reply to author
Forward
0 new messages