create an array of hashes with reduce

[Arnau]

Hi all,

I'm using puppet 5.3. 

I'd like to build proxy_pass array of hashes using the reduce puppet built-in function and picking the values from a nested hash in hiera.

I first played a little bit with the reduce function:

$dirs = ['static','media','photos']

$proxy = $dirs.reduce([ { 'path' => '/', 'url' => "http://localhost:${port}/" } ]) |$memo, $value| { $memo + [ { 'path' => "/$value/", 'url' => '!' } ] }

notify { "MemoOut: $proxy" : ;}

If I puppet apply the above code the output looks like:

Notice: MemoOut: [{path => /, url => http://localhost:189/}, {path => /static/, url => !}, {path => /media/, url => !}, {path => /photos/, url => !}]

Notice: /Stage[main]/Main/Node[default]/Notify[MemoOut: [{path => /, url => http://localhost:189/}, {path => /static/, url => !}, {path => /media/, url => !}, {path => /photos/, url => !}]]/message: defined 'message' as 'MemoOut: [{path => /, url => http://localhost:189/}, {path => /static/, url => !}, {path => /media/, url => !}, {path => /photos/, url => !}]'

To my eyes this is an array of hashes, and that's what I need to pass to the apache vhost define.

So, now I want to pick the array (list of directories) from a nested hash in hiera that look like:

  application:

    apache:

      enable: true

      servername: application

      static_served:

        - static

I have a define that picks the above hash.... The array with the list of directories becomes "${apache['static_served']}" inside teh define. 

I can print it using notice ${apache['static_served']}":

Notice: static_served: application [static]

This looks like an array but is_array says that this is not an array anymore. 

And this is my first question, why it is not an array anymore?

I make this an array using any2array (array un puppet 5) so I can still call the reduce function::

$var=(Array("${apache['static_served']}"))

so the reduce line now looks like:

$_proxy_pass =  $var.reduce([ { 'path' => '/', 'url' => "http://localhost:${port}/" } ]) |$memo, $value| { $memo + [ { 'path' => "/$value/", 'url' => '!' } ] }

And here is where everythiong starts to make even less sense:

If I print the output (using notice $_proxy_pass and )  and I get a weird output:

Notice: /Stage[main]/.../Python::Virtualenv[application]/Notify[PROXY_PASS: [{path => /, url => http://local

host:11080/}, {path => /[/, url => !}, {path => /s/, url => !}, {path => /t/, url => !}, {path => /a/, url => !}, {path => /t/, url => !}, {path => /i/, url => !},

 {path => /c/, url => !}, {path => /]/, url => !}]]/message: defined 'message' as 'PROXY_PASS: [{path => /, url => http://localhost:11080/}, {path => /[/, url => !

}, {path => /s/, url => !}, {path => /t/, url => !}, {path => /a/, url => !}, {path => /t/, url => !}, {path => /i/, url => !}, {path => /c/, url => !}, {path => /

]/, url => !}]'

reduce is taking all letters from "static" instead of 'static" as the first (an uniqe) element from the array.... 

Later, in my code, I configure the proxy_pass in the apache:vhost like:

apache::vhost { "${apache['servername']}":

[...]

        proxy_pass     => "$_proxy_pass",

[...]

But, in the apache file, the proxy pass parameters  looks like

 ProxyPass path url

 ProxyPassReverse path url

 ProxyPass path url

 ProxyPassReverse path url

[...]

So it's picking the keys from the hashes in the above array of hashes and not the values.

I don't understand what is going on. Any help will be appreciated.

Best,

Arnau

[Henrik Lindberg]

On 2018-11-06 17:13, Arnau wrote:
> Hi all,
>
> I'm using puppet 5.3.
>

> I'd like to build proxy_pass array of hashes using the /reduce
> /puppetbuilt-in function
> <https://puppet.com/docs/puppet/5.5/function.html#reduce>and picking the

> values from a nested hash in hiera.
>
> I first played a little bit with the reduce function:
>
> $dirs = ['static','media','photos']
> $proxy = $dirs.reduce([ { 'path' => '/', 'url' =>
> "http://localhost:${port}/" } ]) |$memo, $value| { $memo + [ { 'path' =>
> "/$value/", 'url' => '!' } ] }
> notify { "MemoOut: $proxy" : ;}
>
> If I puppet apply the above code the output looks like:
>

Avoid having notify with a title containing [ ] - it can get very
confusing. Use a title like "testing", and set the message instead.

You give the reduce an array with a hash in it, and you then append to
that array with an array containing a hash. The result will be an
array where each element is a hash.

> Notice: MemoOut: [{path => /, url => http://localhost:189/}, {path =>
> /static/, url => !}, {path => /media/, url => !}, {path => /photos/, url
> => !}]
> Notice: /Stage[main]/Main/Node[default]/Notify[MemoOut: [{path => /, url
> => http://localhost:189/}, {path => /static/, url => !}, {path =>
> /media/, url => !}, {path => /photos/, url => !}]]/message: defined
> 'message' as 'MemoOut: [{path => /, url => http://localhost:189/}, {path
> => /static/, url => !}, {path => /media/, url => !}, {path => /photos/,
> url => !}]'
>

> /To my eyes this is an array of hashes, and that's what I need to pass
> to the apache vhost define./

>
> So, now I want to pick the array (list of directories) from a nested
> hash in hiera that look like:
>
>   application:
>     apache:
> enable: true
>       servername: application
>       static_served:
>         - static
>
> I have a define that picks the above hash.... The array with the list of
> directories becomes "${apache['static_served']}" inside teh define.
> I can print it using notice ${apache['static_served']}":
>
> Notice: static_served: application [static]
>

> This looks like an array but /*is_array */says that this is *not *an

> array anymore.
> And this is my first question, why it is not an array anymore?
>
>
> I make this an array using any2array (array un puppet 5) so I can still
> call the reduce function::
>
> $var=(Array("${apache['static_served']}"))
>

> so the /reduce /line now looks like:
>
You are asking Array to create an array out of a string. It will
construct that as an array where each element is a single character string.

You probably wanted

$var = Array($apache['static_served'])

Which by the way is a no-op since $apache['static_served'] already is an
array (as per the hiera data you supplied).

> $_proxy_pass = $var.reduce([ { 'path' => '/', 'url' =>
> "http://localhost:${port}/" } ]) |$memo, $value| { $memo + [ { 'path' =>
> "/$value/", 'url' => '!' } ] }
>
> And here is where everythiong starts to make even less sense:
>

You are now performing a reduce based on a sequence of single character
strings. This is indeed confusing as you have already gone off the road.

Hope this helps you get back on track.

- henrik

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAM69jx-wKXmP%3DhfPLsMsrYDeOS7TCTQaQ9XyhUaoWJ9uq%3DeZEg%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-users/CAM69jx-wKXmP%3DhfPLsMsrYDeOS7TCTQaQ9XyhUaoWJ9uq%3DeZEg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Arnau]

Hi Henrik,

first of all, thanks for your answer.

El mar., 6 nov. 2018 a las 20:12, Henrik Lindberg (<henrik....@puppet.com>) escribió:

[...]

> $dirs = ['static','media','photos']
> $proxy = $dirs.reduce([ { 'path' => '/', 'url' =>
> "http://localhost:${port}/" } ]) |$memo, $value| { $memo + [ { 'path' =>
> "/$value/", 'url' => '!' } ] }
> notify { "MemoOut: $proxy" : ;}
>
> If I puppet apply the above code the output looks like:

Avoid having  notify with a title containing [ ] - it can get very
confusing. Use a title like "testing", and set the message instead.

Ok, thanks. Then:

 notify {"test $name" :

    message => "static_served: ${apache[static_served]}",

 }

produces:

 defined 'message' as 'static_served: [static]'

I need to enclose the hash in curly braces otherwise it shows the whole hash like:

defined 'message' as 'Message {enable => true, servername => application, static_served => [static]}[static_served]' 

You give the reduce an array with a hash in it, and you then append to
that array with an array containing a hash. The result will be an
array where each element is a hash.

Yes, exaclty. That's what I need (if I have not missunderstood it) to pass tot he proxy_pass parameter in the apache forge module:

 proxy_pass => [
    { 'path' => '/a', 'url' => 'http://backend-a/' },
    { 'path' => '/b', 'url' => 'http://backend-b/' },
]

[...] 

> I make this an array using any2array (array un puppet 5) so I can still
> call the reduce function::
>
> $var=(Array("${apache['static_served']}"))
>
> so the /reduce /line now looks like:
>
You are asking Array to create an array out of a string. It will
construct that as an array where each element is a single character string.

Ah, ok, cause I'm passing the first element of the array and not the array to the Array function? and this is becasue of the quoting, right?

You probably wanted

$var = Array($apache['static_served'])

 

> $_proxy_pass = $var.reduce([ { 'path' => '/', 'url' =>

> "http://localhost:${port}/" } ]) |$memo, $value| { $memo + [ { 'path' =>
> "/$value/", 'url' => '!' } ] }
>
> And here is where everythiong starts to make even less sense:
>

You are now performing a reduce based on a sequence of single character
strings. This is indeed confusing as you have already gone off the road.

So, at the end it was as simple as calling the reduce function like:

$apache['static_served'].reduce

(I as using a more larger hiera data and, in the very first application, the static_serverd array was empty, so reduce was failling all the time cause the array was empty... Damn!!

now the code looks like:

if $apache['static_served'] {

    $_proxy_pass =  $apache['static_served'].reduce([ { 'path' => '/', 'url' => "http://localhost:${port}/" } ]) |$memo, $value| { $memo + [ { 'path' => "/$value/", 'url' => '!' } ] }

  }else{

   $_proxy_pass =  [ { 'path' => '/', 'url' => "http://localhost:${port}/" } ]

  }

  notify { "PROXY: $name" :

    message => $_proxy_pass,

  }

But the reduce is now not working:

Notice: {"path"=>"/", "url"=>"http://localhost:56902/"}

Notify[PROXY: application]/message: defined 'message' as {

  'path' => '/',

  'url' => 'http://localhost:56902/'

}

what am I doing wrong now?

BTW, I find the hash notation/manipulation a little confusing... According to https://puppet.com/docs/puppet/5.4/lang_data_hash.html it must be as simple as:

$key[$value]

And quotes should be used in "keys" when they are strings, but in the above example:

$var = Array($apache['static_served'])

 you are quoting the $value when it's a array and not a string...

what is the generic rule for quoting in hashes?

Hope this helps you get back on track.

- henrik

Thanks again!

Arnau 

[Henrik Lindberg]

On 2018-11-07 10:40, Arnau wrote:
> Hi Henrik,
>
> first of all, thanks for your answer.
>
> El mar., 6 nov. 2018 a las 20:12, Henrik Lindberg

> (<henrik....@puppet.com <mailto:henrik....@puppet.com>>) escribió:

> [...]
>
> > $dirs = ['static','media','photos']
> > $proxy = $dirs.reduce([ { 'path' => '/', 'url' =>
> > "http://localhost:${port}/" } ]) |$memo, $value| { $memo + [ {
> 'path' =>
> > "/$value/", 'url' => '!' } ] }
> > notify { "MemoOut: $proxy" : ;}
> >
> > If I puppet apply the above code the output looks like:
>
> Avoid having  notify with a title containing [ ] - it can get very
> confusing. Use a title like "testing", and set the message instead.
>
>
> Ok, thanks. Then:
>  notify {"test $name" :
>     message => "static_served: ${apache[static_served]}",
>  }
> produces:
>  defined 'message' as 'static_served: [static]'
>

> /I need to enclose the hash in curly braces otherwise it shows the whole
> hash like:/

Not sure - but I simplified your code, and this works:

test.pp
------
$apache = { 'static_served' => [ 'test1', 'test2'] }
$port = 8888

$start_value = [{

'path' => '/',
'url' => "http://localhost:${port}/"
}]

$_proxy_pass = if $apache['static_served'] =~ Array {
$apache['static_served'].reduce($start_value) |$memo, $value| {
$memo + [{'path' => "/${value}/", 'url' => '!' }]
}
} else {
$start_value
}
notice($_proxy_pass)

>> puppet apply test.pp

Notice: Scope(Class[main]): [{path => /, url => http://localhost:8888/},
{path => /test1/, url => !}, {path => /test2/, url => !}]


If you change the last notice to do this:

notice String($_proxy_pass)

You will get better looking output with quotes around strings:

Notice: Scope(Class[main]): [{'path' => '/', 'url' =>
'http://localhost:8888/'}, {'path' => '/test1/', 'url' => '!'}, {'path'
=> '/test2/', 'url' => '!'}]

Which to me looks like what you intended.

The result is the same in both cases, but the output you get by using
String to convert the result before the notice gives you sane
formatting. If just doing a notice or setting the value as a "message"
you will get a backwards compatible transformation to string that does
not show quotes etc. You may not want the quotes in the actual result
you are going to use, but it is valuable when testing/debugging.

Hope this helps you.

Best,
- henrik

>
> BTW, I find the hash notation/manipulation a little confusing...
> According to https://puppet.com/docs/puppet/5.4/lang_data_hash.html it
> must be as simple as:
>
> $key[$value]
>
> And quotes should be used in "keys" when they are strings, but in the
> above example:
>
> $var = Array($apache['static_served'])
>
>  you are quoting the $value when it's a array and not a string...
>
> what is the generic rule for quoting in hashes?
>
>

There is something you must have misunderstood. You need to quote values
when you need to make sure it is a string. Puppet allows using what is
known as bare words - i.e. some characters that make out a word, when
that bare word does not mean anything in particular to puppet.

For example:

notice(hello)

Which works because the word hello is not special to puppet. A bare word
is just a string. This is exactly the same thing:

notice("hello")

Which is the same as:

notice('hello')

The documentation says that you should quote string keys in hashes -
i.e. it is recommended to write:

{"hello" => "world"}

instead of

{ hello => world }

since you never know if bare words will have no meaning to puppet for
ever - by quoting you are telling puppet "hey, this is really a string".

For example - in this example - you *must* quote:

{ "if" => "is the question"}

If you remove the quotes around "if" you will get a syntax error.

Next thing - the difference between $x and "$x" or "${x}" - the $x means
the value of the variable named x. The "$x" means a string where the
part $x is replaced by the value of the variable x, and "${x}" is the
same thing as "$x", but allows more elaborate expression inside the
braces. Always use the "${ }" form.

> Hope this helps you get back on track.
>
> - henrik
>
>
> Thanks again!
> Arnau
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CAM69jx9evODEdpT%2B-aCoHtf6iZc1j1pxguA7%2B6onaBAb31daBA%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-users/CAM69jx9evODEdpT%2B-aCoHtf6iZc1j1pxguA7%2B6onaBAb31daBA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Arnau]

Hi Henrik,

I did not answer before cause I was playing a little more with the code.... thanks a lot for your asnwers and your explanation.

El mié., 7 nov. 2018 a las 11:37, Henrik Lindberg (<henrik....@puppet.com>) escribió:

> what am I doing wrong now?
>
Not sure - but I simplified your code, and this works:

My code is working as well. But, for soe reason, I have to move the notify below in my code ....

 

>
> BTW, I find the hash notation/manipulation a little confusing...
> According to https://puppet.com/docs/puppet/5.4/lang_data_hash.html it
> must be as simple as:
>
> $key[$value]
>
> And quotes should be used in "keys" when they are strings, but in the
> above example:
>
> $var = Array($apache['static_served'])
>
>   you are quoting the $value when it's a array and not a string...
>
> what is the generic rule for quoting in hashes?
>
>
There is something you must have misunderstood. You need to quote values

Thanks for this explanatin!! very clarifying!

Best,

Arnau