Calling hiera functions from inside Ruby templates broken in Puppet 4.x?

[Maura Dailey]

Yes, I know this practice is discouraged in the documentation, but the updated hiera 2.0 documentation also assures me that it's supported and implies no changes to puppet 3.x manifests or templates are required.

This networking.erb file used to work in puppet 3.8 (this is a cruddy example, I use callouts to hiera data EXTENSIVELY in dozens of much more complicated configuration templates):

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=<%= @fqdn %>
GATEWAY=<%= scope.function_hiera(['gateway']) %>
NOZEROCONF=yes

Now in puppet 4.1, I get the following:

Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Function Call, Failed to parse template networking/network.erb:
  Filepath: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/functions/fail.rb
  Line: 3
  Detail: hiera() has been converted to 4x API

I checked the hiera 2.0 documentation on the names of functions here: http://docs.puppetlabs.com/hiera/latest/puppet.html#hiera-lookup-functions

and the hiera 2.0 documention on how to call them from inside templates here: http://docs.puppetlabs.com/hiera/latest/puppet.html#using-the-lookup-functions-from-templates

What am I missing? I tried to read the Ruby source for hints, but I'm more of a Python person and couldn't find any documentation on how to call the 4x API functions correctly. Thanks in advance for any assistance.

[Henrik Lindberg]

You have unfortunately run into a problem with a puppet 3.x function
(template()) making a call to a function that have been migrated to the
4.x function API. In the new API functions are not called via scope, and
a 3x function does not have the needed information to do so easily.

You can get around the problem by using the new templating system EPP
where the logic is written in the Puppet Language. This is also the long
term direction (The more safe, and better specified EPP over ERB where
you are exposed to the internals of the puppet runtime).

The snippet you showed can be written like this in EPP:

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=<%= $fqdn %>
GATEWAY=<%= hiera('gateway') %>
NOZEROCONF=yes

I understand this may be a bit of work when you have many and
complicated templates esp. if you rely on the internals of Puppet.
For regular templates that only access variables, it should be as easy
as replacing a @varname with $varname).

I think a ticket should be logged regarding the difficulty of calling a
4x function (this has popped up in other contexts (where it was possible
to work around the issue more easily)).

What is needed is a calling mechanism that is agnostic; a function that
is written with the 4.x. API simply uses the method
'call_function(<NAME>, <ARG1>, <ARG2>, ...)' and it calls either a 3.x
or a 4.x function. A similar thing is needed in an accessible way from
within a 3.x. function or template (e.g. scope.call_function with the
same signature as in the 4.x API). The fix would entail something like
what I am describing below...

Alternatively, to get past the problem if you do not want to move to EPP
right away here is how to call a 4.x function (but this is not
considered API, as we are planning refactoring how calls are made - they
are done in several different ways atm):

# Note that this requires >= 3.8.1 with future parser or 4.1.0.
#
def call_function(name, args, scope, &block)
# Get the loader that serves the environment - i.e for a call that
# comes from within the environment. This makes all functions in the
# environment (all modules, and environment private function) visible)
# to the caller. (This is somewhat cheating wrt. visibility - i.e.
# 'private' functions which will be supported in a later version).
#
loader = scope.compiler.loaders.private_environment_loader
if loader && func = loader.load(:function, name)
return func.call(scope, *args, &block)
end
# the function was not found... deal with it
end

The above is a modified version of what is used in the puppet runtime,
and should be possible to use in a template - even as a one-liner (in
your case:

scope.compiler.loaders.private_environment_loader.load(:function,
'hiera').call(scope, 'gateway')

Note that the real implementation is also able to call 3.x functions,
and is written to support visibility rules (i.e. handling private
functions. See the method 'call_function' defined in
puppet/pops/evaluator/runtime3_support.rb) if you want to look at the
complete implementation).

Regards
- henrik

>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/5b8e076a-6eb2-4640-89e0-96e64cb3101d%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/5b8e076a-6eb2-4640-89e0-96e64cb3101d%40googlegroups..com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Maura Dailey]

Thank you for your detailed response. It appears the primary problem could be in the hiera documentation, then, as it still refers to calling conventions that are apparently now completely outdated. I've read your blog before but didn't pick up on the meaning of the article about epp templates.

I did a quick test using this syntax, and this appears to work perfectly. This will work for 99% of my existing templates, I think. (I had to switch the calling convention inside the pp file to call epp instead of template before it stopped giving me errors. Oops.)

Thanks for the extra details here. Obviously, this is fairly low level, and I don't want to find things breaking out from under me in a future puppet release! I've gone over my existing templates and the EPP format should be sufficient to meet my needs without more programmatic changes.

[Stephen Gelman]

Unfortunately using EPP is not sufficient for all of our templates.  (In a few places we use some pretty intense logic that we are working to simplify but it will take a while)  Are there plans to make it easier to call hiera from within a template?  Would porting the template function to be a 4.x function change its ability to call the hiera function more easily?  I'm a little frustrated that a huge breaking change like this was made without the behavior either being deprecated first and also without any mention of the breaking change in the puppet 4 release notes.

Thanks,

Stephen

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7f7b4d1b-0299-4925-9298-39dcc36d372e%40googlegroups.com.

[Henrik Lindberg]

On 2015-27-05 20:08, Stephen Gelman wrote:
> Unfortunately using EPP is not sufficient for all of our templates. (In
> a few places we use some pretty intense logic that we are working to
> simplify but it will take a while) Are there plans to make it easier to
> call hiera from within a template?

What I suggested earlier would help - a unified way of calling from 3.x
(method on scope or a helper utility). Also perhaps making such a method
available directly in the template context.

> Would porting the template function
> to be a 4.x function change its ability to call the hiera function more
> easily?

Not by itself since it is the invocation of the ERB that needs to run in
a context where the mechanism to call both 3.x and 4.x is available.
(i.e. the suggestion above).

> I'm a little frustrated that a huge breaking change like this
> was made without the behavior either being deprecated first and also
> without any mention of the breaking change in the puppet 4 release notes.
>

I am very sorry, it was unintentional to break the case of calling
functions from within the templates. This feature is not covered by
tests and we simply did not catch this regression until it bit you.

I am also frustrated as it is not that easy to provide a quick fix for this.

Regards
- henrik

> epp instead of template before it stopped giving me errors. Oops..)

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/7f7b4d1b-0299-4925-9298-39dcc36d372e%40googlegroups.com

> <https://groups.google.com/d/msgid/puppet-users/7f7b4d1b-0299-4925-9298-39dcc36d372e%40googlegroups..com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CAARq2BMLGREUapQ7oZcoP%3DRvp7-o2_KvnDkD7QpZW4%2BjB9it6g%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-users/CAARq2BMLGREUapQ7oZcoP%3DRvp7-o2_KvnDkD7QpZW4%2BjB9it6g%40mail.gmail.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

[Stephen Gelman]

Henrik,

Thanks for the reply.  Now that it is established that this is a problem is there a plan to re-add this functionality?  Should I submit a ticket for it?

Thanks,

Stephen

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/mk7d4p%24bfc%241%40ger.gmane.org.

[Henrik Lindberg]

On 2015-04-06 24:37, Stephen Gelman wrote:
> Henrik,
>
> Thanks for the reply. Now that it is established that this is a problem
> is there a plan to re-add this functionality? Should I submit a ticket
> for it?
>

Here is the ticket that was logged for this. A solution is in the works.
https://tickets.puppetlabs.com/browse/PUP-4753

- henrik

[Henrik Lindberg]

On 2015-13-06 3:59, Henrik Lindberg wrote:
> On 2015-04-06 24:37, Stephen Gelman wrote:
>> Henrik,
>>
>> Thanks for the reply. Now that it is established that this is a problem
>> is there a plan to re-add this functionality? Should I submit a ticket
>> for it?
>>
>
> Here is the ticket that was logged for this. A solution is in the works.
> https://tickets.puppetlabs.com/browse/PUP-4753
>

To close the loop:
There is now a fix for this that will be released in 3.8.2 and 4.2.0.

Scope gets a new method, so the original problematic template line can
be written as:

GATEWAY=<%= scope().call_function('hiera',['gateway']) %>

The new scope.call_function can be used both with and without future
parser in effect, it always takes arguments as specified by 4.x (i.e. no
funny "undef is empty string"), and if needed it will convert to the 3.x
calling convention if the target function is implemented as a 3.x function.