Can custom functions accept arrays as parameters?

[Matthew Probst]

I'm having a difficult time with a custom function:

module Puppet::Parser::Functions 

  newfunction(:subpaths, :type => :rvalue) do |args|

    result = []

    args.flatten.each do |path|

      temp_path = path

      begin

        result << temp_path

        temp_path = File.dirname(temp_path) 

      end while (temp_path != "/" and temp_path != ".")

    end

    return result.uniq

  end

end

If I try calling this in a Puppet manifest:

class testing {

subpaths("/usr/local/bin")

subpaths("/usr/local/bin", "/var/log/foo")

subpaths(["/usr/local/bin","/var/log/foo"])

}

and I get that to run on a host, with no other classes called, I get an error 

 Error 400 on SERVER: can't convert Array into String 

I tried changing my custom function to

module Puppet::Parser::Functions 

  newfunction(:subpaths, :type => :rvalue) do |args|

    return ""

  end

end

and it still does the same thing.

I see in the bug tracker that several times in the past the ability to pass in arrays has been broken through inadvertent changes, but I see no recent discussion on whether you _should_ be able to pass arrays in; in other words, it could be an intentional design, in which case I can just try something else.

However, when I browse the source code in /usr/lib/ruby/site_ruby/1.8/puppet/parser/functions, I don't see any magic in the functions that accept arrays, except that many seem to use .flatten on the args array passed in.

I tried that in my initial attempt, and it seems that it fails before it even gets to calling the custom function.

I've tried looking in the mailing lists and searching the web, but most mentions of this seem to be for older pre-2.x versions of Puppet.

[Andrew Parker]

What version of puppet are you using? Also, what do you get if you run it with --trace?

> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To post to this group, send email to puppe...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-dev+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.

[Matthew Probst]

On Tue, Jul 24, 2012 at 1:17 PM, Andrew Parker <an...@puppetlabs.com> wrote:

What version of puppet are you using? Also, what do you get if you run it with --trace?

Hostnames are edited.  Otherwise cut and pasted.

On the puppet master and on the client:

[root@puppet-client ~]# puppet --version

2.7.18

[root@puppet-master ~]# puppet --version

2.7.18

Here's a run with --trace:

 

[root@puppet-client ~]# puppet agent --test --noop --trace

notice: Ignoring --listen on onetime run

/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:56:in `deserialize'

/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:120:in `find'

/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:196:in `find'

/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:240:in `retrieve_new_catalog'

/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:490:in `thinmark'

/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'

/usr/lib/ruby/1.8/benchmark.rb:307:in `realtime'

/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:489:in `thinmark'

/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:239:in `retrieve_new_catalog'

/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:86:in `retrieve_catalog'

/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:112:in `retrieve_and_apply_catalog'

/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:152:in `run'

/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:43:in `run'

/usr/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:21:in `lock'

/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:43:in `run'

/usr/lib/ruby/1.8/sync.rb:229:in `synchronize'

/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:43:in `run'

/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:95:in `with_client'

/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run'

/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:172:in `call'

/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:172:in `controlled_run'

/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'

/usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:337:in `onetime'

/usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:311:in `run_command'

/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'

/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'

/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'

/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'

/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'

/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:69:in `execute'

/usr/bin/puppet:4

err: Could not retrieve catalog from remote server: Error 400 on SERVER: can't convert Array into String at /etc/puppet/environments/development/modules/testing/manifests/init.pp:7 on node puppet-client

warning: Not using cache on failed catalog

err: Could not retrieve catalog; skipping run

This was run from the client.  The following is in effect for this client right now:

nodes.pp:

node ' somebodys-server-at-somewhere ' {

include testing

}

modules/testing/manifests/init.pp:

# modules/testing/manifests/init.pp:

#

class testing {

$foo = subpaths("/usr/local/bin")

$bar = subpaths("/usr/local/bin", "/var/log/foo")

$baz = subpaths(["/usr/local/bin","/var/log/foo"])

}

init.pp:7 is the third line which tries to assign to $baz.

[Matthew Probst]

And yes, I do understand that the custom function will always run on the puppet master.  It's just that only the config for this particular host originally triggered the problem, so that's where I boiled it down to a simpler test case.

On Tue, Jul 24, 2012 at 1:17 PM, Andrew Parker <an...@puppetlabs.com> wrote:

[Nick Fagerlund]

On Tuesday, July 24, 2012 11:06:29 AM UTC-7, Matthew Probst wrote:

I see in the bug tracker that several times in the past the ability to pass in arrays has been broken through inadvertent changes, but I see no recent discussion on whether you _should_ be able to pass arrays in; in other words, it could be an intentional design, in which case I can just try something else.

Yo, docs team needs to know the answer to this one too.

Puppet's "normal" data types (excluding regexes, which can't be used most places) are:

• Booleans
• The special value "undef"
• Strings 
• Resource references
• Numbers
• Arrays
• Hashes

Are all of these valid to pass to functions?

[Matthew Probst]

On Tue, Jul 24, 2012 at 2:53 PM, Nick Fagerlund <nick.fa...@puppetlabs.com> wrote:

Yo, docs team needs to know the answer to this one too.

Puppet's "normal" data types (excluding regexes, which can't be used most places) are:

• Booleans
• The special value "undef"
• Strings 
• Resource references
• Numbers
• Arrays
• Hashes

Are all of these valid to pass to functions?

And indeed, a few of the built-in functions _do_ seem to accept arrays.  So I'd expect that I can pass one to a custom function.

My first thoughts were that I've made some other unrelated error in syntax or design or Ruby programming, as Ruby is a new language to me.

My second thought was that perhaps I need to send some other symbol to newfunction() to indicate it _can_ accept arrays.  I don't see that mentioned anywhere in the source code for the built-ins, but while I'm experienced with other languages, my inexperience in Puppet and Ruby might be leading me astray.

My third thought was that somehow the functions that are built-in are marked as able to accept arrays by some other portion of the Puppet code that I don't know well enough to find.

The Puppet custom functions page doesn't mention much about it.  I've been doing web searches but am hindered somewhat by my inexperience, in that I don't know what keywords to use, and the keywords I do use are generic enough that they have a hard time drilling down to Puppet results ("object" "reference" "array" "string" "function" "puppet" "ruby").

I'd vote for being able to pass standard Puppet data types into functions if there were a decision to be made, even if they are not default and are somehow restricted.  I'd live with the input into a custom function being unfolded if it's an array, however I see from past bug/trouble reports that others have specifically requested that the ability to pass arrays be restored, after accidental removal of that feature by adding unfolds at various points in the code.

 

And, to keep this on a positive note, the Puppet devs have created a nice piece of software, that seems quite well documented in most ways, I just have a knack for stumbling into trouble.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-dev/-/YvWUzmAOnUEJ.

[Nick Fagerlund]

So I just took a look at how "include" does it:

  1 # Include the specified classes
  2 Puppet::Parser::Functions::newfunction(:include, :doc => "Evaluate one or more classes.") do |vals|
  3     vals = [vals] unless vals.is_a?(Array)

Maybe try that? Or maybe modify what you did and try [args].flatten.each do etc. etc.

[Matthew Probst]

Actually, I had it _that_ way before I tried the ways I posted previously, in imitation of the exact same code.  But, to make sure I wasn't just messing it up, I went ahead and tried it again with this code:

# subpaths.rb

#

module Puppet::Parser::Functions 

  newfunction(:subpaths, :type => :rvalue) do |args|

    args = [args] unless args.is_a?(Array)

    args.each do |path|

      temp_path = path

      begin

        result << temp_path

        temp_path = File.dirname(temp_path) 

      end while (temp_path != "/" and temp_path != ".")

    end

    return result.uniq

  end

end

The result was similar:

At this point I want to give it a try from the puppet master side, to make sure it isn't just some network error.  That trace mentions things related to REST and deserializing, which is not what I expected, though I admit my inexperience.

Remember, in my first post, I mentioned that I also tried it with a function that does no processing on the args, and just returns empty string.  I'll try that again too and get back.

--

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-dev/-/ARmOiWuJuzAJ.

[Matthew Probst]

Ahhhh.  Indeed there is something wrong with my function.  Sorry to bother the folks here, I was mostly just worried that there might have been a design decision against allowing this.  I'll keep adding bits back until it fails.

--

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-dev/-/ARmOiWuJuzAJ.

[Brice Figureau]

The error *happens* on the master, because functions are evaluated
during compilation. But what you see is the exception on the client side
which has little to do with your error.

Run the master with --trace and look to the master log to see the
stacktrace.

> Remember, in my first post, I mentioned that I also tried it with a
> function that does no processing on the args, and just returns empty
> string. I'll try that again too and get back.

Couldn't it be an issue with the function returned value and not its
argument?

Only the full master stacktrace will let us know :)
--
Brice Figureau
My Blog: http://www.masterzen.fr/

[Matthew Probst]

I got it working by using a function from the Puppet Labs standard library as a starting point.  I never did find out exactly what was going on.

I thought that the logs it was giving me in /var/log/puppet were all I could get.  For some reason proper logging never got set up, but the presence of the HTTP logs fooled me into thinking I couldn't get any more.

I may never know what the issue was now.  The specific formula that finally worked was

module Puppet::Parser::Functions

  newfunction(:subpaths, :type => :rvalue) do |arguments|

    input = arguments[0]

    if ! input.is_a?(Array)

      input = [input]

    end

    result = []

    input.each do |path|

[Matthew Probst]

On Tue, Jul 24, 2012 at 3:37 PM, Brice Figureau <brice-...@daysofwonder.com> wrote:

Couldn't it be an issue with the function returned value and not its
argument?

It could, but I thought I had eliminated that possibility by rewriting the function to return a string no matter what the input was, and testing that.

 

Only the full master stacktrace will let us know :)

Which, due to my own competence, seems to never have been recorded.  I trusted the default installation to either "configure logs" or "not configure logs".  Rather, it seems to have "configured HTTP logs" but "not configured other logs".  It's my own assumptions that were at fault.

[Stefan Schulte]

On Tue, Jul 24, 2012 at 10:37:47PM +0200, Brice Figureau wrote:
> On 24/07/12 22:30, Matthew Probst wrote:
> > Actually, I had it _that_ way before I tried the ways I posted
> > previously, in imitation of the exact same code. But, to make sure I
> > wasn't just messing it up, I went ahead and tried it again with this code:
> >
> > # subpaths.rb
> > #
> > module Puppet::Parser::Functions
> > newfunction(:subpaths, :type => :rvalue) do |args|

all arguments that you pass to your custom function are now represented
by args. So if you call your function like

subpaths('foo', 'bar')

args will ['foo', 'bar']. If you call your function like

subpaths(['foo', 'bar'])

then you really just pass one argument to your function, so args[0] is
['foo', 'bar'] and args = [['foo', 'bar']]

> > args = [args] unless args.is_a?(Array)
> > args.each do |path|

if you called your function like subpaths(['foo', 'bar']) then your
first temp_path will be the array ['foo', 'bar']

> > temp_path = path
> > begin
> > result << temp_path
> > temp_path = File.dirname(temp_path)

File.dirname fails if you pass an array with "can't convert Array into
String"

> > end while (temp_path != "/" and temp_path != ".")
> > end
> > return result.uniq

result is only valid inside your args.each block.

> > end
> > end
> >

I hope this helps understanding why this failed. In your last (working
example) you first extracted the first argument with arguments[0] and you
define result outside the each block.

-Stefan

[Matthew Probst]

It's hard to tell whether Stefan or Brice was involved in this
message, but I did eventually through some experimentation figure out
what was needed, and enhanced my actual understanding after I got it
working.


Thanks for the help.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.