Macros in puppet

[Pawel Tomulik]

Hi,

macros in puppets -> http://forge.puppetlabs.com/ptomulik/macro

They're similar to parser functions with three differences:

• macro names may resemble names of puppet variables/bindings, e.g. 'foo::bar::geez'
• stored in a hierarchy of directories, that is (for macro 'foo::bar::geez'):
  

      + lib/puppet/parser/macros/
        + foo/
          + bar/
            + geez.rb

• they're a little bit easier to implement than functions, because arity checking is provided out of the box

In some cases these macros may be more handy than 'params' classes when defining defaults for parameters. For example defaults for defined types which vary from instance to instance, or values which are hard to be computed in puppet DSL may be easily handled with macros.

Leave a comment, if you find it useful or useless.

Regards!
--
Pawel Tomulik

[Erik Dalén]

On 27 January 2014 04:00, Pawel Tomulik <ptom...@meil.pw.edu.pl> wrote:

Hi,

macros in puppets -> http://forge.puppetlabs.com/ptomulik/macro

They're similar to parser functions with three differences:

• macro names may resemble names of puppet variables/bindings, e.g. 'foo::bar::geez'
• stored in a hierarchy of directories, that is (for macro 'foo::bar::geez'):
  

      + lib/puppet/parser/macros/
        + foo/
          + bar/
            + geez.rb

• they're a little bit easier to implement than functions, because arity checking is provided out of the box

While I think that block parameters is nicer than the current puppet method of putting all arguments in an array, there is builtin support for arity checking in puppet 3.x. You specify the arity in the function definition, like :arity => 2.

I think it should be possible for you to do strict arity checking even with just regular blocks as well, not only with lambdas. The block parameter has a arity method that you can use to find out the arity and then have something that runs before the macro in question that checks the parameters given against that, kind of like how the puppet 3.x function calling does it.

 

In some cases these macros may be more handy than 'params' classes when defining defaults for parameters. For example defaults for defined types which vary from instance to instance, or values which are hard to be computed in puppet DSL may be easily handled with macros.

Leave a comment, if you find it useful or useless.

Cool stuff. I like the simpler definition of them with less boilerplate than the current functions.

One thought though when me and others and discussed a new function API for puppet we considered just having a module with methods on it instead. So to define a new function you would just need to open that module and define a new method. I think that would be more straightforward for people that already know Ruby. And probably a lot easier to write unit tests for as they would work just like a unit test for any other method. What do you think about doing something like that to define functions/macros?

 

Regards!
--
Pawel Tomulik

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/1f08465e-0386-4045-8399-26d00888618e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Erik Dalén

[Pawel Tomulik]

W dniu poniedziałek, 27 stycznia 2014 10:14:40 UTC+1 użytkownik Erik Dalén napisał:

On 27 January 2014 04:00, Pawel Tomulik <ptom...@meil.pw.edu.pl> wrote:

Hi,

macros in puppets -> http://forge.puppetlabs.com/ptomulik/macro

They're similar to parser functions with three differences:

• macro names may resemble names of puppet variables/bindings, e.g. 'foo::bar::geez'
• stored in a hierarchy of directories, that is (for macro 'foo::bar::geez'):
  

      + lib/puppet/parser/macros/
        + foo/
          + bar/
            + geez.rb

• they're a little bit easier to implement than functions, because arity checking is provided out of the box

While I think that block parameters is nicer than the current puppet method of putting all arguments in an array, there is builtin support for arity checking in puppet 3.x. You specify the arity in the function definition, like :arity => 2.

I suppose, It's not possible to specify default parameters that way, this was reason for introducing 'parameters' method to Object in ruby 1.9. Macros handle default parameters out of the box (on ruby 1.9+) and you don't have to specify arities/parameter descriptions explicitly. Well, this could be easily implemented also for functions, I believe.

 

I think it should be possible for you to do strict arity checking even with just regular blocks as well, not only with lambdas. The block parameter has a arity method that you can use to find out the arity and then have something that runs before the macro in question that checks the parameters given against that, kind of like how the puppet 3.x function calling does it.

Of course it's possible, maybe not by using arity, but the parameters method (1.9+). It's actually implemented in macros and work seamlesly for lambdas and blocks on ruby 1.9+ (I was quite asleep when writing docs). The arity method (which is the only available method on ruby 1.8) doesn't handle well default parameters:

    irb(main):020:0> def arity(&block); p block.arity; end
    irb(main):021:0> arity do |x,y=1| end
    1
    => 1
    irb(main):021:0> arity do |x| end
    1
    => 1

With lambdas, we just have the arities verified additionally by ruby, so in case my macro's validation lies, we always have this extra guard.

 

 

In some cases these macros may be more handy than 'params' classes when defining defaults for parameters. For example defaults for defined types which vary from instance to instance, or values which are hard to be computed in puppet DSL may be easily handled with macros.

Leave a comment, if you find it useful or useless.

Cool stuff. I like the simpler definition of them with less boilerplate than the current functions.

One thought though when me and others and discussed a new function API for puppet we considered just having a module with methods on it instead. So to define a new function you would just need to open that module and define a new method. I think that would be more straightforward for people that already know Ruby. And probably a lot easier to write unit tests for as they would work just like a unit test for any other method. What do you think about doing something like that to define functions/macros?

 

That looks ok. I suppose it would need some convention for method naming, e.g:

module Functions
  def function_foo() # <- this is puppet function
  end
  def foo() <-this is not
  end
end

You could also make a convention for documentation:

module Functions
  def doc_foo <<-EOT
    blah blah
  end
end

I didn't thought about it this way, but it looks fine. I'm not sure, however, how could you handle environments in this approach. I've seen that currently functions are stored in hashes, one hash per puppet environment.
In my case, I just keep a hash of  Procs for each environment. I also plan to have another hash of macro docs, which will be build only on demand.

 

--
Erik Dalén

[Henrik Lindberg]

On 2014-27-01 4:00, Pawel Tomulik wrote:
> Hi,
>
> macros in puppets -> http://forge.puppetlabs.com/ptomulik/macro
>
> They're similar to parser functions with three differences:
>

> * macro names may resemble names of puppet variables/bindings, e.g.
> 'foo::bar::geez'
> * stored in a hierarchy of directories, that is (for macro

> 'foo::bar::geez'):
>
> + lib/puppet/parser/macros/
> + foo/
> + bar/
> + geez.rb
>

> * they're a little bit easier to implement than functions, because

> arity checking is provided out of the box
>
> In some cases these macros may be more handy than 'params' classes when
> defining defaults for parameters. For example defaults for defined types
> which vary from instance to instance, or values which are hard to be
> computed in puppet DSL may be easily handled with macros.
>

I have been working on a new API for functions that I hope we will be
able to finish for Puppet 4.0. I have just started and have looked at
the work dalen and zaphod42 did earlier.

The need for a new API is because the 3x API requires values to be
presented to functions in a particular way (empty strings instead of
undef is one of those). If we want to fix that we do need to have a new
API. I will come back later and present ideas in more detail (I am not
quite done with exploration yet).

Requirements as I see them:

* Must be redefinable in different environments / be reloadable
* Support fully qualified names
* Support arity (fixed and variable)
* Support default arguments
* Support the new type system with automatic type checking
* Support overloading of one function (i.e. multiple signatures)

* Be easy to write for simple functions
* Easy to test
* Callable direct from Ruby
* Protected from exposure from "too much non API" to reduce future
migration issues

So far, I am leaning against a design that:

* Uses regular ruby methods with regular parameter declaration (easier
for users - no need to parse the array args are internally passed).
* One function per ruby file (easier to autoload)
* A call to a newfunction "factory" method that creates the function
class/module internally - thus enabling using anonymous modules
(more or less required to be able to reload/redefine).
* Creation call looks horrible if all the desired features are
to be passed using a hash of options - hence, for more advanced
options, additional methods are implemented.
* Move Functions out of the Puppet::Parser namespace - they are not part
of the parser!

As an example - here is a simple function from 3x

Puppet::Parser::Functions::newfunction(:sha1, :type => :rvalue,
:arity => 1,
:doc => "Returns a SHA1 hash value from a provided string.") do
|args|
Digest::SHA1.hexdigest(args[0])
end

Rewritten it would be:

Puppet::Functions.create_function(:sha1) do
# Returns a SHA1 hash value from a provided string.
def sha1(str)
Digest::SHA1.hexdigest(str)
end
end

If we want to do more - handle multiple signatures, get automatic type
checking etc. Additional calls that defines those are required.
This part is what I am working on now - something along the lines of:

dispatch(:sha1, 'String[1]') # one arg == non empty string

(I am leaving out lots of detail here because ideas are half baked)

I want to be able to:

* Support simple calls
* Calls to different methods depending on signature
* Support Polymorphic dispatch (based on type of first arg) just
like the future parser/evaluator does.

I am also contemplating if we want to tie function more closely to
the type of the first argument to allow addition of functions with the
same name that operates on a different type - don't know how valuable
that would be in the Puppet Language though.

Hope to have a more complete proposal in a couple of weeks time.

Regards
- henrik

[Pawel Tomulik]

Names with '::'? So probably they can't be just module members (or there must be an additional step of registering these methods under fqdn names).
 

* Support arity (fixed and variable)

It appear that this is quite easy to implement, the idea is just to convert procs/blocks provided by user at definition point to lambas and store lambdas. Basically, I use this code in my implementation of macros: https://gist.github.com/ptomulik/8670700. Of course, if you store puppet functions as a regular methods it becomes even easier.
 

* Support default arguments

Works only on ruby 1.9+ (at least when we speek in terms of lambdas/procs). On 1.8 `|a=nil|` is a syntax error.
 

* Support the new type system with automatic type checking
* Support overloading of one function (i.e. multiple signatures)

That would be great! And shouldn't be so hard to implement, at least on ruby 1.9+.
 

* Be easy to write for simple functions

The current interface is not so scary, the only scary thing is the need to validate arguments manually, which usually takes 85% of the function's code. If there were dedicated validators, it would be just great! I suppose, most of the work could be done by the type system with automatic type checking (if it's applicable to function arguments).
 

* Easy to test

So - special rspec helper for functions?
 

* Callable direct from Ruby

Current functions are actually callable from ruby, aren't they?

[Henrik Lindberg]

On 2014-28-01 17:29, Pawel Tomulik wrote:
> W dniu wtorek, 28 stycznia 2014 16:47:53 UTC+1 użytkownik henrik
> lindberg napisał:

> I have been working on a new API for functions that I hope we will be
> able to finish for Puppet 4.0. I have just started and have looked at
> the work dalen and zaphod42 did earlier.
>
> The need for a new API is because the 3x API requires values to be
> presented to functions in a particular way (empty strings instead of
> undef is one of those). If we want to fix that we do need to have a new
> API. I will come back later and present ideas in more detail (I am not
> quite done with exploration yet).
>
> Requirements as I see them:
>
> * Must be redefinable in different environments / be reloadable
> * Support fully qualified names
>
>
> Names with '::'? So probably they can't be just module members (or there
> must be an additional step of registering these methods under fqdn names).
>

Yes, there must be an extra step - the classes must be anonymous. (easy
to do), and then bound via a name. (Work remains on how exactly this
should work).

> * Support arity (fixed and variable)
>
>
> It appear that this is quite easy to implement, the idea is just to
> convert procs/blocks provided by user at definition point to lambas and
> store lambdas. Basically, I use this code in my implementation of
> macros: https://gist.github.com/ptomulik/8670700. Of course, if you
> store puppet functions as a regular methods it becomes even easier.
>
> * Support default arguments
>
>
> Works only on ruby 1.9+ (at least when we speek in terms of
> lambdas/procs). On 1.8 `|a=nil|` is a syntax error.
>
> * Support the new type system with automatic type checking
> * Support overloading of one function (i.e. multiple signatures)
>
>
> That would be great! And shouldn't be so hard to implement, at least on
> ruby 1.9+.
>

It will be easy to implement using something similar to the polymorphic
dispatch used throughout the future parser/evaluator. It cannot be
specified using ruby methods directly - it is either implemented
with one method where the overaloding logic takes place inside one
method, or the user dispatches to different methods depending on
signature. (That is the idea anyway).

> * Be easy to write for simple functions
>
>
> The current interface is not so scary, the only scary thing is the need
> to validate arguments manually, which usually takes 85% of the
> function's code. If there were dedicated validators, it would be just
> great! I suppose, most of the work could be done by the type system with
> automatic type checking (if it's applicable to function arguments).
>

Yeah, interface is not too bad, but as you say, user is fully
responsible for the handling of arguments and their types.

> * Easy to test
>
>
> So - special rspec helper for functions?
>

I was thinking more about ability to just call it without having to have
scopes and stuff. Unfortunately, with an anonymous class it is not
as easy as just going MyFunc.call(a,b). Can almost get there though.

> * Callable direct from Ruby
>
>
> Current functions are actually callable from ruby, aren't they?
>

yes sure, but only after having the Puppet Runtime jumping through hoops
adding them to a scope (which requires an environment an a compiler, and
known resource types, and...) i.e to get a banana you also get a
gorilla and a jungle filled with wildlife.

- henrik