A note on upcoming module metadata improvements

[Ryan Coleman]

We've been working hard to reduce the complexity involved in publishing modules to the Forge and make it simpler to find great modules. I'm writing today to give you some background on the problems we're working to solve and the approach we'd like to take to help solve them.

To publish a Forge module today, you must provide some metadata in a form on forge.puppetlabs.com and create a file named Modulefile in your module root which contains additional (and in some cases duplicate) metadata. Then you must run `puppet module build` or follow a workflow in Geppetto which both transform the Modulefile into the metadata.json artifact that Puppet and the Forge consume.

On top of that, the types of metadata you can enter into the Modulefile is static and requires an update to Puppet to improve. You've been waiting for a way to find modules that work with your version of Puppet and on the operating systems you run in your datacenter. The Forge is ready to display this information, allow you to filter search results on it and more. The last step is to allow for additional metadata to be supplied by module authors in a single, simple, source of metadata.


So, what are we planning to simplify module publishing and enable better module discovery on Forge?

- The web form on Forge currently required to publish a new module is going away.
- The Modulefile metadata file will be deprecated in the next major release of Puppet
- metadata.json (part of the `puppet module build` output) will become the single-source of metadata
- http://docs.puppetlabs.com/puppet/latest/reference/modules_publishing.html will be updated to reflect these changes and to document brand new metadata keys including:
 - puppet_version : The major.minor versions of Puppet your module supports like 2.7 or 3.3
 - operating_system : OSes your module supports stated like the $operatingsystem Facter values Ubuntu or RedHat
- This new metadata will be used on forge.puppetlabs.com on module pages, search results and will be available over its (soon to be released) REST API.

There are a couple of notable caveats to the above which will be included in the documentation along with much greater detail on each metadata key. For example, we'd like to allow for more flexible expressions in puppet_version (not just major.minor) and we'd like to allow for version comparison in operating_system. We plan to introduce these keys as described above until Forge is ready to accept more complicated expressions. Once able, we'll update the documentation to document the new capabilities and accept both forms of metadata.



Will this break all my modules and when can I start using this stuff?

We know that these changes aren't trivial and even though Puppet and the Forge use metadata.json already, some tools interact with Modulefile directly. We're doing several things to make this transition seamless and pleasant.

First, rest easy knowing that the modules you build with the module tool available in 2.7.14 and later will continue to work. Though you won't be able to express the new metadata or publish to the Forge without visiting the website, older versions of Puppet and Geppetto will continue to create and interact with modules just the same.

Users of Puppet 3.3.0 and later will enjoy extra flexibility during this transition. It includes a change to respect the new metadata fields expressed in metadata.json when building your release tarball based on the metadata entered in Modulefile. Though you won't be living the single source of metadata dream, you can still express new metadata with few manual steps.

The next major version of Puppet will include a greatly revamped Puppet Module Tool that (amongst other improvements) will make it easy to express and validate the new metadata directly in metadata.json. A future release of Geppetto will be the easiest choice when it provides a simple form editor to express, edit and validate the new metadata.

The updated documentation which should be available within the next month. It will also describe these changes in greater detail and provide all the information you need to get started.

Best of all, once module authors begin publishing new releases which contain the new metadata keys, you'll be able to filter your Forge search results on the Puppet versions and operating systems that you care about. Three sources of module metadata will be reduced to one authoritative source that doesn't require an upgrade of your entire Puppet infrastructure to improve. Publishing modules to the Forge will no longer require web-site interaction enabling vastly improved workflows like GitHub-based publishing that we hope to work on in the coming months.

On behalf of all of us at Puppet Labs, I want to thank you for being such an awesome community and I'm looking forward to the next year of module development. You're always welcome to email me directly or find me as ryanycoleman in #puppet on Freenode. There are many more things coming from us in the next few months and I look forward to sharing those with you soon.


--Ryan

[Alessandro Franceschi]

Il giorno mercoledì 16 ottobre 2013 18:07:57 UTC+2, Ryan Coleman ha scritto:

We've been working hard to reduce the complexity involved in publishing modules to the Forge and make it simpler to find great modules. I'm writing today to give you some background on the problems we're working to solve and the approach we'd like to take to help solve them.

To publish a Forge module today, you must provide some metadata in a form on forge.puppetlabs.com and create a file named Modulefile in your module root which contains additional (and in some cases duplicate) metadata. Then you must run `puppet module build` or follow a workflow in Geppetto which both transform the Modulefile into the metadata.json artifact that Puppet and the Forge consume.

On top of that, the types of metadata you can enter into the Modulefile is static and requires an update to Puppet to improve. You've been waiting for a way to find modules that work with your version of Puppet and on the operating systems you run in your datacenter. The Forge is ready to display this information, allow you to filter search results on it and more. The last step is to allow for additional metadata to be supplied by module authors in a single, simple, source of metadata.


So, what are we planning to simplify module publishing and enable better module discovery on Forge?

- The web form on Forge currently required to publish a new module is going away.
- The Modulefile metadata file will be deprecated in the next major release of Puppet
- metadata.json (part of the `puppet module build` output) will become the single-source of metadata
- http://docs.puppetlabs.com/puppet/latest/reference/modules_publishing.html will be updated to reflect these changes and to document brand new metadata keys including:
 - puppet_version : The major.minor versions of Puppet your module supports like 2.7 or 3.3
 - operating_system : OSes your module supports stated like the $operatingsystem Facter values Ubuntu or RedHat
- This new metadata will be used on forge.puppetlabs.com on module pages, search results and will be available over its (soon to be released) REST API.

All this sound great,

as Dolf has written in his reply,  and others like Gareth have suggested at the PuppetConf, a hugely useful key would be something like 'Optional-dependency' with relevant option(s) on the module tool on how to manage it.

Maybe it could also be nice to add a key where are expressed the external resources the module depends upon (either mandatory or optional): most of the times you depend on an apache module, for example, just because you use its apache::vhost and that single dependency more or less easily fixable, may prevent different modules from being used together.

Both these keys would greatly help interoperability among modules, IMHO.

Can open a feature request, if there's not already one.

Al