Module metadata

[Luke Kanies]

So (Paul Lathrop will correct me if I'm wrong, I hope), it looks like
we decided on the following simplistic metadata for modules:

* version
* name
* author
* source (i.e., a URL)
* requires

The requires need to be able to specify either names or names and
version.

We also decided that the data will be stored in a file named something
like 'metadata.json', and the file format will be json.

My plan at this point is to do the requires as either an array, or an
array of arrays:

requires: foo, bar
requires: [foo, 1.0], [bar, 2.0]

Comments?

I'm partway to a first version, anyway.

--
Commit suicide. A hundred thousand lemmings can't be wrong.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

[R.I.Pienaar]

hello,

----- "Luke Kanies" <lu...@madstop.com> wrote:

> So (Paul Lathrop will correct me if I'm wrong, I hope), it looks like
>
> we decided on the following simplistic metadata for modules:
>
> * version
> * name
> * author
> * source (i.e., a URL)
> * requires
>
> The requires need to be able to specify either names or names and
> version.
>
> We also decided that the data will be stored in a file named something
>
> like 'metadata.json', and the file format will be json.
>
> My plan at this point is to do the requires as either an array, or an
>
> array of arrays:
>
> requires: foo, bar
> requires: [foo, 1.0], [bar, 2.0]
>
> Comments?

Can you tell those who aren't at the camp what the useage of this meta data will be, will this for dependencies between modules somehow or is it merely a compile time aid, or something else completely?

I assume all the data will be available in some vars for all classes in teh module as well? I for one would love a $modulename var that's the same in all sub classes.

--
R.I.Pienaar

[Paul Lathrop]

On Thu, Oct 1, 2009 at 3:57 PM, Luke Kanies <lu...@madstop.com> wrote:
> So (Paul Lathrop will correct me if I'm wrong, I hope), it looks like
> we decided on the following simplistic metadata for modules:
>
> * version
> * name
> * author
> * source (i.e., a URL)
> * requires

* license
* platforms

> The requires need to be able to specify either names or names and
> version.

I put version comparators in the "next version" bucket. So, for now,
you should be able to specify "requires MODULE" or "requires MODULE
version VERSION" but not "requires MODULE version greater than
VERSION"

> We also decided that the data will be stored in a file named something
> like 'metadata.json', and the file format will be json.

I'd say we settled on metadata.json, that json is what we will support
at first, but clearly that leaves it open to support other formats
later.

> My plan at this point is to do the requires as either an array, or an
> array of arrays:
>
>   requires: foo, bar
>   requires: [foo, 1.0], [bar, 2.0]
>
> Comments?
>
> I'm partway to a first version, anyway.

You are a machine! I was totally going to hack up a first pass at a
tool this evening. Ah well, c'est la vie.

--Paul

[Paul Lathrop]

On Thu, Oct 1, 2009 at 4:08 PM, R.I.Pienaar <r...@devco.net> wrote:
> Can you tell those who aren't at the camp what the useage of this meta data will be, will this for dependencies between modules somehow or is it merely a compile time aid, or something else completely?

At the moment, the meta data will be used for whatever you use it for.
At our conversation we basically all agreed that a standard for
metadata was a good foundation upon which to build tools and features.
We need to have data there before we start figuring out the best ways
to use it.

> I assume all the data will be available in some vars for all classes in teh module as well?  I for one would love a $modulename var that's the same in all sub classes.

Again, there is not (to my knowledge) a plan to pass this information
into Puppet as yet. Certainly that sounds like a good feature request.

--Paul

[Paul Lathrop]

On Thu, Oct 1, 2009 at 4:13 PM, Paul Lathrop <paul.l...@gmail.com> wrote:
> Again, there is not (to my knowledge) a plan to pass this information
> into Puppet as yet. Certainly that sounds like a good feature request.

Correction. I watched Luke get a fair amount of this integration
hacked up yesterday. Should be awesome.

Whoever @puppetizer is on Twitter brought up a good point - we need to
have Puppet version as one of the very first metadata things we need
to support. I also think we should have a "metadata version" metadata
tag. Yes, I know, we're getting meta-meta here, but it seems like an
important piece of information.

--Paul

[Luke Kanies]

On Oct 1, 2009, at 4:08 PM, R.I.Pienaar wrote:

>
> hello,
>
> ----- "Luke Kanies" <lu...@madstop.com> wrote:
>
>> So (Paul Lathrop will correct me if I'm wrong, I hope), it looks like
>>
>> we decided on the following simplistic metadata for modules:
>>
>> * version
>> * name
>> * author
>> * source (i.e., a URL)
>> * requires
>>
>> The requires need to be able to specify either names or names and
>> version.
>>
>> We also decided that the data will be stored in a file named
>> something
>>
>> like 'metadata.json', and the file format will be json.
>>
>> My plan at this point is to do the requires as either an array, or an
>>
>> array of arrays:
>>
>> requires: foo, bar
>> requires: [foo, 1.0], [bar, 2.0]
>>
>> Comments?
>
> Can you tell those who aren't at the camp what the useage of this
> meta data will be, will this for dependencies between modules
> somehow or is it merely a compile time aid, or something else
> completely?

It will be a first step toward adding the ability to talk about
dependencies between modules, which would be checked at compile time.
We'd also add, eventually, tools for doing general validation outside
of a Puppet compile.

>
> I assume all the data will be available in some vars for all classes
> in teh module as well? I for one would love a $modulename var
> that's the same in all sub classes.

I hadn't thought of that, but it's a good point. I know the
$modulename thing is already a ticket, and the others shouldn't be too
hard, either.

--
The chief lesson I have learned in a long life is that the only way to
make a man trustworthy is to trust him; and the surest way to make him
untrustworthy is to distrust him and show your distrust.
-- Henry L. Stimson

[Luke Kanies]

On Oct 1, 2009, at 4:11 PM, Paul Lathrop wrote:

>
> On Thu, Oct 1, 2009 at 3:57 PM, Luke Kanies <lu...@madstop.com> wrote:
>> So (Paul Lathrop will correct me if I'm wrong, I hope), it looks like
>> we decided on the following simplistic metadata for modules:
>>
>> * version
>> * name
>> * author
>> * source (i.e., a URL)
>> * requires
>
> * license
> * platforms

Got it.

>
>> The requires need to be able to specify either names or names and
>> version.
>
> I put version comparators in the "next version" bucket. So, for now,
> you should be able to specify "requires MODULE" or "requires MODULE
> version VERSION" but not "requires MODULE version greater than
> VERSION"

Right - I've got direct version equality, but that's it.

>
>> We also decided that the data will be stored in a file named
>> something
>> like 'metadata.json', and the file format will be json.
>
> I'd say we settled on metadata.json, that json is what we will support
> at first, but clearly that leaves it open to support other formats
> later.

Yep.

>
>> My plan at this point is to do the requires as either an array, or an
>> array of arrays:
>>
>> requires: foo, bar
>> requires: [foo, 1.0], [bar, 2.0]
>>
>> Comments?
>>
>> I'm partway to a first version, anyway.
>
> You are a machine! I was totally going to hack up a first pass at a
> tool this evening. Ah well, c'est la vie.

I didn't get it done yesterday, but maybe today.

--
This space intentionally has nothing but text explaining why this
space has nothing but text explaining that this space would otherwise
have been left blank, and would otherwise have been left blank.

[Dan Bode]

I wanted to toss a possible use case into the mix:

If puppet had some internal knowledge of module versions, it could provide puppet events some awareness about when puppet source code changes. 

This could be useful in differentiating between events that occurred as a result of a new version of a module versus random events (events not related to source code changes in the module are probably more important)

Anyways, I will open a ticket for this as a feature request if one does not exist already.

regards,

Dan

[Luke Kanies]

On Oct 4, 2009, at 5:17 PM, Dan Bode wrote:

> I wanted to toss a possible use case into the mix:
>
> If puppet had some internal knowledge of module versions, it could
> provide puppet events some awareness about when puppet source code
> changes.
>
> This could be useful in differentiating between events that occurred
> as a result of a new version of a module versus random events
> (events not related to source code changes in the module are
> probably more important)

It would, at the least, be a bit of metadata you could add to the
configuration changes, in addition to the overall repository version
(which would begin to have a bit less meaning if people are keeping
most modules outside the main repo).

You're more talking about a three-way merge operation (current state,
desired state, previous state), which also needs to be supported but
(I think) is a different feature.

>
> Anyways, I will open a ticket for this as a feature request if one
> does not exist already.

That'd be great.

--
It's a small world, but I wouldn't want to paint it.
-- Stephen Wright

[David Schmitt]

Luke Kanies wrote:
> On Oct 4, 2009, at 5:17 PM, Dan Bode wrote:
>
>> I wanted to toss a possible use case into the mix:
>>
>> If puppet had some internal knowledge of module versions, it could
>> provide puppet events some awareness about when puppet source code
>> changes.
>>
>> This could be useful in differentiating between events that occurred
>> as a result of a new version of a module versus random events
>> (events not related to source code changes in the module are
>> probably more important)
>
> It would, at the least, be a bit of metadata you could add to the
> configuration changes, in addition to the overall repository version
> (which would begin to have a bit less meaning if people are keeping
> most modules outside the main repo).
>
> You're more talking about a three-way merge operation (current state,
> desired state, previous state), which also needs to be supported but
> (I think) is a different feature.

From my understanding, Dan wants to distinguish in his reports between
events that were caused by a change in the manifest and events that were
caused because something (or someone) happened to the system (like a
service dying, or a manual change by an unenlightened individual).

(Ab-)Using the module version for this is a cheap way to get 99% there.
the last 1%, that is changes that were made immediately before the
puppet run with the new module version, can probably only caught if you
also have the last applied catalog too. Then you can see whether the
current state is what was applied in the last run.

Regards, DavidS

[Luke Kanies]

The module version doesn't help you there, any more than the
configuration version in 0.25 does, because all you know is what part
of the config says a given fix should happen, not whether that config
has changed or the local state has.

--
Think twice before you speak, and then you may be able to say
something more insulting than if you spoke right out at
once. -- Evan Esar

[David Schmitt]

Without wanting to go into to much local details, Dan has quite the
intelligent report processor, so he should be able to tell whether his
config has changed. Which should give you the "99%" I spoke of.

You seem to imply that that's already possible with 0.25? I seem do have
missed that. Any pointers to docs?

Regards, DavidS

[Luke Kanies]

What are these 'docs' you mention? :)

http://reductivelabs.com/trac/puppet/wiki/ConfigurationReference

Look for 'config_version'.

It should have been mentioned in the change log and release notes and
stuff.

Basically, you tell Puppet how to get the version of your config
repository (note it's the whole repo, not per-module), and that
version gets attached to every resource and every log.

--
It is said that power corrupts, but actually it's more true that power
attracts the corruptible. The sane are usually attracted by other things
than power. -- David Brin

[David Schmitt]

Ah, yes O:-)

> It should have been mentioned in the change log and release notes and
> stuff.

It's neither mentioned in the release notes
(http://reductivelabs.com/trac/puppet/wiki/ReleaseNotes) nor the
CHANGELOG (http://projects.reductivelabs.com/projects/puppet/changelog).
I've checked now.

> Basically, you tell Puppet how to get the version of your config
> repository (note it's the whole repo, not per-module), and that
> version gets attached to every resource and every log.

If I read the docs correctly, it's just output from a script, so one can
get any local "release state" tag/version which makes sense locally.

Regards, DavidS

[Luke Kanies]

Hrm. This must be my fault.

>
>> Basically, you tell Puppet how to get the version of your config
>> repository (note it's the whole repo, not per-module), and that
>> version gets attached to every resource and every log.
>
> If I read the docs correctly, it's just output from a script, so one
> can
> get any local "release state" tag/version which makes sense locally.

Exactly.

--
However beautiful the strategy, you should occasionally look at the
results. -- Sir Winston Churchill

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luke Kanies wrote:
>>>> You seem to imply that that's already possible with 0.25? I seem do
>>>> have
>>>> missed that. Any pointers to docs?
>>> What are these 'docs' you mention? :)
>>>
>>> http://reductivelabs.com/trac/puppet/wiki/ConfigurationReference
>>>
>>> Look for 'config_version'.
>> Ah, yes O:-)
>>
>>> It should have been mentioned in the change log and release notes and
>>> stuff.
>> It's neither mentioned in the release notes
>> (http://reductivelabs.com/trac/puppet/wiki/ReleaseNotes) nor the
>> CHANGELOG (http://projects.reductivelabs.com/projects/puppet/
>> changelog).
>> I've checked now.
>
> Hrm. This must be my fault.

Had an inkling this feature existed - I think we very briefly
discussed it once but had no idea how it works. Can someone add
something to a wiki page? Perhaps the version control one? And add
a section to the ReleaseNotes page please?

Or tell me enough about it so I can.

Regards

James Turnbull

- --
Author of:
* Pro Linux Systems Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBSsuh3iFa/lDkFHAyAQKU+Af/WoNYkKY9Bd+I18zGhdKUuhCIODBmoxoJ
Uf3hRKgXFGlvuGyDihmh8hWSl7UqNAM3i22nZwHcFpD9Muj0hcUkx+Tk0/B+oDKE
zEwa2bxT1gwWOenUbtNx/xwQLfugj5oOfQzkuOqiXNPlK3CiHVMFsYbBsr4tFzbW
oIRDXBQX2pg3sMe6DLq5+gI0eJBl2OBv9ODkFc+MN/yvvi0/fyoaovchLOWJd4Zi
pKxDklixwz6DGFMzQ00UyAJRwiMkisp+nA+oet6xtLNSxgiUp/U+rBPfybk3BUtI
0Z1pLE60GctZBa/VihMSe6tikE3kY/tbozWpSGFn1uvcPErCjUvvAg==
=50LV
-----END PGP SIGNATURE-----

[Luke Kanies]

It's very straightforward - you provide a command in the
'config_version' setting, and Puppet runs that command at compile time
to determine the config version. This version then gets attached to
the catalog and, by extension, to all resources. When these resources
create logs, the version is attached to the log instances (not in
syslog, but the actual objects in memory, which get serialized and
sent in the report to the server).

So, basically, you now have a way to correlate every log message you
receive to a specific configuration version. You just have to define
what 'configuration version' means to you.

If you don't provide a command, then the compile time is used.

--
A classic is something that everybody wants to have read and nobody
wants to read. -- Mark Twain

[James Turnbull]

2009/10/7 Luke Kanies <lu...@madstop.com>:

>
> On Oct 6, 2009, at 1:00 PM, James Turnbull wrote:
>
>>
> It's very straightforward - you provide a command in the
> 'config_version' setting, and Puppet runs that command at compile time
> to determine the config version.  This version then gets attached to
> the catalog and, by extension, to all resources.  When these resources
> create logs, the version is attached to the log instances (not in
> syslog, but the actual objects in memory, which get serialized and
> sent in the report to the server).

So each resource is tagged with this "version"?

> So, basically, you now have a way to correlate every log message you
> receive to a specific configuration version.  You just have to define
> what 'configuration version' means to you.

Is "version" expected to be in some format? Any restricted characters, etc?

> If you don't provide a command, then the compile time is used.

What if the command fails?

Regards

James Turnbull

[Luke Kanies]

On Oct 6, 2009, at 2:47 PM, James Turnbull wrote:

>
> 2009/10/7 Luke Kanies <lu...@madstop.com>:
>>
>> On Oct 6, 2009, at 1:00 PM, James Turnbull wrote:
>>
>>>
>> It's very straightforward - you provide a command in the
>> 'config_version' setting, and Puppet runs that command at compile
>> time
>> to determine the config version. This version then gets attached to
>> the catalog and, by extension, to all resources. When these
>> resources
>> create logs, the version is attached to the log instances (not in
>> syslog, but the actual objects in memory, which get serialized and
>> sent in the report to the server).
>
> So each resource is tagged with this "version"?

Yes, although it's an attribute, not a tag.

>
>> So, basically, you now have a way to correlate every log message you
>> receive to a specific configuration version. You just have to define
>> what 'configuration version' means to you.
>
> Is "version" expected to be in some format? Any restricted
> characters, etc?

Nope; opaque string.

>
>> If you don't provide a command, then the compile time is used.
>
> What if the command fails?

Erm. Bad stuff?

--
Ninety-eight percent of the adults in this country are decent,
hard-working, honest Americans. It's the other lousy two percent that
get all the publicity. But then--we elected them. --Lily Tomlin

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luke Kanies wrote:
> On Oct 6, 2009, at 2:47 PM, James Turnbull wrote:
>
>> 2009/10/7 Luke Kanies <lu...@madstop.com>:
>>> On Oct 6, 2009, at 1:00 PM, James Turnbull wrote:
>>>
>>> It's very straightforward - you provide a command in the
>>> 'config_version' setting, and Puppet runs that command at compile
>>> time
>>> to determine the config version. This version then gets attached to
>>> the catalog and, by extension, to all resources. When these
>>> resources
>>> create logs, the version is attached to the log instances (not in
>>> syslog, but the actual objects in memory, which get serialized and
>>> sent in the report to the server).
>> So each resource is tagged with this "version"?

How's this:

http://reductivelabs.com/trac/puppet/wiki/ReleaseNotes#configuration-versioning

Regards

James Turnbull

- --

Author of:
* Pro Linux Systems Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBSsw1RSFa/lDkFHAyAQIMPAf/c+bqQ+0rey6rXYWwPqhaFp1fZMPK82TY
OGq/tPRDuJ6P91U1qfWvgHc56HZdgmudDLls0GdLGUDNNsMqLoDJcnC6RGldFHt+
ef+GNYqhJrhHSMDqaO+B79C2cKLlKIaz+J08xwPXNrm0yMuN3h579uf5e3lMbIAU
PJk8xJDD1cdH9WU+O4qU0VleL2Dn791xH8Gnqk2lrOWpArKZs/05HtByqVBDeEOr
Cey2iprCQnTybFOADGiEtf3isUiViVv95GXTuAViRJaYAwsQ1NkvIqXOgTDfKG7C
7IIqSQt/Vevkqcy+DOhuJ4WCgw4wcDZK/+tWiYCghCH00ukn3LzAWw==
=44Fd
-----END PGP SIGNATURE-----

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

James Turnbull wrote:
> http://reductivelabs.com/trac/puppet/wiki/ReleaseNotes#configuration-versioning
>

Also sent to:

http://reductivelabs.com/trac/puppet/wiki/PuppetVersionControl

Regards

James Turnbull

- --
Author of:
* Pro Linux Systems Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBSsw6JCFa/lDkFHAyAQJeqggAl3jtugkdVM0nGQLKHF7YsDYp5ccbpn8G
FRVjUbj3RwUvJxw710iYF470I67zui5zeWw6UJTmmZ5gC4ACH5rdtBCeIx6RJoLl
Lf3rRTU3TD3CCA0++KhwQGvh4JhQYAFgexmlxjd22P435Piac7rzFan1SCc7fq/a
oYq32gvkYYiBqJH/hZ+4cFwH+gQ9RN6yduKT3t7BSsMHHN80ao92vNO2aa2QmW6o
X09cfOcFJuVGAsPY5+UF5gr+AhDzStK8eNeL3Y3f74354kw0dJ/1mMM4Q3KH04HX
gDo+pvWCuoU+eqBIbvxk0Y7hHFTK+xYuT8I04KaG/zsT4BO59PARCg==
=QmrI
-----END PGP SIGNATURE-----

[Peter Meier]

Hi

in general I like the idea of the metadata very much and it's great to
see progress in this area. However I see one issue:

>>> My plan at this point is to do the requires as either an array, or an
>>> array of arrays:
>>>
>>> requires: foo, bar
>>> requires: [foo, 1.0], [bar, 2.0]
>>>
>>> Comments?
>>>
>>> I'm partway to a first version, anyway.

What about all the forks? Many people provide their modules as a git
repository and people fork them and start hacking their own changes.
With the linear model of version numbers we don't have any possibility
to differentiate between a module foo that have been forked from 2.0 and
evolved to 2.1 and the original module that might have been improved to
version 2.1 . If you look at the current reality there are plenty of
such modules.

For sure the main goal to fix that is the common modules project,
however if we introduce a formal description which modules work together
or require another in a certain version we should be able to distinguish
forks as well. Otherwise we might end up with a lot of people who take
module foo from x in version 1, which requires module bar in version 2.1
but instead to take the actual version from x they take the fork from y.
So somehow we need to be able to the actual version.

cheers pete

[Luke Kanies]

On Oct 6, 2009, at 11:50 PM, James Turnbull wrote:

>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> James Turnbull wrote:
>> http://reductivelabs.com/trac/puppet/wiki/ReleaseNotes#configuration-versioning
>>
>
> Also sent to:
>
> http://reductivelabs.com/trac/puppet/wiki/PuppetVersionControl

Looks good.

--
I can't understand why a person will take a year to write a novel when
he can easily buy one for a few dollars. -- Fred Allen

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luke Kanies wrote:
> On Oct 6, 2009, at 11:50 PM, James Turnbull wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> James Turnbull wrote:
>>> http://reductivelabs.com/trac/puppet/wiki/ReleaseNotes#configuration-versioning
>>>
>> Also sent to:
>>
>> http://reductivelabs.com/trac/puppet/wiki/PuppetVersionControl

Can we confirm what happens if the command fails? Does it fall back
and ignore config_version value?

I can test if you don't know answer off the top of your head.

Regards

James Turnbull

- --
Author of:
* Pro Linux Systems Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBSszsPSFa/lDkFHAyAQLzMggAshaRgPvAQ/c42BpBvLh93nPhNKy1glmz
X2X60qUiJSesiBjm5nXdxAArVcHBbhAh5hZVz8XIEsMi2zEyrWvLlJhLickiVxGx
BQq1GS2kLomcF5dN4JnxvpdkOgP8/9j52afsMD1+vzcuavCoTEqUjdVMKRiMC4QW
yuSMlq4yHHt+IRUj89kRaRsqPzZFJtL+aimN3tdwsMcY8odYgm/DcMLPbI9AXrY0
f9hOtEgbW01bsRAGeFXwS+Dn0osvYuNxhDHQrvmHeand9GUwZygEf4ka0SPQi9J+
MKRcJeSWLuO5fRp/rbrqBE24hMXBhkcDEDiSNcsfKaSckrx/DScuwA==
=wTmK
-----END PGP SIGNATURE-----

[Luke Kanies]

On Oct 7, 2009, at 12:30 PM, James Turnbull wrote:

>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Luke Kanies wrote:
>> On Oct 6, 2009, at 11:50 PM, James Turnbull wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> James Turnbull wrote:
>>>> http://reductivelabs.com/trac/puppet/wiki/ReleaseNotes#configuration-versioning
>>>>
>>> Also sent to:
>>>
>>> http://reductivelabs.com/trac/puppet/wiki/PuppetVersionControl
>
> Can we confirm what happens if the command fails? Does it fall back
> and ignore config_version value?
>
> I can test if you don't know answer off the top of your head.

I don't know the answer off the top of my head, but I just looked at
the code and it looks like it's a pretty naive implementation. That
should probably be fixed. I'll open a ticket for it.

--
Zeilinger's Fundamental Law:
There is no Fundamental Law.

[Luke Kanies]

Any suggestions for this?

I haven't been worrying too much about parallel versions in forks
because 1) I'm assuming that people will only ever have a single
version installed at a time and 2) Puppet can only deal with a single
module with a given name, so there isn't much opportunity for conflict.

In the end, we can only usefully talk about dependencies within a
subset of modules that exist, and my assumption is that we'll
generally have two subsets we operate on: The modules installed, and
the modules available in whatever repositories we're looking at. It
should be straightforward to manage forks (or not support them) within
that subset.

That being said, I'm happy to find a solution here, but I'm not sure
it fits under the "simplest solution that provides a benefit", which
is what I was shooting for initially.

--
Wear the old coat and buy the new book. -- Austin Phelps

[Peter Meier]

>> What about all the forks? Many people provide their modules as a git
>> repository and people fork them and start hacking their own changes.
>> With the linear model of version numbers we don't have any possibility
>> to differentiate between a module foo that have been forked from 2.0
>> and
>> evolved to 2.1 and the original module that might have been improved
>> to
>> version 2.1 . If you look at the current reality there are plenty of
>> such modules.
>>
>> For sure the main goal to fix that is the common modules project,
>> however if we introduce a formal description which modules work
>> together
>> or require another in a certain version we should be able to
>> distinguish
>> forks as well. Otherwise we might end up with a lot of people who take
>> module foo from x in version 1, which requires module bar in version
>> 2.1
>> but instead to take the actual version from x they take the fork
>> from y.
>> So somehow we need to be able to the actual version.
>
> Any suggestions for this?

Not really. We would need something like a unique identifier to which we
could refer. On the other side it might be sufficient to mention the
refered module with source in the README and if people mix that up...

> I haven't been worrying too much about parallel versions in forks
> because 1) I'm assuming that people will only ever have a single
> version installed at a time and 2) Puppet can only deal with a single
> module with a given name, so there isn't much opportunity for conflict.

more or less yeah, but for example if I look to all the apache modules
out there for example I think that quite everyone is completely different.

> In the end, we can only usefully talk about dependencies within a
> subset of modules that exist, and my assumption is that we'll
> generally have two subsets we operate on: The modules installed, and
> the modules available in whatever repositories we're looking at. It
> should be straightforward to manage forks (or not support them) within
> that subset.

that should be fine.

> That being said, I'm happy to find a solution here, but I'm not sure
> it fits under the "simplest solution that provides a benefit", which
> is what I was shooting for initially.

I don't think it have to be in the very first version and I think we
should start without this issue. But I think some time we should try to
find a good solution for it.

cheers pete

[Luke Kanies]

True; and a basic 'forked-from' attribute wouldn't be bad, I guess.

>
>> I haven't been worrying too much about parallel versions in forks
>> because 1) I'm assuming that people will only ever have a single
>> version installed at a time and 2) Puppet can only deal with a single
>> module with a given name, so there isn't much opportunity for
>> conflict.
>
> more or less yeah, but for example if I look to all the apache modules
> out there for example I think that quite everyone is completely
> different.

I think that's mostly a namespace collision problem - multiple people
making competing modules. That will sort itself out over time as we
have central repositories people can refer to.

>
>> In the end, we can only usefully talk about dependencies within a
>> subset of modules that exist, and my assumption is that we'll
>> generally have two subsets we operate on: The modules installed, and
>> the modules available in whatever repositories we're looking at. It
>> should be straightforward to manage forks (or not support them)
>> within
>> that subset.
>
> that should be fine.
>
>> That being said, I'm happy to find a solution here, but I'm not sure
>> it fits under the "simplest solution that provides a benefit", which
>> is what I was shooting for initially.
>
> I don't think it have to be in the very first version and I think we
> should start without this issue. But I think some time we should try
> to
> find a good solution for it.

I agree. I think a lot of it will get sorted out as people start to
use this minimal behaviour.

--
Always behave like a duck - keep calm and unruffled on the surface but
paddle like the devil underneath. -- Jacob Braude

[David Lutterkort]

On Thu, 2009-10-01 at 15:57 -0700, Luke Kanies wrote:
> So (Paul Lathrop will correct me if I'm wrong, I hope), it looks like
> we decided on the following simplistic metadata for modules:
>
> * version
> * name
> * author
> * source (i.e., a URL)
> * requires

I just looked through gem2rpm to see what metadata we'd need to
automatically generate RPM spec files from module metadata.

>From the above list we'd need name, version, and requires; we'd need two
URL's: what RPM calls 'URL', a pointer to some project page (where
humans would go) and a Source URL, which points to a tarball or similar
of a released module.

In addition:

* Summary: a one line summary of what the module does
* Description: a longer, human readable description of the module
* Required puppet version (this could eventually sprout into
dependencies on specific versions of other packages outside of
puppet if you know your module will trigger bugs in them, e.g.
ruby-selinux >= $version, though that's not needed now; it will
become an issue for modules that contain plugins)
* List of files in the module with an indication of whether they
are 'normal', docs or config - since for now, modules only
contain 'normal' files, besides the license, that can be added
in the future
* Standardize the name of the license file (COPYING or LICENSE)
* a License tag that indicates what license it is. For RPM, it
would be easiest if the values for that tag were directly the
ones from the Short Names column in [1] - if other short names
are used, we'd need to map them, which isn't that big a deal.
But that list needs to be documented on the Wiki.

We should also settle on a recommendation for where to put 3rd party
modules. I cleverly added /usr/share/puppet/modules[2] to the default
module search path way back when - that's what I would use as the
recommended place for 3rd party modules; the generated spec file can
easily place files there.

David

[1] https://fedoraproject.org/wiki/Licensing#Good_Licenses
[2] I am assuming that we will never ever have modules with arch
specific content, i.e. with binaries that need to be built from source

[Luke Kanies]

This shouldn't be that hard to get into the system; I'll do so next
week and resubmit the code.

--
I don't want the world, I just want your half.

[Luke Kanies]

On Oct 8, 2009, at 11:08 AM, David Lutterkort wrote:

I've added some of these to my 'feature/master/module_metadata'
branch, but I didn't add the license tag/fixed name information, nor
have I added the file list.

How would we manage the list of allowed licenses? The goal is to
provide consistency, right, not just restrict what license people can
use?

So do we just have an array somewhere that we update when we discover
a new license we want to use? Or can we seed it with the valid
licenses in RPM?

As to the file list, I'm not really ready to do that. I think the
right answer is some easy way to generate it from git or ruby; having
a fixed file list is just fugly.

--
Experience is that marvelous thing that enables you recognize a
mistake when you make it again. -- F. P. Jones

[David Lutterkort]

On Wed, 2009-10-14 at 17:36 -0700, Luke Kanies wrote:
> How would we manage the list of allowed licenses? The goal is to
> provide consistency, right, not just restrict what license people can
> use?

It doesn't have to be supported by tooling; a Wiki page similar to the
one that Fedora has (or just pointing to that) would be good enough.

> So do we just have an array somewhere that we update when we discover
> a new license we want to use? Or can we seed it with the valid
> licenses in RPM?

The fixed license short names are a convention; if you use an unknown
license, you can still build the RPM. Some of the rpm checking tools
like rpmlint will complain though - I don't think anything like that
needs to be there for puppet modules, as long as the convention is
followed by people (and the ones not following will get a polite
reminder from whoever packages the module as an RPM)

> As to the file list, I'm not really ready to do that. I think the
> right answer is some easy way to generate it from git or ruby; having
> a fixed file list is just fugly.

Yeah, I mostly listed that for completeness - it's a very low priority,
as long a we can easily tell documentation files from 'normal' files
because of their name (again, based on conventions)

David

[Luke Kanies]

Sounds like we're mostly done, then. I think we're going to want a
lot of tooling around this, too, but at least for now we've got
something we can try.

--
There is only one success - to be able to spend your life in your own
way. -- Christopher Morley