Facter version guarantees

[Kylo Ginsberg]

[Splitting this out of the original thread which was the Facter 2.2.0 announcement.]

On Thu, Aug 28, 2014 at 2:58 AM, David Schmitt <da...@dasz.at> wrote:

On 2014-08-28 00:51, Kylo Ginsberg wrote:

But going forward there's a question about how to handle changes to fact
*values*. One proposal is that we identify (and of course test against)
some essential facts that we "care a lot about" (such as
'lsbmajdistrelease") and set some rules, like:

(a) we do not change those in x.y.Z releases
(b) we highlight it when they DO change in x.Y or X releases

Strict semver would suggest that (major) changes to values be marked with an increment in the major version.

There are a couple really good issues here, and I'm torn on the answers:

1) While semver clearly applies to the Facter *API*, I'm not sure if it provides guidance on Facter *values*. Yes, on the one hand, I could argue that the values are part of the Facter contract, but that seems like a contortion of "API". I incline toward treating semver as API-only.

2) More importantly, a strict reading of semver as applied to facts would tie our hands (or force more rapid majors). E.g. I add fact 'foo' in 3.0, and it turns out to work great on RHEL but is incoherent on, say, Windows. But it's now out in the wild, so if I'm completely strict about changes we have to leave it be until 4.0, although that may mean that more and more Windows users painfully write manifests around the 3.0 behavior.

It's roughly (hand wave) that sort of consideration that led to the idea floated above of define some "major" facts (and platforms) where we make guarantees.

To tie (1) and (2) together, there's a case to be made for separating Facter's API guarantees from its fact value guarantees, and perhaps even to tiering the latter. The exact details are up for discussion of course.

Thoughts?

(And regardless, we should add a VERSIONING.md to the facter project so we have a thought-out statement on the subject.)

Kylo

--

Kylo Ginsberg

ky...@puppetlabs.com

Join us at PuppetConf 2014, September 20-24 in San Francisco

Register by September 8th to take advantage of the Final Countdown —save $149!

[Eric Shamow]

Maybe worth considering a separate level of semver/facter guarantees around facts shipped with core Facter and Facter as an API.

As in, you have Facter 2.2.0 with Facter Core Set 1.0.3 or some such. Make it more like stdlib, where you can iterate more quickly on the facts themselves than the framework?

-Eric

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALsUZFEAkxiWg70mH7cTu3TbqSd39PwGRrr0hW3invBkC6Vrjw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Wil Cooley]

On Fri, Aug 29, 2014 at 2:11 PM, Kylo Ginsberg <ky...@puppetlabs.com> wrote:

 

There are a couple really good issues here, and I'm torn on the answers:

1) While semver clearly applies to the Facter *API*, I'm not sure if it provides guidance on Facter *values*. Yes, on the one hand, I could argue that the values are part of the Facter contract, but that seems like a contortion of "API". I incline toward treating semver as API-only.

From the perspective of direct clients of Facter's Ruby API, such as Puppet itself, the values might not be part of the API. (But then again they might -- what if the change broke a provider constraint?)

From the perspective of someone writing manifests, Facter values are part of the API of the Puppet DSL.  From this perspective, Facter's Ruby API is meaningless -- only the existence of facts and values are meaningful. Someone writing a module expects to be able to count on both the existence and the values of core facts.

 

2) More importantly, a strict reading of semver as applied to facts would tie our hands (or force more rapid majors). E.g. I add fact 'foo' in 3.0, and it turns out to work great on RHEL but is incoherent on, say, Windows. But it's now out in the wild, so if I'm completely strict about changes we have to leave it be until 4.0, although that may mean that more and more Windows users painfully write manifests around the 3.0 behavior.

It would depend on how badly broken it is on a particular platform -- If it's so broken as to be unusable, then changing it for that platform should not be a problem. But if it's usable with work arounds or more effort than really needed, then you're probably suck with that behavior. I realize there's plenty of room for interpretation there.

Of course, a strict reading of SemVer reveals that it only applies to APIs -- Not environments, user interfaces, platforms, etc. And that it is not clear about how it applies to dependencies. So, if, for example, you were the *user* of a piece of software and if one of the libraries that software depended on increased a minimum version of one of *its* dependencies, such that the whole thing ceased to work on, say, the version of a language interpreter that ships with your OS, well, that's ok and you're out of luck for several reasons. I found this out myself: https://github.com/gollum/gollum-lib/issues/44

Wil

[Henrik Lindberg]

On 2014-29-08 23:11, Kylo Ginsberg wrote:
> [Splitting this out of the original thread which was the Facter 2.2.0
> announcement.]
>
> On Thu, Aug 28, 2014 at 2:58 AM, David Schmitt <da...@dasz.at
> <mailto:da...@dasz.at>> wrote:
>
> On 2014-08-28 00:51, Kylo Ginsberg wrote:
>
> But going forward there's a question about how to handle changes
> to fact
> *values*. One proposal is that we identify (and of course test
> against)
> some essential facts that we "care a lot about" (such as
> 'lsbmajdistrelease") and set some rules, like:
>
> (a) we do not change those in x.y.Z releases
> (b) we highlight it when they DO change in x.Y or X releases
>
>
>
> Strict semver would suggest that (major) changes to values be marked
> with an increment in the major version.
>
>
> There are a couple really good issues here, and I'm torn on the answers:
>
> 1) While semver clearly applies to the Facter *API*, I'm not sure if it
> provides guidance on Facter *values*. Yes, on the one hand, I could
> argue that the values are part of the Facter contract, but that seems
> like a contortion of "API". I incline toward treating semver as API-only.
>

API is really what we define it to be.
IMO, values should be included. We need to specify what a fact means -
what is is measuring, and ideally how to verify that it measures that.
We also need to specify for which platforms this applies, and for which
platforms the specification is undefined / not applicable /
experimental, or some other such status to indicate that a change can
occur at any time without breaking semver.

IMO we should also type the facts at the level of detail that the Puppet
Type system allows things to be typed (i.e. not just that it is a
string, but also that it is empty / non-empty, an enumeration, a value
range, must match a pattern etc. This is especially important if the
detailed type of the value varies between platforms.

> 2) More importantly, a strict reading of semver as applied to facts
> would tie our hands (or force more rapid majors). E.g. I add fact 'foo'
> in 3.0, and it turns out to work great on RHEL but is incoherent on,
> say, Windows. But it's now out in the wild, so if I'm completely strict
> about changes we have to leave it be until 4.0, although that may mean
> that more and more Windows users painfully write manifests around the
> 3.0 behavior.
>

Surely, there must be some tests that validates the intended behavior.
If there are none, the the behavior is unspecified, and *that* should be
specified. Having done so entitles someone to change it on that platform.

> It's roughly (hand wave) that sort of consideration that led to the idea
> floated above of define some "major" facts (and platforms) where we make
> guarantees.
>

Yes, but they all need to be specified.

> To tie (1) and (2) together, there's a case to be made for separating
> Facter's API guarantees from its fact value guarantees, and perhaps even
> to tiering the latter. The exact details are up for discussion of course.
>
> Thoughts?
>

They were posted above :-)

- henrik

> (And regardless, we should add a VERSIONING.md to the facter project so
> we have a thought-out statement on the subject.)
>
> Kylo
>
> --
> Kylo Ginsberg

> ky...@puppetlabs.com <mailto:ky...@puppetlabs.com>
>
> *Join us at PuppetConf 2014 <http://www.puppetconf.com/>, September
> 20-24 in San Francisco*
> /Register by September 8th to take advantage of the Final Countdown
> <https://www.eventbrite.com/e/puppetconf-2014-tickets-7666774529?discount=FinalCountdown>
> //—//save $149!/

>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-dev+...@googlegroups.com

> <mailto:puppet-dev+...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-dev/CALsUZFEAkxiWg70mH7cTu3TbqSd39PwGRrr0hW3invBkC6Vrjw%40mail.gmail.com

> <https://groups.google.com/d/msgid/puppet-dev/CALsUZFEAkxiWg70mH7cTu3TbqSd39PwGRrr0hW3invBkC6Vrjw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[David Schmitt]

Hi,

I really like the points Eric, Wil and Henrik have brought up. Here're
some more from me.

On 2014-08-29 23:11, Kylo Ginsberg wrote:
> [Splitting this out of the original thread which was the Facter 2.2.0
> announcement.]
>
> On Thu, Aug 28, 2014 at 2:58 AM, David Schmitt <da...@dasz.at
> <mailto:da...@dasz.at>> wrote:
>
> On 2014-08-28 00:51, Kylo Ginsberg wrote:
>
> But going forward there's a question about how to handle changes
> to fact
> *values*. One proposal is that we identify (and of course test
> against)
> some essential facts that we "care a lot about" (such as
> 'lsbmajdistrelease") and set some rules, like:
>
> (a) we do not change those in x.y.Z releases
> (b) we highlight it when they DO change in x.Y or X releases
>
>
>
> Strict semver would suggest that (major) changes to values be marked
> with an increment in the major version.
>
>
> There are a couple really good issues here, and I'm torn on the answers:
>
> 1) While semver clearly applies to the Facter *API*, I'm not sure if it
> provides guidance on Facter *values*. Yes, on the one hand, I could
> argue that the values are part of the Facter contract, but that seems
> like a contortion of "API". I incline toward treating semver as API-only.

That's only semantic BS. The primary consumer of facter is puppet code
writers. If the meaning of return values of something like sprintf would
change, all hell would break loose. Why should it be different for the
return value of $operatingsystem ?

Semver's exclusion of UIs is due to the fact that users are much, *much*
more flexible to react to changes than deployed code.

> 2) More importantly, a strict reading of semver as applied to facts
> would tie our hands (or force more rapid majors). E.g. I add fact 'foo'
> in 3.0, and it turns out to work great on RHEL but is incoherent on,
> say, Windows. But it's now out in the wild, so if I'm completely strict
> about changes we have to leave it be until 4.0, although that may mean
> that more and more Windows users painfully write manifests around the
> 3.0 behavior.
>
> It's roughly (hand wave) that sort of consideration that led to the idea
> floated above of define some "major" facts (and platforms) where we make
> guarantees.
>
> To tie (1) and (2) together, there's a case to be made for separating
> Facter's API guarantees from its fact value guarantees, and perhaps even
> to tiering the latter. The exact details are up for discussion of course.

That is totally legitimate and starting a separate semver stream for the
values just underlines that they too are API (if also perhaps a
different set).

Something that can be done to ease the semver pressure is start
collecting recipes to avoid disaster when the inevitable glitch happens.
For example, if a certain fact produces garbage on a platform and fixing
it would require wide-spread changes, one could add a $FACT_ex fact that
returns the fixed values in a minor version and fold that change into
the normal fact on the next major bump. Or start a versioned facts hash
so that the consumer can choose which version to access. I suggest the
latter because even if the fact-value-semver is decoupled from facter's
semver, manifests will have to deal with a broad range of deployed
versions and having to keep all permutations and bugs straight won't be
easier, just because it now has a version.



Regards, David

--
* Always looking for people I can help with awesome projects *
G+: https://plus.google.com/+DavidSchmitt
Blog: http://club.black.co.at/log/
LinkedIn: http://at.linkedin.com/in/davidschmitt

[Daniele Sluijters]

Lets take is_virtual as an example. That fact, and all the boolean facts, have been strings since the dawn of time. During the first days of creation people wrote manifests that went if $is_virtual == 'false' {}.

By now, most if not all, DSL consumers are aware of the 'correct' way to do this, str2bool($::is_virtual). As long as str2bool() with a bool returns that bool I see no issue with changing all the boolean facts to be actual booleans, even in a .Z release. If you're still comparing it as a string, too bad, we've had a better way for over 2 years now.

What I mean is, I think there's a lot of facts we can type without breaking anything. More and more manifests validate their input using the validate_* and is_* functions in stdlib and leverage the set of 'conversion' functions we have in stdlib to transform input to their actual type.

I also think a lot of active module writers are well aware of the changes coming to Facter and Puppet and we shouldn't be too afraid to use that to our advantage. As long as it's clearly announced, early so people can roll a release with fixes in time for a Facter release, I doubt there's a need for strict semver adherence.

[Erik Dalén]

As a bit of a followup to this discussion I've created this pull request: https://github.com/puppetlabs/facter/pull/777

When fixing support for structured facts I noticed that the current data type bugginess in facter makes it almost unusable with PuppetDB 2.2. As you can't use the < or > operators on strings anymore.

IMO this is more of a bug fix even though it somewhat changes the API for Puppet manifests.

--

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/4cf8a0ff-5fed-4209-b843-e3c34c97d7a7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Erik Dalén

[Felix Frank]

On 08/30/2014 01:00 AM, Henrik Lindberg wrote:
> API is really what we define it to be.
> IMO, values should be included.

Agreed. Facter.value() is very generic and unlikely to undergo any
drastic changes soon (I assume). But it really forms the heart of the
API, as far as I'm concerned, and the values returned in response to
certain arguments are of great import.
If I take the opposite assumption (fact values are not part of the API)
to its extreme, then we could take the liberties of making arbitrary
changes to fact values independently of release versions.

On 08/30/2014 02:59 PM, Daniele Sluijters wrote:
> By now, most if not all, DSL consumers are aware of the 'correct' way
> to do this, str2bool($::is_virtual). As long as str2bool() with a bool
> returns that bool I see no issue with changing all the boolean facts
> to be actual booleans, even in a .Z release. If you're still comparing
> it as a string, too bad, we've had a better way for over 2 years now.

Well, yes, but there should really be a stern warning in the release
notes to that effect, so that users can audit their code for missing
wrappings in their Puppet code.

I don't think this should be a bugfix release, either - even though you
could consider the rampant maltyping to be bugs, then those still have
been introduced years ago, so the fix should be deferred to the next
feature release.

Best,
Felix

[Ken Barber]

> As a bit of a followup to this discussion I've created this pull request:
> https://github.com/puppetlabs/facter/pull/777
>
> When fixing support for structured facts I noticed that the current data
> type bugginess in facter makes it almost unusable with PuppetDB 2.2. As you
> can't use the < or > operators on strings anymore.

+1. Making the types numbers when they really are numbers will aid in
searching in PuppetDB and this is going to benefit a lot of people.
Not only that, they are often more optimal for storage - as they will
be stored as big integers in PostgreSQL versus a string of digits. The
indexes for these columns are more appropriate as well.

ken.

[John Bollinger]

On Saturday, August 30, 2014 3:10:14 AM UTC-5, David Schmitt wrote:

Hi,

I really like the points Eric, Wil and Henrik have brought up. Here're
some more from me.

On 2014-08-29 23:11, Kylo Ginsberg wrote:
> [Splitting this out of the original thread which was the Facter 2.2.0
> announcement.]
>
> On Thu, Aug 28, 2014 at 2:58 AM, David Schmitt <da...@dasz.at
> <mailto:da...@dasz.at>> wrote:
>
>     On 2014-08-28 00:51, Kylo Ginsberg wrote:
>
>
> 1) While semver clearly applies to the Facter *API*, I'm not sure if it
> provides guidance on Facter *values*. Yes, on the one hand, I could
> argue that the values are part of the Facter contract, but that seems
> like a contortion of "API". I incline toward treating semver as API-only.

That's only semantic BS. The primary consumer of facter is puppet code
writers. If the meaning of return values of something like sprintf would
change, all hell would break loose. Why should it be different for the
return value of $operatingsystem ?

Preach it, Brother!

Seriously, the spirit of semver is not well captured by splitting hairs over what actually constitutes the "API" to which it applies.  Semver is about using version numbering that gives users of the software reliable information regarding compatibility between versions, principally with a view toward how that software is used by other software.  Yes, Facter's Ruby API is an aspect of that, but certainly the fact values characteristic of various environments are, too.  The values are central to how Puppet manifests use Facter.

Alternatively, in the spirit of hair splitting, you could argue that because manifests do not use Facter directly, Facter has no responsibility toward them.  Instead, then, it's up to Puppet to maintain its own semver obligations toward manifests, PuppetDB, etc., including the fact variables characteristic of each environment that it exposes to those consumers.  No joy there.

I think the most interesting questions here are actually about bug compatibility and what constitutes bug.  I don't think the intention of semver includes requiring a major version bump to accompany every batch of bug fixes.  I'd argue that Facter providing a fact value inconsistent with that fact's definition certainly does constitute a bug -- or at least, it did at one time.  When that bug persists for so long that a significant body of code comes to rely on it, however, I don't think you can reasonably call it a bug any longer.

 

Semver's exclusion of UIs is due to the fact that users are much, *much*
more flexible to react to changes than deployed code.

> 2) More importantly, a strict reading of semver as applied to facts
> would tie our hands (or force more rapid majors). E.g. I add fact 'foo'
> in 3.0, and it turns out to work great on RHEL but is incoherent on,
> say, Windows. But it's now out in the wild, so if I'm completely strict
> about changes we have to leave it be until 4.0, although that may mean
> that more and more Windows users painfully write manifests around the
> 3.0 behavior.
>

I'll return here to the distinction between feature and bug: that's where I think you can find your wiggle room.  As much as there is to find, anyway.

 

> It's roughly (hand wave) that sort of consideration that led to the idea
> floated above of define some "major" facts (and platforms) where we make
> guarantees.
>

You actually do have some room there, too, in that you could claim -- either globally or on a per-fact basis -- that Facter's behavior is well-defined only on certain specified platforms.  But you can't just declare that retrospectively, not if indeed you were serious about semver in the first place.  And you also need to be prepared for the fact (no pun intended) that such a position is really just a hedge.  People will still be unhappy when fact values change for platforms not on the official support list, and I wouldn't blame them.

 

> To tie (1) and (2) together, there's a case to be made for separating
> Facter's API guarantees from its fact value guarantees, and perhaps even
> to tiering the latter. The exact details are up for discussion of course.

That is totally legitimate and starting a separate semver stream for the
values just underlines that they too are API (if also perhaps a
different set).

It is legitimate from the viewpoint of Facter's semver, but it doesn't really gain any ground outside the moral superiority realm.  You would end up with Puppet manifests depending on fact value semver instead of Facter semver, but you would still have the issue that you cannot change documented behavior without a major bump (in that case to the fact value semver), and you would still have the problem that any such change would break existing manifests.

Moreover, having one piece of software versioned simultaneously by two independent schemes would be a nightmare on many levels.  The only way I could imagine it working would be to split Facter into separate API and fact-evaluation pieces, one for each version number sequence.  Even then, I cannot imagine it working very well because modules will necessarily have direct dependencies on the fact-evaluation version.  You may think we have module compatibility problems now, but they're nothing compared to what we would have if you add the possibility of fact-evaluation-version incompatibility on top.

 

Something that can be done to ease the semver pressure is start
collecting recipes to avoid disaster when the inevitable glitch happens.
For example, if a certain fact produces garbage on a platform and fixing
it would require wide-spread changes,

I think that's a straw man.

If a fact implementation produces garbage on a particular system then that's a bug -- it can be fixed without a major bump.  In the meantime users will not usefully rely on that fact because it's garbage.  Alternatively, if a fact is not meaningful for a particular system, then it can be withdrawn altogether for that system without a major bump because it being computed at all for that system is a bug.  Users cannot usefully rely on such a fact, because that would imply that it is meaningful, contrary to the assumption.  Either way, the breadth of implementation changes that might be necessary to fix any particular issue is not a semver consideration.

John

[Michael Stahnke]

<snip>

That's only semantic BS. The primary consumer of facter is puppet code
writers. If the meaning of return values of something like sprintf would
change, all hell would break loose. Why should it be different for the
return value of $operatingsystem ?

Preach it, Brother!

Seriously, the spirit of semver is not well captured by splitting hairs over what actually constitutes the "API" to which it applies.  Semver is about using version numbering that gives users of the software reliable information regarding compatibility between versions, principally with a view toward how that software is used by other software.  Yes, Facter's Ruby API is an aspect of that, but certainly the fact values characteristic of various environments are, too.  The values are central to how Puppet manifests use Facter.

Alternatively, in the spirit of hair splitting, you could argue that because manifests do not use Facter directly, Facter has no responsibility toward them.  Instead, then, it's up to Puppet to maintain its own semver obligations toward manifests, PuppetDB, etc., including the fact variables characteristic of each environment that it exposes to those consumers.  No joy there.

I think the most interesting questions here are actually about bug compatibility and what constitutes bug.  I don't think the intention of semver includes requiring a major version bump to accompany every batch of bug fixes.  I'd argue that Facter providing a fact value inconsistent with that fact's definition certainly does constitute a bug -- or at least, it did at one time.  When that bug persists for so long that a significant body of code comes to rely on it, however, I don't think you can reasonably call it a bug any longer.

I totally agree with this. I'd consider facter values an API for getting values and making decisions with Puppet code (among other things).  The discussion John brings up about what's a bug is really the bigger issue here, since the behavior had been defined a different way for so long. Obviously this is the type of thing we try to avoid and unfortunately just missed it this time. 

I'd still like us to keep the truthiness in fact values as high (and consistent) as possible.