The host type has unexpected behavior and is not idempotent

[Eli Young]

As per PUP-3901, the host type has some serious issues.  Major issues with the current design:

1. The namevar:
   
   • It's currently the canonical hostname. This means that a hostname can be the canonical representation for at most 1 IP address.  This is a problem if, for example, you want to provide both IPv4 and IPv6 addresses for a hostname.
     
   • Changing it to be the IP address would mean that an IP address could have at most one canonical hostname associated with it.  This is less of an issue, but still not ideal.
     
   • Probably the best solution here is to change this to be both the IP address and the canonical hostname (e.g. "1.2.3.4/example.com"). However...
2. Parsing is flawed:
   
   • Multiple records with the same value for the namevar (currently the canonical hostname) overlap and only one is registered.  Modifying or removing records that overlap behaves inconsistently and, in the case of removal, requires multiple runs to achieve consistency.  Examples in the issue's description.
   • Changing the namevar to be the IP address or both the IP address and the canonical hostname could cause problems on Windows, where the number of hostname aliases per record is limited. This could be resolved by having the provider split a resource into multiple records in the file if the underlying system has alias count limits.

The other issues are all consequences of these two issues:

1. Inconsistent resource modification and removal (examples in the issue's description) is a result of namevar collision.
2. Removal of a hostname causing removal of all the aliases is more of a documentation issue than anything. So long as this is explicitly called out as expected behavior, it's not a problem.

As such, my proposed changes to the host type are to:

1. Change the generated resource namevar (and, by extension, the alias for specified resources) to use both the IP address and the canonical hostname.
2. Fix parsing to handle cases where multiple records specify the same namevar (which, after change #1, would be an IP address and canonical hostname) by merging them into a single resource.
3. Update documentation to to indicate that the hostname aliases are not first-class host items and that, when a hostname is removed, all aliases are removed too.  If the user wants to retain a hostname alias while removing a hostname, they'll need to put it into a different host resource.
   
4. To allow manifests to set relationships to hosts without knowing ahead of time what the IP address is, potentially provide resource aliases with titles set to the hostname and all the hostname aliases. Unfortunately, this runs into an issue when multiple host resources service the same hostname; blindly making resource aliases would result in each trying to alias to the same name, but conditionally aliasing based on if an alias already exists would result in a relationship attaching to different host resources depending on parse order.  This is a problem that I'm not sure how to solve.

Furthermore, since a resolution to this issue would almost definitely be a breaking change, I recommend that we try to get it in for Puppet 4.  If we can figure out a solution for the problem in change #4, I can hammer out a revised type, provider, tests, and documentation ASAP.  Any thoughts?

[John Bollinger]

On Monday, January 26, 2015 at 7:06:33 PM UTC-6, Eli Young wrote:

As per PUP-3901, the host type has some serious issues.  Major issues with the current design:

1. The namevar:
   
   • It's currently the canonical hostname. This means that a hostname can be the canonical representation for at most 1 IP address.  This is a problem if, for example, you want to provide both IPv4 and IPv6 addresses for a hostname.
     
   • Changing it to be the IP address would mean that an IP address could have at most one canonical hostname associated with it.  This is less of an issue, but still not ideal.
     
   • Probably the best solution here is to change this to be both the IP address and the canonical hostname (e.g. "1.2.3.4/example.com"). However...

As far as I can tell, it is a design characteristic of the current hosts file format that it associates each address with exactly one canonical name, and each canonical name with exactly one address.  This is a bit ticklish, though, because there seems to be no canonical reference for the file format itself.  Nevertheless, the Linux manpage for it says it has one line per IP address, and that "[f]or each host a single line should be present [...]" (emphasis added).

I am aware that some resolver implementations (including GNU's, documentation notwithstanding) seem to handle the same canonical name and/or the same alias appearing more than once in the hosts file.  Nevertheless, Puppet's Host type must provide an abstraction that is applicable to all supported systems.  There is a bit of room to fiddle with provider features, but you're contemplating a change much more fundamental than that could accommodate.

Indeed, though the type's documentation merely says that a Host resource represents a "host entry", the longtime design demonstrates that it more specifically represents a mapping from a canonical hostname to properties of that hostname including a network address.  It is implicit in the historic use of hostname alone as namevar that duplicate canonical names cannot be modeled.  That these entries are typically recorded in /etc/hosts (on some systems) is in fact a function of the provider and of the 'target' property, so really the format and allowed usage of particular host files in particular contexts can be only weak guidance for whether the model is appropriate.

Perhaps a better question is "what should a Host resource model"?  With the (canonical) hostname as namevar it models properties of that name, but if the namevar were a composite of name and address then it would model properties of that mapping.  Maybe that would indeed be better, but I don't think it can be taken as a given.
 

1. Parsing is flawed:
   
   • Multiple records with the same value for the namevar (currently the canonical hostname) overlap and only one is registered.  Modifying or removing records that overlap behaves inconsistently and, in the case of removal, requires multiple runs to achieve consistency.  Examples in the issue's description.
   • Changing the namevar to be the IP address or both the IP address and the canonical hostname could cause problems on Windows, where the number of hostname aliases per record is limited. This could be resolved by having the provider split a resource into multiple records in the file if the underlying system has alias count limits.

Objection, Your Honor!  Describing the issue as flawed parsing assumes that the files being parsed are correct, and that they are (intended to be) supported by Puppet, but the validity of both assertions is unclear.  To be sure, Host files containing more than one record bearing the same canonical name do not comply with Puppet's model for host entries.  It is unsurprising that Puppet does not handle such files well, but that could as easily be ascribed to invalid/incompatible files as to flawed parsing.

 

The other issues are all consequences of these two issues:

1. Inconsistent resource modification and removal (examples in the issue's description) is a result of namevar collision.
2. Removal of a hostname causing removal of all the aliases is more of a documentation issue than anything. So long as this is explicitly called out as expected behavior, it's not a problem.

You already covered resource modification and removal as a parsing issue.  I am inclined to say "well, duh!" about removing a hostname having the effect of removing all its aliases -- I cannot imagine a person I would trust to write any manifest code being surprised by that -- but on the other hand, I rarely object to more documentation.

 

As such, my proposed changes to the host type are to:

1. Change the generated resource namevar (and, by extension, the alias for specified resources) to use both the IP address and the canonical hostname.
2. Fix parsing to handle cases where multiple records specify the same namevar (which, after change #1, would be an IP address and canonical hostname) by merging them into a single resource.
3. Update documentation to to indicate that the hostname aliases are not first-class host items and that, when a hostname is removed, all aliases are removed too.  If the user wants to retain a hostname alias while removing a hostname, they'll need to put it into a different host resource.
   
4. To allow manifests to set relationships to hosts without knowing ahead of time what the IP address is, potentially provide resource aliases with titles set to the hostname and all the hostname aliases. Unfortunately, this runs into an issue when multiple host resources service the same hostname; blindly making resource aliases would result in each trying to alias to the same name, but conditionally aliasing based on if an alias already exists would result in a relationship attaching to different host resources depending on parse order.  This is a problem that I'm not sure how to solve.

Furthermore, since a resolution to this issue would almost definitely be a breaking change, I recommend that we try to get it in for Puppet 4.  If we can figure out a solution for the problem in change #4, I can hammer out a revised type, provider, tests, and documentation ASAP.  Any thoughts?

The proposed items (1) and (2) are natural for your characterization of the issue -- both that it is a problem that should be solved and the nature of that problem.  As you observe, such a change could be very disruptive.  Consider, however, the proposition that the issue you describe is a limitation of the Host resource that would be better documented and lived with than reworked as you describe.

It would be worthwhile, I think, to at least fix Host removal so that it removes every entry bearing the Host's canonical name, but I don't think semver conflicts with making such a change in a minor (X.Y.0) release.  People who need to manage configurations that Host cannot support must already do so via File resources (or not at all), so they are not affected anyway.  Additionally, perhaps it would be better to deprecate the Host resource in favor of something different, maybe a "HostEntry", that is not burdened with the same limitations.

As I already said, I have no objection to action item (3), which happens to be independent of everything else.

Action item (4) does not appear to be an issue, supposing that the canonical hostname belonging to a Host resource continues to be exposed as a separate property.  Inasmuch as 'name' would no longer be appropos for that property, perhaps it would be exposed as something like 'canonical_name'.  In that case, you could establish resource relationships via collectors:

Host<| canonical_name == 'db_server.my.com' |> -> Service['my_web_service']

What you could not do without knowing both hostname and address is establish a relationship with a specific Host resource among several for the same canonical name.


John

[Eli Young]

On Tuesday, January 27, 2015 at 2:53:46 PM UTC-8, John Bollinger wrote:

As far as I can tell, it is a design characteristic of the current hosts file format that it associates each address with exactly one canonical name, and each canonical name with exactly one address.  This is a bit ticklish, though, because there seems to be no canonical reference for the file format itself.  Nevertheless, the Linux manpage for it says it has one line per IP address, and that "[f]or each host a single line should be present [...]" (emphasis added).

I've done some investigation into various implementations of DNS resolvers and can say that the documentation (or at least this interpretation thereof) is inaccurate.  getaddrinfo(3), which is used on all platforms including Windows to resolve hostnames, provides a linked list of results and can optionally provide the canonical hostname.  When a hostname is listed in the hosts file multiple times, getaddrinfo(3) provides all the IP addresses that are listed for that host, just like it would if it had to fetch that information from DNS.  There are, however, some differences between platforms as to how canonical names are determined if a host is listed as the canonical name for one IP and an alias for another.  Explanation of this will take some doing, so bear with me here.

If you have the following hosts file:

1.1.1.1    host
1.1.2.2    host
1.1.1.1    other

Doing a lookup on "other" will return a linked list with one element, which contains the IP address 1.1.1.1 and the canonical name "other".  This is true on all platforms I tested (i.e. Linux, FreeBSD, OpenBSD, OS X/Darwin, Windows).  Doing a lookup on "host" will return a linked list with two elements, the first with IP address 1.1.1.1 and canonical name "host", and the second with IP address 1.1.2.2.  This is where things differ.  According to the documentation for all platforms, only the first item in the list is supposed to have the canonical name.  Despite this, the BSD variants (i.e. FreeBSD, OpenBSD, and OS X/Darwin) fill in the canonical name on all elements of the list.  So on Linux and Windows, the second element has no canonical name provided, but on the BSDs "host" is listed for the second element.

If you have this hosts file:

1.1.1.1    host
1.1.2.2    host    other
1.1.1.1    other

Things are a bit different.  The results for looking up "host" are the same on all platforms as they were in the previous example, but when looking up "other" things vary.  On Linux, the first element has an IP address of "1.1.2.2" and a canonical name of "other" and the second element has an IP address of "1.1.1.1" and no canonical name.  On FreeBSD and OpenBSD, the first element has an IP address of "1.1.2.2" and a canonical name of "host" and the second element has an IP address of "1.1.1.1" and a canonical name of "other".  On OS X, the order gets switched up; the first element has an IP address of "1.1.1.1" and a canonical name of "other" and the second element has an IP address of "1.1.2.2" and a canonical name of "other".  On Windows, the first element has an IP address of "1.1.1.1" and a canonical name of "other" and there is no second element because I guess as far as Windows is concerned hostnames can be either canonical or aliases but not both, and canonical takes precedence.

If you have this hosts file:

1.1.1.1    host    other
1.1.2.2    host
1.1.1.1    other

Things get weird.  Windows stands by its stance of "canonical and alias are mutually exclusive" and provides a single element containing "1.1.1.1" and "other".  Linux provides two elements with IP address "1.1.1.1", the first with a canonical name of "other" and the second without.  FreeBSD and OpenBSD provide two elements with IP address "1.1.1.1", the first with canonical name "host" and the second with canonical name "other".  OS X/Darwin, however, does something weird.  It provides three elements, the first two with IP address "1.1.1.1" and canonical name "other", and the third with IP address "1.1.2.2" and canonical name "other".

What?

As far as I can tell, OS X/Darwin's outlook is, "Well, the canonical name was something else sometimes, so I went ahead and resolved that for you too.  Also, I didn't make any way for you to tell which items had the different canonical name.  You're welcome!"  This behavior continues if you use this hosts file:

1.1.1.1    host    other
1.1.2.2    host

In this case, when resolving "other", every platform except for OS X/Darwin provides one element with an IP address of "1.1.1.1" and a canonical name of "host".  By contrast, OS X/Darwin provides two elements, the first with IP address "1.1.1.1" and canonical name "host" and the second with IP address "1.1.2.2" and canonical name "host".


So yeah.  I do think that the host type should support specifying multiple IPs for the same hostname, because every resolver implementation I can track down seems to support that (with the possible exception of Solaris, which I can check tomorrow, though I very much doubt that it'll prove an outlier).  This makes sense as the hosts file is something of a poor man's lightning-fast DNS server.  It may be worth also putting some logic in to detect cases where a hostname is specified as both a canonical name and an alias and throwing a warning or an error.  Also to detect cases (at least on OS X) where an alias is specified for some but not all of the IP addresses listed for its canonical name.

 

Indeed, though the type's documentation merely says that a Host resource represents a "host entry", the longtime design demonstrates that it more specifically represents a mapping from a canonical hostname to properties of that hostname including a network address.  It is implicit in the historic use of hostname alone as namevar that duplicate canonical names cannot be modeled.  That these entries are typically recorded in /etc/hosts (on some systems) is in fact a function of the provider and of the 'target' property, so really the format and allowed usage of particular host files in particular contexts can be only weak guidance for whether the model is appropriate.

By contrast, in Chef the namevar is the IP address.  The fact that the canonical hostname is the namevar here is merely a choice that was made when the type was designed, and I am of the opinion that the design is flawed.
 

Objection, Your Honor!  Describing the issue as flawed parsing assumes that the files being parsed are correct, and that they are (intended to be) supported by Puppet, but the validity of both assertions is unclear.  To be sure, Host files containing more than one record bearing the same canonical name do not comply with Puppet's model for host entries.  It is unsurprising that Puppet does not handle such files well, but that could as easily be ascribed to invalid/incompatible files as to flawed parsing.

This is true, and doesn't really matter for a system entirely provisioned by/with Puppet, but if one is attempting to Puppetize infrastructure that already exists, this may well be a problem, one with no good solution at present other than to fall back to managing the hosts file with a file resource.  This is suboptimal.
 

Additionally, perhaps it would be better to deprecate the Host resource in favor of something different, maybe a "HostEntry", that is not burdened with the same limitations.

I actually suggested this very thing in my most recent comment on the JIRA issue, albeit as part of way to enable relationships without requiring collectors.

[John Bollinger]

On Tuesday, January 27, 2015 at 10:02:06 PM UTC-6, Eli Young wrote:

On Tuesday, January 27, 2015 at 2:53:46 PM UTC-8, John Bollinger wrote:

As far as I can tell, it is a design characteristic of the current hosts file format that it associates each address with exactly one canonical name, and each canonical name with exactly one address.  This is a bit ticklish, though, because there seems to be no canonical reference for the file format itself.  Nevertheless, the Linux manpage for it says it has one line per IP address, and that "[f]or each host a single line should be present [...]" (emphasis added).

I've done some investigation into various implementations of DNS resolvers and can say that the documentation (or at least this interpretation thereof) is inaccurate.  getaddrinfo(3)

DNS resolvers are irrelevant to the hosts file format.  The hosts file is not a DNS data source, even if resolver libraries happen to consult it, too.  In fact, the original objective of DNS was to altogether replace hosts files.  That getaddrinfo(3) can provide multiple records for one canonical name in no way implies that every data source it draws upon is expected to be able to describe such data. Moreover, the official specification for getaddrinfo(3) (from POSIX 1003.1) does not even mention the hosts file, so it is completely up to implementations whether to use such a file, and if so, how.  (POSIX doesn't even require that the function perform resolutions against DNS, though it notes that many implementations in fact do so.)

 

, which is used on all platforms including Windows to resolve hostnames, provides a linked list of results and can optionally provide the canonical hostname.  When a hostname is listed in the hosts file multiple times, getaddrinfo(3) provides all the IP addresses that are listed for that host, just like it would if it had to fetch that information from DNS.

Sorry, you simply cannot support that based only on use of the getaddrinfo(3) interface.  The behavior of that function is implementation-defined with respect to ALL data sources, even DNS.  As you yourself observe, its behavior in fact does differ among implementations.  It is flexible enough to support the behavior you describe, which is common, but it cannot be taken as any sort of specification for hosts files.

I have more regard for your survey of actual implementations, though.

[...]

So yeah.  I do think that the host type should support specifying multiple IPs for the same hostname, because every resolver implementation I can track down seems to support that (with the possible exception of Solaris, which I can check tomorrow, though I very much doubt that it'll prove an outlier).

It is a big step from "known resolvers handle hosts files giving multiple addresses for the same name" (in not altogether consistent ways) to "Puppet's Host resource should be modified to support multiple addresses per name", and even a bigger one to "Host resources should be modified to have a composite namevar."  I did not and do not deny that there is a problem to be solved here, but on the other hand, I don't necessarily agree with your characterization of the problem or your proposed solution.

 

  This makes sense as the hosts file is something of a poor man's lightning-fast DNS server.  It may be worth also putting some logic in to detect cases where a hostname is specified as both a canonical name and an alias and throwing a warning or an error.  Also to detect cases (at least on OS X) where an alias is specified for some but not all of the IP addresses listed for its canonical name.

 

Indeed, though the type's documentation merely says that a Host resource represents a "host entry", the longtime design demonstrates that it more specifically represents a mapping from a canonical hostname to properties of that hostname including a network address.  It is implicit in the historic use of hostname alone as namevar that duplicate canonical names cannot be modeled.  That these entries are typically recorded in /etc/hosts (on some systems) is in fact a function of the provider and of the 'target' property, so really the format and allowed usage of particular host files in particular contexts can be only weak guidance for whether the model is appropriate.

By contrast, in Chef the namevar is the IP address.  The fact that the canonical hostname is the namevar here is merely a choice that was made when the type was designed, and I am of the opinion that the design is flawed.
 

I agree that a different namevar could as easily have been chosen back when the Host resource was designed.  Inasmuch as the Host resource does not support multiple addresses for the same name, and that is a desirable feature, it is inarguable that the design has an unfortunate limitation.  Call it a flaw if you will.

At the time the type was designed, however, hostname as namevar better suited Puppet's capabilities and users' requirements, particularly with respect to resource relationships.  It would be rare for another resource to depend on there being a name mapped to (say) address 10.11.12.13, but it is reasonably common for another resource to depend on a name such as 'myservice.my.com' being mapped to an address.  Only with the advent of the chain operators did Puppet gain the ability to declare relationships other than via the target resource's namevar.

 

Objection, Your Honor!  Describing the issue as flawed parsing assumes that the files being parsed are correct, and that they are (intended to be) supported by Puppet, but the validity of both assertions is unclear.  To be sure, Host files containing more than one record bearing the same canonical name do not comply with Puppet's model for host entries.  It is unsurprising that Puppet does not handle such files well, but that could as easily be ascribed to invalid/incompatible files as to flawed parsing.

This is true, and doesn't really matter for a system entirely provisioned by/with Puppet, but if one is attempting to Puppetize infrastructure that already exists, this may well be a problem, one with no good solution at present other than to fall back to managing the hosts file with a file resource.  This is suboptimal.

Agreed.  I never argued that there wasn't any issue here.

 

 

Additionally, perhaps it would be better to deprecate the Host resource in favor of something different, maybe a "HostEntry", that is not burdened with the same limitations.

I actually suggested this very thing in my most recent comment on the JIRA issue, albeit as part of way to enable relationships without requiring collectors.

I think your suggestion could be tweaked such that introduction of the proposed new resource type didn't require any change to most existing code.  If, just as you suggested, the hypothetical new type, "Hostentry" / "Hostname" / whatever,

1. autogenerated a corresponding Host resource, and
   
2. created a 'before' relationship from itself to the corresponding Host,

then other resources could continue to 'require' Host resources just as they do now.  (I assert that other than as described above, no useful purpose is served by a 'before' edge targeting a Host resource).

Furthermore, with a bit of care in the type implementation, and perhaps a new parameter or two for the Host resource, this scheme could also cause duplicate resource errors to be properly thrown when a manually-declared Host resource conflicts with a Hostentry resource.

Avoiding a breaking change is an attractive feature of this strategy in its own right, but it has the additional advantage that it does not require hurrying to get an implementation into Puppet 4 (or waiting years for Puppet 5).  With P4 imminent, I am uncertain whether there is any chance at this point to get an additional change/features into 4.0.0.


John

[Eli Young]

On Thursday, January 29, 2015 at 7:27:10 AM UTC-8, John Bollinger wrote:

DNS resolvers are irrelevant to the hosts file format.  The hosts file is not a DNS data source, even if resolver libraries happen to consult it, too.  In fact, the original objective of DNS was to altogether replace hosts files.

Mistaken word choice on my part.  I meant hostname resolvers.
 

At the time the type was designed, however, hostname as namevar better suited Puppet's capabilities and users' requirements, particularly with respect to resource relationships.  It would be rare for another resource to depend on there being a name mapped to (say) address 10.11.12.13, but it is reasonably common for another resource to depend on a name such as 'myservice.my.com' being mapped to an address.  Only with the advent of the chain operators did Puppet gain the ability to declare relationships other than via the target resource's namevar.

This makes sense.  Indeed, having a composite namevar would make it less usable, which was something I was trying to reconcile from the beginning (c.f. my thoughts on and questions about resource aliasing).
 

I think your suggestion could be tweaked such that introduction of the proposed new resource type didn't require any change to most existing code.  If, just as you suggested, the hypothetical new type, "Hostentry" / "Hostname" / whatever,

1. autogenerated a corresponding Host resource, and
   
2. created a 'before' relationship from itself to the corresponding Host,

then other resources could continue to 'require' Host resources just as they do now.  (I assert that other than as described above, no useful purpose is served by a 'before' edge targeting a Host resource).

Furthermore, with a bit of care in the type implementation, and perhaps a new parameter or two for the Host resource, this scheme could also cause duplicate resource errors to be properly thrown when a manually-declared Host resource conflicts with a Hostentry resource.

Avoiding a breaking change is an attractive feature of this strategy in its own right, but it has the additional advantage that it does not require hurrying to get an implementation into Puppet 4 (or waiting years for Puppet 5).  With P4 imminent, I am uncertain whether there is any chance at this point to get an additional change/features into 4.0.0.

This is probably the best way forward.  It might also be worth adding an additional dummy provider solely for the autogenerated host resources so that Puppet wouldn't try to use them to modify the hosts file.

[Rob Nelson]

My concern is with accurate modeling of existing state. For example:

[rnelson0@test ~]$ cat /etc/hosts
# HEADER: This file was autogenerated at Thu Jan 22 22:08:17 +0000 2015
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
127.0.0.1       localhost       localhost.localdomain localhost4 localhost4.localdomain4
::1     localhost       localhost.localdomain localhost6 localhost6.localdomain6

1.2.3.4 test
ffff::0001 test
[rnelson0@build profile]$ puppet resource host
host { 'localhost':
  ensure       => 'present',
  host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4'],
  ip           => '127.0.0.1',
  target       => '/etc/hosts',
}
host { 'test':
  ensure => 'present',
  ip     => '1.2.3.4',
  target => '/etc/hosts',
}

Puppet cannot accurately model the existing state, so it is difficult at beast to enforce that state. There are so many edge cases that I feel it may be impossible, but I don't think we've identified them all, much less tested them. Regardless, such a simple hosts file should be enumerable as an accurate model.

I do not have any good answers for this issue, but wanted to point out what I think is a good guiding principle for resolving this. Perhaps picking a single OS, a HostEntry type/provider can be built to try and accurately model state, whether the namevar is the ip, hostname, or something else, until we get it right. As a new type, there would be no deadline for Puppet 4 and hence it could be done right rather than making compromises again.