Facter ec2 facts, should they be redone with describe sdk methods

[Phil Watts]

As a person who uses Puppet Enterprise in ec2 heavily, I've spent a lot of time thinking about, and working around the present ec2 facts. Right now, ec2 facts are merely a flattened version of the ec2 instance metadata, and while it provides quite a lot of information, and a good bit of it being information on would desire, sometimes that information is difficult to access (possibly my ignorance at work). As an example, the fact for the VPC id of an ec2 instance, is ec2_network_interfaces_macs_$Mac address of interface_vpc_id. Although I've not had to consider the implications of many interfaces attached to an instance, this fact is unwieldy, but often a good item to turn configuration on. I've dealt with it, using a simpler custom fact.

require 'facter'

  Facter.add("ec2_vpc_id") do
    confine :cloud_provider => 'aws'
    setcode do
      mac    = Facter.value(:ec2_mac)
      vpc_id_fact = "ec2_network_interfaces_macs_#{mac}_vpc_id"
      Facter.value(vpc_id_fact)
    end
  end

Forgive the cloud_provider confine, it's a relic of the pre factor 2 in PE days, and I haven't gone back to find the appropriate confine for VPC instances yet.

Anyway, my question is, in general, does it seem a worthwhile effort to refactor the base ec2 facts to use parts of the meta data, instead of the more simple flattening of the entire return (which would also remove some of the undesirable facts created). Also has thought been given to using the ruby SDK to invoke describe methods to get new facts into the core. Of particular interest might be a fact, or set of facts around the ec2 instance tags, which are not in the metadata. Below is my hacky way of accomplishing that at present.

require 'facter'
require 'json'
cloud_provider = Facter.value(:cloud_provider)
case cloud_provider
  when 'aws'
    instance_id = Facter.value(:ec2_instance_id)
    osfamily = Facter.value(:osfamily)
    case osfamily
      when 'Debian'
        tags = Facter::Core::Execution.exec("/usr/local/bin/aws ec2 describe-tags --filters \"Name=resource-id,Values=#{instance_id}\" --output json --region us-east-1")
      when 'RedHat'
        tags = Facter::Core::Execution.exec("/usr/bin/aws ec2 describe-tags --filters \"Name=resource-id,Values=#{instance_id}\" --output json --region us-east-1")
    end
      tags_hash = JSON.parse(tags)["Tags"]
    begin
      tags_hash.each do |tag|
    rescue
        Facter.add("ec2_tag_" + tag["Key"]) do
          setcode do
        tag["Value"]
          end
        end
      end
    end
end

Please forgive any ignorance to history, convention, or customs, this is my first post, and I'm pretty new to the world of writing anything in ruby. Thoughts, advice, direction would all be appreciated.

[Michael Smith]

On Mon, Apr 13, 2015 at 4:04 AM, Phil Watts <pwat...@gmail.com> wrote:

As a person who uses Puppet Enterprise in ec2 heavily, I've spent a lot of time thinking about, and working around the present ec2 facts. Right now, ec2 facts are merely a flattened version of the ec2 instance metadata, and while it provides quite a lot of information, and a good bit of it being information on would desire, sometimes that information is difficult to access (possibly my ignorance at work). As an example, the fact for the VPC id of an ec2 instance, is ec2_network_interfaces_macs_$Mac address of interface_vpc_id. Although I've not had to consider the implications of many interfaces attached to an instance, this fact is unwieldy, but often a good item to turn configuration on. I've dealt with it, using a simpler custom fact.

require 'facter'

  Facter.add("ec2_vpc_id") do
    confine :cloud_provider => 'aws'
    setcode do
      mac    = Facter.value(:ec2_mac)
      vpc_id_fact = "ec2_network_interfaces_macs_#{mac}_vpc_id"
      Facter.value(vpc_id_fact)
    end
  end

Forgive the cloud_provider confine, it's a relic of the pre factor 2 in PE days, and I haven't gone back to find the appropriate confine for VPC instances yet.

This particular case may be easier with the structured ec2_metadata fact:

ec2_metadata => {"ami-id"=>"ami-4dbf9e7d", "ami-launch-index"=>"0", "ami-manifest-path"=>"(unknown)", "block-device-mapping"=>{"ami"=>"/dev/sda1", "root"=>"/dev/sda1"}, "hostname"=>"ip-10-0-2-134.us-west-2.compute.internal", "instance-action"=>"none", "instance-id"=>"i-4a26a0bc", "instance-type"=>"t2.micro", "local-hostname"=>"ip-10-0-2-134.us-west-2.compute.internal", "local-ipv4"=>"10.0.2.134", "mac"=>"06:84:3c:20:ab:9e", "metrics"=>{"vhostmd"=>"<?xml version=\"1.0\" encoding=\"UTF-8\"?>"}, "network"=>{"interfaces"=>{"macs"=>{"06:84:3c:20:ab:9e"=>{"device-number"=>"0", "interface-id"=>"eni-89d982ff", "ipv4-associations"=>{"52.11.17.205"=>"10.0.2.134"}, "local-hostname"=>"ip-10-0-2-134.us-west-2.compute.internal", "local-ipv4s"=>"10.0.2.134", "mac"=>"06:84:3c:20:ab:9e", "owner-id"=>"482693910459", "public-hostname"=>"ec2-52-11-17-205.us-west-2.compute.amazonaws.com", "public-ipv4s"=>"52.11.17.205", "security-group-ids"=>"sg-b4c4c4d1", "security-groups"=>"launch-wizard-128", "subnet-id"=>"subnet-c88c2abf", "subnet-ipv4-cidr-block"=>"10.0.2.0/24", "vpc-id"=>"vpc-2c13bd49", "vpc-ipv4-cidr-block"=>"10.0.0.0/16"}}}}, "placement"=>{"availability-zone"=>"us-west-2b"}, "profile"=>"default-hvm", "public-hostname"=>"ec2-52-11-17-205.us-west-2.compute.amazonaws.com", "public-ipv4"=>"52.11.17.205", "public-keys"=>{"0"=>{"openssh-key"=>"..."}}, "reservation-id"=>"r-5f98f853", "security-groups"=>"launch-wizard-128", "services"=>{"domain"=>"amazonaws.com"}}

ec2_metadata.network.interfaces.macs is a hash that can be iterated over in Puppet. This may make some tasks easier.

A brief example of using them to print the vpc-id for each network interface:

$macs = $::facts['ec2_metadata']['network']['interfaces']['macs']

each($macs) |$k, $v| { notice $v['vpc-id'] }

That's an interesting idea, but looks like you need to install the aws tool. That makes it trickier to have as a core fact. 

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/6105274e-752b-41c5-b688-4c485a91da35%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael Smith

Sr. Software Engineer, Puppet Labs

PuppetConf 2015 is coming to Portland, Oregon! Join us October 5-9.

Register now to take advantage of the Early Adopter discount —save $349!

[Phil Watts]

Thanks for the tip regarding the fact hash reference, I'm pretty much just getting started with Ruby, so obviously I'm prone to inefficiencies like this.

As for the tags fact, this is again a manifestation of haste and ignorance. The aws cli could easily be replaced by requiring and aws-sdk gem, but early attempts led to inefficiencies accepted more to prove the concept. I'll certainly refactor these facts, but my general question would be, is there a place I should submit that code? I'm not positive what the right approach would be apart from writing them for my own use.

[Michael Smith]

To be clear, my example of iterating over the interfaces was Puppet code. But you can also access the hash from Ruby.

New custom facts can be distributed as Puppet modules. See https://docs.puppetlabs.com/guides/plugins_in_modules.html. If you have a concrete set of facts you'd like to propose, you can follow up with them or put them up as a PR to Facter, but there's a good chance we'll ask that you release them as a module instead. It depends on what function the facts are trying to fulfill.

[Peter Foley]

Hi Phil,

New custom facts can be distributed as Puppet modules. See https://docs.puppetlabs.com/guides/plugins_in_modules.html. If you have a concrete set of facts you'd like to propose, you can follow up with them or put them up as a PR to Facter, but there's a good chance we'll ask that you release them as a module instead. It depends on what function the facts are trying to fulfill.

If your proposed facts don't make it into the facter core set of facts could I suggest rather than creating a new module you develop a pull request against the puppetlabs-aws module [1] which is currently being worked on.  This module already requires the aws-sdk-core ruby gem you mentioned.

[1] - https://github.com/puppetlabs/puppetlabs-aws

Peter

[Gareth Rushgrove]

You might also be interested in
https://github.com/mrzarquon/puppet-ec2tags, which exposes tags on an
EC2 instance and uses the aws-sdk to do so. You could also use this as
a basis for a separate module exposing other facts based on API calls.

Their are a number of things like this available from the API but not
from the metadata. I'm unsure whether these would be better as facts
in the AWS module, or as separate modules (maybe with a dependency
from the AWS module). My gut feeling is the later.

Gareth

> Peter

>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/83c20dee-01a3-48d7-bc78-a2f99781b6d8%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.



--

Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[Phil Watts]

Thanks Gareth, that looks pretty much like the refactoring I intended to do to converts tag facts from cli to sdk.

I have a bit of an upjumped problem at work that has me distracted from integrating puppet and AWS at the moment, but once that's resolved, I'll start curating a list of sdk available information that may be a good list to compose into a module.

Sent from my iPhone

> You received this message because you are subscribed to a topic in the Google Groups "Puppet Developers" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-dev/Or0ZxTnoFxU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAFi_6yJTN_UPe-1hegP%3D4n4DWqAerUVygJkyWYCJnUvvjb2NjA%40mail.gmail.com.