Unresponsive Nodes - After Stopping AWS Instances w/o Elastic IP

[Brahim Dagher]

I deployed a Puppet Enterprise Master and three Agent nodes in AWS. At the time I did not allocate Elastic IP to the the instances. I got everything up and running and was enjoying Puppet until I stopped the instances.

Now my console shows the nodes as unresponsive.

Here is my attempt thus far in fixing it.

I allocated EIPs to each of the instances and now the external DNS names have changed

MASTER DNS NAME = ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com

AGENT INTERNAL DNS NAME = ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal

_______________________________________________________________

PUPPET.CONF

I changed the puppet.conf file to reflect the new Elastic IPs

AGENTS

[main]

    certname = ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal

    vardir = /var/opt/lib/pe-puppet

    logdir = /var/log/pe-puppet

    rundir = /var/run/pe-puppet

    modulepath = /etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules

    server = ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com

    user  = pe-puppet

    group = pe-puppet

    archive_files = true

    archive_file_server = ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com

[agent]

    report = true

    classfile = $vardir/classes.txt

    localconfig = $vardir/localconfig

    graph = true

    pluginsync = true

    environment = production

MASTER 

[main]

    certname = ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com

    dns_alt_names = ip-zzz-zzz-zzz-zzz,ip-zzz-zzz-zzz-zzz.us-west-2.compute.internal,puppet,puppet.us-west-2.compute.internal$

    vardir = /var/opt/lib/pe-puppet

    logdir = /var/log/pe-puppet

    rundir = /var/run/pe-puppet

    modulepath = /etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules

    server = ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com

    user  = pe-puppet

    group = pe-puppet

    archive_files = true

    archive_file_server = ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com

[master]

    certname = ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com

    ca_name = 'Puppet CA generated on ip-zzz-zzz-zzz-zzz.us-west-2.compute.internal at 2014-05-23 17:37:24 -0400'

    reports = console,puppetdb

    node_terminus = console

    ssl_client_header = SSL_CLIENT_S_DN

    ssl_client_verify_header = SSL_CLIENT_VERIFY

    storeconfigs_backend = puppetdb

    storeconfigs = true

[agent]

    report = true

    classfile = $vardir/classes.txt

    localconfig = $vardir/localconfig

    graph = true

    pluginsync = true

    environment = production

_______________________________________________________________

CERTS

MASTER - wipe the certs

puppet cert clean ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com

AGENTS

sudo rm -f /etc/puppetlabs/puppet/ssl/certs/ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com.pem

I only have a ca.pem in there now

Ran  sudo /opt/puppet/bin/puppet agent -t --> Exiting; no certificate found and waitforcert is disabled

_______________________________________________________________ 

 

 

At this point the Master itself also has an agent that is non responsive.

I can ping the master from the agent and vice versa..

What is missing??

Need a ninja...