Puppet Client using wrong environment.

[Tony G.]

Hi,

I exposed few days ago this situation on the IRC, but unfortunately I haven't found a way to solve it yet.

I run the Puppet Master with multiple environments, and everything is working good but a couple of nodes that are causing me some troubles.

These nodes are pointing to my production environment and I need to point them back to development to grab some changes I made.

I usually follow these steps on the clients once I've tested in my own environment:
1. Stop puppet daemon.
2. Test the new config pointing to development environment.
3. Run puppetd pointing to development.
4. Verify change.
5. Start Puppet.

This is a test I did with a similar node:

Testing pointing to development:

[app03 ~]# /usr/sbin/puppetd --test -vt --environment=development
--snip--
notice: Finished catalog run in 30.42 seconds
[app03 ~]# grep -m2 -P "/development/|/production/" /var/lib/puppet/localconfig.yaml
  file: /opt/puppet/development/site.pp
        file: /opt/puppet/development/modules/ntp/manifests/init.pp

Testing now pointing to production:

[app03 ~]# /usr/sbin/puppetd --test -vt --environment=production
--snip--
notice: Finished catalog run in 30.55 seconds
[eapp03 ~]# grep -m2 -P "/development/|/production/" /var/lib/puppet/localconfig.yaml
  file: /opt/puppet/production/site.pp
        file: /opt/puppet/production/classes/puppet.pp

When doing the same on the node I see the strange behavior this is what I get:

[app01 ~]# /usr/sbin/puppetd --test -vt --environment=development
--snip--
[app01 ~]# grep -m2 -P "/development/|/production/" /var/lib/puppet/localconfig.yaml
        file: /opt/puppet/production/classes/yumrepos.pp
        file: /opt/puppet/production/classes/yumrepos.pp

I've tried:

• Removing the cache in the server ($vardir/yaml/{node,facts}/app01.example.com.yaml) and puppetca --clean app01.example.com
  
• Restarted the server(using passenger).
  
• Removed the client $vardir/*
  
• Reinstalled puppet client.

All of them with the same result.

The nodes with these issues are being hosted with a third party hosting company, don't think there is any relationship with this as communications are made between the client/server, but seems like I can't find any other.

Any suggestion is appreciated,  Thanks for reading this.

This is what I use:

PuppetMaster 0.24.8
Puppetclients 0.24.8
passenger (2.2.4)

puppet.conf:
---snippet---
environments = production,testing,development
    manifest = /opt/puppet/production/site.pp
    modulepath = /opt/puppet/production/modules

[production]
    manifest = /opt/puppet/production/site.pp
    modulepath = /opt/puppet/production/modules

[testing]
    manifest = /opt/puppet/testing/site.pp
    modulepath = /opt/puppet/testing/modules

[development]
    manifest = /opt/puppet/development/site.pp
    modulepath = /opt/puppet/development/modules

---snippet---


--
Tony

[Rob McBroom]

How is the environment determined when you don’t specify one? On my systems, it seems that the `--environment` option is completely ignored. I imagine this is because the external information (from LDAP in my case) always takes precedence.

--
Rob McBroom
<http://www.skurfer.com/>

[Tony G.]

When no environment is defined it uses production, per my notes:

environments = production,testing,development
    manifest = /opt/puppet/production/site.pp
    modulepath = /opt/puppet/production/modules

This is part of [main].

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
Tony

[Tony G.]

Finally found the issue

I never looked into the /etc/puppet directory(my bad), there was the issue, a strange one although but here is what happened, in there were two files(wish I knew why there were that second if we use puppet 0.24.8):

puppet.conf
puppetd.conf

Same content, nothing different, I don't declare the environment in there as I use it in /etc/sysconfig/puppet

[app02 ~]$ grep environment /etc/puppet/puppet.conf /etc/puppet/puppetd.conf
[app02 ~]$ diff /etc/puppet/puppet.conf /etc/puppet/puppetd.conf
[app02 ~]# /usr/sbin/puppetd -t --environment=development
info: Loading fact drac
info: Loading fact rhelver
info: Loading fact sserial
info: Retrieving plugins
info: Loading fact drac
info: Loading fact rhelver
info: Loading fact sserial
info: mount[localhost]: Mounted /
info: mount[modules]: Mounted
info: mount[plugins]: Mounted
info: Caching catalog at /var/lib/puppet/localconfig.yaml
notice: Starting catalog run
info: Loading fact drac
info: Loading fact rhelver
info: Loading fact sserial
info: Sent transaction report in 1.47 seconds
notice: Finished catalog run in 16.85 seconds
[app02 ~]# grep -m2 -P "development|production" /var/lib/puppet/localconfig.yaml
        file: /opt/puppet/production/classes/yumrepos.pp
        file: /opt/puppet/production/classes/yumrepos.pp

Fixing:
[app02 ~]# rm /etc/puppet/puppetd.conf
rm: remove regular file `/etc/puppet/puppetd.conf'? y

[app02 ~]# /usr/sbin/puppetd -t --environment=development
info: Loading fact drac
info: Loading fact rhelver
info: Loading fact sserial
info: Retrieving plugins
info: Loading fact drac
info: Loading fact rhelver
info: Loading fact sserial
info: mount[localhost]: Mounted /
info: mount[modules]: Mounted
info: mount[plugins]: Mounted
info: Caching catalog at /var/lib/puppet/localconfig.yaml
notice: Starting catalog run
info: Loading fact drac
info: Loading fact rhelver
info: Loading fact sserial
notice: //Node[app02.example.com]/common::ldap/Sudo::Config[standard]/Package[sudo-ldap]/ensure: removed
notice: //Node[app02.example.com]/common::ldap/Sudo::Config[standard]/Package[sudo]/ensure: created
info: Sent transaction report in 0.37 seconds
notice: Finished catalog run in 18.09 seconds
[app02 ~]# grep -m2 -P "development|production" /var/lib/puppet/localconfig.yaml
  file: /opt/puppet/development/site.pp
        file: /opt/puppet/development/classes/common.pp

So what is doing puppetd is reading both files and for some reason ignoring the --environment flag via command line or the flag in the /etc/sysconfig/puppet when bringing up the daemon with the /etc/init.d/puppet script.

Doing a strace I see both files are being opened with puppetd:
open("/etc/puppet/puppetd.conf", O_RDONLY) = 3
open("/etc/puppet/puppet.conf", O_RDONLY) = 4

I assume it attempts to read puppetd.conf as it was the old config file, but should it ignore the environment flag as it's doing it?

Thanks

--
Tony