Verdict: Went fine with the usual upgrade teething troubles. (Once I figured those out I reverted to my pre-upgrade VM snapshot on the first upgraded host and there was no hassle the second time.) I went from 3.4.3 to 3.6.2.
The procedure was to upgrade the following rpms on each host (daemon restarts were included), reading logs as I went:
for i in mcollective-puppet-agent hiera facter puppet mcollective; do mco package $i update -F hostname=$SERVER; done
Upgrading mcollective this way meant the last mco package never returned since the daemon was restarted, I'm fine with that.
Then this for the puppetmasters to ensure the passenger-based puppetmasters were restarted:
mco service httpd restart -F hostname=$SERVER; done
These three issues, unfixed, would have been blockers for me doing an upgrade:
1) yumrepo mirrorlist vs baseurl (fixed in puppet 3.5.1)
https://tickets.puppetlabs.com/browse/PUP-2155 (dupe)
https://tickets.puppetlabs.com/browse/PUP-2150
I was using mirrorlist, it was past time to use baseurl anyway. I never saw an error since I was on 3.4.3.
Solution: use baseurl.
2) datacat module 0.4.2 incompatible with puppet >= 3.5 (fixed in datacat 0.5.0)
https://github.com/richardc/puppet-datacat/issues/12
I saw this sort of error on the puppetmaster:
Jun 12 09:48:15 puppetmaster1dev puppet-master[4552]: undefined method `module' for "production":String at /etc/puppet/environments/production/modules/datacat/manifests/init.pp:59 on node
puppetmaster1dev.company.com
Solution: upgrade the datacat module to 0.5.0
3) mcollective service/package trouble with puppet 3.5 (fixed in mcollective-puppet-agent 1.7.1)
https://tickets.puppetlabs.com/browse/MCOP-23
I saw the following errors when checking package status after the initial aborted upgrade of mcollective only, not any mcollective plugins.
a) First, this after a plain upgrade (but without restarting the puppet agent):
$ mco package mcollective status -F hostname=puppetmaster1dev
* [ ==========================================================> ] 1 / 1
puppetmaster1dev.company.com: Could not autoload puppet/type/package: no 'environments' in {:root_environment=>#<Puppet::Node::Environment:0x7fc087f1d070 @manifest="/tmp", @modulepath=[], @watching=true, @name=:"*root*", @config_version=nil>, :current_environment=>#<Puppet::Node::Environment:0x7fc087f1d070 @manifest="/tmp", @modulepath=[], @watching=true, @name=:"*root*", @config_version=nil>} at top of [[0, nil, nil]]
Summary of Arch:
No aggregate summary could be computed
Summary of Ensure:
No aggregate summary could be computed
Finished processing 1 / 1 hosts in 46.00 ms
b) Second, after restarting both mcollective and puppet:
$ mco package mcollective status -F hostname=puppetmaster1dev
* [ ==========================================================> ] 1 / 1
puppetmaster1dev.company.com: undefined method `properties' for nil:NilClass
Summary of Arch:
No aggregate summary could be computed
Summary of Ensure:
No aggregate summary could be computed
Finished processing 1 / 1 hosts in 31.84 ms
Solution: upgrade mcollective-puppet-agent before upgrading/restarting mcollective and puppet
Editorially, this was very relaxing and I should have saved it for Friday afternoon.
> To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/20140612114824.GB25610%40iniquitous.heresiarch.ca.