Puppet upgrade 2.6.12 -> 2.7.10
Thomas Uphill
Prentice Bisbal
I'm having a problem with Puppet as a result of this upgrade. Clients
and server are both at 2.7.10. I see this in /var/log on the clients:
puppet-agent[17028]: Could not retrieve catalog from remote server:
Error 400 on SERVER: No support for http method POST
Running 'puppet agent --test' on a client, I see this:
# puppet agent --test
info: Caching catalog for host02.example.com
info: Applying configuration version '1328204765'
notice: /Stage[main]/Nvidia/Exec[/usr/sbin/nvidia-config-display
enable]/returns: executed successfully
notice: /Stage[main]/Nvidia::Cuda/Service[nvidia]/ensure: ensure changed
'stopped' to 'running'
notice: Finished catalog run in 8.93 seconds
notice: /File[/var/lib/puppet/state/last_run_summary.yaml]/content:
--- /var/lib/puppet/state/last_run_summary.yaml 2012-02-02
13:03:27.220300345 -0500
+++ /tmp/puppet-file20120202-22006-uz0vv5-0 2012-02-02
13:13:55.014216959 -0500
@@ -1,33 +1,33 @@
---
+ events:
+ failure: 0
+ total: 2
+ success: 2
+ changes:
+ total: 2
resources:
- failed_to_restart: 0
+ out_of_sync: 2
total: 206
+ failed: 0
+ scheduled: 0
restarted: 0
- changed: 2
skipped: 6
- scheduled: 0
- failed: 0
- out_of_sync: 2
+ changed: 2
+ failed_to_restart: 0
time:
- sshkey: 0.024306
- service: 2.978418
- config_retrieval: 2.92832803726196
- exec: 1.064473
- last_run: 1328205807
- filebucket: 0.00024
- total: 9.56267703726196
- mount: 0.004192
- mailalias: 0.000462
- file: 1.233224
- package: 1.32159
- host: 0.004381
- ssh_authorized_key: 0.003063
- changes:
- total: 2
- events:
- success: 2
- total: 2
- failure: 0
+ filebucket: 0.000257
+ last_run: 1328206435
+ total: 11.2155618839874
+ service: 3.214659
+ config_retrieval: 3.79004788398743
+ package: 2.165398
+ mailalias: 0.000459
+ mount: 0.0054
+ host: 0.004472
+ exec: 0.998636
+ sshkey: 0.023502
+ file: 1.009792
+ ssh_authorized_key: 0.002939
version:
- config: 1328204723
+ config: 1328204765
puppet: 2.7.10
\ No newline at end of file
err: Could not remove PID file /var/run/puppet/agent.pid
Checking on the server, I see this in /var/log/httpd/error_log:
[ pid=6391 thr=140684595804128 file=ext/apache2/Hooks.cpp:863
time=2012-02-02 12:49:09.606 ]: Unexpected error in mod_passenger: An
error occured while buffering HTTP upload data to a temporary file in
/tmp/passenger.1.0.6370/generation-0/buffered_uploads. The current
Apache worker process (which is running as apache) doesn't have
permissions to write to this directory. Please change the permissions
for this directory (as well as all parent directories) so that it is
writable by the Apache worker process, or set the
'PassengerUploadBufferDir' directive to a directory that Apache can
write to.
Backtrace:
in 'boost::shared_ptr<Passenger::BufferedUpload>
Hooks::receiveRequestBody(request_rec*)' (Hooks.cpp:1230)
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:566)
Checking the permissions down to
/tmp/passenger.1.0.6370/generation-0/buffered_upload:
# cd /tmp
# ls -l passenger.1.0.6370
drwsr-xr-x. 3 root root 4096 Feb 2 12:45 passenger.1.0.6370
I changes the ownership to apache:apache, but still get an error
# chown -R apache:apache passenger.1.0.6370
So then I made all directories in that path read/write for everyone:
# ls -ld /tmp/passenger.1.0.6370
drwsrwxrwx. 3 apache apache 4096 Feb 2 12:45 /tmp/passenger.1.0.637
# ls -ld /tmp/passenger.1.0.6370/generation-0
drwsrwxrwx. 5 apache apache 4096 Feb 2 12:45
/tmp/passenger.1.0.6370/generation-0
# ls -ld /tmp/passenger.1.0.6370/generation-0/buffered_uploads/
drwsrwxrwx. 2 apache apache 4096 Feb 2 13:26
/tmp/passenger.1.0.6370/generation-0/buffered_uploads/
I no longer see the write permissions error in /var/log/httpd/error_log,
but clearly having those directories 777 is probably not a good idea.
SELinux is permissive for now, and I restarted httpd on the puppet
master to make sure it was using the new puppet binaries.
Any ideas?
--
Prentice
Benjamin Rose
Sorry, the first copy sent before I was done typing...
Prentice,
You are likely using a 2.6 client against a 2.7 server, at least that's the error message I received when I updated and had this issue. You said you restarted the puppetmaster, did you restart puppetd on the clients?
Thanks, Ben
Benjamin Rose
Prentice,
Let's examine together.
You stated that puppet and puppetmaster installed RPMs are 2.7.10, and you said you restarted httpd on the puppet master. However, you never said you restarted puppetd on the clients.
Now, given this output:
" # puppet agent --test > info: Caching catalog for host02.example.com <http://host02.example.com> > info: Applying configuration version '1328204765' ...
version: > - config: 1328204723 > + config: 1328204765 > puppet: 2.7.10 "
It looks as if 2.7.10 does indeed work just fine, when run from the command line. However, your failure logs come from /var/log, which is the output of the puppet daemon. In this case, it seems to be that puppetd on the clients has never been restarted, so is still running as 2.6. Humor me and give "service puppet restart" a try on the clients... that fixed this exact problem for me a couple of days ago.
Ben
Ben, You should read at least the first line of an e-mail before replying. The first line of my e-mail:
> I'm having a problem with Puppet as a result of this upgrade. Clients > and server are both at 2.7.10. I see this in /var/log on the clients:
-- Prentice On 02/02/2012 01:34 PM, Benjamin Rose wrote:
> Prentice, > You are likely attempting to run a 2.6 client against a 2.7.10 server.
> Ben > On 01/31/2012 04:27 PM, Thomas Uphill wrote:
>> > Puppet has been upgraded from 2.6.12 to 2.7.10. There are language >> > differences between 2.6 and 2.7 as outlined in the release notes >> > http://projects.puppetlabs.com/projects/puppet/wiki/Release_Notes >> > >> > Check the output of your server for warnings, there are several >> > functions and practices that are now deprecated. >> > Dynamic scope is no longer supported, all variables should be >> > referenced using their fully qualified addresses as outlined >> > here: http://docs.puppetlabs.com/guides/scope_and_puppet.html >> > >> > We will be pushing the new puppet tomorrow, so if you are configured >> > for automatic updates you might wish to disable automatic updates >> > until you can test the new puppet in your environment. >> > >> > We have been using the new version internally since 2.7.6 without >> > issue but were waiting for the 2.7 series to stabilize before >> > releasing it in the wild. >> > >> > Any problems, please report them to the list or create a ticket in trac. >> > >> > Thank you >> >
> I'm having a problem with Puppet as a result of this upgrade. Clients > and server are both at 2.7.10. I see this in /var/log on the clients: > > puppet-agent[17028]: Could not retrieve catalog from remote server: > Error 400 on SERVER: No support for http method POST > > Running 'puppet agent --test' on a client, I see this: > > > # puppet agent --test
> info: Caching catalog for host02.example.com <http://host02.example.com>
Prentice Bisbal
I'm using puppet with Passenger. Just noticed on my server that it
looked like puppetmaster might have been started outside of apache
('service puppet start' instead of 'service httpd start'). Does the
puppetmaster RPM blindly something like 'service puppetmaster restart'
after an upgrade?
My error message from 'puppet agent --test' in my original e-mail was
incorrect. That one showed the error after I loosened the file
restrictions on the /tmp/passenger... directory. Before doing that (And
before turning off SELinux). I was seeing this at the bottom of my
'puppet agent --test' output, which didn't make it into the previous
post, but should have, since this is the important error message:
err: Could not send report: Error 500 on SERVER: <!DOCTYPE HTML PUBLIC
"-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.2.15 (Red Hat) Server at puppet Port 8140</address>
</body></html>
err: Could not remove PID file /var/run/puppet/agent.pid
After further poking around, it looks like this is an SELinux error:
Feb 2 13:52:42 puppetmaster setroubleshoot: SELinux is preventing
/usr/sbin/httpd from read access on the file temp.lWrmuM. For complete
SELinux messages. run sealert -l fde59db8-7d09-4075-baf5-212407840dcd
Prentice Bisbal
> Prentice,
> You stated that puppet and puppetmaster installed RPMs are 2.7.10, and you said you restarted httpd on the puppet master. However, you never said you restarted puppetd on the clients.
> Now, given this output:
> "
> # puppet agent --test
> > info: Applying configuration version '1328204765'
> ...
> version:
> > - config: 1328204723
> > + config: 1328204765
> > puppet: 2.7.10
> "
> It looks as if 2.7.10 does indeed work just fine, when run from the command line. However, your failure logs come from /var/log, which is the output of the puppet daemon. In this case, it seems to be that puppetd on the clients has never been restarted, so is still running as 2.6. Humor me and give "service puppet restart" a try on the clients... that fixed this exact problem for me a couple of days ago.
The errors were being reproduced on the command-line, so they were occuring with 2.7.10, not just the running daemon. See my
most recent e-mail on this thread I accidently posted the output of 'puppet agent --test' AFTER I turned off SELinux and
loosened the permissions on the /tmp dir, so there weren't any errors in the output I posted. I did do 'service puppet restart'
several times in the course of my troubleshooting.
This discussion is moot at this point - I tracked it down to an SELinux error. Damn Passenger!
>> > info: Caching catalog for host02.example.com <http://host02.example.com> <http://host02.example.com>
Thomas Uphill
> most recent e-mail on this thread I accidently posted the output of 'puppet agent --test' AFTER I turned off SELinux and
> loosened the permissions on the /tmp dir, so there weren't any errors in the output I posted. I did do 'service puppet restart'
> several times in the course of my troubleshooting.
>
> This discussion is moot at this point - I tracked it down to an SELinux error. Damn Passenger!
>
When you find the rules you need to change, let us know and we can
update the selinux module if necessary.
policy_module(puppet-passenger,0.3)
require {
type httpd_t;
type puppet_var_lib_t;
type puppet_var_run_t;
type puppet_log_t;
type puppet_port_t;
type lib_t;
type httpd_tmp_t;
type port_t;
}
allow httpd_t puppet_var_lib_t:dir rw_dir_perms;
allow httpd_t puppet_var_lib_t:file manage_file_perms;
allow httpd_t puppet_var_run_t:dir {search getattr};
allow httpd_t puppet_log_t:dir rw_dir_perms;
allow httpd_t puppet_log_t:file rw_file_perms;
allow httpd_t puppet_log_t:file create_file_perms;
allow httpd_t puppet_log_t:file setattr;
allow httpd_t puppet_port_t:tcp_socket name_bind;
allow httpd_t lib_t:file execute_no_trans;
allow httpd_t httpd_tmp_t:sock_file rw_sock_file_perms;
allow httpd_t httpd_tmp_t:sock_file {create unlink setattr};
allow httpd_t self:capability { fowner fsetid sys_ptrace };
allow httpd_t port_t:udp_socket name_bind;
Prentice Bisbal
On 02/02/2012 02:12 PM, Thomas Uphill wrote:
>> The errors were being reproduced on the command-line, so they were occuring with 2.7.10, not just the running daemon. See my
>> most recent e-mail on this thread I accidently posted the output of 'puppet agent --test' AFTER I turned off SELinux and
>> loosened the permissions on the /tmp dir, so there weren't any errors in the output I posted. I did do 'service puppet restart'
>> several times in the course of my troubleshooting.
>>
>> This discussion is moot at this point - I tracked it down to an SELinux error. Damn Passenger!
>>
>
> When you find the rules you need to change, let us know and we can
> update the selinux module if necessary.
>
Thomas,
Will do. I just updated my SELinux policy, but I'm still seeing some
denials. I'll let them collect in my audit log for a while.
Prentice
Brian Epstein
Hash: SHA1
On 01/31/2012 04:27 PM, Thomas Uphill wrote:
> Puppet has been upgraded from 2.6.12 to 2.7.10. ?There are language
> differences between 2.6 and 2.7 ?as outlined in the release notes
> http://projects.puppetlabs.com/projects/puppet/wiki/Release_Notes
Thomas,
What is the best way to ensure puppet stays at 2.6 in our manifests for
the time being? Would something like this work?
package {
"puppet":
ensure => '2.6',
}
Thanks,
ep
- --
Brian Epstein <beps...@ias.edu> +1 609-734-8179
Manager, Network and Security Institute for Advanced Study
Key fingerprint = 128A 38F4 4CFA 5EDB 99CE 4734 6117 4C25 0371 C12A
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAk8v/x4ACgkQYRdMJQNxwSrO0ACgxo1xFcd5v7oeiaV1OMIbvmii
skMAnjASKg4sdNqeyOIz9qsOec3jmrRH
=Ukb3
-----END PGP SIGNATURE-----
Benjamin Rose
Since the default provider for the type "package" on a rhel-like
system is yum, it should be possible to do something akin to:
package {
"puppet.2.6.12.x86_64":
ensure => present;
}
which will run a command similar to "yum -y install $name".
You will still have to be careful to not overwrite during normal
updates, but this can be worked around in other ways.
Ben
Brian Epstein
Hash: SHA1
On 02/06/2012 11:49 AM, Benjamin Rose wrote:
> Since the default provider?for the type "package" on a rhel-like
> system is yum, it should be possible to do something akin to:
>
> package {
> "puppet.2.6.12.x86_64":
> ensure => present;
> }
Ben,
Thanks for your response. Originally, I tried setting the package name
to "puppet-2.6", and it didn't like that at all. I'll try your
suggestion to see if it works instead.
Otherwise, I might just force the version in the repository.
Alternatively, I'll stop wasting time and just update my manifests to 2.7 :)
Thanks,
ep
- --
Brian Epstein <beps...@ias.edu> +1 609-734-8179
Manager, Network and Security Institute for Advanced Study
Key fingerprint = 128A 38F4 4CFA 5EDB 99CE 4734 6117 4C25 0371 C12A
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAk8wBQYACgkQYRdMJQNxwSqj8QCeNj8tTtjjTrTE2lLLH0EIs+4W
S4UAn1gO5f1oUXKGGLnMcTTlXCl8n5me
=3+kW
-----END PGP SIGNATURE-----
Prentice Bisbal
On 02/06/2012 11:26 AM, Brian Epstein wrote:
> On 01/31/2012 04:27 PM, Thomas Uphill wrote:
> > Puppet has been upgraded from 2.6.12 to 2.7.10. ?There are language
> > differences between 2.6 and 2.7 ?as outlined in the release notes
> > http://projects.puppetlabs.com/projects/puppet/wiki/Release_Notes
>
> Thomas,
>
> What is the best way to ensure puppet stays at 2.6 in our
> manifests for
> the time being? Would something like this work?
>
> package {
> "puppet":
> ensure => '2.6',
> }
>
> Thanks,
> ep
>
Brian,
I'm pretty sure you need to match the version exactly or use a regular
expressions.
Prentice
Josko Plazonic
> What is the best way to ensure puppet stays at 2.6 in our manifests for
> the time being? Would something like this work?
>
> package {
> "puppet":
> ensure => '2.6',
> }
I wouldn't do it that way - it might only work if you do not have
other update mechanisms and run updates only through puppet (which
would be an unusual way to do things). Instead push a rule for augeas
to exclude puppet from updates - this has not been tested but should
be approximately correct:
augeas { "Do not update puppet from unsupported repo":
context => "/files/etc/yum.repos.d/puias-6-unsupported.repo",
changes=> "set PUIAS_6_unsupported/exclude puppet"
}
Josko
Thomas Uphill
> augeas { "Do not update puppet from unsupported repo":
> context => "/files/etc/yum.repos.d/puias-6-unsupported.repo",
> changes=> "set PUIAS_6_unsupported/exclude puppet"
> }
However, I'm not sure why you wouldn't just update? Nothing should
break and you will only be issued with warnings for deprecations.
Maybe I'm wrong?
--Thomas.
Brian Epstein
Hash: SHA1
On 02/06/2012 12:47 PM, Josko Plazonic wrote:
> augeas { "Do not update puppet from unsupported repo":
> context => "/files/etc/yum.repos.d/puias-6-unsupported.repo",
> changes=> "set PUIAS_6_unsupported/exclude puppet"
> }
This covers the more general case, I like it. Only question is, what
will that do for the initial puppet install on new builds. I'll try it out.
Thanks,
ep
- --
Brian Epstein <beps...@ias.edu> +1 609-734-8179
Manager, Network and Security Institute for Advanced Study
Key fingerprint = 128A 38F4 4CFA 5EDB 99CE 4734 6117 4C25 0371 C12A
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAk8wJxcACgkQYRdMJQNxwSphxwCfZZEWaQnO9kcifjyPNUPMtp2v
49IAoImFD72JB2yLufbkTuRLyiaqif85
=ugUX
-----END PGP SIGNATURE-----
Brian Epstein
Hash: SHA1
On 02/06/2012 01:57 PM, Thomas Uphill wrote:
> However, I'm not sure why you wouldn't just update? Nothing should
> break and you will only be issued with warnings for deprecations.
My assumptions about incompatibility come from your initial post. Also,
our puppetmaster is a RHEL5 build, so I have to update it before the
clients.
I just figured a short term block would prevent immediate headaches
until we get a chance to update everything to 2.7.
Thanks,
ep
- --
Brian Epstein <beps...@ias.edu> +1 609-734-8179
Manager, Network and Security Institute for Advanced Study
Key fingerprint = 128A 38F4 4CFA 5EDB 99CE 4734 6117 4C25 0371 C12A
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAk8wJ74ACgkQYRdMJQNxwSoLuwCfXzL0kTUvdNLf1uSSwgpc2ZKM
7eoAoJlMMoME1aJpxSzYVltzKkLZPdGN
=X2XZ
-----END PGP SIGNATURE-----
Prentice Bisbal
non-issue.
On 02/06/2012 01:57 PM, Thomas Uphill wrote:
Benjamin Rose
but if you are using the official repos for a new install and running
puppet in %post, 2.7 will already be installed. You can still exclude
it from the repos in yum.repos.d during kickstart's %pre, but you'll
then need another local repo or some sort of mechanism for
distributing only the 2.6 version. If you maintain a local mirror,
just exclude puppet from the rsync or reposync and remove all 2.7
rpms.
In addition, there is a native yumrepo{} type for puppet you can
leverage for creating a new repo. You can also use this mechanism to
disable puppet from the official repos if you don't want to mess with
augeas.
+1 to what everyone else has said so far, the language differences are
minor. My installation, though currently pretty small, actually didn't
need any manifest modifications. The trickiest part is getting
everything up to 2.7 at the same time. Func or mcollective can help
with this.
Ben
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 02/06/2012 12:47 PM, Josko Plazonic wrote:
> > augeas { "Do not update puppet from unsupported repo":
> > context => "/files/etc/yum.repos.d/puias-6-unsupported.repo",
> > changes=> "set PUIAS_6_unsupported/exclude puppet"
> > }
This covers the more general case, I like it. Only question is, what
will that do for the initial puppet install on new builds. I'll try it out.
Thanks,
ep
- --
Brian Epstein <beps...@ias.edu> +1 609-734-8179
Manager, Network and Security Institute for Advanced Study
Key fingerprint = 128A 38F4 4CFA 5EDB 99CE 4734 6117 4C25 0371 C12A
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAk8wJxcACgkQYRdMJQNxwSphxwCfZZEWaQnO9kcifjyPNUPMtp2v
Josko Plazonic
On Mon, Feb 6, 2012 at 14:36, Benjamin Rose <pu...@allmybase.com> wrote:
> +1 to what everyone else has said so far, the language differences are
> minor. My installation, though currently pretty small, actually didn't
> need any manifest modifications. The trickiest part is getting
> everything up to 2.7 at the same time. Func or mcollective can help
> with this.
There should be no need to update everything at the same time. With
puppet what's important is that the puppet server version is >= puppet
client version. So if you update the server you can take time to
update clients.
Josko P.