this is really weird. i've upgraded test machine to 2.6 and started
getting those. virtually any file pulled from puppetmaster has zero
size. it's even true for internal objects like custom functions and
facts. whenever puppet decides that file has actually changed (this
logic works fine) it pulls a new file but somehow ends up with empty
object. see (please compare md5s):
<0>root@judy(1)~$ puppetd --test
notice: Ignoring --listen on onetime run
info: Retrieving plugin
info: /File[/var/lib/puppet/lib]: Storing newly-audited value for
content
info: /File[/var/lib/puppet/lib/facter]: Storing newly-audited value
for content
info: /File[/var/lib/puppet/lib/puppet]: Storing newly-audited value
for content
info: /File[/var/lib/puppet/lib/puppet/parser]: Storing newly-audited
value for content
info: /File[/var/lib/puppet/lib/puppet/parser/functions]: Storing
newly-audited value for content
notice: /File[/var/lib/puppet/lib/puppet/parser/functions/vlanfind.rb]/
content: content changed '{md5}d41d8cd98f00b204e9800998ecf8427e' to
'{md5}2ff6a0929541c75de18fa51ad420fdca'
info: /File[/var/lib/puppet/lib/puppet/provider]: Storing newly-
audited value for content
info: /File[/var/lib/puppet/lib/puppet/provider/sysctl]: Storing newly-
audited value for content
info: /File[/var/lib/puppet/lib/puppet/type]: Storing newly-audited
value for content
info: Loading downloaded plugin /var/lib/puppet/lib/puppet/parser/
functions/vlanfind.rb
notice: Caught INT; calling stop
<0>root@judy(2)~$ cat /var/lib/puppet/lib/puppet/parser/functions/
vlanfind.rb
<0>root@judy(3)~$ md5sum /var/lib/puppet/lib/puppet/parser/functions/
vlanfind.rb
d41d8cd98f00b204e9800998ecf8427e /var/lib/puppet/lib/puppet/parser/
functions/vlanfind.rb
<0>root@judy(4)~$ >/tmp/EMPTY
<0>root@judy(5)~$ md5sum /tmp/EMPTY
d41d8cd98f00b204e9800998ecf8427e /tmp/EMPTY
i'm talking to passenger puppetmaster seated on localhost.
On Mon, 2010-08-02 at 12:15 -0700, asq wrote: > this is really weird. i've upgraded test machine to 2.6 and started > getting those. virtually any file pulled from puppetmaster has zero > size. it's even true for internal objects like custom functions and > facts. whenever puppet decides that file has actually changed (this > logic works fine) it pulls a new file but somehow ends up with empty > object. see (please compare md5s):
> <0>root@judy(1)~$ puppetd --test > notice: Ignoring --listen on onetime run > info: Retrieving plugin > info: /File[/var/lib/puppet/lib]: Storing newly-audited value for > content > info: /File[/var/lib/puppet/lib/facter]: Storing newly-audited value > for content > info: /File[/var/lib/puppet/lib/puppet]: Storing newly-audited value > for content > info: /File[/var/lib/puppet/lib/puppet/parser]: Storing newly-audited > value for content > info: /File[/var/lib/puppet/lib/puppet/parser/functions]: Storing > newly-audited value for content > notice: /File[/var/lib/puppet/lib/puppet/parser/functions/vlanfind.rb]/ > content: content changed '{md5}d41d8cd98f00b204e9800998ecf8427e' to > '{md5}2ff6a0929541c75de18fa51ad420fdca' > info: /File[/var/lib/puppet/lib/puppet/provider]: Storing newly- > audited value for content > info: /File[/var/lib/puppet/lib/puppet/provider/sysctl]: Storing newly- > audited value for content > info: /File[/var/lib/puppet/lib/puppet/type]: Storing newly-audited > value for content > info: Loading downloaded plugin /var/lib/puppet/lib/puppet/parser/ > functions/vlanfind.rb > notice: Caught INT; calling stop > <0>root@judy(2)~$ cat /var/lib/puppet/lib/puppet/parser/functions/ > vlanfind.rb > <0>root@judy(3)~$ md5sum /var/lib/puppet/lib/puppet/parser/functions/ > vlanfind.rb > d41d8cd98f00b204e9800998ecf8427e /var/lib/puppet/lib/puppet/parser/ > functions/vlanfind.rb > <0>root@judy(4)~$ >/tmp/EMPTY > <0>root@judy(5)~$ md5sum /tmp/EMPTY > d41d8cd98f00b204e9800998ecf8427e /tmp/EMPTY
> i'm talking to passenger puppetmaster seated on localhost.
Since 2.6, puppet now streams the file content through rack. It is well possible you're hitting a bug with this new feature.
Can you let us know what version of passenger and rack you are using. It would also be worth upgrading those versions to see if that helps.
Then, you should also run your master with --debug --trace, to see if the master logs anything interesting during one of this specific file transfer. -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org!
On 3 Sie, 09:19, Brice Figureau <brice-pup...@daysofwonder.com> wrote:
> Since 2.6, puppet now streams the file content through rack. It is well
> possible you're hitting a bug with this new feature.
> Can you let us know what version of passenger and rack you are using. It
> would also be worth upgrading those versions to see if that helps.
> Then, you should also run your master with --debug --trace, to see if
> the master logs anything interesting during one of this specific file
> transfer.
On Tue, 2010-08-03 at 03:09 -0700, asq wrote: > On 3 Sie, 09:19, Brice Figureau <brice-pup...@daysofwonder.com> wrote:
> > Since 2.6, puppet now streams the file content through rack. It is well > > possible you're hitting a bug with this new feature.
> > Can you let us know what version of passenger and rack you are using. It > > would also be worth upgrading those versions to see if that helps.
> > Then, you should also run your master with --debug --trace, to see if > > the master logs anything interesting during one of this specific file > > transfer.
> same thing with both ruby and enterprise ruby interpreters.
Ruby version shouldn't matter, I more think this is a puppet rack versioning issue.
Would it be possible to test rack 1.1.0 (which is what I tried when adding this feature)?
Do you have any interesting --debug --trace log on the master side? -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org!
> On 3 Sie, 09:19, Brice Figureau <brice-pup...@daysofwonder.com> wrote:
>> Since 2.6, puppet now streams the file content through rack. It is well >> possible you're hitting a bug with this new feature.
>> Can you let us know what version of passenger and rack you are using. It >> would also be worth upgrading those versions to see if that helps.
>> Then, you should also run your master with --debug --trace, to see if >> the master logs anything interesting during one of this specific file >> transfer.
> On 3 Sie, 09:19, Brice Figureau <brice-pup...@daysofwonder.com> wrote:
> > Since 2.6, puppet now streams the file content through rack. It is well
> > possible you're hitting a bug with this new feature.
> > Can you let us know what version of passenger and rack you are using. It
> > would also be worth upgrading those versions to see if that helps.
> > Then, you should also run your master with --debug --trace, to see if
> > the master logs anything interesting during one of this specific file
> > transfer.
On Tue, 2010-08-03 at 03:22 -0700, asq wrote: > On 3 Sie, 12:09, asq <asqu...@gmail.com> wrote: > > On 3 Sie, 09:19, Brice Figureau <brice-pup...@daysofwonder.com> wrote:
> > > Since 2.6, puppet now streams the file content through rack. It is well > > > possible you're hitting a bug with this new feature.
> > > Can you let us know what version of passenger and rack you are using. It > > > would also be worth upgrading those versions to see if that helps.
> > > Then, you should also run your master with --debug --trace, to see if > > > the master logs anything interesting during one of this specific file > > > transfer.
Can you try the latest 2.6.1rc1 and confirm that it solves your issue? Or alternatively try what Thorsten Biel suggested in another e-mail in this thread (ie getting rid of rack 1.0.1)?
-- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org!
> Can you try the latest 2.6.1rc1 and confirm that it solves your issue?
> Or alternatively try what Thorsten Biel suggested in another e-mail in
> this thread (ie getting rid of rack 1.0.1)?
[Tue Aug 03 16:25:16 2010] [error] [client 10.100.101.15] Premature
end of script headers: vlanfind.rb
[ pid=3397 file=ext/apache2/Hooks.cpp:711 time=2010-08-03
16:25:16.910 ]:
The backend application (process 3608) didn't send a valid HTTP
response. It might have crashed during the middle of sending an HTTP
response, so please check whether there are crashing problems in your
application. This is the data that it sent: [Content-Type]
*** Exception NoMethodError in
PhusionPassenger::Rack::ApplicationSpawner (undefined method `each'
for 4741:Fixnum) (process 3608):
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/request_handler.rb:103:in `process_request'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/request_handler.rb:99:in `each'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/request_handler.rb:99:in `process_request'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/abstract_request_handler.rb:207:in `main_loop'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/application_spawner.rb:118:in `run'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/application_spawner.rb:69:in
`spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/utils.rb:184:in `safe_fork'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/application_spawner.rb:62:in
`spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/application_spawner.rb:45:in
`spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/spawn_manager.rb:159:in `spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/spawn_manager.rb:287:in `handle_spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/abstract_server.rb:352:in `__send__'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/abstract_server.rb:352:in `main_loop'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/abstract_server.rb:196:in `start_synchronously'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/bin/passenger-
spawn-server:61
> > Can you try the latest 2.6.1rc1 and confirm that it solves your issue? > > Or alternatively try what Thorsten Biel suggested in another e-mail in > > this thread (ie getting rid of rack 1.0.1)?
i've tried 2.2.11/1.2.1 combo and unfortunately results are pretty
much the same.
i'll try 2.6.1rc1 tomorrow, maybe there were some other fixes on
passenger.
if there is no success what is current best practice to run large
scale puppetmasters?
(so far we were quite happy on passenger and enterprise-ruby memory
management tweaks)
> i've tried 2.2.11/1.2.1 combo and unfortunately results are pretty > much the same. > i'll try 2.6.1rc1 tomorrow, maybe there were some other fixes on > passenger. > if there is no success what is current best practice to run large > scale puppetmasters? > (so far we were quite happy on passenger and enterprise-ruby memory > management tweaks)
Passenger is still the preferred way to scale a puppetmaster (until I progress on the JRuby front).
Unfortunately rack or passenger tends to break on each new release (and it is worst in case of Puppet _and_ rack upgrades :))
The best to fix this issue is to collect the more log possible (including stack trace) so that we can understand what's going wrong.
Did someone file a puppet bug about this issue? -- Brice Figureau My Blog: http://www.masterzen.fr/
> Passenger is still the preferred way to scale a puppetmaster (until I > progress on the JRuby front).
I've had great success running Puppet 0.25.4, 0.25.5, and 2.6.0 under Nginx and Unicorn. Alas, http://projects.puppetlabs.com/issues/4319 is still in play. I use the highly suspect "give up and use wget" workaround.
>> Passenger is still the preferred way to scale a puppetmaster (until I >> progress on the JRuby front).
> I've had great success running Puppet 0.25.4, 0.25.5, and 2.6.0 under > Nginx and Unicorn. Alas, http://projects.puppetlabs.com/issues/4319 > is still in play. I use the highly suspect "give up and use wget" > workaround.
Use either 2.6.1rc1 or my single-line patch linked to the bug report to get rid of the issue :) IMHO it's better than refactoring your manifests to wget stuff. -- Brice Figureau My Blog: http://www.masterzen.fr/
I'm getting the same error with a new custom fact in <module>/lib. The puppetmaster is running on Apache2 + Passenger (setup works fine in production with 0.24.8).
With Puppet 2.6.0 (Debian package) the file is created empty. With Puppet 2.6.0 + patch from Brice, I get a 'Could not set file on ensure: Error 500' on the client side and the apache logs on the server shows:
[Wed Aug 04 23:01:46 2010] [error] [client 192.168.200.199] Premature end of script headers: vzbackup.rb [ pid=28420 file=ext/apache2/Hooks.cpp:735 time=2010-08-04 23:01:46.242 ]: The backend application (process 28465) didn't send a valid HTTP response. It might have crashed during the middle of sending an HTTP response, so please check whether there are crashing problems in your application. This is the data that it sent: [Content-Type] *** Exception NoMethodError in PhusionPassenger::Rack::ApplicationSpawner (undefined method `each' for 1268:Fixnum) (process 28465): from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/request_ handler.rb:100:in `process_request' from /var/lib/gems/1.8/gems/rack-1.2.1/lib/rack/utils.rb:305:in `each' from /var/lib/gems/1.8/gems/rack-1.2.1/lib/rack/utils.rb:304:in `each' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/request_ handler.rb:96:in `process_request' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/abstract_requ est_handler.rb:207:in `main_loop' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/applicat ion_spawner.rb:120:in `run' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/applicat ion_spawner.rb:65:in `spawn_application' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/utils.rb:252: in `safe_fork' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/applicat ion_spawner.rb:58:in `spawn_application' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/applicat ion_spawner.rb:41:in `spawn_application' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/spawn_manager .rb:150:in `spawn_application' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/spawn_manager .rb:278:in `handle_spawn_application' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/abstract_serv er.rb:352:in `__send__' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/abstract_serv er.rb:352:in `main_loop' from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/abstract_serv er.rb:196:in `start_synchronously' from /var/lib/gems/1.8/gems/passenger-2.2.15/bin/passenger-spawn-server:61
Same thing with 2.6.1rc1 from git unfortunately...
Apart from this, the manifests are compiled and run fine, and the facts that already exist on the client are evaluated correctly.
The server I use for testing is running Ubuntu 10.04, the client is 8.04. I've tried various combinations of Passenger (2.2.7, 2.2.11, 2.2.15) and Rack (1.1.0, 1.2.1), from gems and .deb, without success.
>> Passenger is still the preferred way to scale a puppetmaster (until I >> progress on the JRuby front).
> I've had great success running Puppet 0.25.4, 0.25.5, and 2.6.0 under > Nginx and Unicorn. Alas, http://projects.puppetlabs.com/issues/4319 > is still in play. I use the highly suspect "give up and use wget" > workaround.
Use either 2.6.1rc1 or my single-line patch linked to the bug report to get rid of the issue :) IMHO it's better than refactoring your manifests to wget stuff. -- Brice Figureau My Blog: http://www.masterzen.fr/
-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Wed, 2010-08-04 at 23:34 +1000, Nicolas Brisac wrote: > I'm getting the same error with a new custom fact in <module>/lib. > The puppetmaster is running on Apache2 + Passenger (setup works fine > in production with 0.24.8).
> With Puppet 2.6.0 (Debian package) the file is created empty. > With Puppet 2.6.0 + patch from Brice, I get a 'Could not set file on > ensure: Error 500' on the client side and the apache logs on the > server shows:
> [Wed Aug 04 23:01:46 2010] [error] [client 192.168.200.199] Premature > end of script headers: vzbackup.rb > [ pid=28420 file=ext/apache2/Hooks.cpp:735 time=2010-08-04 > 23:01:46.242 ]: > The backend application (process 28465) didn't send a valid HTTP > response. It might have crashed during the middle of sending an HTTP > response, so please check whether there are crashing problems in your > application. This is the data that it sent: [Content-Type] > *** Exception NoMethodError in > PhusionPassenger::Rack::ApplicationSpawner (undefined method `each' > for 1268:Fixnum) (process 28465):
OK, I just had a look to the passenger code and this is insane. I think the problem is that we return the content-length as an int, but passenger chokes if it gets anything except a string.
Can you try this (untested) patch:
diff --git a/lib/puppet/network/http/rack/rest.rb b/lib/puppet/network/http/rack/rest.rb index e5f50c4..b7e1d97 100644 --- a/lib/puppet/network/http/rack/rest.rb +++ b/lib/puppet/network/http/rack/rest.rb @@ -41,7 +41,7 @@ class Puppet::Network::HTTP::RackREST < Puppet::Network::HTTP::RackHttpHandler unless result.is_a?(File) response.write result else - response["Content-Length"] = result.stat.size + response["Content-Length"] = result.stat.size.to_s response.body = RackFile.new(result) end end
Let me know if that fixes the problem, -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org!
On Wed, 2010-08-04 at 23:34 +1000, Nicolas Brisac wrote: > I'm getting the same error with a new custom fact in <module>/lib. > The puppetmaster is running on Apache2 + Passenger (setup works fine > in production with 0.24.8).
> With Puppet 2.6.0 (Debian package) the file is created empty. > With Puppet 2.6.0 + patch from Brice, I get a 'Could not set file on > ensure: Error 500' on the client side and the apache logs on the > server shows:
> [Wed Aug 04 23:01:46 2010] [error] [client 192.168.200.199] Premature > end of script headers: vzbackup.rb > [ pid=28420 file=ext/apache2/Hooks.cpp:735 time=2010-08-04 > 23:01:46.242 ]: > The backend application (process 28465) didn't send a valid HTTP > response. It might have crashed during the middle of sending an HTTP > response, so please check whether there are crashing problems in your > application. This is the data that it sent: [Content-Type] > *** Exception NoMethodError in > PhusionPassenger::Rack::ApplicationSpawner (undefined method `each' > for 1268:Fixnum) (process 28465):
OK, I just had a look to the passenger code and this is insane. I think the problem is that we return the content-length as an int, but passenger chokes if it gets anything except a string.
Can you try this (untested) patch:
diff --git a/lib/puppet/network/http/rack/rest.rb b/lib/puppet/network/http/rack/rest.rb index e5f50c4..b7e1d97 100644 --- a/lib/puppet/network/http/rack/rest.rb +++ b/lib/puppet/network/http/rack/rest.rb @@ -41,7 +41,7 @@ class Puppet::Network::HTTP::RackREST < Puppet::Network::HTTP::RackHttpHandler unless result.is_a?(File) response.write result else - response["Content-Length"] = result.stat.size + response["Content-Length"] = result.stat.size.to_s response.body = RackFile.new(result) end end
Let me know if that fixes the problem, -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org!
-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
In my site manifest, I have a few webmail servers which have $class =
'webmail' in them and a few which don't have any $class set to
something else. Before I migrated to the nginx+mongrel setup (and
after I reverted back from it), for the nodes that have a $class !=
'webmail', the default imapproxy.conf was being served, but now, I'm
served an emtpy file instead.
The hosts with $class = 'webmail' face no such problem - they receive
the file they were supposed to receive.
($project is set to 'projectA' for all nodes in the manifest.)
I gave this another attempt recently with 2.6.2rc1, nginx, and thin
via config.ru. (not nginx + mongrels like earlier)
Now I know a little more about the problem than I did a month ago:
I use a lot of variables such as $project in my file path array I set
for the source parameter for file type resources:
I get the correct file served when I use the stock WEBrick/
puppetmasterd and I don't get the correct file served with nginx+thin
+config.ru unless I do:
node bar {
$project = "blah"
include imapproxy
}
FWIW, my server-side logs don't show me the $project part in the GETs
at all - the $hostname (and other facts are) is shown.
> In my site manifest, I have a few webmail servers which have $class =
> 'webmail' in them and a few which don't have any $class set to
> something else. Before I migrated to the nginx+mongrel setup (and
> after I reverted back from it), for the nodes that have a $class !=
> 'webmail', the default imapproxy.conf was being served, but now, I'm
> served an emtpy file instead.
> The hosts with $class = 'webmail' face no such problem - they receive
> the file they were supposed to receive.
> ($project is set to 'projectA' for all nodes in the manifest.)
I suspect that's what I've been seeing with 2.6.(0|1) + unicorn + nginx + 2.6.0 clients and also with 2.6.(0|1) + passenger + apache.
Glad (FAVO 'glad', obviously) it's not just me.
-- John Hawkes-Reed
-- Future Publishing Limited (registered company number 2008885) and Future Publishing (Overseas) Limited (registered company number 06202940) are wholly owned subsidiaries of Future plc (registered company number 3757874). Future Publishing Limited, Future Publishing (Overseas) Limited and Future plc are all incorporated in England and Wales and share the same registered address at Beauford Court, 30 Monmouth Street, Bath BA1 2BW.
This email and any files transmitted with it are confidential. If you have received this email in error please notify the sender and then delete it immediately. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Future.
The recipient should check this email and any attachments for the presence of viruses. Future accepts no liability for any damage caused by any virus transmitted by this email.
Future may regularly and randomly monitor outgoing and incoming emails (including the content of them) and other telecommunications on its email and telecommunications systems. By replying to this email you give your consent to such monitoring.
