getting empty files from fileserver
asq
getting those. virtually any file pulled from puppetmaster has zero
size. it's even true for internal objects like custom functions and
facts. whenever puppet decides that file has actually changed (this
logic works fine) it pulls a new file but somehow ends up with empty
object. see (please compare md5s):
<0>root@judy(1)~$ puppetd --test
notice: Ignoring --listen on onetime run
info: Retrieving plugin
info: /File[/var/lib/puppet/lib]: Storing newly-audited value for
content
info: /File[/var/lib/puppet/lib/facter]: Storing newly-audited value
for content
info: /File[/var/lib/puppet/lib/puppet]: Storing newly-audited value
for content
info: /File[/var/lib/puppet/lib/puppet/parser]: Storing newly-audited
value for content
info: /File[/var/lib/puppet/lib/puppet/parser/functions]: Storing
newly-audited value for content
notice: /File[/var/lib/puppet/lib/puppet/parser/functions/vlanfind.rb]/
content: content changed '{md5}d41d8cd98f00b204e9800998ecf8427e' to
'{md5}2ff6a0929541c75de18fa51ad420fdca'
info: /File[/var/lib/puppet/lib/puppet/provider]: Storing newly-
audited value for content
info: /File[/var/lib/puppet/lib/puppet/provider/sysctl]: Storing newly-
audited value for content
info: /File[/var/lib/puppet/lib/puppet/type]: Storing newly-audited
value for content
info: Loading downloaded plugin /var/lib/puppet/lib/puppet/parser/
functions/vlanfind.rb
notice: Caught INT; calling stop
<0>root@judy(2)~$ cat /var/lib/puppet/lib/puppet/parser/functions/
vlanfind.rb
<0>root@judy(3)~$ md5sum /var/lib/puppet/lib/puppet/parser/functions/
vlanfind.rb
d41d8cd98f00b204e9800998ecf8427e /var/lib/puppet/lib/puppet/parser/
functions/vlanfind.rb
<0>root@judy(4)~$ >/tmp/EMPTY
<0>root@judy(5)~$ md5sum /tmp/EMPTY
d41d8cd98f00b204e9800998ecf8427e /tmp/EMPTY
i'm talking to passenger puppetmaster seated on localhost.
anyone has seen anything similiar?
Brice Figureau
Since 2.6, puppet now streams the file content through rack. It is well
possible you're hitting a bug with this new feature.
Can you let us know what version of passenger and rack you are using. It
would also be worth upgrading those versions to see if that helps.
Then, you should also run your master with --debug --trace, to see if
the master logs anything interesting during one of this specific file
transfer.
--
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!
asq
> Since 2.6, puppet now streams the file content through rack. It is well
> possible you're hitting a bug with this new feature.
>
> Can you let us know what version of passenger and rack you are using. It
> would also be worth upgrading those versions to see if that helps.
>
> Then, you should also run your master with --debug --trace, to see if
> the master logs anything interesting during one of this specific file
> transfer.
passenger 2.2.7
ruby:
ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux]
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux], MBARI 0x6770,
Ruby Enterprise Edition 20090928
same thing with both ruby and enterprise ruby interpreters.
Brice Figureau
Ruby version shouldn't matter, I more think this is a puppet rack
versioning issue.
Would it be possible to test rack 1.1.0 (which is what I tried when
adding this feature)?
Do you have any interesting --debug --trace log on the master side?
Thorsten Biel
Hi,
Try deleting the old Rack version (gem cleanup).
Had a similar problem, it looks as if rack 1.0.1 doesn't play nice with Puppet 2.6.
Cheers,
Thorsten
asq
Could not find file_content for '/plugins/puppet/parser/functions/
template_search.rb'
see logs: http://pastie.org/private/tx8intebzefhymcu5miqw
config: http://pastie.org/private/7eugfclhbhsdwuq5gj6za
file exists: http://pastie.org/private/ms67m4z4rhzdsucpawfja
asq
> Ruby version shouldn't matter, I more think this is a puppet rack
> versioning issue.
>
> Would it be possible to test rack 1.1.0 (which is what I tried when
> adding this feature)?
> Do you have any interesting --debug --trace log on the master side?
Brice Figureau
I think you're seeing bug #4319:
http://projects.puppetlabs.com/issues/4319
Can you try the latest 2.6.1rc1 and confirm that it solves your issue?
Or alternatively try what Thorsten Biel suggested in another e-mail in
this thread (ie getting rid of rack 1.0.1)?
asq
> I think you're seeing bug #4319:http://projects.puppetlabs.com/issues/4319
>
> Can you try the latest 2.6.1rc1 and confirm that it solves your issue?
> Or alternatively try what Thorsten Biel suggested in another e-mail in
> this thread (ie getting rid of rack 1.0.1)?
master/4319) i'm getting passenger error:
[Tue Aug 03 16:25:16 2010] [error] [client 10.100.101.15] Premature
end of script headers: vlanfind.rb
[ pid=3397 file=ext/apache2/Hooks.cpp:711 time=2010-08-03
16:25:16.910 ]:
The backend application (process 3608) didn't send a valid HTTP
response. It might have crashed during the middle of sending an HTTP
response, so please check whether there are crashing problems in your
application. This is the data that it sent: [Content-Type]
*** Exception NoMethodError in
PhusionPassenger::Rack::ApplicationSpawner (undefined method `each'
for 4741:Fixnum) (process 3608):
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/request_handler.rb:103:in `process_request'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/request_handler.rb:99:in `each'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/request_handler.rb:99:in `process_request'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/abstract_request_handler.rb:207:in `main_loop'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/application_spawner.rb:118:in `run'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/application_spawner.rb:69:in
`spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/utils.rb:184:in `safe_fork'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/application_spawner.rb:62:in
`spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/rack/application_spawner.rb:45:in
`spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/spawn_manager.rb:159:in `spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/spawn_manager.rb:287:in `handle_spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/abstract_server.rb:352:in `__send__'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/abstract_server.rb:352:in `main_loop'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/lib/
phusion_passenger/abstract_server.rb:196:in `start_synchronously'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/bin/passenger-
spawn-server:61
rack 1.1.0
Brice Figureau
> On 3 Sie, 13:01, Brice Figureau <brice-pup...@daysofwonder.com> wrote:
>
> > I think you're seeing bug #4319:http://projects.puppetlabs.com/issues/4319
> >
> > Can you try the latest 2.6.1rc1 and confirm that it solves your issue?
> > Or alternatively try what Thorsten Biel suggested in another e-mail in
> > this thread (ie getting rid of rack 1.0.1)?
>
> with patch applied (http://github.com/masterzen/puppet/tree/tickets/
> master/4319) i'm getting passenger error:
Which means it now tries to send the file. You just hit another issue
which is discussed in this thread:
http://groups.google.com/group/puppet-users/browse_thread/thread/543510e15f06074c#
asq
> Which means it now tries to send the file. You just hit another issue
i've tried 2.2.11/1.2.1 combo and unfortunately results are pretty
much the same.
i'll try 2.6.1rc1 tomorrow, maybe there were some other fixes on
passenger.
if there is no success what is current best practice to run large
scale puppetmasters?
(so far we were quite happy on passenger and enterprise-ruby memory
management tweaks)
Brice Figureau
Passenger is still the preferred way to scale a puppetmaster (until I
progress on the JRuby front).
Unfortunately rack or passenger tends to break on each new release (and
it is worst in case of Puppet _and_ rack upgrades :))
The best to fix this issue is to collect the more log possible
(including stack trace) so that we can understand what's going wrong.
Did someone file a puppet bug about this issue?
--
Brice Figureau
My Blog: http://www.masterzen.fr/
Richard Crowley
> progress on the JRuby front).
I've had great success running Puppet 0.25.4, 0.25.5, and 2.6.0 under
Nginx and Unicorn. Alas, http://projects.puppetlabs.com/issues/4319
is still in play. I use the highly suspect "give up and use wget"
workaround.
Brice Figureau
Use either 2.6.1rc1 or my single-line patch linked to the bug report to
get rid of the issue :)
IMHO it's better than refactoring your manifests to wget stuff.
Nicolas Brisac
The puppetmaster is running on Apache2 + Passenger (setup works fine in production with 0.24.8).
With Puppet 2.6.0 (Debian package) the file is created empty.
With Puppet 2.6.0 + patch from Brice, I get a 'Could not set file on ensure: Error 500' on the client side and the apache logs on the server shows:
[Wed Aug 04 23:01:46 2010] [error] [client 192.168.200.199] Premature end of script headers: vzbackup.rb
[ pid=28420 file=ext/apache2/Hooks.cpp:735 time=2010-08-04 23:01:46.242 ]:
The backend application (process 28465) didn't send a valid HTTP response. It might have crashed during the middle of sending an HTTP response, so please check whether there are crashing problems in your application. This is the data that it sent: [Content-Type]
*** Exception NoMethodError in PhusionPassenger::Rack::ApplicationSpawner (undefined method `each' for 1268:Fixnum) (process 28465):
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/request_handler.rb:100:in `process_request'
from /var/lib/gems/1.8/gems/rack-1.2.1/lib/rack/utils.rb:305:in `each'
from /var/lib/gems/1.8/gems/rack-1.2.1/lib/rack/utils.rb:304:in `each'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/abstract_request_handler.rb:207:in `main_loop'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/application_spawner.rb:120:in `run'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/application_spawner.rb:65:in `spawn_application'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/utils.rb:252:in `safe_fork'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/application_spawner.rb:58:in `spawn_application'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/rack/application_spawner.rb:41:in `spawn_application'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/spawn_manager.rb:150:in `spawn_application'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/spawn_manager.rb:278:in `handle_spawn_application'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/abstract_server.rb:352:in `__send__'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/abstract_server.rb:352:in `main_loop'
from /var/lib/gems/1.8/gems/passenger-2.2.15/lib/phusion_passenger/abstract_server.rb:196:in `start_synchronously'
from /var/lib/gems/1.8/gems/passenger-2.2.15/bin/passenger-spawn-server:61
Same thing with 2.6.1rc1 from git unfortunately...
Apart from this, the manifests are compiled and run fine, and the facts that already exist on the client are evaluated correctly.
The server I use for testing is running Ubuntu 10.04, the client is 8.04.
I've tried various combinations of Passenger (2.2.7, 2.2.11, 2.2.15) and Rack (1.1.0, 1.2.1), from gems and .deb, without success.
Here is my current gems list on the server:
actionmailer (2.3.8)
actionpack (2.3.8)
activerecord (2.3.8)
activeresource (2.3.8)
activesupport (2.3.8)
fastthread (1.0.7)
passenger (2.2.15)
rack (1.2.1)
rails (2.3.8)
rake (0.8.7)
I'll take any suggestion!
Thanks,
--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
> I'm getting the same error with a new custom fact in <module>/lib.
> The puppetmaster is running on Apache2 + Passenger (setup works fine
> in production with 0.24.8).
>
> With Puppet 2.6.0 (Debian package) the file is created empty.
> With Puppet 2.6.0 + patch from Brice, I get a 'Could not set file on
> ensure: Error 500' on the client side and the apache logs on the
> server shows:
>
> [Wed Aug 04 23:01:46 2010] [error] [client 192.168.200.199] Premature
> end of script headers: vzbackup.rb
> [ pid=28420 file=ext/apache2/Hooks.cpp:735 time=2010-08-04
> 23:01:46.242 ]:
> The backend application (process 28465) didn't send a valid HTTP
> response. It might have crashed during the middle of sending an HTTP
> response, so please check whether there are crashing problems in your
> application. This is the data that it sent: [Content-Type]
> *** Exception NoMethodError in
> PhusionPassenger::Rack::ApplicationSpawner (undefined method `each'
> for 1268:Fixnum) (process 28465):
OK, I just had a look to the passenger code and this is insane.
I think the problem is that we return the content-length as an int, but
passenger chokes if it gets anything except a string.
Can you try this (untested) patch:
diff --git a/lib/puppet/network/http/rack/rest.rb b/lib/puppet/network/http/rack/rest.rb
index e5f50c4..b7e1d97 100644
--- a/lib/puppet/network/http/rack/rest.rb
+++ b/lib/puppet/network/http/rack/rest.rb
@@ -41,7 +41,7 @@ class Puppet::Network::HTTP::RackREST < Puppet::Network::HTTP::RackHttpHandler
unless result.is_a?(File)
response.write result
else
- response["Content-Length"] = result.stat.size
+ response["Content-Length"] = result.stat.size.to_s
response.body = RackFile.new(result)
end
end
Let me know if that fixes the problem,
Nicolas Brisac
Merci :)
Current setup:
Puppet 2.6.0 (.deb from Sid)
Passenger 2.2.15 and Rack 1.2.1 (gems)
Cheers,
Nico
Naresh V.
My setup:
gems:
rack (1.0.1)
rails (2.3.4)
rake (0.8.7)
activerecord (2.3.8, 2.3.4)
activeresource (2.3.8, 2.3.4)
activesupport (2.3.8, 2.3.4)
mongrel (1.1.5)
RPMs:
puppet-server-2.6.0-0.7.el5 (tmz's)
I patched the files installed via the RPM to test out nginx+mongrel as
follows (taken from http://projects.puppetlabs.com/issues/4319 ):
http://pastie.org/1114966
The problem I'm facing is:
My modules have 'source' like:
source => [
"puppet:///imapproxy/$project/imapproxy.conf.
$hostname",
"puppet:///imapproxy/$project/imapproxy.conf-$class",
"puppet:///imapproxy/imapproxy.conf.$hostname",
"puppet:///imapproxy/imapproxy.conf",
],
in my files directory, I have:
/projectA/imapproxy.conf-webmail
/projectA/imapproxy.conf
In my site manifest, I have a few webmail servers which have $class =
'webmail' in them and a few which don't have any $class set to
something else. Before I migrated to the nginx+mongrel setup (and
after I reverted back from it), for the nodes that have a $class !=
'webmail', the default imapproxy.conf was being served, but now, I'm
served an emtpy file instead.
The hosts with $class = 'webmail' face no such problem - they receive
the file they were supposed to receive.
($project is set to 'projectA' for all nodes in the manifest.)
Is anybody else able to replicate this?
-Naresh.
Naresh V.
I gave this another attempt recently with 2.6.2rc1, nginx, and thin
via config.ru. (not nginx + mongrels like earlier)
Now I know a little more about the problem than I did a month ago:
I use a lot of variables such as $project in my file path array I set
for the source parameter for file type resources:
> source => [
> "puppet:///imapproxy/$project/imapproxy.conf.$hostname",
And in my site manifest, I tend to have
class foo {
$project = "blah"
include imapproxy
}
node bar {
include foo
}
I get the correct file served when I use the stock WEBrick/
puppetmasterd and I don't get the correct file served with nginx+thin
+config.ru unless I do:
node bar {
$project = "blah"
include imapproxy
}
FWIW, my server-side logs don't show me the $project part in the GETs
at all - the $hostname (and other facts are) is shown.
[2010-10-06 00:04:35] bar - - [06/Oct/2010:00:04:35 CDT] "GET /
production/file_metadata/users//sudoers.xen-3 HTTP/1.1" 404 48
[2010-10-06 00:04:35] bar - - [06/Oct/2010:00:04:35 CDT] "GET /
production/file_metadata/users//sudoers- HTTP/1.1" 404 43
[2010-10-06 00:04:35] bar - - [06/Oct/2010:00:04:35 CDT] "GET /
production/file_metadata/users//sudoers HTTP/1.1" 200 336
There should've been a "GET /production/file_metadata/users/blah/
sudoers.xen-3" etc.)
Thank you,
Naresh.
Peter Meier
> at all - the $hostname (and other facts are) is shown.
>
> [2010-10-06 00:04:35] bar - - [06/Oct/2010:00:04:35 CDT] "GET /
> production/file_metadata/users//sudoers.xen-3 HTTP/1.1" 404 48
> [2010-10-06 00:04:35] bar - - [06/Oct/2010:00:04:35 CDT] "GET /
> production/file_metadata/users//sudoers- HTTP/1.1" 404 43
> [2010-10-06 00:04:35] bar - - [06/Oct/2010:00:04:35 CDT] "GET /
> production/file_metadata/users//sudoers HTTP/1.1" 200 336
>
> There should've been a "GET /production/file_metadata/users/blah/
> sudoers.xen-3" etc.)
http://projects.puppetlabs.com/issues/4922
~pete
John Hawkes-Reed
Aha!
I suspect that's what I've been seeing with 2.6.(0|1) + unicorn + nginx +
2.6.0 clients and also with 2.6.(0|1) + passenger + apache.
Glad (FAVO 'glad', obviously) it's not just me.
--
John Hawkes-Reed
--
Future Publishing Limited (registered company number 2008885) and Future Publishing (Overseas) Limited (registered company number 06202940) are wholly owned subsidiaries of Future plc (registered company number 3757874). Future Publishing Limited, Future Publishing (Overseas) Limited and Future plc are all incorporated in England and Wales and share the same registered address at Beauford Court, 30 Monmouth Street, Bath BA1 2BW.
This email and any files transmitted with it are confidential. If you have received this email in error please notify the sender and then delete it immediately. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Future.
The recipient should check this email and any attachments for the presence of viruses. Future accepts no liability for any damage caused by any virus transmitted by this email.
Future may regularly and randomly monitor outgoing and incoming emails (including the content of them) and other telecommunications on its email and telecommunications systems. By replying to this email you give your consent to such monitoring.
*****
Save resources: think before you print.