puppet master and puppet client running very slow
sanjiv.singh
I have configured puppet 0.25.4 setup.
since last few days , puppet master and client running very slow .It
seems puppet client taking soo much time in taking configuration from
master (compilation manifest ) ,specially for those node(puppet
client ) for which large manifest class to be compiled and to be
deployed .
what can i do to over come this issue ?
Thanks,
Sanjiv Singh
Impetus Infotech
Daniel Maher
What sorts of steps have you already taken in order to investigate the
problem? Does the Puppetmaster have enough RAM? It is otherwise
overloaded from a system load perspective? What about those two factors
on the client(s)? In other words, have you eliminated all of the
standard system-based considerations?
If so, I would recommend running the client manually with test and debug
enabled, and taking a look at the output there. As well, if it's
possible, you may wish to do the same for the master daemon - that will
give you a much better idea of where along the process any potential
hangs are occurring.
Good luck !
--
Daniel Maher
� makin' plans now to live on Mars 'cuz I got Earth on lock. �
sanjiv.singh
I tried to run puppet client
/usr/sbin/puppetd --test --debug
there are more entries as following :
.
.
.
.
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
its throwing same entries for long time (in minutes), prior it take
few seconds to get catalog from server and get deployed.
and
even when i fired puppetrun on master.
it takes long time to trigger puppet client (listening to master).
what cuold be the issue?
thanks,
Sanjiv singh
Ken Barber
files that get imported)? For example - are you pulling in a large
file of all your nodes or something? I recall there has been problems
in the past with parsing large files - its fixed now though.
The obvious suggestion is to upgrade :-).
ken.
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>
sanjiv singh
i m using puppet 0.25.4 .
since last few days, client and master are very slow puppet client
taking more time in geeting configuration from puppet master.
specially for those servers for which long manifest to be deployed .
thanks,
Sanjiv Singh
Impetus Infotech
sanjiv.singh
my manifest contains number of file (flat and templete ) and for
pulling some jars on client.
prior few days , same manifest took 2-3 minutes , now its taking 20
minutes on all puppet clients.
it seems(not sure), puppet client is not bottleneck .
puppet master is an issue , bcuz all puppet client facing same
problem.
and
there is sufficient RAM and storage on puppet master.
and
puppet master hardly serving few (4-5) puppet client at a time in my
environment.
feel free to ask , if some in put require to solve issue
thanks,
Sanjiv Singh
Ken Barber
> pulling some jars on client.
I guess I was looking to find out the specifics of how big these
manifests and templates are in total on the nodes you are having
specific performance issues with. You say 'big manifests' but this is
a relative statement :-).
If 'nothing has changed' (please confirm this with your svn/git
history) then it must be something else other then content or puppet
itself ....
Without seeing your content its a hard one to trace via a mailing
list, there could be a network issue and some function you are trying
to execute could be lagging or having trouble resolving dns etc. Try
running your puppetmasterd in the foreground using something like:
puppetmasterd --no-daemonize --debug --trace --logdest console
And as you do a single run, see if it gives you any clues as to what
its blocking on during those 20 minutes.
ken.
sanjiv.singh
thanks for quick reply.
as per yr suggestion ,
sudo /usr/sbin/puppetmasterd --no-daemonize --debug --trace --logdest
console
fired in puppet master........
there is noting seems wrong in debug trace...., only taking time to
do stuff.
these is some clue for you that may help you to find out root cause.
i have configured puppet master to log in file /var/log/puppet/
masterhttp.log
.
.
.
.
.
[2011-07-04 10:57:18] DEBUG close: 10.122.130.1:45575
[2011-07-04 10:57:24] DEBUG Puppet::Network::HTTP::WEBrickREST is
invoked.
[2011-07-04 10:57:24] 209.251.184.24 - - [04/Jul/2011:10:57:24 EDT]
"GET/production/file_metadata/production/tomcat-web.xml HTTP/1.1" 200
391
[2011-07-04 10:57:24] - -> /production/file_metadata/production/tomcat-
web.xml
[2011-07-04 10:57:24] DEBUG close: 209.251.184.24:20132
[2011-07-04 10:57:24] DEBUG accept: 10.122.130.1:8767
[2011-07-04 10:56:33] ERROR OpenSSL::SSL::SSLError:
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:
44:in `accept'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:
44:in `listen'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:
42:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:
41:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:
41:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:
41:in `listen'
/usr/lib/ruby/1.8/thread.rb:135:in `synchronize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:
38:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:131:in
`listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:146:in
`start'
/usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:128:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/application/
puppetmasterd.rb:122:in `main'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in
`send'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in
`run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in
`exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'
/usr/sbin/puppetmasterd:66
there are in log repetted error trace for each file puppet master
serving to client.
thanks,
Sanjiv singh
Luke Bigum
How long does your master think that catalog compilation is taking?
Look for these lines:
Jul 4 15:13:53 puppetmaster puppet-master[26138]: Compiled catalog
for puppetclient in environment live in 0.12 seconds
And then the time for the agents applying that catalog:
Jul 4 15:13:55 puppetclient puppet-agent[24198]: Finished catalog run
in 1.22 seconds
-Luke
sanjiv.singh
Hi luke,
thanks for quick reply.
as i m able to see that puppet master taking around 7-8 seconds to
compiler catalog
and puppet client taking around 1106 second to apply catalog .
seems strange , because some manifest and configuration (on both
master and client) working fine with dozens of servers concurrently.
Since Sunday getting strange behavior........
feel free to ask, if some more input required to resolve d issue.
thanks,
Sanjiv Singh
Ken Barber
the result ... (not sure if that flag was round then).
ken.
Patrick
I'm pretty sure those entries you are looking at are file resources. In general, file resources scale very poorly with webrick. Even without webrick, they still tend to perform poorly.
My advice:
Stop using webrick
Take a look at recuse=>remote if you use recursive file copies a lot
Make sure you're not swapping and that you have enough ram
Consider copying big files or directories with a lot of files with a package or tar+wget.
Peter Meier
Hash: SHA1
> as i m able to see that puppet master taking around 7-8 seconds to
> compiler catalog
> and puppet client taking around 1106 second to apply catalog .
run the client with --summarize and see on which resource types it is
spending most of the time, then you can probably figure out what is wrong.
also analyzing the reports with something like the dashboard or the
foreman can help to trace these issues.
Certainly something within your environment or its dependencies has
changed. Puppet is heavily calling out to tools like facter, package
managers, service scripts etc. which all can easily affect your puppet runs.
~pete
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk4R+mcACgkQbwltcAfKi3/d/QCeODSqHvF8OxULMcbw6gfuEOT9
noMAn2b0DySrqrZ27kA6Xfvb1WaVWKdX
=pHX/
-----END PGP SIGNATURE-----
vagn scott
File system not full?
No errors in log files, like a disk might be failing?
Load average reasonable?
Any network changes?
Do you have a switch plugged into itself?
what does tcpdump show? Normal activity?
--
vagn
On 07/05/2011 12:11 AM, sanjiv.singh wrote:
> some times puppet client fails with following.......
>
> debug: Loaded state in 0.01 seconds
> debug: Using cached certificate for ca
> debug: Using cached certificate for pb11-9.domain.local
> debug: Using cached certificate_revocation_list for ca
> debug: catalog supports formats: b64_zlib_yaml marshal pson raw yaml;
> using pson
> err: Could not retrieve catalog from remote server: Could not intern
> from pson: expected value in object at '"File[/var/lib/'!
> warning: Not using cache on failed catalog
> err: Could not retrieve catalog; skipping run
>
> thanks,
> Sanjiv Singh
>
Peter Meier
Hash: SHA1
> File: 1539.03
This indicates clearly a problem in serving files, so does the other
error you posted.
As others asked: Are you still running with Webrick (means default
puppetmaster without setting up passenger/mongrel)? And how many
clients are connecting to the server how frequently?
~pete
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk4SpsUACgkQbwltcAfKi3+AeQCfVKw/DBKcMIlCzMfGK7gF0xpK
0BYAn3ph1iWV8YjO5WxuEzm7bhucvAU+
=Aj9t
-----END PGP SIGNATURE-----
sanjiv.singh
thanks for quick reply.
ya i am using Webrick default with puppetmaster.
As per performance guild line,for servicing more nodes it is
recommended to move on passenger/mongrel. surely i will do this.
There are around 100+ servers in my environment.but at most puppet
master serving to 10-12 nodes (varies) concurrently.
Thanks,
Sanjiv Singh
Impetus Info-tech.
On Jul 5, 10:53 am, Peter Meier <peter.me...@immerda.ch> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> > File: 1539.03
>
> This indicates clearly a problem in serving files, so does the other
> error you posted.
>
> As others asked: Are you still running with Webrick (means default
> puppetmaster without setting up passenger/mongrel)? And how many
> clients are connecting to the server how frequently?
>
> ~pete
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
sanjiv.singh
I am using environment option in /etc/puppet/puppet.conf for
supporting various releases.
like
.
.
[release1]
modulepath = /usr/local/applications/release1
[release2]
modulepath = /usr/local/applications/release2
[release3]
modulepath = /usr/local/applications/release3
.
.
As i added new release [YYYY] with addition of few files/templates and
jars,
it seems following issue coming with only new release [YYYY].
what can be various way with puppet , if i wanna to do alot file
serving for puppet client ?
thanks,
Sanjiv Singh
Impetus Infotech
Ken Barber
improvements in newer revisions), and use something like
Apache/Passenger as a delivery mechanism.
One question - does this happen each time you run it on the same box?
If there is no new changes - is the performance still terrible or is
it only for systems where there is a new change?
I'm just pondering also - did you introduce a new large file? I'm just
wondering if this time is spent doing an md5 sum of a huge file for
file {} comparison. You see the file provider will do an md5
comparison to make sure a a file that already exists doesn't get
installed again. If the file is large - then it takes a long time.
Does this slowness happen even when it appears no changes need to be
made? If you think a file is a culprit you can emulate the md5
behaviour by running 'time md5sum bigassfile.iso' to see if it takes a
long time.
Otherwise its the delivery of such a large file ... which might be the
case, and you definitely should switch out your stack.
I see you have 211 resources or so ... so its not the quantity of
files here that is the problem I would guess.
> There are around 100+ servers in my environment.but at most puppet
> master serving to 10-12 nodes (varies) concurrently.
Webrick at best can manage 1 node concurrently properly. So the minute
you fire up more then 1 it will slow down as it tries to serve each
client when available. Switch to Apache/Mongrel :-).
ken.