using inetd/xinetd with puppetrun?
Larry Ludwig
100MB+ on a VPS that has only 256 MB (which isn't that uncommon) is
just plain awful. Yes I know I can make puppetd into a cron job but
then loose the ability to run puppet via puppetrun command.
I haven't gone down this road yet, but I'm wondering if I can use
inetd/xinetd to fire up a puppetrun request?
Adam Jacob
Why not use Capistrano instead?
A simple cap recipe for this might be:
role :puppet, server1, sever2, server3
namespace :puppet do
desc <<-DESC
Run puppet on each. You can set the Puppet Server with the SERVER
env variable.
You can specify a --tags value with TAGS.
DESC
task :default do
puppet_server = nil
if ENV.has_key?("SERVER")
puppet_server = ENV["SERVER"]
else
puppet_server = `hostname -f`.chomp!
end
tags = ""
if ENV.has_key?("TAGS")
tags = "--tags #{ENV["TAGS"]}"
end
sudo("sleep $(expr $RANDOM \% 10); sudo puppetd --onetime
--verbose --ignorecache --server #
{puppet_server} #{tags}")
end
end
That is untested, but cribbed from working code (that uses iClassify
to get on-the-fly lists of nodes to run puppet on.)
That would allow you to trigger puppet in the same way that puppetrun
will, but without the overhead of a running puppetd.
Adam
--
HJK Solutions - We Launch Startups - http://www.hjksolutions.com
Adam Jacob, Senior Partner
T: (206) 508-4759 E: ad...@hjksolutions.com
Ohad Levy
for me, the main reason to run puppet as a daemon mode was to be able to execute it remotely and to have a randomize drift between client runs (i.e. after some time each client is fetching the configs in a more or less random time).
I would be happy to have an xinetd ( or similar ) option officially support, or maybe there is a way to reduce puppet footprint (split between puppet daemon and puppet actual client?).
Ohad
Joel Krauska
I'm about to cron this:
perl -e "sleep int(rand(20));" ; puppetd --onetime
Where 20 (in seconds) is closer to 25min, and the cron runs on the half hour..
What is the puppetd client holding on to in between runs?
Shouldn't it just be a scheduler anyway?
--joel
Evan Hisey
>
> I'm equally frustrated.
>
> I'm about to cron this:
> perl -e "sleep int(rand(20));" ; puppetd --onetime
>
> Where 20 (in seconds) is closer to 25min, and the cron runs on the half hour..
>
> What is the puppetd client holding on to in between runs?
> Shouldn't it just be a scheduler anyway?
>
of the servers or workstations. Even the puppetmasterd is not really
using much in the way of resource when idle. I am using 2.24.4 on
CentOS boxes with the EPEL puppet rpms.
Evan
Duncan Hill
The original poster said it was chewing 100MB+ on a 256 MB VM.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd
On a smaller machine, that is a significant chunk of RAM.
Larry Ludwig
makes it also use more memory (which is not that uncommon)
Here is from one VPS.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31695 0.1 15.6 127668 82284 ? Ss Jun23 2:21 /usr/
bin/ruby /usr/sbin/puppetd
Joel Krauska
> The original poster said it was chewing 100MB+ on a 256 MB VM.
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd
Here's an example host of mine right now.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29102 root 15 0 132m 53m 2504 S 0 0.3 0:03.74 ruby
We tend to use certain machines as memory caches and try to max out
our memory usage on them.
So it's not just low memory machines, but you also get problems when
you're trying to use memory efficiently.
When there's low memory available, the background puppetd won't even fire.
--joel
Evan Hisey
does tend to do that. I wonder if puppet does not like virtual hosts.
I saw another postin gon the list were a user was having high memory
issues on a Virtual host.
Evan
RijilV
> I was unaware of that issue, but not not really suprised. Low memory
> does tend to do that. I wonder if puppet does not like virtual hosts.
> I saw another postin gon the list were a user was having high memory
> issues on a Virtual host.
We're not running it on any virtual hosts and are seeing large memory
footprints as well. Also seeing really large footprints for the
puppetmaster (though I don't care as much about that).
Sorry about the formatting.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 8399 0.8 2.0 166316 83484 ? Ssl 17:25 0:19
/usr/bin/ruby /usr/sbin/puppetd
puppet 15543 0.0 5.3 303764 216352 ? Ssl Jun24 1:04
/usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel
--masterport 18140 --pidfile /var/run/puppet/puppetmaster-18140.pid
puppet 15569 0.0 2.1 172812 85180 ? Ssl Jun24 0:38
/usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel
--masterport 18141 --pidfile /var/run/puppet/puppetmaster-18141.pid
puppet 15595 0.1 3.1 214396 126456 ? Ssl Jun24 2:09
/usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel
--masterport 18142 --pidfile /var/run/puppet/puppetmaster-18142.pid
puppet 15621 0.0 2.5 191304 103760 ? Ssl Jun24 0:37
/usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel
--masterport 18143 --pidfile /var/run/puppet/puppetmaster-18143.pid
puppet 15647 0.0 1.7 157968 70560 ? Ssl Jun24 0:43
/usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel
--masterport 18144 --pidfile /var/run/puppet/puppetmaster-18144.pid
puppet 15673 0.0 1.8 164452 76900 ? Ssl Jun24 0:32
/usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel
--masterport 18145 --pidfile /var/run/puppet/puppetmaster-18145.pid
puppet 15699 0.0 1.8 161176 73764 ? Ssl Jun24 0:44
/usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel
--masterport 18146 --pidfile /var/run/puppet/puppetmaster-18146.pid
puppet 15725 0.0 2.4 184812 97252 ? Ssl Jun24 0:41
/usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel
--masterport 18147 --pidfile /var/run/puppet/puppetmaster-18147.pid
Mike Renfro
> I was unaware of that issue, but not not really suprised. Low memory
> does tend to do that. I wonder if puppet does not like virtual hosts.
> I saw another postin gon the list were a user was having high memory
> issues on a Virtual host.
It's certainly not a guarantee that you'll have memory problems on
low-memory virtual hosts:
=====
# free
total used free shared buffers cached
Mem: 262292 252920 9372 0 6052 46736
-/+ buffers/cache: 200132 62160
Swap: 1048568 99448 949120
# uname -r
2.6.18-4-xen-686
# puppetd --version
0.23.1
# ps aux | grep [p]uppetd
root 21479 0.2 17.6 57608 46248 ? Ss Jun04 69:35 ruby
/usr/sbin/puppetd --server gold.cae.tntech.edu --factsync
=====
Most of my memory on this box is sucked up by Zope/Plone and Apache, not
by Puppet.
--
Mike Renfro / R&D Engineer, Center for Manufacturing Research,
931 372-3601 / Tennessee Technological University
Russ Allbery
> We're not running it on any virtual hosts and are seeing large memory
> footprints as well. Also seeing really large footprints for the
> puppetmaster (though I don't care as much about that).
puppetmaster still leaks like a sieve for us. :/ We have to restart it
nightly to keep the system from running out of memory, and even that
doesn't always work. It's a problem that's been going on for a long time,
unfortunately, and it's not at all clear what's causing it.
We're currently running a slightly patched version of 0.24.4.
--
Russ Allbery (r...@stanford.edu) <http://www.eyrie.org/~eagle/>
Peter Meier
>> We're not running it on any virtual hosts and are seeing large memory
>> footprints as well. Also seeing really large footprints for the
>> puppetmaster (though I don't care as much about that).
>
> puppetmaster still leaks like a sieve for us. :/ We have to restart it
> nightly to keep the system from running out of memory, and even that
> doesn't always work. It's a problem that's been going on for a long time,
> unfortunately, and it's not at all clear what's causing it.
>
> We're currently running a slightly patched version of 0.24.4.
I have the same problem on centos 5 boxes, x86_64 as well i386, all
running in xenguests and 0.24.4 (as well the guests). I deploy a cron on
puppetmasters to restart puppetmaster every 4 hours. Just to be on the
safe side. As well I'm having some big problems with certain modules and
use cases [1]. However currently I simply get problems like these
working with workarounds. :-/
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6562 root 15 0 183m 96m 2260 S 0 32.2 6:35.53 puppetd
on a host with 300mb ram.
I couldn't yet figure out what is causing this problem, but it certainly
exists. Unfortunately I don't know yet a lot how to debug problems like
these and I'm hoping that problems like these might go away with the
switch away from xmlrpc, as I could see a lot of memory problems there.
but now i'm just guessing.
however I certainly don't have anymore any corrupted file due to memory
leaks! :)
if anybody would like to dig into this problem and would like to have
data from hosts i'm having access to it, please ping me on irc
(duritong). I'd really like to help on this problem.
greets pete
Luke Kanies
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 6562 root 15 0 183m 96m 2260 S 0 32.2 6:35.53 puppetd
>
> on a host with 300mb ram.
This is a pretty clear indication that the next release should be
focused on ram usage.
--
A person's maturity consists in having found again the seriousness one
had as a child, at play. --Friedrich Nietzsche
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Ohad Levy
I for one, don't care that much how much RAM a puppet master is using (as long as its stable), because that's a service, which needs system resources to perform its duty,however, as far as for the client, I cant find any good argument why it should use a lot of memory when all its suppose to do is to wait for puppetrun requests and count until its next run.
It would be really nice, if puppetd could release its system resources when in idle mode.
Thanks,
Ohad
Russ Allbery
> I think that there is a bit difference between a puppetmaster and a
> client in terms of ram usage.
>
> I for one, don't care that much how much RAM a puppet master is using
> (as long as its stable), because that's a service, which needs system
> resources to perform its duty,however, as far as for the client, I cant
> find any good argument why it should use a lot of memory when all its
> suppose to do is to wait for puppetrun requests and count until its next
> run.
I care about both because the memory consumption of the puppetmaster
daemon is currently the limiting factor on how many of them I can run on a
single system, which is currently my performance limitation in scaling
Puppet. Right now, we can't run our infrastructure reliably on a single
puppetmaster server in large part because the system has 4GB of memory,
which isn't sufficient for even 10 puppetmasterds restarted daily.
When we try to put another system with only 2GB of memory into the
rotation, it doesn't last long enough to be useful before it runs out of
memory, although we could probably play more games with balancing between
the two.
We have new hardware about to be racked, but really, I don't think
puppetmaster should be this heavy.
Joel Krauska
puppetd --onetime --no-client
does not appear to work as I would expect.. (run once and exit)
How do you folks running puppetd client in crond call it?
--test works, but it's verbose...
--joel
AJ
/usr/sbin/puppetd -t --noop --puppetdlockfile
/var/lib/puppet/state/puppetcronlock, is how I run it
Duncan Hill
>
> Hi
>
>>> We're not running it on any virtual hosts and are seeing large memory
>>> footprints as well. Also seeing really large footprints for the
>>> puppetmaster (though I don't care as much about that).
>>
>> puppetmaster still leaks like a sieve for us. :/ We have to restart it
>> nightly to keep the system from running out of memory, and even that
> I have the same problem on centos 5 boxes, x86_64 as well i386, all
> running in xenguests and 0.24.4 (as well the guests). I deploy a cron on
> puppetmasters to restart puppetmaster every 4 hours. Just to be on the
Puppetmaster 1 - catalog server.
VIRT 99.6, RES 82.
Puppetmaster 2 - file server.
VIRT 52M, RES 37M
I've also got a restart cron job, but it only smacks the catalog
server, not the file server. CentOS 5, x86, VMware Server guest.
Before I split the file server out (and I sent a patch to dlutter for
the RHEL init script, perhaps it belongs elsewhere), I'd routinely see
puppetmasterd at 200+ MB of memory on a VM that had 256. Add in the
backup software taking VIRT 139/RES 4.5, and a postgres instance for
storeconfigs, and I'd also be 200 MB into swap, and compile times
would hit hundreds of seconds. Even with the cron to restart it, this
happened.
So perhaps splitting the file server out has benefits beyond file
serving hogging CPU on the catalog generation?
Evan Hisey
> daemon is currently the limiting factor on how many of them I can run on a
> single system, which is currently my performance limitation in scaling
> Puppet. Right now, we can't run our infrastructure reliably on a single
> puppetmaster server in large part because the system has 4GB of memory,
> which isn't sufficient for even 10 puppetmasterds restarted daily.
>
> When we try to put another system with only 2GB of memory into the
> rotation, it doesn't last long enough to be useful before it runs out of
> memory, although we could probably play more games with balancing between
> the two.
>
> We have new hardware about to be racked, but really, I don't think
> puppetmaster should be this heavy.
What is case you to need to 10 puppetmasterds on a server? From what
I have seen on the list, a single instance can handle a very large
work load. The usual trick for multiple puppetmasterds is when they
are in split out over several regions and there is a link speed issue.
There may be a better way to handle you setup, or I need to be aware
of why you need to do that as I push to spread puppet out to more of
our areas.
Evan
Russ Allbery
> What is case you to need to 10 puppetmasterds on a server? From what I
> have seen on the list, a single instance can handle a very large work
> load.
A single instance wasn't even remotely handling a large work load for us.
Running multiple puppetmasterds behind Apache to load-balance helped
considerably. We're planning on switching to Pound for the load-balancing
since we've heard it works a lot better.
> The usual trick for multiple puppetmasterds is when they are in split
> out over several regions and there is a link speed issue. There may be
> a better way to handle you setup, or I need to be aware of why you need
> to do that as I push to spread puppet out to more of our areas.
It's possible that you've just not scaled quite as far as we have. I'm
not sure. We don't have that many nodes but they're nearly all different
from each other and we do a very wide variety of stuff to them.
lsdb:/srv/puppet-ca/ssl/ca# find signed -type f -print | wc -l
310
windlord:~/work/puppet> find . -name \*.pp -print | wc -l
302
windlord:~/work/puppet> find . -name \*.pp -print0 | xargs -0 wc -l | tail -1
24797 total
windlord:~/work/puppet> find */*/files -type f -print | wc -l
5489
On the average, it takes fifteen seconds to compile a manifest, which is
already too slow for my taste.
henson2:/var/log> grep 'Compiled configuration' puppetmaster.log | sed -e 's/.* in //' -e 's/ seconds.*//' | awk '{ count++; total += $1 } END { print total / count }'
15.853
Russ Allbery
> Before I split the file server out (and I sent a patch to dlutter for
> the RHEL init script, perhaps it belongs elsewhere), I'd routinely see
> puppetmasterd at 200+ MB of memory on a VM that had 256. Add in the
> backup software taking VIRT 139/RES 4.5, and a postgres instance for
> storeconfigs, and I'd also be 200 MB into swap, and compile times would
> hit hundreds of seconds. Even with the cron to restart it, this
> happened.
>
> So perhaps splitting the file server out has benefits beyond file
> serving hogging CPU on the catalog generation?
That's entirely possible. We do a *lot* of file service.
Russ Allbery
> windlord:~/work/puppet> find */*/files -type f -print | wc -l
> 5489
Sorry, that's a bogus number. I always forget about Subversion's annoying
overhead. Here's the correct number.
windlord:~/work/puppet> find */*/files -name .svn -prune -o -type f -print | wc -l
1566
Evan Hisey
was asking. I like learning from others mistakes/success. !0 puppet
masters on one server still strikes me as counter productive from a
speed stand point. Each instance is then increasing the load by N on
the server, or has this not been a facter so much as the memory foot
print?
Evan
Russ Allbery
> We have definitely not scaled out that far yet, part of the reason I was
> asking. I like learning from others mistakes/success. !0 puppet masters
> on one server still strikes me as counter productive from a speed stand
> point. Each instance is then increasing the load by N on the server, or
> has this not been a facter so much as the memory foot print?
The load is a bit of an issue but hasn't been as much of a factor as the
memory footprint. The system has two CPUs and eight logical processes
with hyperthreading, and we definitely saw a performance increase when
going from five puppetmasters to ten.
Of course, with less beefy hardware, it would make sense to spread out
across more machines instead, but it's getting to the point where that's
almost a standard server box.
Peter Meier
> 15.853
server1# grep 'Compiled catalog' /var/log/puppet/puppetmaster.log | sed
-e 's/.* in //' -e 's/ seconds.*//' | awk '{ count++; total += $1 } END
{ print total / count }'
14.2223
running 4 instances load balanced with nginx.
server2# grep 'Compiled catalog' /var/log/puppet/puppetmaster.log | sed
-e 's/.* in //' -e 's/ seconds.*//' | awk '{ count++; total += $1 } END
{ print total / count }'
65.5803
currently only running 1 instance with webrick. soon switching to nginx
and as well 4 instances. will see if that changes anything.
both are yet only serving a bunch of hosts. more to come soon.
actually I don't really care how long it takes to compile the catalogs
(only when testing new features :P ) but more than a minute seems also
long for me. 15 seconds are currently ok.
btw: in which Version changed the log entry from Compiled configuration
to Compiled catalog?
greets pete
Duncan Hill
> The original poster said it was chewing 100MB+ on a 256 MB VM.
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd
>
> On a smaller machine, that is a significant chunk of RAM.
Coming to a VM near you, When Puppetmasterd Goes Bad. Starring Puppet
0.24.4, CentOS 5.1 and VMware Server 1.04.
Mem: 340780k total, 337032k used, 3748k free, 156k buffers
Swap: 819192k total, 409596k used, 409596k free, 9268k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17923 puppet 18 0 400m 249m 2828 R 51.0 74.9 80:56.51 puppetmasterd
6856 puppet 15 0 56484 6144 1456 S 2.3 1.8 524:38.33 puppetmasterd
The first one is the manifest puppetmaster. The second is the file server.
Larry Ludwig
> The load is a bit of an issue but hasn't been as much of a factor as the
> memory footprint. The system has two CPUs and eight logical processes
> with hyperthreading, and we definitely saw a performance increase when
> going from five puppetmasters to ten.
hyperthreading != multi-core CPUs. In some cases hyperthreading
performs much worse when enabled. I don't know about with a
puppetmaster server setup, but you may want to test this.
Also the newer Xeons (woodcrests and newer) have a much lower power
consumption, and blow the pants off of the older Xeons.
Right now from what I see, I believe puppetmaster needs much more
optimization than anything else.
-L
--
Larry Ludwig
Empowering Media
1-866-792-0489 x600
Managed and Unmanaged Xen VPSes
http://www.hostcube.com/
Russ Allbery
> I assume you know this but will throw it out just in case.
>
> hyperthreading != multi-core CPUs. In some cases hyperthreading
> performs much worse when enabled. I don't know about with a
> puppetmaster server setup, but you may want to test this.
Yeah, I know. We haven't experimented yet with this. We might with the
new system.
Peter Meier
Luke Kanies wrote:
> On Jun 25, 2008, at 1:22 PM, Peter Meier wrote:
>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>
>>
>> 6562 root 15 0 183m 96m 2260 S 0 32.2 6:35.53 puppetd
>>
>> on a host with 300mb ram.
>
>
> This is a pretty clear indication that the next release should be
> focused on ram usage.
should I open a ticket, wich tracks a bit the problems and points raised
in this discussion? so we have something that these points don't get lost.
greets pete
Luke Kanies
> should I open a ticket, wich tracks a bit the problems and points
> raised
> in this discussion? so we have something that these points don't get
> lost.
Yes.
--
Chase after truth like hell and you'll free yourself, even though
you never touch its coat-tails. -- Clarence Darrow
Peter Meier
>> should I open a ticket, wich tracks a bit the problems and points
>> raised
>> in this discussion? so we have something that these points don't get
>> lost.
>
>
> Yes.
I tried to summarize the recent discussion in bug #1395 [1]. I tried to
collect the interesting points of the discussion and tried to summarize
the interesting data.
I hope I noticed everything and that the important data is in the bug
report.
For me it is the first attempt for this kind of bugreport. I hope it is
useful and that it might help to investigate the problem further.
I'd like to help to investigate on this problem, as I have a lot of xen
guests having not much memory, which should do just a bunch of little
tasks. (dns for example) However I still like to manage them with
puppet, which is kind of problematic at the current memory usage.
As I don't have much of experience in debugging of ruby applications
especially on memory usage, it would be nice to have some people with
experience joining the investigation(s). If somebody teaches me to
collect any data needed from running puppetds, I'm happy to provide you
with additional data.
greets pete