I don't like how much of a pig puppetd uses memory. puppetd using
100MB+ on a VPS that has only 256 MB (which isn't that uncommon) is
just plain awful. Yes I know I can make puppetd into a cron job but
then loose the ability to run puppet via puppetrun command.
I haven't gone down this road yet, but I'm wondering if I can use
inetd/xinetd to fire up a puppetrun request?
On Sun, Jun 22, 2008 at 7:57 AM, Larry Ludwig <larry...@gmail.com> wrote:
> I don't like how much of a pig puppetd uses memory. puppetd using > 100MB+ on a VPS that has only 256 MB (which isn't that uncommon) is > just plain awful. Yes I know I can make puppetd into a cron job but > then loose the ability to run puppet via puppetrun command.
> I haven't gone down this road yet, but I'm wondering if I can use > inetd/xinetd to fire up a puppetrun request?
Why not use Capistrano instead?
A simple cap recipe for this might be:
role :puppet, server1, sever2, server3
namespace :puppet do
desc <<-DESC Run puppet on each. You can set the Puppet Server with the SERVER env variable.
You can specify a --tags value with TAGS. DESC task :default do puppet_server = nil if ENV.has_key?("SERVER") puppet_server = ENV["SERVER"] else puppet_server = `hostname -f`.chomp! end tags = "" if ENV.has_key?("TAGS") tags = "--tags #{ENV["TAGS"]}" end sudo("sleep $(expr $RANDOM \% 10); sudo puppetd --onetime --verbose --ignorecache --server # {puppet_server} #{tags}") end end
That is untested, but cribbed from working code (that uses iClassify to get on-the-fly lists of nodes to run puppet on.)
That would allow you to trigger puppet in the same way that puppetrun will, but without the overhead of a running puppetd.
Adam
-- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: a...@hjksolutions.com
I'm also not happy with puppet memory footprint, What is the common practice in other places? do you run puppet from a cronjob or do you run it as a daemon?
for me, the main reason to run puppet as a daemon mode was to be able to execute it remotely and to have a randomize drift between client runs (i.e. after some time each client is fetching the configs in a more or less random time).
I would be happy to have an xinetd ( or similar ) option officially support, or maybe there is a way to reduce puppet footprint (split between puppet daemon and puppet actual client?).
On Sun, Jun 22, 2008 at 11:59 PM, Adam Jacob <a...@hjksolutions.com> wrote:
> On Sun, Jun 22, 2008 at 7:57 AM, Larry Ludwig <larry...@gmail.com> wrote:
> > I don't like how much of a pig puppetd uses memory. puppetd using > > 100MB+ on a VPS that has only 256 MB (which isn't that uncommon) is > > just plain awful. Yes I know I can make puppetd into a cron job but > > then loose the ability to run puppet via puppetrun command.
> > I haven't gone down this road yet, but I'm wondering if I can use > > inetd/xinetd to fire up a puppetrun request?
> Why not use Capistrano instead?
> A simple cap recipe for this might be:
> role :puppet, server1, sever2, server3
> namespace :puppet do
> desc <<-DESC > Run puppet on each. You can set the Puppet Server with the SERVER > env variable.
> You can specify a --tags value with TAGS. > DESC > task :default do > puppet_server = nil > if ENV.has_key?("SERVER") > puppet_server = ENV["SERVER"] > else > puppet_server = `hostname -f`.chomp! > end > tags = "" > if ENV.has_key?("TAGS") > tags = "--tags #{ENV["TAGS"]}" > end > sudo("sleep $(expr $RANDOM \% 10); sudo puppetd --onetime > --verbose --ignorecache --server # > {puppet_server} #{tags}") > end > end
> That is untested, but cribbed from working code (that uses iClassify > to get on-the-fly lists of nodes to run puppet on.)
> That would allow you to trigger puppet in the same way that puppetrun > will, but without the overhead of a running puppetd.
On Tue, Jun 24, 2008 at 8:56 PM, Ohad Levy <ohadl...@gmail.com> wrote: > I'm also not happy with puppet memory footprint, What is the common practice > in other places? do you run puppet from a cronjob or do you run it as a > daemon?
> for me, the main reason to run puppet as a daemon mode was to be able to > execute it remotely and to have a randomize drift between client runs (i.e. > after some time each client is fetching the configs in a more or less random > time).
> I would be happy to have an xinetd ( or similar ) option officially support, > or maybe there is a way to reduce puppet footprint (split between puppet > daemon and puppet actual client?).
> Ohad
> On Sun, Jun 22, 2008 at 11:59 PM, Adam Jacob <a...@hjksolutions.com> wrote:
>> On Sun, Jun 22, 2008 at 7:57 AM, Larry Ludwig <larry...@gmail.com> wrote:
>> > I don't like how much of a pig puppetd uses memory. puppetd using >> > 100MB+ on a VPS that has only 256 MB (which isn't that uncommon) is >> > just plain awful. Yes I know I can make puppetd into a cron job but >> > then loose the ability to run puppet via puppetrun command.
>> > I haven't gone down this road yet, but I'm wondering if I can use >> > inetd/xinetd to fire up a puppetrun request?
>> Why not use Capistrano instead?
>> A simple cap recipe for this might be:
>> role :puppet, server1, sever2, server3
>> namespace :puppet do
>> desc <<-DESC >> Run puppet on each. You can set the Puppet Server with the SERVER >> env variable.
>> You can specify a --tags value with TAGS. >> DESC >> task :default do >> puppet_server = nil >> if ENV.has_key?("SERVER") >> puppet_server = ENV["SERVER"] >> else >> puppet_server = `hostname -f`.chomp! >> end >> tags = "" >> if ENV.has_key?("TAGS") >> tags = "--tags #{ENV["TAGS"]}" >> end >> sudo("sleep $(expr $RANDOM \% 10); sudo puppetd --onetime >> --verbose --ignorecache --server # >> {puppet_server} #{tags}") >> end >> end
>> That is untested, but cribbed from working code (that uses iClassify >> to get on-the-fly lists of nodes to run puppet on.)
>> That would allow you to trigger puppet in the same way that puppetrun >> will, but without the overhead of a running puppetd.
On Wed, Jun 25, 2008 at 11:15 AM, Joel Krauska <jkrau...@gmail.com> wrote:
> I'm equally frustrated.
> I'm about to cron this: > perl -e "sleep int(rand(20));" ; puppetd --onetime
> Where 20 (in seconds) is closer to 25min, and the cron runs on the half hour..
> What is the puppetd client holding on to in between runs? > Shouldn't it just be a scheduler anyway?
How are you guess running puppetd? It has almost no footprint on any of the servers or workstations. Even the puppetmasterd is not really using much in the way of resource when idle. I am using 2.24.4 on CentOS boxes with the EPEL puppet rpms.
> On Wed, Jun 25, 2008 at 11:15 AM, Joel Krauska <jkrau...@gmail.com> wrote:
>> I'm equally frustrated.
>> I'm about to cron this: >> perl -e "sleep int(rand(20));" ; puppetd --onetime
>> Where 20 (in seconds) is closer to 25min, and the cron runs on the half hour..
>> What is the puppetd client holding on to in between runs? >> Shouldn't it just be a scheduler anyway?
> How are you guess running puppetd? It has almost no footprint on any > of the servers or workstations. Even the puppetmasterd is not really > using much in the way of resource when idle. I am using 2.24.4 on > CentOS boxes with the EPEL puppet rpms.
The original poster said it was chewing 100MB+ on a 256 MB VM. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd
On a smaller machine, that is a significant chunk of RAM.
> > On Wed, Jun 25, 2008 at 11:15 AM, Joel Krauska <jkrau...@gmail.com> wrote:
> >> I'm equally frustrated.
> >> I'm about to cron this:
> >> perl -e "sleep int(rand(20));" ; puppetd --onetime
> >> Where 20 (in seconds) is closer to 25min, and the cron runs on the half hour..
> >> What is the puppetd client holding on to in between runs?
> >> Shouldn't it just be a scheduler anyway?
> > How are you guess running puppetd? It has almost no footprint on any
> > of the servers or workstations. Even the puppetmasterd is not really
> > using much in the way of resource when idle. I am using 2.24.4 on
> > CentOS boxes with the EPEL puppet rpms.
> The original poster said it was chewing 100MB+ on a 256 MB VM.
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd
> On a smaller machine, that is a significant chunk of RAM.
On Wed, Jun 25, 2008 at 9:31 AM, Duncan Hill <bajand...@googlemail.com> wrote: > The original poster said it was chewing 100MB+ on a 256 MB VM. > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd
Here's an example host of mine right now.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 29102 root 15 0 132m 53m 2504 S 0 0.3 0:03.74 ruby
We tend to use certain machines as memory caches and try to max out our memory usage on them. So it's not just low memory machines, but you also get problems when you're trying to use memory efficiently.
When there's low memory available, the background puppetd won't even fire.
On Wed, Jun 25, 2008 at 11:42 AM, Joel Krauska <jkrau...@gmail.com> wrote:
> On Wed, Jun 25, 2008 at 9:31 AM, Duncan Hill <bajand...@googlemail.com> wrote: >> The original poster said it was chewing 100MB+ on a 256 MB VM. >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd
> Here's an example host of mine right now.
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 29102 root 15 0 132m 53m 2504 S 0 0.3 0:03.74 ruby
> We tend to use certain machines as memory caches and try to max out > our memory usage on them. > So it's not just low memory machines, but you also get problems when > you're trying to use memory efficiently.
> When there's low memory available, the background puppetd won't even fire.
> --joel
I was unaware of that issue, but not not really suprised. Low memory does tend to do that. I wonder if puppet does not like virtual hosts. I saw another postin gon the list were a user was having high memory issues on a Virtual host.
> I was unaware of that issue, but not not really suprised. Low memory > does tend to do that. I wonder if puppet does not like virtual hosts. > I saw another postin gon the list were a user was having high memory > issues on a Virtual host.
We're not running it on any virtual hosts and are seeing large memory footprints as well. Also seeing really large footprints for the puppetmaster (though I don't care as much about that).
> I was unaware of that issue, but not not really suprised. Low memory > does tend to do that. I wonder if puppet does not like virtual hosts. > I saw another postin gon the list were a user was having high memory > issues on a Virtual host.
It's certainly not a guarantee that you'll have memory problems on low-memory virtual hosts:
RijilV <rij...@gmail.com> writes: > We're not running it on any virtual hosts and are seeing large memory > footprints as well. Also seeing really large footprints for the > puppetmaster (though I don't care as much about that).
puppetmaster still leaks like a sieve for us. :/ We have to restart it nightly to keep the system from running out of memory, and even that doesn't always work. It's a problem that's been going on for a long time, unfortunately, and it's not at all clear what's causing it.
We're currently running a slightly patched version of 0.24.4.
>> We're not running it on any virtual hosts and are seeing large memory >> footprints as well. Also seeing really large footprints for the >> puppetmaster (though I don't care as much about that).
> puppetmaster still leaks like a sieve for us. :/ We have to restart it > nightly to keep the system from running out of memory, and even that > doesn't always work. It's a problem that's been going on for a long time, > unfortunately, and it's not at all clear what's causing it.
> We're currently running a slightly patched version of 0.24.4.
I have the same problem on centos 5 boxes, x86_64 as well i386, all running in xenguests and 0.24.4 (as well the guests). I deploy a cron on puppetmasters to restart puppetmaster every 4 hours. Just to be on the safe side. As well I'm having some big problems with certain modules and use cases [1]. However currently I simply get problems like these working with workarounds. :-/
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
I couldn't yet figure out what is causing this problem, but it certainly exists. Unfortunately I don't know yet a lot how to debug problems like these and I'm hoping that problems like these might go away with the switch away from xmlrpc, as I could see a lot of memory problems there. but now i'm just guessing.
however I certainly don't have anymore any corrupted file due to memory leaks! :)
if anybody would like to dig into this problem and would like to have data from hosts i'm having access to it, please ping me on irc (duritong). I'd really like to help on this problem.
This is a pretty clear indication that the next release should be focused on ram usage.
-- A person's maturity consists in having found again the seriousness one had as a child, at play. --Friedrich Nietzsche --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
I think that there is a bit difference between a puppetmaster and a client in terms of ram usage.
I for one, don't care that much how much RAM a puppet master is using (as long as its stable), because that's a service, which needs system resources to perform its duty,however, as far as for the client, I cant find any good argument why it should use a lot of memory when all its suppose to do is to wait for puppetrun requests and count until its next run.
It would be really nice, if puppetd could release its system resources when in idle mode.
> This is a pretty clear indication that the next release should be > focused on ram usage.
> -- > A person's maturity consists in having found again the seriousness one > had as a child, at play. --Friedrich Nietzsche > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com
"Ohad Levy" <ohadl...@gmail.com> writes: > I think that there is a bit difference between a puppetmaster and a > client in terms of ram usage.
> I for one, don't care that much how much RAM a puppet master is using > (as long as its stable), because that's a service, which needs system > resources to perform its duty,however, as far as for the client, I cant > find any good argument why it should use a lot of memory when all its > suppose to do is to wait for puppetrun requests and count until its next > run.
I care about both because the memory consumption of the puppetmaster daemon is currently the limiting factor on how many of them I can run on a single system, which is currently my performance limitation in scaling Puppet. Right now, we can't run our infrastructure reliably on a single puppetmaster server in large part because the system has 4GB of memory, which isn't sufficient for even 10 puppetmasterds restarted daily.
When we try to put another system with only 2GB of memory into the rotation, it doesn't last long enough to be useful before it runs out of memory, although we could probably play more games with balancing between the two.
We have new hardware about to be racked, but really, I don't think puppetmaster should be this heavy.
>>> We're not running it on any virtual hosts and are seeing large memory >>> footprints as well. Also seeing really large footprints for the >>> puppetmaster (though I don't care as much about that).
>> puppetmaster still leaks like a sieve for us. :/ We have to restart it >> nightly to keep the system from running out of memory, and even that > I have the same problem on centos 5 boxes, x86_64 as well i386, all > running in xenguests and 0.24.4 (as well the guests). I deploy a cron on > puppetmasters to restart puppetmaster every 4 hours. Just to be on the
Puppetmaster 1 - catalog server. VIRT 99.6, RES 82.
Puppetmaster 2 - file server. VIRT 52M, RES 37M
I've also got a restart cron job, but it only smacks the catalog server, not the file server. CentOS 5, x86, VMware Server guest.
Before I split the file server out (and I sent a patch to dlutter for the RHEL init script, perhaps it belongs elsewhere), I'd routinely see puppetmasterd at 200+ MB of memory on a VM that had 256. Add in the backup software taking VIRT 139/RES 4.5, and a postgres instance for storeconfigs, and I'd also be 200 MB into swap, and compile times would hit hundreds of seconds. Even with the cron to restart it, this happened.
So perhaps splitting the file server out has benefits beyond file serving hogging CPU on the catalog generation?
> I care about both because the memory consumption of the puppetmaster > daemon is currently the limiting factor on how many of them I can run on a > single system, which is currently my performance limitation in scaling > Puppet. Right now, we can't run our infrastructure reliably on a single > puppetmaster server in large part because the system has 4GB of memory, > which isn't sufficient for even 10 puppetmasterds restarted daily.
> When we try to put another system with only 2GB of memory into the > rotation, it doesn't last long enough to be useful before it runs out of > memory, although we could probably play more games with balancing between > the two.
> We have new hardware about to be racked, but really, I don't think > puppetmaster should be this heavy.
What is case you to need to 10 puppetmasterds on a server? From what I have seen on the list, a single instance can handle a very large work load. The usual trick for multiple puppetmasterds is when they are in split out over several regions and there is a link speed issue. There may be a better way to handle you setup, or I need to be aware of why you need to do that as I push to spread puppet out to more of our areas.
"Evan Hisey" <ehi...@gmail.com> writes: > What is case you to need to 10 puppetmasterds on a server? From what I > have seen on the list, a single instance can handle a very large work > load.
A single instance wasn't even remotely handling a large work load for us. Running multiple puppetmasterds behind Apache to load-balance helped considerably. We're planning on switching to Pound for the load-balancing since we've heard it works a lot better.
> The usual trick for multiple puppetmasterds is when they are in split > out over several regions and there is a link speed issue. There may be > a better way to handle you setup, or I need to be aware of why you need > to do that as I push to spread puppet out to more of our areas.
It's possible that you've just not scaled quite as far as we have. I'm not sure. We don't have that many nodes but they're nearly all different from each other and we do a very wide variety of stuff to them.
lsdb:/srv/puppet-ca/ssl/ca# find signed -type f -print | wc -l 310
"Duncan Hill" <bajand...@googlemail.com> writes: > Before I split the file server out (and I sent a patch to dlutter for > the RHEL init script, perhaps it belongs elsewhere), I'd routinely see > puppetmasterd at 200+ MB of memory on a VM that had 256. Add in the > backup software taking VIRT 139/RES 4.5, and a postgres instance for > storeconfigs, and I'd also be 200 MB into swap, and compile times would > hit hundreds of seconds. Even with the cron to restart it, this > happened.
> So perhaps splitting the file server out has benefits beyond file > serving hogging CPU on the catalog generation?
That's entirely possible. We do a *lot* of file service.
>> What is case you to need to 10 puppetmasterds on a server? From what I >> have seen on the list, a single instance can handle a very large work >> load.
> A single instance wasn't even remotely handling a large work load for us. > Running multiple puppetmasterds behind Apache to load-balance helped > considerably. We're planning on switching to Pound for the load-balancing > since we've heard it works a lot better.
>> The usual trick for multiple puppetmasterds is when they are in split >> out over several regions and there is a link speed issue. There may be >> a better way to handle you setup, or I need to be aware of why you need >> to do that as I push to spread puppet out to more of our areas.
> It's possible that you've just not scaled quite as far as we have. I'm > not sure. We don't have that many nodes but they're nearly all different > from each other and we do a very wide variety of stuff to them.
> lsdb:/srv/puppet-ca/ssl/ca# find signed -type f -print | wc -l > 310
> On the average, it takes fifteen seconds to compile a manifest, which is > already too slow for my taste.
> henson2:/var/log> grep 'Compiled configuration' puppetmaster.log | sed -e 's/.* in //' -e 's/ seconds.*//' | awk '{ count++; total += $1 } END { print total / count }' > 15.853
We have definitely not scaled out that far yet, part of the reason I was asking. I like learning from others mistakes/success. !0 puppet masters on one server still strikes me as counter productive from a speed stand point. Each instance is then increasing the load by N on the server, or has this not been a facter so much as the memory foot print?
"Evan Hisey" <ehi...@gmail.com> writes: > We have definitely not scaled out that far yet, part of the reason I was > asking. I like learning from others mistakes/success. !0 puppet masters > on one server still strikes me as counter productive from a speed stand > point. Each instance is then increasing the load by N on the server, or > has this not been a facter so much as the memory foot print?
The load is a bit of an issue but hasn't been as much of a factor as the memory footprint. The system has two CPUs and eight logical processes with hyperthreading, and we definitely saw a performance increase when going from five puppetmasters to ten.
Of course, with less beefy hardware, it would make sense to spread out across more machines instead, but it's getting to the point where that's almost a standard server box.
|
