Puppetmaster leaving files open with "too many files open" error

[treydock]

Every few days I have to restart the puppetmaster process due to this
error

"Could not read YAML data for node .... Too many open files".

I have 9 clients being controlled by Puppet and each one of those will
generate the same error once this happens. Running 'lsof' during this
time shows me 100's of entries like this

"puppetmas 10443 puppet 14u unix
0xffff810014c7e180 17057251 socket".

The only way to remedy this is by restarting the puppetmaster
process. Once I restart the puppet master process it seems that a new
lsof entry is created and stays present for every client check.

All clients and puppetmaster are running 2.6.8 on CentOS 5.6 x64. Any
ideas? If more output and debug info is needed please let me know.
Also if switching to a mysql data store could get rid of this then
that would work as I was planning on it eventually.

Thanks
- Trey

[Nathan Clemons]

I'm pretty sure there is an open bug on this, I was looking around on this the other day. It's related to using stored configs and Puppet/ruby not closing the connections to the database servers.

--

Nathan Clemons

http://www.livemocha.com

The worlds largest online language learning community

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[treydock]

What is the current issue number? The only one I could find that
mentions the same problem is this , http://projects.puppetlabs.com/issues/7203.

Would any changes to my current puppet.conf (see below) make this
problem no longer an issue? I'm not suing storeconfig, and not really
sure if I need to be doing anything as far as saving the inventory
information. My setup is very basic, but right now can't run for more
than 2 or 3 days without having to restart the puppetmaster service.

[master]
certname=puppetmaster
reports = http, store

# Configure Inventory Service
facts_terminus = inventory_active_record
dbadapter = mysql
dbname = puppet_inv
dbuser = puppet_inv
dbpassword = cabb0ac9ed925867f93587ccfb147ebf15dec8a3
dbserver = localhost
dbsocket = /var/lib/mysql/mysql.sock


Thanks
- Trey

On May 21, 6:08 am, Nathan Clemons <nat...@livemocha.com> wrote:
> I'm pretty sure there is an open bug on this, I was looking around on this
> the other day. It's related to using stored configs and Puppet/ruby not
> closing the connections to the database servers.
>
> --

> Nathan Clemonshttp://www.livemocha.com

[Todd Zullinger]

treydock wrote:
> What is the current issue number? The only one I could find that
> mentions the same problem is this , http://projects.puppetlabs.com/issues/7203.

https://bugzilla.redhat.com/show_bug.cgi?id=572722 and
http://projects.puppetlabs.com/issues/3693 are both relevant here.

I'd love to see a fix in either the rails activerecord component or in
puppet. Seeing that older puppet versions worked with the rails stack
in EPEL, it seems like it should be possible to make that work again
with some changes to puppet.

--
Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is strangely absurd to suppose that a million human beings
collected together are not under the same moral laws which bind them
separately.
-- Thomas Jefferson

[Nathan Clemons]

http://projects.puppetlabs.com/issues/3238 is the issue I was thinking of, but 3693 is probably a duplicate of that.

These bugs have been open a really long time. Outside of lowering the MySQL timeout value, are there any workarounds for this problem? I can't really understand how Zynga can be managing as many hosts via Puppet as they do without being affected by a bug like this, unless they're not using stored configs (which would surprise me).

--

Nathan Clemons

http://www.livemocha.com

The worlds largest online language learning community

[Todd Zullinger]

Nathan Clemons wrote:
> http://projects.puppetlabs.com/issues/3238 is the issue I was
> thinking of, but 3693 is probably a duplicate of that.
>
> These bugs have been open a really long time. Outside of lowering
> the MySQL timeout value, are there any workarounds for this problem?
> I can't really understand how Zynga can be managing as many hosts
> via Puppet as they do without being affected by a bug like this,
> unless they're not using stored configs (which would surprise me).

Sadly, I don't know of any decent workarounds. I ended up rolling a
rails-2.3.5 for EL-5 in the production instances I wanted to use
storedconfigs. I would love to see a workaround or a patch for either
the activerecord or puppet packages. I know that either way this can
be fixed will get into EPEL shortly, as it's a very annoying bug that
was introduced with the update from 0.24 to 0.25.

--
Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I figure that if God actually does exist, He's big enough to
understand an honest difference of opinion.
-- Isaac Asimov

[treydock]

I don't know how helpful this would be but for now I think I found a
"workaround" of sorts for my small environment.

I commented out the following in my puppet.conf...

facts_terminus = inventory_active_record
dbadapter = mysql
dbname = puppet_inv
dbuser = puppet_inv
dbpassword = cabb0ac9ed925867f93587ccfb147ebf15dec8a3
dbserver = localhost
dbsocket = /var/lib/mysql/mysql.sock

I got rid of all that and after 2 days of puppetmaster running I only
see a few entries when doing "lsof | grep puppetmas" rather than 100s.

I'm also not doing stored configs but I think those lines above were
still the problem as it was puppet interfacing with MySQL.

What potential benefit would be lost from not using MySQL for
"facts_terminus" ?

- Trey

On May 24, 7:17 pm, Todd Zullinger <t...@pobox.com> wrote:
> Nathan Clemons wrote:
> >http://projects.puppetlabs.com/issues/3238is the issue I was

>  application_pgp-signature_part
> < 1KViewDownload

[Leon Johnson]

I've noticed the same thing with our puppetmaster server.

We're still in testing so dramatic changes to puppet.conf are not a problem. But commenting lines out doesn't fix the problem, it just turns on the feature that was being broken by too many files being open.

Trey: We've been testing on 12 nodes and get this error after about two days as well. Wondering if this could still be an issue with 100s of nodes.