[Puppet Users] Puppetmaster with stored configs leaks file descriptors on CentOS 5

[Arnaud Gomes-do-Vale]

Hi folks,

When I enabled stored configs on my puppet master, it began keeping lots
of open connections to the MySQL server (as far as I understand it
opened one connection for each client run and never closed them).

I set up the server to close idle connections after 5 minutes. Now the
database server does close the connections but the puppet master keeps
them in CLOSE_WAIT state and eventually runs out of file descriptors.

Here is a sample of the output of lsof on the puppet master:

.../...
puppetmas 16418 puppet 617u IPv4 2683389 TCP puppetmaster.ircam.fr:57097->mysql2.ircam.fr:mysql (CLOSE_WAIT)
puppetmas 16418 puppet 618u IPv4 2683494 TCP puppetmaster.ircam.fr:57099->mysql2.ircam.fr:mysql (CLOSE_WAIT)
puppetmas 16418 puppet 619u IPv4 2683506 TCP puppetmaster.ircam.fr:57100->mysql2.ircam.fr:mysql (CLOSE_WAIT)
puppetmas 16418 puppet 620u IPv4 2684180 TCP puppetmaster.ircam.fr:57108->mysql2.ircam.fr:mysql (CLOSE_WAIT)
puppetmas 16418 puppet 621u IPv4 2683726 TCP puppetmaster.ircam.fr:57102->mysql2.ircam.fr:mysql (CLOSE_WAIT)
.../...

The matching sockets don't appear on the MySQL server.

The puppet master runs CentOS 5 with (pieces of) EPEL; here are the
installed versions of the relevant packages:

puppet-0.25.4-1.el5
puppet-server-0.25.4-1.el5
rubygem-activeresource-2.1.1-1.el5
rubygem-activesupport-2.1.1-2.el5
rubygem-activerecord-2.1.1-2.el5
rubygems-1.3.1-1.el5
rubygem-rails-2.1.1-2.el5
ruby-1.8.5-5.el5_4.8
ruby-mysql-2.7.3-1.el5

And the puppetmaster config :

[puppetmasterd]
node_terminus = ldap
ldapserver = nsldap.ircam.fr
ldapbase = ou=Hosts,dc=ircam,dc=fr
modulepath = /etc/puppet/modules:/usr/share/puppet:/var/lib/puppet/modules

storeconfigs = true
dbserver = mysql2.ircam.fr
dbadapter = mysql
dbname = XXXXXXXXXXXXXXXX
dbuser = XXXXXXXXXXXXXXXX
dbpassword = XXXXXXXXXXXXXXXXXXXXXXXXXXX

reports = store,log,rrdgraph
rrdgraph = true

--
Arnaud

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Brice Figureau]

Hi,

On Tue, 2010-04-27 at 15:05 +0200, Arnaud Gomes-do-Vale wrote:
> Hi folks,
>
> When I enabled stored configs on my puppet master, it began keeping lots
> of open connections to the MySQL server (as far as I understand it
> opened one connection for each client run and never closed them).
>
> I set up the server to close idle connections after 5 minutes. Now the
> database server does close the connections but the puppet master keeps
> them in CLOSE_WAIT state and eventually runs out of file descriptors.
>
> Here is a sample of the output of lsof on the puppet master:
>
> .../...
> puppetmas 16418 puppet 617u IPv4 2683389 TCP puppetmaster.ircam.fr:57097->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet 618u IPv4 2683494 TCP puppetmaster.ircam.fr:57099->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet 619u IPv4 2683506 TCP puppetmaster.ircam.fr:57100->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet 620u IPv4 2684180 TCP puppetmaster.ircam.fr:57108->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> puppetmas 16418 puppet 621u IPv4 2683726 TCP puppetmaster.ircam.fr:57102->mysql2.ircam.fr:mysql (CLOSE_WAIT)
> .../...
>
> The matching sockets don't appear on the MySQL server.
>
> The puppet master runs CentOS 5 with (pieces of) EPEL; here are the
> installed versions of the relevant packages:
>
> puppet-0.25.4-1.el5
> puppet-server-0.25.4-1.el5
> rubygem-activeresource-2.1.1-1.el5
> rubygem-activesupport-2.1.1-2.el5
> rubygem-activerecord-2.1.1-2.el5

Can you try with a newer rails/active record?
This one is a little bit old, and I'm not sure there aren't some bugs.
I vaguely remember that puppet required rails 2.2, but I might be wrong.
--
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!

[Scott Smith]

You might also try the latest ruby-mysql gem.

[Arnaud Gomes-do-Vale]

Brice Figureau <brice-...@daysofwonder.com> writes:

> Can you try with a newer rails/active record?
> This one is a little bit old, and I'm not sure there aren't some bugs.
> I vaguely remember that puppet required rails 2.2, but I might be wrong.

Will try. I was hoping I could get away with using whatever is in EPEL;
has anybody tested this kind of setup?

--
Arnaud

[Christopher]

Hello...

On Tue, 2010-04-27 at 15:05 +0200, Arnaud Gomes-do-Vale wrote:

> Hi folks,
>
> When I enabled stored configs on my puppet master, it began keeping lots
> of open connections to the MySQL server (as far as I understand it
> opened one connection for each client run and never closed them).
>
> I set up the server to close idle connections after 5 minutes. Now the
> database server does close the connections but the puppet master keeps
> them in CLOSE_WAIT state and eventually runs out of file descriptors.
>

I ran into this same thing. Update (rebuild) to the following rpms
versions from fedora 11:

rubygem-activerecord.noarch 1:2.3.2-4.fc11
rubygem-activesupport.noarch 1:2.3.2-2.fc11
rubygem-rack.noarch 1.0.0-1.fc11

For various reasons, these versions cannot be pushed into EPEL.

Christopher McCrory
"The guy that keeps the servers running"

chri...@pricegrabber.com
http://www.pricegrabber.com

Let's face it, there's no Hollow Earth, no robots, and
no 'mute rays.' And even if there were, waxed paper is
no defense. I tried it. Only tinfoil works.

[Todd Zullinger]

Arnaud Gomes-do-Vale wrote:
> Brice Figureau <brice-...@daysofwonder.com> writes:
>
>> Can you try with a newer rails/active record? This one is a little
>> bit old, and I'm not sure there aren't some bugs. I vaguely
>> remember that puppet required rails 2.2, but I might be wrong.

Well, puppet 0.25 doesn't refuse to run with rails 2.1 and 0.25 got
some patches to allow better support for 2.1, so it's not completely
"unsupported" from a user perspective. The wiki does list >= 2.2.2
for 0.25 and <= 2.2.2 for 0.24 though.

> Will try. I was hoping I could get away with using whatever is in
> EPEL; has anybody tested this kind of setup?

It's definitely unfortunate that what is in EPEL suffers from this
bug. I would love to see either puppet or rails there patched to work
around this. I filed https://bugzilla.redhat.com/572722 for this
issue but haven't heard anything back from the rails packagers yet.

There was a bug in the Puppet Labs redmine about these leaks, but I
think it was closed. I need to find that bug again and reopen it.

--
Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A neurosis is a secret you don't know you're keeping.
-- Kenneth Tynan

[Arnaud Gomes-do-Vale]

Christopher <chri...@pricegrabber.com> writes:

> I ran into this same thing. Update (rebuild) to the following rpms
> versions from fedora 11:
>
> rubygem-activerecord.noarch 1:2.3.2-4.fc11
> rubygem-activesupport.noarch 1:2.3.2-2.fc11
> rubygem-rack.noarch 1.0.0-1.fc11

Thanks, this fixed the issue.

--
Arnaud

[Jason Koppe]

I'm having the same issue on CentOS5.2 with the following packages:

yum
puppet-0.25.5-1.el5
puppet-server-0.25.5-1.el5
rubygems-1.3.1-1.el5
ruby-shadow-1.4.1-7.el5
ruby-devel-1.8.6.111-1
ruby-1.8.6.111-1
ruby-ri-1.8.6.111-1
ruby-docs-1.8.5-5.el5_3.7
ruby-augeas-0.3.0-1.el5
ruby-irb-1.8.6.111-1
ruby-rdoc-1.8.6.111-1
ruby-mode-1.8.5-5.el5_3.7
libselinux-ruby-1.33.4-5.5.el5
ruby-libs-1.8.6.111-1

gem
activerecord (2.3.8, 2.3.4)
activeresource (2.3.8)
activesupport (2.3.8, 2.3.4)
mysql (2.8.1)
rack (1.1.0)
rails (2.3.8)

Suggestions?

--
Jason Koppe
Jason.Rob...@gmail.com
Cell (210) 445-8242