err: Too many open files - /var/lib/puppet/state/state.yaml.tmp
Colm O'Connor
err: Too many open files - /var/lib/puppet/state/state.yaml.tmp
Does anybody know what might be causing it?
I'm also getting "err: State got corrupted". Everything does seem to
be kind of working, but it's sort of worrying getting these new errors.
James Turnbull
What version? What platform?
Regards
James Turnbull
--
Puppet Labs - http://www.puppetlabs.com
C: 503-734-8571
Héctor Rivas Gándara
>> err: Too many open files - /var/lib/puppet/state/state.yaml.tmp
>> Does anybody know what might be causing it?
>>
I am having the same problem. I think that it is a socket leak in
puppetmaster (it does not properly closes sockets).
To workaround the problem you can increase the ulimit for open files
(/etc/security/limits.conf) and restart puppetmaster periodically
(¿each 6h?).
I am using puppet 2.6.1rc3 on Debian Lenny with ruby 1.8.7.
I use mongrel+apache and storeconfigs+mysql
I made it fail executing puppetd in a loop in two nodes to test
another bug. lsof output is:
# lsof +p 26100|grep socket | wc
1017 8137 77326
# lsof +p 26100
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby 26100 puppet cwd DIR 254,0 4096 2 /
ruby 26100 puppet rtd DIR 254,0 4096 2 /
ruby 26100 puppet txt REG 254,0 5288 115042
/usr/bin/ruby1.8
ruby 26100 puppet mem REG 254,0 88968 172152
/lib/libnsl-2.7.so
ruby 26100 puppet mem REG 254,0 2138416 165012
/usr/lib/libmysqlclient.so.15.0.0
ruby 26100 puppet mem REG 254,0 78976 264552
/usr/lib/ruby/1.8/x86_64-linux/mysql.so
ruby 26100 puppet mem REG 254,0 23560 266160
/usr/lib/ruby/1.8/x86_64-linux/http11.so
ruby 26100 puppet mem REG 254,0 8208 264470
/usr/lib/ruby/1.8/x86_64-linux/digest/md5.so
ruby 26100 puppet mem REG 254,0 10784 264492
/usr/lib/ruby/1.8/x86_64-linux/shadow.so
ruby 26100 puppet mem REG 254,0 44448 264465
/usr/lib/ruby/1.8/x86_64-linux/bigdecimal.so
ruby 26100 puppet mem REG 254,0 12720 264467
/usr/lib/ruby/1.8/x86_64-linux/digest.so
ruby 26100 puppet mem REG 254,0 1667176 171245
/usr/lib/libcrypto.so.0.9.8
ruby 26100 puppet mem REG 254,0 330880 171246
/usr/lib/libssl.so.0.9.8
ruby 26100 puppet mem REG 254,0 298568 264491
/usr/lib/ruby/1.8/x86_64-linux/openssl.so
ruby 26100 puppet mem REG 254,0 15184 279484
/usr/lib/ruby/1.8/x86_64-linux/racc/cparse.so
ruby 26100 puppet mem REG 254,0 14312 169673
/usr/lib/gconv/UTF-16.so
ruby 26100 puppet mem REG 254,0 16624 264477
/usr/lib/ruby/1.8/x86_64-linux/iconv.so
ruby 26100 puppet mem REG 254,0 19256 264485
/usr/lib/ruby/1.8/x86_64-linux/strscan.so
ruby 26100 puppet mem REG 254,0 271320 264480
/usr/lib/ruby/1.8/x86_64-linux/nkf.so
ruby 26100 puppet mem REG 254,0 12512 264487
/usr/lib/ruby/1.8/x86_64-linux/syslog.so
ruby 26100 puppet mem REG 254,0 10928 264475
/usr/lib/ruby/1.8/x86_64-linux/etc.so
ruby 26100 puppet mem REG 254,0 5152 264476
/usr/lib/ruby/1.8/x86_64-linux/fcntl.so
ruby 26100 puppet mem REG 254,0 45824 264483
/usr/lib/ruby/1.8/x86_64-linux/socket.so
ruby 26100 puppet mem REG 254,0 93504 165898
/usr/lib/libz.so.1.2.3.3
ruby 26100 puppet mem REG 254,0 37856 264489
/usr/lib/ruby/1.8/x86_64-linux/zlib.so
ruby 26100 puppet mem REG 254,0 15888 264488
/usr/lib/ruby/1.8/x86_64-linux/thread.so
ruby 26100 puppet mem REG 254,0 122848 264486
/usr/lib/ruby/1.8/x86_64-linux/syck.so
ruby 26100 puppet mem REG 254,0 20768 264484
/usr/lib/ruby/1.8/x86_64-linux/stringio.so
ruby 26100 puppet mem REG 254,0 1375536 172156
/lib/libc-2.7.so
ruby 26100 puppet mem REG 254,0 534736 172148
/lib/libm-2.7.so
ruby 26100 puppet mem REG 254,0 39112 172150
/lib/libcrypt-2.7.so
ruby 26100 puppet mem REG 254,0 14616 172154
/lib/libdl-2.7.so
ruby 26100 puppet mem REG 254,0 130114 172171
/lib/libpthread-2.7.so
ruby 26100 puppet mem REG 254,0 946760 169053
/usr/lib/libruby1.8.so.1.8.7
ruby 26100 puppet mem REG 254,0 119288 172167
/lib/ld-2.7.so
ruby 26100 puppet DEL REG 254,2 37750356
/var/run/nscd/dbeKJQ5D
ruby 26100 puppet mem REG 254,2 217016 12928501
/var/cache/nscd/services
ruby 26100 puppet DEL REG 254,2 37750355
/var/run/nscd/dblolCC6
ruby 26100 puppet DEL REG 254,2 37750354
/var/run/nscd/db3DHabz
ruby 26100 puppet mem REG 254,0 25700 164974
/usr/lib/gconv/gconv-modules.cache
ruby 26100 puppet 0r CHR 1,3 565 /dev/null
ruby 26100 puppet 1w CHR 1,3 565 /dev/null
ruby 26100 puppet 2w CHR 1,3 565 /dev/null
ruby 26100 puppet 3r FIFO 0,6 1514186 pipe
ruby 26100 puppet 4w FIFO 0,6 1514186 pipe
ruby 26100 puppet 5u unix 0xffff81002f412440 1514460 socket
ruby 26100 puppet 6u IPv4 1514464 TCP
localhost:18140 (LISTEN)
ruby 26100 puppet 8w REG 254,6 1929 4436700
/srv/mnt/dsk/cgxappsvg1-dlv00data/cgx1/puppet/var/log/rails.log
ruby 26100 puppet 9u unix 0xffff81008f9caf40 1518960 socket
ruby 26100 puppet 10u unix 0xffff81004bc656c0 1531736 socket
ruby 26100 puppet 11u unix 0xffff81003c53ac40 1525135 socket
ruby 26100 puppet 12u unix 0xffff81004bc65c40 1527344 socket
ruby 26100 puppet 13u unix 0xffff81002f4129c0 1763840 socket
ruby 26100 puppet 14u unix 0xffff81002f413d00 1536103 socket
ruby 26100 puppet 15u unix 0xffff81002f4134c0 1536500 socket
ruby 26100 puppet 16u unix 0xffff81008f9ca440 1540994 socket
ruby 26100 puppet 17u unix 0xffff81008f9ca9c0 1562781 socket
ruby 26100 puppet 18u unix 0xffff8100aad0ec40 1569133 socket
ruby 26100 puppet 19u unix 0xffff81003c53b740 1658921 socket
ruby 26100 puppet 20u unix 0xffff81003c53a140 1691003 socket
ruby 26100 puppet 21u unix 0xffff81003c53a400 1693685 socket
ruby 26100 puppet 22u unix 0xffff81002f412f40 1731935 socket
... etc...
--
Atentamente
Héctor Rivas
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>
Héctor Rivas Gándara
> To workaround the problem you can increase the ulimit for open files
> (/etc/security/limits.conf) and restart puppetmaster periodically
> (¿each 6h?).
About this solution, I think that I found another bug in
puppetmasterd: http://projects.reductivelabs.com/issues/4707
If you change the ulimits probably you should start the daemon using
"sudo -u puppet <initscript>".
--
Atentamente
Héctor Rivas