Interesting MQ error

[Kevin Esteb]

I captured this error when running the mq under my supervisor process.

[2009/10/25 00:14:06][mq] error:
[2009/10/25 00:14:06][mq] error: ==============================
[2009/10/25 00:14:06][mq] error: MQ Crashed: 2009-10-25 00:14:06
[2009/10/25 00:14:06][mq] error: ==============================
[2009/10/25 00:14:06][mq] error:
[2009/10/25 00:14:06][mq] error: Trace begun at /wise/sbin/mq.pl line 283
[2009/10/25 00:14:06][mq] error: main::__ANON__('Unable to save message in /data/db/mq/msg-F7BAF9CC-C135-11DE-9117-CF0FB73A093F.txt: Too many open files at /usr/lib/perl5/site_perl/5.8.8/POE/Component/MessageQueue/Storage/FileSystem.pm line 361.^J') called at /usr/lib/perl5/site_perl/5.8.8/POE/Component/MessageQueue/Storage/FileSystem.pm line 361
[2009/10/25 00:14:06][mq] error: POE::Component::MessageQueue::Storage::FileSystem::_write_message_to_disk('POE::Component::MessageQueue::Storage::FileSystem=HASH(0x9225f18)', 'POE::Session=ARRAY(0x9326cf4)', 'POE::Kernel=ARRAY(0x8298ac4)', 'HASH(0x92b64b8)', '_write_message_to_disk', 'POE::Session=ARRAY(0x933ef8c)', undef, '/usr/lib/perl5/site_perl/5.8.8/POE/Component/MessageQueue/Storage/FileSystem.pm', 210, '_expire', 'POE::Component::MessageQueue::Message=HASH(0xd0ca4c4)') called at /usr/lib/perl5/site_perl/5.8.8/POE/Session.pm line 484
[2009/10/25 00:14:06][mq] error: POE::Session::_invoke_state('POE::Session=ARRAY(0x9326cf4)', 'POE::Session=ARRAY(0x933ef8c)', '_write_message_to_disk', 'ARRAY(0xb3499b4)', '/usr/lib/perl5/site_perl/5.8.8/POE/Component/MessageQueue/Storage/FileSystem.pm', 210, '_expire') called at /usr/lib/perl5/site_perl/5.8.8/POE/Kernel.pm line 1079
[2009/10/25 00:14:06][mq] error: eval {...} at /usr/lib/perl5/site_perl/5.8.8/POE/Kernel.pm line 1065
[2009/10/25 00:14:06][mq] error: POE::Kernel::_dispatch_event('POE::Kernel=ARRAY(0x8298ac4)', 'POE::Session=ARRAY(0x9326cf4)', 'POE::Session=ARRAY(0x933ef8c)', '_write_message_to_disk', 1, 'ARRAY(0xb3499b4)', '/usr/lib/perl5/site_perl/5.8.8/POE/Component/MessageQueue/Storage/FileSystem.pm', 210, '_expire', 1256454846.08554, 13077499) called at /usr/lib/perl5/site_perl/5.8.8/POE/Resource/Events.pm line 265
[2009/10/25 00:14:06][mq] error: POE::Kernel::_data_ev_dispatch_due('POE::Kernel=ARRAY(0x8298ac4)') called at /usr/lib/perl5/site_perl/5.8.8/POE/Loop/Select.pm line 328
[2009/10/25 00:14:06][mq] error: POE::Kernel::loop_do_timeslice('POE::Kernel=ARRAY(0x8298ac4)') called at /usr/lib/perl5/site_perl/5.8.8/POE/Loop/Select.pm line 336
[2009/10/25 00:14:06][mq] error: POE::Kernel::loop_run('POE::Kernel=ARRAY(0x8298ac4)') called at /usr/lib/perl5/site_perl/5.8.8/POE/Kernel.pm line 1317
[2009/10/25 00:14:06][mq] error: POE::Kernel::run('POE::Kernel') called at /wise/sbin/mq.pl line 318
[2009/10/25 00:14:06][mq] error: Couldn't open crashlog '/data/db/mq/crashed.log': Too many open files
[2009/10/25 00:14:06][mq] error:
[2009/10/25 00:14:06][mq] error: ==============================
[2009/10/25 00:14:06][mq] error: MQ Crashed: 2009-10-25 00:14:06
[2009/10/25 00:14:06][mq] error: ==============================
[2009/10/25 00:14:06][mq] error:
[2009/10/25 00:14:06][mq] error: Trace begun at /wise/sbin/mq.pl line 283
[2009/10/25 00:14:06][mq] error: main::__ANON__('Unable to save message in /data/db/mq/msg-F7BAF9CC-C135-11DE-9117-CF0FB73A093F.txt: Too many open files at /usr/lib/perl5/site_perl/5.8.8/POE/Component/MessageQueue/Storage/FileSystem.pm line 361.^J') called at /usr/lib/perl5/site_perl/5.8.8/POE/Resource/Events.pm line 279
[2009/10/25 00:14:06][mq] error: POE::Kernel::_data_ev_dispatch_due('POE::Kernel=ARRAY(0x8298ac4)') called at /usr/lib/perl5/site_perl/5.8.8/POE/Loop/Select.pm line 328
[2009/10/25 00:14:06][mq] error: POE::Kernel::loop_do_timeslice('POE::Kernel=ARRAY(0x8298ac4)') called at /usr/lib/perl5/site_perl/5.8.8/POE/Loop/Select.pm line 336
[2009/10/25 00:14:06][mq] error: POE::Kernel::loop_run('POE::Kernel=ARRAY(0x8298ac4)') called at /usr/lib/perl5/site_perl/5.8.8/POE/Kernel.pm line 1317
[2009/10/25 00:14:06][mq] error: POE::Kernel::run('POE::Kernel') called at /wise/sbin/mq.pl line 318
[2009/10/25 00:14:06][mq] error: Unable to save message in /data/db/mq/msg-F7BAF9CC-C135-11DE-9117-CF0FB73A093F.txt: Too many open files at /usr/lib/perl5/site_perl/5.8.8/POE/Component/MessageQueue/Storage/FileSystem.pm line 361.
[2009/10/25 00:14:07][mq] warn: process exited
[2009/10/25 00:14:07][Supervisor] warn: mq has exited
[2009/10/25 00:14:07][Supervisor] error: exit code: 25, was not recognized for mq. restarting not attempted
[2009/10/25 00:14:07][mq] warn: stopping session

[David Snopek]

Hrm. Are you using Throttled? Assuming you are using Throttled with
a max of 2 (per the default) this shouldn't be caused by the MQ. If
not, or with a very high max (not recommended) then you may have to
increase the number of allowed open files via ulimit -n. This could
also be caused by another process opening too many files, but you are
usually only limited per login session (via pam_limits.so) so you
would have to be running the MQ and some other file intense process in
the same login, which seems unlikely. OR you could be hitting the
system wide hard limit, although, that's also very unlikely because
the number is usually really, really huge (cat /proc/sys/fs/file-max).

Probably not a very helpful response, but let me know if any of those
possibilities make you think of anything..

Happy Hacking,
David Snopek.

2009/10/27 Kevin Esteb <kes...@wsipc.org>:

[Kevin Esteb]

I am using the supplied mq.pl so nothing special is being done with file limits. The mq will run for months and then die for unknown reasons, so I started to run it under a supervisor and captured this error. I thought it was interesting and passed it along. The supervisor will now recognized exit code 25 and restart the mq.