Reindex error: "failed to open stream: Too many open files"

[Vincent]

Hi there!

Now, I thought I had everything working with creating and restoring
backups, reindexing by executing "php symfony search:populate
QubitSearch" and so on...

However... now I received the following error some time after
executing the reindex statement:

-------------
Warning: Zend_Search_Lucene_Storage_Directory_Filesystem::require_once
(Zend/Search/Lucene/Storage/File/Filesystem.php): failed to open
stream: Too many open files in /path/icaatom-1.0.8.1/plugins/
sfLucenePlugin/lib/vendor/Zend/Search/Lucene/Storage/Directory/
Filesystem.php on line 183

Fatal error:
Zend_Search_Lucene_Storage_Directory_Filesystem::require_once():
Failed opening required 'Zend/Search/Lucene/Storage/File/
Filesystem.php' (include_path='/path/icaatom-1.0.8.1/plugins/
sfLucenePlugin/lib/vendor:/path/icaatom-1.0.8.1/lib/vendor/symfony/lib/
plugins/sfPropelPlugin/lib/vendor:/path/icaatom-1.0.8.1:/path/
icaatom-1.0.8.1/lib/vendor/symfony/lib:/path/icaatom-1.0.8.1/lib/
vendor/symfony/lib/plugins/sfPropelPlugin/lib/task:/usr/local/lib/
php') in /path/icaatom-1.0.8.1/plugins/sfLucenePlugin/lib/vendor/Zend/
Search/Lucene/Storage/Directory/Filesystem.php on line 183
[root@localhost icaatom-1.0.8.1]#
-------------

When I opened ICA AtoM, only 43 of the 480 archival descriptions were
visible. I have executed the statement while I had ICA AtoM open in a
browser, but closing this browser and re-run the statement did not
make any difference.

Anyone any idea how to close some files?

[Jesús García Crespo]

Hi Vincent!

You can increase this limit running:

$ ulimit -n 10000

to set 10,000 as a new limit.

To get the current limit:

$ ulimit -n

Regards,

--
Jesús García Crespo

[David Juhasz]

On 23-Nov-09, at 9:37 AM, Jesús García Crespo wrote:

You can increase this limit running:

$ ulimit -n 10000

to set 10,000 as a new limit.

To get the current limit:

$ ulimit -n

Please note that some systems have other ceilings that will limit the number of open files (e.g. kern.maxfiles and kern.maxfilesperproc on Mac OS X [1]).

[1] => http://krypted.com/mac-os-x/maximum-files-in-mac-os-x/

David Juhasz,

Software Engineer, Artefactual Systems Inc.

http://www.artefactual.com | P: 604.527.2056 | F: 604.521.2059

[Vincent]

Thanks, setting ulimit solved one problem, but now I hit another one.
Instead of 43 archival descriptions, there are 84 loaded. The error in
the shell:

-----
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried
to allocate 53 bytes) in /path/icaatom-1.0.8.1/lib/model/om/
BaseInformationObject.php on line 73
[root@localhost icaatom-1.0.8.1]
-----

By the way, it is about 480 archival descriptions in a backup file
with a size of 1,7 MB. I use Fedora 10. I have looked at
http://krypted.com/mac-os-x/maximum-files-in-mac-os-x/ shortly. Maybe
I have to look at it closer (and I haven't google the error yet), but
this is the situation at this moment...

[Jesús García Crespo]

Hi Vincent

I am not sure if I told you this before, because I saw this problem many times in the last weeks. PHP installations normally come with a memory limit to prevent bad written scripts for eating up all available memory on a server.

Five days ago I wrote a message that will help you:

http://groups.google.com/group/ica-atom-users/msg/1a2e67ac919032a7

Please, tell me if you find more difficulties. Take into account that message talks about a memory_limit problem with data-load task. Of course, after you modify your PHP memory_limit, you should run rebuild index task.

Best regards,

--
Jesús García Crespo

[David at Artefactual]

Hi Vincent,

Also see:
http://www.qubit-toolkit.org/wiki/index.php?title=PHP_script_execution_limits


Regards,

[David at Artefactual]

Hi Vincent,

Also this issue also has more information about the file limit
problem:
http://code.google.com/p/qubit-toolkit/issues/detail?id=1054

[Vincent]

Hi Jesus and David

I got the backup restored by first setting memory_limit to a higher
value, then reboot and then setting ulimit to a much higher value.
(First I left the reboot-step out, but I got the same result as
before.

Putting back the backup took at least 3 hours (maybe a bit more), but
now I have something.

Thanks for all the information!