PHP error log shows "Index is under processing now"

[alx....@gmail.com]

Hi everyone,

Since yesterday i started noticing that when i try to perform a search
in our 1.0.9 installation i get an error 500. When i go looking into
the PHP error log file i see the message

"Index is under processing now"

Is this some kind of automatic processing on the Lucene index files or
is this some weird behavior/error?


Cheers,

Alexandre Paes

[Jesús García Crespo]

Hi Alex,

Did you try to run the debug mode (qubit_dev.php)? What does it say? I think that this could be related to the "open too many files" issue.

I would try to optimize your index:

$ php symfony search:optimize QubitSearch

Regards,

--
Jesús García Crespo,
Software Engineer, Artefactual Systems Inc.
http://www.artefactual.com | +1.604.527.2056

[alx....@gmail.com]

Hi Jesús,

I edited the index.php to enable debug as stated on the wiki and got
the following backtrace:

# at ()
in SF_ROOT_DIR/plugins/sfLucenePlugin/lib/vendor/Zend/Search/
Lucene.php line 297 ...

294. }
295. }
296. } else {
297. throw new Zend_Search_Lucene_Exception($e-
>getMessage(), $e->getCode(), $e);
298. }
299. }
300.

# at
Zend_Search_Lucene::getActualGeneration(object('xfLuceneEnhancedFilesystem'))
in SF_ROOT_DIR/plugins/sfLucenePlugin/lib/vendor/Zend/Search/
Lucene.php line 529 ...

526. // Mark index as "under processing" to prevent
other processes from premature index cleaning
527.
Zend_Search_Lucene_LockManager::obtainReadLock($this->_directory);
528.
529. $this->_generation =
self::getActualGeneration($this->_directory);
530.
531. if ($create) {
532. require_once 'Zend/Search/Lucene/
Exception.php';

# at Zend_Search_Lucene-
>__construct(object('xfLuceneEnhancedFilesystem'), )
in SF_ROOT_DIR/plugins/sfLucenePlugin/lib/engine/
xfLuceneEngine.class.php line 133 ...

130.
131. if (file_exists($this->location . '/segments.gen'))
132. {
133. $this->index = new Zend_Search_Lucene($fs, false);
134. }
135. else
136. {

# at xfLuceneEngine->open()
in SF_ROOT_DIR/lib/search/QubitSearch.class.php line 28 ...

25. protected function initialize()
26. {
27. $this->setEngine(new
xfLuceneEngine(sfConfig::get('sf_data_dir').'/index'));
28. $this->getEngine()->open();
29. }
30.
31. /**

# at QubitSearch->initialize()
in SF_ROOT_DIR/plugins/sfSearchPlugin/lib/index/
xfIndexCommon.class.php line 57 ...

54.
55. $this->setLogger(new xfLoggerBlackhole);
56.
57. $this->initialize();
58. }
59.
60. /**

I also tried to optimize the index using the command line you stated
but i get a huge red box with:

Index is under processing now

If i go to the data/index directory i find several files, the
following files called out to me:

optimization.lock.file
read.lock.file
read-lock-processing.lock.file
segments_49y
segments_49z
segments.gen
write.lock.file

I have googled around and can see some posts from early 2009 with
people with similar problems on other applications and such but
curiously enough none of the articles i found had any explanation or
solution to this problem. Do any of you guys have any ideas? How could
i check that 'too many open files' problem?

Sorry for the huge post and thanks for trying to help me with this
issue,

Alexandre Paes

On 22 Set, 22:20, Jesús García Crespo <je...@artefactual.com> wrote:
> Hi Alex,
>

[Jesús García Crespo]

Hi Alex,

On Thu, Sep 23, 2010 at 1:12 PM, alx....@gmail.com <alx....@gmail.com> wrote:

If i go to the data/index directory i find several files, the
following files called out to me:

optimization.lock.file
read.lock.file
read-lock-processing.lock.file
segments_49y
segments_49z
segments.gen
write.lock.file

I have googled around and can see some posts from early 2009 with
people with similar problems on other applications and such but
curiously enough none of the articles i found had any explanation or
solution to this problem. Do any of you guys have any ideas? How could
i check that 'too many open files' problem?

Unfortunately, I don't know too much about Zend Lucene internals, but it seems that your index has a problem with the locking mechanism. I'd suggest you to search for a way to force the unlocking at Zend Lucene documentation. If you don't find anything interesting, I would try to empty the index directory and create a new index.

Regards,

 

[alx....@gmail.com]

Hi again Jesús,

I've looked at the Zend Lucene and tried to unlock the index by using
Zend_Search_Lucene_Storage_File_Filesystem::unlock() but i couldn't
get the Zend_Search_Lucene_Storage_File constructor to grab the file
handle for the index so i decided and simply rebuilt the index, took a
little bit of time but it's better than messing up the index without
knowing what i might be messing with.

Thanks for the help,

Alexandre Paes

On 23 Set, 12:33, Jesús García Crespo <je...@artefactual.com> wrote:
> Hi Alex,
>

[alx....@gmail.com]

Oooops, little correction, only now have i looked at the output from
the index rebuild and noticed that i get a segmentation fault a few
minutes into the index rebuild process... Any idea how i might debug
this and find out what went wrong?

Thanks,

Alexandre Paes

On 23 Set, 12:33, Jesús García Crespo <je...@artefactual.com> wrote:
> Hi Alex,
>

[Jesús García Crespo]

Hi Alex,

On Thu, Sep 23, 2010 at 3:03 PM, alx....@gmail.com <alx....@gmail.com> wrote:

Oooops, little correction, only now have i looked at the output from
the index rebuild and noticed that i get a segmentation fault a few
minutes into the index rebuild process... Any idea how i might debug
this and find out what went wrong?

This is an outstanding issue that is being really difficult to get rid of because it seems a problem with the PHP core. However, you can restart the index build process from the point that it segfaults thanks to a simple patch that I wrote some weeks ago.

Please see this message for some more details:

http://groups.google.com/group/ica-atom-users/msg/71459e677de40b9e

Regards,

 

[alx....@gmail.com]

Hi Jesús,

sorry to be bothering again, i've read the message you pointed me to
but i'm not exactly sure how to use the

(ioOffset and actorOffset; value '-1' means ignore all items)

that you mention. After applying this patch how should i call this on
the command line? from the failed rebuild process i can't seem to gain
any information about either ioOffset nor actorOffset, can you please
elaborate a bit on what these values are and where can i get
information about them?

Thanks again,

Alexandre Paes

On 23 Set, 14:21, Jesús García Crespo <je...@artefactual.com> wrote:
> Hi Alex,
>

[Jesús García Crespo]

Hi Alex,

On Thu, Sep 23, 2010 at 4:44 PM, alx....@gmail.com <alx....@gmail.com> wrote:

(ioOffset and actorOffset; value '-1' means ignore all items)

that you mention. After applying this patch how should i call this on
the command line? from the failed rebuild process i can't seem to gain
any information about either ioOffset nor actorOffset, can you please
elaborate a bit on what these values are and where can i get
information about them?

Yes, actually I expected this question, :-).

The first thing you have to know is that search:populate task will index both actors and information objects, in this order. After this patch is applied, you'll see that when a new item is indexed, an enumeration is appended to the output line. I will give you two different situations in order to show you how to restart the process:

1st case) the index process segfaults while actors are being indexed:

- Actor 1 (1/3)

- Actor 2 (2/3)

Segmentation fault!

What to do? Run: php symfony search:populate QubitSearch --actorOffset=2

(actors index will be restarted from the third position)

2nd case) the index process segfaults while information objects are being indexed:

- Actor 1 (1/3)

- Actor 2 (2/3)

- Actor 3 (3/3)

- Information object 1 (1/18)

- Information object 2 (2/18)

- Information object 3 (3/18)

- Information object 4 (4/18)

Segmentation fault!

What to do? Run: php symfony search:populate QubitSearch --actorOffset=-1 --ioOffset=4

(actors will be ignored this time and information object index will start from the fifth position) 

Of course, if you got a segfault and you are planning to restart the process, please do not remove your data index directory. But, if you are planning to start the index build process from scratch, to remove the data/index contents will make the build process to run faster.

After the index is built, I'd suggest you to optimize it!

Please, let me know if you need more help.

[alx....@gmail.com]

Hi again Jesús,

using this patch everything went quite smooth, the rebuild process
still segfaulted (now i noticed it always segfaults on the same
information object, maybe something wrong with the record?) but after
running the rebuild process with the new command line options
everything completed without any more hickups.

I would just like to thank you very much for helping me out with these
issues.


Regards,

Alexandre Paes

On 23 Set, 16:26, Jesús García Crespo <je...@artefactual.com> wrote:
> Hi Alex,
>

> On Thu, Sep 23, 2010 at 4:44 PM, alx.p...@gmail.com <alx.p...@gmail.com>wrote:
>
> > (ioOffset and actorOffset; value '-1' means ignore all items)
>
> > that you mention. After applying this patch how should i call this on
> > the command line? from the failed rebuild process i can't seem to gain
> > any information about either ioOffset nor actorOffset, can you please
> > elaborate a bit on what these values are and where can i get
> > information about them?
>
> Yes, actually I expected this question, :-).
>
> The first thing you have to know is that search:populate task will index
> both actors and information objects, in this order. After this patch is
> applied, you'll see that when a new item is indexed, an enumeration is
> appended to the output line. I will give you two different situations in
> order to show you how to restart the process:
>
> 1st case) the index process segfaults while actors are being indexed:
>
> - Actor 1 (1/3)
> - Actor 2 (2/3)
> Segmentation fault!
>

> What to do? Run: *php symfony search:populate QubitSearch --actorOffset=2*

> (actors index will be restarted from the third position)

> *
> *

> 2nd case) the index process segfaults while information objects are being
> indexed:
>
> - Actor 1 (1/3)
> - Actor 2 (2/3)
> - Actor 3 (3/3)
> - Information object 1 (1/18)
> - Information object 2 (2/18)
> - Information object 3 (3/18)
> - Information object 4 (4/18)
> Segmentation fault!
>

> What to do? Run: *php symfony search:populate QubitSearch --actorOffset=-1
> --ioOffset=4*