Lily snapshot r4706: index stuck in BUILD_REQUESTED stage

[Michael Kurze]

Hello again,

After doing a parallel import of a large data set without any index configured (just FYI: I had set mqFeeder enabled=false as that option had tickled me), I tried building an index over my data.

The setup: CDH3b4, hbase and lily with a 3-node zk-quorum, and hbase on a non-standard znode.

First of all, I got an error trying to install this indexer configuration which worked fine on lily-0.2.1 (and on lily-0.3, at least using the lily-provided minicluster):
https://github.com/michaelku/bugzilla_etl/blob/master/configuration/lily/indexer/indexerconf.xml

========
[INFO ][10:50:41,648][main      ] org.lilyproject.client.LilyClient - Current Lily servers = [10.2.72.183:42883]

Failed to parse & build the indexer configuration.

If this problem occurs because no Lily node is available
or because certain field types or record types do not exist,
then you can skip this validation using the option --force

Indexer configuration: Error in the configuration.
null
null
========


After that, I retried it using "--force" and got the index to install.
When I updated the index to request a rebuild using map-reduce, it got stuck in this state:


========
Number of indexes: 1

development
  + General state: ACTIVE
  + Update state: SUBSCRIBE_AND_LISTEN
  + Batch build state: BUILD_REQUESTED
  + Queue subscription ID: IndexUpdater_development
  + SOLR shards:
    + development_1: http://[host]:1337/solr/
========


From there, it does not seem to progress. I can find no errors in the jobtracker logs or in the lily logs.
Also, I cannot update the index to the DELETE_REQUESTED state:

========
org.lilyproject.indexer.model.api.IndexUpdateException: Cannot move index build state from BUILD_REQUESTED to BUILD_REQUESTED
    at org.lilyproject.indexer.model.impl.IndexerModelImpl.updateIndex(IndexerModelImpl.java:248)
    at org.lilyproject.indexer.admin.cli.UpdateIndexCli.run(UpdateIndexCli.java:103)
    at org.lilyproject.cli.BaseCliTool.runBase(BaseCliTool.java:158)
    at org.lilyproject.cli.BaseCliTool.start(BaseCliTool.java:50)
    at org.lilyproject.indexer.admin.cli.UpdateIndexCli.main(UpdateIndexCli.java:34)
========


I should mention that the first full rebuild I attempted failed because of wrong permissions in our hadoop setup.
Getting stuck, I manually deleted that index configuration from zookeeper before trying another rebuild. That got stuck again without error messages then.

Which also makes me wonder, is there a safe way to "reset" the lily state in zk?

Thanks for your help, have a nice Weekend!
Michael

[Evert Arckens]

Hi Michael,

I'm not sure yet why you would get stuck in the BUILD_REQUESTED state.
Are you sure the solr setup is correct? And have you put the "mqFeeder enabled" option back to false (not really needed though for triggering the map-reduce rebuild).

But the exception you're seeing when trying to go to the DELETE_REQUESTED state is because we currently do not allow this transition when the Batch build state is in BUILD_REQUESTED or BUILDING. We could consider to allow this transition, but it needs some extra thought first.

Regards,
Evert.

--
Evert Arckens

http://outerthought.org/
Open Source Content Applications
Makers of Kauri, Daisy CMS and Lily

[Bruno Dumon]

Hi,

Sorry for the late reply, but if you would happen to have this
environment still around, it would be interesting to check the
lily-server log file, it should contain a stacktrace of the error in
the indexer configuration, most likely following on the message
"Problem starting index updater for index".

Unfortunately, lily-add-index currently does not print it out (I'll
look into adding an option for this).

On Fri, Mar 11, 2011 at 9:17 PM, Michael Kurze
<mr.micha...@googlemail.com> wrote:

--
Bruno Dumon
Outerthought
http://outerthought.org/

[Michael Kurze]

Hi Bruno,

On Tuesday, March 22, 2011 6:55:39 PM UTC+1, Bruno Dumon wrote:

Hi,

Sorry for the late reply, but if you would happen to have this
environment still around, it would be interesting to check the
lily-server log file, it should contain a stacktrace of the error in
the indexer configuration, most likely following on the message
"Problem starting index updater for index".

sorry, I don’t have the environment anymore. In the future I’ll make a point of archiving all logs. But I am pretty sure there was no stack trace in the lily logs.

Anyway, since that event was preceded by a misconfiguration of mine (wrong permissions on mapred folder in hdfs),
and a full reimport seems to have solved things, I am fine with the current state of affairs.

What I still would suggest though is an admin command to allow resetting zookeeper state in the event of such problems, or to force an index removal.
 
As always, thanks for both your help,
Michael