Indexes stuck in INSTALLED status

[Brandon Dean]

I've been struggling with my indexes getting stuck in the INSTALLED status with no apparent way to get them out of that status.  After much frustration, I started with a completely fresh install of JanusGraph 0.1.1, left the configuration as is, and then followed the steps from the documentation exactly to create a new composite index.  I was able to successfully create my index and it moved into the REGISTERED status and then the ENABLED status after issuing a REINDEX command.

1490727 [gremlin-server-session-1] INFO  org.janusgraph.graphdb.database.management.GraphIndexStatusWatchemposite do not currently have status REGISTERED: name=ENABLED

I then attempted to follow the documentation to delete this same index using the following steps:

gremlin> :remote connect tinkerpop.server /opt/vdp/janus/conf/remote.yaml session

gremlin> :remote console

gremlin> m = graph.openManagement()

gremlin> i = m.getGraphIndex('byNameComposite')

gremlin> m.updateIndex(i, SchemaAction.DISABLE_INDEX).get()

gremlin> m.commit()

Rather than moving to a DISABLED state though, my index is now back in the INSTALLED state and I am unable to move it to REGISTERED or DISABLED.  I've tried issuing additional commands to DISABLE or even REGISTER the index but they all time out and the status never changes from INSTALLED.

gremlin> m = graph.openManagement()

gremlin> i = m.getGraphIndex('byNameComposite')

gremlin> i.getIndexStatus(m.getPropertyKey('name'))

==>INSTALLED

gremlin> m.rollback()

gremlin> m.updateIndex(i, SchemaAction.REMOVE_INDEX).get()

Update action [REMOVE_INDEX] cannot be invoked for index with status [INSTALLED]

I'm currently the only user on this server.  I've performed rollbacks just to be certain and insured that there are no other open instances.  Anytime I issue an awaitGraphIndexStatus it times out and the status never changes.  What's the proper way to get this index to move out of the INSTALLED status?  Alternatively, what might be blocking that from occuring if it is supposed to happen automatically?

gremlin> graph.getOpenTransactions()

==>standardjanusgraphtx[0x18489ed6]

gremlin> graph.tx().rollback()

==>null

gremlin> graph.getOpenTransactions()

==>standardjanusgraphtx[0x18489ed6]

gremlin> graph.openManagement().getOpenInstances()

==>0a7f01141301102-dcwidphiat002-edc-nam-gm-com1(current)

gremlin> graph.openManagement().awaitGraphIndexStatus(graph, 'byNameComposite').call()

Script evaluation exceeded the configured 'scriptEvaluationTimeout' threshold of 30000 ms or evaluation wa request [graph.openManagement().awaitGraphIndexStatus(graph, 'byNameComposite').call()]: sleep interrupte

Any help for the new guy (me) would be very much appreciated!

[Brandon Dean]

After some further investigation, I realized I did apparently have one transaction that was holding things up.  Using graph.tx().rollback() had no effect so I used this method I found after some searching:

gremlin> size = graph.getOpenTransactions().size();

==>1

gremlin> for(i=0;i<size;i++) {graph.getOpenTransactions().getAt(0).rollback()}

==>null

That alone didn't make any difference but after doing that I ran the following commands and my index finally moved from INSTALLED to REGISTERED:

gremlin> mgmt = graph.openManagement()

gremlin> byName= mgmt.getGraphIndex('byNameComposite');

gremlin> propkey = mgmt.getPropertyKey('name');

gremlin> byName.getIndexStatus(propkey);

==>INSTALLED

gremlin> mgmt.updateIndex(byName, SchemaAction.REGISTER_INDEX).get()

gremlin> byName.getIndexStatus(propkey);

==>INSTALLED

gremlin> mgmt.commit()

gremlin> mgmt = graph.openManagement()

gremlin> byName.getIndexStatus(propkey);

==>INSTALLED

gremlin>  mgmt.awaitGraphIndexStatus(graph, 'byNameComposite').call()

==>GraphIndexStatusReport[success=true, indexName='byNameComposite', targetStatus=REGISTERED, notConverged={}, converged={name=REGISTERED}, elapsed=PT0.008S]

It's not clear to me whether calling awaitGraphIndexStatus actually triggered anything or (if as I suspect) it is just monitoring the status and I needed to wait a little longer.

I also need to do some more testing with the disable process because none of this addresses the fact that when I tried to disable my index it got stuck in the INSTALLED state rather than moving to a DISABLED state as I expected.

[Rafael Fernandes]

I've helped another user that asked me me in private exactly the same issue as yourself and my solution worked for him... you may wanna try it as well:

Let me know if it works so I can push a "workaround" section on documentation...
see below my email

=======
Reindexing has been tough for me too, so I had to write few wrapper classes and tons of debugging to make sure I didn't miss anything, but what I found out was that, some of my transactions were still opened, remember, titan/Janus open several threads and they might have opened transactions that you don't know about...
I'd suggest the following, if you can:
1) stop all your app servers
2) use gremlin console or write a simple java program
3) execute steps 29.1.5.1, http://docs.janusgraph.org/latest/index-admin.html
3.1) set this storage.parallel-backend-ops=false
3.2) set this ids.num-partitions=1
3.3) depending how large your graph is, this shouldn't take long...
4) once your indexes are done, bring up your servers.

That should do the trick, it did for me...
Enjoy,
Rafa

[Brandon Dean]

Thanks Rafael, I did see your post and I gave those settings a shot but that alone didn't seem to resolve the problem.  Following the steps in my last post seems to be a repeatable solution to getting indexes out of the INSTALLED status though.

I did some more testing on the disable process and it worked fine this time as long as I:

1. performed a commit after issuing the DISABLE_INDEX
2. insured that I had no open transactions
3. waited patiently for the logs to indicate that the index had in fact been disabled before trying to perform any other operations on the index
   

At this point I feel like I know how to get things back on track, even though I still don't have a good understanding of how they got off track in the first place.  Thanks again for your response!

[Ajay Choudary]

@Brandon, I also got the same issue in 0.2.0 

And tried the steps in your answer, it worked for me as well.

But no clue what went wrong earlier. 

Strange coincidence is first tried `m = graph.openManagement()` in failed scenario and `mgmt = graph.openManagement()` used later   :')

[dengzim...@gmail.com]

this is a problem caused by multi-thread, and it may be fixed by : https://github.com/JanusGraph/janusgraph/pulls?q=is%3Aopen+is%3Apr

if can't be fixed, i will lookout the source code to find the bug.

[kumargau...@gmail.com]

I was able to resolve this issue-

1. 

graph.getOpenTransactions().forEach { tx -> tx.rollback() }

2. 

mgmt = graph.openManagement()

mgmt.getOpenInstances() //all open instances

mgmt.getOpenInstances().forEach {

  if (it.reverse().take(1) != ")") {

    mgmt.forceCloseInstance(it)

  }

}

mgmt.commit()

3.

ManagementSystem.awaitGraphIndexStatus(graph, 'quadTextString').status(SchemaStatus.REGISTERED).call()

ManagementSystem.awaitGraphIndexStatus(graph, 'quadTextString').status(SchemaStatus.ENABLED).call()