Reindexing in Solr
Carl Hall
Ian Boston
This is relatively easy to do, and once I have done KERN-1957 I was
intending to look at the issue of migration and rebuilding, which
would give you the feeds you needed.
However, I dont think its appropriate to start adding new features
into version 1 at this late stage. V1 is already in RCs. There are a
number of critical things effecting other institutions that have
already been pushed out by the managed project, so adding a large
feature should come after fixing those issues.
If you are talking about post V1 (I hope you are), then I dont see any
mention of reindexing in the Roadmap outlined by the URG, so in theory
a new feature of this scale is already out of scope for the next
version. (at least by the managed project)
If you are looking for a local solution, then investigate in more
detail why the lock would not release. What process/thread had hold of
the lock? When was it created? The Solr indexer is single threaded so
unless you have some code that writes to Solr independently or creates
threads, then the most likely cause is that you reconfigured a bundle
and caused the indexer to die leaving the lock behind. The same
problem happens if you cause Jackrabbit to re-start. Its Lucene lock
hangs around and never gets released.
The normal way out of this is to verify what process had the lock and
if that process is not running (you shut the JVM down so its not,
right), then remove the lock file and start the process up again.
Ian
> --
> You received this message because you are subscribed to the Google Groups
> "Sakai Nakamura" group.
> To post to this group, send email to sakai-...@googlegroups.com.
> To unsubscribe from this group, send email to
> sakai-kernel...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/sakai-kernel?hl=en.
>
Clay Fenlason
On Thu, Aug 18, 2011 at 3:50 AM, Ian Boston <i...@tfd.co.uk> wrote:
> ... I dont see any
> mention of reindexing in the Roadmap outlined by the URG, so in theory
> a new feature of this scale is already out of scope for the next
> version. (at least by the managed project)
At the top of that same document [1] you'll see a couple notes to the
effect that URG discussions have not yet included technical
priorities, nor even detailed technical study of its items, and that
what's there should not be confused with a project plan at this stage.
We're particularly mindful of the fact that server team priorities
have not yet been factored in.
I don't know anything about the issue Carl raises, but I did want to
say that ruling anything out of scope because it doesn't show up on
that Confluence page is premature.
~Clay
Carl Hall
Ian Boston
Content, see [1] done in the last hour.
It has unit test coverage but I havent validated absolutely that it works.
You need to write a bundle that does:
Session session = repository.loginAdministrative();
session.getAuthorizableManager().refreshAll();
session.getContentManager().refreshAll();
It will add events to the Solr queue (at upto 60K events/s), and those
will then be indexed in batches of about 200, but thats going to take
a bit longer.
Although the patch adds just under 400 lines, and touches most files
in core, there are no code changes to Nakamura, and no changes to any
code that can be accessed from Nakamura, so I felt this was Ok to add.
If you need to re-index Acls, just ask.
Please test on Oracle.
I have tested on all the other DBs. Cassandra and HBase dont have this
feature at the moment.
HTH
Ian
1. https://github.com/ieb/sparsemapcontent/commit/de5c3a905ba4c719d3958d0f0b420b30c104fb55