warning about SOLR core already exist when upgrading to dspace 5.3/5.4

[Francis Brouns]

Hi all,

we are in the process of upgrading our Dspace 1.8 server to DSpace 5.x. Both in the upgrade to DSpace 5.3 and dspace 5.4 we encountered warnings about solr core already existing. In our DSpace 1.8 server we did not use SOLR. Consequently I am importing legacy log files from 2003 onwards. After having imported a couple of 2003, 2004, etc, I ran the solr sharding. 

When I look at the solr statistics directory, I can see the new cores. The statistics page does show figures with the various categories, so, I can see total count, top visits, etc.

So, the sharding seemed to go without problems, except that a couple of times the same warning is shown in the solr.log file

2015-12-04 16:02:22,852 INFO  org.apache.solr.core.SolrCore @ [statistics] webapp=/solr path=/update params={wt=javabin&version=2} status=0 QTime=0 

2015-12-04 16:02:22,857 INFO  org.apache.solr.handler.admin.CoreAdminHandler @ core create command instanceDir=statistics&name=statistics-2003&action=CREATE&dataDir=/dspace/dspace/solr/statistics-2003/data&wt=javabin&version=2

2015-12-04 16:02:22,857 WARN  org.apache.solr.handler.admin.CoreAdminHandler @ Creating a core with existing name is not allowed

2015-12-04 16:02:22,858 ERROR org.apache.solr.core.SolrCore @ org.apache.solr.common.SolrException: Core with name 'statistics-2003' already exists.

at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:555)

Is there something I should do about the warning, something I have to correct? How can I check whether the 2003 statistics are present? 

Strangely enough the warning is only about the 2003 core and not about the other 3 cores.

I might have run the sharding before optimising the solr indexes. Could that have caused this warning?

Running dspace 5.4 on a Linux server (SLES), jdk 7, tomcat 7, Oracle10g. This same warning also occurs in dspace 5.3.

kind regards,

Francis Brouns

[helix84]

Try taking a look here:

https://jira.duraspace.org/browse/DS-2212
https://jira.duraspace.org/browse/DS-2488

One of them is still an open issue.


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

[Francis Brouns]

Dear helix84,

thanks for your prompt reply. I was aware of both issues and we ran into comparable problems with solr failing at various moments when we tried to use Tomcat 8 and JDK8. With Tomcat 8 and JDK8 we encounter many memory and MMap errors and SOLR would fail repeatedly, occassionally crashing and leaving files in the temp directory. When the latter happens, SOLR does not recover.

We now downgraded to Tomcat7 and JDK7 and no longer run into the creation and memory errors. The sharding process itself does not report any errors, the cores are created at file system. Would you recommend that we not import our old log files and/or do no shard the core into years? It would mean that we lose our statistics and that would be a pity.

best wishes, Francis

[helix84]

> Strangely enough the warning is only about the 2003 core and not about the other 3 cores.

I'm pretty sure that's because it's the oldest and first in order to be created.

In fact, I think I encountered this error myself when sharding, but I
don't remember the specifics anymore. It wasn't hard to resolve, but I
kept a backup of the unsharded core.

> thanks for your prompt reply. I was aware of both issues and we ran into
> comparable problems with solr failing at various moments when we tried to
> use Tomcat 8 and JDK8. With Tomcat 8 and JDK8 we encounter many memory and
> MMap errors and SOLR would fail repeatedly, occassionally crashing and
> leaving files in the temp directory. When the latter happens, SOLR does not
> recover.

I'm not aware of any unresolved problems reported on Tomcat 8 / Java
8, so if you have them, please do report the details into Jira. Sooner
or later everyone will upgrade and we'll want them resolved. Also
mention which JDK you're using.

Personally, I'm running on OpenJDK 7 and Tomcat 8.0.14 without any problems.

> We now downgraded to Tomcat7 and JDK7 and no longer run into the creation
> and memory errors. The sharding process itself does not report any errors,
> the cores are created at file system. Would you recommend that we not import
> our old log files and/or do no shard the core into years? It would mean that
> we lose our statistics and that would be a pity.

I would not recommend to import from the logs because you'll lose man
fields only captured in Solr, not in the logs.
Of course, if you lost some data, you may choose to import only events
from the lost time range from logs into Solr.

If you have a backup of the unsharded statistics core around, I'd
compare whether number of records there matches the sum of records in
your individual sharded cores. Run a "*:*" query and look at
"numFound", or do that in the Solr admin interface.

[Francis Brouns]

Hi helix84,

sorry, for my omission. Until now we have not been using SOLR, only the legacy statistics. Therefore I am converting and importing our logfiles to switch to the new SOLR statistics conform the instructions in the DSpace installation manual.

We are running a SLES Linux server, Sun/Oracle JDK 1.7, Tomcat 7.0.64, Oracle10g (on a separate dedicated DB server) and upgrading from Dspace 1.8.3 to Dspace 5.4.

In September I started testing the upgrade with the DSpace 5.3 release that was available then, on a newly installed server, using Tomcat 8 and Sun/Oracle JDK 1.8. We ran into multiple problems with SOLR that I reported in the forum. I will enter these into Jira. After contacting @Mire we decided to downgrade Tomcat and Java and that seems to prevent the many memory and MMap issues.  

best wishes, Francis Brouns 

[helix84]

Yes, in your case it makes sense to reimport statistics from logs and
try sharding again.

Make sure to delete the statistics-[20xx] cores before sharding again.
Keep a copy of the unsharded core around in case sharding fails. And
compare the number of events before/after sharding, as I suggested.