keeping stats index across upgrade

[Monika Mevenkamp]

From the message below I understand that I will use my solr stats if I upgrade from 1.8 to 5

Is that true ? 

If so, is tere a way around it ? 

Monika

—

Monika Mevenkamp

Digital Repository Infrastructure Developer

Princeton University

Phone: 609-258-4161

Skype: mo-meven

On Oct 30, 2015, at 8:13 AM, gko...@gmail.com wrote:

Sean:

I had the same problem upgrading from DSpace 1.8.2 to DSpace 5.2.  The older indexes won't work when trying to upgrade (even if you manually try to upgrade them).  I ended up dropping my indexes and rebuilding them from scratch after the upgrade.  That meant losing my statistics, but I had that information stored elsewhere, so we went ahead with a clean slate for the upgrade.

George Kozak

Cornell University

Sent from my iPad

On Oct 29, 2015, at 12:23 PM, Sean Carte <sean....@gmail.com> wrote:

I feel like I'm rapidly losing my mind while trying to upgrade a DSpace 1.7.3 installation to 5.3.

I copied the entire c:/dspace directory from the old to the new server (on which I have previously successfully build and installed 5.3), dropped the dspace database and restored a copy from the old server, and then ran ant update. Everything seemed to go well until solr, which failed with the error copied below.

I've followed the instructions at https://wiki.duraspace.org/display/DSDOC5x/Upgrading+DSpace:
- I downloaded lucene-core-3.5.0.jar to the install directory and ran ant update again, but got the same error.

- I tried manually updating the indices:
java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader /dspace/solr/statistics/data/index/
java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader /dspace/solr/oai/data/index/

no errors, or any feedback, and ant update still fails.

Is there anything else I can do?

This is the error:

check_solr_index:
     [echo] Checking if the Solr index at /dspace/solr/statistics/data/index/ is >= Solr 3.5.0
     [java] Java Result: 1

BUILD FAILED
C:\Users\cartes\Desktop\dspace-5.3-src-release\dspace\target\dspace-installer\build.xml:974: The following error occurred while executing this line:
C:\Users\cartes\Desktop\dspace-5.3-src-release\dspace\target\dspace-installer\build.xml:1061: ERROR occurred while checking Solr index version:
Exception in thread "main" java.io.IOException: Could not read Lucene segments files in C:\dspace\solr\statistics\data\index
        at org.dspace.app.util.IndexVersion.getIndexVersion(IndexVersion.java:141)
        at org.dspace.app.util.IndexVersion.main(IndexVersion.java:59)
Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="C:\dspace\solr\statistics\data\index\segments_1")
        at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
        at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41)
        at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
        at org.apache.lucene.store.DataInput.readLong(DataInput.java:154)
        at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:425)
        at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:454)
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:906)
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:752)
        at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:450)
        at org.dspace.app.util.IndexVersion.getIndexVersion(IndexVersion.java:136)
        ... 1 more

Sean

-- 

-- 
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To post to this group, send email to dspac...@googlegroups.com.
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To post to this group, send email to dspac...@googlegroups.com.
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

________________ 

Monika Mevenkamp

mo.m...@gmail.com

http://mo-meven.tumblr.com/

http://mcmprogramming.com/mo.meven/

[Andrea Schweer]

Hi Monika,

On 31/10/15 03:26, Monika Mevenkamp wrote:

From the message below I understand that I will use my solr stats if I upgrade from 1.8 to 5

Is that true ?

In most cases, you should be fine and should not lose any stats data. There are two things that need to be done to the Solr indexes for upgrading 1.8->5:

1. Upgrade the file format on disk to one compatible with the newer version of Solr included in DSpace 5.
2. Re-index your usage statistics core to account for differences in the Solr schema between DSpace 1.8 and DSpace 5. The discovery core (if you're even using discovery in your 1.8 instance) will probably need to be re-indexed too, but that is not an issue since the re-index can pull the information it needs from elsewhere - ie, database and extracted fulltext files. The usage statistics data is stored only in Solr, so the situation is slightly different there.
   

For the first point, there is an auto-migration process in DSpace 5 that will automatically try and upgrade all your Solr indexes to what DSpace 5 needs during the ant update step. Sean and George ran into problems with this process it looks like, but it has worked well for me for several DSpace instances and presumably for many other folks too. So hopefully it will "just work" for you too. There are some hints in the upgrade docs for troubleshooting some of the problems: https://wiki.duraspace.org/display/DSDOC5x/Upgrading+DSpace#UpgradingDSpace-ManuallyUpgradingSolrIndexes

For the second point, there is a reindex command in DSpace 5.3 that you can use. It's documented here: https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics+Maintenance#SOLRStatisticsMaintenance-ReindexSOLRstatistics,forupgradesorwhenevertheSolrschemaforstatisticsischanged (linked from the DSpace 5 upgrade instructions).

Obviously the usual rules apply, test your upgrade on a copy of your data first, make sure you have current backups, etc.

If you're nervous about the process, I guess you could try and backport / customise the code for step 2 and use it to export your usage stats from 1.8 then import into 5.
https://github.com/DSpace/DSpace/blob/dspace-5_x/dspace-api/src/main/java/org/dspace/util/SolrImportExport.java

cheers,
Andrea

-- 
Dr Andrea Schweer
IRR Technical Specialist, ITS Information Systems
The University of Waikato, Hamilton, New Zealand
+64-7-837 9120