Can't get statistics to work after upgrading from 1.8.x to 5.x

[Tiago Delboni]

Hi!

I'm working on upgrading DSpace from 1.8.2 to 5.4. I guess was able to put everything working but the statistics. Everytime I click the Statistics menu, in JSPUI, DSpace come out with a Internal Server Error.
Bellow is the relevant part gattered from dspace.log. Any ideas how to fix it?

Also, I would like to know if I can keep the historical statistics I have on 1.8.x. What I did, trying to achieve this, was to copy the contents from [dspace1.8]/log and [dspace1.8]/solr/statistics/data to [dspace5.4] and ran [dspace5.4]/bin/dspace solr-reindex-statistics.

2015-12-17 13:15:12,864 WARN  org.dspace.app.webui.servlet.InternalErrorServlet @ :session_id=0CFF63BC0604F2B56DD0EE375E575ED9:internal_error:-- URL Was: https://dspace-novo.almg.gov.br/jspui/statistics
-- Method: GET
-- Parameters were:

java.lang.NullPointerException
        at org.dspace.app.webui.servlet.StatisticsServlet.showStatistics(StatisticsServlet.java:121)
        at org.dspace.app.webui.servlet.StatisticsServlet.doDSPost(StatisticsServlet.java:68)
        at org.dspace.app.webui.servlet.StatisticsServlet.doDSGet(StatisticsServlet.java:49)
        at org.dspace.app.webui.servlet.DSpaceServlet.processRequest(DSpaceServlet.java:119)
        at org.dspace.app.webui.servlet.DSpaceServlet.doGet(DSpaceServlet.java:67)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
        at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:844)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:673)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1526)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1482)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
2015-12-17 13:15:12,865 WARN  org.dspace.app.webui.util.UIUtil @ Unable to send email alert
java.lang.NullPointerException
        at org.dspace.storage.rdbms.DatabaseManager.queryTable(DatabaseManager.java:230)
        at org.dspace.content.DSpaceObject$MetadataCache.retrieveMetadata(DSpaceObject.java:1330)
        at org.dspace.content.DSpaceObject$MetadataCache.get(DSpaceObject.java:1265)
        at org.dspace.content.DSpaceObject.getMetadata(DSpaceObject.java:676)
        at org.dspace.content.DSpaceObject.getMetadata(DSpaceObject.java:585)
        at org.dspace.content.DSpaceObject.getMetadataFirstValue(DSpaceObject.java:653)
        at org.dspace.eperson.EPerson.getFirstName(EPerson.java:772)
        at org.dspace.eperson.EPerson.getFullName(EPerson.java:748)
        at org.dspace.app.webui.util.UIUtil.sendAlert(UIUtil.java:419)
        at org.dspace.app.webui.servlet.InternalErrorServlet.doGet(InternalErrorServlet.java:54)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:721)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:468)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:391)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:318)
        at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:439)
        at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:382)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
        at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:844)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:673)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1526)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1482)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

[Monika Mevenkamp]

I am getting ready to do the same uproar - and I pieced together some notes based on
wiki-doc: https://wiki.duraspace.org/display/DSDOC5x/Upgrading+DSpace  


you can test whether your solr version is outdated (probably you are ok there since you say  everything but stats works)

# test that index is outdated   anything smaller than 3.5
/dspace/bin/dspace dsrun org.dspace.app.util.IndexVersion /dspace/solr/statistics/data/index

but if your versions are off do the following - get yourself an uptodate  lucene-core-3.5.0.jar

# upgrade statistics index
# in home directory we have the jar parked
java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader  /dspace/solr/statistics/data/index/

# expect to see 3.5
/dspace/bin/dspace dsrun org.dspace.app.util.IndexVersion /dspace/solr/statistics/data/index 

#update index
ant update_solr_indexes
/dspace/bin/dspace dsrun org.dspace.app.util.IndexVersion /dspace/solr/statistics/data/index 

.......
Monika Mevenkamp
http://tinyurl.com/mevenkamp

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To post to this group, send email to dspac...@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

[Tiago Delboni]

Hi Monika!

Index version looks good:

dspace@dspace-novo:/opt/dspace/deploy/bin> ./dspace dsrun org.dspace.app.util.IndexVersion /opt/dspace/deploy/solr/statistics/data/index

4.10.2

2015-12-17 13:53 GMT-02:00 Monika C. Mevenkamp <mon...@princeton.edu>:

You need to upgrade your statistics and solr index ? 

I am getting ready to do the same uproar - and I pieced together some notes based on 

wiki-doc: https://wiki.duraspace.org/display/DSDOC5x/Upgrading+DSpace   

you can test whether your solr version is outdated (probably you are ok there since you say  everything but stats works) 

# test that index is outdated   anything smaller than 3.5

/dspace/bin/dspace dsrun org.dspace.app.util.IndexVersion /dspace/solr/statistics/data/index

but if your versions are off do the following - get yourself an uptodate  lucene-core-3.5.0.jar

# upgrade statistics index

# in home directory we have the jar parked

java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader  /dspace/solr/statistics/data/index/

# expect to see 3.5

/dspace/bin/dspace dsrun org.dspace.app.util.IndexVersion /dspace/solr/statistics/data/index  

#update index 

ant update_solr_indexes

/dspace/bin/dspace dsrun org.dspace.app.util.IndexVersion /dspace/solr/statistics/data/index  

—

Monika Mevenkamp

Digital Repository Infrastructure Developer

Princeton University

Phone: 609-258-4161

Skype: mo-meven

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To post to this group, send email to dspac...@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

--

Tiago M Delboni

+55 31 8898-5873 | mobile

tdelboni | skype

br.linkedin.com/in/delboni | linkedin

facebook.com/tiago.delboni | facebook

[Andrea Schweer]

Hi,

On 18/12/15 04:39, Tiago Delboni wrote:

Hi!

I'm working on upgrading DSpace from 1.8.2 to 5.4. I guess was able to put everything working but the statistics. Everytime I click the Statistics menu, in JSPUI, DSpace come out with a Internal Server Error.
Bellow is the relevant part gattered from dspace.log. Any ideas how to fix it?

Also, I would like to know if I can keep the historical statistics I have on 1.8.x. What I did, trying to achieve this, was to copy the contents from [dspace1.8]/log and [dspace1.8]/solr/statistics/data to [dspace5.4] and ran [dspace5.4]/bin/dspace solr-reindex-statistics.

The statistics page you're referring to (compare http://demo.dspace.org/jspui/statistics) is not related to the Solr-based usage statistics at all. Instead, this page is showing the legacy statistics populated from the log files and the .dat / .html files generated from the log files via the stat-* commands listed in the "Legacy statistics" section here: https://wiki.duraspace.org/display/DSDOC5x/Command+Line+Operations#CommandLineOperations-Legacystatistics

My guess is that you didn't copy over the contents of the report directory (that holds these .dat and .html files); looking at the code at the line below supports that. The location of the report directory is configured in dspace.cfg:
https://github.com/DSpace/DSpace/blob/dspace-5_x/dspace/config/dspace.cfg#L1741

So you might like to check what you have yours set to and copy the directory across from your 1.8 [dspace] directory tree to your 5.x directory tree.

2015-12-17 13:15:12,864 WARN  org.dspace.app.webui.servlet.InternalErrorServlet @ :session_id=0CFF63BC0604F2B56DD0EE375E575ED9:internal_error:-- URL Was: https://dspace-novo.almg.gov.br/jspui/statistics
-- Method: GET
-- Parameters were:

java.lang.NullPointerException
        at org.dspace.app.webui.servlet.StatisticsServlet.showStatistics(StatisticsServlet.java:121)

https://github.com/DSpace/DSpace/blob/dspace-5_x/dspace-jspui/src/main/java/org/dspace/app/webui/servlet/StatisticsServlet.java#L121

cheers,
Andrea

-- 
Dr Andrea Schweer
IRR Technical Specialist, ITS Information Systems
The University of Waikato, Hamilton, New Zealand
+64-7-837 9120

[Tiago Delboni]

Hi Andrea, kudos!

All I had to do was run the legacy commands and create the missing "reports" directory - thank you!

Now, if these are the legacy statistics, where are the new one and how can I import old stat into it?

dspace@dspace-novo:/opt/dspace/deploy/bin> ./dspace stat-initial
dspace@dspace-novo:/opt/dspace/deploy/bin> ./dspace stat-general
dspace@dspace-novo:/opt/dspace/deploy/bin> ./dspace stat-monthly
dspace@dspace-novo:/opt/dspace/deploy/bin> ./dspace stat-report-initial
Unable to write to output file /opt/dspace/deploy/reports//report-2005-1.html
dspace@dspace-novo:/opt/dspace/deploy/bin> mkdir /opt/dspace/deploy/reports
dspace@dspace-novo:/opt/dspace/deploy/bin> ./dspace stat-report-initial
dspace@dspace-novo:/opt/dspace/deploy/bin> ./dspace stat-report-general
dspace@dspace-novo:/opt/dspace/deploy/bin> ./dspace stat-report-monthly

[geneviev...@anu.edu.au]

On your communities, collections or items pages in the default UI installation you want to look for a blue icon with vertical bars of varying heights.  Click on this and it will take you to the stats page for that section.

To import the old statistics you want to use 'stats-log-converter' on your old dspace.log files, and then run 'stats-log-importer' to import them into solr.