DSpace 7 - Importing solr usage statistics

[Agustina Martinez]

Dear all,

We are currently testing exporting our current DSpace 5.10 Solr statistics and importing them into our 7.4 test environment, following the process outlined in the migration page: https://wiki.lyrasis.org/display/DSDOC7x/Upgrading+DSpace.

Our statistics core is huge - close to 300M solr documents, and we were wondering whether this process could take place way before our actual live migration, to minimise downtime for our users. We will be starting with an empty statistics core in a dedicated Solr server. Could we safely perform the following:

1. As close as possible to production migration time export the statistics core using the "solr-export-statistics -i statistics" command
2. Use a test DSpace 7.x installation, with the Solr instance that will be used for the migration to import our exported statistics

Once we have undertaken the production migration, we could then safely run the command "solr-upgrade-statistics-6x -i statistics" to update our pre-DSpace 6 legacy identifiers.

Is this an approach that could be taken to minimise the amount of very time consuming jobs that are needed as part of the migration? Or do any of the Solr commands rely on looking at a migrated DSpace 7.x database.

Best and many thanks,

Agustina

[Tim Donohue]

Hi Agustina,

It appears (to me) that only the "solr-export-statistics" and "solr-import-statistics" scripts only depend on Solr (not the database).  But, the "solr-upgrade-statistics-6x" requires database access (as it uses the database to migrate stats from old IDs to new UUIDs).

So, I think your approach would work.  You should be able to export/import with only Solr around.  You only need the database available once you are ready to upgrade the statistics. 

Technically, the database is minimally used to just lookup objects by their old ID to find the new UUID.  

So, it could even be possible to migrate some of your statistics​ using a copy of the production database... and then migrate the remainder after​ you upgrade production.  For instance, you should be able to run the upgrade statistics script multiple times in order to migrate those stats in batches (e.g. "solr-upgrade-statistics-6x --num-rec 100000" should just migrate 100,000 records... running it again should migrate the next 100,000 it finds, etc.)  Doing the migration in that way may mean that there's fewer statistical records to migrate once you upgrade your production to 7.x

Hopefully if others have done this before they can chime in with additional feedback.  But, based on my reading of the code, this seems like it should work.

Tim

---

From: dspac...@googlegroups.com <dspac...@googlegroups.com> on behalf of Agustina Martinez <amgc...@gmail.com>
Sent: Tuesday, October 25, 2022 3:51 PM
To: DSpace Technical Support <dspac...@googlegroups.com>
Subject: [dspace-tech] DSpace 7 - Importing solr usage statistics

 

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/a272bafc-d7f1-4155-a9ac-5b1aca09eb65n%40googlegroups.com.

[Agustina Martinez]

That is great to hear Tim! I thought that that would be the case re the solr commands not making use of the DB, just the legacy converter one as it needs to ID to UUID matching.

It is very good news that we can perform this process before the actual migration as it would minimise a lot the time where stats are not showing for our users. 

Hearing from anybody that has attempted this approach would be useful

Thank you,

Agustina