Migrate database from Dspace 4 to Dspace 6

[Jose Javier Hernández]

I am trying to migrate a database from DSpace 4 to DSpace 6. I exported the DSpace 4 database and imported it locally in my DSpace 6.3 installation. The database information is as follows

database information

Then I do:

/dspace/bin/dspace database repair
/dspace/bin/dspace database migrate

Database URL: jdbc:postgresql://172.18.0.2:5432/dspace
Migrating database to latest version AND running previously "Ignored" migrations... (Check logs for details)
Migration exception:
java.sql.SQLException: Flyway migration error occurred
    at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:673)
    at org.dspace.storage.rdbms.DatabaseUtils.main(DatabaseUtils.java:187)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
    at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
Caused by: org.flywaydb.core.api.FlywayException: Validate failed: Detected applied migration not resolved locally: 4.1
    at org.flywaydb.core.Flyway.doValidate(Flyway.java:1008)
    at org.flywaydb.core.Flyway.access$100(Flyway.java:72)
    at org.flywaydb.core.Flyway$1.execute(Flyway.java:921)
    at org.flywaydb.core.Flyway$1.execute(Flyway.java:917)
    at org.flywaydb.core.Flyway.execute(Flyway.java:1373)
    at org.flywaydb.core.Flyway.migrate(Flyway.java:917)
    at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:662)
    ... 7 more

Any help? Thanks in advance.

[Tim Donohue]

Hi Jose,

That error is FlywayDB.org (the system we use to update the DSpace Database) telling you that the migration named "4.1" is not found. 

In your pastebin, it looks like that "4.1" migration is a custom migration called "Upgrade to Dspace Vlir schema" and is listed as "Missing".

My guess is you added a custom database migration to your codebase at some point (it says it ran July 1, 2016), and this custom migration has not been copied over into your DSpace 6 codebase.

Good luck,

Tim

---

From: dspac...@googlegroups.com <dspac...@googlegroups.com> on behalf of Jose Javier Hernández <jjbeni...@gmail.com>
Sent: Wednesday, December 4, 2019 9:19 AM
To: DSpace Technical Support <dspac...@googlegroups.com>
Subject: [dspace-tech] Migrate database from Dspace 4 to Dspace 6

 

--
All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/5b196f92-d3fb-4e97-bae0-c31411abbd87%40googlegroups.com.

[Carolyn Brown]

As of DSpace 5 (and above) the DSpace code will automatically upgrade your database.  By default, this will occur AUTOMATICALLY when you restart Tomcat.  So, the steps to manually upgrade the database by running commands on the command-line are not needed.

1) Shut down Tomcat

2) Import your database.

3) If using postgres, make sure you run the following command after importing the database:

CREATE EXTENSION pgcrypto;

4) Then restart Tomcat back on (the database will be upgraded automatically).

--

All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/5b196f92-d3fb-4e97-bae0-c31411abbd87%40googlegroups.com.

--

Carolyn Brown Ray

Web Architect

Office of Information Technology

Muhlenberg College

484-664-3646

[Jose Javier Hernández]

Hi Tim, thanks for your answer.

What can I do to solve this?

Is this lost migration in the DSpace installer or only in the old codebase dspace?

Is it possible to skip that migration and then do it manually?

To unsubscribe from this group and stop receiving emails from it, send an email to dspac...@googlegroups.com.

[Tim Donohue]

Jose,

SQL migrations in DSpace 6.x are stored under the [src]/dspace-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/ directory:

https://github.com/DSpace/DSpace/tree/dspace-6_x/dspace-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres

That error seems to saying that you previously ran a custom migration from that folder which started with "V4.1".  I do not know what that migration is, or what it does, as it is *not* an out-of-the-box DSpace migration.  Therefore, your first step should be to try to find that migration file that was previously run (maybe it is in your old codebase?).  It probably will be named something like "V4.1__Upgrade_to_Dspace_Vlir_schema.sql" based on what your "./dspace database info" command is returning.

If you can find this SQL file and place it in the above directory alongside your other migrations, then this will fix the error.  This is the best solution, as it will ensure Flyway is working properly & is able to validate your database table against *all* the migrations previously run on this table.

The odd thing to me here is that you said you are upgrading from DSpace 4 to 6.  So, I don't understand how you have old DSpace 4.x Flyway migrations (as DSpace 4 didn't use Flyway *at all*, that wasn't introduced until DSpace 5).  Are you *sure* this is a DSpace 4 installation?  It seems to be acting more like a DSpace 5.0 installation -- which would imply your old code should have that migrations folder above, and you can simply move your custom migration to the 6.x source code and rebuild.

Tim

---

From: dspac...@googlegroups.com <dspac...@googlegroups.com> on behalf of Jose Javier Hernández <jjbeni...@gmail.com>

Sent: Wednesday, December 4, 2019 10:46 AM

To: DSpace Technical Support <dspac...@googlegroups.com>

Subject: Re: [dspace-tech] Migrate database from Dspace 4 to Dspace 6

 

To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/f0f6b455-fa20-429e-b4d0-2c161144d9e6%40googlegroups.com.

[Jose Javier Hernández]

Tim,

I'm really not worried about that data "V4.1__Upgrade_to_Dspace_Vlir_schema.sql". I deleted that entry in the schema_version table.

In my database I have other tables of my own, which in that version had a structure and now has some changes, such as UUID for identifiers.

Now I get this error:

Caught exception:
java.lang.RuntimeException: Error attempting to update Bitstream Format and/or Metadata Registries
 at org.dspace.storage.rdbms.DatabaseRegistryUpdater.updateRegistries(DatabaseRegistryUpdater.java:93)
 at org.dspace.storage.rdbms.DatabaseRegistryUpdater.afterMigrate(DatabaseRegistryUpdater.java:121)
 at org.flywaydb.core.internal.command.DbMigrate$3.doInTransaction(DbMigrate.java:250)
 at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
 at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:246)
 at org.flywaydb.core.Flyway$1.execute(Flyway.java:959)

 at org.flywaydb.core.Flyway$1.execute(Flyway.java:917)
 at org.flywaydb.core.Flyway.execute(Flyway.java:1373)
 at org.flywaydb.core.Flyway.migrate(Flyway.java:917)
 at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:662)

 at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:576)
 at org.dspace.storage.rdbms.DatabaseUtils.main(DatabaseUtils.java:221)

 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
 at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)

Caused by: org.dspace.core.exception.DatabaseSchemaValidationException: The schema validator returned: Missing column: item_uuid in public.list_document
 at org.dspace.core.Context.init(Context.java:170)
 at org.dspace.core.Context.<init>(Context.java:126)
 at org.dspace.storage.rdbms.DatabaseRegistryUpdater.updateRegistries(DatabaseRegistryUpdater.java:59)
 ... 17 more

public.list_document is one of those tables

How can I do to pass the data from that format to the new one?

To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/f0f6b455-fa20-429e-b4d0-2c161144d9e6%40googlegroups.com.

[Jose Javier Hernández]

Tim;

I deleted my own tables manually, and the migration says it was done, but when I check the tables they don't have the UUID column as identifier.

Migration shouldn't add UUID to tables?

PD: you're right the database is from DSpace 5

El miércoles, 4 de diciembre de 2019, 12:52:05 (UTC-5), Tim Donohue escribió:

To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/f0f6b455-fa20-429e-b4d0-2c161144d9e6%40googlegroups.com.

[Tim Donohue]

Jose,

That error doesn't make sense to me, as there's no "list_document" table in DSpace.  There's also no column called "item_uuid" in *any* DSpace table. This is sounding like you've either installed a third-party add-on *or* you've customized DSpace with new tables & columns.

This error seems to be coming from Hibernate (which manages the database in DSpace 6).  It seems to be saying that some Java Bean is expecting there to be an "item_uuid" column in the "list_document" table.  But, neither of those things should exist in out-of-the-box DSpace 6.  You could check your DSpace logs to see if other errors from Hibernate are appearing there to give you more clues.

I'm unfortunately not sure how to solve this error as I don't know what these tables/columns are.  If these are from a third-party add-on, you might want to see if there's a newer version of that add-on that is compatible with DSpace 6.  It might simply be that you are using custom code / addon that is only compatible with DSpace 5 and you need to update it.

Tim

---

From: dspac...@googlegroups.com <dspac...@googlegroups.com> on behalf of Jose Javier Hernández <jjbeni...@gmail.com>

Sent: Wednesday, December 4, 2019 3:26 PM

To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/ac8dab55-2eb5-4648-af17-09e21586ef8e%40googlegroups.com.

[Jose Javier Hernández]

"list_document" is a table made by me, I have added some new features that include another 5 tables.

If I delete those tables from the database I am migrating it works.

But I really need to migrate the data they have to the new structure in DSpace 6.

I guess I will have to study the flyway documentation to achieve it.

Thank you very much Tim, thanks for your time !!!

To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/ac8dab55-2eb5-4648-af17-09e21586ef8e%40googlegroups.com.