High level overview of migration

[kco...@hmdc.harvard.edu]

Rationale:
Dataverse v4 is a complete redesign, including the database schema. Because of this moving from v3.x to v4.x requires a migration, involving mapping 3.x tables to v4.x tables, cleaning or adjusting certain types and fields, and dropping others.

High level procedure:
1. Upgrade from v3.x to v3.6.3.
2. Create a test copy of the v3.6.3 database.
3. Create a test or target v4.x installation.
4. Perform test migrations from a copy of the v3.x database until all issues are resolved.
    -create successful checkpoint copies of the migrated database along the way to make iterating easier.
5. Once you have a successful procedure, including cleaning data, plan your actual production migration.

Migration Scripts, Status of Migration:
Our migration scripts are what we used to migrate our own production environment to v4.0. They are a collection of database and shell scripts that migrate components of our application, roughly corresponding to the main objects: Dataverses, Datasets, Files, Users.
They are in their original state and could use some polishing.

The migration scripts and instructions can be found on GitHub here:
https://github.com/IQSS/dataverse/tree/develop/scripts/migration

Presteps, involving extracting data from v3.6.3 database are here:
https://github.com/IQSS/dataverse/blob/develop/scripts/migration/migration_presteps.txt

Migration instructions are here:
https://github.com/IQSS/dataverse/blob/develop/scripts/migration/migration_instructions.txt

Potentially an Iterative Process:
For a relatively small, simple installation, the migration may complete smoothly with few if any extra steps. For larger installations, especially those using custom metadata, there may be some iteration required as non-conforming data such as dates, duplicate email addresses, or custom fields are encountered. For this reason it is important to plan ahead: allow time to work out the steps for your particular installation, save your work as you go so you can restart at the last successful point.