Data Migration Support

[Paul Marrington]

The Adept Object Database stores Java objects directly into persistent storage. It's not just the primary way of accessing stored data, in fact it's the only way. This makes for a fast efficient system with one major drawback - data migration. If you change the contents or order of persistent fields in a Java DAO or any of it's internal objects then all databases using those objects will fail when they attempt to use the updated code.

This is a common problem with databases. The resolution is called migration. In the relational database world the DBA will write a migration script. It's then a manual process to ensure that the migration script is run on production data at the same time as the new code is promoted. In the shrink-wrap software world it's the responsibility of the install program to recognise an upgrade and migrate the client data.

The Adept Library has a Migration class that does most of the work for you once you have completed some manual steps:

1. Recognise that a DTO needs to change. This isn't as easy as it sounds, since a DTO can have POJO fields - and if those fields change the DTO is no longer valid.
2. If it doesn't yet exist, create a new class called Migrations in the same package as the DTO that is to change.
3. Inside the Migration class create a public static class extending DAO with the same name as the class that requires migration. If the migrating class is itself an inner class, match the class path inside Migration.
4. Copy the code from the class being migrated into the newly created inner class as a further depth of inner class.
5. Change the name of the class just copied to something unique. I use the date or release number.
6. Make it static.
7. Have it implement Migration.Interface.
8. Implement the migrate() method that will return the new changed DAO.
9. If there is a previous migration for this DAO, change it's migrate method to return this DAO rather than the external one.
10. Implement the code changes to the original external DAO.

This is actually easier than it sounds. The best way, as always, is to provide an example:

// File DAOTest.java
public class DAOTest extends TestCase
  {
    static class TestDAO extends DAO
      {
        /***/ public String string;
        /***/ public int integer;
        /***/ public float real;
        /***/ public static class PrimaryIndex
            extends Index
          { /***/ public String string; }
        /***/ public static class SecondaryIndex
            extends Index
          { /***/ public int integer; }
        /***/ public static class TertiaryIndex
            extends Index
          {
            /***/ public int integer;
            /***/ public String string;
          }
      }
  }

.......................................

// File Migrations.java
public class Migrations
  {
    /**
     * Migration for DAOTest DAOs - note that these
     * are inner class DAOs.
     */
    public static class DAOTest
      {
        /** Test migration path */
        public static class TestDAO extends DAO
          {
            /** First on migration path */
            public static class V050511a extends DAO
                implements Migration.Interface
              {
                /***/ public String string;
                /***/ public int integer;
                /***/ public float real;
                /***/ public static class PrimaryIndex
                    extends Index
                  { /***/ public String string; }
                /***/ public static class SecondaryIndex
                    extends Index
                  { /***/ public int integer; }
                /***/ public static class TertiaryIndex
                    extends Index
                  {
                    /***/ public int integer;
                    /***/ public String string;
                  }
                
                /**
                 * @see Migration.Interface#migrate()
                 */
                public DAO migrate()
                  {
                   /*
                    * Originally created DAOTest - 
                    * changed when new migration added
                    */
                    V050511b testDAO = new V050511b();
                    /*
                     * Use DTO deep copy since objects
                     * will be so similar.
                     */
                    if (dto == null)
                      dto = new DTO( this, testDAO);
                    dto.copy( testDAO, this);
                    /*
                     * This is where we would do the
                     * processing that is the essence
                     * of migration. If we added fields
                     * we may need to fill them here.
                     */
                    testDAO.integer++;
                    /*
                     * Lucky last - we return the newly
                     * created replacement DAO.
                     */
                    return testDAO;
                  }
                private static DTO dto;
              }
              
            /** Second on migration path */
            public static class V050511b extends DAO
                implements Migration.Interface
              {
                /***/ public String string;
                /***/ public int integer;
                /***/ public float real;
                /***/ public static class PrimaryIndex
                    extends Index
                  { /***/ public String string; }
                /***/ public static class SecondaryIndex
                    extends Index
                  { /***/ public int integer; }
                /***/ public static class TertiaryIndex
                    extends Index
                  {
                    /***/ public int integer;
                    /***/ public String string;
                  }
              
                /** @see Migration.Interface#migrate() */
                public DAO migrate()
                  {
                    com.marringtons.object.DAOTest.TestDAO
                      testDAO = new
                        com.marringtons.object.DAOTest.TestDAO();
                    if (dto == null)
                      dto = new DTO( this, testDAO);
                    dto.copy( testDAO, this);
                    testDAO.real++;
                    return testDAO;
                  }
                private static DTO dto;
              }
          }
      }
  }

 

The Act of Migration

The worst migration case I've personally experienced was with an accounting program. When upgrading a 5 year old system it required me to install two complete intermediate versions of the software, migrate to them in sequence then install the latest version and migrate a third time.

Adept migration is more transparent. Your setup code must call Migration.database(), giving it the name of the database to migrate. It will review every persistent class and record the operation, so only the parts of the migration path not yet performed will be done. As an example: suppose over 10 releases, a particular DAO has changed 5 times (shame on you). Any user could be upgrading from an earlier version to the latest at that time and migrate again later. The migration code will account for this and pick up the migration thread from where it left off the last time the software was upgraded.

Migration Limitations

1. It only works with classes that inherit from DAO. If a POJO field inside a DAO changes, you will need to create a new POJO with a new name and migrate from the old to the new manually in the DAO migrate(). This is a good reason to inherit from DAO for any persistent non-trivial data object.
2. It must be called by a single threaded setup routine to stop any other possible simultaneous access of the database being migrated.
3. The developer is responsible for providing a migration path before changing a DAO or a POJO inside a DAO. This goes against the practice of changing internals without concern for effect. If you have application specific POJOs inside DAOs, comment the source so they will not be changed by accident. Better still, have them inherit from DAO or copy them to a DAO as part of the persistence code.