Migrations status update & feedback request

[Jeffrey Becker]

I'm almost done my infrastructure work & unit tests on the DDL generation portion of NHibernate needed to support my proposed migrations feature.  As of now I've got an interface:

public interface IDdlOperation

{

    IEnumerable<string> GetStatements(NHibernate.Dialect.Dialect dialect);

}

I've re-implemented the ddl generation in the Mapping namespace as a series of implementations of this interface.  Each of these classes accepts a model which contains the relevant information for building the sql statements.  As much as possible database object names are passed to these classes pre-qualified and pre-quoted.  The relevant mapping classes have been changed from implementing IRelationalModel to having methods that produce one or more relevant IDdlOperation instances.  The Configuration class was updated to use these methods.

While I finish out the unit tests and get it all staged into a coherent set of commits for a pull request I'd like some feedback on the following topics:

Currently the Configuration/Mapping classes quote and qualify objects using the dialect methods which results in dialect specific quoting.  I this does not present a problem in the existing system because analysis always happens at the same time as execution and the target dialect is known.  In a world with coded migrations a migration could be generated against a one dialect and executed against another with incompatible quoting.  Should these methods be changed to quote and qualify using back-tick quoting and the operations be changed to mangle those quotes into dialect specific quoting?

Where is the dividing line between NHibernate and non-core Migrations features if any? I see this framework eventually containing components such as versioning strategies, a fluent migration building DSL, a migration execution framework, and a component which generates/executes coded migrations within the package manager console.  At a minimum that 3rd piece needs to be a separate assembly because of its dependency on visual studio.  How to people feel about the rest?  Should I aim to include some / all of that within Nhibernate or split it out into a separate assembly?

How do people feel about a fluent DSL for building the coded migrations?  How do we make this into something which is both expressive enough for people to write a migration themselves and flexible enough that they don't have to resort to issuing raw sql for everything?

[Ricardo Peres]

Hi, Jeffrey!

Need help with this? Do you have a public repository?

RP

[Jeffrey Becker]

I've just been slammed recently. 

The repo is up on my github at https://github.com/jeffreyabecker/nhibernate-core. The commits are a bit of a stream-of-consciousness mess at the moment. 

The core ddl generation rewrite into the DdlOperation framework is in there.  Currently I've been playing around with encapsulating that stuff into a Migration class but I'm not sure I'm happy about how that's going.  

[Ricardo Peres]

And do you have something that describes your "vision", what functionality you'd like to have, so that we can discuss them?

RP

[Jeffrey Becker]

The primary goal is to make a framework which supports coded migrations between versions of a Mapping, independent of the underlying database & schema.  The framework should have the following major features: 

• Unified DDL generation across NHibernate / DDL from schema export should match the DDL from Migrations for trivial cases
  
• Be flexible enough that consumers can encapsulate dialect specific operations in a testable fashion
  
• A Domain specific language / fluent API for coded migrations 
  
• A Code generator which creates Migration classes based on such an api
  
• A migration execution framework which compares Mappings in the DB to Current and selects a set of migrations to execute at run-time.
  

Some background:  In the past I've worked on a product where we used NHibernate while supporting several different databases in the product.  The data migrations were one of the hairiest parts of the application. SchemaUpdate was virtually useless because some changes required dropping and rebuilding tables which it messed up badly.  Versioning was inconsistent at best; Mostly the current version was kept in a table except for version 3.12 where it was in an extended property on SqlServer, in a table comment on Oracle, and in the table for the rest. The migrations themselves quickly became a rats-nest of bad, conditional DDL building.  

[Jeffrey Becker]

I've gone back, re-forked from nhibernate/nhibernate-core repo and cleaned up my commits significantly.  There are currently 3 branches in my repo at https://github.com/jeffreyabecker/nhibernate-core/.  

The main branch is migrations.  This is the branch that will eventually end up in the pull request.

The ddl-generation branch is the work on abstracting ddl-generation out into a unified framework.  Instead of the various mapping classes implementing IRelationalModel, DDL generation is abstracted out into a set of models and operations.  A model class is a DTO which summarizes all the information an operation might need to know in order to generate the DDL.  This framework is in-turn used by the existing schema creation utils and the fluent DSL to produce uniform output. Each operation is responsible for generating the appropriate set of DDL statements for the given dialect.  This will make it possible for migration writers to implement operations not supplied by NHibernate in a cross-platform manner.

The migration-dsl branch is concerned with the fluent dsl for building migrations

[Ricardo Peres]

I'm listening! Will have a look! ;-)

RP

[Jeffrey Becker]

In terms of the nitty-gritty of how to go about generating the migrations I really like EF's take on this.  The version history table for EF contains a snapshot of the mapping.  When the user requests a migration generation the tooling pulls that snapshot and uses the differences to generate a migration class.  In NHibernate this could easily be achieved by serializing out the Configuration instance.  I like this approach because it allows the migrations framework to more safely generate drop statements.  If the table isn't in the mapping any more its probably been dropped.  If the migration writer doesn't want that dropped then they should delete the drop statement.

One place for improvement over EF is definitely in the version history table.  The version strategy is pretty much hard-coded.  Everything is stored in a table with a hard coded name and specific schema.  No allowance is made for having a different table name, or having that table live in a specific schema.  The table itself contains enough information that multiple separate apps could easily be deployed to a single database but they'll all share the history table. More over it doesn't allow for integrating existing versioning methods. Ideally I'd like an interface around this so people with existing versioning strategies could try to integrate them.  

[Stuart Duncan]

Hi, I know this an old thread now, but has there been any progress on these features?

Thanks,

Stuart

[Jeffrey Becker]

Basically, a number of people felt this was 'reinventing the wheel'.  I've got a similar project using fluent migrator up on my github. A lot of work remains in that repo.  Basically the migration generator works by looking at two Configuration instances and generating FluentMigrator objects off that.   My big-picture todo currently consists of:

Figure out configuration snapshot storage.  Basically we need to serialize a snapshot of the configuration for differential migrations. EF puts this information in the version table in the DB.  FluentMigrator doesnt currently have the extensiblity to do that. However I'm not even sure this is a good idea.

Write a full differential Migration generator.  Right now the generator only produces full-create migrations.

Work out tooling. None of this is particularly useful without some tooling.  Making a unit test to generate your migration feels not-right.  This brings up the question of how to get an instance of the Configuration.  Personally, I just inherit configuration and encapsulate my setup in the constructor of that class.  But that is certainly not how the docs describe the process so I imagine it unusual.  

Further discussion and pull requests are welcome!

[Jeffrey Becker]

I'm beginning to have concerns about the viability of FluentMigrator as a project. The project seems pretty much abandoned. The main github repo is under an account that no longer takes pull requests. It took 4 months for someone to get around to approving my post on the google group. Since then, its had 29 views (mostly me trying to remember what I wrote) and 0 replies.  

Given that people seemed to like the overall effect of the work on DDL generation code is this something worth re-investigating as part of NHibernate proper?

[Gunnar Liljas]

There seems to be some activity, although it's moving at a snail's pace.

/G

--

---
You received this message because you are subscribed to the Google Groups "nhibernate-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nhibernate-develo...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.