DM migration project for GAE

[Max]

Hi all, 

I am planning to create a new open source data migration project to better manage app engine data. Basically you will need to checkout the project and deploy to your application (of course to a non-default version with admin access only). It will support some CRUD operations which are not supported by admin console at the moment, for example, edit text fields, add new fields to an entity which doesn't exist in meta data, upload / download file from a blob fields etc. Other than that, I will also try to support some of SQL for data migration. It will use task (pipeline) API to perform operations over large set of data and notify admins on the result via email (won't write anything to datastore). For example: UPDATE person SET firstName = 'Max' WHERE firstName IS NULL or DELETE FROM person WHERE NOT EXISTS (SELECT 1 FROM apartment WHERE apartment.owner = person.name )

It will also try to implement some new data migration syntax specific for app engine. For example, INDEX person (firstName) WHERE age > 25 will iterate over all person entities and for each of them partial index firstName is indexed when age is more than 25. 

Some health check functions may also be helpful. For example: CHECK UNIQUE person (firstName) will iterate over all person entities to make sure firstName is not duplicated (will need an index to perform this check, and only indexed entities will be checked). Other examples include CHECK TYPE person (image) blob and CHECK NOT NULL person (image)

Other than the application, there will also be a library available to be imported in your application directly. One use case is you can have a cron job to update something and then perform some health check on your data. 

Any other things you think will be valuable to implement?

Please please please do let me know if there is already a similar one. Don't want to reinvent the wheel. 

[Pascal Voitot Dev]

I'm interesting by your work because I'm thinking about data migration issues in a more generic way for Siena!

regards
Pascal

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Vmm_vyvAip4J.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Max]

Thanks Pascal, 

What's your plan to add DM features to Siena? Since it's designed not only for appengine, are you planning to have a GAE specific DM lib or a generic one working for all DBs?

[Pascal Voitot Dev]

I plan to have a generic one at the end but as the problem is complex, I will begin with the best implementation for Siena which is GAE! GAE to GAE begin :)

Moreover, in the same kind of idea as yours, I'd like to propose a way of writing "migration scripts" that would be managed by an application responsible of the migration...

Pascal

On Sat, Aug 13, 2011 at 7:30 PM, Max <theb...@gmail.com> wrote:

Thanks Pascal, 

What's your plan to add DM features to Siena? Since it's designed not only for appengine, are you planning to have a GAE specific DM lib or a generic one working for all DBs?

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/8-7g_4fXLxcJ.

[Ronoaldo José de Lana Pereira]

Hi all,

I'm interested on such an API too. Max, I had the same idea that you got: write some SQL-like administrative commands to allow some usefull complex queries, like counting, aggregating, updating schema changes, re-indexing, etc. Also, I'm considering to use the pipeline API, but I'm not sure about how to make this in java, since the Java Pipeline API is very odd.

Do you want any help on the project? Do you plan to write it in Java or Python (I can help with both :) )?

Looking forward to help with the project.

Best Regards,

-Ronoaldo

[Max]

hi Pascal, 

Don't quite understand on "migration script", do you have any example of this?

[Max]

Thanks Ronoaldo, 

I use python just on some simple scripts and never tried complicated projects with full set unit test / continuous integration setup, so most likely will start off with Java. 

My plan is to keep bumping this post for another several days until someone tells us there is already a similar one exists. Another purpose is to gather more info on what is really in need. 

Let's keep in touch!

[Pascal Voitot Dev]

My scripts are like your pseudo SQL commands but I would write those commands in a DSL designed for our needs. Not SQL because SQL is not functional and is quite limited in representing potential data transformations.

Pascal

On Mon, Aug 15, 2011 at 5:53 PM, Max <theb...@gmail.com> wrote:

hi Pascal, 

Don't quite understand on "migration script", do you have any example of this?

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bZtYsHrR9YYJ.

[Max]

sounds good, any plan to start? and any concrete example to look forward?

[Pascal Voitot Dev]

No examples, just ideas till now :)

We should define a few usecases and look at the existing projects even in SQL world!

Migration between DBs means 2 things IMO:
- An existing model evolves (rename field, change type of a field ...) -> this case is certainly the simplest in a first approach
- You have an existing DB and need to transform it into another DB by migrating data from the first to the second performing operations on it. A field in the destination DB could be the result of an operation on one or several fields from the originating DB -> this case is much more complex and complete

Do you see other cases?

Pascal

On Wed, Aug 17, 2011 at 6:42 PM, Max <theb...@gmail.com> wrote:

sounds good, any plan to start? and any concrete example to look forward?

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/MI_jgQyqGkEJ.