Large sets of bootstrapped data

[Colin Harrington]

So how do people accomplish bootstrapping large sets of data for a Grails app?

I've got a large set of data that can be pulled remotely from external servers over the course of 10 minutes that I want to have cached so we can cut that time down considerably.

We don't have Schema versioning in place yet, but we want to accomplish something that works with our setup when we start using Liquibase/Autobase, etc.  Is Schema Versioning neccesary now to make this possible?

Any recommendations?

Thanks in advance

Colin Harrington
colin.ha...@gmail.com

[Ted Naleid]

We just use a mysql dump file.  The dump file has the tables that liquibase needs to determine the schema version of the data, so if the dump I'm using is a little old, I can just run the migrations that are missing.  Every once in a while I take a new snapshot and use that as my default.   I sent a script out to the GUM list a few months back to Mike that we use to randomize the data that might be useful if you need to anonymize things.

That's by far the quickest way to get a ton of bootstrapped data in place.

Alternatively, if you want to actually run a process to import stuff, I posted some stuff a few weeks ago that is worth looking at for speeding imports of large sets of data up considerably from what a default grails implementation will give you (while still staying within the grails domain objects).

http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/

-Ted

--
You received this message because you are subscribed to the "Groovy Users of Minnesota" group.
 
To post to this group, send email to groo...@googlegroups.com
To unsubscribe from this group, send email to groovymn-u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/groovymn?hl=en

[Jean Barmash]

Colin,

I don't have schema versioning issues yet, so what I did is use Ted's
RunScript.groovy script (my copy attached) to run a script that reads
several Excel files and creates domain objects as needed (some using
Ted's Build-Test-Data plugin, so I am a bit immune to *some* schema
changes). My version of the script is one I use with 1.2M4, though I
think it will work with previous versions (i.e. 1.1) as well.

This allows me to start the app, and then load the data from a
separate script separately. I have a bunch of checks and
transactional handling in the code that allows me to rerun the script
several times if there are issues.

Thanks,

Jean