Database initialization and migrations

[Daniel Blanco]

Hi everybody!

I posted this yesterday in the mgo-users group, but since more people read here, I'll give it a try:

I'm wondering, what do people use to init database data? I'm speaking of master collections (countries, ACL (roles, features, main admin user...) and so)

Coming from PHP, I've used PHPMig and Laravel Artisan migrations (both for seeding and schema changes). How do you achieve this with MongoDB and Go?

The idea is to have a directory with a file per db operation (bulk field updates, master collection data insertion...) and run them when the app inits (or on demand via CLI command)

A collection called migrations will keep track of which ones have already been applied. 

I've been thinking about it, and my three ideas are: 

1. Have .js files in the directory, and run them from my Go app using mgo.Database.Run(bson.M{"eval", jsCode}) , where jsCode would be the file contents, something along db.users.insert({name: "admin", email: "ad...@app.com", ...}

Pros: very easy to implement.

Cons: eval() has been deprecated in MongoDB 3.0. It bypasses the app logic that could be present when CRUD operations are triggered. 

2.  Regular  JSON files in the directory. A JSON document modeling a operation ("insert", "update"...), containing an array with data to be unmarshaled to the model structs, and persisted to database using the app repositories (userRepo.Insert(userData))

Pros: uses app regular path to resource creation / deletion / update

Cons: could be that in the future, some changes are introduced in the code (model structs) that break JSON unmarshaling to the structs.

3. Extended JSON files (http://docs.mongodb.org/manual/reference/mongodb-extended-json/) Unmarshal them to raw bson and insert / update without using the app repositories

Pros: independence of the app code (no need for JSON / Go struct parity)

Cons: bypasses app logic (as point 1). Need to third party package to support extended JSON unmarsharling. 

4. Do it in the code. Create a function for each migration / seeding, with an array of structs containing the data to insert, or updates to perform, just like what's used in unit tests (http://dave.cheney.net/2013/06/09/writing-table-driven-tests-in-go)

Pros: same as solution 2

Cons: data in the code, doesn't look very good to me.

Any insights? 

Thanks in advance!

Dani

[Daniel Theophanes]

I have one line of business application written in Go where the (SQL) schema is specified in the application. When I update the schema I also add new entries to an alter function.
Db.Alter(12, schema.AlterAddColumn, "account.invoice")
Db.Alter(12, schema.AlterAddColumn, "account.invoice_date")

Then on application startup it runs the alter versions above the current in order.


-Daniel

[Ian Davis]

On Thu, Mar 12, 2015, at 11:42 AM, Daniel Blanco wrote:

I'm wondering, what do people use to init database data? I'm speaking of master collections (countries, ACL (roles, features, main admin user...) and so)

 

I haven't used this myself but I came across it today while looking for something else. I thought it looked interesting.

 

http://godoc.org/github.com/eaigner/jet

 

There seem to be a few other packages on godoc:

 

http://godoc.org/?q=sql+migrate

 

All the best,

 

Ian

 

 

 

[Shawn Milochik]

I've used this a little, and made a small contribution. It's a nice project:

https://github.com/rubenv/sql-migrate

It can be used either as a library or a stand-alone binary, so you can use it with Go and non-Go projects.

[Camilo Aguilar]

I found existing solutions doing too much or being unfriendly when used as libraries with embedded assets. So I wrote my own, so far only Postgres is supported: https://github.com/c4milo/migrator.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Blanco]

Thank you all for the replies. I take note on your suggestions. 

The thing is I'm using MongoDB, and haven't found anything for it. I know schema changes are not needed, but data seeding and batch updates (related to new features) are.

I guess I'll go with something similar to what Daniel commented (migrations as functions in the code)

If I find something, I'll add it here.

Regards,

[Gustavo Niemeyer]

Hey Daniel,

[copying from the response @ mgo-users]

The suggested approach of using a collection that describes the current version of the schema, and then having a selection of functions that know how to move from X to X+1, is a good one and may be applied to any database.

One thing to keep in mind when applying that concept to MongoDB is that it is not transactional, meaning a migration could be interrupted mid-way through and leave an inconsistent state behind. For that reason, the migration functions need to be aware of such mid-state, and cope well with it.

Another relevant point is that MongoDB can handle uneven schemas, and people often use that to do such migrations live. For example, instead of replacing an old field by a new field all at once, and having to stop serving clients while that happens, one might opt to teach the application to handle both the old and the new format while the migration is taking place (either in a one-off batch, or on an as-needed basis).

--

gustavo @ http://niemeyer.net

[Daniel Blanco]

Hi Gustavo!

First, thanks for the reply. 

I've mentioned both schema migrations and database seeding, and may be I shouldn't.

What I really want to do, at this time, is just seeding. I have some collections with the app default users, roles and features (access management), countries, currencies... you know.

How would you approach filling these collections with default data in an automated way?

Thanks again!

[Gustavo Niemeyer]

Until there is a reason to do more than that, I'd just go for the obvious one: a few Go slices in a .go file.

[Daniel Blanco]

That's what I'll do then. Thanks! 

--

-Dani-