Mnesia tables initialized at start-up

[Przemysław Dąbek]

Hi,

I decided to make a pull request: https://github.com/evanmiller/ChicagoBoss/pull/28

Please, take a look and write what you think about that idea.

Regards,

Przemysław

[Evan Miller]

Is creating a schema non-destructive? I am not familiar with managing Mnesia

2011/9/7 Przemysław Dąbek <przemysl...@gmail.com>

--
Evan Miller
http://www.evanmiller.org/

[Przemysław Dąbek]

I checked mnesia sources in my erlang distribution.

mnesia_schema module contains ensure_no_schema function. If schema exists, error will be returned, so it is non-destructive behaviour.

[Graeme Defty]

In my initialisation routines i provided init/0 and reinit/0.

init catches a schema present error and returns an error message

reinit removes any existing schema

I think the code illustrated is cool (being driven from the models) but as it is mnesia-only (and at that, only really applicable to initial stages of development) I am not convinced it should be part of the CB framewrork.

Neat though!
 
g
_________________________________________-

2011/9/12 Przemysław Dąbek <przemysl...@gmail.com>

[Przemysław Dąbek]

I haven't tried to write something similiar in other db adapters but I think it should be possible.

graeme defty, functions init/0 and reinit/0 seems to be quite good solution to initliaze database.

What I really miss in CB are migrations and schema available in Ruby on Rails. I think that it would be nice to develop something similiar.

[Evan Miller]

2011/9/12 Przemysław Dąbek <przemysl...@gmail.com>

I haven't tried to write something similiar in other db adapters but I think it should be possible.

graeme defty, functions init/0 and reinit/0 seems to be quite good solution to initliaze database.

What I really miss in CB are migrations and schema available in Ruby on Rails. I think that it would be nice to develop something similiar.

To work with other DB adapters we will need type hints for the models. E.g.

-module(book, [Id, AuthorName, NumberOfPages]).

-type({author_name, varchar, 255}).

-type({number_of_pages, integer}).

Not a bad idea as we could use this information to have a more intelligent admin interface.

I am not opposed to incorporating a migration framework but I've never really found a system I liked. It's a hard problem, especially in production.

[Graeme Defty]

I have just developed something modelled on Rails' migrations.

basically, in the internal table where I keep the counters for the various application tables, I have put a record which keeps the database version. Each time I do an upgrade which requires schema changes, I add a function to my 'dbutils' module which performs the necessary. As many of these as necessary are called to get from where we are to where w want to be.

I did think about asking Evan to add get_dbversion/0 and set_dbversion/1 to the DB interface so that each db adapter could handle these in it's own way, but the code that updates the schema is so db-specific (hich is why it is in dbutils ;-) ) that in the end it did not seem very worthwhile.

g

(I could post the code from home if there is any interest)
__________________________________________

2011/9/13 Przemysław Dąbek <przemysl...@gmail.com>

[Kai Janson]

Hi Evan, hi Graeme,

I am not do sure that a migration system like

In ruby on rails is something that I really miss

In CB.  The migrations in RoR are usually quite a mess,

especially when more than one dev is working on db

related code and migrations.  IF there would be a system

that would be far superior to RoR's migrations, that would be

cool.  But just to have a migration system like RoR, no thanks.

Ever since CB 0.6.2 I am using mongo db with it and I am

really happy about not to have to deal with db migrations.

If there was a system for db migrations then I would like to

use it From within the admin interface and not from those

little pesky migration files like in RoR.

...just my $0.02...

--Kai

Sent from my iPhone

[Evan Miller]

If we were to do migrations it would go something like this. DB tables would be versioned (e.g. voters_v123) and the ID would contain the version number (as it already contains the model name... e.g. "voter-1-v123"). Models would have migration functions, e.g.

migrate(v123, v124, OldAttributes) ->

   NewAttributes;

...

During querying CB would ask for *next* version than the highest it knew about. If there was a *new* record it would read the *old* record and treat it as read-only. If there was not a new schema it would read and write the old (current) record as usual. If it received a request for an *old* record (version older than what it knew about) it would retrieve the old record and run the migration function before application processing. Then upon save it would save to the *new* schema. This way you could have gradual migrations across a cluster and not need to run ALTER TABLE on the database... you'd just transition to the new table via application saves. When all nodes were migrated you could start deleting migrated records from the old schema or force a migration of the remaining records and then drop the table (via admin).

As an application developer you would just need to write an attribute migration function for each model version and also handle the extra case of read-only records... perhaps I'd slip it in as a validation error so that you wouldn't need to alter any controller code.

2011/9/12 Kai Janson <kot...@gmail.com>

[Graeme Defty]

This sounds much nicer.

See how easy it is when you can change the framework  ;-)

Also, I assume your expression "When all nodes were migrated ..." means when all nodes are updated with the latest software release, yes?

g
____________________________________________________________

2011/9/13 Evan Miller <emmi...@gmail.com>

[Evan Miller]

Yes I meant updated to the latest app :)

Sent from my iPhone

[Przemysław Dąbek]

Kai Janson, I agree that migration files in RoR is a mess but in long-term development it's a nice way to make some changes with database.

I don't want an exact copy of RoR's migrations in CB. I'd like to see something new too.

graeme defty, I like a possibility to change framework - write some functionality that I need. I can change framework or write extra libraries.

For me RoR seems to be too complex to play easily with code in it. However it's a mature framework and I can write my own libraries.

What I like the most in CB is speed. I start my app and it's ready to use. In RoR 3.x it takes a while to start.

[Manuel A. Rubio "Bombadil"]

Hi,

El Tuesday 13 September 2011 09:30:03 escribió:
> > If we were to do migrations it would go something like this. DB tables
> > would be versioned (e.g. voters_v123) and the ID would contain the
> > version number (as it already contains the model name... e.g.
> > "voter-1-v123"). Models would have migration functions, e.g.
> >
> > migrate(v123, v124, OldAttributes) ->
> >
> > NewAttributes;

sounds good! i have used RoR migrations a lot, in little and big projects, and
really, is a mess. We don't get a real order, and a fast view of tables, and
database structure (or schema).

But, the proposal, it could be:

- schema/
- my_connection/ (by example PostgreSQL)
- my_table/
- 1.erl (.yml, ...)
- 2.erl
- 3.erl
- ... and so on
- my_other_table/
- 1.erl
- my_other_connection/ (by example Riak, is possible?)
- ...

Have a module to "update", "redo" a specific version, "downgrade" to a
specific version and "status" (to see the general table and database status).

About table, to migrations, should be a table by connection, and contents a
field with "table -> version" pair of values.

I could help you to do it (if is a good idea for all).

Regards!

--
Manuel A. Rubio "Bombadil"
Usuario de GNU/Linux #323628 acorde a http://counter.li.org/
Técnico en Admin. Sistemas Informáticos

[Przemysław Dąbek]

Interesting proposal for me. 

Personally I prefer timestamp filenames (something like 20110913153300.erl) because of sorting issues by filename in system.

[Colin 'Logan' Campbell-McPherson]

Timestamps are great for sorting and avoiding conflicts in version
control. However they make tab completion a bitch. If there was some
kind of relatively uniquely identifying prefix to the migrations it'd
be handy.

2011/9/13 Przemysław Dąbek <przemysl...@gmail.com>:

[Manuel A. Rubio "Bombadil"]

I though that this question could be optional. You could like use serial
numbers: 0001, 0002, 0003, ...; or numbers: 1, 2, 3, 4, ...; or timestamps:
20110913153300, 201109131611, ... It could be optional... only needs see the
table (migrations table) and check what migrations is not done, to do it.

It could be sorter in integer way and, in this form, the three options are
valid, isn't?

[Evan Miller]

Not sure I understand your proposal... will these files perform general DB tasks (create table, drop column, etc)? Or migrate individual records? Or both?

 

About table, to migrations, should be a table by connection, and contents a
field with "table -> version" pair of values.

I could help you to do it (if is a good idea for all).

Regards!

--
Manuel A. Rubio "Bombadil"
Usuario de GNU/Linux #323628 acorde a http://counter.li.org/
Técnico en Admin. Sistemas Informáticos

[Manuel A. Rubio "Bombadil"]

El Tuesday 13 September 2011 17:11:19 escribió:
> > But, the proposal, it could be:
> >
> > - schema/
> >
> > - my_connection/ (by example PostgreSQL)
> >
> > - my_table/
> >
> > - 1.erl (.yml, ...)
> > - 2.erl
> > - 3.erl
> > - ... and so on
> >
> > - my_other_table/
> >
> > - 1.erl
> >
> > - my_other_connection/ (by example Riak, is possible?)
> >
> > - ...
>

> Not sure I understand your proposal... will these files perform general DB
> tasks (create table, drop column, etc)? Or migrate individual records? Or
> both?

These files will perform general DB tasks. Each file, will contents a simil to
a SQL instruction (or equivalent for another kind of databases as NoSQL if
necessary).

[Evan Miller]

Hmm. These sorts of things should probably be integrated with OTP appup files somehow:

http://www.erlang.org/doc/man/appup.html

Need to give this some more thought.