Feature Proposal: per-settings fake migration override

[Flavio Curella]

I'm not sure if this warrants a DEP, so I've decided to post it to the list first. If this needs to be a DEP, just let me know and I'll follow the DEP process.

Rationale

-------------

Assume the following scenario:

1. At some point in history, project A was deployed in production and was very successful.

2. The project created a table called `transactions_log`, which is now considerably big and full of very valuable data, that we want to retain.

3. Because of $reasons$, project B gets started as a future replacement of project B

4. Project B wants to reuse `transactions_log` and the `transaction.models.Log` model unaltered

What we have is a 'split' between what the initial migration for the `transaction.models.Log` model should do, between dev and production:

* On local dev, it simply creates the table

* On production, the migration must be faked.

Traditionally, we would go to the person in charge of migration and tell them 'Hey, migration `transactions.0001_initial` must be faked'.

But this seems error-prone. That information ('fake this one specific migration') could be lost in the communication workflow, and imposes the cognitive burden of having to remember another special case.

Proposed solution

------------------------

Assume the following settings files:

```

# settings/local.py

# ..usual list of settings...

```

```

# settings/production.py

MIGRATION_FAKE = (

    ('transactions', '0001_initial),

)

```

* On `django-admin.py migrate --settings=settings.local`, the migration `transactions.0001` will be faked only if the `--fake` flag is passed (previous behavior).

* On `django-admin.py migrate --settings=settings.production`, the migration `transactions.0001` will *always* be faked. The new setting will act as an ad-hoc, per-migration override of the global `--fake` option.

I have a branch up as a Proof of Concept at https://github.com/fcurella/django/tree/feature/adhoc-fake (no test or doc changes yet). The code change necessary seems to be minimal, let me know if you think it's something worth having.

Thank you,

–Flavio.

[Markus Holtermann]

Hi Flavio,

This sounds like a very risky and scary proposal, to be honest. The recommended way would indeed be to fake the initial migration(s) on production.

Since it seems to me that this would only ever be needed once (you said devs should just run the migration locally and the table is too big for any changes) I'd probably go with the one time fake.

Alternatively, as of Django 1.9 you could move the transaction log model into a dedicated app and disable migrations for that app.

As a last resort, why not make the model unmanaged (Meta.manage=False) and apply the respective operations manually with RunSQL operations?

That are the ideas that just came to mind, there are probably others as well.

/Markus

[Shai Berger]

Hi,

On Saturday 19 September 2015 14:02:38 Markus Holtermann wrote:
> On September 19, 2015 6:38:59 AM GMT+10:00,
> Flavio Curella <flavio....@gmail.com> wrote:
> >
> ># settings/production.py
> >MIGRATION_FAKE = (
> >
> > ('transactions', '0001_initial),
> >
> >)
>

> This sounds like a very risky and scary proposal, to be honest. The
> recommended way would indeed be to fake the initial migration(s) on
> production.
>
> Since it seems to me that this would only ever be needed once (you said
> devs should just run the migration locally and the table is too big for
> any changes) I'd probably go with the one time fake.
>
> Alternatively, as of Django 1.9 you could move the transaction log model
> into a dedicated app and disable migrations for that app.
>
> As a last resort, why not make the model unmanaged (Meta.manage=False) and
> apply the respective operations manually with RunSQL operations?
>
> That are the ideas that just came to mind, there are probably others as
> well.
>

My instinctive reaction was that

A) This is a highly specialized case, which doesn't warrant framework support;

B) I'd solve it (at the project level) by putting some identifying setting in
settings/production.py, something along the lines of

# settings/production.py
FAKE_MIGRATIONS_FOR_PROD = True

and add to the initial migration a RunPython operation which looks for this
setting and raises an exception if it's there. Just to make the failure
explicit -- if the migration tried to create an existing table, it would fail
anyway.

Note that, if the model is in its own app, migrations already have a mechanism
in them to auto-fake the migrations of an app where all model tables exist.

(and note that this thread is already drifting into django-users' domain)

Shai.

[Flavio Curella]

On Saturday, September 19, 2015 at 6:03:06 AM UTC-5, Markus Holtermann wrote:

Hi Flavio,

This sounds like a very risky and scary proposal, to be honest. The recommended way would indeed be to fake the initial migration(s) on production.

Could you elaborate on this point? I have the feeling my proposal might have problematic consequences I have not thought of, and you did.

 

Since it seems to me that this would only ever be needed once (you said devs should just run the migration locally and the table is too big for any changes) I'd probably go with the one time fake.

Alternatively, as of Django 1.9 you could move the transaction log model into a dedicated app and disable migrations for that app.

As a last resort, why not make the model unmanaged (Meta.manage=False) and apply the respective operations manually with RunSQL operations?

That are the ideas that just came to mind, there are probably others as well.

I think my mistake here was calling this proposal a 'feature', when it's more of an enhancement.

There are currently many ways to handle this and similar scenario. The goal of my proposal was to enable a more convenient way.

[Flavio Curella]

On Saturday, September 19, 2015 at 8:39:54 AM UTC-5, Shai Berger wrote:

Hi,

On Saturday 19 September 2015 14:02:38 Markus Holtermann wrote:
> On September 19, 2015 6:38:59 AM GMT+10:00,
> Flavio Curella <flavio....@gmail.com> wrote:
> >
> ># settings/production.py
> >MIGRATION_FAKE = (
> >
> >    ('transactions', '0001_initial),
> >
> >)
>
> This sounds like a very risky and scary proposal, to be honest. The
> recommended way would indeed be to fake the initial migration(s) on
> production.
>
> Since it seems to me that this would only ever be needed once (you said
> devs should just run the migration locally and the table is too big for
> any changes) I'd probably go with the one time fake.
>
> Alternatively, as of Django 1.9 you could move the transaction log model
> into a dedicated app and disable migrations for that app.
>
> As a last resort, why not make the model unmanaged (Meta.manage=False) and
> apply the respective operations manually with RunSQL operations?
>
> That are the ideas that just came to mind, there are probably others as
> well.
>

My instinctive reaction was that

A) This is a highly specialized case, which doesn't warrant framework support;

Other scenarios came to my mind, but I didn't mentioned them to keep the proposal simple.

One other common scenario is for when you have to run slight different SQL in production versus development, for example having to `CREATE INDEX ... CONCURRENTLY`, which is a quite common necessity.

That said, if you think this proposal will encourage bad practices, I'm with you and I'd rather drop it than provide an easy way for users to shoot themselves in the foot.