Migrations in Django 1.7 make unit testing models harder

[Bernie Sumption]

Hi Django devs,

I've just started a new project in 1.7b, and the development experience working with unit tests on models is more complicated than it was in 1.6. It's all down to how the throwaway test databases are created. In 1.6, the create_test_db function "Creates a new test database and runs syncdb against it." In 1.7, it runs "migrate".

While generally speaking, migrate is the new syncdb, this behaviour is not ideal for tests. In 1.6 "syncdb" created a database reflecting the current state of the models in models.py. "migrate" creates a database reflecting the state of the models at the last time makemigrations was run. If you're doing TDD and constantly making small changes to your models then runnning unit tests, you have to run makemigrations before each test run to get your tests to work. You therefore end up with many tiny migration files representing the minute-by-minute history of development.

I came up with a pretty effective workaround that is working for me, but I thought I'd post this here as others are sure to encounter this issue, and I think that it makes more sense for the behaviour produced by this workaround to be the default for running tests.

If makemigrations has not yet been run, the "migrate" command treats an app as unmigrated, and creates tables directly from the models just like syncdb did in 1.6. I defined a new settings module just for unit tests called "settings_test.py", which imports * from the main settings module and adds this line:

MIGRATION_MODULES = {"myapp": "myapp.migrations_not_used_in_tests"}

Then I run tests like this:

DJANGO_SETTINGS_MODULE="myapp.settings_test" python manage.py test

This fools migrate into thinking that the app is unmigrated, and so every time a test database is created it reflects the current structure of models.py.

So my feature request is as follows:

If the new behaviour is by design and considered desirable, then it is a big change from the previous version and should be prominently documented in the migrations and testing pages, along with the workaround. I'm happy to write this documentation if that's the way you want to go.

However, if the new behaviour is not by design but just a by-product of the new migrations feature, I suggest making the workaround the default behaviour. I don't (yet!) know enough about Django internals to volunteer for this however.

Thanks for your time,

Bernie     :o)

[Andrew Godwin]

Yes, the new behaviour is by design, in the sense that the workaround you mentioned will be deprecated in 1.9 (along with all syncdb-related functionality). This way, tests always run against the version of your models that production would run, so you don't run the risk of the tests passing locally as they're not using migrations, pushing it live, and then things failing in production as the tables don't match.

I'm happy to take a small patch to the docs to mention that you need to run makemigrations for tests to see your database changes, but the workaround shouldn't be in there - as I said, it's already part of the deprecation cycle.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/00e9a053-3e61-4c5d-8fcc-5a4d67deab38%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marc Tamlyn]

Do we have an equivalent of south's --update? This would mean you don't get many files. We don't want to make it too hard for people to work in a strict TDD fashion.

M

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1ur1EAxBoDC4O_-LD6B_CsGv3iE7jxCZRts-7ORfKKc%2Buw%40mail.gmail.com.

[Florian Apolloner]

On Tuesday, March 25, 2014 7:14:55 PM UTC+1, Andrew Godwin wrote:

Yes, the new behaviour is by design, in the sense that the workaround you mentioned will be deprecated in 1.9 (along with all syncdb-related functionality).

What exactly will get deprecated here?

[Mark Lavin]

Andrew,

Can you clarify exactly what is deprecated in this work-around? I don't see anything in the deprecation timeline to remove MIGRATION_MODULES nor any pending deprecations related to its usage. It seems like could probably be replaced by something that uses the app-loading/app-configs instead but it does look like it's been done.

Best,

Mark

[Shai Berger]

On Tuesday 25 March 2014 20:36:39 Marc Tamlyn wrote:
> Do we have an equivalent of south's --update? This would mean you don't get
> many files. We don't want to make it too hard for people to work in a
> strict TDD fashion.
>

+1

Shai.

[Andrew Godwin]

So, the functionality whereby you can have apps which do not use migrations (i.e. that use the old creation backends) is meant to go away in 1.9 (i.e. the standard three-release deprecation cycle). Most of the side-effects of this are detailed in https://docs.djangoproject.com/en/dev/internals/deprecation/#deprecation-removed-in-1-9 but not this one - the fact that setting an empty MIGRATION_MODULE means it falls back to the old syncdb method (in 1.9, this would just make Django think that the app had no migrations at all and do nothing, and probably raise a warning or error).

The setting itself isn't deprecated, just this undocumented behaviour, so I haven't put a warning next to it in the docs.

As for the general TDD problem, we do have the squashing functionality now so the --update problem is not quite as bad, but I'm not averse to something like it appearing in 1.8 (1.7 is too frozen now to add it IMO).

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/201403252101.52683.shai%40platonix.com.

[Michael Manfre]

I just read the deprecation timeline and the very brief mention of syncdb command and signals going away doesn't really seem to sufficiently detail the "side-effects" you mention. Anyone who hasn't read your email is going to be unpleasantly surprised. I also don't see any deprecation warnings in the code about this either.

> As for the general TDD problem, we do have the squashing functionality now so the --update problem is not quite as bad, but I'm not averse to something like it appearing in 1.8 (1.7 is too frozen now to add it IMO).

If I read your statement correctly, you are asking for some one to create a release blocker ticket because taking the position that TDD will be painful in 1.7 and might improve for 1.8 seems wrong.

Regards,

Michael Manfre

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upm2g84E6JymtQUgrmBh0vSdfaLU5gNCC-EEopTGx%3D7AQ%40mail.gmail.com.

[Mark Lavin]

I don't see any meaningful notes about apps being required to ship migrations beginning with 1.9. I do see deprecation notes about the syncdb signals changing and the syncdb command itself is clearly deprecated. This legacy behavior is handled by sync_apps in the migrate command but there aren't any deprecation warnings raised when it executes. I don't see anything which points to this behavior being deprecated other than this thread.

[Andrew Godwin]

I'll update the deprecation document to include more direct information about DatabaseCreation and the legacy app sync method.

I'm not sure "TDD is a bit harder" is a release blocker - the TDD I do generally doesn't have that much model creation, and it's relatively easy to just run makemigrations to get new migrations for models, especially now it just does the whole project at once. Unless I can see this framed as being a significant impact to development process, I'm not going to hold up the release because of it...

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/262a32fd-fd5c-495b-a0ba-878372d3f7ea%40googlegroups.com.

[Florian Apolloner]

On Tuesday, March 25, 2014 9:12:42 PM UTC+1, Andrew Godwin wrote:

So, the functionality whereby you can have apps which do not use migrations (i.e. that use the old creation backends) is meant to go away in 1.9.

Uhm, strong -1 here unless you have really convincing arguments, I really wouldn't like to require migrations for all the models we have in our own testsuite. Especially since they are somewhat transient and don't have to care about migrations.

Florian

[Andrew Godwin]

We can possibly work around it for the test suite, but the intention was that all apps people build on Django would require migrations by that point, as then we can get rid of a whole load of legacy code. There's nothing to stop us just making something that just migrates models for tests without files on disk, but it'd still have to be making Migration objects in memory behind the scenes due to any potential dependency/foreignkey issues (the syncdb code only gets around this by doing everything in one huge transaction, which isn't the model that SchemaEditor supports).

Separate from all that, though, we can't promote adding random strings to MIGRATION_MODULES as the suggested way to "get around" migrations for tests. In my opinion, the whole point of migrations is that you know you have the same schema everywhere, and it's especially important you use them during tests.

Andrew

[Bernie Sumption]

we can't promote adding random strings to MIGRATION_MODULES as the suggested way to "get around" migrations for tests.

I agree, my workaround is a hack. It would be better to introduce a flag or setting designed specifically for this use case.

 

In my opinion, the whole point of migrations is that you know you have the same schema everywhere, and it's especially important you use them during tests.

If you view tests as a verification tool that is used before deployment or committing to check that the system is working as desired then yes, this is true. If you're practicing TDD, then tests are something else too - they're a development environment. They're the primary way of interacting with your code. Add a field, run tests. Rename it, run tests. Change its options, run tests.

The fact is, Django 1.6 and South supported this use case very well, Django 1.7 does not.

[Andrew Godwin]

When I practice TDD I write the test to spec, and then write the model and view code, so I usually have about the same amount of model changes as otherwise (as, having written the tests, I usually have a clearer idea of what fields I need). I agree that if you're incrementally writing tests on top of models, however, it could be extra verbose, but bear in mind that having hundreds of migrations on 1.7 is easily solved by a single `squashmigrations` command - much easier than on South (and performance on large migration sets should also be improved).

I'd be willing to keep the current contract of "things without a migrations directory don't get migrated", but I suspect you're doing things on apps that already have migrations (which makes my reticence to add a setting even bigger - if you "syncdb" an app with migrations to a main database, you have forever ruined the migration tracking on it). Would that work? Or do you want to do this on apps which are already several migrations along? (If that's the case, I suspect we might have to look at re-introducing --update, which is going to be near-impossible to do before release with the new creation system).

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ac7d9f74-691f-4ec8-8900-434f8e0a93ff%40googlegroups.com.

[Bernie Sumption]

I'd be willing to keep the current contract of "things without a migrations directory don't get migrated", but I suspect you're doing things on apps that already have migrations (which makes my reticence to add a setting even bigger - if you "syncdb" an app with migrations to a main database, you have forever ruined the migration tracking on it). Would that work? Or do you want to do this on apps which are already several migrations along? (If that's the case, I suspect we might have to look at re-introducing --update, which is going to be near-impossible to do before release with the new creation system).

You're right, I'm creating an app with migrations but want to be able to run tests without migrations when I'm mid-development. I also agree that if I'm running integration tests before deployment, or a non-developer is running tests before installing something written by someone else then those tests should use migrations, so it's not as simple as "don't use migrations in tests".

How about specifying whether a database is migrated as part of the database definition in the settings module? That way a database would either be migrated or not, and if a developer wants to set up an alternative non-migrated database for development there's a supported way of doing it.

Something like:

# in settings_bernie_dev.py

from .settings import *

DATABASES = {

    'default': {

        'ENGINE': 'django.db.backends.sqlite3',

        'NAME': os.path.join(BASE_DIR, 'testdb.sqlite3'),

        'USE_MIGRATIONS': False,

    }

}

Bernie     :o)

[Shai Berger]

I think a flag on the test command would be much more appropriate for the use-
cases you describe -- we don't want to send the message that disabling
migrations in tests is ok for a setup, although it may be ok for a given test-
run (basically, like running test for just a single app is ok).

Shai.

[Andrew Godwin]

If I need to take either of these options I'd tend towards Shai's one - we don't want to allow people to just disable migrations on a per-database basis, that's bound to get someone into trouble. That said, the flag is going to be weird to explain to people.

Just to establish a baseline, would you say that adding an "--update" command to makemigrations - which rolls the changes into existing migrations if it can - would solve your problem here? If that's the case, we can work towards that and figure out something simpler for the meantime.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1520941.Sdtj0HqCMr%40deblack.

[Bernie Sumption]

The problem with --update is that if overwrites the most recent migration, then it might be used to modify a committed and distributed migration, which is a Bad Thing. The flag would probably be useful to people with my use case, if they trust themselves to use the flag with care and remember to not use it immediately after committing. Was this the reason that you didn't carry --update from South to Django?

Entertainingly, I was about to defend my original proposal, but have just realised that for the last few days while we've been having this conversation, I have not been remembering to run makemigrations before committing, thereby falling into exactly the trap that you predicted my behaviour would produce!

Given that --update and my original proposal both have significant dangers if not used properly, my new thought is to write a "safe update migrations" script that used Git to delete all uncommitted migrations and runs makemigrations again. This could give me the best of both worlds - tests that "just work", no build up of many tiny migrations files, and no risk of trashing the committed migration history.

I'm coming to think that there's no change that could be made to Django core that would magically fix this without side effects, so perhaps it should be left to the community to create and share scripts that work for each VCS?

Bernie     :o)

[Bernie Sumption]

OK, it turns out that the "safe update migrations script" too simple to even qualify as a "script":

git clean myapp/migrations -f && python manage.py makemigrations

Perhaps the solution is to document this on the testing page as a solution to the "accumulation of many small migrations during development" problem?

Bernie     :o)

[Marc Tamlyn]

That script would be bad if you'd run any of those migrations against your development db (yes it should be "throwaway" or rebuildable but...)

Personally, I'm strongly in favour of Shai's suggestion and also in favour of --update, mainly as I like being able to capture (most) migrations has logical bits of work as it aids code review. My pull requests don't include every iteration of the code I wrote to get the final version, so why should my migrations?

M

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e15603d7-ed14-4d18-b36b-925aaf824bce%40googlegroups.com.

[Bernie Sumption]

That script would be bad if you'd run any of those migrations against your development db (yes it should be "throwaway" or rebuildable but...)

I'd think the same could be said of --update? As I understand it, --update is the equivalent of deleting the most recent migration and recreating it. If you'd applied the most recent migration to your development database and then you --update that migration, you'd need to rebuild the database.

[Marc Tamlyn]

South's `--update` also rolled the previous migration back, changed it and then reapplied it to the current database.

M

On 28 March 2014 10:48, Bernie Sumption <ber...@berniecode.com> wrote:

That script would be bad if you'd run any of those migrations against your development db (yes it should be "throwaway" or rebuildable but...)

I'd think the same could be said of --update? As I understand it, --update is the equivalent of deleting the most recent migration and recreating it. If you'd applied the most recent migration to your development database and then you --update that migration, you'd need to rebuild the database.

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f3e1b67b-f1fc-4c94-964f-23ec7fb27755%40googlegroups.com.

[Bernie Sumption]

South's `--update` also rolled the previous migration back, changed it and then reapplied it to the current database.

OK, in that case I can very much see how it's useful for people who develop against a persistent database. That's probably most people.

Anyway, the result of this thread for me is that I now consider my original request to be obsolete, as the "git clean" thing is a simple way of getting the behaviour I want for my own style of TDD without hacks.

Thanks for your time.

Bernie     :o)

[Andrew Godwin]

Yes, --update is very risky if you run it on migrations that are already committed and pushed, but the main reason I left it out of 1.7 was complexity (because makemigrations is now much more intelligent, updating and adding a foreignkey into a migration might introduce a new dependency or force a new migration anyway). Given that we have the ability to safely squash large numbers of small migrations down into one with squashmigrations and distribute that to fix the many-small-migrations problem, I considered it pretty low priority, though I have a rough idea of how I could make it work (I'd have to load up the autodetector with the existing migrations already loaded in as a halfway state and then run it from there, which should produce the right result).

Anyway, if you're retracting your original request, I'm happy to leave this for the 1.7 release; I don't think there's a good solution that Django core can implement effectively. This reminds me of when people used to ask me to automatically stop their developers writing conflicting migrations - the solution varies from company to company and often isn't technical but just education or communication.

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8e3ecf3c-aa05-4e3d-b905-3260b093e046%40googlegroups.com.

[Harry Percival]

Am just working on updating my book on TDD to django 1.7 based on the beta.   Currently half-way thru, not run into any problems because I don't use migrations until a later chapter, but when I do I will run into the same problems Bernie mentions.

Will share more once I've finished the rewrites, but from what I see so far, I think I'd personally prefer to be able to run my tests without having to remember to call makemigrations every time.  some kind of customisable option?  either a command-line flag for the test runner, or maybe a setting in settings.py, eg MIGRATIONS_OFF_FOR_TESTS = True?

personally i'd like the default to be true, but i can appreciate other people will have different workflows / assumptions.

[Andrew Godwin]

No, there is no way to turn off migrations for tests - some of the core tests won't work without them turned on, in fact, and adding that option would be weird (why only tests? what would it do? how do you load data in now initial_data is gone?). The only complaint I've seen - the one that Bernie brought up originally, that it's "extra work" to run makemigrations before each test run - doesn't really hold water with me, as the alternative options mean you could run the tests and have them pass WITHOUT HAVING THE RIGHT MIGRATIONS - and so you're not testing part of your codebase.

Hell, you can alias together makemigrations and test if you want, that'll save you the typing. This might make a few more migrations than normal, but you could quickly point out that squashmigrations exists to deal with this problem and move on.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a2897318-ec81-4acd-b201-2591042db1a0%40googlegroups.com.

[Harry Percival]

I suspect you're probably right.  Having to run makemigrations in between making changes to model code and running tests isn't the end of the world i suppose.  Will know better what I'm talking about  when I've actually got to that part of the book...

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/PWPj3etj3-U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uoqPk47Zbd7fDA9pP3VyJ5xw%2B4UciBffD-jgKq88i4MJQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
------------------------------
Harry J.W. Percival
------------------------------
Twitter: @hjwp
Mobile:  +44 (0) 78877 02511
Skype:         harry.percival

[Harry Percival]

OK, I've now run into the problem IRL, and sure enough, it's different to what i'm used to.  Am trying to overcome my knee-jerk reactions of "why did it change! i hate it!".  ignoring that for a moment then:

One thing I did find surprising was that I'm going thru TDD in small steps like this:

    class Item(models.Model):
        pass

- create migration 001-initial, with Item and auto-id

    class Item(models.Model):
        text = models.TextField()

- create migration 0002, adding the item field.  here's where we get into the discussion of too many migrations, wanting to squash, etc etc etc.

leaving that aside for a moment, i was taken aback by this:

You are trying to add a non-nullable field 'text' to item without a default;
we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows)
 2) Quit, and let me add a default in models.py

I thought TextField did have a default, the empty string?

[Ryan Hiebert]

I thought TextField did have a default, the empty string?

Like every other field, the "default default" is None (NULL). 

[Harry Percival]

Ah, so the reason I was confused is because it *looks* like the default is the empty string, because that's what you get if you initialise an object, by default. But at the database level, the default for the column is NULL.  Is that right?

So, I realise we're getting sidetracked here, but,  how does this fit with the fact that `null=False` is the default for all Field types?

On 29 March 2014 23:37, Ryan Hiebert <ry...@ryanhiebert.com> wrote:

I thought TextField did have a default, the empty string?

Like every other field, the "default default" is None (NULL). 

--

You received this message because you are subscribed to a topic in the Google Groups "Django developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/PWPj3etj3-U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABpHFHQK9UdEK6x4Mb3eDqXd5f%3D9egzJp5A21UZYXAQJw8HRDw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Shai Berger]

On Sunday 30 March 2014 15:08:17 Harry Percival wrote:
> Ah, so the reason I was confused is because it *looks* like the default is
> the empty string, because that's what you get if you initialise an object,
> by default. But at the database level, the default for the column is NULL.
> Is that right?
>
> So, I realise we're getting sidetracked here, but, how does this fit with
> the fact that `null=False` is the default for all Field types?
>

Simply: The "default default" is that fields are required.

But this is very deep in django-users territory.

Shai.

[Andrew Godwin]

You're roughly right, yes. String fields are a little odd, though, in that you can have them blank=True without null=True and then the default should be an empty string (Django's separation of blank and null irks me still) - the migrations should correctly detect this and insert blank strings for you then. If not, open a bug report!

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/201403301513.23581.shai%40platonix.com.

[Harry Percival]

Just found out that you can make Django behave in the "old way" by just deleting the migrations folder when you first do your `startapp`, and then you can pretend migrations don't exist until you need them.  Don't think that's necessarily a good idea though...

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/PWPj3etj3-U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1urwhyZkfF9smfeeSOeS_8ejFdKmKo3X%3D4yfWd8DGU_%3DTA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Andrew Godwin]

Yes, you can - the "migrations or not" switch is currently "is there a migrations directory", but bear in mind this will eventually turn from "use the old way" to "ignore it completely", probably also in 1.9.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CACFvh99EEyphjkE7%2BY7GawvkvhRNRrAVPOebj17T-OT8Z3HpBA%40mail.gmail.com.