Remove automatic date-naming of migrations (00XX_auto_YYYMMDD)

[Adam Johnson]

Hi djangonauts,

In a blog post earlier this year I outlined my technique to prevent Django creating migration files with automatic date names. I had a lot of response with other techniques and ended up adding two more techniques to the post. It's at https://adamj.eu/tech/2020/02/24/how-to-disallow-auto-named-django-migrations/ .

The problem with such migration names:

When you run Django’s manage.py makemigrations, it will try to generate a name for the migration based upon its contents. For example, if you are adding a single field, it will call the migration 0002_mymodel_myfield.py. However when your migration contains more than one step, it instead uses a simple ‘auto’ name with the current date + time, e.g. 0002_auto_20200113_1837.py. You can provide the -n/--name argument to makemigrations, but developers often forget this.

Naming things is a known hard problem in programming. Having migrations with these automatic names makes managing them harder: You can’t tell which is which without opening them, and you could still mix them up if they have similar names due to being generated on the same day.

Django *currently* sets the migration name to something other than the automatic date name in two cases:

• If the migration contains a single operation, it uses a name based on that operation, e.g. 00023_remove_model_field, or 0024_add_model_field (but not for all operation types)
  
• If the migration consists *only* of model creation operations, it combines their operation names names, which come as just the lower-cased model names e.g. 0025_book_author_sale

I opened a PR to expand the operation naming for the single case to cover all built-in operations: https://github.com/django/django/pull/12131 

I'd like to propose using this new full coverage of operation naming to remove the "auto_YYYYMMDD" behaviour, and instead always combine operations' "suggested migration names" up until a limit of say 60 characters. I made a commit for that here: https://github.com/adamchainz/django/commit/c2bc6893a05c2c8099e1fb68e688618446641ed6

This would lead to migration names such as:

• 0026_remove_book_title_add_book_description
  

Whilst perhaps long, they're explict and imo easier to work with than the auto_YYYYMMDD behaviour.

Mariusz wrote on the PR:

Personally, I'm against removing auto named migrations. IMO chaining operation names is even more confusing.

--

Adam

[Caio Ariede]

I think making it possible to customize how migration names are generated has a great value. Changing something from “auto” to some better/descriptive name has its value, but I can also see people creating their own strategies for naming migrations.

I work on a project where migration names are fixed to “auto”. We use a similar technique to those mentioned in the blog post to do that. The reason is that we want to force developers to get conflicts (git) on migration names during the review process, rather than having to handle migration merging manually during deploy

--

Caio

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM10ACO6wEMZXukN6_xy766Bxa8PQOc7teFEQhRmzWxqwA%40mail.gmail.com.

[Claude Paroz]

Le mercredi 22 avril 2020 15:06:12 UTC+2, Adam Johnson a écrit :

I'd like to propose using this new full coverage of operation naming to remove the "auto_YYYYMMDD" behaviour, and instead always combine operations' "suggested migration names" up until a limit of say 60 characters. I made a commit for that here: https://github.com/adamchainz/django/commit/c2bc6893a05c2c8099e1fb68e688618446641ed6

...

Mariusz wrote on the PR:

Personally, I'm against removing auto named migrations. IMO chaining operation names is even more confusing.  

An alternative could be to ask the user in interactive mode (and keep the current behaviour in non-interactive mode).

Claude

[Asif Saif Uddin]

I Like the idea of Claude.

Asif

[Jon Dufresne]

> I'd like to propose using this new full coverage of operation naming to remove the "auto_YYYYMMDD" behaviour

+1 on this proposal. As you've noted in your blog post, I find the default auto name extremely opaque. Projects I work on normally have an internal policy to change these names to something more informative. Inevitably, this means there is always a bit of a learning process for new team members. If Django handled this more automatically, it would remove some of this learning process.

> and instead always combine operations' "suggested migration names"

This sounds reasonable to me. If there is concern about words blending into one another, we could separate them by a double underscore: 0026_remove_book_title__add_book_description. I would personally be fine with either.

> up until a limit of say 60 characters

Did you have a suggestion for this situation? Revert back to auto-naming or request the user to name the migration?

--

[Andrew Godwin]

I am a little mixed on this change - the behaviour during the initial development was indeed to concatenate names like this, albeit only for adding fields or models; I thought it looked unwieldy, which is why I added the "auto" name.

That said, the number is the only part that actually matters, and while the date is sometimes useful for people to resolve merge conflicts, I don't think it's better than more informative autogenerated names, so I'm happy to go with the change.

(60 is a bit long though, maybe we can bump it down to something a bit shorter?)

Andrew

--

[Adam Johnson]

Thank you all for the feedback.

Re: Andrew:

(60 is a bit long though, maybe we can bump it down to something a bit shorter?)

Sure, how about 42? 👽

Re: Jon:

Did you have a suggestion for this situation? Revert back to auto-naming or request the user to name the migration?

The patch as-is accumulates suggested names from operations in order, until the length is >=60 characters. Later operations just aren't mentioned.

Re: Claude:

An alternative could be to ask the user in interactive mode (and keep the current behaviour in non-interactive mode).

I'd be up for this if we "pre-fill" the input with the auto-generated name, to make it easy to continue when we can suggest a reasonable name. Forcing users to type a name from scratch could be annoying especially when iterating on a new feature and dropping/rebuilding the respective migration.

Re: Caio:

I work on a project where migration names are fixed to “auto”. We use a similar technique to those mentioned in the blog post to do that. The reason is that we want to force developers to get conflicts (git) on migration names during the review process, rather than having to handle migration merging manually during deploy

Yes, this is a secondary problem with migrations, trying to keep the history linear via git.

I've used an alternative solution where a second file is created in the migrations folder called "latest" or similar, that simply contains the name of the latest migration file. This forces a conflict.

Although that's an alternative proposal, I think adding something like that to Django could be a good idea. It's better than forcing all migrations to have meaningless names.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1urecp9nGUqwrsxYox%2BtWzv5FcBBWbANg7E9PGAtUG4m_A%40mail.gmail.com.

--

Adam

[Jure Erznožnik]

I too am in favour of this change.

But I read Andrew's comment differently: it's maybe not the 60 characters, but some sort of "aggregation" we could be looking for. So maybe instead of "0026_remove_book_title_add_book_description" we could have "0026_book_add_remove" - especially when multiple fields were involved.

Alternatively, makemigrations could also generate multiple migrations (by default?) in order to obtain the goal of meaningful names? Have a parameter to force combining?

LP,
Jure

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM2HA%2BKPdrnedORUxBvgeAdR-%3DWNb3HvyhsQZgtdbwDnvw%40mail.gmail.com.

[Adam Johnson]

But I read Andrew's comment differently: it's maybe not the 60 characters, but some sort of "aggregation" we could be looking for. So maybe instead of "0026_remove_book_title_add_book_description" we could have "0026_book_add_remove" - especially when multiple fields were involved.

I feel an implementation for anything like this would be very complicated, especially when factoring in RunPython, RunSQL, and custom operations. I don't think it's worth the effort, considering that it's a best "first guess" for the name, and developers can change them if they feel they aren't descriptive (though the point of this change is that they often don't).

Alternatively, makemigrations could also generate multiple migrations (by default?) in order to obtain the goal of meaningful names? Have a parameter to force combining?

I don't think that's ideal because migrations are atomic (by default) on databases that support atomic DDL. Splitting them for the sake of slightly better automatic naming unnecessarily breaks this guarantee.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d29fe147-9c96-b616-3ef2-a8c419326f4c%40gmail.com.

--

Adam

[אורי‎]

On Wed, Apr 22, 2020 at 4:06 PM Adam Johnson <m...@adamj.eu> wrote:

Hi djangonauts,

In a blog post earlier this year I outlined my technique to prevent Django creating migration files with automatic date names. I had a lot of response with other techniques and ended up adding two more techniques to the post. It's at https://adamj.eu/tech/2020/02/24/how-to-disallow-auto-named-django-migrations/ .

The problem with such migration names:

When you run Django’s manage.py makemigrations, it will try to generate a name for the migration based upon its contents. For example, if you are adding a single field, it will call the migration 0002_mymodel_myfield.py. However when your migration contains more than one step, it instead uses a simple ‘auto’ name with the current date + time, e.g. 0002_auto_20200113_1837.py. You can provide the -n/--name argument to makemigrations, but developers often forget this.

Actually I would prefer all the automatically-generated migration file names to be ‘auto’ name with the current date + time, e.g. 0002_auto_20200113_1837.py (except the initial migration which could remain "0001_initial.py"). I don't think there is a need to name the migrations and actually migration names such as "0004_user_has_confirmed_email.py" (which is one of our migrations in Speedy Net) makes it look like someone named it specifically, although it's also automatically generated by Django. Also I prefer the file name to contain the date the migration was generated. Most of our migrations, at least 98% I think, were generated automatically by running "make_migrations" and I would like to keep it this way. Only rarely I would find a need to manually write a migration. If the file name is not like ‘auto’ name with the current date + time, it looks like a migration which was written by a developer.

אורי

[Fran Hrženjak]

I’ve been in both situations more than once: 

1. wanting to just run makemigrations and never look back,
   
2. naming each migration and reminding people to do the same.
   

Small teams / solo = 1, Large teams = 2.

Perhaps a new setting?

AUTO_NAME_MIGRATIONS = None  # makemigrations fails if no name is provided

AUTO_NAME_MIGRATIONS = ‘django.db.migrations.suggest_name’  # default value

AUTO_NAME_MIGRATIONS = ‘my.stuff.name_migration’  # custom callable

In this custom callable the developers can decide exactly how to join the suggested names, what is their char limit (60, 42, 254), what to do if char limit is exceeded (raise or fall back to timestamp), etc.

To me this whole feature feels like a bit on the “advanced” side, so a custom callable seems more appropriate then too much hand-holding.

+1 for “auto_” prefix by default.

My filename char limit would be 255 :)

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABD5YeEHR6XxPDY_YbdFXW_C9M5JSy3DAwD3FNg%2Bzd-pV%3DSXLw%40mail.gmail.com.

[Adam Johnson]

Re: Uri:

If the file name is not like ‘auto’ name with the current date + time, it looks like a migration which was written by a developer.

I think making a distinction between "auto generated" and "hand written" migrations is a bad idea. Django's makemigrations is a code generator, but as a developer you're still responsible for the code. The autodetector isn't perfect, and perhaps never can be. Even "simple" cases like adding a through table to a ManyToManyField are autodetected "incorrectly" ( a real migration needs SeparateDatabaseAndState https://docs.djangoproject.com/en/3.0/howto/writing-migrations/#changing-a-manytomanyfield-to-use-a-through-model ).

I fear marking "auto generated" migrations differently would just encourage (more) lax use of migrations files without reading their content, which invites more risk for data loss and anger with Django for not being perfect.

Re: Fran:

Perhaps a new setting?
...

In this custom callable the developers can decide exactly how to join the suggested names, what is their char limit (60, 42, 254), what to do if char limit is exceeded (raise or fall back to timestamp), etc.

I agree that you might want different behaviour in bigger teams. But I don't think we needd a new setting. If you want to customize the auto-naming, there are enough places you can hook into the process already. I documented three solutions in my blog post.

Another place you can directly change the automatic names is in makemigrations.Command.write_migration_files, which receives a dict of app labels to migration objects. You can customize migration.name there (in a custom subclassed command) before calling super() to affect the output names.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAM2o%3DwMicQZA0PjnvmBSEUyvtAJMDrL19yem3qdPTXXg8M%3D2Sw%40mail.gmail.com.

--

Adam

[אורי‎]

On Sat, Apr 25, 2020 at 5:48 PM Adam Johnson <m...@adamj.eu> wrote:

Re: Uri:

If the file name is not like ‘auto’ name with the current date + time, it looks like a migration which was written by a developer.

I think making a distinction between "auto generated" and "hand written" migrations is a bad idea. Django's makemigrations is a code generator, but as a developer you're still responsible for the code. The autodetector isn't perfect, and perhaps never can be. Even "simple" cases like adding a through table to a ManyToManyField are autodetected "incorrectly" ( a real migration needs SeparateDatabaseAndState https://docs.djangoproject.com/en/3.0/howto/writing-migrations/#changing-a-manytomanyfield-to-use-a-through-model ).

I fear marking "auto generated" migrations differently would just encourage (more) lax use of migrations files without reading their content, which invites more risk for data loss and anger with Django for not being perfect.

As a developer I would like to know who generated the code. If a migration is auto-generated, I would like to know that. I checked and auto-generated migrations in my project have a comment such as "# Generated by Django 2.1.15 on 2020-01-21 15:31". In most cases auto-generated migrations are good and don't need to be edited by a developer. So, I would prefer the migration file names to be auto-generated too (‘auto’ name with the current date + time).

[Tom Forbes]

I would be in favour of Adam’s proposed changes. Adding more context to them and stripping the appended timestamp would make them more user friendly.

> As a developer I would like to know who generated the code. If a migration is auto-generated, I would like to know that. I checked and auto-generated migrations in my project have a comment such as "# Generated by Django 2.1.15 on 2020-01-21 15:31". 

A couple of things:

1. Your VCS will tell you who generated the code

2. The name of the file including “auto” in it doesn’t mean that it wasn’t completely edited by a developer afterwards, making the “auto” useless in knowing who generated the code

A common operation when looking at a list of migrations is to try and find a specific one. I think this is a much more common occurrence than looking at the list of migrations to know how many have been (possibly) generates by Django.

This would make the common case more convenient and simpler, which is good.

Tom

On 25 Apr 2020, at 17:45, אורי <u...@speedy.net> wrote:



--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABD5YeFvWJSotjqhB_L%3Dt9i%3Dx3%3DRt%2Bh-S_gYCAgTV3C8RFMjwg%40mail.gmail.com.

[Adam Johnson]

Since this proposal seems to be broadly supported, I've created a corresponding ticket and pull request: https://code.djangoproject.com/ticket/31516 and https://github.com/django/django/pull/12799 .

Thanks all.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CC4B31C7-EC63-4531-BF69-3672B9909133%40tomforb.es.

--

Adam