Increase default integer keys to 64 bits

[Christophe Pettus]

tl;dr: Introduce new field types to handle auto-incremented ID fields, change the PostgreSQL backend to use the preferred syntax

--

One of the most common issues my company runs into on Django sites is that models.AutoField defaults to a 32-bit integer (int32). 2^31-1 possible entries is just not that many anymore, and by the time the developers realize this and need to move to a 64 bit integer key, it's too late to do so conveniently, because expanding the field is very painful (in PostgreSQL, at least).

While models.AutoBigField exists, it's barely mentioned in examples, and is often overlooked.

Changing AutoField to 64 bits would result in all kinds of breakage; at best, a lot of very unplanned and expensive migrations.

My proposal is:

1. Create two new field types to represent auto-incrementing primary keys. I'd suggest IdentityField and SmallIdentityField for int64 and int32, respectively.

2. Change all examples to use SerialField instead of AutoField.

3. As a side note, switch the PostgreSQL backend to use the standard "GENERATED BY DEFAULT AS IDENTITY" syntax. This became available in PostgreSQL version 10, but the previous version (9.6) reaches EOL in November 2021.

4. At some point in the future, deprecate AutoField and AutoBigField.

This would result in new projects getting 64 bit primary keys by default. I think that's a positive. For small tables, the size difference hardly matters; for big tables, we have saved a major foot-gun of either integer exhaustion or a very expensive data migration problem.

--

Comments?

--
-- Christophe Pettus
x...@thebuild.com

[Curtis Maloney]

I recall a discussion some time about about adding a setting to control which field to use for the default PK... seems there's some overlap here.

--

C

-- 

You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/F4A50AB6-8CC1-4B6E-BD1B-B865F53A645A%40thebuild.com.

[Tom Forbes]

Better than that: 3.2 is the first step to changing the default. See “customising the type of primary keys” in the release notes (https://docs.djangoproject.com/en/dev/releases/3.2/).

“GENERATED BY DEFAULT AS IDENTITY” would be a nice improvement though.

Tom

On 29 Jan 2021, at 01:58, Curtis Maloney <cur...@tinbrain.net> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/50ec245f-21a4-4ab9-ac51-55be2c969310%40www.fastmail.com.

[charettes]

As Tom said Django 3.2 supports swapping the default primary key of models so the integer exhaustion part of your suggestion should be addressed

Regarding identity columns support there's already a ticket opened about it on Trac[0] where you might want to chime in.

Even #30511 if doesn't land in the near future it shouldn't be too hard to write a field class that uses "GENERATED BY DEFAULT AS IDENTITY" and point DEFAULT_AUTO_FIELD to it.

> At some point in the future, deprecate AutoField and AutoBigField.

That would have a large impact on most Django project so unless we can find a really smooth upgrade path (e.g. for migrations) we should avoid doing that.

Cheers,

Simon

[0] https://code.djangoproject.com/ticket/30511

[Christophe Pettus]

> On Jan 29, 2021, at 07:40, charettes <chare...@gmail.com> wrote:
>
> As Tom said Django 3.2 supports swapping the default primary key of models so the integer exhaustion part of your suggestion should be addressed

That's not particularly related. The issue isn't that there isn't any way to get a 64 bit key; there is, of course, using AutoBigField. It's that the default, based on all of the documentation and code samples available, is to get a 32 bit key using AutoField, and that's a foot-gun with a long-delayed firing time.

The essence of the proposal is to make the public default for new projects 64 bit keys; we're doing developers a disservice by making the default path 32 bit keys.

[Adam Johnson]

64 bits is the default for new projects. See the 3.2 release notes: https://docs.djangoproject.com/en/dev/releases/3.2/

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/C0092F9E-71ED-463D-A0A8-4F00D8F00889%40thebuild.com.

--

Adam