[Django] #27996: Add pgcrypto extension and GEN_RANDOM

Django

unread,

Mar 29, 2017, 5:51:23 PM3/29/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------
Reporter: Paolo | Owner: Paolo Melchiorre
Melchiorre |
Type: New | Status: assigned
feature |
Component: | Version: master
contrib.postgres | Keywords: uuid extension
Severity: Normal | function random postgresql
Triage Stage: | cryptography
Unreviewed | Has patch: 1
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
After the introduction of the UUID Field in Django 1.8, I believe that
django.contrib.postgres could benefit from some custom functions based on
the pgcrypto extension of PostgreSQL (see
https://www.postgresql.org/docs/9.6/static/pgcrypto.html). That kind of
functions would be very helpful for apply a migration that adds a unique
non-nullable field to a table with existing rows.

Starting from "Migrations that add unique fields" (see
https://docs.djangoproject.com/en/dev/howto/writing-migrations
/#migrations-that-add-unique-fields) I speed up the `gen_uuid` using
`GEN_RANDOM_UUID` function, I change the function from:

{{{
import uuid

def gen_uuid(apps, schema_editor):
MyModel = apps.get_model('myapp', 'MyModel')
for row in MyModel.objects.all():
row.uuid = uuid.uuid4()
row.save(update_fields=['uuid'])
}}}

to

{{{
from django.contrib.postgres.functions import RandomUUID

def gen_uuid(apps, schema_editor):
MyModel = apps.get_model('myapp', 'MyModel')
MyModel.objects.update(uuid=RandomUUID())
}}}

Using this function on my system the time to migrate more than 10000
objects decreased from

{{{
real 0m15.988s
user 0m10.680s
sys 0m0.508s
}}}

to

{{{
real 0m2.957s
user 0m1.736s
sys 0m0.072s
}}}

I already implemented a solution for thi feature and I'm crating a related
pull request.

--
Ticket URL: <https://code.djangoproject.com/ticket/27996>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Mar 29, 2017, 6:08:28 PM3/29/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Paolo Melchiorre):

I just submitted a full featured pull request on ghithub
https://github.com/django/django/pull/8265

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:1>

Django

unread,

Mar 30, 2017, 3:18:10 AM3/30/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Keywords: uuid extension | Triage Stage: Accepted
function random postgresql |
cryptography |

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Claude Paroz):

* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:2>

Django

unread,

Mar 30, 2017, 3:46:24 AM3/30/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Paolo Melchiorre:

Old description:

New description:

After the introduction of the UUID Field in Django 1.8, I believe that
django.contrib.postgres could benefit from some custom functions based on
the pgcrypto extension of PostgreSQL (see
https://www.postgresql.org/docs/9.6/static/pgcrypto.html). That kind of
functions would be very helpful for apply a migration that adds a unique
non-nullable field to a table with existing rows.

Starting from "Migrations that add unique fields" (see
https://docs.djangoproject.com/en/dev/howto/writing-migrations
/#migrations-that-add-unique-fields) I speed up the `gen_uuid` using

`GEN_RANDOM_UUID` function changing it from:

{{{
import uuid

def gen_uuid(apps, schema_editor):
MyModel = apps.get_model('myapp', 'MyModel')
for row in MyModel.objects.all():
row.uuid = uuid.uuid4()
row.save(update_fields=['uuid'])
}}}

to

{{{
from django.contrib.postgres.functions import RandomUUID

def gen_uuid(apps, schema_editor):
MyModel = apps.get_model('myapp', 'MyModel')
MyModel.objects.update(uuid=RandomUUID())
}}}

Using this function on my system the time to migrate more than 10000
objects decreased from

{{{
real 0m15.988s
user 0m10.680s
sys 0m0.508s
}}}

to

{{{
real 0m2.957s
user 0m1.736s
sys 0m0.072s
}}}

I already implemented a solution for thi feature and I'm crating a related
pull request.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:3>

Django

unread,

Mar 30, 2017, 3:58:24 AM3/30/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Paolo Melchiorre):

* has_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:4>

Django

unread,

Mar 30, 2017, 4:00:12 AM3/30/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Paolo Melchiorre):

* has_patch: 0 => 1

Old description:

> After the introduction of the UUID Field in Django 1.8, I believe that
> django.contrib.postgres could benefit from some custom functions based on
> the pgcrypto extension of PostgreSQL (see
> https://www.postgresql.org/docs/9.6/static/pgcrypto.html). That kind of
> functions would be very helpful for apply a migration that adds a unique
> non-nullable field to a table with existing rows.
>
> Starting from "Migrations that add unique fields" (see
> https://docs.djangoproject.com/en/dev/howto/writing-migrations
> /#migrations-that-add-unique-fields) I speed up the `gen_uuid` using

> `GEN_RANDOM_UUID` function changing it from:

>
> {{{
> import uuid
>
> def gen_uuid(apps, schema_editor):
> MyModel = apps.get_model('myapp', 'MyModel')
> for row in MyModel.objects.all():
> row.uuid = uuid.uuid4()
> row.save(update_fields=['uuid'])
> }}}
>
> to
>
> {{{
> from django.contrib.postgres.functions import RandomUUID
>
> def gen_uuid(apps, schema_editor):
> MyModel = apps.get_model('myapp', 'MyModel')
> MyModel.objects.update(uuid=RandomUUID())
> }}}
>
> Using this function on my system the time to migrate more than 10000
> objects decreased from
>
> {{{
> real 0m15.988s
> user 0m10.680s
> sys 0m0.508s
> }}}
>
> to
>

> {{{
> real 0m2.957s
> user 0m1.736s
> sys 0m0.072s
> }}}
>
> I already implemented a solution for thi feature and I'm crating a
> related pull request.

New description:

After the introduction of the UUID Field in Django 1.8, I believe that
django.contrib.postgres could benefit from some custom functions based on
the pgcrypto extension of PostgreSQL (see
https://www.postgresql.org/docs/9.6/static/pgcrypto.html). That kind of
functions would be very helpful for apply a migration that adds a unique
non-nullable field to a table with existing rows.

Starting from "Migrations that add unique fields" (see
https://docs.djangoproject.com/en/dev/howto/writing-migrations
/#migrations-that-add-unique-fields) I speed up the `gen_uuid` using

`GEN_RANDOM_UUID` function changing it from:

{{{
import uuid

def gen_uuid(apps, schema_editor):
MyModel = apps.get_model('myapp', 'MyModel')
for row in MyModel.objects.all():
row.uuid = uuid.uuid4()
row.save(update_fields=['uuid'])
}}}

to

{{{
from django.contrib.postgres.functions import RandomUUID

def gen_uuid(apps, schema_editor):
MyModel = apps.get_model('myapp', 'MyModel')
MyModel.objects.update(uuid=RandomUUID())
}}}

Using this function on my system the time to migrate more than 10000
objects decreased from

{{{
real 0m15.988s
user 0m10.680s
sys 0m0.508s
}}}

to

{{{
real 0m2.957s
user 0m1.736s
sys 0m0.072s
}}}

I already implemented a solution for thi feature and I've created a
[https://github.com/django/django/pull/8265 PR]

--

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:5>

Django

unread,

Apr 6, 2017, 4:11:54 AM4/6/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Markus Holtermann):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:6>

Django

unread,

Apr 6, 2017, 8:50:55 AM4/6/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Paolo Melchiorre):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:7>

Django

unread,

Apr 7, 2017, 3:37:05 AM4/7/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Keywords: uuid extension | Triage Stage: Ready for
function random postgresql | checkin
cryptography |

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Marc Tamlyn):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:8>

Django

unread,

Apr 25, 2017, 8:41:05 PM4/25/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Reporter: Paolo Melchiorre | Owner: Paolo
| Melchiorre

Type: New feature | Status: closed
Component: contrib.postgres | Version: master
Severity: Normal | Resolution: fixed

Keywords: uuid extension | Triage Stage: Ready for
function random postgresql | checkin
cryptography |

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham <timograham@…>):

* status: assigned => closed
* resolution: => fixed

Comment:

In [changeset:"fcb5dbfec0542faaa1b0adad754a1caf1bcd65e2" fcb5dbfe]:
{{{
#!CommitTicketReference repository=""
revision="fcb5dbfec0542faaa1b0adad754a1caf1bcd65e2"
Fixed #27996 -- Added RandomUUID function and CryptoExtension to
contrib.postgres.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:9>

Django

unread,

May 15, 2017, 9:26:47 AM5/15/17

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by GitHub <noreply@…>):

In [changeset:"650bf6714d5648811a8bc08ce53b7fa86cc38d40" 650bf671]:
{{{
#!CommitTicketReference repository=""
revision="650bf6714d5648811a8bc08ce53b7fa86cc38d40"
Refs #27996 -- Skipped RandomUUID test on PostgreSQL 9.3.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:10>

Django

unread,

Jul 28, 2020, 3:57:53 AM7/28/20

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak <felisiak.mariusz@…>):

In [changeset:"628c4a26eee93417bb7161aa393dffb3f4c146b2" 628c4a2]:
{{{
#!CommitTicketReference repository=""
revision="628c4a26eee93417bb7161aa393dffb3f4c146b2"
Refs #27996 -- Doc'd no extension required for RandomUUID() on PostgreSQL
13+.

https://www.postgresql.org/docs/13/functions-uuid.html
https://www.postgresql.org/docs/13/pgcrypto.html#id-1.11.7.34.10.5
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:11>

Django

unread,

Jul 31, 2020, 5:31:04 AM7/31/20

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Iuri de Silvio):

Hi! Today I updated my master and test migrations failed because of
https://github.com/django/django/pull/13241 . You checked for
`is_postgresql_13` even if the connection is not postgresql. It fails with
psycopg2 installed but using another db (sqlite in my case).

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:12>

Django

unread,

Jul 31, 2020, 5:46:26 AM7/31/20

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Iuri de Silvio):

For completeness, here is the stacktrace.

{{{
Traceback (most recent call last):
File "runtests.py", line 571, in <module>
failures = django_tests(
File "runtests.py", line 313, in django_tests
failures = test_runner.run_tests(
File "django/django/test/runner.py", line 707, in run_tests
old_config = self.setup_databases(aliases=databases)
File "django/django/test/runner.py", line 626, in setup_databases
return _setup_databases(
File "django/django/test/utils.py", line 170, in setup_databases
connection.creation.create_test_db(
File "django/django/db/backends/base/creation.py", line 65, in
create_test_db
call_command(
File "django/django/core/management/__init__.py", line 168, in
call_command
return command.execute(*args, **defaults)
File "django/django/core/management/base.py", line 394, in execute
output = self.handle(*args, **options)
File "django/django/core/management/base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "django/django/core/management/commands/migrate.py", line 92, in
handle
executor = MigrationExecutor(connection,
self.migration_progress_callback)
File "django/django/db/migrations/executor.py", line 18, in __init__
self.loader = MigrationLoader(self.connection)
File "django/django/db/migrations/loader.py", line 53, in __init__
self.build_graph()
File "django/django/db/migrations/loader.py", line 210, in build_graph
self.load_disk()
File "django/django/db/migrations/loader.py", line 112, in load_disk
migration_module = import_module(migration_path)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in
import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in
_find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File "django/tests/postgres_tests/migrations/0001_setup_extensions.py",
line 22, in <module>
needs_crypto_extension = not connection.features.is_postgresql_13
AttributeError: 'DatabaseFeatures' object has no attribute
'is_postgresql_13'
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:13>

Django

unread,

Jul 31, 2020, 5:53:42 AM7/31/20

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by felixxm):

Iuri, thanks for the report, fix is ready
[https://github.com/django/django/pull/13261 PR].

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:14>

Django

unread,

Jul 31, 2020, 6:38:27 AM7/31/20

to django-...@googlegroups.com

#27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres
-------------------------------------+-------------------------------------

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by GitHub <noreply@…>):

In [changeset:"88af11c58baf0eae2fa947a5f0733906ffe6bb38" 88af11c]:
{{{
#!CommitTicketReference repository=""
revision="88af11c58baf0eae2fa947a5f0733906ffe6bb38"
Refs #27996 -- Fixed postgres_tests crash if not running with PostgreSQL.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/27996#comment:15>

Reply all

Reply to author

Forward

[Django] #27996: Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django

Django