Django backend for Psycopg 3

[Daniele Varrazzo]

Hello,

I have maintained psycopg2 in the last >10 years and in the last year I have been developing a new adapter, Psycopg 3, using the experience made so far in the area of connecting Python to PostgreSQL.

https://www.psycopg.org/psycopg3/

https://www.psycopg.org/psycopg3/docs/

One of the goals of the project is to make upgrading from psycopg2 easy, so the project retains the familiar DBAPI 2.0 interface. However, in order to use more advanced PostgreSQL features (prepared statements, binary parameters...) it makes use of server-side parameters binding, which here and there have slightly different behaviour from the client-side binding used in psycopg2.

I have written a first cut of a Django backend for PostgreSQL based on Psycopg 3, which is available as an external module at <https://github.com/dvarrazzo/django-psycopg3-backend>. Using the Django test suite was immensely helpful to find the missing features and the different behaviour: thank you very much! As much as I could, I have changed the adapter to make default choices to reproduce the behaviour of psycopg2. However there are a few cases where the PostgreSQL-related code requires some changes, which makes it compatible with both psycopg2 and > 2.

I have a set of changesets, based on the stable/3.2.x branch, which make the test suite pass with both psycopg 2 and 3; they can be previewed at <https://github.com/django/django/compare/stable/3.2.x...dvarrazzo:psycopg3-support>. Most of the changes are import-related, breaking the assumption that using Postgres implies psycopg2 installed (bopth in the Django code and in its tests). A few changes are needed to accommodate differences in the behaviour of PostgreSQL when using server-side binding. A writeup with the description of these changes is available at <https://www.psycopg.org/articles/2021/08/02/psycopg3-django-driver/>.

I would like to propose these changes for inclusion in Django. I would say that the Psycopg 3 backend module should remain an external package until the adapter becomes more stable and the go-to choice for PostgreSQL. Some of the changes proposed for Django can be implemented in different ways: for instance, instead of importing the Range object from psycopg2 (and now 3, with imports that in my changeset are conditional), Django could have its own Range object implementation and the two backends can be configured to use it instead of the driver-provided ones. Yesterday someone made me aware that the GROUP BY problem was found in the Oracle backend too and solved in a different way (https://code.djangoproject.com/ticket/27632).

So, I would like to know, first, if the Django project has interest in accepting changes with the goal of supporting Psycopg > 2 and server-side parameters binding for PostgreSQL (which would also help other potential backends for drivers using the same parameters passing mechanism, such as asyncpg) and if so what is the best way to propose them. I can propose a set of stand-alone merge requests, where the code and the single alternative implementations can be reviewed.

Thank you very much!

-- Daniele

[Samuel Smith]

I can't speak for the Django project or Foundation, but would like to
thank you for all of your work on this project!

--
Sam

> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com
> <mailto:django-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/f82d221e-fc12-4131-9c81-c38b77746c1an%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/f82d221e-fc12-4131-9c81-c38b77746c1an%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Markus Holtermann]

Hi Daniele,

wow, these are wonderful news! Thank you for all your time on psycopg2 and now 3!

From where I'm standing, I agree, having the psycopg3 Django backend as an external package for now sounds good. It's much easier to keep updated and maintained, without having to go through all of Django's hoops.

As for the changes to Django, I guess we should at least merge those changes that remove the requirement of having psycopg2 installed, even though it's unused. So, namely in the contrib.postgres area. Instead, we should try to use either psycopg2 or psycopg3.

Cheers,

Markus

> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to django-users...@googlegroups.com.

[Chew Kok Hoor]

Thanks and Great work, Daniele Varrazzo!

And thanks to Django maintainers, and other related open-source projects.

You all make this world a better place!

Regards,

    Kok Hoor

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/7584e71f-d4de-42ad-8297-28eac17a3ec7%40www.fastmail.com.