Supporting a wider range of awesome PostgreSQL features in Django - including hstore, JSON and full text search

[Federico Capoano]

I found this on django's blog:

https://www.kickstarter.com/projects/mjtamlyn/improved-postgresql-support-in-django

Very interesting!

Anybody knows if they intend to redevelop everything from scratch or reuse and improve existing implementations?

Federico

[Russell Keith-Magee]

Marc will be the ultimate authority on this, but from my conversations with him, I believe that his intention is to:

 * use existing implementations where possible, 

 * merge the best parts of existing implementations where multiple options exist, 

 * rebuild implementations that need significant improvement, and 

 * build from scratch where no existing solutions exist.

Yours,

Russ Magee %-)

[Matthieu Rigal]

Is it somehow planned that Django Users can dialog with him to express some desires ?

For example, I would be very happy to see a reimplementation of the annotate and aggregate functions, as they make the queries so slow by grouping by all the fields that I always have to use raw SQL to group only by id.

In one case in my admin, it is trying to group by 40 fields (I have 5 foreign keys and more than 20 fields) instead of 4 IDs and it requires 6 seconds instead of 160 ms...

All the best to Marc

[Russell Keith-Magee]

On Mon, Mar 10, 2014 at 7:22 PM, Matthieu Rigal <matt....@googlemail.com> wrote:

Is it somehow planned that Django Users can dialog with him to express some desires ?

For example, I would be very happy to see a reimplementation of the annotate and aggregate functions, as they make the queries so slow by grouping by all the fields that I always have to use raw SQL to group only by id.

In one case in my admin, it is trying to group by 40 fields (I have 5 foreign keys and more than 20 fields) instead of 4 IDs and it requires 6 seconds instead of 160 ms...

Again, I can't speak for Marc; once the Kickstarter has finalised, I'm sure he'll be in contact with backers to describe the process he'll take. I'd be guessing it's either going to be through his blog, or through django-developers (and possibly both, depending on the level of interaction you're looking for).

That said, the issue you describe sounds like it's outside the scope of what he's looking at. The annotate query you describe may be sub-optimal, but it does *work*. Marc's project is looking at adding the features that PostgreSQL has, but aren't currently accessible using Django's core feature set. This means new field types, new index types, and so on. This is all possible by making a concerted effort to use the features Django already provides, like custom fields and custom lookups.

What you've described is a specific SQL query optimization that presumably would have applicability across multiple database backends. If you've got a proposal for what this would look like at an API level, feel free to propose it. However, keep in mind that Django's ORM is very deliberately *not* a SQL generation engine. There are very few "SQL" concepts at the API level, and discussing "grouping" clauses is very much in that space. You may find that the answer to your question is "use raw SQL". 

Yours,

Russ Magee %-)