Google Summer of Code Updates - Class based indexes

[akki]

Hi

Here is the list of tasks that I have done related to my GSoC project till now.

• Introducing the django.db.models.indexes module.
• Writing the BaseIndex class - kind of skeleton for all the index classes.
• Writing the Index class for the default (B-Tree) indexes.
• Add CreateIndex/DropIndex migration operation for adding and removing the indexes.
• Add methods to schema for the same.
• Writing corresponding tests for migrations and schema.
• Create docs for indexes and the new operations.

#26709 is the ticket  corresponding to the current checkpoint. The code can be found at https://github.com/django/django/pull/6726.

Regards
Akshesh (akki)

[akki]

Progress on previous PR:

• Design changes in ModelState.options.indexex and RemoveIndex migration
• Improved algorithm for index naming
• Other cleanups and cosmetic changes

Other than that I have started working on the second checkpoint as well which is to add support for creating class based index in the Meta class. The code can be seen at https://github.com/akki/django/pull/1/

Regards
akki

[akki]

This week's work on previous PR:

• Change in signature of Index class
• Write tests for the Index class
• Address all issues on PR

The previous PR has been merged. Now indexes can be added using the AddIndex migration.


The next checkpoint is to add support for using Index classes from model's Meta class. Corresponding ticket - #26808. The code is almost ready, I will be sending a pull request soon.

[thinkwel...@gmail.com]

Will it be possible to create indexes other than btree? In reviewing the code it doesn't look like it but I hope I'm wrong. It'd be awesome to support the many more specific index options without using run_sql.

[Tim Graham]

Support for more index types is planned. This is only part 1 of the work. Please see https://gist.github.com/akki/7fd50505928dac58dc350e6cb186a404 for the timeline.

[thinkwel...@gmail.com]

Wonderful!  I make heavy use of btree_gin in one of my projects; will be great to have these new Index classes!

[akki]

• Restructure index migrations
  PR  - https://github.com/django/django/pull/6866
  Modified the internal working of migrations/schema methods related to indexes. This mainly aims at stabilising the newly added migrations for Index class.

• Introduce Meta.indexes
  PR ready for review at https://github.com/django/django/pull/6857.
  Ticket - https://code.djangoproject.com/ticket/26808

[akki]

• Restructure index migrations - Polished according to all reviews, the PR has been merged.
• Introduce Meta.indexes - Updated PR according to suggestions as per reviews.
• Improve schema's index name creating algorithm - #24442 !6898.
  
• Add support for indexes to inspectdb - This depends on !6857 so I will send a PR once indexes are added to Meta class; Current work on my local branch - https://github.com/akki/django/commits/gsoc-indexes-inspectdb

[akki]

• Introduce Meta.indexes
  PR - https://github.com/django/django/pull/6857.
  Ticket - https://code.djangoproject.com/ticket/26808.
  Made a design change so that index can drop it's model attribute by introducing Index.set_name_with_model method
  Wrote some left out tests for related_dependencies.
  
  
• Fixed #24442 - Improve schema's index name creating algorithm
  PR  - https://github.com/django/django/pull/6898
  Updated according to reviews and suggestions.
  
  
• Resolved bug when unique_together was dropped along with it's field
  PR - https://github.com/django/django/pull/6926.
  Initially wrote a temporary fix (which left out some other corner cases) to the problem (afterwards I realised the better fix and sent PR).
  Other comments about the issue on the ticket - https://code.djangoproject.com/ticket/26180.
  Resolved a similar issue that had crept up for Meta.indexes - updated !6857 accordingly.
  
  

[akki]

• Fixed #24442 - Improved schema's index name creating algorithm
  Updated !6898 to take all column names into account while creating index name

• Add support for column ordering (ASC/DESC) in class based indexes
  Somewhat provides the feature requested in #20888
  WIP code on my branch - https://github.com/akki/django/commits/gsoc-indexes-order
  Modifying introspection of all databases to return column order as well (required for testing) -- implementation for Oracle remaining.
  

• Implement Hash index class
  WIP implementation at https://github.com/akki/django/commits/gsoc-indexes-hash
  Currently implemented only for postgresql because that's the only backend which fully supports it AFAIK (MySQL supports for some storage engines but neither for MyISAM nor InnoDB).

[Jani Tiainen]

Hi,

On 26.07.2016 04:28, akki wrote:

• Fixed #24442 - Improved schema's index name creating algorithm
  Updated !6898 to take all column names into account while creating index name

• Add support for column ordering (ASC/DESC) in class based indexes
  Somewhat provides the feature requested in #20888
  WIP code on my branch - https://github.com/akki/django/commits/gsoc-indexes-order
  Modifying introspection of all databases to return column order as well (required for testing) -- implementation for Oracle remaining.
  

If you get stuck with Oracle I can lend a hand to help to figure out the details.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/be05e9f0-ed44-4c73-aa41-b02349584e29%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Jani Tiainen

[akki]

Hi Jani

Thanks a lot for offering help. I did face some issues with the Oracle setup in the beginning but was able to resolve them yesterday. :)

You are most welcomed to review the PR and offer any suggestions for improvement at https://github.com/django/django/pull/6982/. The work related to the feature is in progress but the introspection-related work is mostly complete.

Regards

Akshesh(akki)

[akki]

• Add support for column ordering (ASC/DESC) in class based indexes
  

• Completed - tests, docs, code
  PR - https://github.com/django/django/pull/6982
  
  
• Implement Hash index class
  Add capability to get type of index in introspection (required for testing this feature)
  Completed - Add tests, docs, code
  PR - https://github.com/django/django/pull/6995
  Ticket - https://code.djangoproject.com/ticket/26974
  
  
• Some minor changes in !6857

Other than these, I also had a look at btree_gin indexes and extensions but haven't started the implementation yet.

[akki]

• Introduce Meta.indexes - !6857, #26808
  The PR has been merged. Now class based indexes can be added using Meta.indexes.
  
  
• Add support for column ordering (ASC/DESC) in class based indexes - !6982, #20888
  Resolve test failures on Oracle
  Update according to review suggestions
  
  
• Resolve bug when unique_together was dropped along with it's field - #26180
  Solve with another approach by introducing [Add|Remove]UniqueTogether operations and removing generate_altered_unique_together method of migrations.Autodetector
  PR - https://github.com/django/django/pull/7038
  
  
• Add support for GIN indexes
  Add BtreeGinExtension operation to contrib.postgres to create btree_gin extension
  Add GinIndex to contrib.postgres.indexes
  Ticket - https://code.djangoproject.com/ticket/27030
  PR - https://github.com/django/django/pull/7046
  

[akki]

• Add support for column ordering (ASC/DESC) in class based indexes - !6982, #20888
  

• The PR has been merged. More on using it over here.
  
  
• Resolve bug when unique_together was dropped along with it's field - !7038, #26180
  Added tests for the new operation. WIP.
  
  
• Add support for GIN indexes - !7046, #27030
  You can use GIN indexes now using the new GinIndex index class (and install btree_gin extension using the BtreeGinExtension operation, if required.)
  
  
• Take indexes into account in inspectdb command
  Updated/completed earlier code
  Ticket - https://code.djangoproject.com/ticket/27060
  PR - https://github.com/django/django/pull/7083
  
  
  

[thinkwel...@gmail.com]

Thank you, akki, for this project! Have enjoyed following your progress. It'll be something I'm going to make significant use of.

[akki]

Hi

The GSoC 2016 period has almost reached it's completion and I have completed my submission. You can find my work submission here - https://gist.github.com/akki/f45c7cf5db30c025f49d862d82dd259a.

I enjoyed a lot working on this project and am very excited about everyone making use of class based indexes. There is some work remaining which would be helpful to extend support for other index types and add new index classes. I will continue working on it and try to finish it soon. Meanwhile, you can use the current indexes from the master repo i.e. development version (and report bugs/provide feedback).

Many thanks to everyone for their involvement and valuable suggestions throughout the summer. :)

Cheers

akki

[Josh Smeaton]

Great work Akki! I know there hasn't been a lot of activity on this thread, but I'm really looking forward to using the new indexing features, and I bet there are a lot of others. It's also been fun to follow along with your progress as you've been pushing many smaller commits rather than one giant one at the end. From where I sit this looks like an extremely successful GSOC project, well done!

[Cheng Chi]

Great work, akki!

I think you should add it to 1.11 Changelog as major new feature, and add yourself to authors list as well.

[akki]

Thanks a lot Josh! Actually it was this mail sent by you that made me interested in this idea and later take it up as a project for GSoC, so a big THANK YOU to you. :)

Great work, akki!

I think you should add it to 1.11 Changelog as major new feature, and add yourself to authors list as well. 

 

Thanks. I have sent a PR to add it to the release docs, you can have a look at it at https://github.com/django/django/pull/7144.