GSoC 2020 Proposal for ModelStates in Migrations

197 views
Skip to first unread message

Sanskar Jaiswal

unread,
Mar 23, 2020, 12:59:55 PM3/23/20
to django-d...@googlegroups.com
Hey everyone,

Here is my proposal for GSoC. My project is based upon making use of ModelState during the migrated phase of the project, rather than rendering fake Models.

Feedback and criticism is highly appreciated.

Thanks!
Kind regards
Sanskar

Javier Buzzi

unread,
Mar 23, 2020, 5:14:04 PM3/23/20
to Django developers (Contributions to Django itself)
I don't see how this solves anything. At the end of the day these "real" models, or at the very least, not "fake" models will not have any custom queryset/manager/properties/method you add to it. Can you explain what it is you're trying to accomplish?

charettes

unread,
Mar 23, 2020, 5:48:43 PM3/23/20
to Django developers (Contributions to Django itself)
Javier

> I don't see how this solves anything. At the end of the day these "real" models, or at the very least, not "fake" models will not have any custom queryset/manager/properties/
method you add to it. Can you explain what it is you're trying to accomplish?

I don't think that's the goal of the project in the first place.

To me the abstract section clearly highlights why this is useful. Right now the schema editor requires access to *rendered* model classes which is quite slow as it involves dynamically creating (or recreating) Python classes and all its dependency chain on most alterations during migration application.

It's way cheaper to create and alter ModelState instances (simple class instances) than creating classes hierarchy over and over again.

Sanskar Jaiswal

unread,
Mar 23, 2020, 5:51:55 PM3/23/20
to django-d...@googlegroups.com
Hey Javier!

So currently, during migrations, specifically during the state_forwards and database_forwards methods, Django renders a fake model and passes it down to the SchemEditor which is an expensive operation and slows down the process. The idea is to use ModelStates instead of Models, as their reduced API make them much more efficient.

The initial problem was filed in this ticket: code.djangoproject.com/ticket/29898

Kind regards
Sanskar

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3509e4d0-e99c-4ff3-a4d0-45a5c3a988e3%40googlegroups.com.

charettes

unread,
Mar 23, 2020, 6:07:32 PM3/23/20
to Django developers (Contributions to Django itself)
Hello Sanskar,

Thank you for your submission, from a quick look it seems to be heading in the right direction.

Would it be possible to break the large overview section into more granular sections where you describe how you roughly plan to tackle each of the point mention?

I personally have doubts about your desire to make minimal change to the schema editor and offload the logic to database_forwards instead of the other way around. That doesn't seem to be in line with a future deprecation of support for direct model passing. I'd much rather see us move all methods to (project_state, model_state, *other_state, **other_fields_and_flags) signatures and provide helpers to help third-party backends migrate (e.g. a method decorator that renders mode) then embedding logic for both paths in the schema editor and adding schema specific logic to Operation.database_forwards.

Simon

Javier Buzzi

unread,
Mar 23, 2020, 6:09:50 PM3/23/20
to Django developers (Contributions to Django itself)
Understood. Thanks.

Sanskar Jaiswal

unread,
Mar 23, 2020, 6:51:27 PM3/23/20
to django-d...@googlegroups.com
Hey Simon,

Thank you for your feedback! It helps a lot. I shall look into your doubts right now, and edit my proposal accordingly.

Kind regards
Sanskar

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

Sanskar Jaiswal

unread,
Mar 25, 2020, 6:05:22 AM3/25/20
to django-d...@googlegroups.com
Hey Simon

Just so that I am more clear, is it favourable to move all logic that ModelOperations and FieldOperations implement right now to ProjectState and ModelState?

Kind regards
Sanskar

charettes

unread,
Mar 25, 2020, 2:27:12 PM3/25/20
to Django developers (Contributions to Django itself)
Are you thinking about the Operation.state_forwards logic? I guess it would make it easier to test state alterations in isolation but I don't think it's necessary for this already large project.

Cheers,
Simon

Le mercredi 25 mars 2020 06:05:22 UTC-4, Sanskar Jaiswal a écrit :
Hey Simon

Just so that I am more clear, is it favourable to move all logic that ModelOperations and FieldOperations implement right now to ProjectState and ModelState?

Kind regards
Sanskar
On Tue, 24 Mar 2020 at 04:20, Sanskar Jaiswal <jaiswal....@gmail.com> wrote:
Hey Simon,

Thank you for your feedback! It helps a lot. I shall look into your doubts right now, and edit my proposal accordingly.

Kind regards

Sanskar

On Tue, Mar 24, 2020 at 3:37 AM charettes <char...@gmail.com> wrote:
Hello Sanskar,

Thank you for your submission, from a quick look it seems to be heading in the right direction.

Would it be possible to break the large overview section into more granular sections where you describe how you roughly plan to tackle each of the point mention?

I personally have doubts about your desire to make minimal change to the schema editor and offload the logic to database_forwards instead of the other way around. That doesn't seem to be in line with a future deprecation of support for direct model passing. I'd much rather see us move all methods to (project_state, model_state, *other_state, **other_fields_and_flags) signatures and provide helpers to help third-party backends migrate (e.g. a method decorator that renders mode) then embedding logic for both paths in the schema editor and adding schema specific logic to Operation.database_forwards.

Simon

Le lundi 23 mars 2020 12:59:55 UTC-4, Sanskar Jaiswal a écrit :
Hey everyone,

Here is my proposal for GSoC. My project is based upon making use of ModelState during the migrated phase of the project, rather than rendering fake Models.

Feedback and criticism is highly appreciated.

Thanks!
Kind regards
Sanskar

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

Sanskar Jaiswal

unread,
Mar 25, 2020, 7:10:40 PM3/25/20
to django-d...@googlegroups.com
Hey Simon

I have made some changes to the proposal. I kindly request you to take a look at it and give any feedback you can :)

Kind Regards 
Sanskar



To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/fbf6ba88-2adc-4dd7-ac82-ecea6d1be313%40googlegroups.com.

charettes

unread,
Mar 25, 2020, 8:35:03 PM3/25/20
to Django developers (Contributions to Django itself)
Thanks Sanskar, this iteration is much better.

Now that you have a bit more details about your proposed changes breakdown I'd give a bit more examples of what a particular operation currently looks like now and you plan to change it. For example, you could describe how an AlterField operation currently alter the state/database forwards with the involved function calls and how you plan to changes things.

Detailing a bit more how the related lookups cache invalidation would be busted and populated when a specific operation is performed would go a long way to support your proposal.

To summarize, a few examples of what you have in mind and how things would interact together would be appreciated. This is a quite a complex problem to solve so a solid initial plan and understanding of the problem will make a huge difference down the road.

Cheers,
Simon

Le mercredi 25 mars 2020 19:10:40 UTC-4, Sanskar Jaiswal a écrit :
Hey Simon

Sanskar Jaiswal

unread,
Mar 26, 2020, 4:01:05 PM3/26/20
to django-d...@googlegroups.com
Hi Simon

Taking advice into account, I have made some more changes to my proposal. Please check it out and criticize it wherever you think I could improve it as always. :)

Also, could you kindly give some kind of idea as to when the new mappings should be populated? I think it makes sense to do so whenever we run makemigrations. Do you have a better idea?

Thanks!
Kind Regards
Sanskar

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3b9516a5-3c0e-4707-b142-1eb8e091168b%40googlegroups.com.

charettes

unread,
Mar 26, 2020, 10:40:59 PM3/26/20
to Django developers (Contributions to Django itself)
Thanks for the examples Sanskar.

I think the mapping should be kept ProjetState instances and invalidated when state alterations are performed. In an ideal world I think that ProjectState should have methods mapping to all Operation subclasses state_forwards and these state_forwards methods should only proxy ProjectState methods.

The CreateModel.state_forwards method is a good example of that[0]; it calls into ProjectState.add_model where all the storage and cache invalidation logic is encapsulated[1]. If ProjectState had methods such as rename_model, alter_field, and such it would make it easily testable in isolation and allow it to maintain it's related object cache, or mapping as you call it, more easily.

Simon


Le jeudi 26 mars 2020 16:01:05 UTC-4, Sanskar Jaiswal a écrit :
Hi Simon

Taking advice into account, I have made some more changes to my proposal. Please check it out and criticize it wherever you think I could improve it as always. :)

Also, could you kindly give some kind of idea as to when the new mappings should be populated? I think it makes sense to do so whenever we run makemigrations. Do you have a better idea?

Thanks!
Kind Regards
Sanskar

Sanskar Jaiswal

unread,
Mar 27, 2020, 3:29:11 AM3/27/20
to django-d...@googlegroups.com
Hey Simon,

While I certainly see the benefits of this, I still fail to understand how this would help ModelState get access to related db_types, (the problem that the mapping in my proposal aims to solve). Am I missing something here? Some more clarification/explanation would be highly helpful.

Kind regards
Sanskar

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/668c02b4-8a27-43fd-b449-53121d8e3a73%40googlegroups.com.

Sanskar Jaiswal

unread,
Mar 28, 2020, 2:58:22 PM3/28/20
to django-d...@googlegroups.com
Hey Simon,

Could you kindly reply? 😅

Thanks
Kind regards
Sanskar

Daryl

unread,
Mar 28, 2020, 3:45:30 PM3/28/20
to django-d...@googlegroups.com
Hi Sanskar,

Please be patient with everyone - during this time, some countries are in terrible shape, others are coping well.
In addition, even without Covid19, it is impossible to know what is going on with anybodies situation, and we are (almost) all donating our time.

Regards.
D



--
-- 
======================
Daryl Egarr,  Director
Kawhai Consultants Ltd
Cell       021 521 353
da...@kawhai.net
======================

Sanskar Jaiswal

unread,
Mar 28, 2020, 3:50:53 PM3/28/20
to django-d...@googlegroups.com
Sorry, I didn’t mean to sound impatient . I understand that this pandemic is disrupting life everywhere and I hope everyone is safe and healthy.

I will keep this in mind next time. Again, my apologies.

Kind regards
Sanskar

charettes

unread,
Mar 28, 2020, 5:32:46 PM3/28/20
to Django developers (Contributions to Django itself)
Hey Sanskar,

As Daryl mentioned don't expect an immediate response from me or anyone on this list.

> I still fail to understand how this would help ModelState get access to related db_types, (the problem that the mapping in my proposal aims to solve).

If project state maintains a map of (app_label, model_name): [(from_fields, app_label, model_name, to_fields), ...] you'll be able to resolve related db types easily by doing project_state.models[to_app_label, to_model_name].fields[to_field].db_type(connection).

As long as this map is kept up to date on model alterations the schema editor will be able to easily get access to related db types.

Note that you'll likely need to maintain both a forward and backward relation map because for operations alterations of fields referenced by foreign keys.

Simon


Le vendredi 27 mars 2020 03:29:11 UTC-4, Sanskar Jaiswal a écrit :
Hey Simon,

While I certainly see the benefits of this, I still fail to understand how this would help ModelState get access to related db_types, (the problem that the mapping in my proposal aims to solve). Am I missing something here? Some more clarification/explanation would be highly helpful.

Kind regards
Sanskar

Sanskar Jaiswal

unread,
Mar 29, 2020, 2:53:44 AM3/29/20
to django-d...@googlegroups.com
Hey Simon,

Thank you for this clarification, I shall make some changes to my proposal accordingly. 
Again, my apologies if I seemed impatient.

Kind regards
Sanskar

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b4628e57-95e8-4390-be21-5ba5b2bf2f8d%40googlegroups.com.

Sanskar Jaiswal

unread,
Mar 29, 2020, 2:56:51 PM3/29/20
to django-d...@googlegroups.com
Hey Simon

I have made some more changes yet again. I would love for you to take one final look at it and again give me your valuable feedback before I make the final submission on the GSoC portal. Looking forward to hearing from you and working with everyone in the summer. 😁
https://gist.github.com/aryan9600/b1c2eaf445006c17e02e7677cf1098d5

Thanks
Kind Regards
Sanskar
Reply all
Reply to author
Forward
0 new messages