Re: Is there a recommended pattern for creating "replaceable" apps?

[Russell Keith-Magee]

On Tue, Nov 20, 2012 at 6:28 AM, Chris Cogdon <ch...@cogdon.org> wrote:

Hi folks!

I'm creating an application to manage an art show (with art pieces to view and sell, silent and voice auctions, and a sales process). To have this work, it of course needs to keep track of people (including names, addresses, email, etc). But to make this more useful to anyone, I want to include the ability to replace the included "Person" model with another, should the implementor so choose.

What I've done so far is to split the Person model into another app (called Peeps) and removed all but a few necessary linkages between the Artshow and Peeps app. It was reasonably simple, but a few things seem "dirty" so I'm wondering if anyone else has a more experienced or authoritative suggestion for this.

 

Ok… so… In Django 1.5, yes there is… unofficially.

Now - I must stress -- everything I'm about to say is 100% undocumented, and 100% experimental. If it breaks, you get to keep all the shiny pieces :-) I'm only mentioning it because I (personally) need people to experiment in this space in order to prove to the core team that the feature is safe for public consumption.

With that caveat in place:

In Django 1.5 (i.e., the current development branch), we've added the ability to add swappable User models. This means you can replace Django's builtin User model with any User model you want -- for example, a user model that uses 'email' as the unique identifier, or one that captures API credentials rather than first and last name.

The dirty secret is that when I implemented this feature for contrib.auth, I did so in a way that was completely independent of the User model itself. There aren't any explicit references to contrib.auth in the main model code that makes swappable User models possible. In theory *any* model can be declared as swappable, which will allow the end user to define that in their project the "X" model will be performed by model "Y", and any foreign keys will be re-routed appropriately.

The magic sauce: On the model you want to be swappable (Person, in your case), add a Meta declaration:

class Person(Model):

    class Meta:

        swappable = 'CUSTOM_PERSON_MODEL'

When you synchronise your database, sycndb will look for a setting called 'CUSTOM_PERSON_MODEL'; if it exists, it won't sync Person into the database, and will replace any references to Person with references to the model defined in CUSTOM_PERSON_MODEL. 

See the development docs for custom user models to see how this works in practice, and adapt for your purposes. I'm intentionally going to leave the rest vague. Call it an entrance exam -- if you can't work out what's going on from what I've given you here and reading Django's source code, you probably shouldn't be playing around with this sort of experimental feature. However, I'm happy to answer specific questions if you get stuck deep in the grass.

Once again, I must stress that this is unofficial, and you'll be in experimental territory if you do this. The reason this is unofficial is because some on the core team are concerned about reopening old wounds -- essentially, swapping models can lead to all sorts of headaches, and the Django project has had these headaches in the past. Consider, for example, if a user changes the CUSTOM_PERSON_MODEL after they've synchronised. Hilarity *will* ensue. There's also problems related to forms and setting an implied contract around your swappable model.

However, it *can* work; and to my mind, we're all consenting adults, so as long as we all understand the potential pitfalls, this is the sort of thing that may be useful. 

If I haven't scared you off by this point, I certainly hope you'll have a tinker and see if this might be palatable for your project.

Yours,

Russ Magee %-)

[Chris Cogdon]

On Monday, November 19, 2012 9:36:40 PM UTC-6, Russell Keith-Magee wrote:

See the development docs for custom user models to see how this works in practice, and adapt for your purposes. I'm intentionally going to leave the rest vague. Call it an entrance exam -- if you can't work out what's going on from what I've given you here and reading Django's source code, you probably shouldn't be playing around with this sort of experimental feature. However, I'm happy to answer specific questions if you get stuck deep in the grass.

Once again, I must stress that this is unofficial, and you'll be in experimental territory if you do this. The reason this is unofficial is because some on the core team are concerned about reopening old wounds -- essentially, swapping models can lead to all sorts of headaches, and the Django project has had these headaches in the past. Consider, for example, if a user changes the CUSTOM_PERSON_MODEL after they've synchronised. Hilarity *will* ensue. There's also problems related to forms and setting an implied contract around your swappable model.

However, it *can* work; and to my mind, we're all consenting adults, so as long as we all understand the potential pitfalls, this is the sort of thing that may be useful. 

If I haven't scared you off by this point, I certainly hope you'll have a tinker and see if this might be palatable for your project.

Sorry, not scared me off yet :) I'll go take a look at that code. I was aware of the replaceable User model, but didn't consider that perhaps this was also setting a standard framework for any replaceable model.

If the framework is simple enough, I could gear my own application to use it and dropship code from 1.5 into my own project, much like Django dropshipped dictconfig for the logging module.

Thanks for the reference, and I'll shoot you questions if/when I have them.

[Marc Aymerich]

wow, I'm also taking notes of this :)

--
Marc

[Chris Cogdon]

Alright! I've gone through the documentation, and waded through a lot of the Django source code that implements the AUTH_USER_MODEL setting (via Meta.swappable, and Model._meta.swapped, etc) and while I am not saying that this isn't a good idea, it seems very heavyweight for what I was trying to do. In particular, there are a lot of places in the contrib.admin code that checks for model._meta.swapped and disables functionality if it indeed is swapped out, even outside the case of the User model.

Ie, I fear that if I use this paradigm, I'll continually be discovering circumstances where this or that neato feature in the admin screens stops working for me.

What Meta.swappable seems to promote is the idea of being able to replace a model with another, while still being able to access that model in all the same locations it would have been visible in before being swapped, AND ensuring that whenever any application requests a auth.User object, they actually get an instance of the replaced object (with some limitations). 

That's totally awesome, but far more than what I needed, which was a way to allow my app (artshow) to have a reference to a settings-configurable Person model with no changes to either the artshow app, or the app that supplies the Model. Ie, I'm looking for a configurable "has-a" relationship, than a configurable "is-a" relationship. "has-a" seems a lot simpler!

So, I'd like to present to you what I've done with my own code. I've gone through several iterations and have boiled it down to what I think is the simplest possible representation. Feedback and comments are gratefully received, and if you think it's useful, I'll provide a HOWTO incorporating all the (useful) feedback.

Requirements for a solution:

• Zero changes are allowed to the component/application that is providing the model (the target)
• All references to the configurable model in the source app are foreign keys (models.ForeignKey, models.OneToOneField, etc)

Implementation:

• The documentation for the source app provides a list of "attribute, field and method" requirements for the target. If the target cannot provide these, a "Proxy Model" needs to be created.
• If the source app needs to run any search queries on the target, a proxy is certainly required to provide methods to return tuples of strings (for "search_fields" in admin models) and a Q object (for searching via querysets) 
• A setting is provided to indicate which target model is providing the functionality. This points to either the TargetModel or the ProxyModel if required.
• The model module for the source app creates an alias for the replaceable model using: ReplaceableModel = models.loading.get_model ( *settings.SETTING_NAME.split('.',1) )
• This works better than using settings.SETTING_NAME everywhere, as it allows views, etc, to use "models.ReplaceableModel.objects..." etc.
• The ProxyModel is created specifically to match the requirements of the source app, and the features of the target app. This is set up as a Meta.proxy = True
• If the source app needs to display some kind of attribute in templates, and it is not provided by the Target Model, a method (or property) is created in the proxy.
• If the source app needs to create different kinds of searches in admin screens, since search_fields needs real fields to search on, a method on the proxy is created. returning a tuple of strings: e.g.: @staticmethod def get_blah_search_fields ( prefix ): return ( prefix+"attr1", prefix+"attr2" )
• If the source app needs to create queryset style fields, a similar method is provided returning a Q object. e.g.: @staticmethod def get_blah_search_q ( search_str, prefix ): return Q(**{prefix+"attr1__icontains":search_str} | Q(**{prefix+"attr2__icontains":search_str})
• __unicode__ can be overridden if a customized default representation is required.

In addition, the source app probably needs a good way of finding and attaching instances of the Replaceable Model. With no change, the above will supply a selection box or a pop-up window (with raw_id_fields), but I've found using django-ajax-selects to be excellent. If the TargetModel supplies a LookupChannel for those objects, you'll need to create a new LookupChannel with a 'model' attribute of "ReplaceableModel" instead. Usually simply inheriting from the existing LookupChannel and overriding the model attribute works. Otherwise the LookupChannel will return the wrong kinds of objects and you won't be able to do assignments, etc.

Now, the above isn't really documentation, I've got a branch with all the above changes checked into github that people can poke at:

https://github.com/chmarr/artshow-jockey/tree/peeps2

The parts relevant to this discussion are:

• ARTSHOW_PERSON_CLASS in artshowproject.common_settings.py
  
• AJAX_LOOKUP_CHANNELS in the same, and in particular the "artshow_person" channel, which points to artshow_shims.lookups
  
• A simple contact manager application called "peeps". This is suppled as the "default" implementation for Person. I've coded this in a way that peeps knows nothing about any other application.
• The entire artshow_shim component/app, which provides:
• models.Person, which inherits from peeps.models.Person, a customised __unicode__ method, and two search methods
• lookups.PersonLookup, which inherits from peeps.lookups.PersonLookup, and simply overrides the "model" attribute
• artshow.models.Person is implemented as:
• Person = models.loading.get_model ( *settings.ARTSHOW_PERSON_CLASS.split('.',1) )
  
• Inside views, etc, for artshow, if I ever need to to a search on a model that includes fields from the "person" it has a foreign key with, instead of doing search_fields = ( "attr1", "attr2", "person__name", "person__email" ) I do search_fields = ( "attr1", "attr2" ) + models.Person.get_search_fields ( "person__" )
• Similarly for filters, etc: Artist.objects.filter ( Q(artistname__icontains=txt) | models.Person.get_search_q ( txt, "person__" ) )
• You can see some of the above two cases in the artshow/admin.py file

Advantages:

• Zero changes to the target implementation
• The "source app" doesn't need to be modified if the implementation details in the target implementation change. Simply, the proxy model is updated to provide the necessary functionality. 

Disadvantages:

• This makes some queries (eg: the search_fields, and Q filters) a touch more complex. Ie, if you're making a query that makes assumptions of the internals of the replaceable module, make sure that they're either simple enough to be handled with a property that emulates the behavior, which generally works if you're only seeking one attribute, or use methods that return customisable tuples for your purposes.
  

I'd love to hear your comments! I apologise if people were eagerly awaiting documentation on how to make swappable models useful for their own projects! :)