Django archive models/tables

619 views
Skip to first unread message

Lemuel Formacil

unread,
Aug 27, 2014, 4:15:24 PM8/27/14
to python-mena
So, I have many Django models (so, many database tables) and these models/tables are now getting big that it's starting to affect performance.  I want to create an "archive" table for each of these models so I can move some old records, soft-deleted records, etc. away from my main table for the "active" records and into the "archive" table for my model.

How would you guys do this in Django so that a model will have 2 tables: one for the active records and another for the archived records?  Preferably, I will maintain only one model definition for both tables to make sure their schema is the same.  The logic for moving the records from the active table and into the archive table, and vice versa, will be in the app so preferably the archive table can be accessed using the queryset API as well.  Any ideas?

I'm thinking, at the end of my models file, I can write some model factory function (and call it) that will introspect the current module and for each instance of a model class (or some base or mixin for "archived" models) I construct an <OriginalModelName>Archive model class.  This way, I only have one model class to define, while, in theory, in my head at least, syncdb and migrate will pick up the factory generated models and create or migrate the archive tables.

Does the above sounds like a good idea?  Is there a more elegant, django-idiomatic way to do this?


-- 
Lemuel

azizmb.in

unread,
Aug 28, 2014, 2:07:04 AM8/28/14
to Lemuel Formacil, python-mena
Couldn't you also do this by creating a database adapter? Small differences, if you move from archive table, to archive database, then you could switch reads/writes to the archive db.



--
You received this message because you are subscribed to the Google Groups "from mena import python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-mena...@googlegroups.com.
To post to this group, send email to pytho...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Abde Sayedna,
Aziz M. Bookwala

Lemuel Formacil

unread,
Aug 28, 2014, 2:47:24 AM8/28/14
to azizmb.in, python-mena
I don't know.  What's the db adapter for?  What can it do?  Is that the db engine or db router?

My primary requirement now is having the "archive" tables created automatically on syncdb.

Thanks.

-- 
Lemuel

Abd Allah Diab

unread,
Aug 28, 2014, 4:53:37 AM8/28/14
to Lemuel Formacil, azizmb.in, python-mena
I've done it before but in a semi automated way, here's a sample gist: https://gist.github.com/mpcabd/c7bd45126babe4650902.

To archive an instance I call:
ArchivedMyModel.create_from_instance(instance)
instance.delete()

To unarchive an instance I call:
MyModel.create_from_archived(archived_instance)
archived_instance.delete()

The only problem here is that when you have foreign keys with specific related_name attribute you'll have to include %(class)s in it so that Django can replace it with the appropriate class name of either the model or the archived model.

And I usually have a cron job that calls a management command that archives instances that should be archived (incomplete old transactions, deleted objects, etc.)

You can - I think, I'm not really sure because I didn't explore it - try to fiddle with meta classes to create something fully automated, but you'll have to be careful as Django uses meta classes to define models and their attributes. If you had the time to explore this please share what you find with us ;)

Hope this helps you Lemuel.

Abd Allah Diab, Software Engineer.
Dubai, United Arab Emirates
Mobile: +971 555 44 82 04
mpc...@gmail.com
http://mpcabd.igeex.biz

 LinkedIn  Blog RSS  Twitter

Lemuel Formacil

unread,
Sep 1, 2014, 1:43:06 AM9/1/14
to Abd Allah Diab, azizmb.in, python-mena
Thanks, Abd Allah.

I guess the only way is to really define a model for the archive tables.  I was looking for a way actually to not define the model for the archive table because I have a lot of models that need archiving and I don't want to double the class definitions in my models file.  I was wondering if there's an option in the Meta inner class that I can set to make the model have more than 1 table (I see an auto_created Meta option that takes a model -- I believe it's used for the m2m tables -- but I'm not sure if I can use it and if there's a way to pass to it the same class being built).

Anyway, this is what I have at the moment, which I added to the bottom of my models file:

import inspect


for klass_name, klass in inspect.getmembers(sys.modules[__name__], inspect.isclass):
    if issubclass(klass, BaseOfModelsIWantArchived) and not klass.Meta.abstract:
        new_class_name = '%sArchive' % (klass_name,)
        new_class = type(new_class_name, (klass,), {'__module__': __name__})
        globals()[new_class_name] = new_class

with the above code at the bottom of models.py file all my models that extends BaseOfModelsIWantArchived say, a model of MyModel(BaseOfModelsIWantArchived ), will automatically get a MyModelArchive class.

You guys find anything wrong with the above code?  (I'm still partial to the use of inspect).

-- 
Lemuel
Reply all
Reply to author
Forward
0 new messages