Request for comments - django-cutemodel (model logging and field change auditing)

[Cal Leeming [Simplicity Media Ltd]]

Hi guys,

We have just released a new module that allows for lightweight/easy relational event logging and auditing field changes.

Our use case was to satisfy four main requirements;

* Log events directly from models, whilst keeping a relational link to the row that triggered the event

* Keep track of field changes (storing the old/new value).

* Provide a scalable/easy to use API that allowed access to this information

* Ensure module was a drop-in replacement (only change required is to subclass CuteModel)

The code itself has been dragged out of our existing projects, cleaned up slightly, and released as open source.

Full documentation and source code has been made available here:

https://github.com/foxx/django-cutemodel

Particular care has been made to ensure the code is able to scale up to many millions of rows, and we have not yet had any issues.

In future, we'd love to add some extra features and do a bit more tidy up - so any testing/feedback/features suggestions/comments would be appreciated.

Cheers

Cal

[Cal Leeming [Simplicity Media Ltd]]

On Mon, Sep 10, 2012 at 11:07 PM, Kurt Pruhs <kpr...@gmail.com> wrote:

Hey Cal,

This looks like a great tool. I know I've implemented code like this in another project. I was planning on doing a field change audit module for an application I'm currently working on. I will definitely look at django-cutemodel and see if it works for what I need, and how I can contribute.
My current project is a time clock system for human resources to manage hourly workers. We need field change auditing for security, we need the ability to produce reports from it, and ability to restore from audit history (in case of record tampering).

I think django-cutemodel would be a pretty good fit for this requirement, although it doesn't yet have any sort of administration interface to produce reports, and the documentation isn't exactly great.

Restore from audit history functionality is something we need for ourselves too, so I've raised an issue (will prob fix that sometime this week);

https://github.com/foxx/django-cutemodel/issues/1

We also need to have the ability to bind the Django User objects together with the event/field change - but some of our clients don't use the Django built-in user/auth stuff - so I need to have a little think about how to satisfy both.

https://github.com/foxx/django-cutemodel/issues/2

Basically, all actions are logged and nothing is deleted or over-written. When changes are made, the original record is marked as a parent archive. The modified record is a dup of the parent, but with the changes.

The approach of duplicating the row individually could work in some cases, but if there was any unique index constraints then I'm guessing the rows would have to be moved into a different table. If this were the case, you'd need twice the amount of tables, which in itself may be undesirable, plus another which stores the change relationships, plus any changes would have to be done twice.

Instead, django-cutemodel doesn't require a sub-table for every model, and the tables don't need to be modified if one of your models changes - it has the following;

- table map (stores unique table names) - allows for super fast lookup instead of full text scan

- model map (stores unique model + app + db names) - allows for super fast lookup instead of full text scan

- fieldchange (stores fieldname+old/new value, target model+pk, and timestamp)

- event (stores event message, log level, target model+pk and timestamp)

Anyway, I'm rambling. If you would like to chat about this let me know. I will update this group, or contact you when I start working on the change audit code

Sure thing, if you think of any additional changes please feel free to fire them over (it'd be great to see others using this in the wild!)

 

Kurt Pruhs
Utah State University
Programing & Design Team

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/OQzlNQjqtbYJ.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[jondykeman]

Hello,

I am in a very similar situation. I would in an environment that deals with sensitive data collection. Everything has to be two-factor authenticated, in the secure server zone etc. As part of this we need logging of every action ever taken, by whom, when, and what the changes were.

At first I implemented my own custom solution which was less than ideal, but it worked. That was a model of creating a new record for each field any time there was a change to the value, but via a lot of manual checking code. 

I am not starting to migrate to a slicker solution. I am taking advantage of django-reversion.

https://github.com/etianen/django-reversion

This provides row level auditing out of the box, and then you just need to take advantage of the pre_commit_signal to track field changes as well.

@receiver(reversion.pre_revision_commit)

def it_worked(sender, **kwargs):

currentVersion = kwargs.pop('versions')[0].field_dict

pastVersion = reversion.get_for_object(kwargs.pop('instances')[0])[0].field_dict

changes = set(currentVersion.items()) - set(pastVersion.items())

changedVars = []

for var in changes:

changedVars.append(var[0])

comment = "Changed: %s" % ", ".join(changedVars)

revision = kwargs.pop('revision')

revision.comment = comment

revision.save()

kwargs['revision'] = revision

Rather than the string tracking which fields I am going to switch to a dict of changed or not for each variable.

I will check out cutemodel for sure and let you know.

Thanks,

JD

[Cal Leeming [Simplicity Media Ltd]]

Thanks for letting me know about django-reversion, it has made for interesting reading.

From what I can see there are two big differences between them;

* CuteModel is designed with performance/scalability in mind (as some of our projects are tipping into the 700+mil row count and rising)

* CuteModel is designed to be as simple and easy as possible - where as django-reversion left me feeling a bit confused.

Looking at django-reversion, it certainly looks close to cutemodel, but there are a few differences;

* Changes are serialized into the database, this adds a significant extra size and CPU overhead (Version.serialized_data)

* Object references are stored as a TextField, this is not good for performance (Version.object_id)

* Creates a new serialized object every time a row is added - this is really really not good for performance(*.VERSION_ADD)

* Requires an additional model for every model you have if you want to store custom meta data (cutemodel only requires 1)

* Uses signal rather than overriding the subclass - although this is probably a better approach - thoughts anyone??

* In some ways the API is quite nice, but in others it seems a bit clunky.

* Requires you to create initial revisions dump - again, not good if you have a lot of rows

That being said - there are definitely some good points from that app, and a lot of features that would be great for CuteModel, such as;

* Grouping together field changes into a 'revision'

* Better low level API support

* Ability to revert a change

I'm certainly going to add those into the todos list - thanks for your feedback!

Cal

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/ZZF9MuCal70J.

[Derek]

Cal

Great post; I think you summed up my feelings about django-reversion as well, although articulated extremely clearly.

If CuteModel (where does that name come from??) can address the issue of reverting a change to a record (or, even better, all changes made at one time to a record), then I think it will be a great alternative option for many folk.

Cheers,

Derek

[Cal Leeming [Simplicity Media Ltd]]

On Thu, Sep 13, 2012 at 9:42 AM, Derek <game...@gmail.com> wrote:

Cal

Great post; I think you summed up my feelings about django-reversion as well, although articulated extremely clearly.

If CuteModel (where does that name come from??)

Cute was the first word that came to my head, thought it sounded catchy

can address the issue of reverting a change to a record (or, even better, all changes made at one time to a record), then I think it will be a great alternative option for many folk.

Yeah after the last few posts, I realised there were a few structural changes that needed to be made - one of those being that field changes need to be grouped by a Revision() object, so you can revert the entire revision or just bits from it.

I also figured that the API needs to expose data to the original save() method call, such as the revision object, so they can integrate this into their existing application (for example, if they have 3 models that get saved in 1 form save, they might want to take all 3 IDs, then create an extra revision layer that lets them revert them all in one go - rather than try and normalize this, exposing the revision object would give the user enough control)

 

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/csvT01soOo8J.