Intermittent IntegrityError on Model Save with auto_now and auto_now_add Fields

[William Palin]

Hello Django community,

We are reaching out after encountering a persistent and elusive issue that manifests as an IntegrityError during the save operation of a Django model. This problem has been sporadically occurring since last year and has successfully stumped four seasoned Django developers on our team. The error seems to involve the non-update of auto_now fields upon model save, leading us to suspect a deeper issue within Django, though we are cautious about drawing premature conclusions. Given the complex and intermittent nature of this bug, we are seeking insights, advice, or any form of guidance from the community.

Issue Overview:

Our model inherits date_created and date_modified fields from an abstract base class designed to standardize these timestamps across our models. Here is how the abstract base class is defined:

class AbstractDateTimeModel(models.Model):

    """An abstract base class for most models"""

    

    date_created = models.DateTimeField(

        help_text="The moment when the item was created.",

        auto_now_add=True,

        db_index=True,

    )

    date_modified = models.DateTimeField(

        help_text="The last moment when the item was modified. A value in year"

                  " 1750 indicates the value is unknown",

        auto_now=True,

        db_index=True,

    )

    class Meta:

        abstract = True

The Problem:

Intermittently, the .save() method triggers an IntegrityError, apparently because the auto_now field (date_modified) does not get created. A specific instance of this error showed that while date_created is correctly populated, date_modified remains null, which directly leads to the IntegrityError upon attempting to insert the record:

[datetime.datetime(2023, 10, 2, 14, 33, 49, 99833, tzinfo=datetime.timezone.utc), None, ...]

'INSERT INTO "search_docket" ("date_created", "date_modified",... ]

What We've Tried:

• Investigated the possibility of an issue with update_fields being non-null and excluding date_modified during .save(), but confirmed through Sentry logs that update_fields was indeed None in all instances of the error.
• Attempted to reproduce the issue in a controlled environment without success, leaving us without a clear direction for a solution.

Request for Help:

We're wondering if this could point to an undocumented edge case in Django's auto_now and auto_now_add implementation or a specific database behavior under certain conditions. Any advice on further debugging steps, experiences with similar issues, or knowledge of potential Django nuances that we might not be considering would be incredibly valuable.

We appreciate your time and any feedback or suggestions you can offer.

Thanks

Bill 

django v5.0.2 

python 3.12.2

db is postgres

[Matthew Pava]

Hi Bill,

We ended up using a package called Django-audit-log: https://pypi.org/project/django-audit-log/.

It’s outdated now, but we used the source code for their CreatingUserField, LastUserField, CreationDateTimeField, and ModificationDateTimeField. More modern packages may have enhanced features than these.

 

In your case, I may just add a save method to the abstract model and add timezone.now() to the corresponding fields. If there’s not a pk, then populate the created date field.

 

You are right, though. Perhaps the Django community can explain the issue between auto_now and auto_now_add fields.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a788c065-3a96-472c-9b41-7c2aebca6967n%40googlegroups.com.

[Michael Lissner]

Looks like this issue isn't affecting lots of folks, since nobody is piping up, so I'll just add that if anybody arrives here in the future, we'll be tracking this in a public issue here: 

https://github.com/freelawproject/courtlistener/issues/3359

My theory is it has something to do with race conditions/timings and async, since I think it started around the time we started switching to async, but others on the team tell me that's a red herring (it's not the cause).

Anyhow, if anybody ever arrives here in the future, we fixed it by overriding the save method to catch IntegrityErrors. It's pretty lame to have code like this in our save method, but it seems to be working:

try:
    # Without a transaction wrapper, a failure will invalidate outer transactions
    with transaction.atomic():
        super(Docket, self).save(
            update_fields=update_fields, *args, **kwargs
        )
except IntegrityError:
    # Temporary patch while we solve #3359
    # If the error is not related to `date_modified` it will raise again
    self.date_modified = timezone.now()
    super(Docket, self).save(
        update_fields=update_fields, *args, **kwargs
    )

Anyhow, thanks for reading, everybody.