Proposed change in ORM model save logic
Barry Johnson
Trac ticket #33191 was recently closed as a “wontfix”, but Carlton encouraged bringing the matter to this group. The actual issue is a small one, but it seems that Django could do better than it is doing today with one small change, and avoid the need for a counter-intuitive "x = x" statement added to application code.
The claim is that Django is unnecessarily clearing the cached entry for an in-memory related object at a time that cache does not need to be cleared, and that this cache clearing can result in unwanted lazy reads down the line. A one-line fix for the problem is suggested.
Many apologies in advance for the extreme length of this post. The requested change is small and subtle (although quite beneficial in some cases); we must dive deep into code to understand WHY it is important.
Background
———————
My team is developing a major retail point of sale application; it’s currently live in nearly 1,000 stores and we’re negotiating contracts that could bring it to another 10,000 stores across the US over the next three to five years before going world-wide. The backend is Django and PostgreSQL.
One of the typical problems we face has to do with importing data from existing systems. This data usually comes to us in large sets of flat ASCII files, perhaps a gigabyte or few at a time. We have to parse this incoming data and fill our tables as quickly as possible. It’s not usual for one such data import to create tens of millions of rows across a more than a hundred tables. Our processes to do this are working reasonably well.
In many cases, the incoming stream of data generates related records in several tables at a time. For example, our product import populates related data across a dozen tables organized into parent/child hierarchies that are up to four levels deep. The incoming data is grouped by product (not by functional data type); the rows and columns of the records for a single product will create ORM objects scattered across those dozen ORM models. The top-level model, Product, has four child models with references to Product; and each of those child models may have other child tables referring back to THOSE models, etc.
If the database schema sounds surprisingly complex, it is. Big box retail is more complex than most people realize. Each store typically carries 30,000 to 70,000 individual products; some are even larger. Some of the retail chains to which we market our systems have more than $1 billion USD per year in sales.
To be efficient, we process this incoming data in chunks: We may instantiate ORM objects representing, say, 5,000 products and all of their related child objects. For example, we’ll create a new instance of a Product model, then instantiate the various children that reference that product. So a very typical pattern is something like
product = Product(values)
pv = ProductVariant(product=product, **more_values)
upc = UPC(product_variant=product_variant, **upc_values)
It’s not that simple, of course; we’re reading in sequential data that generates multiple instances of the children in loops, etc., but essentially building a list of products and the multi-level hierarchy of child objects that dependent upon each of those products. We then use bulk_create() to create all of the top-level products in one operation, then bulk_create() the various lists of first-level children, then bulk_create the children of that second level, etc. LOTS of bulk creates.
Prior to Django 3.2, we had to explicitly set the associated “_id” field for each child’s reference to a parent after the list of parents were bulk_created. Using our examples above, we would see things like:
bulk_create(list_of_products)
for each variant in list_of_product_variants:
variant.product_id = variant.product.id
bulk_create(list_of_product_variants)
for each upc in list_of_upc_entries:
upc.product_variant_id = upc.product_variant.id
bulk_create(list_of_upc_entries)
[…]
Again, this is somewhat simplifying the code, but the key takeaway is that older versions of Django required us to manually pick up the primary key value from recently created instances and set the associated “_id” value in each instance pointing at the recently created objects.
As expected, setting the “_id” value of a foreign key field clears the internal cache entry containing the reference to the parent object. We would fully expect that if we said “variant.product_id = 10”, that “variant.product” would be considered “not yet loaded”, and a reference to variant.product would generate a lazy read. We don’t disagree with that interpretation or behavior, and that is the existing behavior that Carlton implicitly referenced when he closed the ticket in question as “wontfix”.
But…
Relatively Recent Django Changes
——————————————————
In the Django 3.2 timeframe, Ticket #29497 (“Saving parent object after setting on child leads to an unexpected data loss in bulk_create()”) has affected code in this area. Consider this sequence of code:
my_parent = Parent()
my_child = Child(parent=parent)
my_parent.save()
my_child.save()
(Yes, our real business case involves use of bulk_create() for thousands of objects at a time, but this simpler example using only .save() illustrates the problem in a stripped-down manner.)
What is the expected result in the database for the child’s foreign key to the parent? Well… We think most application programmers would expect the row for child object to contain a foreign key refererence to the parent object.
Instead, the pre-3.2 Django behavior was that the child would be saved with a null value in the foreign key reference to parent… Because, when we save the parent (and internally populate the parent’s primary key with a newly assigned ID), Django is unaware of the in-memory reference from child to parent, and cannot automatically set child_parent_id = child.parent.id, so it remains a null value.
If the foreign key from child to parent is required, the save of the child object will fail because of the null value, and the developer will quickly realize something is wrong. However, if the foreign key to the parent is NOT required, then the child WILL be quietly saved to the database with a null foreign key to the parent... and an unwary developer will wind up with unexpected null values.
This was considered an unintentional loss of data and reported on ticket #29497.
The solution to this problem was clever: It recognizes that this is a case where my_child.parent_id and my_child.parent.pk are out of sync with one another (something we normally never see in Django), and that this particular case can be detected and solved. In particular, the logic implemented as part of the ticket effectively changes the internal priority of these two related values.
Prior to 3.2, the code to build the SQL INSERT statement for saving the child would always use the “parent_id” attribute regardless of whether “parent” was an ORM instance or not. The new 3.2 code does a quick initial test: If the “parent” attribute contains an ORM object with a non-null primary key AND if “parent_id” does not have a value, then set parent_id = parent.pk and proceed with the remainder of the save.
In other words, if we don’t have a value in “parent_id” but we DO have a value in “parent.pk”, then fix parent_id before we move on to build the SQL. Nice!
But…
The change we are requesting
——————————————————
For those that want to see the code in this area, check out the method _prepare_related_fields_for_save() in db.models.base.py. The
specific line of code in question is:
setattr(self, field.attname, obj.pk)
It uses an internal “setter” to set “parent_id” to the primary key of the referenced object.
Note this point carefully: The value of “obj” is the referenced instance of the parent in question. Just a few lines earlier
in the code, “obj” is set with this statement:
obj = getattr(self, field.name, None)
When we chase down this getter, it is returning the value in self._state.fields_cache[fieldname]. In other words, it’s the actual instance of the parent referenced in memory from the child. The application-level code has clearly and intentionally made this reference from child to parent, and the fact that we saved the parent to the database doesn’t (well, shouldn’t) affect that relationship. It certainly would NOT affect the relationship if the parent was updated instead of inserted, and we would expect consistent behavior from .save() regardless of insert vs. update.
So, back to the narrow problem. This line of code:
setattr(self, field.attname, obj.pk)
causes the internal cached reference from parent to child to be cleared!
Well, yes, of course it does. This setattr() operation is effectively doing the same thing as
child.parent_id = 10
which we’ve already said OUGHT to invalidate an existing reference to “child.parent”.
But here’s the crux of the argument: The code within _prepare_related_fields_for_save() ISN’T application-level code clearly altering the value of an ID field. Instead, it’s internal Django code attempting to reconcile a mismatch between the cached ID of the parent (child.parent_id) and primary key value of the in-memory reference to a model instance (child.parent.pk). And most importantly, it’s reconciling that difference by believing that the referenced instance is more accurate and should be preserved. Yet the very act of preserving that referenced instance’s primary key is removing the reference to the very instance that is considered more accurate.
Wait a moment — to preserve a pointer to the referenced instance, we’re going to pluck one tiny value out of it AND THEN throw the referenced instance away! To those of my generation, this reminds us of a 50+ year old infamous quotation, “To save the village we had to destroy it.”
Why does it matter?
————————————
If the sole use of the parent and child objects in this example was to get data saved to the database, it really wouldn’t matter.
However, in our specific use case, we have to record certain types of operations in an internal log. (It’s an “inventory transaction register”; whenever we change something in inventory that has an effect on the value of inventory, we must log that change). That logging logic is actually relative to ORM instances that are two levels down from the top in our hierarchy of parent/child objects being bulk created. We have a method to which we pass each of these not-top-level objects, and the log entries includes pieces of data from those objects’ parent and grandparent objects.
When we access the parent object through the child we’re logging, we wind up causing lazy reads of the objects we just saved to the database (and whom we still have in memory, and whom we THOUGHT were still referenced from our child objects). In our case, we actually get 2 lazy reads for each store-specific product being imported; 50,000 items being imported for a four-store chain causes an extra 400,000 unnecessary SQL SELECT statements. Removing these potential lazy reads is the performance improvement we’d like to share with the world.
And perhaps just as importantly, it avoids differences in behavior caused when a referenced object is updated vs. inserted into the database during a save. When it is updated, there is no need to alter the primary key, so the referenced object remains referenced. But if the referenced object is inserted, the internal code that updates the primary key for that instance removes the reference. We normally don’t like these types of behavioral differences in Django – we want .save() to behave the same way regardless of inserting vs. updating.
There is a workaround
——————————————————
There is a fairly simple workaround, and it harkens back to the days of pre-3.2 Django.
my_parent = Parent()
my_child = Child(parent=parent)
my_parent.save()
my_child.parent = my_child.parent # Add this line to work around the problem.
my_child.save()
Yes, we can do this. This rather unintuitive statement (that appears to be useless) obtains the internal cached reference to the parent object, then set both child.parent_id and the internal cached reference back to that instance. (The code in the setter first sets child.parent_id (which clears the cached entry) then inserts the originally referenced parent back into the internal cache.)
This workaround greatly resembles the statement from days of old: (“child.parent_id = child.parent.id”), but has the additional benefit of keeping our in-memory reference to parent (and to all of the objects that parent itself might be referencing as well), and avoiding lazy reads in the future.
Our suggested change
——————————————————
After this very long-winded explanation, we are proposing a small change that makes the workaround unnecessary while still avoiding the potential performance hit from lazy queries.
Let’s look back to our friend _prepare_related_fields_for_save().
We argue there is a STRONG belief that preparing an object for saving to the database should not substantially alter the content of that object. Yet this method does not adhere to that belief: While the results saved to the database are correct, it has altered the in-memory object by discarding the internal reference to a parent object that the application level code has carefully set.
In general, we like what _prepare_related_fields_for_save() is doing — we like the fact that it’s reconciling the difference between the copy of the referenced object’s primary key (child.parent_id) and the in-memory reference (child.parent.pk). We agree that this method is doing the right thing by using the primary key value from the referenced object. We simply do not think that the parent instance should be removed from the internal cache as part of this change.
Instead of resolving this difference in keys by executing:
setattr(self, field.attname, obj.pk)
We believe it should instead:
setattr(self, field.name, obj)
This would effectively replace the pseudo-code
child.parent_id = child.parent.pk
With
child.parent = child.parent
It replaces the value of child.parent_id with child.parent.pk, but without the side effect of discarding the in-memory relation of the child and parent instances.
Finally, there is a small performance improvement that could be made to this suggestion.
If we look at what the current
setattr(self, field.attname, obj.pk)
actually does, we have to look at the __set__() method of class ForeignKeyDeferredAttribute in django.db.models.fields.related_descriptors.py.
The code in this method is:
def __set__(self, instance, value):
if instance.__dict__.get(self.field.attname) != value and self.field.is_cached(instance):
self.field.delete_cached_value(instance)
instance.__dict__[self.field.attname] = value
It does what we expect: If the “parent_id” value is changing and there is a cached in-memory reference, then clear the cached entry before setting the integer value.
But note that the value of the “parent_id” attribute is really stored in the dictionary instance.__dict__[self.field.attname]
Our proposed change back in _prepare_related_fields_for_save() has the goal of setting the value of the integer “parent_id” while preserving the cached in-memory reference to the parent.
Instead of doing
setattr(self, field.name, obj)
(Which accomplishes the goal by essentially clearing the cache, then restoring it)
We could just do this:
self.__dict__[self.field.attname] = obj.pk
It is a slight denormalization of the logic that manages the value of the deferred foreign key attribute, but it has a tiny performance improvement by not removing, then replacing a dictionary entry.
What are WE doing about this problem?
—————————————————————
We’re going to be staying with Django 3.2 until the next long-term support release is available, so this change has no immediacy for our particular application. However, we think it is a nice extension that removes a non-obvious behavior in Django 3.2 (the .save() method can sometimes discard in-memory references to other ORM objects) and seems like it enhances the magic of the ORM framework.
In the meantime, we’ll be plugging non-intuitive lines of code that look like
child.parent = child.parent
into about fifty locations throughout our app, as well as training a team of 20 server-side developers to watch out for this edge case in any future code they write. It’s all doable, and we’ll do it… but we really like Django because it does “automagically” handle so many little database oddities and nuances. Over time, we’d like to get rid of “surprise” code like these lines that an inexperienced developer might remove because they look like they do nothing.
IF YOU’VE REACHED THE END
—————————————————
Thank you! I’m amazed you stuck through it all.
What do you think?
Ian Foote
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/596D3E68-4A37-423D-AEDF-64D2A5658DE8%40epicor.com.
Barry Johnson
Agreed! Not being sure just how widely spread accessing the underlying __dict__ is throughout the code, I wasn't sure which way to go. Although it may be a -measurable- improvement in performance, it's miniscule.
Aymeric Augustin
On 15 Oct 2021, at 08:49, Barry Johnson <bajo...@epicor.com> wrote:
Instead of resolving this difference in keys by executing:
setattr(self, field.attname, obj.pk)
We believe it should instead:
setattr(self, field.name, obj)
Aymeric.
Carlton Gibson
Adam Johnson
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/59a3158f-2552-4f33-9db3-8c5711dd6a91n%40googlegroups.com.