Add signals for QuerySet bulk operations such as `delete`, `update, `bulk_create`

[Byron Ruth]

My use case is for regenerating aggregate data cache at a table level. Simply calling a single signal after a bulk operation is complete would enable invalidating such aggregate cache. There is not a very clean alternate solution to this problem unless using database triggers which calls an external script that invalidates the cache.

[Kääriäinen Anssi]

A somewhat different proposal is in ticket #17824: Add generic pre/post_modify signal. I think the generic
"object modified" signals would fit your use case very well.

The idea is that there would be just one signal which would be fired for any data modifying ORM operation.
The arguments for it:
- fired for every operation modifying data
- one signal to listen to for all data modifications
The likely counter-arguments:
- duplicates the existing signals
- the callbacks end up being a big "switch statement", and thus you end up separating save, delete etc
anyways.
- the API isn't good enough

From performance perspective there should be no big problems: the signal is given an iterable as
"objs_modified" argument. For .update() for example, where you don't want to fetch all the objects for
performance reasons, you could just pass qs.filter(update_filters) as the modified objects. This way
there would be no performance penalty, except if there is actual use of the signal.

I would like to see a generic pre/post modify signal, as I think it is much easier to use than using
the pre/post save/delete + m2m_changed signals. However, I do not feel strongly at all about this, just
something I would find useful. I believe having total control of all data modifying operations using Django
signals would be a welcome addition for many users.

- Anssi
________________________________________
From: django-d...@googlegroups.com [django-d...@googlegroups.com] On Behalf Of Byron Ruth [bjr...@gmail.com]
Sent: Sunday, March 25, 2012 17:46
To: django-d...@googlegroups.com
Subject: Add signals for QuerySet bulk operations such as `delete`, `update, `bulk_create`

My use case is for regenerating aggregate data cache at a table level. Simply calling a single signal after a bulk operation is complete would enable invalidating such aggregate cache. There is not a very clean alternate solution to this problem unless using database triggers which calls an external script that invalidates the cache.

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/DAaTRIau8h8J.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Byron Ruth]

Sure I guess you _could_ take a generic signals approach to handling all modify operations, but for each operation context is important. The state of the object(s) matters when performing operations. The `pre_save` and `post_save` signals for example have a `created` flag which allows for filtering out the new objects versus existing ones. Likewise, having a distinct pair of delete signals ensures you don't make the mistake of performing operations on objects that are now deleted.

Ultimately my request is for batch or bulk-level signals. Up until recently, the lack of API for bulk operations in the ORM has been a non-starter for doing any kind of ETL for large amounts of data (not that I ever expected Django to have suitable APIs or performance for ETL pipelines). But now using it's at least reasonable to argue to include in a pipeline if there would be a suitable set of bulk-level signals when doing batch inserts, updates and deletes.

On Sunday, March 25, 2012 6:23:46 PM UTC-4, Anssi Kääriäinen wrote:

A somewhat different proposal is in ticket #17824: Add generic pre/post_modify signal. I think the generic
"object modified" signals would fit your use case very well.

The idea is that there would be just one signal which would be fired for any data modifying ORM operation.
The arguments for it:
  - fired for every operation modifying data
  - one signal to listen to for all data modifications
The likely counter-arguments:
  - duplicates the existing signals
  - the callbacks end up being a big "switch statement", and thus you end up separating save, delete etc
anyways.
  - the API isn't good enough

From performance perspective there should be no big problems: the signal is given an iterable as
"objs_modified" argument. For .update() for example, where you don't want to fetch all the objects for
performance reasons, you could just pass qs.filter(update_filters) as the modified objects. This way
there would be no performance penalty, except if there is actual use of the signal.

I would like to see a generic pre/post modify signal, as I think it is much easier to use than using
the pre/post save/delete + m2m_changed signals. However, I do not feel strongly at all about this, just
something I would find useful. I believe having total control of all data modifying operations using Django
signals would be a welcome addition for many users.

 - Anssi
______________________________​__________
From: django-developers@​googlegroups.com [django-developers@​googlegroups.com] On Behalf Of Byron Ruth [bjr...@gmail.com]

Sent: Sunday, March 25, 2012 17:46

To: django-developers@​googlegroups.com

Subject: Add signals for QuerySet bulk operations such as `delete`, `update, `bulk_create`

My use case is for regenerating aggregate data cache at a table level. Simply calling a single signal after a bulk operation is complete would enable invalidating such aggregate cache. There is not a very clean alternate solution to this problem unless using database triggers which calls an external script that invalidates the cache.

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To view this discussion on the web visit https://groups.google.com/d/​msg/django-developers/-/​DAaTRIau8h8J.

To post to this group, send email to django-developers@​googlegroups.com.
To unsubscribe from this group, send email to django-developers+unsubscribe@​googlegroups.com.

[Anssi Kääriäinen]

On 03/26/2012 06:34 AM, Byron Ruth wrote:
> Sure I guess you _could_ take a generic signals approach to handling
> all modify operations, but for each operation context is important.
> The state of the object(s) matters when performing operations. The
> `pre_save` and `post_save` signals for example have a `created` flag
> which allows for filtering out the new objects versus existing ones.
> Likewise, having a distinct pair of delete signals ensures you don't
> make the mistake of performing operations on objects that are now deleted.

The pre/post_modify signal would have the context available. (parameters
action='save/delete/bulk_create/update' and action_args). It would be
kind of all the current signals compressed into one. If that is a good
design or not is a good question. I think it would be useful, but I
would not be surprised if other core developers have different opinion
of such a signal.

The reason the generic signal would be useful is that more often than
not you are interested in _all_ the data modifying operations done to a
model, not just in the save or delete or update subset. Hence, one mount
point for all the operations. What is done in the signal could be very
different for the .update() case than for the .save() case, but on the
other hand there are cases where the generic signal handler would reduce
the amount of work. For search engine reindexing (Haystack for example),
you would just do reindex(modified_objects) for all cases except delete.
Similar for cache invalidation. So, such a signal could simplify some
common operations.

Having signals for all data modifying operations is in my opinion
important. It is of course possible to achieve this without generic
signals by just adding pre/post update / bulk_create signals. The
bulk_create signal is problematic because you do not have the auto-pk
values available even in post_bulk_create signal. For PostgreSQL and
Oracle having the PKs would be possible using the "RETURNING ID" syntax.
SQLite could be hacked to support returning the IDs (there are no
concurrent transactions, hence you know what the IDs will be, or at
least I think this is the case). But for MySQL I do not know of any way
of returning the IDs. Except for locking the table for the duration of
the insert...

- Anssi

[Byron Ruth]

I guess this would not be too different from how the `m2m_changed` signal works. The arguments would (at a bare minimum) include the `sender`, `action`, and `using`, then depending on the `action` other arguments may be passed in.

I would imagine if this approach would be taken, migrating existing signals over would follow the normal deprecation policy. But it could be introduced for the bulk-level operations initially without breaking any compatibility.