Deleting entry with associations and callbacks

[Michele Balistreri]

Hello,

I have a model with associations, where the associated fields are dependent on the given entry (quite a common case). If I delete an entry of said model, I want to cascade this to all dependencies. This is of course easily done at the DB-level with "ON DELETE CASCADE" on the foreign key constraint.

My problem now is, I need callbacks to be executed on deletion to remove associated binary files stored outside the database also for the associated entities. How can Ecto handle this case? I currently delete this way:

from(r in MyModel, where: r.id == ^my_id) |> Repo.delete_all

since I receive the id from the outside world, and I do not want to fetch a row to delete it. This does not trigger any callback at all, even on the main model (but this is documented, so no bug here)

Postgres would offer a LISTEN/NOTIFY mechanism which would make the entire callback implementation more reliable (because the DB would be triggering them, even if the change happens, for example, using a database administration interface) but could be quite db-specific if other databases do not support that. Can/should I access this mechanism directly, by taking hold of the connection pool myself or is there any (planned) facility for this?

Regards,

Michele

[José Valim]

Michele,

> since I receive the id from the outside world, and I do not want to fetch a row to delete it. This does not trigger any callback at all, even on the main model (but this is documented, so no bug here)

Without fetching the data, I am not sure how Ecto could solve this. If we were to invoke callbacks in those, we would need to fetch the data, so why not have yourself fetching the data explicitly?

Right now we haven't planned such functionality for Ecto but we could support cascade deletes in the future. However, that would still require fetching the data if you want callbacks to be invoked.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "elixir-ecto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-ecto...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michele Balistreri]

Hello José,

well, I do not fetch the data because there is no support for cascading anyway, so the callback would be no use for now. Fetching the data is not a problem per-se, I can do it.

A question that remains open, is if the LISTEN/NOTIFY callback mechanism of Postgres (http://www.postgresql.org/docs/6.4/static/sql-listen.htm) can be taken advantage of at the framework level. Postgrex already supports this, but Ecto does not expose this functionality, and I am not sure how I could bypass Ecto in this case and especially I am not sure if this is a good idea at all (bypassing Ecto... using LISTEN/NOTIFY is quite a good idea for callbacks imho).

Regards,

Michele

[José Valim]

well, I do not fetch the data because there is no support for cascading anyway, so the callback would be no use for now. Fetching the data is not a problem per-se, I can do it.

Ah, right. So what I meant is for you to fetch the data and explicitly delete it. Then, in the parent model, you could fetch and delete the children on before_delete, which would then delete the binary data.

 

A question that remains open, is if the LISTEN/NOTIFY callback mechanism of Postgres (http://www.postgresql.org/docs/6.4/static/sql-listen.htm) can be taken advantage of at the framework level. Postgrex already supports this, but Ecto does not expose this functionality, and I am not sure how I could bypass Ecto in this case and especially I am not sure if this is a good idea at all (bypassing Ecto... using LISTEN/NOTIFY is quite a good idea for callbacks imho).

Well, you can always use the adapter directly in Ecto, so maybe it is worth giving LISTEN/NOTIFY a try? We have no plans to include it in Ecto but someone has to try it before we decide if it is a good idea or not. :)

[Michele Balistreri]

Ok I understand now what you meant, I can implement cascading using callbacks myself more of less. It is a good point actually.

However I will first try the LISTEN/NOTIFY mechanism. I hope to get this working in the next days. If you are interested about the outcome of the experiment to decide if this should be integrated in Ecto, then I will keep you updated.

[José Valim]

Please keep us updated! :)

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Jason S]

Very interested in the LISTEN/NOTIFY with ecto as well :)

[Michele Balistreri]

Hello all,

so, I have done my research and the first successful experiments (mostly in IEx). I will develop a functional library but I'd like some inputs from you guys. First, here are my findings:

1) To use LISTEN/NOTIFY, you need to be sure there is a connection where the LISTEN commands have been issued. This is important, because LISTENs are per-connection. My approach here would be to have a dedicated process keeping this one single connection alive and restarting it as needed, reissuing the LISTEN commands on each reconnect. This would happens outside and independent from the connection pool used for querying.

2) NOTIFY, obviously, is issued by the database. This means triggers in the database would need to be created manually, possibly as part of a migration. NOTIFY has, other than the channel name, also a payload. Its content would be just forwarded. I think most of the times this would be an ID though.

My idea for this would be to have the following structure (more or less):

Supervisor (one_for_one)

 * -- Agent storing channel/callbacks tuples

 * -- Supervisor (all_for_one)

      * -- Postgrex.Connection

      * -- Process sending LISTEN commands and waiting for notifications and starting tasks to invoke callbacks

The process that sends LISTEN commands must also be the one waiting because Postgrex sends the notification to that process. This means adding/remove callbacks dynamically would require supporting some other messages as well in that process.

My doubts are about the API. 

1) How should I store callbacks? Just funs or {module, function_name} couples? Or both?

2) Should the API be declarative (with macros you could invoke when defining a model for example) or dynamic (add_handler/remove_handler, etc). When all listeners and callbacks are defined at compile time, maybe I can even avoid having an Agent since I could store channel/callbacks as module attributes. In that case only the {module, function_name} form for callbacks would be possible I think. Correct me if I am wrong.

I tend for a declarative API, since I do not see it particularly useful to add/remove callbacks dynamically. Triggers in the database are usually not created dynamically either.

Another question is: I can publish this a project totally independent from Ecto. The only dependency would really be Postgrex (other DBMS could be supported as well, but MySQL, notably, would require polling since it does not have NOTIFY). José, do you think it makes sense to add this to Ecto instead (a sort of Ecto.Live or Ecto.Events module)?

Regards,

Michele

[José Valim]

My doubts are about the API. 

1) How should I store callbacks? Just funs or {module, function_name} couples? Or both?

APIs are typically anonymous_fun or {module, function_name, args}. You would usually prepend or append your own argument to args.

2) Should the API be declarative (with macros you could invoke when defining a model for example) or dynamic (add_handler/remove_handler, etc). When all listeners and callbacks are defined at compile time, maybe I can even avoid having an Agent since I could store channel/callbacks as module attributes. In that case only the {module, function_name} form for callbacks would be possible I think. Correct me if I am wrong.

I can't give much advice here besides to have a explicitly separation between how it works and how the user defines them. For example, if the callbacks are static, you still need to retrieve those callbacks and start something with those callbacks at some point. Make this inner part work first and you will have solid foundation and likely good tests and then worry about the declarative part later, as it will likely be just "sugar".

It is also note worthing that are things like code reloading in Elixir, so even if you go with module declarations, what happens when the module is reloaded?

One final note: your design looks good! You should consider in having the listener processes already executing the callbacks instead of spawning tasks since the connection already does the multiplexing. This could allow the user to organize the callbacks explicitly, for example, those 3 callbacks in this listener process, this other 2 in that one, etc.

José, do you think it makes sense to add this to Ecto instead (a sort of Ecto.Live or Ecto.Events module)?

Not right now, thanks. :) We have other priorities that I want to get into Ecto before we worry about any notify/listen layer. But eventually we want to add a Phoenix.PubSub that relies on Postgres and this may be just the thing. :)

[José Valim]

Excellent!

The most serious problem with the project at the moment is that there are no automatic tests. The reason for this, is that I should mock a connection and probably give up using functions from the Postgrex.Connection module directly to achieve this. I will see what I can do. I have done quite a lot of manual tests however, so I can say it works.

You should definitely have some integration tests too, they have helped up a lot in Ecto.