Agent watchers on Refs

[Stuart Sierra]

Hi Rich, all,

Ever since the new implementation of watchers I've been itching to try
out Cells again. It worked great for agents. I ran into a snag,
though, when I tried to make cells out of refs. Here's an example.

I make two refs:

(def r1 (ref 1))
(def r2 (ref (+ 10 @r1)))

Add a watcher so that r2 gets updated whenever r1 changes:

(add-watcher r1 :send (agent nil) (fn [a r] (dosync (ref-set r2 (+
10 @r)))))

Seems to work:

(println @r1 @r2)
;;=> 1 11
(dosync (ref-set r1 2))
(println @r1 @r2)
;;=> 2 12

Unless I go too fast:

(do (dosync (ref-set r1 3)) (println @r1 @r2))
;;=> 3 12

And I have to wait for the watcher to catch up:

(println @r1 @r2)
;;=> 3 13

I guess I was hoping that the watcher on a ref would be called
synchronously in the same transaction that modified the ref. I'm not
sure if that's possible, or even desirable. I found a way around, but
I had to abandon watchers in the process.

So I suppose my question is, do watchers have to be agents?

-Stuart Sierra

[Rich Hickey]

Were watchers synchronous, they would have to run post-transaction
(else a watcher action failure could cause a transaction rollback,
leaving already notified watchers confused). Being post-transaction
would mean that the refs could have been changed by another
transaction in the interim.

So, to answer your question, no, watchers don't have to be agents, and
while there would be some tighter guarantees were they not, there's no
getting around the essential asynchrony of a multi-threaded system.

I understand that makes it harder to copy synchronous single-threaded
Cells to Clojure, but I wonder if that is the right approach. I guess
what I am hoping for is for someone to re-imagine cells in a Clojure-
like way, i.e. not a stop-the-world model.

Chouser had an interesting use-case (with watchers on agents) where
async notification made it difficult to determine when a generative
process completed.

So, I could go back to watchers are functions, and have a set
deterministic time when they would run. For refs that would be post-
transaction. People that wanted async notification would have to send
to agents manually.

Anyone else trying out the watcher system? Now's the time to chime in
with feedback.

Thanks,

Rich

[Tom Ayerst]

I used a couple of watchers to push model updates onto the EDT in a Swing app; it 'just worked'.  I haven't pushed the envelope on it but  the abstraction is clear and clean IMHO.

Tom

2009/1/26 Rich Hickey <richh...@gmail.com>

[MikeM]

> Were watchers synchronous, they would have to run post-transaction
> (else a watcher action failure could cause a transaction rollback,
> leaving already notified watchers confused). Being post-transaction
> would mean that the refs could have been changed by another
> transaction in the interim.

I've been looking at the source to make sure I understand the trade-
off you're describing. In LockingTransaction, ref.notifyWatches is
called immediately after ref.validate for each ref. So it seems that a
watcher could be triggered with a value that then gets rolled back if
a subsequent validator throws. So even with the current watcher
implementation, a watcher could be called with a value that is not
consistent with the actual value of the ref when the transaction is
done. This doesn't seem much different from the scenario you're
describing. Is this correct?

>
> So, to answer your question, no, watchers don't have to be agents, and
> while there would be some tighter guarantees were they not, there's no
> getting around the essential asynchrony of a multi-threaded system.

After looking at how validators and watchers are invoked, I'm thinking
the current implementation already provides both synchronous and
asynchronous notification - synch notification via a validator, and
asynch notification via a watcher agent. Of course, the validator
isn't intended to be used as a watcher, but I wonder if you could
generalize the notion of validator so that this is acceptable usage.
The value provided to the validator may not represent the actual value
post-transaction (due to roll-back), but this is the cost of getting
synchronous notification. The watcher notification could be moved to
post-transaction, with the possibility that another transaction could
occur before the watcher runs, but this is the cost of asynch
notification.

[Tom Ayerst]

I thought the validator just set "Agent has errors" and you have to check it explicitly.

Tom

2009/1/27 MikeM <michael.m...@invista.com>

[MikeM]

On Jan 27, 10:08 am, Tom Ayerst <tom.aye...@gmail.com> wrote:
> I thought the validator just set "Agent has errors" and you have to check it
> explicitly.

Vars, refs and agents can have validators. A validator on a ref can
prevent the ref from taking on a new value by throwing an exception,
which will cause a roll-back of the transaction.

[Rich Hickey]

On Jan 27, 8:49 am, MikeM <michael.messini...@invista.com> wrote:
> > Were watchers synchronous, they would have to run post-transaction
> > (else a watcher action failure could cause a transaction rollback,
> > leaving already notified watchers confused). Being post-transaction
> > would mean that the refs could have been changed by another
> > transaction in the interim.
>
> I've been looking at the source to make sure I understand the trade-
> off you're describing. In LockingTransaction, ref.notifyWatches is
> called immediately after ref.validate for each ref. So it seems that a
> watcher could be triggered with a value that then gets rolled back if
> a subsequent validator throws. So even with the current watcher
> implementation, a watcher could be called with a value that is not
> consistent with the actual value of the ref when the transaction is
> done. This doesn't seem much different from the scenario you're
> describing. Is this correct?
>
>

No, because the watchers currently use agent sends, which are held
until post-transaction.

>
> > So, to answer your question, no, watchers don't have to be agents, and
> > while there would be some tighter guarantees were they not, there's no
> > getting around the essential asynchrony of a multi-threaded system.
>
> After looking at how validators and watchers are invoked, I'm thinking
> the current implementation already provides both synchronous and
> asynchronous notification - synch notification via a validator, and
> asynch notification via a watcher agent. Of course, the validator
> isn't intended to be used as a watcher, but I wonder if you could
> generalize the notion of validator so that this is acceptable usage.

I don't think so. Use of validators for anything other than purely-
functional validation is not going to be supported.

Rich