A tiny producer-consumer framework
Ken Wesson
`(agent
(fn c# [~item]
~@body
c#)))
(defmacro defconsumer [name item & body]
`(def ~name (consumer ~item ~@body)))
(defn feed [consumer & values]
(doseq [v values] (send-off consumer apply [v])))
Nine lines of code.
user=> (defconsumer foo [x] (println x))
#'user/foo
user=> (feed foo 7)
#<Agent #<user$c__1481__auto__ user$c__1481__auto__@1b030d8>>
user=> (feed foo 42 196)
#<Agent #<user$c__1481__auto__ user$c__1481__auto__@1b030d8>>
(and appearing at System/out)
7
42
196
I used send-off because your typical producer/consumer queue context
involves I/O- rather than CPU-bound work. Since agent sends from the
same thread are processed in the order sent, the whole thing works as
expected with one producer thread; with multiple producer threads the
inputs from separate threads will get interleaved but those from any
single one will be in sequence.
Eric Schulte
A while back I implemented something similar; a propagator system using
agents fed with `send', coming in at a slightly more verbose ~35 lines
of code.
http://cs.unm.edu/~eschulte/research/propagator/
Cheers -- Eric
Ken Wesson
> Nice concise example,
Thanks.
> A while back I implemented something similar; a propagator system using
> agents fed with `send', coming in at a slightly more verbose ~35 lines
> of code.
>
> http://cs.unm.edu/~eschulte/research/propagator/
Interesting. But I have some questions ...
> (defn set-cell "Set the value of a cell" [cell value]
> (send cell (fn [_] value)))
Why (fn [_] value) instead of (constantly value)? OK, actually
(constantly value) is textually longer, but I'd argue it might be
clearer. And it works; (constantly foo) accepts all arities. It's
something like (fn [& _] foo).
A question for Clojure's developers: why does restart-agent require
the agent to be in a failed state? That seems to be a superfluous
requirement. Atoms have reset! and refs ref-set to clobber the old
value; agents would have something analogous if restart-agent worked
on agents that weren't failed too.
> (defmacro propagator "Define a new propagator."
> [name in-cells out-cells & body]
> `(do
> (defn ~(with-meta name
> (assoc (meta name)
> :in-cells in-cells :out-cells out-cells))
Whoa. Why not
(defmacro propagator "Define a new propagator."
[name in-cells out-cells & body]
`(do
(defn ~(vary-meta name assoc
:in-cells in-cells :out-cells out-cells)
?
> (doseq [cell# ~in-cells] (add-neighbor cell# ~name))
There's no forward-declaration of add-neighbor before this.
> ~name))
Usually a deffoo form returns the new var, not just a symbol. This can
be fixed with
(declare add-neighbor)
(defmacro propagator "Define a new propagator."
[name in-cells out-cells & body]
`(let [v# (defn ~(vary-meta name assoc
:in-cells in-cells :out-cells out-cells)
~in-cells ~@body)
(doseq [cell# ~in-cells] (add-neighbor cell# ~name))
v#))
which captures the return value of the defn and returns it after the doseq.
> (defmacro run-propagator
> "Run a propagator, first collect the most recent values from all
> cells associated with the propagator, then evaluate."
> [propagator]
> `(let [results# (apply ~propagator (map deref (:in-cells ^#'~propagator)))]
> (doseq [cell# (:out-cells ^#'~propagator)]
> (when (not (= @cell# results#))
> (send cell# (fn [_#] results#))))
> results#))
Why not use your already-defined set-cell function on that second-to-last-line?
Also your syntax for accessing the metadata seems a bit odd. Why not
(meta ~propagator)?
Good use of metadata, though.
> (defmacro add-neighbor "Add a neighbor to the given cell."
> [cell neighbor]
> `(add-watcher
> ~cell :send
> (agent nil :validator (fn [_#] (do (future (run-propagator ~neighbor)) true)))
> (fn [_# _#])))
What is "add-watcher"? I'm somewhat familiar with add-watch, whose
syntax would be somewhat different. There'd be nothing between the
watcher key :send and the (fn ...), and that function would take four
arguments.
I'd have probably used something like:
(defmacro add-neighbor "Add a neighbor to the given cell."
[cell neighbor]
`(add-watch
~cell :send
(fn [_# _# _# _#]
(future (run-propagator ~neighbor))))
You could also get rid of the need to forward-declare add-neighbor by
moving the propagator macro last.
Convention also is that you'd call cell defcell and propagator
defpropagator, since they expand to def forms.
Now some more general notes:
Having multiple out cells is interesting, but maybe redundant. Even
with only a single out cell, the same effect can be had if you have an
in cell, a bunch of out cells, and for each out cell a propagator that
just returns the in cell's value.
At that point the distinction between cells and propagators can go
away: every cell has a propagator that sets its own value if any of
its in-cells change. A cell that's intended to be an input can have an
empty set of in-cells, which will result in it never doing anything
since it will never be triggered.
Of course the effect of that is to turn it into an almost-spreadsheet.
Giving the cells a spatial organization is the last step to making it
a spreadsheet. :)
I hope you don't think all of that was too critical. It's very interesting work.
Eric Schulte
> On Sat, Jan 22, 2011 at 11:26 AM, Eric Schulte <schult...@gmail.com> wrote:
>> Nice concise example,
>
> Thanks.
>
>> A while back I implemented something similar; a propagator system using
>> agents fed with `send', coming in at a slightly more verbose ~35 lines
>> of code.
>>
>> http://cs.unm.edu/~eschulte/research/propagator/
>
> Interesting. But I have some questions ...
>
>> (defn set-cell "Set the value of a cell" [cell value]
>> (send cell (fn [_] value)))
>
> Why (fn [_] value) instead of (constantly value)? OK, actually
> (constantly value) is textually longer, but I'd argue it might be
> clearer. And it works; (constantly foo) accepts all arities. It's
> something like (fn [& _] foo).
>
I agree constantly would have been a better choice, had I known it
existed.
>
> A question for Clojure's developers: why does restart-agent require
> the agent to be in a failed state? That seems to be a superfluous
> requirement. Atoms have reset! and refs ref-set to clobber the old
> value; agents would have something analogous if restart-agent worked
> on agents that weren't failed too.
>
>> (defmacro propagator "Define a new propagator."
>> [name in-cells out-cells & body]
>> `(do
>> (defn ~(with-meta name
>> (assoc (meta name)
>> :in-cells in-cells :out-cells out-cells))
>
> Whoa. Why not
>
> (defmacro propagator "Define a new propagator."
> [name in-cells out-cells & body]
> `(do
> (defn ~(vary-meta name assoc
> :in-cells in-cells :out-cells out-cells)
>
> ?
>
Again, I work with the functions I am aware of. Thanks for these
pointers, they are very helpful especially as it can be difficult to
learn new functions once a sufficient subset has been discovered.
>
>> (doseq [cell# ~in-cells] (add-neighbor cell# ~name))
>
> There's no forward-declaration of add-neighbor before this.
>
>> ~name))
>
> Usually a deffoo form returns the new var, not just a symbol. This can
> be fixed with
>
> (declare add-neighbor)
>
> (defmacro propagator "Define a new propagator."
> [name in-cells out-cells & body]
> `(let [v# (defn ~(vary-meta name assoc
> :in-cells in-cells :out-cells out-cells)
> ~in-cells ~@body)
> (doseq [cell# ~in-cells] (add-neighbor cell# ~name))
> v#))
>
> which captures the return value of the defn and returns it after the doseq.
>
Agreed, that is an improvement.
>
>> (defmacro run-propagator
>> "Run a propagator, first collect the most recent values from all
>> cells associated with the propagator, then evaluate."
>> [propagator]
>> `(let [results# (apply ~propagator (map deref (:in-cells ^#'~propagator)))]
>> (doseq [cell# (:out-cells ^#'~propagator)]
>> (when (not (= @cell# results#))
>> (send cell# (fn [_#] results#))))
>> results#))
>
> Why not use your already-defined set-cell function on that second-to-last-line?
>
> Also your syntax for accessing the metadata seems a bit odd. Why not
> (meta ~propagator)?
>
Agreed (meta ~propagator) is much more clear.
>
> Good use of metadata, though.
>
>> (defmacro add-neighbor "Add a neighbor to the given cell."
>> [cell neighbor]
>> `(add-watcher
>> ~cell :send
>> (agent nil :validator (fn [_#] (do (future (run-propagator ~neighbor)) true)))
>> (fn [_# _#])))
>
> What is "add-watcher"? I'm somewhat familiar with add-watch, whose
> syntax would be somewhat different. There'd be nothing between the
> watcher key :send and the (fn ...), and that function would take four
> arguments.
>
This was an old project (last spring) and it's version of clojure
(org.clojure/clojure "1.1.0-alpha-SNAPSHOT") is not even resolved any
longer by lein. It appears that the add-watcher function has been
deprecated and replaced by add-watch, which doesn't require that the
function return true allowing the simpler construction you suggest
below.
>
> I'd have probably used something like:
>
> (defmacro add-neighbor "Add a neighbor to the given cell."
> [cell neighbor]
> `(add-watch
> ~cell :send
> (fn [_# _# _# _#]
> (future (run-propagator ~neighbor))))
>
> You could also get rid of the need to forward-declare add-neighbor by
> moving the propagator macro last.
>
> Convention also is that you'd call cell defcell and propagator
> defpropagator, since they expand to def forms.
>
Ah, this is a big oversight on my part. Thanks for mentioning.
I've implemented your suggestions, and committed the results to
http://gitweb.adaptive.cs.unm.edu/propagator.git
>
> Now some more general notes:
>
> Having multiple out cells is interesting, but maybe redundant. Even
> with only a single out cell, the same effect can be had if you have an
> in cell, a bunch of out cells, and for each out cell a propagator that
> just returns the in cell's value.
>
> At that point the distinction between cells and propagators can go
> away: every cell has a propagator that sets its own value if any of
> its in-cells change. A cell that's intended to be an input can have an
> empty set of in-cells, which will result in it never doing anything
> since it will never be triggered.
>
Interesting point. I think your suggestion would lead to simpler code,
but IMO the existing design is simpler conceptually, I like the divide
of propagators as functions and cells as data, rather than every
function having an associated piece of data (namely its output).
I'm not sure which implementation would lead to a more efficient running
system. Having fewer larger propagators which can set the values of
many cells at once, or more smaller propagators each of which only sets
the value for a single cell.
>
> Of course the effect of that is to turn it into an almost-spreadsheet.
> Giving the cells a spatial organization is the last step to making it
> a spreadsheet. :)
>
That would be a great application for this system. Each cell of the
spreadsheet could be a cell, and each formula could be a propagator.
I've implemented this and it seems to work well, I've committed this to
the repo above, and posted the spreadsheet code with example usage into
a gist at https://gist.github.com/792968
>
> I hope you don't think all of that was too critical. It's very interesting work.
Not at all, I appreciate the code review, your function suggestions and
convention reminders were very useful.
Many Thanks -- Eric
Ken Wesson
> Ken Wesson <kwes...@gmail.com> writes:
>> Why (fn [_] value) instead of (constantly value)? OK, actually
>> (constantly value) is textually longer, but I'd argue it might be
>> clearer. And it works; (constantly foo) accepts all arities. It's
>> something like (fn [& _] foo).
>
> I agree constantly would have been a better choice, had I known it
> existed.
Ah. I thought you might have known about it, but not that the function
it emitted would accept arities other than zero.
>> Whoa. Why not
>>
>> (defmacro propagator "Define a new propagator."
>> [name in-cells out-cells & body]
>> `(do
>> (defn ~(vary-meta name assoc
>> :in-cells in-cells :out-cells out-cells)
>>
>> ?
>>
>
> Again, I work with the functions I am aware of.
Ah. Funnily enough it wasn't long ago that I was "caught" actually
reimplementing vary-meta because I didn't know about it.
> Thanks for these pointers, they are very helpful especially as it can
> be difficult to learn new functions once a sufficient subset has been
> discovered.
True, that. Though I'd phrase it more as "difficult to discover new
functions". When you don't know how to do X or feel it should be a
one-liner, if not a single function call, in core, you go trawling
through http://clojure.github.com/clojure/clojure.core-api.html or use
find-doc and apropos at the REPL to try and find something that fits,
or at least simplifies the job. On the other hand when you can slap X
together in two seconds flat it may not occur to you to look. The more
functions you do know and the more adept you've become at combining
them in various ways, the less likely it is that you *won't* think of
at least one way to build X quickly, unless X is something
particularly big and complicated like a major new piece of GUI
functionality or an FTP server or something, and then you're more
likely to look for a third-party library (and, probably, a Java rather
than a Clojure library at that).
I find the cheatsheet at http://clojure.org/cheatsheet useful
sometimes for this, if I want to check if core has a function related
to a category of use (seqs, say, or atoms) that might be useful.
The cheatsheet has a couple of problems. For example, vary-meta isn't
in there. ;) In fact there's no section on metadata at all, just a few
scattered related things like *print-meta* and the reader macros
(which are out of date -- ^ for meta and #^ for with-meta rather than
just ^ for with-meta, nothing for meta). Also:
* deref/@ is listed under refs, instead of somewhere general or
repeated also for atoms and agents.
* delay is listed under laziness but force under other/misc instead
of with delay
* the special forms list is incomplete, missing at least . and set!.
The Java interop section lists set! but not . and the vars section
doesn't list set!.
Nonetheless, keeping those caveats in mind I find it useful.
>> What is "add-watcher"? I'm somewhat familiar with add-watch, whose
>> syntax would be somewhat different. There'd be nothing between the
>> watcher key :send and the (fn ...), and that function would take four
>> arguments.
>
> This was an old project (last spring) and it's version of clojure
> (org.clojure/clojure "1.1.0-alpha-SNAPSHOT") is not even resolved any
> longer by lein. It appears that the add-watcher function has been
> deprecated and replaced by add-watch, which doesn't require that the
> function return true allowing the simpler construction you suggest
> below.
What, it had combined validator and watch functions in one back then?
> Interesting point. I think your suggestion would lead to simpler code,
> but IMO the existing design is simpler conceptually, I like the divide
> of propagators as functions and cells as data, rather than every
> function having an associated piece of data (namely its output).
That's OK.
> I'm not sure which implementation would lead to a more efficient running
> system. Having fewer larger propagators which can set the values of
> many cells at once, or more smaller propagators each of which only sets
> the value for a single cell.
It's likely to depend on how many distinct cells (may sometimes not
all have the same value) you sometimes need to set to identical values
all at once. If you have few such, you don't lose much by having a
one-propagator-one-output-cell rule. If you have many such, you
probably do. (If you have multiple cells that never have distinct
values, they can be replaced by a single cell.)
> That would be a great application for this system. Each cell of the
> spreadsheet could be a cell, and each formula could be a propagator.
> I've implemented this and it seems to work well, I've committed this to
> the repo above, and posted the spreadsheet code with example usage into
> a gist at https://gist.github.com/792968
Cool.
I notice the page at the original URL you posted hasn't been updated though.
>> I hope you don't think all of that was too critical. It's very interesting work.
>
> Not at all, I appreciate the code review, your function suggestions and
> convention reminders were very useful.
>
> Many Thanks -- Eric
You're welcome.
Eric Schulte
>> spreadsheet could be a cell, and each formula could be a propagator.
>> I've implemented this and it seems to work well, I've committed this to
>> the repo above, and posted the spreadsheet code with example usage into
>> a gist at https://gist.github.com/792968
>
> Cool.
>
I think this example is just a couple of lines of Swing code away from
being a generally useful spreadsheet application.
>
> I notice the page at the original URL you posted hasn't been updated though.
>
I've moved everything (repo and documentation) over to repo.or.cz
http://repo.or.cz/w/propagator.git
Thanks again for the feedback -- Eric