core.async: communicating termination

[vemv]

Consider a happy, oblivious producer of values:

(def c (chan 42))

(go (while true (>! c (rand))))

It doesn't know where/now the channel is consumed (which part of the point of channels/queues). However, we do know that at some point, nobody will need the result of our producing, so we should stop our activity.

It seems to me that a natural way to tell the producer to stop is to close the channel. However:

* there's nothing like a clojure.core.async/chan-closed? fn, AFAICT
* The >! fn unconditionally returns nil, whether the channel is open or not. Only the blocking nature of the call can potentially vary - which is not particularly useful for the purpose.

What would be an adequate way to indicate termination? Just telling the producer to stop (e.g. (reset! keep-producing false)) would break the indirection one gains by using channels.

What if >! returned true/false depending on whether the value was put (because the channel was open)?

[Alex Miller]

I haven't been in any discussions about this with the team, so it's entirely possible I am ignorant of some established future direction... but

- "closed?" seems useful to me. (It may also be useful to ask "drained?" (closed + empty).)

- >! returning an indicator of success also seems useful. However, since it executes asynchronously, I suspect adding that functionality may not be feasible (or may be feasible but incur a synchronization or complexity cost that is unacceptable). 

If you'd like to file a ticket at http://dev.clojure.org/jira/browse/ASYNC that would help to track the request. 

With respect to other strategies for dealing with this:

1) You could simply ignore the producer. If no one is requesting new values, the producer will not be run and no thread time should be consumed (there is of course heap being used). Not really recommending this but it's an option!

2) You could create a control channel for the producer (and other interested processes) that could transmit a stop message. Then alts! on the control channel in combination with the >! so that you're waiting on either the need to stop or the ability to put a new value in the channel.

Alex

[Michał Marczyk]

I think you could pass the producer a higher-order (chan 1) with the
actual communication channel placed in the buffer.

Then at each step the producer would do

(when-let [c (<! communication-channel)]
(>! c (rand))
(>! communication-channel c))

Once you close communication-channel, the put succeeds, but then the
take on communication-channel returns nil and the producer stops.

Cheers,
M.

> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Timothy Baldridge]

It's interesting to note that blocking go blocks can actually be GC'd. For example, run this in a repl:

(loop [] (let [c (chan)] 

         (go (>! c 42))) 

      (recur))

On my box the CPU and memory spikes, but after I CTRL+C and kill the loop, the GC kicks in and collects all the channels and gos

When go's are parked, they are put into a queue on the channel, thus when the channel is unreachable (besides inside the go block) both are collected, and the thread of execution is effectively terminated. 



Timothy

--
“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Michał Marczyk]

Hi Timothy,

That's very cool and very good to know!

Cheers,
M.

[Michał Marczyk]

As for the producer/consumer scheme with the higher-order channel, it
works in the single-producer case:

https://gist.github.com/michalmarczyk/5980097

(Actually this is simplified a bit in that it's also single consumer,
but that's not a feature of the approach, but rather my quick & dirty
poc.)

Multi-producer, single-consumer could be done dually.

[Brandon Bloom]

Wouldn't closed and drained predicates introduce race conditions?

(Sorry for brevity, on my phone)

[Michał Marczyk]

On 12 July 2013 01:08, Michał Marczyk <michal....@gmail.com> wrote:
> (Actually this is simplified a bit in that it's also single consumer,
> but that's not a feature of the approach, but rather my quick & dirty
> poc.)

Single consumer, multiple producers:

https://gist.github.com/michalmarczyk/5980700

[vemv]

If one used those preds to try putting values on the channels one is asking about, then yes, that would generate a classic nasty check-then-act scenario:

(when-not (closed? c) (>! c 42)) ;; Value could be never put, in face of interleavings

Programmers experienced in concurrency should have developed a sensibility against that kind of code anyway.

However, if what one is trying is to stop putting values on the channel, I see no possible race conditions.

[Brandon Bloom]

> However, if what one is trying is to stop putting values on the channel, I see no possible race conditions.

Querying the state of a channel at worst leads to race conditions and at best leads to bad design.

You're only supposed to close a channel from the producer side. So if you're the only writer, then you know if you've closed the channel or not. If there are multiple writers, then need to be coordinated in some way. Typically, they would alt! against reading from a control channel and writing to the output channel. When you get a shutdown signal from the control channel, you stop writing.

[Víctor M. Valenzuela]

Querying the state of a channel at worst leads to race conditions and at best leads to bad design.

The open/closed state of a channel is simple to reason about (it only changes, if ever, once), unlike the rest of state that is associated to a given channel...

You're only supposed to close a channel from the producer side.

Why?

--

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/_KzEoq0XcHQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Brandon Bloom]

>> You're only supposed to close a channel from the producer side.

> Why?

First, an appeal to authority: http://golang.org/pkg/builtin/#close

Notice the type of the argument: chan<-

That's a write-only port. Go's type system allows automatic coercion from read/write ports to constrained read or write only ports. However, you can't coerce a constrained port back to an unconstrained one.

See also the discussion here: https://groups.google.com/d/topic/golang-nuts/pZwdYRGxCIk/discussion

There are a few points discussed there, but in summary: It is a programming error to write a message to a closed channel. "close" is not a resource cleanup operation, it is a control signal. It "flows" in the same direction as the messages sent on the channel itself. If the receiver were allowed to close the channel, then the sender would have no way of avoiding a closed/write race condition.

This brings up another issue: Currently writing to a closed channel is a no-op, but it probably should throw an exception. Similarly, closing a closed channel is a no-op, but also probably should throw an exception. Both are things that a well behaved sender should never do, since they know when they close, so they know not to keep putting stuff in there. Or, they are warned of the impending close by some coordination process & the same rules apply.

[mstump]

This brings up another issue: Currently writing to a closed channel is a no-op, but it probably should throw an exception. Similarly, closing a closed channel is a no-op, but also probably should throw an exception. Both are things that a well behaved sender should never do, since they know when they close, so they know not to keep putting stuff in there. Or, they are warned of the impending close by some coordination process & the same rules apply.

Sorry to resurrect an old thread, but why not return true on successful put, false otherwise? That would allow for the following:

   (loop []

     (let [results (get-metrics metrics)]

       (when (async/>! mchan results)

         (recur))))

You could terminate blocks simply by closing the channel. You also avoid the messy higher order (quit) channel.