core.async: Unbound channels

[Ernesto Garcia]

You can create a unbound channel with (chan), but not if you use a transducer; (chan nil (filter odd?)) will raise an error that no buffer is provided. Why is this the case?

Why the enforcement of all channels to be bound? In a program, there will be channels that propagate to other channels, so only channels at the boundaries would require to be bound?

Channel limits are also much dependent on the particular process and environment. How would we write generic code that creates channels, if those need to be bound to limits unknown?

Thanks,

Ernesto

[Ghadi Shayban]

(chan) is not a channel with an unbounded buffer. It is a channel with no buffer and needs to rendezvous putters and takers 1-to-1.  (Additionally it will throw an exception if more than 1024 takers or putters are enqueued waiting)

[Ernesto Garcia]

Thanks for your response, it is important to know. (Sorry for my lexical typo: unbounded. I didn't realize it derives from the verb bound, not bind!)

My question on channel boundaries still holds though. Why the enforcement of boundaries always?

[Matching Socks]

Ernesto, you may be interested in the informative response to this enhancement request, https://clojure.atlassian.net/browse/ASYNC-23, "Support channel buffers of unlimited size".  Anyway, if you do not want to think very hard about buffer size, you can specify a size of 1.  It does not limit the number of items the producer may put on the channel, but it affects how soon (and how often) the producer's puts must wait for a consumer.

[Ernesto Garcia]

On Thursday, July 4, 2019 at 4:24:33 PM UTC+2, Matching Socks wrote:

Ernesto, you may be interested in the informative response to this enhancement request, https://clojure.atlassian.net/browse/ASYNC-23, "Support channel buffers of unlimited size".

An example: Here a function that makes a request, and returns a channel with responses with the same referenceId as the request. The function taps a mult-channel, but I don't now what buffer to use.I would expect that the original mult-channel already implements some buffering limits.

(defn request [client events-mult-channel req]

  (let [ref-id (.getReference req)

        result-channel (chan 16 (filter #(= ref-id (.getReference %))))]

    (tap events-mult-channel result-channel)

    (.send client req)

    result-channel))

 

Anyway, if you do not want to think very hard about buffer size, you can specify a size of 1.  It does not limit the number of items the producer may put on the channel, but it affects how soon (and how often) the producer's puts must wait for a consumer.

I see, puts start to block when the buffer is full or when no buffer. Isn't the 1024-limit on blocked puts an effective limit, like a buffer size?

[Matching Socks]

"Effective" is in the eye of the beholder.  The 1024 limit helps surface bugs wherein more than a thousand threads are blocked for lack of a certain channel's buffer space.  But the 1024 limit does not pertain if 1 thread would like to do thousands of puts for which there is no buffer space.  In the latter case, that thread is in a loop that does 1 put at a time, and therefore only 1 put, at any given moment, counts against the 1024 limit.

[Ernesto Garcia]

I see. Bufferless channels are meant to be used within the core.async threading architecture, where there will be a limited number of blocked puts and takes. At the boundaries, channels with dropping or sliding windows can be used for limiting work.

So, my original question actually turns into: Why do channels with a transducer need a buffer? Is it just a limitation of the implementation, or does it have a conceptual reason behind it?

[Alex Miller]

Expanding transducers (like mapcat) can produce multiple output values per input value, and those have to have someplace to go.

[Ernesto Garcia]

Thanks Alex!

Correct, the channel implementation takes care that "transduced" channels always pass elements through the transducer and the buffer. Also, a FixedBuffer allows running out of limit for those cases, see this example with a FixedBuffer of size 1 making space for 4 elements:

(def c (chan 1 (mapcat seq)))

=> #'psdk.hack-config/c

(put! c "hola")

=> true

(-> c .buf .buf)

=> (\a \l \o \h)

However, the blocking semantics of a channel change if a buffer is enforced. You can't have a "transduced" channel for which all puts will block for a take. Not sure if this is too limiting for any practical case.

For the case of expanding transducers, I am not sure if there would be sensible semantics for channel operations without a buffer.