[ANN] Transducers library

[Jack Firth]

Rebellion now provides the rebellion/streaming/transducer library for processing streams of data using transducers.

A transducer is a kind of sequence transformation that comes with guarantees about memory usage and about how much of the sequence is consumed in between each produced output. Transducers are first class objects that chain together fluently, without any macros involved. For example, this code defines a function that finds and reports all of the lines in a string that are longer than 80 characters and it uses no macros at all:

(require rebellion/streaming/reducer
         rebellion/streaming/transducer)

(define (report-all-long-lines str)
  (transduce str
             (batching into-line)
             enumerating
             (filtering long-line?)
             #:into (into-for-each report-long-line)))

(define (long-line? e)
  (define line (enumerated-element e))
  (> (string-length line) 80))

(define (report-long-line e)
  (define line (enumerated-element e))
  (define number (enumerated-position e))
  (printf "Line ~a is longer than 80 characters: ~a\n" number line))

You may find transducers more useful than for-style loops in the following cases:

• The for/xxx form you need doesn't exist
• You want to sort elements or search for the N largest or N smallest elements
  
• You're reading data from external sources and want control over when resources are acquired and released
• You're processing a stream that's too large to fit into memory
• You care a lot about processing your streams in only one pass
  
• You want to combine elements into batches or group them by key
• You want to remove duplicate elements
  
• You prefer writing sequence transformations in a point-free style
• You already like reducers
• You've got a rebellious streak

For some real-world examples of transducers in action, I cannot thank Sam Phillips enough for putting together this gist containing scripts he wrote for his day job using transducers.

[William G Hatch]

On Wed, Oct 23, 2019 at 12:39:43PM -0700, Jack Firth wrote:
>and it uses *no macros at all*:

Gotcha! You're using the `define` and `#%app` macros in this example,
aren't you!

More seriously, ever since I watched a Rich Hickey talk about
transducers I've wanted Racket to have them. Thanks for making this;
I'm excited to use it.