Generators vs streams (incl. new library)

[Siddhartha Kasivajhula]

Hi folks,

Here is a by-no-means definitive, and yet hopefully useful library for working with generators:

https://docs.racket-lang.org/generator-util/index.html

In writing this library, I thought about the cases in which one might use a generator vs a stream in practice. Since Racket provides such great interfaces for working with streams, why not just always use them, and maybe convert generators into streams using utilities like in-producer? It somehow felt wrong to do that, so I did some digging to see what people might have said on the subject. I found a paper about it, but it didn't really get into this particular point.

In the end I seemed to converge on that while streams are great for lazy semantics in general, generators are a more natural fit when, in addition to requiring lazy semantics:
1. We are working with state and side effects (although of course there's nothing except good sense preventing us from writing streams that deal in side effects), or
2. In cases where function semantics are more natural, rather than sequence semantics, and finally, and probably the most important

3. When we need constant memory guarantees, for instance, while processing large datasets for some "big data" style task. For this last point, since streams memoize return values, their memory requirements should grow linearly, is that right?

Anyway, whatever the reason one might use them, it just felt like working with generators could be a little more convenient, hence this library. There are 2 main caveats with using these utilities: (1) they don't support coroutines, i.e. cases where you need to send data to an "online" generator, vs just consuming from it (does anyone know if there is a way to do generator delegation in Racket, like in python?), and (2) the provided "constructors" aren't lazy, so they are best used to manipulate existing generators rather than truly construct generators from scratch for production usecases (not sure this pattern of usage would ever really come up, but just for symmetry with e.g. streams).

Finally, for reference, quickcheck and rackcheck also seem to provide some useful generator-related utilities (not sure if they are limited to testing usecases).

-Sid