Transform one message into many

[Marko Topolnik]

I need to transform a channel's output so that one incoming message produces many outgoing messages. Looking into the implementation of map* and filter* I didn't get an idea how I could achieve this. Any help here, apart from "manually" shoveling messages over using a receive-all?

[Zach Tellman]

You almost never want to use receive-all + enqueue, because that breaks the auto-closing behavior.  If you want multiple channels from a given stream that are consumed separately, you can (fork ...) the channel or (map* identity ...) over it multiple times.  If that's not what you want, you'll have to explain what you're trying to do in more detail.

[Marko Topolnik]

I have set up a path involving a map* stage, which most of the time needs to just transform one message into another, but I have a special case where one message must be transformed into a stream of several messages. 

In more detail, I have an API defined where the structure of messages that my client dequeues from the channel is already specified. They notify the consumer about changes in the state of a system. But when a new consumer arrives, I want to replay the messages that are necessary for that consumer to initialize its copy of system's state.

Now, my map* stage is stateful: it remembers the current system state and suppresses duplicate messages (there is another filter* downstream that discards nil messages). That state is private to the map* stage, so I want that stage to be the source of all the state-establishing messages and want to trigger this by a special message that will be enqueued into the map* stage.

More generally, a stateful map*/filter* combination is very close to playing the role of an actor and I would like to leverage this fully. I'm sure my immediate goal could be realized by different routes, but this one seems the most elegant at the moment.

[Marko Topolnik]

In simplest terms possible, I need a mapcat*.

[Zach Tellman]

Ah, I see, I was thinking one stream into many streams.  Luckily, there already is a mapcat*, which is a composition of map* and concat*.

Zach

[Marko Topolnik]

Thanks. I need a better way to browse the library... I jumped into the code to filter* with the intention to look for concat* amongst its neighbors, and didn't realize I wasn't in core, but in channel.

[Zach Tellman]

http://ztellman.github.com/lamina/

[Marko Topolnik]

Thanks... the problem with that one is "something goes here" :) I tried to use it for a while, but found myself resort to browsing the code anyway. I'll just have to keep in mind the consequences of Potemkin :)

If you'd like a helping hand with writing the docs, let me know; although I doubt how useful I can be.

[Wai Lee Chin Feman]

I just wanted to say: thanks for raising this question.  I hadn't used mapcat before.  But now I know.

Also, sometimes you wind up having one event propagate to many using the following pattern:

(siphon (filter* #(...) origin-channel) first-propagation)

(siphon (filter* #(...) origin-channel) second-propagation)

(siphon (filter* #(...) origin-channel) third-propagation)

Which might be more idiomatic depending what you're doing.