Hi,
A simplified view of my Akka use case involves a "producer" actor that is producing messages that are sent on to other parts of the my actor system. The producer is fetching data from MongoDB. Right now, the "consumer" actors are not able to keep up with the rate of the producer and thus I eventually face running out of heap space.
I've been reading up strategies with regard to throttling but none of them, in my mind, would work in my scenario. The issue is that my producer is getting it's data using ReactiveMongo - I'm creating a tailable cursor. When data is encountered, it is being delivered asynchronously via a callback (using Play! Iteratees):
cursor = Some(messages.find(new QueryBuilder(), QueryOpts().tailable.awaitData))
cursor.get.enumerate.apply(Iteratee.foreach({
doc =>
val json = BSONConversions.fromBSON(doc)
receiver ! Notification(json)
}))
One strategy I've seen relates to consumers actually informing the producer that they are ready for work, but I can't see how I can do this with the streaming tailable cursor approach.
As I'm new to Akka, I'm still getting my head around the concurrency model. In a non-actor model, I guess you could consider throttling the thread handling the callback from the tailable cursor - however I'm not sure this makes sense in this Akka context.
Tools such as ReactiveMongo are purposely designed for non-blocking usage scenarios, which in turn makes them play well in an Akka environment. Thus, I feel that there must be (and needs to be) a good pattern for this type of scenario. Any ideas on how I might deal with this?
Thanks,
Andrew