Bunch of Stateful Consumers?

[Stefan Siprell]

Hey everybody,

I have been trying to wrap my head around this problem: I want a bunch (as in millions) of stateful agents which communicate asynchronously via a reactor. All of the blogs and comments refer to Reactor as super-fast  stateless event processor and the examples imply that very few object-instances of consumer-classes do the lifting of all the events. 

If the events remain stateless and immutable and I crank up the number of consumer instances to the millions and each has its own internal state: will it scale? Is this something you guys han in mind building this nice piece of tech?

Regards

Stefan

[Jon Brisbin]

Stefan,

Is this something you guys han in mind building this nice piece of tech?

The answer is essentially a heavily asterisked: "not exactly". :)

You can have as many stateful components as you have resources for, of course, but the problem becomes the internal Registry that holds Selector + Consumer assignments. The current implementation is designed to work with a relatively small number of Consumers (I've tested 10's of 1000's of Consumers but iterating over those begins to tax the system). There is a cache inside the Registry which is a Map which gets zapped when adding or removing Consumers. The limitations of putting objects into a Map, then, also apply to a Registry. There's no real "limit" other than that we keep all Registrations in an internal array (because it's faster and cheaper) and we use a Map as a cache. If lots of Consumers are being added or removed frequently, then you'll see a large hit on iterating over the Registrations to cache the proper ones, etc...

If a small number of Consumers is responsible for addressing a large number of stateful components, it's not necessary to have a Consumer assignment per stateful component. If you don't want a single Consumer to do that, you can easily partition the Consumers to address a slice of the hash range for a key in the same way that Dynamo systems do consistent hashing for partitioning data. You're still getting the async handoff it's just that the individual components aren't themselves a Consumer in this case.

Unless we implement our own Map that works differently than HashMap, I'm afraid we'll always be limited by what you can do with that.

Thanks!

Jon Brisbin | Reactor Project Lead

http://about.me/jbrisbin | @j_brisbin

[Stefan Siprell]

Hey Jon,

first of all: Thanks for the answer. It was pretty much what I was hoping for. Of course there is no hard limit, but the design gives some boundaries -  thanks for elaborating. The agents have an intrinsic proximity, so a partitioning of agents is reasonable. I guess a stream / graph setup would be necessary to "multiplex" events to surrounding reactors, when the event occurred too close to a proximity border.  The consumer would handle all of the updates for the agents in the 'hood. I kind of like this approach - does it make sense?

Stefan