CQRS for a browser game, handling big message spike during "world generation"

[Ben Fischer]

Hi,

I am currently evaluating if CQRS can be used successfully for a space-oriented browser game.

The normal "game flow" (building or researching something, statistics, etc.) seems to be easy to model, but I am having a hard time modeling the aggregates and their corresponding commands + events which are necessary to build the universe.

If, for example, the universe has 10 galaxies, each with say 500 solar systems, each with 10 planets. Every galaxy, solar system and planet has additional properties.

Not a small amount of data...as soon as it gets projected in the view store (inmemory-based) it's not really a problem anymore.

My previous/first attempt was to model universe, galaxy, solar system and planet as aggregates with a CreateUniverseCommand + UniverseCreatedEvent which would create 10 CreateGalaxyCommand's -> GalaxyCreatedEvent which would create 500 CreateSolarSystemCommands and so on...

That would fill my command queue instantly with a few dozen thousand commands, which doesn't feel right.

In my opinion, Planet just has to be an aggregate (is referenced by other aggregates, so if I understand correctly that precludes having Planet as entity inside another aggregate), but as long as I have Planet as aggregate, I won't get fewer than 50.000 events (in said example, maybe more, maybe less in real usage). 

Maybe games are not the best scenario for CQRS, but I imagine other application types have similar problems with bulk input / spikes / whatever-it's-called, so maybe someone can push me in the right direction :)

Regards, Ben

[Greg Young]

The event store can handle 50k events in about 5 seconds on my laptop...

Is it really an issue?

Greg

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Ben Fischer]

I didn't have the time yet to read about your new event store. Not sure if my event store is the real problem, didn't do enough profiling to be sure. 

I suspect the accumulated latency for each round-trip to the azure service bus (command queue) is the real culprit.

[Greg Young]

At sounds quite likely. But I'm general 50k done rarely is really not that much. To give you an idea, for financial data its common to take that sustained per second. Focus on your distances and likely you can handle it easily

[Greg Young]

If your data is in memory can I ask why you need a service bus?

[Tom Janssens]

This might be a bit off-topic, but might I suggest this presentation to get you started: http://www.slideshare.net/wooga/erlang-the-big-switch-in-social-games

Op vrijdag 28 september 2012 16:49:13 UTC+2 schreef Ben Fischer het volgende:

[Ben Fischer]

Thanks, nice to hear that I may not be that far off than I initially thought ;)

Maybe I should describe my current (more or less experimental) architecture.

- Worker role, reads commands from mentioned service bus queue, executes command handlers, writes resulting events in event store and publishes them on service bus

- "UI role", hosts the Game UI, receives events from service bus and updates in-memory view store. During role start, the message log (based on Lokad.CQRS Sample message store) is read and the view store is built to the current state.

Service bus is needed because of the two different roles. I don't want to combine the UI and the worker so the alternative would be to utilize the projections / generate the views from the worker and save them to blob storage. In doing that, the service bus may not be needed anymore. But you got me thinking here...my initial reason (well at least partially, in-memory was also just appealing because of speed ;) ) was that I thought I would have a few million instead of ~100k planets, which would result in massive amounts of storage transactions (each planet must have it's own view), since that's not the case anymore blob-based view storage may be an option. But even ~100k view updates would take quite some time.

Another potential problem may be, that later in the development, multiple parallel universes (like tenants, they are more or less dedicated (not quite sure yet)) shall be possible. My fear was, that if one universe gets created (with mentioned ~50-100k messages), the other universes' performance may be drastically reduced. But, like tenants, I guess I could have a command queue for each universe, that should solve that problem.

I am unsure if I fully grasped the concept of bounded concepts yet, but if I have a bounded context "Universe" for the gameplay elements (1 for each universe, like tenants) and 1 BC for all cross-cutting/universe-overlapping elements (chat, accounts) and one queue for each BC (and thus for each universe), the performance shouldn't be a problem.

Sorry for rambling, I have a hard time keeping all the cqrs topics in my mind, maybe I am thinking too much in "old ways" ;)

[Ben Fischer]

If I were to transform each Universe into a Bounded Context (well each universe would be an instance of the Universe BC), how would they get created? Universes can be created from inside the application (by an admin).

One way could be that this creates an CreateUniverseContextCommand and the corresponding event would then create the Universe BC. After an application restart (or crash, version upgrade, whatever) the main bounded context goes through the main domain log and for each UniverseContextCreatedEvent the Universe BC gets (re)created. The Universe BC's would then access their own BC domain log, read from their own queue etc etc.

Would that make sense?

@Greg, you asked why I used a service bus. Based on my explanation, would you discourage me from doing so?