Google Groups

Re: [DDD/CQRS] Tech Question - ZeroMQ and event replication


Rinat Abdullin Dec 2, 2011 9:40 AM
Posted in group: DDD/CQRS
Kelly,

Using Azure SB will break the technology golden rule by increasing number of technologies the project depends upon :)

While by using spirits I can actually reduce the number of tech (replacing already used in-memory and file-based queues with sockets) at the same time with improving throughput. Besides this will bring closer the experience of Rackspace and Azure deployments (allowing to have same maintenance tools).

Sounds like an easy choice for me :)

Best,
Rinat

On Fri, Dec 2, 2011 at 11:18 PM, kell.s...@gmail.com <kell.s...@gmail.com> wrote:
Azure Service Bus sounds like a natural fit here when hosted on Azure. Some people are publishing an absolutely massive amount of messages through it so scaling of messages shouldn't be a problem there. Much better in throughput and scale than Azure Queues from my understanding. 

For my Continuous Client demo I wired up Azure SB manually in 2 hours. Using the SDK likely even faster :)

Later,
Kell

On 2011-12-02, at 7:05 AM, Rinat Abdullin <rinat.a...@gmail.com> wrote:

Hi Tom,

We're doing slightly simpler approach, the brute force way. In short - if system is under load (queues get packed), we just provision more workers (virtual machines) to help deal with the work. Properly designed systems can handle 250x increase easily. Less thorough systems do only 15x increase (or much less).

Once the work has been dealt with - de-provision the excessive workers (and pay only for the hours that have been used - that's the beauty of the cloud computing :)

However, one of the primary bottlenecks of this approach is that when you ask for more workforce, it also increases the overall amount of messages that fly through the system (starting to kill primary queues, brokers, IO etc).

The idea with these small socket-driven spirits is to have everything constructed in a way, that when new workforce comes in, it also brings in more infrastructure capacities with itself, increasing in total throughput, redundancy and reliability. Somewhat similar to p2p networks (but dead-simple in topology). Sounds crazy, I know :)

Best,
Rinat





On Fri, Dec 2, 2011 at 5:52 PM, Tom Janssens <d4s...@gmail.com> wrote:
Hey Rinat,

The spirits sound like an interesting approach... Here's an idea, although it will probably be to complicated:

If a spirit is under heavy load, do a multicast, requesting spirits with duplicate functionality that do not have much load. When you find one, morph it into the one that you need...
(And add a chaos monkey as well, so you are sure everything works as expected.)

On the other hand, you might just as well contact google and ask them to license their tech ;)

Tom