Is reliability important in this scenario? I treat things like MSMQ/RabbitMQ as total opposites from ZeroMQ. I use ZeroMQ when fast is more important than reliability because it's not a reliable messaging solution.
In our system we are using both reliable and unreliable messaging. Similar to the concept of separating reads and writes we optimize the communications choice based on the needs.
Look forward to hearing about your ZeroMQ experiences :)
Later,
Kell
Generate events first? (Y/N) y Time to store 4093664 messages (612,931488037109 MB) : 60000,4318 msecs, msgs/sec = 68227,2423246127, MB/sec = 10,2154512834208 Reading back messages Time to read 4093664 messages : 81044,6355 msecs, msgs/sec = 50511,2272360087
Oddly enough, reading is slower, even though I read only half the amount of bytes I write in this particular case... However, I get 60 times the speed you get ?
Am I missing something here, or is there a problem with your implementation ?
If you want to verify: create a new console app, use nuget to add Scritchy (make sure you ref the latest 0.1.0.15), and copy/paste the code from this gist:
https://gist.github.com/1418094
Tom
Once again, great post ! (Still would love to see the blog post ;). So, you are saying one should shard in hardware (i.e. RAID), that makes a lot of sense: just get faster HW as dev cost is usually more expensive then HW.
thanks!
Tom
This message was typed on a mobile
Then there's the whole issue of how you serialize your payloads.
XML serialization can be an order of magnitude of bloat on the
underlying data, so you could be effectively at 5MBps.
Not to say that everyone's doing XML serialization.
Still, whether it's 5 or 50, that's for all your users and all their
data.
Most people don't think about scaling at the network level (because of
the prefix "giga" which is translated to "so much I don't even need to
think about it").
Some more food for thought :-)
Udi
On Dec 9, 6:21 am, Tim Gebhardt <t...@gebhardtcomputing.com> wrote:
> Hey Tom, sorry for the late reply.
>
> It's not even that the sharding is necessary if your message rates can keep
> up:
> A gigabit ethernet interface has a theoretical sustained write rate of 125
> megabytes per second:http://www.google.com/search?q=1000+Mb/s+in+MB/s
>
> That's a LOT of data. I work for a financial brokerage where we have a lot
> of incoming and outgoing customer market data (with a lot of duplication
> included) and the last time I looked at our firewall logs I think our
> external interface had an average sustained data rate of about 70 MB/s.
>
> But even the most reasonable enterprise-class hard drives can keep up with
> that sustained write rate (making the assumption that your input and output
> messaging rates are about the same).http://www.tomshardware.com/charts/enterprise-hard-drive-charts-2010/...
Perfect timing, Rinat!
We had a very important stream go corrupt yesterday. This would have been nice then. Instead, I rebuilt the stream to version-1, to another container and overwrote.
I'm moving to sample project thus weekend. :)
Tom,It's too early to write a post, because topology is currently not fixed. I'm adjusting it on daily basis, as more systems get proper ES support and/or are brought to CQRS world. In a sense, it is just like neural system in a growing organism - continuously shifting and changing (multiple times a week sometimes)I have multiple masters, because different systems "source" different event streams. Off-cloud systems have local event streams, which are then pushed over, while Cloud systems store event streams in the cloud storage.Replicators run everywhere, where I can start them (in Azure/Rackspace and on dedicated DC). In fact, the backup that was saving my life today, was not a backup, but a replica from reporting system.I don't aim for centralized topology, but rather a distributed network of dead-simple elements, that can replicate and heal as needed (and preferably survive nuclear attack on half of the world, while coming alive and sending autonomous domains back into the past to prevent this from happening)Hope this helpsBest,Rinat
Crap rinat is building skynet
Sent from my iPad