Google Groups

Re: Tech Question - ZeroMQ and event replication


Udi Dahan Dec 10, 2011 7:12 AM
Posted in group: DDD/CQRS
On the whole gigabit ethernet thing - most developers aren't coding at
the level of ethernet.
TCP is usually as low as people go, and that gets around 40%
utilization of the underlying ethernet (at best), so you're at 50MBps.

Then there's the whole issue of how you serialize your payloads.
XML serialization can be an order of magnitude of bloat on the
underlying data, so you could be effectively at 5MBps.

Not to say that everyone's doing XML serialization.

Still, whether it's 5 or 50, that's for all your users and all their
data.
Most people don't think about scaling at the network level (because of
the prefix "giga" which is translated to "so much I don't even need to
think about it").

Some more food for thought :-)

Udi


On Dec 9, 6:21 am, Tim Gebhardt <t...@gebhardtcomputing.com> wrote:
> Hey Tom, sorry for the late reply.
>
> It's not even that the sharding is necessary if your message rates can keep
> up:
> A gigabit ethernet interface has a theoretical sustained write rate of 125
> megabytes per second:http://www.google.com/search?q=1000+Mb/s+in+MB/s
>
> That's a LOT of data.  I work for a financial brokerage where we have a lot
> of incoming and outgoing customer market data (with a lot of duplication
> included) and the last time I looked at our firewall logs I think our
> external interface had an average sustained data rate of about 70 MB/s.
>
> But even the most reasonable enterprise-class hard drives can keep up with
> that sustained write rate (making the assumption that your input and output
> messaging rates are about the same).http://www.tomshardware.com/charts/enterprise-hard-drive-charts-2010/...
>
> Spinning rust hard disks are pretty fast at streaming reads and writes,
> it's the seeks the kill you.  But if you're doing event sourcing then
> you're just streaming and some pretty reasonable hard ware should be able
> to keep up.
>
> Just food for thought.
>
> Tim
>
>
>
>
>
>
>
> On Sat, Dec 3, 2011 at 3:45 AM, Tom Janssens <d4sk...@gmail.com> wrote:
> > Hey Tim,
>
> > Once again, great post ! (Still would love to see the blog post ;). So,
> > you are saying one should shard in hardware (i.e. RAID), that makes a lot
> > of sense: just get faster HW as dev cost is usually more expensive then HW.
>
> > thanks!
>
> > Tom
>
> > This message was typed on a mobile