another performance related question

[eugene]

Hello,

I've read that Meteor has been "tested up to 5,000 clients per node
receiving 1 msg/sec each".

I noticed meteor is based on 'select' and thus the latency is expected
to increase O(n) along with the number of concurrent clients. Had
anyone tested the latency of let's say 10,000 or 20,000 clients
receiving occasional messages. In other words what do you expect a
likely bottleneck would be?

Thanks a lot,
Eugene

[Andrew Betts]

Eugene,

Yes, Meteor is single threaded and uses select to poll connections.
While the resource use does increase linearly with both the number of
concurrently connected clients and also the rate of messages sent, it
can scale further in terms of messages than in terms of clients. So a
large number of messages being sent to a small number of clients
scales better than a small number of messages sent to a large number
of clients.

There was some discussion some time ago about re-factoring Meteor into
a multi-threaded version but I have not heard anything about this
recently.

Andrew

[eugene]

Thanks a ton, Andrew.

Unfortunately the system I'm architecting is characterized by a large
number of mostly idle clients. Your response confirmed my suspicion
that in it's present form Meteor will not scale enough. I like it's
simplicity and flexibility though. I'm inclined to prototype the
system using Meteor and work on it's re-factoring later, betting that
few of thousands of clients Meteor handles with ease will buy me
enough time.

I'd appreciate any suggestions on re-factoring Meteor to use
asynchronous event loop model, like Event::Lib or maybe Danga::Socket
-- both make use of epoll on Linux and kqueue on FreeBSD.

Regards,
Eugene

[Andrew Betts]

Hi Eugene,

I've no opinion either way, as I've not used either of them. Feel
free to experiment and submit your patches though - I'd be very
pleased to get your input.

Andrew