How elastic is the system?

[todd]

I was also wondering how quickly new nodes can be brought into the
system and how it would handle nodes that fail frequently, that is
would a lot of churn in nodes kill the system or does expect a
relatively stable cluster?

thanks

[Anand Kesari]

Hi Todd,

S4 supports two modes of operation (static and elastic). A static S4
cluster runs with a fixed set of machines and cannot tolerate node
failures. S4 also supports running an elastic cluster that uses
ZooKeeper to manage a pool of active machines. The example we have
provided runs in the static mode.

Churn would not "kill" the system in terms of making it come to a halt
or run out of resources. However, this would cause a input events to
be lost (i.e. not received and processed). Currently, upon node
failure, all state based on past events on that node is lost. When a
new node takes over, the state is constructed from the events that it
then receives. So the accuracy of results in the presence of churn
depends on the amount of state that Processing Elements have to
maintain: if this is relatively small, then churn does not affect the
application very much; if not, then the results of the computation may
be inaccurate.

--
anand