scalability of serf

[jst...@tacc.utexas.edu]

Hi,

Are there any write-ups (formal or otherwise) on the scalability of Serf? I'm familiar with the original SWIM paper which has nice performance results regarding message load, expected detection time and false-positives. The prototype presented there is relatively small, but I've seen it stated that Serf scales well to 5 digit numbers of servers, including minimal bandwidth consumption. I would be very interested in any analyses along these lines.

Thanks very much in advance.

Best,

Joe Stubbs

[Armon Dadgar]

Hey Joe,

Unfortunately there are no write ups about the scalability of Serf. It is very closely modeled on SWIM,

with bits taken from Bimodal Multicast. Most of the scalability testing comes from real-world deploys

of both Serf and Consul.

Best Regards,

Armon Dadgar

--
You received this message because you are subscribed to the Google Groups "Serf" group.
To unsubscribe from this group and stop receiving emails from it, send an email to serfdom+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joe Stubbs]

Thanks very much Armon. Is there any more detail you can provide on large real world deploys? Even confirmation of total number of nodes would be useful. 

Thanks again, 

Joe 

[Armon Dadgar]

Hey Joe,

It has been successfully deployed at about the 10K node scale. I’m not sure on the exact count of machines however.

There are many many smaller deploys in the 1K-5K node case as well.

Best Regards,

Armon Dadgar

[Joe Stubbs]

That's great, thanks again! 

[Luca Clementi]

Hey Joe,
I did run Serf for few days on a 20K nodes cluster, while the cluster
was getting installed.
After few hiccups (that you can find here
https://github.com/hashicorp/serf/issues/269), it was running quite
smoothly.

I haven't had a chance to run it in production yet, although we are planning to.


Luca

[Joe Stubbs]

Very cool, Luca. Can you share anything more about your deployment? We would be interested in the network topology details (data centers, firewalls, etc) and any performance metrics you may have regarding bandwidth consumption, convergence times, and so on, if you are measuring them. Also, is this pure serf or are you deploying Consul?

Thanks again!
Joe

________________________________________
From: Luca Clementi <luca.c...@gmail.com>
Sent: Friday, March 20, 2015 1:15 PM
To: Armon Dadgar
Cc: Joe Stubbs; ser...@googlegroups.com

[Luca Clementi]

On Mar 20, 2015 2:08 PM, "Joe Stubbs" <jst...@tacc.utexas.edu> wrote:
>
> Very cool, Luca. Can you share anything more about your deployment? We would be interested in the network topology details (data centers, firewalls, etc) and any performance metrics you may have regarding bandwidth consumption, convergence times, and so on, if you are measuring them. Also, is this pure serf or are you deploying Consul?
>

Hey,
I only tested the software for few days and the cluster was mostly
idle, getting set up for acceptance tests, so take this info with the
consideration it deserves...

We were running serf on a flat 10ge network (all the nodes were in the
same network segment) so no routing no firewall.

If i remember well i noticed a cpu consumption of about 5 minutes a
day by the daemon on each node.

Bandwidth i have no clue i think there is a formula to calculate that
(and i dont think it represent an issue considering serf message size
and the fact that we are running on 10GE).

We are planning to use it to deliver user messages from our compute
nodes to our service nodes, i did some preliminary (and rough) tests
and noticed that the delivery of messages is basically instantaneous,
most of the nodes get the message in around a second or two, and some
of that delay might have been caused by the central rsyslogd receiving
all messages from all the nodes.

The critical issue is how is it going to perform under load when most
of our compute nodes CPU load will be consistently 100%?
This will introduce high latency in processing serf messages on all
nodes and it _might_ cause node flapping (going in and out of the
cluster).
^^^^^^^^^^^^^^^^^^^^^^^
ATT: These are all my speculations I have no real number for this.

Since we are not planning to actively use cluster membership (who is
joining and who is leaving) this should not be a problem for us, in
fact messages will be delivered anyway thanks to the gossip protocol
characteristics (even if some nodes flap).


I hope that helps,
Luca

[Joe Stubbs]

Thanks for all the information Luca - it is very helpful. Good luch with the project. It sounds very interesting!

Best,

Joe

________________________________________
From: Luca Clementi <luca.c...@gmail.com>

Sent: Saturday, March 21, 2015 1:33 AM
To: Joe Stubbs
Cc: Armon Dadgar; ser...@googlegroups.com

Subject: Re: scalability of serf

[Vasileios Anagnostopoulos]

Hi,

as far as the scalability of Sef I have some observations which I would be glad to hear any objections

The bigger the deployment => 

the bigger the probability of an element to fail=>

the bigger the frequency of failures =>

the bigger the flooding with messages =>

the bigger the delay to report?

[Armon Dadgar]

Hey Vasileios,

I think the first few parts of that are correct. Given an constant mean-time-to-failure,

the larger the cluster size, the more often you will experience failures. Additionally,

as the cluster grows the update dissemination time grows logarithmically with it.

However, for clusters of even thousands of nodes, delivery is still sub-second. 

There is a simulator here where you can play with the properties and it see what that

does to dissemination time:

https://www.serfdom.io/docs/internals/simulator.html

Hope that helps!

Best Regards,

Armon Dadgar

From: Vasileios Anagnostopoulos <fithi...@gmail.com>
Reply: Vasileios Anagnostopoulos <fithi...@gmail.com>>
Date: May 14, 2015 at 7:27:34 AM
To: ser...@googlegroups.com <ser...@googlegroups.com>>
Subject:  Re: scalability of serf

[Vasileios Anagnostopoulos]

I have to check then.

Thank you for your professional contribution.

--

Dr. Vasileios Anagnostopoulos (MSc,PhD)

Researcher/Developer

ICCS/NTUA 9 Heroon Polytechneiou Str., Zografou 15773 Athens,Greece

T (+30) 2107723404 M (+30) 6936935388

E va...@mail.ntua.gr<mailto:va...@mail.ntua.gr> www.ntua.gr<http://www.ntua.gr/>

[Eric Anderson]

Hi All,

I'm dropping in to revive this question.  I'm looking at using Serf in a limited-bandwidth context, and I'm interested in the performance/load implications.  I've got a pretty good understanding of how SWIM works, but I'm curious about how exactly Serf differs from the original SWIM algorithm / protocol.  Is that documented anywhere, and/or is it easy to glean from the code?

Thanks!

Eric