Trying to understand Clojure/Java concurrency performance?

[Nick Pavlica]

Dear Clojure/Java Experts,

  Earlier today I posted a question to the Immutant google group asking them a question about large numbers of concurrent websocket connections and their server.  A member of the group kindly responded with a link (https://github.com/ptaoussanis/clojure-web-server-benchmarks)  to some fairly recent benchmarks that included a number of Clojure/Java servers.  After looking at the numbers in the benchmark, I was a little disappointed to see that they were only serving 60K connections as compared to other solutions like Erlang which seem to be capable of 1-2+ million connections on similar hardware.  The difference on the surface seems dramatic, and I was wondering if someone could help clarify the numbers for me.  It makes me wounder if there is a JVM solution that can even meet these numbers half way, or if I'm missing something?   

  

Many Thanks!

--Nick

####### Discussion from the Immutant Google group ######################

All,

  I'm a new Clojure developer, and I'm looking for a webserver that can handle a large number of concurrent websocket connections.  I understand that this question is a little/very vague, and is dependent on many factors like the OS, it's Kernal, amount of RAM in the system, etc.  However, there are a number of generalized claims out there, and I was wondering if anyone has done any basic testing/benchmarking with Immutant/Undertow?

Examples:

- http://www.http-kit.org/600k-concurrent-connection-http-kit.html

- http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html

- https://en.wikipedia.org/wiki/C10k_problem

- https://vimeo.com/44312354

- And so on ...

Thanks!

--Nick

--------------------------------------------------------------------------------------------------

Hi Nick, 

Here are some fairly recent benchmarks that are relevant: 

https://github.com/ptaoussanis/clojure-web-server-benchmarks 

--Sven 

-------------------------------------------------------------------------------------------------

[Nick Pavlica]

All,

I posed this question with a little more detail in the Quasar/Pulsar group in hopes that they may have some insight into my question (https://groups.google.com/forum/#!topic/quasar-pulsar-user/l8ZX7pk9bkI) because they are more focused on that domain.  Hopefully, my question is clear, if not please help me clarify it.

Thanks Again

-- Nick

[Moe Aboulkheir]

Nick,

(There's a lot to understand about those benchmarks, and I haven't really spent time with them, or wrk2, so feel free to ignore)

On Wed, Oct 7, 2015 at 2:13 AM, Nick Pavlica <lin...@gmail.com> wrote:

After looking at the numbers in the benchmark, I was a little disappointed to see that they were only serving 60K connections

I don't see that in those results.  There are definitely a lot of numbers bigger than 60k here: https://github.com/ptaoussanis/clojure-web-server-benchmarks/blob/master/results/60k-keepalive/20150217-13-18-table.txt - and in the http-kit link you included. 

 

as compared to other solutions like Erlang which seem to be capable of 1-2+ million connections on similar hardware.  

Do you have a reference for that outside of the video you linked?  I haven't watched it, but the description mentions "tuning and patching the BEAM emulator and FreeBSD kernel".

Take care,

Moe

"60k ought to be enough for anybody"

[Thomas Heller]

FWIW getting 1mil+ connections will requires some OS level tuning even in Erlang. So you are not going to get that in a benchmark that is not set up for this.

In the JVM you are going to run into GC problems eventually, depending on how much state/memory you keep per connection. It might work with something Azul [1] offers but the standard JVM still has no comparable GC. You can do some heavy tuning of the different standard GCs but that is a science in itself and can not be answered without a deep understanding of your app architecture.

Erlang was specifically designed for this purpose and makes some things significantly simpler, so it is going to be much more straightforward writing such a system. Erlang is also pretty fun.

As you can see in the http-kit benchmark, the application itself barely uses any CPU. Most of it is GC, so this is going to be your worst enemy as you are going to run into some hefty GC pauses on large heaps, which will affect ALL connections. Erlang does not suffer from this problem.

I'm a big fan of using the right tool for the job, in this case I would recommend Erlang and only Erlang. Unless you want to write C, which is going to be much much harder and more prone to error. You can also use Erlang to handle all the connection stuff and interface it to a Clojure app that does the DB work, you do not have to write everything in Erlang.

Just my 2 cents,

/thomas

PS: I have not written such a system in either language, so everything is just an educated guess.

[1] https://www.azul.com/

[Thomas Heller]

Just realized that I misread the http-kit benchmark, but still stand by my arguments. GC problems are pretty much directly tied to the size of the heap. So if you'd re-run the http-kit benchmark with a 8gb heap instead of 3gb things would probably look different. How much heap you are going to need depends on your app. Also there is this whole aspects of latency vs. throughput.

 

I would recommend looking at everything Martin Thompson, Gil Tene or Cliff Click have written or spoken about, they are the real Java experts. Although they are not necessarily talking about networked server, pretty much all of it is till going to apply to an app of that scale. Are you sure you are going to need that scale? 1mil connections is a pretty ambitious goal.

/thomas

[Nick Pavlica]

All,

  Thanks for the great reply's thus far!  They have helped me get a better idea of what the issues may be on the JVM.  

  "Are you sure you are going to need that scale? 1mil connections is a pretty ambitious goal."

  I'm not currently planning on 1-2 million connections on a single server at the moment.  I really wish I had those problems, but I would like to count on being able to achieve 100-200K on a single reasonably sized server.  Even if I could achieve 1-2 million connections on a server, I'm not sure it's the best idea to do so.  It seems like allot of coupling of the service to a single endpoint.  I sounds like Elxir/Erlang+x, or possibly Go, may be better at handling the concurrency components of my app until the JVM ships with a GC that's better suited to this kind of work.  I know that they are going to ship a new GC in 1.9, but I don't know if it will help out. 

Thanks!

-- Nick

[Timothy Baldridge]

I wouldn't be so fast to discount the JVM. http-kit isn't exactly the only game in town when it comes to websockets on the JVM. And also it's written by a rather small team. I'd investigate other tools (Netty?) before discounting an entire platform due to a quick glance at a single library. 

Also, in this day and age, I question why anyone would need 200k connections on a single box. Shard the server, use AWS auto scaling, there are many ways to solve this sort of problem besides (as you mentioned) putting all your eggs in one basket. 

In the end any platform is going to be so taxed by 1mil or even 200k connections that you are going to need to scale out to more boxes to do anything besides handle ping requests. 

So if you're doing something like a chat server, that's one thing, but if you're doing actual processing, handling that many connections on a box seems like premature optimization. 

Timothy

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Moe Aboulkheir]

On Wed, Oct 7, 2015 at 8:40 PM, Nick Pavlica <lin...@gmail.com> wrote:

  I'm not currently planning on 1-2 million connections on a single server at the moment.  I really wish I had those problems, but I would like to count on being able to achieve 100-200K on a single reasonably sized server.  Even if I could achieve 1-2 million connections on a server, I'm not sure it's the best idea to do so.  It seems like allot of coupling of the service to a single endpoint.  I sounds like Elxir/Erlang+x, or possibly Go, may be better at handling the concurrency components of my app until the JVM ships with a GC that's better suited to this kind of work.  I know that they are going to ship a new GC in 1.9, but I don't know if it will help out. 

Whether GC is going to be a problem for you for other reasons, your post contained a link to a benchmark which surpassed the number you quoted by an order of magnitude - 600,000.  Disregarding whether http-kit is a sensible/safe library choice, its benchmark is readily understandable.  The github webserver benchmarks also contain many numbers above 200,000 qps, for a variety of other webservers - right? 

http://www.http-kit.org/600k-concurrent-connection-http-kit.html

I don't think the problem exists as you're describing it.

Take care,

Moe

[Nick Pavlica]

Hi Timothy & Moe,

  Thanks for your feedback!  I realize that we are speaking very generally on this thread, and know that there are many ways to solve the complex problems we face as developers.  I'm trying to understand the current know limitations of using great tools like Clojure to solve the problems I'm facing.  Clojure is bound by the JVM  which wasn't directly designed to address high concurrency needs in the same way that Erlang/Go/etc. are.  My experience with Java/JVM in the past was that it was more focused on raw throughput, and less about high levels of concurrency.  I was hoping that there were some good high concurrency solutions in place.  That's what prompted my original question about the only benchmarks that I could find.  If you have any additonal benchmarks, papers, etc., to share on this subject, it would be great! 

Timothy Wrote:

        "I'd investigate other tools (Netty?) before discounting an entire platform"

         -  I'm open to any suggestions.  However, I believe that Aleph is build on Netty.

         "I question why anyone would need 200k connections on a single box, use AWS auto scaling"

         -  There are many reasons, but in my case I'm paying for each server, and I have to maintain not only the ones that I need to get the work done, but ones I need for redundancy etc.  There is a huge business case for doing the same work on one server that  would otherwise take two or three.  An AWS server equipped similarly to the one in the benchmark is running about $1200/month, adding two more would obviously increase my costs by $2400/month or $28,000/year.   

Mo Wrote:

         "The github webserver benchmarks also contain many numbers above 200,000 qps, for a variety of other webservers - right?"

         - Yes, they are really great at throughput, but that's not necessarily the primary metric that I'm looking for.  I'm trying to determine if they are reliable and perform reasonably well with many connections.  I realize that http-kit made a claim of 600K connections in 2013.  But I can't find any other supporting evidence that it can do that reliably.  It also seems to have stagnated a bit as a project which is unfortunate. 

Thanks Again for your input and guidance!

-- Nick

        

On Tuesday, October 6, 2015 at 9:57:45 PM UTC-6, Nick Pavlica wrote:

[Moe Aboulkheir]

On Wed, Oct 7, 2015 at 7:47 PM, Moe Aboulkheir <m...@nervous.io> wrote:

Nick,

(There's a lot to understand about those benchmarks, and I haven't really spent time with them, or wrk2, so feel free to ignore)

On Wed, Oct 7, 2015 at 2:13 AM, Nick Pavlica <lin...@gmail.com> wrote:

After looking at the numbers in the benchmark, I was a little disappointed to see that they were only serving 60K connections

I don't see that in those results.  There are definitely a lot of numbers bigger than 60k here: https://github.com/ptaoussanis/clojure-web-server-benchmarks/blob/master/results/60k-keepalive/20150217-13-18-table.txt - and in the http-kit link you included. 

This was a mistake on my part - the tests are bound at 60k connections.  The numbers I am talking about here are queries/second.

Take care,

Moe

[Charles Harvey III]

Have you had a look at Chronicle? They have built up an entire infrastructure that is off-heap so there is no GC.

http://chronicle.software/products/chronicle-engine/

https://github.com/OpenHFT/Chronicle-Engine

There is a free version and a paid for version. They have broken it into parts so you can use just the off-heap Map/Set/Logger/Queue.

There isn't a web server, just a processing engine, so you would be rolling a lot of it on your own. But it sure is fast.

[Andrey Antukh]

Hi Nick.

I think that you are taking wrong conclusions on wrong assumptions. IMHO

I'm not trying defend clojure or erlang, I like them both. I try to explain me:

When you trying to scale one single machine to 1-2M persistent connections, the main problem will  not be the virtual machine (jvm or beam), will be your operating system (among other factors mentioned later). Additionally, if you will try to scale in this manner in AWS, good luck, but I'm pretty sure you will found you pay much more that you have expected, because virtual machines on aws does not behaves like bare metal and will introduce additional IO latencies that you do not have expecting now.

Scaling horizontally does not implies buy more and more expensive servers. Implies buy more cheaper servers. In my personal and real experience... we have migrated one application from one monolitic high power server to 5-10 small servers (depending on demand), and we have started to have less applications errors, less timeouts, increase the maximum number of possible persistent connections.

Why? The JVM/Beam is not he only thing that runs on your machine, with bigger machine the operating system should handle much more IO, memory for connections, socket buffers, etc... that will implies big latencies and other consequences. You should do very high fine tuning of the operating system for make your OS not obstacle. And forget do it on virtual environment (aws)... Independently if you will use erlang or clojure, vertical scale is not a good approach in much of cases.

Later, you are taking a decision taking microbenchmarks. Is well know that micro benchmarks are very less precise that you are expecting. Have the ability to handle 2M persistent connections does not depends only on the server/language and how they handle the "hello world" request/response, where are other players in the equation: persistence and your business logic, operating system tuning, virtual/bare metal hardware, network hardware...

 

Do you want also have the persistence layer on the same server or in a separated one? Is a persistence layer also accepts 2M persistent connections? I don't mean nothing with it, only I want to say that 1-2M connections is very very big number of connections and is usually reached in very specific situations (for single machine). And the first challenge for to have an application that scales vertically to 2M connections is your code, not the language and its vm...

And finally, having everything in one server make it less fail tolerant, if the machine is gone, everything is gone... 

The golden rule is: "Divide and conquer".

My two cents.

Andrey.

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Andrey Antukh - Андрей Антух - <ni...@niwi.nz>

http://www.niwi.nz

https://github.com/niwinz

[Christopher Small]

I'd just like to point something which may have been glossed over from Timothy's post (and now Andrey's as well).

> ... if you're doing actual processing, handling that many connections on a box seems like premature optimization.

> Have the ability to handle 2M persistent connections does not depends only on the server/language and how they handle the "hello world" request/response, where are other players in the equation: persistence and your business logic, operating system tuning, virtual/bare metal hardware, network hardware...

If you're interested in the question of a maximum number of connections from a purely theoretical standpoint, then great; it's an interesting question. But if you're really thinking about it from the perspective of making an infrastructure decision for a real world project, I'd have to agree with Andrey and Timothy here. In most real world scenarios, you're not going to be able to get even close to those upper limits, and you'll see other aspects of the runtimes become much more important. So I wouldn't put too much weight on this particular question in isolation. If your application's processing per connection really is super minimal with almost 0 real work being done (as in a chat server, as mentioned), then it's a factor worth considering. But most of the time we need to do things with our data, so it's important that we consider the bigger picture.

I'll also echo the sentiment that scaling horizontally over vertically has a lot of advantages, so it's worth considering designing your architecture in this direction regardless of which language you choose.

Best

Chris