HTTP server performance

[Mirek Suk]

Hi everyone.

AKKA 2.5.9 / HTTP 10.0.11

As part of prototyping architecture, I ran some benchmarks (including http) few months ago.
A simple hello world (request handler is completed with result immediately) had nice performance ~2.7m requests/minute. with 1,6m r/m when response included ask to sharded actor.

Recently I re-did those benchmarks to find out around ~9x performance drop on original test system.
Rolled prototype back to point of original test with similar result.
Built scala http sample with similar result.
Reinstalled JVM with similar result.
Randomly got performance back, until system restart (yeah like what ?)

On EC2-T2.medium I can do ~7k req/s to "hello world" page with 100% CPU usage (2 cores).
No akka.http configuration is changed and system does no background processing.
Default dispatcher is unchanged (only due to clustering throughput = 10).

Is this kind of performance expected ?
Is there something I can try ?

Thanks,
M.

[Mike Nielsen]

Did you try it on a new instance of the same size?

At my work we occasionally see EC2 instances start getting really poor performance.

When we replace them with a new instance (same instance type) performance comes back.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Viktor Klang]

Mirek,

As a general note: Benchmarking using default configuration is almost pointless* since a default configuration is—by definition, and by necessity—a configuration which will not be *optimal for a specific use-case*.

*It can be useful as a baseline for tuning.

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Mirek Suk]

There is not much to change in akka http server configuration. Care to share any tips ?

I've tried fidling with io.tcp with little luck. (accept batches etc...)
EC2 instances are restarted during every deployment.

Anyways I'm currently in process of migration of http interface to finangle as it seems to have ~2.5x better throughput with less cpu consumed.
But I still wonder where the performance I initially measured went. (I blame the meltdown patch on windows secretly lol)

Thanks

[Viktor Klang]

The top hit for me when searching for "tuning akka http" is: https://stackoverflow.com/questions/41297334/akka-http-performance-tuning

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Mirek Suk]

Thanks, but been there seen that, I would not be here without simple google search.

Guy used blocking operations which limited his throughput. We - as far as I know and as far cpu usage says - don't.
For http benchmark i use - as I mentioned - simple complete("hello world")
For normal operation we use ask pattern which creates temporary actor and complete requests in async manner.

M.

[Viktor Klang]

Ok. So besides the common dispatcher tuning tips, JVM tuning tips, and Linux IO tuning tips there's also a lot of settings for Akka Http specifically.

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Mirek Suk]

Only a few of them could affect CPU consumption and throuput, unless they mean something else.
Besides that as I mentioned in op throuput degradation was "random" and on same system. Tunning (in java) usually means ~50%. As I mentioned I'm experiencing 9x performance drop from initial benchmarks.

As a sidenote
JVM tuning: min, max memory = 2gb (bit op, but due to sharding state),disable shared memory (akka is distributed as docker)
Linux IO tuning: netty (both v3 and v4 as akka uses v3) epoll loop used.

Performance degradation happened on windows server machine used for initial evaluation. I sadly never did benchmarks on EC2 linux instances initially as we were bound for Azure, so my reference point sucks a bit.
I still do consider 7k requests with 1000milicores on 2xcpu a very miserable result.

My experience with scala/java is very limited - 2 months (c++ guy here) so my expectations may be above what jvm is capable of.
Benchmark from someone else would probably help, but akka http benchmarks are scarce.

Besides that I'm unsure why do I answer to non constructive posts. Probably because I needed help which won't be found here.

Thanks.

[Viktor Klang]

What dispatcher tuning did you experiment with?

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Mirek Suk]

I doubt dispatcher is problem as forkjoinpool with default (and lesser settings due to tests required by ops) were tried.
No other dispatches would make sense in our case.

As a sidenote grpc.io endpoint which does
- route to akka cluster instance where state is via sharding actor (artery is used)
- persistent actor modifies state (cassandra cluster)
- route response back and respond to grpc call

This endpoint has performance of ~6k req/sec (akka is cpu bound here) which (considering the amount of work) is "nice" (once again I see performance metrics in c++ so I'm biased, but rapid development in akka somewhat erases some issues for me)

So I don't see where http endpoint could spend so much time if it has only ~15% more performance over grpc.

Once again my main issue here is original vs current tests.
Sadly I can find proof for both cases - akka http unacceptable/acceptable - online. One here, second on unrelated site.
However nothing still explains performance degradation, except for my error.
https://groups.google.com/forum/#!topic/akka-user/qhZlh0KBl2A
https://www.techempower.com/benchmarks/

I did not study methodologies used to validate any of those benchmarks.
Possibly akka team should do that so we have reference point.

M.

[Viktor Klang]

On Tue, Feb 20, 2018 at 11:46 PM, Mirek Suk <mira....@gmail.com> wrote:

I doubt dispatcher is problem as forkjoinpool with default (and lesser settings due to tests required by ops) were tried.

What do you mean by "lesser settings"? How did changing pool size and/or throughput affect your bench?

What tool are you using to bench it with?

 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Mirek Suk]

Lesser = less maximum threads. Akka has minimum which is "high" for 2 core machines. = in error margin difference +-~5%
throuput on default displtcher changed to 100 and 1 (clustering changes this to 10 from 1)
1 = worse performance (around -20%, but once again consider artery in play, with default remoting I assume it would be worse)
100 = in error margin.

I even tried to create "local" actorSystem for services which are local to each akka pod/node (in jvm process) instead of using clusered one. made no difference.
Noticeable difference in grpc was
https://grpc.io/grpc-java/javadoc/io/grpc/ServerBuilder.html#directExecutor--
around 10% in overall cpu consumption. meaning routing "something" from netty thread to executor is somewhat slow. (considering we only do ask in grpc)

Tools
- jmeter (for "hello world" benchmark)
- custom device simulator. We're prototyping IOT server app.

M.

[Viktor Klang]

What does wrk2 say w.r.t. throughput?

Are you testing pipelined requests or new connections for each request? Are you testing over network or not?

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√