Payara Configuring for 20,000 Transactions/Second

[Edwin Amoakwa]

Hello,

I have a simple Restful application (JAX-RS)  that serves as an api-end point to other application and need to achieve at least 20,000 transactions per second. 

The simplest end endpoint just takes a url parameter and persist it to the database using JPA. 

I have taken all the under-listed steps, and still unable to achieve an average of 1,500 transaction per second. 

Actually, I get timeout errors seconds after starting load test, which is also less than http-timeout configuration of 30 secs

Any further recommendations on how to achieve 20,000 transactions will be greatly appreciated.

Fine-Tuning 

I followed this tutorial http://blog.payara.fish/fine-tuning-payara-server-in-production

and fined-tuned it with the following configuration

JVM Options

Changed to -Xmx4g

Changed to -Xms4g

EJB Pool Settings

Initial and Minimum Pool Size: 2000

Maximum Pool Size: 5000

Pool Resize Quantity: 8

HTTP Network Listeners

Http Max Connections : 5000

Timeout : 30 sec

Thread Pools

Max Thread Pool Size: 500

Min Thread Pool Size: 100

JDBC Connection Pool

Initial and Minimum Pool Size : 1500

Maximum Pool Size: 2000

Pool Resize Quantity:50

Clustering & Load Balancing

I followed the clustering and load balancing tutorials at http://blog.payara.fish/creating-a-simple-cluster-with-payara-server and http://blog.payara.fish/payara-server-load-balancing-with-nginx 

I have created three (3) instances in one cluster, and load balance to all the three instances

Code Changes/Reactive Imprementation

I followed  some recommendation of @OMihalyi at https://www.youtube.com/watch?v=P8I7ard9p9w on reactive techniques and changed all the rest api to use AsyncResponse of JAX-RS 2.0.   

[Ondrej Mihályi]

Hi Edwin,

It would be interesting to see the code of your application.

In order to rule out that any pool limits the throughput, I would try increasing maximum sizes of all connection and thread pools to a value that is permitted by the OS (for threads this can grow even to millions, for connections this can be tens of thousands and more). Also, try monitoring the pool statistics (either by JMX, REST endpoints in Admin console, or Admin console GUI) to find out how much the pools are utilized. If the number of busy threads / used connections is at the maximum size allowed, you definitely need to raise the maximum or add more nodes to the load-balancer.

If you are using any background tasks (e.g. via ManagedExecutorService or @Asyncrhonous EJB methods), you should also configure the thread pools for those, as they use another thread pool as the HTTP listener. You can find the configuration in Concurrency configuration in Admin Console.

Also monitor heap usage in JVMs, as GC may be a big factor slowing your app down. You can monitor heap usage either using JMX, or by logging JVM usage using the -Xloggc JVM option (but logging to a file may further slow your app down). Also check if you don't log too much into the logs, as also degrades performance.

Ondrej

[Mike Croft]

It may be useful to look at the request tracing service, which Ondrej has blogged about, to help you see which part of your scenario is taking the most time:
http://blog.payara.fish/request-tracing-service-in-payara-server-payara-micro

It's very important to make changes based on data and evidence, rather than making assumptions, since things might get worse instead of better! As Ondrej says, verbose GC is incredibly useful and I would recommend enabling it in production from day 1. The overhead is so low as to be almost negligible and the benefits can be huge:
http://stackoverflow.com/a/15319823/212224

[Edwin Amoakwa]

Hello,

I modified the code to this simple version, and removed all database/JPA stuff.

It just logs the parameter and return true to the caller.

And I still get timeout seconds after starting the test.

Not sure what I am still missing. I can share the access to the server if needed

** Thread Pool settings is as below

Max Thread Pool Size: 5000

Min Thread Pool Size: 1000

/**

 *

 * @author Edwin

 */

@Path("api")

@Stateless

public class SimpleRest

{

    private static final Logger LOG = Logger.getLogger(SimpleRest.class.getName());

    

    @Inject private CrudService crudService;

    @Resource

    ManagedExecutorService managedExecutorService;

    

    @GET

    @Path("onboard")

    @Produces(MediaType.TEXT_PLAIN)

    @Consumes(MediaType.APPLICATION_FORM_URLENCODED)

    public void onboard(@Suspended final AsyncResponse asyncResponse, @QueryParam("phoneNo") final String phoneNo)

    {

        asyncResponse.setTimeoutHandler(new TimeoutHandler()

        {

            @Override

            public void handleTimeout(AsyncResponse asyncResponse)

            {

                LOG.info("Time out occured processing ... " + phoneNo);

            }

        });

        

         managedExecutorService.execute(new Runnable()

        {

            @Override

            public void run()

            {

                try

                {

                    

                    LOG.info("Received for processing ... " + phoneNo);

                    asyncResponse.resume(Response.ok(true).build());

                    

                } catch (Exception ex)

                {

                    asyncResponse.resume(Response.ok(false).build());

                }

            }

        });

    }

    

    

}

[Andy Bailey]

Hi Edwin,

I would recommend reducing those thread pool limits as they can cause things to slow down a lot because of Thread context thrashing.

Part of the problem here is that standard Jersey, while it does a lot of the work for you, it does add a lot of overhead.

I recommend using a Java APM (Application Performance Monitor) such as Glowroot to see where the bottlenecks may lie.

At the end of the day 20k transactions/second is ambitious and not something you may achieve simply by load balancing.

You might want to look at using gRPC which uses binary protocols for web services rather than the entire HTTP stack.

[Ondrej Mihályi]

Hi Edwin,

Andy might be right about the number of threads. I know I recommended increasing the size of the pools, but I forgot about context switching when too many threads are running at the same time. It really depends on where the bottleneck is - if it is the CPU, then you need fewer threads to avoid frequent context switching. If it is the resources (e.g. the DB), then increasing the number of threads may help to avoid having most of the threads blocked, waiting for resources.

The most important thing is to identify what is slowing things down. For that, Payara offers request tracing service, which I mentioned earlier - just configure a threshold for the length of your requests, and the information about long running requests will be dumped to the log. From there, you will find out where your requests spend most of the time.

Ondrej

[Edwin Amoakwa]

Hello Andy,

First of all, I agree that 20K transactions per second is very very ambitions. This record was set because a component of the system was implemented using Vert.x, which easily achieved that without much configuration at all. Hopefully, this number might be reviewed downwards during the final system audit and quality assurance.

We used newrelic.com and that was helpfull in giving some insight. Thanks for introducing https://glowroot.org I will spend some time with it in the coming weeks.

As to the problem being Jersey, not to sure about that assertion, because we did a very simple test (in JavaSE)  using NettyHttpContainerProvider of Jersey and the numbers where far higher that what we achieved using JavaEE/Payara. However, re-writing the code that style wasn't just an option, because we used several API's of JavaEE.

In my opinion, it think it has a lot more to do with the http-provider (grizzly)  or the the container than Jersey.

Will also take a look at  gRPC.

Andy, Thanks for your contribution.