HikariCP get Connection takes very long

[Madhuri Inuganti]

Hello,

Hortonworks Schema Registry Version : - 0.5.4

Kubernetes Application 

Database : Mysql (db.r3.large 8 gig)

We have Hortonworks Schema registry deployed on our dev stage environments and works fine. However we tried a production roll yesterday and it was throttled and couldn't handle the load.  Our new relic logs show below. 

HikariCP get Connection is taking the most time up to 27 seconds under load. We process a load of 1200 records per second on average. We are wondering if we should increase the connection pool size and if so how do we configure that. 

Looking at the source code, we see queryTimeoutInSecs: 30 is configurable. but we don't see a way to configure the connection pool size. And even wondering if that is the right thing to do or if there is anything else to handle the volume.

Category Slowest components Count Duration % Java HikariDataSource.getConnection() 1 27,100 ms 96%