We have quite a unique problem. We run KillBill version 20.10 as a high availability set-up on AWS. We have KillBill as containers in Fargate and then database Aurora MySQL. Database version of MySQL was 5.6 but we now migrated to 5.7.
We used AWS Green-Blue deployment so new 5.7 version was created and then there was a switchover. Everything seemed to work, but we got this in Killbill health check
"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)", "java.lang.Thread.run(Thread.java:748)" ],
"cause" : {
"message" : "Communications link failure with primary. No active connection found for replica. ",
"stack" : [ "org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:175)", "org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110)", "org.mariadb.jdbc.MariaDbDataSource.getConnection(MariaDbDataSource.java:316)", "com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:444)", "com.zaxxer.hikari.pool.BaseHikariPool$1.run(BaseHikariPool.java:419)", "java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)", "java.util.concurrent.FutureTask.run(FutureTask.java:266)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ],
"cause" : {
"message" : "Communications link failure with primary. No active connection found for replica. ",
"stack" : [ "org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:477)", "org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:175)", "org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:162)", "org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:99)", "org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:477)", "org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)", "org.mariadb.jdbc.MariaDbDataSource.getConnection(MariaDbDataSource.java:313)", "com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:444)", "com.zaxxer.hikari.pool.BaseHikariPool$1.run(BaseHikariPool.java:419)", "java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)", "java.util.concurrent.FutureTask.run(FutureTask.java:266)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ],
"cause" : {
"message" : "Could not connect to HostAddress{host='
ssdatabase-name-green.eu-central-1.rds.amazonaws.com', port=3306}.
ssdatabase-name-green.eu-central-1.rds.amazonaws.com",
"stack" : [ "org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:175)", "org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.connException(ExceptionMapper.java:83)", "org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:422)", "org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:155)", "org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:207)", "org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:159)", "org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:99)", "org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:477)", "org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)", "org.mariadb.jdbc.MariaDbDataSource.getConnection(MariaDbDataSource.java:313)", "com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:444)", "com.zaxxer.hikari.pool.BaseHikariPool$1.run(BaseHikariPool.java:419)", "java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)", "java.util.concurrent.FutureTask.run(FutureTask.java:266)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ],
"cause" : {
"message" : "
ssdatabase-name-green.eu-central-1.rds.amazonaws.com",
"stack" : [ "java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)", "java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)", "java.net.Socket.connect(Socket.java:589)", "org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:452)", "org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:419)", "org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:155)", "org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:207)", "org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:159)", "org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:99)", "org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:477)", "org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)", "org.mariadb.jdbc.MariaDbDataSource.getConnection(MariaDbDataSource.java:313)", "com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:444)", "com.zaxxer.hikari.pool.BaseHikariPool$1.run(BaseHikariPool.java:419)", "java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)", "java.util.concurrent.FutureTask.run(FutureTask.java:266)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ]
In short, we would be interested to knowing how to change the Database address where KillBill is trying to connect? We cannot understand how this has even been changed as we did nothing to update it.
I don't know if I managed to explain the problem really well, anyway all help appreciated as now after an attempt to fix by re-starting both instances, we are in a situation where every KillBill operation gives 401 :(