KillBill gives just 401 after MySQL database update

21 views
Skip to first unread message

Jani Helin

unread,
Mar 2, 2023, 6:07:21 AM3/2/23
to Kill Bill users mailing-list
Hey Killbill Gurus,

We have quite a unique problem. We run KillBill version 20.10 as a high availability set-up on AWS. We have KillBill as containers in Fargate and then database Aurora MySQL. Database version of MySQL was 5.6 but we now migrated to 5.7.

We used AWS Green-Blue deployment so new 5.7 version was created and then there was a switchover. Everything seemed to work, but we got this in Killbill health check 

"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)", "java.lang.Thread.run(Thread.java:748)" ],
      "cause" : {
        "message" : "Communications link failure with primary. No active connection found for replica. ",
        "stack" : [ "org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:175)", "org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110)", "org.mariadb.jdbc.MariaDbDataSource.getConnection(MariaDbDataSource.java:316)", "com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:444)", "com.zaxxer.hikari.pool.BaseHikariPool$1.run(BaseHikariPool.java:419)", "java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)", "java.util.concurrent.FutureTask.run(FutureTask.java:266)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ],
        "cause" : {
          "message" : "Communications link failure with primary. No active connection found for replica. ",
          "stack" : [ "org.mariadb.jdbc.internal.failover.AbstractMastersListener.throwFailoverMessage(AbstractMastersListener.java:477)", "org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.checkInitialConnection(MastersSlavesListener.java:175)", "org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:162)", "org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:99)", "org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:477)", "org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)", "org.mariadb.jdbc.MariaDbDataSource.getConnection(MariaDbDataSource.java:313)", "com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:444)", "com.zaxxer.hikari.pool.BaseHikariPool$1.run(BaseHikariPool.java:419)", "java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)", "java.util.concurrent.FutureTask.run(FutureTask.java:266)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ],
          "cause" : {
            "message" : "Could not connect to HostAddress{host='ssdatabase-name-green.eu-central-1.rds.amazonaws.com', port=3306}. ssdatabase-name-green.eu-central-1.rds.amazonaws.com",
            "stack" : [ "org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:175)", "org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.connException(ExceptionMapper.java:83)", "org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:422)", "org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:155)", "org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:207)", "org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:159)", "org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:99)", "org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:477)", "org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)", "org.mariadb.jdbc.MariaDbDataSource.getConnection(MariaDbDataSource.java:313)", "com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:444)", "com.zaxxer.hikari.pool.BaseHikariPool$1.run(BaseHikariPool.java:419)", "java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)", "java.util.concurrent.FutureTask.run(FutureTask.java:266)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ],
            "cause" : {
              "message" : "ssdatabase-name-green.eu-central-1.rds.amazonaws.com",
              "stack" : [ "java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)", "java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)", "java.net.Socket.connect(Socket.java:589)", "org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:452)", "org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:419)", "org.mariadb.jdbc.internal.protocol.AuroraProtocol.loop(AuroraProtocol.java:155)", "org.mariadb.jdbc.internal.failover.impl.AuroraListener.reconnectFailedConnection(AuroraListener.java:207)", "org.mariadb.jdbc.internal.failover.impl.MastersSlavesListener.initializeConnection(MastersSlavesListener.java:159)", "org.mariadb.jdbc.internal.failover.FailoverProxy.<init>(FailoverProxy.java:99)", "org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:477)", "org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)", "org.mariadb.jdbc.MariaDbDataSource.getConnection(MariaDbDataSource.java:313)", "com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:444)", "com.zaxxer.hikari.pool.BaseHikariPool$1.run(BaseHikariPool.java:419)", "java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)", "java.util.concurrent.FutureTask.run(FutureTask.java:266)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ]


I have replaced the DB name with "ssdatabase-name-green.eu-central-1.rds.amazonaws.com". Anyway the problem seems to be that KillBill trieds to connect to the temporary "green" database, and we cannot figure out how to get KillBill to update the address.

In short, we would be interested to knowing how to change the Database address where KillBill is trying to connect? We cannot understand how this has even been changed as we did nothing to update it.

I don't know if I managed to explain the problem really well, anyway all help appreciated as now after an attempt to fix by re-starting both instances, we are in a situation where every KillBill operation gives 401 :( 

Br,
Jani Helin

Jani Helin

unread,
Mar 2, 2023, 6:43:28 AM3/2/23
to Kill Bill users mailing-list
Update to anybody reading this: In the end the problem was in AWS environment set-up, where unlucky DNS cache update hit us at the worst time. We fixed by creating new configuration for the AWS task with correct value for the "KB_org_killbill_dao_url" environment variable.

So in the end this was AWS environment related problem, not a problem with Killbill itself diretly.

karan bansal

unread,
Mar 2, 2023, 7:46:02 AM3/2/23
to Kill Bill users mailing-list
Hi Jani,

Glad to know that it worked. Please let me know in case you face any further issue.

Regards
Karan
Reply all
Reply to author
Forward
0 new messages