Sudden increase in MySQL prepared statements with Kill Bill

[Pardos Dedondo]

Hello everyone,

I’m running Kill Bill 0.24.9 on Tomcat 8.5.98 with MySQL 8.0.32, and I recently started experiencing the following error:

Can't create more than max_prepared_stmt_count statements

What puzzles me is that nothing has changed in my setup or workload compared to the past. For a long time, the system was stable around ~14,000 prepared statements (with MySQL’s default max_prepared_stmt_count set to 16382), but now the counter keeps growing until it hits the limit.

Here is my current configuration:

• org.killbill.dao.maxActive = 200

• all prepStmtCacheSize = 500

• MySQL max_prepared_stmt_count = 16382

I followed the usual checks:

• Monitored Prepared_stmt_count over time

• Verified the number of active connections in the pool

• Reviewed JDBC driver parameters (useServerPrepStmts, cachePrepStmts, etc.)

• Confirmed that there were no changes in Kill Bill, plugins, or workload

So far, I don’t see anything unusual, and the increase in prepared statements seems to have started happening “out of the blue.”

Has anyone else encountered this behavior with Kill Bill 0.24.x or with MySQL Connector/J? Could this be related to a driver update or a subtle configuration change in how prepared statements are handled (server-side vs. client-side)?

Any insights or suggestions would be greatly appreciated.

Thanks a lot,

Pardos

[karan bansal]

Hi Pardos,

is this issue happening at any particular time/during any particular process OR is it happening randomly. Could you please confirm what all plugins are installed in your KB setup.

Regards

Karan

[Pardos Dedondo]

Hi Karan,

We noticed a significant increase during the phase of massive subscription creation, whereas throughout the rest of the day the growth is more gradual and not as sudden as in the first case. Also, we rarely observe a decrease in the number of prepared statements in cache.

[karan bansal]

Hi Pardos,

Have you tried setting the prepStmtCacheSize to a higher number like 1000. 

Regards

Karan