Killbill consuming CPU - 100% CPU Load at all times

[Kostas Galanis]

Hi we are facing an issue in our production setup with high CPU Load. The systems are in 100% CPU Load at all times.

The setup:

-Cloud provider: Microsoft Azure

-Installation Type: Docker in Virtual machine.

-Virtual Machines: 2nodes with 8vCPU/32BG RAM each.

-Database: Azure MySQL Flexible Server. The load in the database is low at all times so we let's remove it from the equation.

-shiroNbIterations: Default

-Customers: about 2500K with 2 Subscriptions that renewed every month in different dates but one of them is renewed at the 2nd day every month.

-Plugins:

  --All Plugins are using different Azure MySQL Flexible instance in the same region. again the database is sleeping (no load or blocking queries have been detected)

  --Catalog Plugin with historic versions per customer. 

  --Invoicing Plugin and Payments Plugin. We got notified for purchases and renewals 

  --Stripe Plugin is enabled.

  --Subscriptions Plugin. we got notified for all immediate actions in subscriptions such as cancellations with future events

Killbill

kbVersion": 0.22.32

apiVersion": 0.53.17

pluginApiVersion: 0.26.4

commonVersion: 0.24.21

platformVersion: 0.40.13

The problem is pretty much explained in the title and in the first sentence. We are facing 100% CPU Load (meaning that with 8vCores 800% CPU) in both nodes at all times. 
The busses (main and external) have less than 5 items in the queue and IN PROCESSING state, we don't have items with past effective_date in AVAILABLE State but we see that most of the times the system cannot process and re-queuing the events.

We found an item in the bus history that have been process more than 700 times. 

We support both payments as Topups (purchase without subscriptions) and payments with purchase a subscriptions but we don't currently have more than 20 transactions per day. 

Is there any suggestion in the configurations or any idea on why we see this CPU Load? 

Thank you,

[stephane brossier]

Is your deployment vanilla, or do you run custom code (e.g plugins)? Also, if this is the case, can you do a jstack and see what is currently happening?

--
You received this message because you are subscribed to the Google Groups "Kill Bill users mailing-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to killbilling-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/2e3607a3-d1e8-43c9-9ce3-2ead9561e861n%40googlegroups.com.