KillBill did not generate the Invoice/Payment for the next month

[Pereira]

We have an account configured to bill every day 23 of the month, this account was billing normally for months but suddenly it did not create the invoice for the next month (June this year) and in the following months it continued not to charge either, and I can't see any error log in the month in which the last charge occurred or in the recurring months in which the Invoice should have been created, I also don't see records in the KAUI Timeline, can you guide me on possible causes or tables/evidence that I can look at to try to find the cause?

Thanks. 

[karan bansal]

Hi Pereira,

Could you please confirm your install setup ( Tomcat/AWS/Docket etc. ). I would need the Kill Bill log files ( not necessarily error logs ), for example catalina.out in case of Tomcat installation. Also, please let me know the account ID facing the issue. 

What happens if you try to do a dry run for a future date? Does it show earlier missing charges in that case? 

Please also could you query all blocking states for the particular account using this endpoint https://killbill.github.io/slate/account.html#retrieve-blocking-states-for-account and send me the output. 

Regards
Karan

[Pereira]

Hi Karan, I sent you some evidence on private replying only to you, did you receive them?

Thanks

[karan bansal]

Hi Pereira,

Looks like I didn't get it. Could you please send it to the group, that way anyone on the team can have a look if needed. 

Regards

Karan

[Pereira]

- The blocking state output you requested:

[

{

"blockedId": "631bbd11-d0b2-4096-8052-6c01f2031639",

"stateName": "ENT_STARTED",

"service": "entitlement-service",

"isBlockChange": false,

"isBlockEntitlement": false,

"isBlockBilling": false,

"effectiveDate": "2022-05-23T21:47:12.644Z",

"type": "SUBSCRIPTION",

"auditLogs": []

},

{

"blockedId": "9de6cd7e-1141-4790-bd21-e9e02eddbb31",

"stateName": "SUSPENDED",

"service": "overdue-service",

"isBlockChange": false,

"isBlockEntitlement": false,

"isBlockBilling": false,

"effectiveDate": "2022-09-24T21:47:17.000Z",

"type": "ACCOUNT",

"auditLogs": []

},

{

"blockedId": "9de6cd7e-1141-4790-bd21-e9e02eddbb31",

"stateName": "__KILLBILL__CLEAR__OVERDUE_STATE__",

"service": "overdue-service",

"isBlockChange": false,

"isBlockEntitlement": false,

"isBlockBilling": false,

"effectiveDate": "2022-09-27T22:00:44.000Z",

"type": "ACCOUNT",

"auditLogs": []

}

]

- There's also the KAUI timeline for the account that stopped billing if this happens somehow

The INVOICE was not generated on 2023-06-23 and for the subsequent months. We manually triggered the invoice (where you see "INSERTED by violet") on the right side, after doing that KB was able to fix and generate the following INVOICES by itself and the payment for them also worked after that. So looks like for some reason KB did not generate the Invoice for Month 6, it got stuck and the following months were stuck as well. Any ideas why?

About the logs, wouldn't be possible to send you everything since we have a bunch of log files packed on the same day, can you guide me in things to look for in particular? As I said I can't see any logs in general, looking for AccountId and other Account Attributes.

Also, is there any internal table that we can take a look at to find useful information related to this?

[karan bansal]

Hi Pereira,

Does that mean that after you triggered the invoice manually, the further process of invoice generation resumed normally? If yes, then it is possible that the system faced some unexpected issue when trying to generate invoice for 23rd June, for example, a database lock, that caused the invoice generation to fail as well as to not update the future events/dates properly, consequently future invoices also were not generated.  

The only way to confirm what happened would be to check the logs from 22nd/23rd June, may be use the account ID to filter down the logs, though I doubt such old logs will still be there. As long as the invoice generation resumed fine, I would not read too much into this issue. 

Also, would suggest you to look into upgrading to the latest version of Kill Bill, where we have added resilience for some unexpected failures.

Regards

Karan

[Pereira]

Yes, after triggering the Invoice manually things worked just fine.

About the logs, couldn't find anything in the whole month of June, using accId or other account attributes.

That's just one case we found from nowhere, considering we couldn't determine the root cause we're at least trying to map the affected accounts, do you know some query we can run against KB tables or a APIs to search accounts that are in this state? that is, the INVOICE was not generated. By tracking that we should be able to manually fixing them.

Thank you for now Karan.