Want to Contribute Kafka Consumer Plugin

[Prashant Kumar]

Hello Kill Bill Team, Shaun Forgie
I'm excited to make a valuable contribution to the community by sharing a Kafka Consumer Plugin that I believe will be beneficial.

The motivation behind this plugin is to address a specific use case, as illustrated in the architecture diagram below. In the Kill Bill ecosystem, services typically interact with Kill Bill through its API. Each service operates within its own tenant in Kill Bill, with each tenant having its own set of accounts. These accounts are subscribed to plans using unique external keys.
In many scenarios, services need to push account events using the record usage for the subscription API. This involves regular API calls to Kill Bill, as depicted in the diagram.

In the event of a production outage in Kill Bill due to external issues such as development changes or failures in external services, there may be a period during which Kill Bill is inaccessible. Although developers can quickly revert changes and restore Kill Bill, there is a gap in service availability during the downtime. This poses a significant challenge, particularly for other services that continue to operate and attempt to push metrics using the API.

Each service typically has its own client implementation to interact with Kill Bill, and these implementations may or may not include retry mechanisms. As a result, the inability to access Kill Bill during downtime can lead to operational nightmares for developers and businesses alike. It impacts the overall product services and disrupts business operations.

To address this challenge, we proposed the use of Kafka, where every service can publish its metrics to a Kafka topic. Kill Bill can then consume these messages using the Kafka Consumer Plugin. Even in the event of downtime in Kill Bill, other services can continue to push their usage metrics to the Kafka topic. When Kill Bill resumes operation, it can still consume the messages that were pushed during the downtime. This ensures greater reliability of the service and minimises disruptions to business operations.

I'm looking to contribute a Kafka Consumer Plugin to the community. This plugin focuses solely on consuming usage metrics, as it's the most frequently used API when other services are pushing metrics. Additionally, other developers can create similar Kafka consumer plugins for their use cases by referencing this plugin. Could you please guide me on how to get started with contributing a plugin to the community?

Thank you,
Prashant Kumar

[Shaun Forgie]

HI Prashant,

Firstly it is good to see you are keen to contribute to the Kill Bill product. It is good for all organisations in the community to have useful contributions like this being planned.

Documentation on plugins can be found here https://docs.killbill.io/latest/plugin_introduction. So familiarity with java based OSGi modules will be important. Started templates are available in github.

You have provided some high level generic ideas about how the plugin would pull data out of a Kafka queue which is a good start...but further details are required. 

I would like to hear more about how your design works for more than one billing tenant, what sorts of validation is going to be applied to usage messages, the message formats and how errors are to be handled.

The solution should also be aware of tenant specific usage billing settings that relate to past billing periods and have rules about accepting forward dated transactions.

In terms of protecting the integrity of the billing data you will also need to address problems related to processing the same message more than once and dealing with the accidental resubmission of duplicate usage records into the kafka queue.

You will also need to support different run time topologies as more than one instance of Kill Bill can be running. This will mean potentially more than one consumer could be connected to kafka at any one time.

Look forward to discussing this more with you.

Rgs

Shaun

--
You received this message because you are subscribed to the Google Groups "Kill Bill users mailing-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to killbilling-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/74b91c05-62ae-4889-b6ec-4bf102824e47n%40googlegroups.com.

--

Shaun Forgie [Principal] - Method Maker Ltd
57a McIntyre Rd, Manukau 2022, Auckland, New Zealand
Mobile +64 21 666 910

[Prashant Kumar]

Hello Shaun Forgie and the Kill Bill team,
Thanks for you reference. I already created kafka Consumer Plugin which using java based OSGI module. Currently, we are utilising this plugin for our internal use cases, and we believe it would be valuable to contribute this plugin to the wider community.

Lets come to each queries:
1. I would like to hear more about how your design works for more than one billing tenant, what sorts of validation is going to be applied to usage messages, the message formats and how errors are to be handled.

The solution should also be aware of tenant specific usage billing settings that relate to past billing periods and have rules about accepting forward dated transactions.

Our design follows this structure: Each service, particularly those whose subscriptions and accounts are associated with a specific tenant, pushes messages in JSON format similar to the push usage API. These messages include additional fields such as apiKey. Utilising the API Key, we retrieve the tenant, which require in setting up the PluginCallContext.
With the OSGIKillBillAPI interface, we can directly push this usage data using osgiKillbillAPI.getUsageUserApi().recordRolledUpUsage().
Here's an example JSON payload which look similar to Record usage for a subscription
{
  "subscriptionId": "1b652c50-6ced-4335-bd6b-daf70a6c65e0",
  "trackingId": "1",
  "apiKey": "router",
  "unitUsageRecords": [
    {
      "unitType": "txnCount",
      "usageRecords": [
        {
          "recordDate": "2024-04-20T02:59:33.147",
          "amount": 25
        }
      ]
    },
    {
      "unitType": "txnVolume",
      "usageRecords": [
        {
          "recordDate": "2024-04-20T02:59:33.147",
          "amount": 4000
        }
      ]
    }
  ]
}

Here's another aspect to consider: Other services responsible for pushing usage data only have knowledge of the external Subscription Key used to create the subscription. These services should not rely on the getSubscriptionByExternalKey API call to retrieve the subscription ID. It's more efficient for each service to manage its subscription information rather than relying on Kill Bill to fetch the subscription ID through an API call as our idea is to not depend on API calls.

Furthermore, we've made an enhancement where each service is treated as its own tenant. Therefore, instead of making an API call to retrieve the tenant ID, we pass the tenant ID directly from the service itself to reduce  getTenant calls at the consumer end. However, this approach can be adjusted to include the API key if necessary.
During Kafka consumption, we first retrieve the subscription using the external Subscription ID. This step  itself do the validation
that, that particular sub belong to tenant or not. and also if that tenant exist or not by throwing the UsageApiException, SubscriptionApiException and TenantApiException. If the tenant and the subscription is valid, we proceed to push the usage data directly using osgiKillbillAPI.getUsageUserApi().recordRolledUpUsage().
Since we're using the osgiKillbillAPI interface, it handles the tenant-specific usage billing settings seamlessly.
Here's an example of the JSON payload:
{
  "externalSubKey": "Service1_merchantId_planName",
  "trackingId": "1",
  "tenantId": "9a652a50-6ced-4335-bd6b-daf70a6c65e0",
  "unitUsageRecords": [
    {
      "unitType": "txnCount",
      "usageRecords": [
        {
          "recordDate": "2024-04-20T02:59:33.147",
          "amount": 25
        }
      ]
    },
    {
      "unitType": "txnVolume",
      "usageRecords": [
        {
          "recordDate": "2024-04-20T02:59:33.147",
          "amount": 4000
        }
      ]
    }
  ]
}

Now come to next query regarding 
2. In terms of protecting the integrity of the billing data you will also need to address problems related to processing the same message more than once and dealing with the accidental resubmission of duplicate usage records into the kafka queue

Since if duplicate message come, there tracking id will remain same. So, if two message processing parallel, only one usage will be added. Hence idempotency will be maintained.

3. You will also need to support different run time topologies as more than one instance of Kill Bill can be running. This will mean potentially more than one consumer could be connected to kafka at any one time.

To address the query regarding multiple instances running in parallel and connecting to Kafka, let's delve into how consumers interact with Kafka. Each consumer is associated with a group ID, which automatically manages consumption across multiple instances. In the case of our KafkaConsumer Plugin, all instances will share the same group ID.

Now, consider there are three partitions for a topic, each containing its own set of messages. If there's only one instance available, all three partitions will be assigned to a single instance, effectively allowing one consumer to consume from all partitions. 

However, if two consumers/instance are present. Since the consumer group id remain same,  one partition will be assigned to one instance, while the other two partitions will be assigned to the second instance. Similarly, if there are three instances, each partition will be assigned to one instance. This distribution process is also known as horizontal scaling.

In the event that multiple instances are in operation, there's no issue because each message resides in only one partition. Even if the same message is inadvertently pushed twice into different partitions and consumed by two different consumers, only one instance will process it since they share the same tracking ID. This ensures that duplicate processing of the same message is effectively prevented.

                           .                 



Hope, I explained all your queries. Please let me know if you have any query regarding this. And also let me know if we are ready to contribute.

Thanks and regards

Prashant Kumar

[Prashant Kumar]

Hi KillBill Team,
Any update on this?

Thanks and Regards
Prashant Kumar

[Pierre-Alexandre Meyer]

Hi Prashant,

This is exciting work!

What kind of support are you looking for from the core team? Have you already published the code to GitHub? As a next step, maybe share the link here for the community to take a look?

Kind regards,

To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/08767b6f-026d-471f-88a3-767fd7652aaen%40googlegroups.com.

--

Pierre

[Prashant Kumar]

Hi Pierre,
For our business use case, we've created a plugin in our private Bitbucket repository. Should I first add the code to my personal GitHub repository before sharing it with you for community contribution?

Waiting for your response
Thanks and Regards
Prashant

[Pierre-Alexandre Meyer]

Hi Prashant,

GitHub isn't a requirement, you can share the code via other source code repositories.

To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/7a4d6c5e-a98e-48a6-9608-28e45704117an%40googlegroups.com.

--

Pierre

[Shaun Forgie]

HI Prashant,

In terms of protecting the integrity of the data within the system it would also be useful to ensure that usage records can not be recorded against unit types that are not defined in the catalog. As new versions of catalogs can be loaded over time you will also need to ensure that usage unit types defined in previous catalogs can also be accepted. This will be important when processing usage records when a new billing catalog is published.

An existing issue around recording usage records with the wrong usage unit type can be found here https://github.com/killbill/killbill/issues/2013

It may also be worth considering implementing most of the core logic in a generic message broker consumer plugin to capture generalised system validations and error handling that are generic across all message brokers. Where possible put this into a core / common area so that it can be easily reused by other message broker implementations. For example ActiveMQ and RabbitMQ are likely to be very common alternatives that others are using.

Rgs

Shaun

To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/54ce3111-c588-46cb-879a-7a800dde0ed2n%40googlegroups.com.

[Prashant Kumar]

Hi Pierre, Shaun, killbill team
I have implemented validation for the usage unit to ensure that it is only added to the rolled_up_usage table if it is part of the active plan. I have pushed my code to my GitHub repository:  prashant280920/killbill-kafka-consumer-plugin. 

Please review the code and let me know how I can contribute this plugin to the project. I believe this will be a valuable contribution for users who need stream integration with Killbill instead of relying on API calls to push usage. This approach ensures 100% usage consumption during downtime and can serve as a reference for others to write similar plugins for their own use cases.

Looking forward to your feedback.
Best regards,

Prashant Kumar

[stephane brossier]

Hi Prashant,

I took a quick look, good work!

One thing regarding the groupId (which is currently commented in your pom.xml) and also the package name you are using: Since you own this plugin, it may make sense to use your own groupId (and ideally matching java package) so you can release the plugin in public sonatype.

If you are interested in a more detailed review, I could provide some comments, but I would need you to open a PR.

Thanks,

Stéphane

To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/cbe2db84-a7f8-42f1-8d1e-435b9fbe9de0n%40googlegroups.com.

[Shaun Forgie]

HI Prashant,

A good start indeed. I agree with Stephane in that you should adjust your package naming to reflect your contribution. Would your employer cashfree mind you setting up a github account to host this code and or using their domain name in the package name? 

I have also created a few issues / enhancement requests to track.

https://github.com/prashant280920/killbill-kafka-consumer-plugin/issues/2

Rgs

Shaun 

To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/CAKuqY316_T--ur3OM1v5FHJWFsSqhqm116OcupNsTV66n6rtbw%40mail.gmail.com.

[Prashant Kumar]

Hi Stephane,
I understand the changes I need to make regarding the group ID and package name. However, I need clarification on where to raise the PR. Can you please guide me on the appropriate repository or platform for submitting my pull request?

Thanks and regards
Prashant Kumar

[Prashant Kumar]

Got It
Okay just a quick question. How will this count as a contribution to your organization if the entire codebase is in my repository? How will others become aware of this plugin?

Thanks and regards

Prashant Kumar

[stephane brossier]

The PR could be between two branches within your own repo just so we have a way to look at a 'diff' and provide comments - that is if you want us to provide comments.

Thanks,

Stéphane

To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/0497833d-07f0-4352-8a08-3e80d36e644fn%40googlegroups.com.

[stephane brossier]

Hi Prashant,

We can certainly create a fork of your repo on our 'killbill' organization to bring more visibility if you there is any interest on your end.

If so, let's first allow for a review, and then we can go ahead and create the fork.

Stéphane

To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/ffb25cd1-3784-407a-b226-0234d257ad37n%40googlegroups.com.

[Prashant Kumar]

Hi Stephane, Shaun, and the Kill Bill Team,

I hope this email finds you well.

Stephane as per your suggestion regarding the group id I have migrated our entire codebase to the official Cashfree open-source repository with the group Id changes Please go through this PR -  Pull Request #1.

Shaun, regarding your concern about unit validation, I have addressed it in the following issue: Usage Unit Validation Issue. The fix is included in the same pull request:  Pull Request #1.

Please review the PR at your earliest convenience and share any feedback you might have. Additionally, let me know when you're ready to fork this repository, so we can mark this as our first open-source contribution to the Kill Bill community.

Looking forward to your input.
Best regards,

Prashant Kumar

[Prashant Kumar]

Hi Stephen,

Our team has a small query: Would it be better if you create the killbill-kafka-consumer-plugin repository in the Killbill organization and we fork it into our Cashfree-OSS GitHub? This way, we can raise PRs directly in the Killbill repo, making it look like we have contributed to the Killbill project. This approach seems more efficient than you forking our repository and having to update it each time whenever we update our repo.

What do you think?

Thanks and Regards
Prashant Kumar

[Prashant Kumar]

Hi Stephen, Shaun, killbill team

Any update on this?

Waiting for your response

Thanks and Regards

Prashant Kumar