Kill Bill Footprint for Production

[chez...@gmail.com]

Hi Pierre,
I'm a developer from NETS Singapore and Hoan is our team Architect.

We would like to know what is a optimum production hardware sizing for Kill Bill in Tomcat 8.5.5.

Estimated:

Number of Tenants: 1800
Accounts per Tenants: 50

Kill Bill highly use cases:
1. Invoice Creation
2. Payment through plugin similar to killbill-hello-world-java-plugin structure.

Best Regards
Chester

[Pierre-Alexandre Meyer]

Hi Chester,

This is quite difficult to answer in the abstract, as it heavily depends on a number of factors, including catalog configuration for instance: if the BCD of all 90k accounts is on the same day, you will have monthly spikes on that day during invoice generation, while if the BCD is spread out, the load will be distributed throughout the month.

Additionally, it depends on your deployment strategy (cloud or on-premise). Some cloud solutions will require specific tuning: MySQL RDS and Aurora RDS both require different Kill Bill and hardware settings to perform optimally for instance.

In general though, while Kill Bill scales horizontally, you typically only need a few mid-sized instances (2 to 5 of modern CPUs, 2GB-4GB heap). The database is eventually the bottleneck, so I encourage you to take into account your growth projection over the next 12 months and over provision it.

Happy to go into specifics if you can share more details about your setup (type of hardware available, expected throughput/latency numbers, expected load, BCD alignment, Kill Bill version, bus and notification queue configuration, etc.). The easiest though is probably to setup a load testing environment and verify the numbers you are getting (see also this blog post: http://killbill.io/blog/performance-numbers/).

Does that help?

--
You received this message because you are subscribed to the Google Groups "Kill Bill users mailing-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to killbilling-users+unsubscribe@googlegroups.com.
To post to this group, send email to killbilling-users@googlegroups.com.
Visit this group at https://groups.google.com/group/killbilling-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/4ecfbe09-3606-4f02-bb71-06e7c2ae962d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Pierre

[Chez Liew]

Hi Pierre,

Thank you for your initial suggestions.

Some of the details are:

Deployment will be on-premise, and MySQL RDS cluster of 3.

Killbill version: 0.18.3

2 instances of Kill Bill

Expected able to handle at least 60tps

We do not have bus and notification queue configuration.

Best Regards

Chester

[Pierre-Alexandre Meyer]

On Mon, May 15, 2017 at 11:28 AM, Chez Liew <chez...@gmail.com> wrote:

Deployment will be on-premise, and MySQL RDS cluster of 3.

Just to be sure I understand correctly: the Kill Bill servers will be in your own datacenter, while you will use RDS on AWS? Even with AWS Direct Connect, you will likely face long latencies for database operations. I would strongly recommend co-locating the servers and the database in the same region instead, either cloud or on on-premise, but not both.

Also, could you clarify "cluster of 3"? Do you mean one master and two read replicas? Typically, a single master/slave combo is enough.

Do you have experience with MySQL RDS? There are quite a few gotchas to be aware of to get good performance (e.g. pre-provisioned I/O is a must). Doing some pre-launch performance tests is highly encouraged to validate the setup.

Killbill version: 0.18.3

I would suggest to deploy the latest version of Kill Bill 0.18.x instead. 0.18.6 for instance featured lots of performance improvements and the upcoming 0.18.9 will further reduce GC overhead.

 

2 instances of Kill Bill

Expected able to handle at least 60tps

What kind of traffic is it? All mixed operations (GET/POST)? If you expect 60 payments per second for instance, given the typical gateway latencies, 2 instances won't be enough.

Also, what is the expected peak?

We do not have bus and notification queue configuration.

I don't understand this statement: Kill Bill cannot work without its internal event bus and notification queues. Make sure to take a look at https://github.com/killbill/killbill/wiki/Kill-Bill-Bus-and-Notification-Queue-Configuration, especially if the DNS of the Kill Bill servers won't be static (cloud scenario, where instances go up and down).

--

Pierre

[Chez Liew]

Hi Pierre,

Sorry on the Mysql RDS typo, we are not using any AWS. 

Database is on-premise too, else the setup would sounds really weird.

We are using latest MySQL Replication, its up to our DBA to setup.

If the master db is down, one of the two slaves will automatically promote as a master. 

We will try move to 0.18.6, given that it will not break our existing kb-plugins.

Regarding the internal event bus and notification queues configuration, I do not recall explicitly configure anything. I assume Kill Bill there's default value somewhere.

60tps would be mixed operations, as I read through your performance-numbers load test, sustaining 133 req/sec.

Since we will not be having hardware as powerful upfront, so expected less than half of your result, I presume is reasonable? 

Regards

Chester

[Pierre-Alexandre Meyer]

On Tue, May 16, 2017 at 1:58 AM, Chez Liew <chez...@gmail.com> wrote:

Sorry on the Mysql RDS typo, we are not using any AWS. 

Database is on-premise too, else the setup would sounds really weird.

Great :-) 

We will try move to 0.18.6, given that it will not break our existing kb-plugins.

0.18.x upgrades should be fairly easy. In any case, it's good practice in general to keep up with patch upgrades.

Regarding the internal event bus and notification queues configuration, I do not recall explicitly configure anything. I assume Kill Bill there's default value somewhere.

Yes, there's indeed a default, but it might not always work depending on your setup (more details in the wiki page I sent). Make sure to go through http://docs.killbill.io/latest/userguide_deployment.html#_pre_and_post_deployment_checklist as well.

60tps would be mixed operations, as I read through your performance-numbers load test, sustaining 133 req/sec.

Since we will not be having hardware as powerful upfront, so expected less than half of your result, I presume is reasonable? 

That load test was simulating a specific scenario (e-commerce payments), with a specific set of config, hardware, etc. With the limited information I have, I have no way of guaranteeing any number for your setup, the easiest would be to try it (see https://github.com/killbill/killbill/wiki/Kill-Bill-Profiling). I would also work backwards from your requirements (what is your expected traffic pattern?), to size the environment correctly.

Hope that helps,

--

Pierre