Question about expected speed of killbill requests

[daniel.n...@gmail.com]

When running the killbill application and database containers on my laptop and doing a GET on http://localhost:8080/1.0/kb/subscriptions?externalKey on an existing subscription I am seeing speeds around 0.1 seconds.

When deploying the killbill application on a container on one server and with a mysql server on a separate server I am seeing speeds on the same endpoint around 0.5 to 0.9 seconds. Sometimes up to 3 seconds.

What is the expected speed of this request?

Are they tweeks i can do to speed up the process.
Both the servers are on the same 1G network and have plenty of CPU,RAM and so on

Thanks
/Daniel

[karan bansal]

Hi Daniel,

What are the response times for ping between the containers on same machine vs separate servers. Also, do you see any performance difference between the first call vs the subsequent calls to the subscription in the same session?

Regards

Karan

[daniel.n...@gmail.com]

Hi Karan

Here is a ping flood from the killbill server to the database server

10000 packets transmitted, 10000 received, 0% packet loss, time 2920ms
rtt min/avg/max/mdev = 0.073/0.211/5.386/0.194 ms, ipg/ewma 0.292/0.188 ms

the request seems to vary between 0m0.461s and 0m0.801s at the moment

I am using curl to test so not sure how to test with the same session

time curl -v \
        -u x:x \
        -H "X-Killbill-ApiKey: x" \
        -H "X-Killbill-ApiSecret: x" \
        -H "Accept: application/json" \
        'https://xxxx/1.0/kb/subscriptions?externalKey=2467249'

[daniel nielsen]

Hi Karen

Did you get time to look at this?

Thanks

/Daniel

[karan bansal]

Hi Daniel,

Thank you for bumping it up. In the initial email you have mentioned about querying the subscriptions. Are the response times for all the endpoints ( with mysql on separate server ) similar OR is it quicker for some of them or higher for others. 

Regards

Karan

[daniel nielsen]

Hi Karen

Sorry for replying so late. I dont think i got the email when you replied.


We have one server running the killbill containers
p-killbill01
And another running a reverseproxy via nginx
p-proxy-i with domain  killbill.dk-tv.net


this is my test curl to test the speed.

curl -w 'Testing Website Response Time for :%{url_effective}\n\nLookup Time:\t\t%{time_namelookup}\nConnect Time:\t\t%{time_connect}\nPre-transfer Time:\t%{time_pretransfer}\nStart-transfer Time:\t%{time_starttransfer}\n\nTotal Time:\t\t%{time_total}\n' \
    -u xxx:xxx \
    -H "X-Killbill-ApiKey: xxx" \
    -H "X-Killbill-ApiSecret: xxx" \
    -H "Accept: application/json" \
-o /dev/null \
    'https://killbill.dk-tv.net:8080/1.0/kb/subscriptions?externalKey=1934614'


Doing a curl on p-proxy-i(https) to killbill.dk-tv.net gives me this (1,5 - 1,8s)

Lookup Time:            0.003162
Connect Time:           0.003570
Pre-transfer Time:      0.044797
Start-transfer Time:    1.605804

Total Time:             1.606888
Doing a curl on p-proxy-i(https) to p-killbill01  gives me the same (1,5 - 1,8s)

Lookup Time:            0.003530
Connect Time:           0.005144
Pre-transfer Time:      0.059687
Start-transfer Time:    1.623905

Total Time:             1.624415


Doing curl on p-killbill01 to p-killbill01 is also the same

Lookup Time:            0.005528
Connect Time:           0.006114
Pre-transfer Time:      0.118487
Start-transfer Time:    1.611047

Total Time:             1.611436


p-killbill01 is running nginx listning to 8080 on https and passing it on to the killbill containers as http also on port 8080

entering the nginx container and running the curl against killbill:8080 gives me the same

Lookup Time:            0.014816
Connect Time:           0.015588
Pre-transfer Time:      0.016235
Start-transfer Time:    1.564737

Total Time:             1.565089

entering one of the killbil containers and running curl against localhost is also the same

Lookup Time:            0.006815
Connect Time:           0.007471
Pre-transfer Time:      0.007836
Start-transfer Time:    1.575847

Total Time:             1.576454

so just wanted to be sure that it wasnt related to my reverseproxy setup

To answer your original question it seems like its the same error on all the containers.
I have 6 running and have run this script

for i in {1..6}; do echo killbill_$i ; docker exec killbill_$i curl -s -w '%{time_total}\n' -u xxx:xxx -H "X-Killbill-ApiKey: xxx" -H "X-Killbill-ApiSecret: xxx" -H "Accept: application/json" -o /dev/null 'http://localhost:8080/1.0/kb/subscriptions?externalKey=1934614' ; done

killbill_1
1.561865
killbill_2
1.642079
killbill_3
1.541519
killbill_4
1.546816
killbill_5
1.512320
killbill_6
1.672892


Another aspect is that i have 3 environment. All duplicated with seperate database, application servers and reverseproxys.
The above is in production, on qa and staging the time is around 0.3-0.4s

[daniel nielsen]

This has been resolved.
I will post my times her incase anyone is have the same question.

For some reason i suddenly saw the times going down to 0.5 seconds.
I think there has been some issue on the underlying hardware

But i also followed your advice and changed KILLBILL_SECURITY_SHIRO_NB_HASH_ITERATIONS

This has change the times to 0.100755

Thank you for you help