/usr/lib/cgi-bin/astpp/astpp-cdr-xml.cgi

[deanskis]

Is anyone using ASTPP 2.3? and have over 200 concurrent calls?

Our system load is struggling bad. 

We have 4x Freeswitch servers. 1x ASTPP server.  1x MYSQL server AWS  2x Apache load sharing servers.

And our system can't handle anymore then 150 calls!!!!!!

/usr/lib/cgi-bin/astpp/astpp-cdr-xml.cgi  seems to be the issue. it is putting huge load on the server. I have had ASTPP look at it and they blamed the MYSQL they did some modification but it did not change anything. so that was a waste of money. 

I have had 2 more ASTPP / Freeswitch experts look at it. We ended up adding more servers to handle that specific resource to see if that would help. But still the same. 

I am at my wits end. 

Surely ASTPP is not restricted to 200 calls while free switch can handle 1000's.

All our servers sit at around 1 to 4% load but the server that handles the perl CGI script will go up to 95% with 150 + calls.    Currently it has 12 cores and 32GB memory. 

the perl CGI script /usr/lib/cgi-bin/astpp/astpp-cdr-xml.cgi   is the main cause and would love if anyone can shed some light on it. 

Dean 

[Fozzy]

I think and this is just my opinion and no means a solution but maybe a nudge in the right direction, there is a fundamental issue with script-to-db processing issue here. MySQL DB is fast and if Freeswitch interacted directly with the DB and not via a xml script then this would not be an issue I feel. The process of processing information is too slow and resource hungry.

A call is initiated, XML / CURL needs to interact with DB for account numbers, permission, then do LCR (although takes milliseconds is still a process that could be made shorter), call is ended and the process started again with XML / CURL to DB with CDR info. This times 200 and no wonder the server with more than enough resources wants to die.

The guys at ASTPP and mad geniuses and I'm sure they can relook at the way Freeswitch talks to MySQL.

Sorry if this does not help but felt I should mention a few things.

[deanskis]

Thanks Fozzy, I had a meeting with them today to try and find a solution. It seems because they use apache to process the cdr files in to the DB. Apache can not handle 500+ requests it just dies.  so we are getting closer to the problem.  But i think your thinking is spot on! 

[Samir Doshi]

If the issue belongs to apache then there must be delay or fail case or over load issue with call processing as that is also using apache scripts to build dynamic dialplans. Are you facing any issue in initiating and connecting calls to gateway? 

Best Regards
--
Samir Doshi
iNextrix Technologies Pvt. Ltd.
http://www.inextrix.com

Disclaimer:
The information contained in this communication is confidential and may be legally privileged. It is intended solely for the use of the individual or entity to whom it is addressed and others authorised to receive it. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking action in reliance of the contents of this information is strictly prohibited and may be unlawful.

On Mon, Jan 18, 2016 at 1:29 PM, deanskis <de...@flashtel.com.au> wrote:

Thanks Fozzy, I had a meeting with them today to try and find a solution. It seems because they use apache to process the cdr files in to the DB. Apache can not handle 500+ requests it just dies.  so we are getting closer to the problem. 

On Monday, January 18, 2016 at 4:43:28 PM UTC+10, Fozzy wrote:

I think and this is just my opinion and no means a solution but maybe a nudge in the right direction, there is a fundamental issue with script-to-db processing issue here. MySQL DB is fast and if Freeswitch interacted directly with the DB and not via a xml script then this would not be an issue I feel. The process of processing information is too slow and resource hungry.

A call is initiated, XML / CURL needs to interact with DB for account numbers, permission, then do LCR (although takes milliseconds is still a process that could be made shorter), call is ended and the process started again with XML / CURL to DB with CDR info. This times 200 and no wonder the server with more than enough resources wants to die.

The guys at ASTPP and mad geniuses and I'm sure they can relook at the way Freeswitch talks to MySQL.

Sorry if this does not help but felt I should mention a few things.

On Monday, 18 January 2016 06:09:48 UTC+2, deanskis wrote:

Is anyone using ASTPP 2.3? and have over 200 concurrent calls?

Our system load is struggling bad. 

We have 4x Freeswitch servers. 1x ASTPP server.  1x MYSQL server AWS  2x Apache load sharing servers.

And our system can't handle anymore then 150 calls!!!!!!

/usr/lib/cgi-bin/astpp/astpp-cdr-xml.cgi  seems to be the issue. it is putting huge load on the server. I have had ASTPP look at it and they blamed the MYSQL they did some modification but it did not change anything. so that was a waste of money. 

I have had 2 more ASTPP / Freeswitch experts look at it. We ended up adding more servers to handle that specific resource to see if that would help. But still the same. 

I am at my wits end. 

Surely ASTPP is not restricted to 200 calls while free switch can handle 1000's.

All our servers sit at around 1 to 4% load but the server that handles the perl CGI script will go up to 95% with 150 + calls.    Currently it has 12 cores and 32GB memory. 

the perl CGI script /usr/lib/cgi-bin/astpp/astpp-cdr-xml.cgi   is the main cause and would love if anyone can shed some light on it. 

Dean 

--
=====================================================================
Documentation : https://astppdoc.atlassian.net/
Please contact at sa...@inextrix.com for commercial support.
---
You received this message because you are subscribed to the Google Groups "ASTPP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astpp+un...@googlegroups.com.
To post to this group, send email to as...@googlegroups.com.
Visit this group at https://groups.google.com/group/astpp.

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/bcdc8fd7-d8c1-43cb-aa98-8e46163174f1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[voipguy]

I don't see Apache causing this issue, but I can not say 100% as I've not seen this in real life. When the load on the server gets high is this affecting anything? Call quality, speed of accessing portal etc? 

I noted that the load you are giving %, I take it your taking this from "top", what is the actual load of the server? Is it above 12 when you think its overloaded?

[deanskis]

Hi Voipguy,

Yes calls will start to fail because the server refuses the http connection for the new XML. 

When you start to have over 150 + calls APACHE & MYSQL resources go through the roof as it can not handle that many HTTP requests.  After talking with hardik from astpp it seems there is only one queue or something along those lines so the CDR files get backed up. i am not a tech but i have over 5 techs working on this. Hardik from ASTPP is investigating as well. 

And yes everything starts to deteriorate that why separated the process's 

[deanskis]

Hi Samir, 

Only when there is 150+ calls phone calls will start to fail HTTP request the server will start rejecting them due to high load. 

anything under 150 everything is smooth. 

[Jason]

What is the spec of your apache servers?

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/25e36d7b-a273-41d9-823f-3e0f7b7bd502%40googlegroups.com.

[deanskis]

12gb RAM and 12 cores

[Samir Doshi]

Do you have dedicated database server and is it in same network where apache server located? 

Best Regards
--
Samir Doshi
iNextrix Technologies Pvt. Ltd.
http://www.inextrix.com

Disclaimer:
The information contained in this communication is confidential and may be legally privileged. It is intended solely for the use of the individual or entity to whom it is addressed and others authorised to receive it. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking action in reliance of the contents of this information is strictly prohibited and may be unlawful.

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/7f23c78e-deec-41c6-9b1a-a3441ecd705b%40googlegroups.com.

[Jason]

Are these physical servers or hosted on digital ocean etc?

You have more than enough power to cope well beyond what your having issues with if you are getting full resources.

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/CAFABgc001a9LHd1fibiWe6ExFK_0vPaJhcEigaXBpTKUiaBHKw%40mail.gmail.com.

[deanskis]

Softlayer bare metal servers

[deanskis]

Hi Samir, 

to give you a better understanding.

First we had one server 8gb ram 8 cores or something like that. Normal ASTPP installation. 

When we were reaching 100+ calls we seen a high load on the server. We thought that was weird but anyways i then decided to add more free-switch servers as we  were targeting other country's anyways. 

Once we did that we were getting more calls with new customers  150 to 200 concurrent calls. We then noticed the CGI script was causing huge load in Apache and MySQL. 

So we thought we would use AWS RDS  MYSQL which a lot of company's use when they have intensive resources.  That did not help the problem. So we then decided to separate Apache and have the CDR processing server on its own,.  that still did nothing it would still go up to 95% CPU  when it hit 150 calls.

I have done everything possible. I have even had ASTPP look at it. they blamed the MySQL. done some tuning..but it did not help. waste of $60.00

It seems freeswitch having to send XML files via HTTP and not directly to the database does not seem the ideal solution. Free-switch on a single server can handle 1 million calls (if tuned right of-course and supported hardware) . But ASTPP seems to have not thought about server performance processing 100's or 1000's or 10,0000 of XML requests. instead of going direct to DB. 

We also noticed when you have fail-over gateways in trunk settings if we are getting 50 to 100 calls at a time and just say 50 of them were disconnected it would try on all failovers and that caused us major load as well. it would be good if there was a setting it only failovers if the gateway was not responding it would try to fail over not fail over on disconnected numbers.

We also did some other testing with nginx etc but nothing worked. 

[Jason]

If you used AWS RDS you would need to also host apache and the rest of your setup on EC2 to get best speeds. In In a nutshell all servers should be in the same location. Or you will have issues.

What is the load of the server with 12 cores? Is it going over 12? (Not % but load)

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/0e61134a-88a8-4437-96a8-434c6152a9e0%40googlegroups.com.

[deanskis]

Voipguy, i agree but we had issues with both setups.. so it makes no difference.  we have better results with RDS to be honest. 

the problem is as you probably know. Freeswitch does not talk with MYSQL it is via a xml script. You then have to have a server to process those XML and talk to DB with 1000's of XML's i can't see how that would work smoothly?   it is very common for company's to have switches in different locations. so i am not trying to do the impossible. I think if you had free-switch talk directly to a DB it would not be a big issue.  it would cut out the middle man so to speak. 

Yes it was going over 12. 

[Jason]

Yes I know you can have in various locations but apache can be funny with longer latencies. Astpp must be able to step in here, what is the most there system is doing in Iive production?

If your example is the normal then we are currently currently screwed and I need to look for an alternative.

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/ac1b652b-85dc-429d-ad2e-3a49e866cd43%40googlegroups.com.

[deanskis]

Good questions, I would love to know what there live production stats are. 

I can also confirm we reinstalled ASTPP at one stage just in case there was an issue. Still same result. 

It is very depressing to be honest. I have customers but i can't support them..And i have spent $1000s so far for nothing basically. 

I do have a very good guy that knows ASTPP and freeswitch inside out. He was one of the original techs who coded ASTPP front end to PHP from perl? i think something like that. 

he has suggested that i get the whole CDR processing rewritten and have freeswitch talk directly to mysql. But its going to cost me $4000! as its a big job. 

Not sure what to do . 

[Jason]

Hum, I've also been today about another flaw that calls are rated after they finish and not live. Thus leaving you open to fraud. Might have to go back to a2billing at this rate. :(

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/eda75d36-4eec-4c43-b54e-0666c586f71d%40googlegroups.com.

[deanskis]

I dd notice that as well...

Could you imagine the load on the servers if it constantly kept rating ?  it would be millions of XML files lol 

I don't know a2billing thats my issue lol Well i know if it. but it looks like such a huge system and need a degree to use it. 

Also please dont get me wrong, i know this is opensource and i know a lot of people have worked hard and invested a lot of time in to the system. And i am grateful for this software. But i would rather be paying ASTPP for modifications and extra things i want then paying to be able to make calls! 

[Fozzy]

So exactly what I suspected about script-to-db being slow.

I did some reading about Freeswitch and MySQL and see that MySQL is not natively supported but via ODBC. PostgresSQL is though. Besides this I think if the XML or SQL transacts could be written within Freeswitch confs bypassing the need for Apache to do the work then the entire ASTPP would be so much better.

Perhaps in planning for ASTPP v 3?

[deanskis]

I like your thinking fozzy, ASTPP v3  Next week will be great! haha..

[Jason]

A2billing is ok but doesn't offer everything such as limiting calls per second. I bought the extra offering of this but later found out it only limits per server not account! Useless! The benefit of a2billing is its been around a long time so major issues have long been sorted.

From the start astpp seemed better I have already paid for work from them but with the issues your having I think I need to stop and rethink before I envest more. Unless someone can reassure me..

--

=====================================================================
Documentation : https://astppdoc.atlassian.net/
Please contact at sa...@inextrix.com for commercial support.
---
You received this message because you are subscribed to the Google Groups "ASTPP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astpp+un...@googlegroups.com.
To post to this group, send email to as...@googlegroups.com.
Visit this group at https://groups.google.com/group/astpp.

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/bebc0338-3896-422d-b7de-373574143755%40googlegroups.com.

[deanskis]

I have given ASTPP access to my systems to prove it to them. but i have not had a response last 2 days. So that's a little disappointing

Not expecting a fix but i would sure love reassurance that they can see the issue! 

[Samir Doshi]

>> It seems freeswitch having to send XML files via HTTP and not directly to the database does not seem the ideal solution. Free-switch on a single server can handle 1 million calls (if tuned right of-course and supported hardware) . But ASTPP seems to have not thought about server performance processing 100's or 1000's or 10,0000 of XML requests. instead of going direct to DB. 

Yeah we are using xml cdr module and scripts to process the CDR because before inserting any cdr in database we are processing for free packages if any created, balance deduction, customer, reseller and provider billing calculations. We cannot avoid this else reports will be wrong.

Just a side now, in older version we tried to process CDRs with ESL connection but it was not that much reliable and we moved back to xml-cdr. If anyone have better suggestion to handle cdr processing in faster and reliable way then we are open for suggestion. 

>> We also noticed when you have fail-over gateways in trunk settings if we are getting 50 to 100 calls at a time and just say 50 of them were disconnected it would try on all failovers and that caused us major load as well. it would be good if there was a setting it only failovers if the gateway was not responding it would try to fail over not fail over on disconnected numbers.

Current dialplan is having continue_on_fail=true variable set. That means for any fail calls, it will try for failover trunk / gateway. In future version, we are planning to add flexibility for admin to select failover dispositions. 

https://wiki.freeswitch.org/wiki/Variable_continue_on_fail

>> If you used AWS RDS you would need to also host apache and the rest of your setup on EC2 to get best speeds. In In a nutshell all servers should be in the same location. Or you will have issues.

Yeah that's why I asked you question if you have all servers in same network. For your current case, it may possible that your apache server and mysql server connection is getting slower when high load and that's causing the issue. 

For production, we have many clients running ASTPP with 300-400 concurrent calls on single server smoothly. So by considering that 150 CC on distributed servers are very low and must now create issue. 

>> I did some reading about Freeswitch and MySQL and see that MySQL is not natively supported but via ODBC. PostgresSQL is though. Besides this I think if the XML or SQL transacts could be written within Freeswitch confs bypassing the need for Apache to do the work then the entire ASTPP would be so much better.

Perhaps in planning for ASTPP v 3?

As mentioned in previous points, give us ideas on making it faster and reliable. We will be glad to work on that. 

>> I have given ASTPP access to my systems to prove it to them. but i have not had a response last 2 days. So that's a little disappointing

Not expecting a fix but i would sure love reassurance that they can see the issue! 

I am also ASTPP team member ;). I am replying you here. I looked at your server and then only asked you the question about server location and network. 

>> I have done everything possible. I have even had ASTPP look at it. they blamed the MySQL. done some tuning..but it did not help. waste of $60.00

Our team tried out best to look at possible option to resolve the issue in 1 hour in on demand support contract. Apart from that we are also here in community forum trying to help you out to rectify the problem and that might be helpful to others. 

Would be better rather considering it as waste of money, consider it as contribution in building system. ;) 

Best Regards
--
Samir Doshi
iNextrix Technologies Pvt. Ltd.
http://www.inextrix.com

Disclaimer:
The information contained in this communication is confidential and may be legally privileged. It is intended solely for the use of the individual or entity to whom it is addressed and others authorised to receive it. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking action in reliance of the contents of this information is strictly prohibited and may be unlawful.

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/b494b6cf-deb7-4b21-8f80-915e0b597bb9%40googlegroups.com.

[deanskis]

Thanks for your post Samir,

But i think you are missing a few points.

I did originally have it all on ONE server when the issues started. So you keep saying because we separated it is now an issue is just not true. 

as for contribution, Well Samir, I would be more then happy to contribution  and pay ASTPP for many modifications, but to be honest, to contribute  to be able to make calls on a soft switch? I don't think you will find anyone to over the moon for that. 

[Samir Doshi]

Hi Dean, 

>>I did originally have it all on ONE server when the issues started. So you keep saying because we separated it is now an issue is just not true. 

I never said that you separated server and that is causing the problem. I simply tried to find the clue of your problem,

>> If you used AWS RDS you would need to also host apache and the rest of your setup on EC2 to get best speeds. In In a nutshell all servers should be in the same location. Or you will have issues.

Yeah that's why I asked you question if you have all servers in same network. For your current case, it may possible that your apache server and mysql server connection is getting slower when high load and that's causing the issue. 

>> as for contribution, Well Samir, I would be more then happy to contribution  and pay ASTPP for many modifications, but to be honest, to contribute  to be able to make calls on a soft switch? I don't think you will find anyone to over the moon for that. 

You saying like you are not able to make single call from system and I said you about contribution. :) 

Anyways here are few things, you should try, 

1. Put Apache and MySQL on same server.

2. Put all your servers in same network. 

3. If any limitation with bandwidth between mysql and apache servers then maybe increase it.

Best Regards
--
Samir Doshi
iNextrix Technologies Pvt. Ltd.
http://www.inextrix.com

Disclaimer:
The information contained in this communication is confidential and may be legally privileged. It is intended solely for the use of the individual or entity to whom it is addressed and others authorised to receive it. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking action in reliance of the contents of this information is strictly prohibited and may be unlawful.

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/aa09af9a-a2fb-4cb2-955b-a5fb8690ed50%40googlegroups.com.

[Fozzy]

No points have been missed here. Samir has in some words agreed that the current process could be made better but the point is it is currently working well. Samir simply asked if you have ideas to contribute and not necessarily your time and genius code then it may go along way to make the system better. The whole point of open source besides the fact that it is free is for the world of like minded people with ideas to come together and build an amazing system. If Linus had all the ideas then he would not have released linux to us and have the world make it the OS it is today (That runs the world).

ASTPP is a powerful system that has given myself and many others a easy platform to enter and successfully run a telco like the big boys do. If we have an idea lets share it. If we have some code, share. Its only going to make your life a whole lot better on profitable.

This got off topic a little but is it possible for the same code that runs from Apache to process the calls and CDR be done from Freeswitch itself?

[deanskis]

HI Fozzy, I agree with you but i dont think the process is working well. Not when you have a large customer base. if you can call 200 calls large. 

At the end of the day if ASTPP said they need $1000 to fix the issue i would pay it in a heart beat.  But my point to the contribute is that the person who first looked at, had no idea and still took the money i would of been happy to pay more then 1 hr i would of paid 10 if i had to....if they knew what was wrong.. I dont mind spending money to fix things. But i hate spending money for no reason :)  thats just me personally. 

Samir, The suggestions you have suggested have all been tried and tested. It's simply just not going to work for me unless i get the code changed. But thank you for your suggestions. 

[deanskis]

One thing i did notice, If you have 1 or 2 calls per second the system handles it fine. but if you start getting 10 or 15 calls per second (not every second - every now and then) thats when the load starts to get high. 

in our conversation with ASTPP. They did advised something about queue's the script only has 1 or something along those lines.  Samir, Is it possible for ASTPP to add more queue's? as that should reduce load dramatically? 

If ASTPP can come up with a temp solution i would be happy to pay for it. 

On Wednesday, January 20, 2016 at 6:38:01 PM UTC+10, Fozzy wrote:

[Fozzy]

I also agree with you. Its nice to agree... he he. Perhaps for your spent moneys ASTPP could demonstrate to you a working environment with over 200+ concurrent calls using the same code that is available for download. If I was a large telco business looking into a CMS for VoIP billing then I would want these assurances else specify on the home page of ASTPP the limited CC of ASTPP. It says on their site that it can handle 10s of thousands of calls but this is the ability of Freeswitch standalone.

Seems there is no quick fix for you Deankis. Wish I could help. Maybe a workaround would be multiple VM's of ASTPP on your bare metal and load balance the customers on each switch. This could be a accounting and admin nightmare but perhaps it could all be managed from a single interface like WHMCS.

[Fozzy]

Hmmm I'm currently sitting with 30+ calls and my server load is 0.67 and average of 0.42. Does not go over 1. My system is Debian 7 on a VM by the way.

[Fozzy]

See screenshot.

[deanskis]

Yes if my system has 30 calls... it hardly moves.... just when it goes to around 130 to 150 mark it starts to load big time and then when i hit 200 server load goes through the roof and things will stop working. 

[Samir Doshi]

@Dean, We will be in touch to look at that issue. 

@Fozzy,  If I was a large telco business looking into a CMS for VoIP billing then I would want these assurances else specify on the home page of ASTPP the limited CC of ASTPP. It says on their site that it can handle 10s of thousands of calls but this is the ability of Freeswitch standalone.

Where do you see "It says on their site that it can handle 10s of thousands of calls"? 

Best Regards
--
Samir Doshi
iNextrix Technologies Pvt. Ltd.
http://www.inextrix.com

Disclaimer:
The information contained in this communication is confidential and may be legally privileged. It is intended solely for the use of the individual or entity to whom it is addressed and others authorised to receive it. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking action in reliance of the contents of this information is strictly prohibited and may be unlawful.

To view this discussion on the web visit https://groups.google.com/d/msgid/astpp/35295265-2832-4e6f-a960-715d1607f52b%40googlegroups.com.

[Fozzy]

Oh my bad. Should have checked first hey. That info is on Freeswitch's site I think.

[Fozzy]

From Freeswitch.org

Performance

• Tested under load for over 100 hours
• 10,000,000+ calls
• At rates exceeding 50 CPS

Performance will vary depending on application. You will need to test for your particular situation.

On Wednesday, 20 January 2016 14:20:09 UTC+2, Samir Doshi wrote:

[deanskis]

Samir got in touch the other day, i just sent him the network details so he can have a look 

To satisfy ASTPP and some members here. I reinstalled ASTPP on a single server with 18 cores. 

I was able to get up 300 calls and it died on my again.  

So hopefully ASTPP can find a solution. 

[voipguy]

check out my post about a fix for this:

https://groups.google.com/forum/#!topic/astpp/p5SclYqta7s

[Chandan Singh]

We have same issue .. i have reported previously  but no solution.. Please check attached file ..

 

[ALG]

Hi Everyone,

Is there any update with this issue?

Deankis. How many Maximum concurrent calls can handle with your clustered servers?

Thanks,

ALG