performance testing of MMS

[thangtq]

Hi Amit Bhayani,

I saw your test results about performance of Mobicents Media Server.

But I have some problems when testing.  MMS has errors ENDPOINT_NOT_AVAILABLE

The test was made for 100 concurrent announcements of an average length of announcement 5 Seconds.

Could I need some tunning in file "deploy/server-beans.xml " ?

[Amit Bhayani]

Which Mobicents Media Server are you using? The latest release is 3.0.0.CR1 http://code.google.com/p/mediaserver/downloads/list

Please use this one.

Amit.

[yulian_o]

Hello
You should preconfigure appropriate number of endpoints.Otherwise when all endpoints will be used you will get endpoint not available.
So simply check what endpoints are you using and increase their size in config.

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi  yulian_o,

Although I increased the numbers of endpoints to [1..600] and memory(java -Xmx3072m), but the problem  ENDPOINT_NOT_AVAILABLE still existed.

Could I need make a new performance test with the lasted version- MMS-3.0.0.CR1 ?

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
1) use 3.0.0.CR1 release so in case of problem debug will be usefull.
2) How are you testing the load,please provide scenario
3) Please provide log file of mms

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

you can see scenario, log in the attached files.

errors occurred in MMS of IVR Client System.

Regards,

Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
1) What is IVR Demo? Is it jslee demo or restcomm application
2) How much concurrenct active announcement are you trying to generate , currently i dont see that its higher then 5. If its the case why do you start mms with 450+ endpoints?
3) You are starting to get exception for DLCX ( no call id ) , so it looks like connections are not deleted, therefore while reusing the same endpoints you may get different errors. If you are using Restcomm use latest trunk since in earlier releases this problem was known problem but then it was resolved.
Best regards
Yulian oifa

[Tran Quyet Thang]

Hi Yulian,

Thanks you for replying me.
I built from JSR309Demo, and tried to generate 30 cps.

I think have errors with MMS(of IVRClient System) , you can see the differents with "success call" and "error call" in the attached file.

Incase of "error call" , connectionIdentifier-157dbd887173d4 is reused in MMS when IVRClient send the second "send MGCP Request 2" command.
It's will be regarding to error "ERROR MGCP tx=800003108 Failed" in MMS.

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
There is jsr 309 demo for servlets , restcomm and mgcp
Which one of them you are using.

The problem is in client since when CRCX is sent no C ( connection id ) parameter should be sent , it will be ignored anyway
Therefore when deletion occure it deletes connection it thinks it got and not the actually created.

Please post which example you are using and i will try to contact a team that developed example to solve the issue

Best regards
Yulian Oifa

[yulian_o]

Hello
Please follow https://code.google.com/p/mediaserver/issues/detail?id=86&thanks=86&ts=1350378415
for duplicate connection id problem

Best regards
Yulian Oifa

[yulian_o]

Ok Lets summarize a problem for now
1) its possible to have 2 connection with same id in mgcp , which is bug , and may happen on fast servers ( which is capable to create 2 connections in one nano ).
2) DLCX is using invalid call id from client side , see request 800003108.its C:157f4b034cad74 while for all previous requests call id was 157dcb3540c9f4.

Waiting for an update on what example you are using.
Best regards
Yulian oifa

[thangtq]

Hi Yulian,

I used the jsr 309 demo for servlets.

In the attached files previously, you can saw in "error call" or "success call", the ConnectionIDs are generated by MMS.

IVRClient

CRCX 799999994 mobicents/bridge/$@10.50.145.32:2427 MGCP 1.0

C: 157dcb3540c9f4

MMS

200 799999994 Success

I:157dbd85b9c9fc

Z:mobicents/bridge/4...@10.50.145.32:2427

I think that , the duplicate connection id problem is happend on the MMS side ?

[yulian_o]

Hello
Yes i guess it is , i have opened an issue for that , see link https://code.google.com/p/mediaserver/issues/detail?id=86&thanks=86&ts=1350378415
In any case there is a problem with call id which is invalid in dlcx , therefore exception occures.
So there are 2 problems , one on mms side with id generation , second with invalid call id with servlets jsr 309 example
I will update servlets team with this problem and will resolve the id problem on mms side

Best regards
Yulian Oifa

[George Vagenas]

Hi,

Can you try with the latest MSS snapshot https://mobicents.ci.cloudbees.com/view/SIP%20Servlets/job/Mobicents-SipServlets-Release/lastSuccessfulBuild/artifact/mss-2.0.0.FINAL-jboss-as-7.1.2.Final-1350275431.zip, in this snapshot you will find the media-jsr309-servlet example.

In the latest MSS snapshot, media-jsr309-servlet example there are some bug fixes and also is configured to use the latest MMS 3.0.0.CR1.

George

--
George Vagenas

[yulian_o]

Hello
Are you using this servets version : http://sourceforge.net/projects/mobicents/files/Mobicents%20Sip%20Servlets/Mobicents%20Sip%20Servlets%202.0.0.FINAL/
If not please update to this one , and post logs with this mss

Best regards
Yulian Oifa

[thangtq]

Hi Yulian,George

I'm using version mss-1.7.0.FINAL.

I'll try with version 2.0.0.Final of MSS, and notify to you late.

Any update on the duplicate connection ID prolem. let me know soon.

Regards,
Thangtq,

[Tran Quyet Thang]

Hi Yulian,

I tried with MSS-2.0.0.Final and "jsr-309-driver-3.0.0.CR1.jar", "mgcp-driver-3.0.0.CR1.jar" .

The duplicate connection id proleam is still existed in MMS-3.0.0.CR1.

You can see it happened with the call id - 1614eb54662aec and duplicated connection id-1614cdd2ee9b4c in the attached files.

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
Duplicate connection id may be resolved only on mms side , i will update you as soon as this will be modified ( hope tomorrow )

Best regards
Yulian Oifa

[yulian_o]

Hello
Please try to use Revision c5b62aece91b
Let me know about the status
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

I think the problem is remained .
you can see the call - with callid: 1762d4191247cc.

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
That means you are running 3.0.CR1 release
Take a look at : http://code.google.com/p/mediaserver/source/browse/controls/mgcp/src/main/java/org/mobicents/media/server/mgcp/controller/MgcpConnection.java
you can see that connection id starts with 1 and incremented for each connection , therefore there is no chance it will get to high numbers.

You should git the repository and then mvn clean install in root folder , after that you will have binary in bootstrap/target/.. folder

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

I have errors in mvn , when building a binary MSS-c5b62aece91b.

Could you send me the binary file ? for speed up the performance test.

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
Are you compiling it in linux?
If on windows you can run : mvn clean install -Dmaven.test.skip=true

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

According to you helps.

I built the binary version of MMS on windows successful, buit It can't be run on my linux server.

My linux server can't be connect to the internet, therefore there's no way to build from the linux server.

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
It says that you have invalid character in server-beans.xml please check the file.
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

Thanks you, the problem-duplicate connection id seem solved. But other errors arose.

1. MMS has stop processing MGCP messages when I've enable DEBUB mode from file MMS_HOME/conf/log4j.xml.

10:22:26,950 INFO  [MainDeployer] [[[[[[[[[ Mobicents Media Server: release.version=3.0.0.CR1 Started ]]]]]]]]]
10:22:38,112 DEBUG [MgcpProvider] Receive  message 123 bytes length
10:22:38,115 ERROR [MgcpProvider] Could not process message
java.lang.UnsupportedOperationException
        at java.nio.ByteBuffer.array(ByteBuffer.java:959)
        at org.mobicents.media.server.mgcp.MgcpProvider$Receiver.perform(MgcpProvider.java:346)
        at org.mobicents.media.server.mgcp.MgcpProvider$MGCPHandler.receive(MgcpProvider.java:271)
        at org.mobicents.media.server.io.network.UdpManager$PollTask.perform(UdpManager.java:408)
        at org.mobicents.media.server.scheduler.Task.run(Task.java:117)
        at org.mobicents.media.server.scheduler.Scheduler$WorkerThread.run(Scheduler.java:368)

2. sometime , I need disable DTMF inband(RTP) , how i can do ?


v=0
o=- 1350874880762 1 IN IP4 10.58.44.205
s=Mobicents Media Server
c=IN IP4 10.58.44.205
t=0 0
m=audio 65192 RTP/AVP 97 8 0 101
a=rtpmap:97 l16/8000
a=rtpmap:8 pcma/8000
a=rtpmap:0 pcmu/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15

3. When I run several IVR Client instances , MMS is only stable in 1-2 hours, after that it occurs errors .

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
1) It will be solved in next release , this error is due to changing parsing of mgcp messages.
2) Currently there is no option to disable out of band dtmf
3) Please provide errors output , or at least small chunk of log file , because i can not parse huge file

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

when I running multiple instances of IVRClient(with different local MGCP Port) connecting to MMS.
at the same time, "tx" will be the same with different IVRClient instances in MMS side regarding to the cause of MMS instable.




13:10:35,065 INFO  [MGCP] tx=193633795 Started, message= MDCX mobicents/bridge/323@10.50.145.32:2427, call agent = /10.50.145.32:15341
13:10:35,065 INFO  [MGCP] tx=193633795 Started, message= MDCX mobicents/bridge/259@10.50.145.32:2427, call agent = /10.50.145.32:18641
13:10:35,065 INFO  [MGCP] tx=193615402 Started, message= MDCX mobicents/bridge/3...@10.50.145.32:2427, call agent = /10.50.145.32:13391
13:10:35,066 INFO  [MGCP] tx=193631638 Started, message= MDCX mobicents/ivr/3...@10.50.145.32:2427, call agent = /10.50.145.32:18291
13:10:35,066 INFO  [MGCP] tx=193615403 Started, message= CRCX mobicents/ivr/$@10.50.145.32:2427, call agent = /10.50.145.32:13391
13:10:35,066 INFO  [MGCP] tx=193631639 Started, message= MDCX mobicents/bridge/3...@10.50.145.32:2427, call agent = /10.50.145.32:18291
13:10:35,068 INFO  [MGCP] tx=193633794 was executed normaly
13:10:35,068 ERROR [MGCP] tx=193633795 Failed
org.mobicents.media.server.mgcp.tx.cmd.MgcpCommandException
    at org.mobicents.media.server.mgcp.tx.cmd.ModifyConnectionCmd$Modifier.perform(ModifyConnectionCmd.java:117)

    at org.mobicents.media.server.scheduler.Task.run(Task.java:117)
    at org.mobicents.media.server.scheduler.Scheduler$WorkerThread.run(Scheduler.java:368)

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
Well it indeed cause a problem
Please follow : https://code.google.com/p/mediaserver/issues/detail?id=88&thanks=88&ts=1350983703
I will also update the post when the issue will be resolved ( i will also bring back debug info temporary ) , hope will find some time this evening.

Btw please introduce yourself , and you plan of usage of mms , goals you are planning to gain.

Best regards
Yulian Oifa

[yulian_o]

Hello
Please check with the following release : https://code.google.com/p/mediaserver/source/detail?r=8bbe72c7530fce97f5f12749d5cf8b0c75e151da

It should solve a problem of duplicate transaction id from different sources and should allow log of mgcp requests

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,


- I'll make a high loading test again with this release.
- I work for a telecom operator.
- We're trying to build a small IVRProxy=SIPProxy(MSS) + RTPProxy(MMS) open system.
The system is a brigde between our core system(GMSC, Media Server) and IVR Sub cps (content provides).


Regards,
Thangtq,

[George Vagenas]

Hi Thangtq,

Have you checked Restcomm?

George

--
George Vagenas

[yulian_o]

Hello
Please let me know when you have results,
Please also provide a test plan
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi George,

I only make some initial tests with Restcomm.

I'll try Restcomm loading test  late.

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[George Vagenas]

That would be nice. please make sure you get the latest snapshot from here : https://mobicents.ci.cloudbees.com/view/RestComm/job/RestComm/lastSuccessfulBuild/artifact/

Let me know if you need anything there.

George

--
George Vagenas

[yulian_o]

Hello Thangtq
Do you have some update about perfomance update or at least status of your tests / approximate date when you will have results?

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

I'll update results next week.

My targets will be following things:
    100 cps from 1 -> 2 BHCA.
    20 cps in normal rate.

MMS can be working well in these situations.

any new errors, I'll notify immediately.

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
You mean 100 calls per second?
What is the call duration?

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

yes,
cps ~ calls per second.

the call durations are less than 6 seconds.



Regards,
Thangtq,

[yulian_o]

Hello
The most important thing is number of concurrently active endpoints / connections / calls.
Basically currently you make a load test on mgcp requests.

I would recommend you to use more realistic scenario.
Average call is around 5 min , so if you want to gain 600 concurrent call you should make 1 new call / second and play some long announcement .
More advanced options :  record part of calls , and may be use dtmf generation and their detection on other side.

Last thing i would recommend to use at least one human call ( some person should make calls from time to time to media server to listen to announcement ) ,
this way you will not only see the load but also have some idea about audio quality.

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

Firstly, I do a simple performance test. after that we can move on with complicate tests.
For my project , I'll working alone....

You can review the first sequence diagram , I sent to you.

In this performance test, I already set up a play announcement (in short time- 5s, it can be increasing) scenario from the server side whenever the client make a call.

Therefore from the client side, we can test record part of calls, audio quality (from the recorded files)...





Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[Tran Quyet Thang]

Hi Yulian,

MMS has MGCP issues on Performance test. It's may be congested UDP.


You can review with the tx=319313788.

the client already sent :

2012-10-25 17:43:23,401 DEBUG org.mobicents.protocols.mgcp.stack.TransactionHandler Send command event to /10.50.145.32, message
DLCX 319313788 mobicents/bridge/4...@10.50.145.32:2427  MGCP 1.0
C:1859d9240b7f5c
I:3faf
 
and retry again:

2012-10-25 17:43:27,403 WARN org.mobicents.protocols.mgcp.stack.TransactionHandler message =
DLCX 319313788 mobicents/bridge/4...@10.50.145.32:2427  MGCP 1.0
C:1859d9240b7f5c
I:3faf
  local Tx ID = 319313788 Remote Tx ID = 5409 Sending the Command 2


MMS is congested with duplicated MGCP messages in second and thrown errors.


17:43:28,586 DEBUG [MgcpProvider] Parsing message: DLCX 319313788 mobicents/bridge/4...@10.50.145.32:2427  MGCP 1.0
C:1859d9240b7f5c
I:3faf
 
17:43:28,587 DEBUG [MgcpProvider] Parsing message: DLCX 319313788 mobicents/bridge/4...@10.50.145.32:2427  MGCP 1.0
C:1859d9240b7f5c
I:3faf
 
17:43:43,447 ERROR [MGCP] tx=319313788 Failed
org.mobicents.media.server.mgcp.tx.cmd.MgcpCommandException
        at org.mobicents.media.server.mgcp.tx.cmd.DeleteConnectionCmd$Delete.perform(DeleteConnectionCmd.java:157)

        at org.mobicents.media.server.scheduler.Task.run(Task.java:117)
        at org.mobicents.media.server.scheduler.Scheduler$WorkerThread.run(Scheduler.java:368)

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
From mms point of view its not an error , it notifies that it can not find call id and then reverts back.
The problem is that message is received by mms 5 seconds after it was sent from mss.
This can be due to high mgcp load on mms server ( more then 25 requests in 10ms ) , in this case you should increase mgcp pool size from 25 to lets say 50 or even higher.
If this network problem that you should increase network throughput.Please pay attention that mms has ability to use 2 interfaces for network, one local and one remote ,
so you can use 2 network cards and put mgcp traffic on local interface while sending rtp data over remote interface , this will imporve network throughput.
Please let me know if you have some question.
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

How about mechanism of managing 'rtp connection pool' and initializing for the pool?
 
In high loading case, MMS has errors on 'rtp connection pool'.


09:38:49,481 DEBUG [ResourcesPool] Allocated new rtp connection,pool size:864,free:0
java.net.SocketException: Too many open files
        at org.mobicents.media.server.impl.rtp.RTPDataChannel.bind(RTPDataChannel.java:267)
        at org.mobicents.media.core.connections.RtpConnectionImpl.onCreated(RtpConnectionImpl.java:351)
        at org.mobicents.media.core.connections.BaseConnection.bind(BaseConnection.java:214)
        at org.mobicents.media.core.endpoints.BaseEndpointImpl.createConnection(BaseEndpointImpl.java:200)
        at org.mobicents.media.core.endpoints.BaseSplitterEndpointImpl.createConnection(BaseSplitterEndpointImpl.java:81)
        at org.mobicents.media.server.mgcp.controller.MgcpEndpoint.createConnection(MgcpEndpoint.java:170)
        at org.mobicents.media.server.mgcp.tx.cmd.CreateConnectionCmd$Preprocessor.perform(CreateConnectionCmd.java:286)

        at org.mobicents.media.server.scheduler.Task.run(Task.java:117)
        at org.mobicents.media.server.scheduler.Scheduler$WorkerThread.run(Scheduler.java:368)

09:38:49,484 ERROR [MGCP] tx=289032 Failed
org.mobicents.media.server.mgcp.tx.cmd.MgcpCommandException


Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[Tran Quyet Thang]

Hi Yulian,

For processing MGCP messages from the client side with 'jsr-309-driver-3.0.0.CR1.jar', what needs to be done to improve the performance  ? 

I think it must be some kind of pool here.


Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
Please take a look at mgcp pool size for mgcp server side :
https://code.google.com/p/mediaserver/source/browse/bootstrap/src/main/config/server-beans.xml#83


On  client side at least in mgcp driver i dont see any limitations , except buffer size for single message which should be enough.
Best regards
Yulian Oifa

[yulian_o]

Hello
There is pool management for rtp connections
However in java each open socket creates one file descriptor.
Same with playing announcement ( which opens wav file ) , and also there are some files opened by default which are required to run java.

There is an operating system limit per user that defines how much files can be opened.
To see it run ulimit -n.
I guess you will get 1024 , so in high loads you will get this number.
You can set ulimit -n 4096 to increase it up to 4096.

Please note that you should also check overall limit ( not just your user ) :  cat /proc/sys/fs/file-max
the value should be much greater then the limit you set , otherwise os will not have space to open its files.
You can increase it also by setting start descriptors in /etc/security/limits.conf

Best regards
Yulian Oifa

[yulian_o]

One more comment about file descriptor limits.
ulimit sets the file limits only for specific sesssion , so after restart the limit goes back to initial value.
To set it as static value you can go to /etc/security/limits.conf
and add 2 raws :
USER soft nofile 4096
USER hard nofile 4096

where USER is the username you are using to run mms.
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

smt I saw MGCP messages congested on the client side for exp:  'tx 830662670'.

The client is received the MGCP reponse- later 4 seconds. It's cause of retrying from the client side.

I think it must be some kind of pool for processing MGCP messages from the client.



log of MMS:
mms-server]$ grep 830662670 nohup.out
15:27:25,221 DEBUG [MgcpProvider] Parsing message: DLCX 830662670 mobicents/bridge/23...@10.50.145.31:12427  MGCP 1.0
15:27:25,221 INFO  [MGCP] tx=830662670 Started, message= DLCX mobicents/bridge/23...@10.50.145.31:12427, call agent = /10.50.145.31:14747
15:27:25,232 INFO  [MGCP] tx=830662670 was executed normaly
15:27:29,420 DEBUG [MgcpProvider] Parsing message: DLCX 830662670 mobicents/bridge/23...@10.50.145.31:12427  MGCP 1.0
15:27:29,420 INFO  [MGCP] tx=830662670 Started, message= DLCX mobicents/bridge/23...@10.50.145.31:12427, call agent = /10.50.145.31:14747
15:27:29,427 ERROR [MGCP] tx=830662670 Failed
15:27:29,432 INFO  [MGCP] tx=830662670 Rolled back

Log of MSS Client with ''jsr-309-driver-3.0.0.CR1.jar':
the first time
2012-10-31 15:27:25,218 DEBUG org.mobicents.protocols.mgcp.stack.TransactionHandler Send command event to /10.50.145.31, message
DLCX 830662670 mobicents/bridge/23...@10.50.145.31:12427  MGCP 1.0
C:1a29ce03190441
I:10ba90

retrying
2012-10-31 15:27:29,404 WARN org.mobicents.protocols.mgcp.stack.TransactionHandler message =
DLCX 830662670 mobicents/bridge/23...@10.50.145.31:12427  MGCP 1.0
C:1a29ce03190441
I:10ba90
 
 local Tx ID = 830662670 Remote Tx ID = 174099 Sending the Command 2

2012-10-31 15:27:29,450 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 200 830662670 Success


Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
Can you confirm that response is actually sent to wire from mms and received by client server.
It can be done by using for example tshark capture on client side.
This is really important , since we will be sure its client side and not network congestion or mms buffering , etc.

Thanks best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

I already checked in my captured file, I'm using tcpdump in Redhat.

you can see and search tx=830662670  in the attached file.



Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Thank for you reply
Please follow : https://code.google.com/p/mediaserver/issues/detail?id=94&thanks=94&ts=1351676294

It will take me some time to check the mgcp stack and found why this limitation occured
Thanks and best regards
Yulian Oifa

[yulian_o]

Hello
Please try with release http://code.google.com/p/mediaserver/source/detail?r=20f2137f80b072c6d33c6dc47c79488991111f8c

There was no any limit in mgcp stack but as it appears it reused the mgcp buffer for all messages which could lead to overwritten messages while 2 or more messages where arriving faster then in 20ms.
You can also set http://code.google.com/p/mediaserver/source/browse/client/mgcp/driver/src/main/java/org/mobicents/protocols/mgcp/stack/JainMgcpStackImpl.java#453 to increase the frequency of reqading loop ( set it higher then 50 which is default = one nessages in 20ms)

Please update the post with results so i can confirm the issue
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

The problem is continue existed. It's also marked with 'Duplicate Request' in wireshark.

I think , We need a pool or separate processing buffer and decode packets.



Regards,
Thangtq,

[yulian_o]

Hello
1) I can not find any of duplicate requests , can you point me to some of them.
Are there a lot of them , does they occures in specific time frames , since i want to compare it with network transfer graph ( number of bytes transferred ).
2) Did you set http://code.google.com/p/mediaserver/source/browse/client/mgcp/driver/src/main/java/org/mobicents/protocols/mgcp/stack/JainMgcpStackImpl.java#453 value and what is the number you set?

About your recommendation about processing buffer and decoding : i dont think it work since if the message will not get on time to higher level ( transaction manager ) transaction will resend the message again ,and thats what we want to ommit.
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

1) you can see my attached picture.
2) I already set to 1000 tps, and try to change BUFFER_SIZE, but nothing change.

For processing buffer and  decoding:
I also faced this problem in the past. when the client taking a long time to decode MGCP Messages. At the same time, the messages-UDP bytes come too fast, the buffer will be overwrite.

Redhat OS:

netstat -su
Udp:
    545360183 packets received
    82948792 packets to unknown port received.
    373414 packet receive errors
    698016479 packets sent

Regards,
Thangtq,

[yulian_o]

Hello
Ok i have found how to filter on duplicate packets ( there is such filter on mgcp ).
I will try to separate reading data from wire and parsing messages to 2 stages in mgcp client and will let you know.

Best regards
Yulian Oifa

[yulian_o]

Hello
Please try to use release http://code.google.com/p/mediaserver/source/detail?r=dcd10023062c25e717f2928c1f333c60c13d628c
I have separated reading data from udp channel and mgcp messages parsing , also previously there was single thread which allowed parsing mgcp messages, currently its 2 by default and configurable value ( in stack config file ) , depending on number of cpu cores you can increase it to increase parsing throughtput

Please let me know about the status
Best regargs
Yulian Oifa

[yulian_o]

Hello
Please use release 14305fc37e2c
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

the duplicated tx problem for multi-Clients is repeated again with this release.

nohup2:DLCX 168872528 mobicents/bridge/1...@10.50.145.32:2427  MGCP 1.0
nohup2:2012-11-05 09:41:49,213 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 200 168872528 Success
nohup3:CRCX 168872528 mobicents/bridge/$@10.50.145.32:2427 MGCP 1.0
nohup3:2012-11-05 09:41:49,410 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 200 168872528 Success
nohup6:DLCX 168872528 mobicents/bridge/4...@10.50.145.32:2427  MGCP 1.0
nohup6:2012-11-05 09:42:07,450 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 200 168872528 Success
nohup7:DLCX 168872528 mobicents/ivr/8...@10.50.145.32:2427  MGCP 1.0
nohup7:DLCX 168872528 mobicents/ivr/8...@10.50.145.32:2427  MGCP 1.0
nohup7: local Tx ID = 168872528 Remote Tx ID = 77 Sending the Command 2
nohup7:2012-11-05 09:41:49,929 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 516 168872528 Could not find this call with specified identifier
nohup9:DLCX 168872528 mobicents/ivr/5...@10.50.145.32:2427  MGCP 1.0
nohup9:2012-11-05 09:42:13,171 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 200 168872528 Success

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[Tran Quyet Thang]

Hi Yulian,

new errors in MMS3:

10:12:01,695 ERROR [Task] java.lang.NullPointerException
10:12:02,111 ERROR [RTPDataChannel] java.nio.channels.ClosedChannelException
10:12:02,111 ERROR [RTPDataChannel] java.nio.channels.ClosedChannelException
10:12:32,851 ERROR [RTPDataChannel] java.nio.channels.ClosedChannelException

--
Hi

Regards,
Thangtq,

[yulian_o]

What do you mean by duplicate tx id problem?
You mean that they are not handled correctly by mms or same message is resent by client?
In case 1 i dont see anything special about mms in this case, the only thing in your post is invalid call id which may be correct.\
If its case 2 please provider wireshark capture , and post number of threads you allowed for mgcp message parsing.

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

I'll list some errors from client side with MGCP-driver.

1. Mechanism for generating tx, its will be duplicated with multi-clients running on same server.
public class DriverImpl implements Driver, JainMgcpListener{
...
public synchronized int getNextTxID() {
        return ++txID;
    }


2. perfomance for processing and decoding MGCP messages from the client side is not improve (number of theards from 20 -> 100).

3. duplicated tx is remained: For exp: tx = 185782216

4. new exception with ParserThread

 2012-11-05 13:52:12,063 ERROR Unexpected exception occured:  (stack.JainMgcpStackImpl$ParserThread,494,run )nullorg.mobicents.protocols.mgcp.stack.MessageHandler.scheduleMessages(MessageHandler.java:137)
org.mobicents.protocols.mgcp.stack.JainMgcpStackImpl$ParserThread.run(JainMgcpStackImpl.java:488)


Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
Lets start with null pointer exception
Please modify the following in your local copy :
Instead of 
else
     logger.error(e);

in https://code.google.com/p/mediaserver/source/browse/scheduler/src/main/java/org/mobicents/media/server/scheduler/Task.java#127

Please put

else
{
   e.printStackTrace();
   logger.error(e);
}

As i saw the error occured only once in 2 logs , so i guess it will take some time to find it once again.
ClosedChannelException we will handle after that.

Best regards
Yulian Oifa

[yulian_o]

Hello
Where did you got

public class DriverImpl implements Driver, JainMgcpListener{
...
public synchronized int getNextTxID() {
        return ++txID;
    }

from?
In mgcp driver you have provider implment which has

public int getUniqueTransactionHandler()
it can be optimized by even currently it does not produces same transaction id.

If you have several mgcp stacks running on same machine then each of them have different ip address/port pair therefore it does not really matters if they use same tx id or not.
about 4 : i have updated issue 94 with the finding and will post a solution for that.

Best regards
Yulian Oifa

[yulian_o]

Hello
Please update to revision http://code.google.com/p/mediaserver/source/detail?r=592d8fe515aa
So the exception on client side will be eliminated

Please let me know if you got null pointer on server side , so i can update the code
Best regards
Yulian Oifa

[yulian_o]

Hello
I have made one more update , please test revision 89eef9605be8
Currently mms client stack is splitted into 3 stages
1) reading data
2) parsing message
3) submitting messages to listeners

I have added third stage so it can be sure that listeners are not the problem of resent messages.
Please let me know with results
Best regards
Yulian oifa

[Tran Quyet Thang]

Hi Yulian,

I found a new error from the client side:

1. Errors when releasing connection:
2012-11-06 15:27:25,089 INFO org.mobicents.javax.media.mscontrol.spi.DriverImpl MediaSession3068 NetworkConnection3079 Releasing....
org.mobicents.fsm.UnknownTransitionException: close
        at org.mobicents.fsm.State.signal(State.java:113)
        at org.mobicents.fsm.FSM.signal(FSM.java:151)
        at org.mobicents.javax.media.mscontrol.networkconnection.NetworkConnectionImpl.release(NetworkConnectionImpl.java:266)
        at org.mobicents.servlet.sip.conference.server.media.PhoneConferenceParticipant.leave(PhoneConferenceParticipant.java:118)
        at org.mobicents.servlet.sip.conference.server.media.Conference.removeParticipant(Conference.java:136)
        at org.mobicents.servlet.sip.conference.server.media.Conference.removeParticipant(Conference.java:124)
        at org.mobicents.servlet.sip.conference.server.media.ConferenceMng.removeConference(ConferenceMng.java:198)

2.the duplicated tx problem
exp: with tx = 278047706

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
Lets summarize for now
1) There are RTPDataChannel and null pointer exceptions on mms which most of chances occures when deleting connection. Currently i am waiting from you for stackTrace for null pointer exception as stated in previous posts ( see several post up ). This will allow to resolve the issue.
2) There is jsr fsm exception , i have opened issue for it : https://code.google.com/p/mediaserver/issues/detail?id=96&thanks=96&ts=1352192381
Will close it today and let you know
3) There still exists duplicate requests , but much less , as i see from pcap in 50 seconds with around 320-330 requests/responses or 160-170 requests per second there currently are 10 resent requests ( you can apply filter mgcp.req.dup to see them ).
I need following information from you to get better understanding on that packets : please restart a server , run a test and check number of bad packets ( netstat ) i want to see if all packets are read from wire now and simply there is no enough time to process them .this will eliminate one of 2 stages that can cause the problem ( reading from wire , and parsing ) , latest stage is running as separate stage and is not important in this case since it occures after timers are cancelled.

Waiting for your reply for 1 and 3
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

update error log of NullPointerException from MMS side.

17:01:08,863 ERROR [MGCP] tx=285885130 Failed
java.lang.NullPointerException
        at org.mobicents.media.server.mgcp.tx.cmd.DeleteConnectionCmd$Delete.perform(DeleteConnectionCmd.java:180)
        at org.mobicents.media.server.scheduler.Task.run(Task.java:117)
        at org.mobicents.media.server.scheduler.Scheduler$CriticalWorkerThread.run(Scheduler.java:471)

Regards,
Thangtq,

[yulian_o]

Hello
Please try to use release fffa58414846
It should resolve null pointer exception with mms and jsr 309 fsm state problem.
Please leave the servers some time under load to verify that those exceptions are not occuring anymore , waiting for confirmation on that.

Also waiting for your reply on part 3 :

3) There still exists duplicate requests , but much less , as i see from pcap in 50 seconds with around 320-330 requests/responses or 160-170 requests per second there currently are 10 resent requests ( you can apply filter mgcp.req.dup to see them ).
I need following information from you to get better understanding on that packets : please restart a server , run a test and check number of bad packets ( netstat ) i want to see if all packets are read from wire now and simply there is no enough time to process them .this will eliminate one of 2 stages that can cause the problem ( reading from wire , and parsing ) , latest stage is running as separate stage and is not important in this case since it occures after timers are cancelled.

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

I found a strange problem for duplicated request from  the client side:

- the client already received Reponse 200 OK from the MMS server but It's still resend the request.

#the client send the first delete request
2012-11-07 11:30:52,186 DEBUG org.mobicents.protocols.mgcp.stack.TransactionHandler Send command event to /10.50.145.31, message
DLCX 419563947 mobicents/bridge/2...@10.50.145.31:12427  MGCP 1.0
C:1c42f008544ff7
I:b80fc
#received the first reponse ok.
2012-11-07 11:30:52,212 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Processing response message:200 419563947 Success
I:b80fc
P:PS=207, PR=229

#the client send the second delete request
2012-11-07 11:30:52,488 DEBUG org.mobicents.protocols.mgcp.stack.TransactionHandler Send command event to /10.50.145.31, message
DLCX 419564065 mobicents/bridge/2...@10.50.145.31:12427  MGCP 1.0
C:1c42f008544ff7
I:b80fc

#received the second reponse error.
2012-11-07 11:30:52,516 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 516 419564065 Could not find this call with specified identifier

For the part 3, we'll check late, because I can't restart my server now .

Regards,
Thangtq,

[yulian_o]

Hello
You can see that 2 requests are with different request id.
That means that stack received 2 requests for deleting connection and of course sent them to wire.
MMS responded as it should in this case.
That means that requests are coming from upper layers on client side and since jsr 309 does not generates dlcx by itself but only when network connection release is called i guess that its app that sends 2 requests.
What is the application you are using? Is it some sort of JSR 309 example or its something that you wrote?

Best regards
Yulian Oifa

[Tran Quyet Thang]

 
Hi ,

I'll continue trace to root of the problem.

Please fix the follow errors:

1.
org.mobicents.fsm.UnknownTransitionException: PLAY_COMPLETED

        at org.mobicents.fsm.State.signal(State.java:113)
        at org.mobicents.fsm.FSM.signal(FSM.java:151)

        at org.mobicents.javax.media.mscontrol.mediagroup.PlayerImpl.fireEvent(PlayerImpl.java:415)
        at org.mobicents.javax.media.mscontrol.mediagroup.PlayerImpl.processMgcpCommandEvent(PlayerImpl.java:533)
        at org.mobicents.javax.media.mscontrol.spi.DriverImpl.processMgcpCommandEvent(DriverImpl.java:240)
        at org.mobicents.protocols.mgcp.stack.JainMgcpStackProviderImpl$DispatcherThread.run(JainMgcpStackProviderImpl.java:367)

2. when I using: "logger.debug("DeleteConnectionResponse:"+response);"

2012-11-07 16:02:44,331 ERROR org.mobicents.protocols.mgcp.stack.JainMgcpStackProviderImpl Unexpected exception occured:
java.lang.ArrayIndexOutOfBoundsException: 228
        at jain.protocol.ip.mgcp.message.parms.RegularConnectionParm.toString(RegularConnectionParm.java:86)
        at jain.protocol.ip.mgcp.JainMgcpEvent.BuildListParmLine(JainMgcpEvent.java:117)
        at jain.protocol.ip.mgcp.message.DeleteConnectionResponse.toString(DeleteConnectionResponse.java:97)
        at java.lang.String.valueOf(String.java:2838)
        at java.lang.StringBuilder.append(StringBuilder.java:132)
        at org.mobicents.javax.media.mscontrol.networkconnection.DeleteConnectionResponseHandler.processMgcpResponseEvent(DeleteConnectionResponseHandler.java:47)
        at org.mobicents.javax.media.mscontrol.spi.DriverImpl.processMgcpResponseEvent(DriverImpl.java:270)
        at org.mobicents.protocols.mgcp.stack.JainMgcpStackProviderImpl$DispatcherThread.run(JainMgcpStackProviderImpl.java:381)

Regards,
Thangtq,

[Tran Quyet Thang]

Hi Yulian,

I build my app from JSR 309 example.

From the client side, the buffer is overwrite , it's not pass to passing stage.

The first request
2012-11-07 16:57:43,647 DEBUG org.mobicents.protocols.mgcp.stack.TransactionHandler Send command event to /10.50.145.43, message
DLCX 436735395 mobicents/bridge/3...@10.50.145.43:2427  MGCP 1.0
C:45d3a547feeaf
I:b7481
 localTID:436735395 remoteTID:436915907

received response but i can't see the passing stage
2012-11-07 16:57:43,658 DEBUG org.mobicents.protocols.mgcp.stack.JainMgcpStackImpl receiveBuffer hex dump:000000000  32 30 30 20 34 33 36 37  33 35 33 39 35 20 53 75 |200.436735395.Su|
000000010  63 63 65 73 73 0A 49 3A  62 37 34 38 31 0A 50 3A |ccess.I:b7481.P:|
000000020  50 53 3D 33 34 39 2C 20  50 52 3D 31 37 32 0A                                                      |PS=349,.PR=172.|

the duplicated request
2012-11-07 16:57:47,649 WARN org.mobicents.protocols.mgcp.stack.TransactionHandler message =
DLCX 436735395 mobicents/bridge/3...@10.50.145.43:2427  MGCP 1.0
C:45d3a547feeaf
I:b7481
 
 local Tx ID = 436735395 Remote Tx ID = 436915907 Sending the Command 2
2012-11-07 16:57:47,680 DEBUG org.mobicents.protocols.mgcp.stack.JainMgcpStackImpl receiveBuffer hex dump:000000000  35 31 36 20 34 33 36 37  33 35 33 39 35 20 43 6F |516.436735395.Co|
000000010  75 6C 64 20 6E 6F 74 20  66 69 6E 64 20 74 68 69 |uld.not.find.thi|
000000020  73 20 63 61 6C 6C 20 77  69 74 68 20 73 70 65 63 |s.call.with.spec|
000000030  69 66 69 65 64 20 69 64  65 6E 74 69 66 69 65 72 |ified.identifier|
000000040  0A 49 3A 62 37 34 38 31  0A                      |.I:b7481.|
 
2012-11-07 16:57:47,680 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 516 436735395 Could not find this call with specified identifier

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
Can you confirm that message is not arriving to MessageHandler parsing even after the message is resend?
I mean do you see anywhere in the log DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 200 436735395 .... after the resent happened?
If its not the case then its not the time parsing problem but some exception that appears and not logged.
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

Yes, Its already arrived to the receiverbuffer but not to MessageHandler .

I only see from hexdump, and can't see anywhere.


Regards,
Thangtq,

[Tran Quyet Thang]

Hi Yulian,

some results are changed when I make tunning both sides (client, server).

1. I changed the follow things:

From the client side:
           I increased "parserThreadPoolSize" to 1000.
From the server side:
          I increased "poolSize" to 2000 in file "server-beans.xml"

2. the  problem is regarding to MMS now. Maybe MMS is congested and NullPointerException is existed

you can check with tx=437389358.

the client sent 2 requests and the server received 2 too, but processed slowly (5 second to response).

11:06:03,490 DEBUG [MgcpProvider] Parsing message: CRCX 437389358 mobicents/bridge/$@10.50.145.43:2427 MGCP 1.0
11:06:03,490 INFO  [MGCP] tx=437389358 Started, message= CRCX mobicents/bridge/$@10.50.145.43:2427, call agent = /10.50.145.43:12341
11:06:11,095 INFO  [MGCP] tx=437389358 was executed normaly
11:06:11,118 DEBUG [MgcpProvider] Parsing message: CRCX 437389358 mobicents/bridge/$@10.50.145.43:2427 MGCP 1.0
11:06:11,118 INFO  [MGCP] tx=437389358 Started, message= CRCX mobicents/bridge/$@10.50.145.43:2427, call agent = /10.50.145.43:12341
11:06:11,419 INFO  [MGCP] tx=437389358 was executed normaly


Regards,
Thangtq,

--

[yulian_o]

Hello
Please set parserThreadPoolSize to number of cpu cores you have on your server.
poolSize in mms set to 100.2K mgcp requests means it should be capable to handle around 200K messages per second , thats not something that is possible.

As it looks like you brought the memory to limits which caused mms to froze for 8 seconds , after that some resources where freed and it continue to work.
Please do not try to use "CRAZY NUMBERS" since it will not lead to anything good.Each server has it resources limit and you can not grow beyound them.
Let me know if the error occures again with poolsize=100.

Now about mgcp client :
You confirmed that message simply disappeared between reading it from wire ( which was confirmed ) and parsing it.so its not the load issue but some client bug which should be found.
after reading data from wire , it gets packet representation and puts it to queue. then parsing thread takes it from queue and sends its to message handler. message handler calls piggyDismount and then prints header.

the only thing i suspect currently is piggyDismount function which has try finally without catch clause.
In any case i would request from you the following :
1) I need you to log hex data 3 times , first time before if case https://code.google.com/p/mediaserver/source/browse/client/mgcp/driver/src/main/java/org/mobicents/protocols/mgcp/stack/JainMgcpStackImpl.java#361
here you should print receive buffer content

second time before offering it to queue https://code.google.com/p/mediaserver/source/browse/client/mgcp/driver/src/main/java/org/mobicents/protocols/mgcp/stack/JainMgcpStackImpl.java#368
here you should print pr.getRawData()

third time when taking from queue : https://code.google.com/p/mediaserver/source/browse/client/mgcp/driver/src/main/java/org/mobicents/protocols/mgcp/stack/JainMgcpStackImpl.java#493
here you should change while content to following :
PacketRepresentation pr=waitingQueue.take();
log pr.getRawData();
messageHandler.schedulerMessages(pr);

2) in MessageHandler class , piggyDismount function you have try finally clause , please add to it catch(Exception e) and log error if it occures.

Hope this will help us to find the problem with missing packets.
Let me know if you have some questions.
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

I changed parserThreadPoolSize =16 and poolsize=100.

MMS has the following errors:
1.NullPointerException
09:35:19,935 DEBUG [ResourcesPool] Allocated new rtp connection,pool size:131,free:9
        at org.mobicents.media.server.component.audio.CompoundSplitter$InsideMixTask.perform(CompoundSplitter.java:151)
        at org.mobicents.media.server.scheduler.Task.run(Task.java:117)
        at org.mobicents.media.server.scheduler.Scheduler$WorkerThread.run(Scheduler.java:447)
09:35:19,936 DEBUG [ResourcesPool] Released rtp connection,pool size:131,free:10
09:35:19,936 ERROR [Task] java.lang.NullPointerException

2. java.nio.channels.ClosedChannelException
09:36:28,692 DEBUG [ResourcesPool] Released local connection,pool size:236,free:57
        at sun.nio.ch.DatagramChannelImpl.ensureOpenAndUnconnected(DatagramChannelImpl.java:691)
        at sun.nio.ch.DatagramChannelImpl.connect(DatagramChannelImpl.java:703)
        at org.mobicents.media.server.impl.rtp.RTPDataChannel$RxTask.perform(RTPDataChannel.java:540)
09:36:28,692 DEBUG [ResourcesPool] Released local connection,pool size:236,free:58

3.UnknownTransitionException with state as "closed" from the client side

2012-11-09 09:40:41,861 ERROR DeleteConnectionResponseHandler  (networkconnection.DeleteConnectionResponseHandler,52,processMgcpResponseEvent )closedorg.mobicents.fsm.State.signal(State.java:113)
org.mobicents.fsm.FSM.signal(FSM.java:152)
org.mobicents.javax.media.mscontrol.networkconnection.DeleteConnectionResponseHandler.processMgcpResponseEvent(DeleteConnectionResponseHandler.java:50)
org.mobicents.javax.media.mscontrol.spi.DriverImpl.processMgcpResponseEvent(DriverImpl.java:272)
org.mobicents.protocols.mgcp.stack.JainMgcpStackProviderImpl$DispatcherThread.run(JainMgcpStackProviderImpl.java:387)

Please fix these errors, after that I can move next stage for testing.

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[Tran Quyet Thang]

Hi Yulian,

For finding the problem missing packets , Could you fix the errors below ?

with this MMS version , I can't generated 50 cps in a hour ~ 180000 calls.


Regards,
Thangtq,

[yulian_o]

Hello
Please be patient with this problems i am currently under big pressure in my main project.
I will solve it till the end of week.
Please notice that those issue do not influent perfomance , therefore i would recommend to move forward with debugging on client as i recommended.

Best regards
Yulian Oifa

[yulian_o]

Hello
Please use revision 679b3c6121d6
If everything is fine on server side please follow instruction i gave for client side to debug a problem

Best regards

[Tran Quyet Thang]

Hi,

I'm busy this week,

I'll resume this task soon.

Regards,
Thangtq,

[Tran Quyet Thang]

Hi Yulia,

MMS is more stable. I don't see errors now. I'm tracing some errors from the client side.

What is the meaning of "Loss of lower layer connectivity" in  MGCP reponse from the server side ?




2012-12-03 14:11:55,128 DEBUG org.mobicents.protocols.mgcp.stack.JainMgcpStackImpl poll PacketRepresentation from the Queue:000000000  44 4C 43 58 20 33 38 32  31 20 6D 6F 62 69 63 65 |DLCX.3821.mobice|
000000010  6E 74 73 2F 62 72 69 64  67 65 2F 36 33 35 40 31 |nts/bridge/635@1|
000000020  30 2E 35 30 2E 31 34 35  2E 33 32 3A 32 34 32 37 |0.50.145.32:2427|
000000030  20 4D 47 43 50 20 31 2E  30 0A 49 3A 36 30 31 64 |.MGCP.1.0.I:601d|
000000040  0A 45 3A 39 30 32 20 4C  6F 73 73 20 6F 66 20 6C |.E:902.Loss.of.l|
000000050  6F 77 65 72 20 6C 61 79  65 72 20 63 6F 6E 6E 65 |ower.layer.conne|
000000060  63 74 69 76 69 74 79 0A                          |ctivity.|
 
2012-12-03 14:11:55,128 DEBUG MessageHandler  (stack.MessageHandler,154,scheduleMessages )Message header: DLCX 3821 mobicents/bridge/6...@10.50.145.32:2427 MGCP 1.0
2012-12-03 14:11:55,128 DEBUG MessageHandler  (stack.MessageHandler,169,scheduleMessages )Processing command message = DLCX remote Tx = 3821
2012-12-03 14:11:55,129 DEBUG MgcpMessageParser  (parser.MgcpMessageParser,74,parse )Read header: DLCX 3821 mobicents/bridge/6...@10.50.145.32:2427 MGCP 1.0 from message:DLCX 3821 mobicents/bridge/6...@10.50.145.32:2427 MGCP 1.0
I:601d
E:902 Loss of lower layer connectivity



Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
That means either rtp timeout ( if enabled )
or connection state timeout , for example if it was half open and did not changed to opened for state timeout period

Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

Please fix GENERATOR below, it could be duplicated from the client side.

public abstract class TransactionHandler implements Runnable {
        /** Logger instance */
        private static final Logger logger = Logger.getLogger(TransactionHandler.class);
 
        private static int GENERATOR = (int) (System.currentTimeMillis() & 999999999);

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[Tran Quyet Thang]

Hi Yulian,

the Performance issue problem is still existed in the client side.

I don't understand, what's the real problem, the client's taken 2 seconds for moving to MessageHandle.

2012-12-07 14:23:23,863 DEBUG org.mobicents.protocols.mgcp.stack.JainMgcpStackImpl receiveBuffer hex dump:000000000  32 30 30 20 39 34 30 36  31 31 34 36 37 20 53 75 |200.940611467.Su|
2012-12-07 14:23:25,809 DEBUG MessageHandler  (stack.MessageHandler,154,scheduleMessages )Message header: 200 940611467 Success
2012-12-07 14:23:25,812 DEBUG MessageHandler  (stack.MessageHandler,246,scheduleMessages )Processing response message:200 940611467 Success
2012-12-07 14:23:25,848 DEBUG MgcpMessageParser  (parser.MgcpMessageParser,74,parse )Read header: 200 940611467 Success from message:200 940611467 Success
2012-12-07 14:23:25,851 DEBUG TransactionHandler  (stack.TransactionHandler,671,receiveResponse )process for remoteTID:806376324 TransactionHandle:806376324 commandEvent txHandle:940611467

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[Tran Quyet Thang]

Hi Yulian,

I found the problem of the client side.

all most is regarding to MessageHandler 

2012-12-07 17:27:10,346 DEBUG MessageHandler  (stack.MessageHandler,154,scheduleMessages )Message header: 200 681110748 Success
2012-12-07 17:27:12,407 DEBUG MessageHandler  (stack.MessageHandler,261,scheduleMessages )processing time of MMS:elapsedTime:2119 message:200 681110748 Success

2012-12-07 17:27:10,364 DEBUG MessageHandler  (stack.MessageHandler,154,scheduleMessages )Message header: 200 681110746 Success
2012-12-07 17:27:12,422 DEBUG MessageHandler  (stack.MessageHandler,261,scheduleMessages )processing time of MMS:elapsedTime:2112 message:200 681110746 Success


2012-12-07 17:27:10,354 DEBUG MessageHandler  (stack.MessageHandler,154,scheduleMessages )Message header: 200 681110744 Success
2012-12-07 17:27:12,400 DEBUG MessageHandler  (stack.MessageHandler,261,scheduleMessages )processing time of MMS:elapsedTime:2118 message:200 681110744 Success




Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
Please try to use the latest trunk : http://code.google.com/p/mediaserver/source/detail?r=315358e0936e4e364f97db53cb46fe802165b591

I have checked all the files under stack and util folders , organized sending queues to reduce synchronized usage and sync issues.
if you are still having the problem please measure execution time of :
http://code.google.com/p/mediaserver/source/browse/client/mgcp/driver/src/main/java/org/mobicents/protocols/mgcp/parser/MgcpMessageParser.java?r=315358e0936e4e364f97db53cb46fe802165b591

actually only Utils , and Message Parses classes are still may be problematic and possibly be optimized.
In case of respose 2 functions are used in all process.MgcpMessageParser.parse , and utils.splitStringBySpace.
So i want to measure their response time and see where the problem is.

Let me know the results
Best regards
Yulian Oifa

[thangtq]

Hi Yulian,

Please review the following results:

many errors coming:
2012-12-10 14:53:11,309 ERROR org.mobicents.protocols.mgcp.utils.PacketRepresentationFactory UtilsFactory underflow. Count = 1

Some performance issue regarding to ConcurrentLinkedList , please replace with java.util.concurrent.ConcurrentLinkedQueue

2012-12-10 14:56:02,932 DEBUG org.mobicents.protocols.mgcp.stack.JainMgcpStackImpl receiver PacketRepresentation from the Queue:000000000  32 30 30 20 31 34 37 35  31 39 30 35 38 20 53 75 |200.147519058.Su|
2012-12-10 14:56:05,018 DEBUG org.mobicents.protocols.mgcp.stack.JainMgcpStackImpl decoding Packetfrom the Queue:000000000  32 30 30 20 31 34 37 35  31 39 30 35 38 20 53 75 |200.147519058.Su|

2012-12-10 14:55:10,357 DEBUG org.mobicents.protocols.mgcp.stack.JainMgcpStackImpl receiver PacketRepresentation from the Queue:000000000  32 30 30 20 31 34 37 35  30 37 36 30 39 20 53 75 |200.147507609.Su|
2012-12-10 14:55:12,402 DEBUG org.mobicents.protocols.mgcp.stack.JainMgcpStackImpl decoding Packetfrom the Queue:000000000  32 30 30 20 31 34 37 35  30 37 36 30 39 20 53 75 |200.147507609.Su|

MessageHandler.scheduleMessages also has performance issue.

2012-12-10 14:59:10,616 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 200 147550199 Success
2012-12-10 14:59:12,858 DEBUG MessageHandler  (stack.MessageHandler,307,scheduleMessages )received a completed response:elapsedTime: 8082   parsedTime:2242 message:200 147550199 Success


2012-12-10 14:57:53,633 DEBUG org.mobicents.protocols.mgcp.stack.MessageHandler Message header: 200 147541229 Success
2012-12-10 14:57:56,009 DEBUG MessageHandler  (stack.MessageHandler,307,scheduleMessages )received a completed response:elapsedTime: 2430   parsedTime:2376 message:200 147541229 Success

Regards,
Thangtq,

[yulian_o]

Hello
1) ignore util factory underflow its not an error , but shows that it increases the pool size.
2) Where do you put log raws for receiver PacketRepresentation from the Queue and decoding Packetfrom the Queue

Best regards
Yulian oifa

[thangtq]

Hi Yulian,

Please review the below:

class JainMgcpStackImpl
if (length != 0) {
                        pr = this.prFactory.allocate();
                        receiveBuffer.get(pr.getRawData(), 0, length);
                                                logger.debug("receiver PacketRepresentation from the Queue:"
                                                    + Utils.hexDump(pr.getRawData()));
                        pr.setLength(length);                       
                        pr.setRemoteAddress(address);                       
                                                pr.receivedTime=System.currentTimeMillis();
                        inputQueue.offer(pr);                       
                    }




class DecodingThread

public void run() {
            while(active)
                try {
                                PacketRepresentation pr=inputQueue.take();
                                if(pr==null) continue;
                                logger.debug("decoding Packetfrom the Queue:"
                                + Utils.hexDump(pr.getRawData()));
                    messageHandler.scheduleMessages(pr);
                }

Regards,
Thangtq,

[yulian_o]

Hi
that does not shows that there is perfomance problem with concurrent list.
you should understand that after taking the packet it should be decoded , so if there is a lot of packets in queue because of slow parsing queue will be full and therefore time will be as you see.
if you will write 2 raws , before take and after take you will see that time for take is low , same with offer.
Also checking for null after take is useless , thats because take will never return null , it blocks untill new pr is available , in difference with poll which does not blocks.

so basically there still is parsing problem , i will check it out
Best regards
Yulian Oifa

[thangtq]

Hi Yulian,

In the past, I already met null, performance issues with ConcurrentLinkedList ,  you should replace with java.util.concurrent.ConcurrentLinkedQueue.

grep "takenTime:[2-9][0-9][0-9][0-9]" nohup.out | more
 takenTime:2023
 takenTime:2021
 takenTime:2116
 takenTime:2111
 takenTime:2161
 takenTime:2178
 takenTime:2210
 takenTime:2233
 takenTime:2254

if (length != 0) {
                        pr = this.prFactory.allocate();
                        receiveBuffer.get(pr.getRawData(), 0, length);
                                                logger.debug("receiver PacketRepresentation from the Queue:"
                                                    + Utils.hexDump(pr.getRawData()));
                        pr.setLength(length);                       
                        pr.setRemoteAddress(address);                       
                                                pr.receivedTime=System.currentTimeMillis();

                                                pr.takingTime=System.currentTimeMillis();
                        inputQueue.offer(pr);                       
                    }

class DecodingThread
public void run() {
            while(active)
                try {
                                PacketRepresentation pr=inputQueue.take();

//                                if(pr==null) continue;

                                logger.debug("decoding Packetfrom the Queue:"
                                + Utils.hexDump(pr.getRawData())

                                        + " takenTime:"+(System.currentTimeMillis()-pr.takingTime));
                    messageHandler.scheduleMessages(pr);



Regards,
Thangtq,

[yulian_o]

Hi
Check this :

if (length != 0) {
                        pr = this.prFactory.allocate();
                        receiveBuffer.get(pr.getRawData(), 0, length);
                                                logger.debug("receiver PacketRepresentation from the Queue:"
                                                    + Utils.hexDump(pr.getRawData()));
                        pr.setLength(length);                       
                        pr.setRemoteAddress(address);                       
                                                pr.receivedTime=System.currentTimeMillis();

                        inputQueue.offer(pr);  
logger.debug("Offer time:" +(System.currentTimeMillis()-pr.receiveTime));                    
                    }

class DecodingThread
public void run() {
            while(active)
                try {
                              

                pr.takingTime=System.currentTimeMillis();
                PacketRepresentation pr=inputQueue.take();

//                            

    if(pr==null) continue;

                                logger.debug("decoding Packetfrom the Queue:"
                                + Utils.hexDump(pr.getRawData())

                                        + " takenTime:"+(System.currentTimeMillis()-pr.takingTime));
                    messageHandler.scheduleMessages(pr);

This way you can see how much time take and offer functions takes.
Best regards
Yulian Oifa

[yulian_o]

Hello
Please update to latest trunk
I have modified Mgcp Parser class and checked timing it works much better now.
However there is still utils class usage for parsing parameters , i will update those functions in next release.
Best regards
Yulian Oifa

[Tran Quyet Thang]

Hi Yulian,

Please check this version carefully.

ConcurrentLinkedList has performance isusses

public E take()
    {
            takeTime=System.currentTimeMillis();
        aquireAccess();
        ......
        releaseAccess();
                logger.debug("takeTime:"+(System.currentTimeMillis()-takeTime));
        return currValue;
    }

2012-12-11 14:58:26,213 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:5196
2012-12-11 14:58:26,251 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:5234
2012-12-11 14:58:26,283 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:5266
2012-12-11 14:58:26,328 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:114
2012-12-11 14:58:26,350 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:67
2012-12-11 14:58:26,356 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:50
2012-12-11 14:58:26,440 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:110
2012-12-11 14:58:26,472 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:116
2012-12-11 14:58:26,477 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:120
2012-12-11 14:58:26,553 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:112


2012-12-11 14:58:26,213 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList offerTime:0
2012-12-11 14:58:26,214 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList offerTime:0


the client can't parse sdp from MMS. I can't run my test application.

2012-12-11 14:58:26,432 DEBUG SipApplicationSessionImpl  (session.SipApplicationSessionImpl,405,setLastAccessedTime )Re-Scheduling sip application session 9f69635c-2f78-4d3d-a39d-6bc1bbe56ce1:VIVRClientTest to expire in 3 minutes
t=0 0
m=audio 65530 RTP/AVP 97 8 0 101
a=rtpmap:97 l16/8000
a=rtpmap:8 pcma/8000
a=rtpmap:0 pcmu/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
 
2012-12-11 14:58:26,477 DEBUG org.mobicents.protocols.mgcp.parser.MgcpMessageParser Read session description: v=0o=- 1355212706464 1 IN IP4 10.50.145.32s=Mobicents Media Serverc=IN IP4 10.50.145.32t=0 0m=audio 65530 RTP/AVP 97 8 0 101a=rtpmap:97 l16/8000a=rtpmap:8 pcma/8000a=rtpmap:0 pcmu/8000a=rtpmap:101 telephone-event/8000a=fmtp:101 0-15
2012-12-11 14:58:26,477 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList offerTime:0
2012-12-11 14:58:26,477 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:120
java.text.ParseException: v=0o=- 1355212706464 1 IN IP4 10.50.145.32s=Mobicents Media Serverc=IN IP4 10.50.145.32t=0 0m=audio 65530 RTP/AVP 97 8 0 101a=rtpmap:97 l16/8000a=rtpmap:8 pcma/8000a=rtpmap:0 pcmu/8000a=rtpmap:101 telephone-event/8000a=fmtp:101 0-15
        at gov.nist.core.LexerCore.createParseException(LexerCore.java:761)
        at gov.nist.javax.sdp.parser.ProtoVersionFieldParser.protoVersionField(ProtoVersionFieldParser.java:66)
        at gov.nist.javax.sdp.parser.ProtoVersionFieldParser.parse(ProtoVersionFieldParser.java:71)
        at gov.nist.javax.sdp.parser.SDPAnnounceParser.parse(SDPAnnounceParser.java:113)
        at javax.sdp.SdpFactory.createSessionDescription(SdpFactory.java:129)
        at org.mobicents.javax.media.mscontrol.networkconnection.SdpPortManagerImpl.setLocalDescriptor(SdpPortManagerImpl.java:302)
        at org.mobicents.javax.media.mscontrol.networkconnection.CreateConnectionResponseHandler.processMgcpResponseEvent(CreateConnectionResponseHandler.java:57)
        at org.mobicents.javax.media.mscontrol.spi.DriverImpl.processMgcpResponseEvent(DriverImpl.java:270)
        at org.mobicents.protocols.mgcp.stack.JainMgcpStackProviderImpl$DispatcherThread.run(JainMgcpStackProviderImpl.java:362)
javax.sdp.SdpParseException: Could not parse message
        at javax.sdp.SdpFactory.createSessionDescription(SdpFactory.java:132)
        at org.mobicents.javax.media.mscontrol.networkconnection.SdpPortManagerImpl.setLocalDescript




Regards,
Thangtq,

--
Hi

Regards,
Thangtq,

[yulian_o]

Hello
1) please change https://code.google.com/p/mediaserver/source/browse/client/mgcp/driver/src/main/java/org/mobicents/protocols/mgcp/parser/MgcpMessageParser.java#101
sdp.append(line);
to
sdp.append(line).append("\r\n");

i will commit it later.

2)Please take a look on your parse log.

2012-12-11 14:58:26,477 DEBUG org.mobicents.protocols.mgcp.parser.MgcpMessageParser Read session description: v=0o=- 1355212706464 1 IN IP4 10.50.145.32s=Mobicents Media Serverc=IN IP4 10.50.145.32t=0 0m=audio 65530 RTP/AVP 97 8 0 101a=rtpmap:97 l16/8000a=rtpmap:8 pcma/8000a=rtpmap:0 pcmu/8000a=rtpmap:101 telephone-event/8000a=fmtp:101 0-15
2012-12-11 14:58:26,477 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList offerTime:0
2012-12-11 14:58:26,477 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:120

it took for take 120ms , BUT , as you can see the packet has just been offered and therefore 120 means it was in blocking stage for 120ms. Actuall run starting from packet offered and packet taken is 0 ms as you see.
Offer time is always 0 because it never block , just writes the data to list.
First numbers are high because it took time from stack started and first messages arrived.
To get a real number you should see how much time it takes when server is under load , also another point is that mms uses this class in all critical processes including scheduler and as you see there you dont have perfomance issues.
Please also note that there is at least 4 usages of Concurrent linked list in stack , first for sending queue , second for receiving queue , and send working fine , if you had a problem with this class you should met this problem while sending data also , isnt it?

See here :
https://code.google.com/p/mediaserver/source/browse/client/mgcp/driver/src/main/java/org/mobicents/protocols/mgcp/stack/JainMgcpStackImpl.java#122
And here:
https://code.google.com/p/mediaserver/source/browse/client/mgcp/driver/src/main/java/org/mobicents/protocols/mgcp/stack/JainMgcpStackProviderImpl.java#77

Best regards
Yulian oifa

[Tran Quyet Thang]

Hi Yulian,

My server is already under load (5 cps). But takeTime is so long, not stable.

2012-12-11 16:34:30,284 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:5181
2012-12-11 16:34:30,313 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:5210
2012-12-11 16:34:30,322 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:5219
2012-12-11 16:34:30,397 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:111
2012-12-11 16:34:30,422 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:98
2012-12-11 16:34:30,425 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:76
2012-12-11 16:34:30,509 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:110
2012-12-11 16:34:30,542 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:117
2012-12-11 16:34:30,544 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:118
2012-12-11 16:34:30,620 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:99
2012-12-11 16:34:30,640 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:95
2012-12-11 16:34:30,643 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:97
2012-12-11 16:34:30,731 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:111
2012-12-11 16:34:30,745 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:102
2012-12-11 16:34:30,747 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:102
2012-12-11 16:34:30,840 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:109
2012-12-11 16:34:30,864 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:115
2012-12-11 16:34:30,867 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:117
2012-12-11 16:34:31,280 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:439
2012-12-11 16:34:31,306 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:438
2012-12-11 16:34:31,310 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:441
2012-12-11 16:34:31,312 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:31
2012-12-11 16:34:31,313 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:1
2012-12-11 16:34:31,314 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:2
2012-12-11 16:34:31,320 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:5
2012-12-11 16:34:31,322 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:2
2012-12-11 16:34:31,324 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:13
2012-12-11 16:34:31,326 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:14
2012-12-11 16:34:31,348 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:22
2012-12-11 16:34:31,349 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:23
2012-12-11 16:34:31,351 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:0
2012-12-11 16:34:31,351 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:24
2012-12-11 16:34:31,352 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:0
2012-12-11 16:34:31,352 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:0
2012-12-11 16:34:31,355 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:0
2012-12-11 16:34:31,355 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:32
2012-12-11 16:34:31,355 DEBUG org.mobicents.protocols.mgcp.stack.ConcurrentLinkedList takeTime:0


new errors:
2012-12-11 16:34:36,013 ERROR org.mobicents.protocols.mgcp.stack.TransactionHandler Coud not parse message:
java.text.ParseException: Unexpected event name AU/oc
        at org.mobicents.protocols.mgcp.stack.NotifyHandler.decodeCommand(NotifyHandler.java:72)
        at org.mobicents.protocols.mgcp.stack.TransactionHandler.receiveRequest(TransactionHandler.java:611)
        at org.mobicents.protocols.mgcp.stack.MessageHandler.scheduleMessages(MessageHandler.java:279)
        at org.mobicents.protocols.mgcp.stack.JainMgcpStackImpl$DecodingThread.run(JainMgcpStackImpl.java:460)

one more- null error(from the past, I searched some old-log files), regarding to ConcurrentLinkedList

14:58:11,709 ERROR [MGCP] tx=965221207 Failed
java.lang.NullPointerException
        at org.mobicents.media.server.component.audio.CompoundOutput.stop(CompoundOutput.java:81)
        at org.mobicents.media.server.impl.resource.dtmf.DetectorImpl.deactivate(DetectorImpl.java:154)
        at org.mobicents.media.core.endpoints.MediaGroup.releaseDtmfDetector(MediaGroup.java:171)
        at org.mobicents.media.core.endpoints.MediaGroup.releaseAll(MediaGroup.java:333)
        at org.mobicents.media.core.endpoints.BaseEndpointImpl.deleteConnection(BaseEndpointImpl.java:235)
        at org.mobicents.media.core.endpoints.BaseMixerEndpointImpl.deleteConnection(BaseMixerEndpointImpl.java:92)
        at org.mobicents.media.core.connections.BaseConnection.releaseConnection(BaseConnection.java:398)
        at org.mobicents.media.core.connections.LocalConnectionImpl.onClosed(LocalConnectionImpl.java:150)
        at org.mobicents.media.core.connections.BaseConnection.close(BaseConnection.java:248)
        at org.mobicents.media.core.endpoints.BaseEndpointImpl.deleteConnection(BaseEndpointImpl.java:213)
        at org.mobicents.media.server.mgcp.controller.MgcpEndpoint.deleteConnection(MgcpEndpoint.java:211)
        at org.mobicents.media.server.mgcp.tx.cmd.DeleteConnectionCmd$Delete.perform(DeleteConnectionCmd.java:185)
        at org.mobicents.media.server.scheduler.Task.run(Task.java:117)
        at org.mobicents.media.server.scheduler.Scheduler$CriticalWorkerThread.run(Scheduler.java:471)
14:58:11,711 ERROR [Task] java.lang.NullPointerException


In my opinion , you'd replace ConcurrentLinkedList with a matured - java.util.concurrent.ConcurrentLinkedQueue

Regards,
Thangtq,

--
Hi

Regards,
Thangtq,