Tuning of Jain SIP Stack to avoid retransmissions (Jain SLEE)
659 views
Skip to first unread message
Klein Stephan
May 2, 2012, 9:11:10 AM5/2/12
to mobicent...@googlegroups.com
Hi mobicents community,
I’m new to Mobicents Jain SLEE and currently evaluating how it performs as a HA AS for SIP based Applications.
Therefore we are executing High Load tests to find the maximum performance limits of the platform on a carrier grade hardware:
HP DL380 G7
24 Intel Xeon CPU @ 3.33 GHz
48 GB RAM
I’m using Mobicents Jain Slee 2.6.0 Final (no cluster – default mode) and sip-11-ra provided with this package and the sip-b2bua example from mobicents as application.
JVM is tuned according to our experience with other Jain SLEE platforms.
Now here comes the issue:
Until 600 CPS the SIPP-Traffic is stable.
At 600 CPS the INVITE retransmission counter starts to increase steadily. At a rather constant rate INVITEs are simply not answered (with trying) and SIPP has to retransmit them – this happens steadily (not only during full GC).
Overall at 600 CPS this happens for ~1 % of calls whereas 99% of calls are answered within 100ms – so there are still enough resources, also CPU is rather low at this rate.
A tshark interface trace shows that those INVITEs are received the Jain SLEE Server’s IP Interface (so its not a connection or sipp issue)
When enabling the Jain SIP Message Logging (gov.nist.javax.sip.LOG_MESSAGE_CONTENT) it can be seen that those INVITEs are NOT shown in the log – only the retransmissions after are shown and processed fast.
So I assume that the SIP stack somehow swallows or ignores those INVITEs (even before logging them)
Also interesting is that when starting the traffic already with 600 CPS it takes some time (~20000 messages) until the retransmissions start.
This may indicate that some queue/buffer exceeds its size.
I already tried to tune some of the following preconfigured sipra.properties:
THREAD_POOL_SIZE from 8 to 64
RECEIVE_UDP_BUFFER_SIZE and SEND_UDP_BUFFER_SIZE to the double and triple value.
MAX_SERVER_TRANSACTIONS and MAX_CLIENT_TRANSACTIONS to the double value
None of the changes prevented or delayed the retransmissions.
So I’m asking if someone experienced the same or similar issue and knows how to deal with it ? Are there further sip-stack properties which are worth tuning ? Maybe someone knows how and what areas are worth investigating ? (maybe sip stack provides some possibility to detect buffer overflows ?) Is it possible that some mobicents extension of the sip-stack interferes here ?
Thanks in advance for any feedback.
Best regards,
Stephan Klein – Kapsch CarrierCom
---
P.S: A special hello to our colleagues from Lithuania ;)
The information contained in this e-mail message is privileged and
confidential and is for the exclusive use of the addressee. The person
who receives this message and who is not the addressee, one of his
employees or an agent entitled to hand it over to the addressee, is
informed that he may not use, disclose or reproduce the contents
thereof, and is kindly asked to notify the sender and delete the e-mail
immediately.
Eduardo Martins
May 2, 2012, 2:09:50 PM5/2/12
to mobicent...@googlegroups.com
Hi, since we are close to 2.7 release, you may want to go for the current snapshot instead, it includes many enhancements and fixes (possibly more bugs too, but don't tell that to anyone). It simplifies jsip stack logging ( details at http://code.google.com/p/mobicents/issues/detail?id=3112 ), by redirecting configuration directly to log4j. You can get it at
http://hudson.jboss.org/hudson/view/Mobicents/job/Mobicents-Slee-2.x-Release/lastSuccessfulBuild/artifact/mobicents-jainslee-2.7.0-SNAPSHOT-jboss-5.1.0.GA.zip
For a load test you should go for the production log4j config, which can be found at:
To use simply overwrite jboss-5.1.0.GA/server/default/conf/jboss-log4j.xml with it.
Then you can apply the normal configuration tweaks for performance, this is what I do before running load tests:
sed -e 's|<property name="collectStats">true</property>|<property name="collectStats">false</property>|' -e 's|<property name="timeBetweenLivenessQueries">60</property>|<property name="timeBetweenLivenessQueries">0</property>|' -e 's|<property name="confirmSbbEntityAttachement">true</property>|<property name="confirmSbbEntityAttachement">false</property>|' -i jboss-5.1.0.GA/server/default/deploy/mobicents-slee/META-INF/jboss-beans.xml
Mobicents SLEE performance is very bound to memory available, allocate the max you can in your JAVA_OPTS, here is what I usually set for my 8 GB laptop (you could go further near the limit with a dedicated box, but 48GB is a lot of memory):
export JAVA_OPTS="-Xmn256m -XX:CMSInitiatingOccupancyFraction=80 -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -XX:MaxPermSize=256M -Xmx7000M -Xms512M -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true"
Changing JSIP number of threads, or the SLEE event router threads (config found in the jboss-beans.xml referenced above), will probably not help much, or anything at all, but you can always trial different values. Go with steps that double the previous values. The same for the buffer sizes you referred.
With respect to SIPP, don't go for max candidate load right away, let the server's JVM warm up, increase the load progressively.
We get around 500 cps for such test, with an old server, which having only 10 GB RAM and an outdated processor, should mean your server, after tweaked, should go way higher. If not that would be very interesting feedback, which should definitely be investigated much deeper.
By the way, once you tweak it, could you check out also the sip-uas load test, very curious what such server would manage... And we could possibly come up with an updated doc/tutorial for performance tweaking.
Eduardo Martins
May 2, 2012, 2:15:07 PM5/2/12
to mobicent...@googlegroups.com
Also, retransmissions are a known indicator of reaching about 80% of the performance limit for Mobicents SLEE SIP apps, with our test server, dunno if that will be similar for your case, and that it's an important feedback to extract from your experiences too.
Vilius Panevėžys
May 3, 2012, 4:51:27 AM5/3/12
to mobicent...@googlegroups.com
On Wed, 2 May 2012 15:11:10 +0200
Klein Stephan <Stepha...@kapsch.net> wrote:
> P.S: A special hello to our colleagues from Lithuania ;)
--
Vilius Panevėžys
Elitnet
Vilius Panevėžys
May 3, 2012, 5:05:54 AM5/3/12
to mobicent...@googlegroups.com
On Wed, 2 May 2012 19:09:50 +0100
Eduardo Martins <emma...@gmail.com> wrote:
> Mobicents SLEE performance is very bound to memory available,
> allocate the max you can in your JAVA_OPTS
the longer the collections.
Given the goal is minimal latency, not max throughput (which is the
case for telco systems), I think it's better to asses the size of
required heap based on the set of specific applications to be run and
the expected load. Allowing for some reserve, of course, but throwing
at it all you've got doesn't sound good with regard to minimizing GC
pauses, which can often be the main obstacle in achieving _stable_ and
low latency.
--
Vilius Panevėžys
Elitnet
Eduardo Martins
May 3, 2012, 5:55:38 AM5/3/12
to mobicent...@googlegroups.com
In our tests the heap size we use is not big enough to accomodate jain sip stack needs of memory (very mem hungry best), for instance in the sip uas you can run forever with lowest latency at 80% max perf, but when you start reaching 1100 cps the stack grows so much that GC becomes a harder process, and retransmissions start showing up, and latency goes up. My guess is that if we had a bigger heap we could increase load, jsip stack would eat more memory, but GC would not need to go hard, thus low latency would be kept... I guess his results will clarify that ;-)
Vilius Panevėžys
May 3, 2012, 6:36:51 AM5/3/12
to mobicent...@googlegroups.com
Well, I agree that lack of free memory would cause an even greater GC
impact than a heap that is just bigger than required. I just don't
think that using all the available memory is a good thing to do
(unless the amount of memory is really needed for the given setup).
Anyway, this ends with the rule that there is no universal rule.
Testing and measuring - the only way to find the optimal settings for a
given setup.
--
Vilius Panevėžys
Elitnet
On Thu, 3 May 2012 10:55:38 +0100
impact than a heap that is just bigger than required. I just don't
think that using all the available memory is a good thing to do
(unless the amount of memory is really needed for the given setup).
Anyway, this ends with the rule that there is no universal rule.
Testing and measuring - the only way to find the optimal settings for a
given setup.
--
Vilius Panevėžys
Elitnet
On Thu, 3 May 2012 10:55:38 +0100
Eduardo Martins
May 3, 2012, 7:34:45 AM5/3/12
to mobicent...@googlegroups.com
Just a clarification, we go for max memory possible only on our RHEL QA Labs boxes, cause it is possible to use almost all memory when using it as dedicated server just for SLEE, and cause we know that 9/10GB is not enough to explore all the performance we could achieve. I don't think setting the heap for 48 GB will make sense, it's very probable that 16-32 GB will at some point mean no more performance, due to CPU ;-)
stepha...@kapsch.net
May 3, 2012, 10:11:08 AM5/3/12
to mobicents-public
Thanks for all the comments and tips.
I followed them and this is the new test profile:
- Upgraded to Mobicents 2.7.0 Snapshot - Build 1241
- Used jboss-log4j.xml.production (also before Debug was deactivated)
- Applied your changes to jboss-beans.xml (sed commands)
- Switched to sip-uas testapp from examples
- Additional JVM Opts (had the same before - well performing on other
JainSLEE): -Xms6G -Xmx6G -XX:NewSize=384M -XX:MaxNewSize=384M -
XX:PermSize=512M -XX:MaxPermSize=512M -XX:+UseLargePages -
XX:LargePageSizeInBytes=32M -XX:+UseParNewGC -XX:YoungPLABSize=192k -
XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=75 -XX:+UseConcMarkSweepGC -
XX:CMSMarkStackSize=32M -XX:CMSMarkStackSizeMax=32M -XX:
+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -
XX:CMSInitiatingOccupancyFraction=50 -XX:
+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC
Results:
- Smooth execution up to 1000 CPS (fast responses and no
retransmissions)
- At 1100 CPS Retransmissions start to happen continously (always -
not only at full GC)
- Overall (after 100000 sessions with 1100 CPS) about 1% of INVITEs
were retransmitted, whereas 99% where answered fast
- CPU Load was only about ~15% at 1100 CPS!
Interpretation/Comments:
- SIP-UAS Application Scenario produces about half the workload
compared to SIP-B2BUA, this explains the raise from 600 CPS before to
1100 CPS now
- At 1100 the system does not reach its limit at all. Memory Profile
looks smooth and CPU Load is rather low. Nevertheless retransmissions
show that it does not make sense to raise the load, as the border of
High Availablity seems to be reached here.
So i would like to ask you some further questions:
- Eduardo, I understood that you are interested in the results. Please
let me know how i can contribute here and document them for the
mobicents community.
- Any other ideas to push the limit and avoid retransmissions ?
- Is it normal that SIP Apps (like sip-uas) do not respond with 100
Trying? I noticed this also for sip-b2bua so i added some code to the
sbb so that 100 Trying is sent. Now with sip-uas i have the same issue
and i would prefer to get 100 Trying to have a clean scenario.
ty
-- Stephan
On 2 Mai, 20:15, Eduardo Martins <emmart...@gmail.com> wrote:
> Also, retransmissions are a known indicator of reaching about 80% of the
> performance limit for Mobicents SLEE SIP apps, with our test server, dunno
> if that will be similar for your case, and that it's an important feedback
> to extract from your experiences too.
>
> -- Eduardo
> ..............................................http://emmartins.blogspot.com
> > On Wed, May 2, 2012 at 2:11 PM, Klein Stephan <Stephan.Kl...@kapsch.net>wrote:
>
> >> Hi mobicents community,****
>
> >> ** **
>
> >> ** **
>
> >> HP DL380 G7****
>
> >> 24 Intel Xeon CPU @ 3.33 GHz****
>
> >> 48 GB RAM****
>
> >> ** **
>
> >> JVM is tuned according to our experience with other Jain SLEE platforms.*
> >> ***
>
> >> ** **
>
> >> Now here comes the issue:****
>
> >> Until 600 CPS the SIPP-Traffic is stable.****
>
> >> ** **
>
> >> ** **
>
> >> ** **
> >> *
>
> >> This may indicate that some queue/buffer exceeds its size.****
>
> >> ** **
>
> >> THREAD_POOL_SIZE from 8 to 64****
>
> >> MAX_SERVER_TRANSACTIONS and MAX_CLIENT_TRANSACTIONS to the double value**
> >> **
>
> >> ** **
>
> >> None of the changes prevented or delayed the retransmissions.****
>
> >> ** **
>
> >> ** **
>
> >> Thanks in advance for any feedback.****
>
> >> ** **
>
> >> Best regards,****
>
> >> Stephan Klein – Kapsch CarrierCom****
>
> >> ---****
>
> >> ** **
>
> >> P.S: A special hello to our colleagues from Lithuania ;)****
I followed them and this is the new test profile:
- Upgraded to Mobicents 2.7.0 Snapshot - Build 1241
- Used jboss-log4j.xml.production (also before Debug was deactivated)
- Applied your changes to jboss-beans.xml (sed commands)
- Switched to sip-uas testapp from examples
- Additional JVM Opts (had the same before - well performing on other
JainSLEE): -Xms6G -Xmx6G -XX:NewSize=384M -XX:MaxNewSize=384M -
XX:PermSize=512M -XX:MaxPermSize=512M -XX:+UseLargePages -
XX:LargePageSizeInBytes=32M -XX:+UseParNewGC -XX:YoungPLABSize=192k -
XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=75 -XX:+UseConcMarkSweepGC -
XX:CMSMarkStackSize=32M -XX:CMSMarkStackSizeMax=32M -XX:
+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -
XX:CMSInitiatingOccupancyFraction=50 -XX:
+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC
Results:
- Smooth execution up to 1000 CPS (fast responses and no
retransmissions)
- At 1100 CPS Retransmissions start to happen continously (always -
not only at full GC)
- Overall (after 100000 sessions with 1100 CPS) about 1% of INVITEs
were retransmitted, whereas 99% where answered fast
- CPU Load was only about ~15% at 1100 CPS!
Interpretation/Comments:
- SIP-UAS Application Scenario produces about half the workload
compared to SIP-B2BUA, this explains the raise from 600 CPS before to
1100 CPS now
- At 1100 the system does not reach its limit at all. Memory Profile
looks smooth and CPU Load is rather low. Nevertheless retransmissions
show that it does not make sense to raise the load, as the border of
High Availablity seems to be reached here.
So i would like to ask you some further questions:
- Eduardo, I understood that you are interested in the results. Please
let me know how i can contribute here and document them for the
mobicents community.
- Any other ideas to push the limit and avoid retransmissions ?
- Is it normal that SIP Apps (like sip-uas) do not respond with 100
Trying? I noticed this also for sip-b2bua so i added some code to the
sbb so that 100 Trying is sent. Now with sip-uas i have the same issue
and i would prefer to get 100 Trying to have a clean scenario.
ty
-- Stephan
On 2 Mai, 20:15, Eduardo Martins <emmart...@gmail.com> wrote:
> Also, retransmissions are a known indicator of reaching about 80% of the
> performance limit for Mobicents SLEE SIP apps, with our test server, dunno
> if that will be similar for your case, and that it's an important feedback
> to extract from your experiences too.
>
> -- Eduardo
>
>
>
>
>
>
>
> On Wed, May 2, 2012 at 7:09 PM, Eduardo Martins <emmart...@gmail.com> wrote:
> > Hi, since we are close to 2.7 release, you may want to go for the current
> > snapshot instead, it includes many enhancements and fixes (possibly more
> > bugs too, but don't tell that to anyone). It simplifies jsip stack logging
> > ( details athttp://code.google.com/p/mobicents/issues/detail?id=3112),
> > by redirecting configuration directly to log4j. You can get it at
>
> >http://hudson.jboss.org/hudson/view/Mobicents/job/Mobicents-Slee-2.x-...
>
>
>
>
>
>
> On Wed, May 2, 2012 at 7:09 PM, Eduardo Martins <emmart...@gmail.com> wrote:
> > Hi, since we are close to 2.7 release, you may want to go for the current
> > snapshot instead, it includes many enhancements and fixes (possibly more
> > bugs too, but don't tell that to anyone). It simplifies jsip stack logging
> > ( details athttp://code.google.com/p/mobicents/issues/detail?id=3112),
> > by redirecting configuration directly to log4j. You can get it at
>
>
> > For a load test you should go for the production log4j config, which can
> > be found at:
>
> > jboss-5.1.0.GA/server/default/deploy/mobicents-slee/log4j-templates/jboss-l og4j.xml.production
>
> > To use simply overwrite jboss-5.1.0.GA/server/default/conf/jboss-log4j.xmlwith it.
> > For a load test you should go for the production log4j config, which can
> > be found at:
>
> > jboss-5.1.0.GA/server/default/deploy/mobicents-slee/log4j-templates/jboss-l og4j.xml.production
>
>
> >> Hi mobicents community,****
>
> >> ** **
>
> >> I’m new to Mobicents Jain SLEE and currently evaluating how it performs
> >> as a HA AS for SIP based Applications.****
> >> I’m new to Mobicents Jain SLEE and currently evaluating how it performs
>
> >> Therefore we are executing High Load tests to find the maximum
> >> performance limits of the platform on a carrier grade hardware:****
> >> Therefore we are executing High Load tests to find the maximum
>
> >> ** **
>
> >> HP DL380 G7****
>
> >> 24 Intel Xeon CPU @ 3.33 GHz****
>
> >> 48 GB RAM****
>
> >> ** **
>
> >> I’m using Mobicents Jain Slee 2.6.0 Final (no cluster – default mode) and
> >> sip-11-ra provided with this package and the sip-b2bua example from
> >> mobicents as application.****
> >> I’m using Mobicents Jain Slee 2.6.0 Final (no cluster – default mode) and
> >> sip-11-ra provided with this package and the sip-b2bua example from
>
> >> JVM is tuned according to our experience with other Jain SLEE platforms.*
> >> ***
>
> >> ** **
>
> >> Now here comes the issue:****
>
> >> Until 600 CPS the SIPP-Traffic is stable.****
>
> >> At 600 CPS the INVITE retransmission counter starts to increase steadily.
> >> At a rather constant rate INVITEs are simply not answered (with trying) and
> >> SIPP has to retransmit them – this happens steadily (not only during full
> >> GC).****
> >> At 600 CPS the INVITE retransmission counter starts to increase steadily.
> >> At a rather constant rate INVITEs are simply not answered (with trying) and
> >> SIPP has to retransmit them – this happens steadily (not only during full
>
> >> Overall at 600 CPS this happens for ~1 % of calls whereas 99% of calls
> >> are answered within 100ms – so there are still enough resources, also CPU
> >> is rather low at this rate.****
> >> Overall at 600 CPS this happens for ~1 % of calls whereas 99% of calls
> >> are answered within 100ms – so there are still enough resources, also CPU
>
> >> ** **
>
> >> A tshark interface trace shows that those INVITEs are received the Jain
> >> SLEE Server’s IP Interface (so its not a connection or sipp issue)****
> >> A tshark interface trace shows that those INVITEs are received the Jain
>
> >> When enabling the Jain SIP Message Logging
> >> (gov.nist.javax.sip.LOG_MESSAGE_CONTENT) it can be seen that those INVITEs
> >> are NOT shown in the log – only the retransmissions after are shown and
> >> processed fast.****
> >> When enabling the Jain SIP Message Logging
> >> (gov.nist.javax.sip.LOG_MESSAGE_CONTENT) it can be seen that those INVITEs
> >> are NOT shown in the log – only the retransmissions after are shown and
>
> >> ** **
>
> >> So I assume that the SIP stack somehow swallows or ignores those INVITEs
> >> (even before logging them)****
> >> So I assume that the SIP stack somehow swallows or ignores those INVITEs
>
> >> ** **
>
> >> Also interesting is that when starting the traffic already with 600 CPS
> >> it takes some time (~20000 messages) until the retransmissions start. ***
> >> Also interesting is that when starting the traffic already with 600 CPS
> >> *
>
> >> This may indicate that some queue/buffer exceeds its size.****
>
> >> ** **
>
> >> I already tried to tune some of the following preconfigured
> >> sipra.properties:****
> >> I already tried to tune some of the following preconfigured
>
> >> THREAD_POOL_SIZE from 8 to 64****
>
> >> RECEIVE_UDP_BUFFER_SIZE and SEND_UDP_BUFFER_SIZE to the double and triple
> >> value.****
> >> RECEIVE_UDP_BUFFER_SIZE and SEND_UDP_BUFFER_SIZE to the double and triple
>
> >> MAX_SERVER_TRANSACTIONS and MAX_CLIENT_TRANSACTIONS to the double value**
> >> **
>
> >> ** **
>
> >> None of the changes prevented or delayed the retransmissions.****
>
> >> ** **
>
> >> So I’m asking if someone experienced the same or similar issue and knows
> >> how to deal with it ? Are there further sip-stack properties which are
> >> worth tuning ? Maybe someone knows how and what areas are worth
> >> investigating ? (maybe sip stack provides some possibility to detect buffer
> >> overflows ?) Is it possible that some mobicents extension of the sip-stack
> >> interferes here ?****
> >> So I’m asking if someone experienced the same or similar issue and knows
> >> how to deal with it ? Are there further sip-stack properties which are
> >> worth tuning ? Maybe someone knows how and what areas are worth
> >> investigating ? (maybe sip stack provides some possibility to detect buffer
> >> overflows ?) Is it possible that some mobicents extension of the sip-stack
>
> >> ** **
>
> >> Thanks in advance for any feedback.****
>
> >> ** **
>
> >> Best regards,****
>
> >> Stephan Klein – Kapsch CarrierCom****
>
> >> ---****
>
> >> ** **
>
> >> P.S: A special hello to our colleagues from Lithuania ;)****
Klein Stephan
May 3, 2012, 10:26:54 AM5/3/12
to mobicent...@googlegroups.com
Find attached the test results, statistics from SIPP as well as a screenshot from jvisualvm which shows CPU and Memory Usage of two similar subsequent tests.
> >> I'm using Mobicents Jain Slee 2.6.0 Final (no cluster - default mode) and
> >> Stephan Klein - Kapsch CarrierCom****
> >> sip-11-ra provided with this package and the sip-b2bua example from
> >> mobicents as application.****
>
> >> JVM is tuned according to our experience with other Jain SLEE platforms.*
> >> ***
>
> >> ** **
>
> >> Now here comes the issue:****
>
> >> Until 600 CPS the SIPP-Traffic is stable.****
>
> >> At 600 CPS the INVITE retransmission counter starts to increase steadily.
> >> At a rather constant rate INVITEs are simply not answered (with trying) and
> >> SIPP has to retransmit them - this happens steadily (not only during full
> >> mobicents as application.****
>
> >> JVM is tuned according to our experience with other Jain SLEE platforms.*
> >> ***
>
> >> ** **
>
> >> Now here comes the issue:****
>
> >> Until 600 CPS the SIPP-Traffic is stable.****
>
> >> At 600 CPS the INVITE retransmission counter starts to increase steadily.
> >> At a rather constant rate INVITEs are simply not answered (with trying) and
> >> GC).****
>
> >> Overall at 600 CPS this happens for ~1 % of calls whereas 99% of calls
> >> are answered within 100ms - so there are still enough resources, also CPU
>
> >> Overall at 600 CPS this happens for ~1 % of calls whereas 99% of calls
> >> is rather low at this rate.****
>
> >> ** **
>
> >> A tshark interface trace shows that those INVITEs are received the Jain
> >> SLEE Server's IP Interface (so its not a connection or sipp issue)****
>
> >> When enabling the Jain SIP Message Logging
> >> (gov.nist.javax.sip.LOG_MESSAGE_CONTENT) it can be seen that those INVITEs
> >> are NOT shown in the log - only the retransmissions after are shown and
>
> >> ** **
>
> >> A tshark interface trace shows that those INVITEs are received the Jain
> >> SLEE Server's IP Interface (so its not a connection or sipp issue)****
>
> >> When enabling the Jain SIP Message Logging
> >> (gov.nist.javax.sip.LOG_MESSAGE_CONTENT) it can be seen that those INVITEs
Andrew Miller
May 3, 2012, 10:32:20 AM5/3/12
to mobicent...@googlegroups.com
On the 100 Trying issue; from a SIP protocol view-point a 100 is only
required if you are not going to respond within 200 ms to the INVITE.
From experience with other SIP stacks and with Mobicents SIP Servlets,
we have found that it gives more consistent performance to always send a
100 Trying. This also has the effect that you don't have to start a
timer to send a 100 later - so it makes the UAs job easier.
Andy Miller
Crocodile RCS Ltd.
>>>> I�m new to Mobicents Jain SLEE and currently evaluating how it performs
required if you are not going to respond within 200 ms to the INVITE.
From experience with other SIP stacks and with Mobicents SIP Servlets,
we have found that it gives more consistent performance to always send a
100 Trying. This also has the effect that you don't have to start a
timer to send a 100 later - so it makes the UAs job easier.
Andy Miller
Crocodile RCS Ltd.
>>>> as a HA AS for SIP based Applications.****
>>>> Therefore we are executing High Load tests to find the maximum
>>>> performance limits of the platform on a carrier grade hardware:****
>>>> ** **
>>>> HP DL380 G7****
>>>> 24 Intel Xeon CPU @ 3.33 GHz****
>>>> 48 GB RAM****
>>>> ** **
>>>> I�m using Mobicents Jain Slee 2.6.0 Final (no cluster � default mode) and
>>>> Therefore we are executing High Load tests to find the maximum
>>>> performance limits of the platform on a carrier grade hardware:****
>>>> ** **
>>>> HP DL380 G7****
>>>> 24 Intel Xeon CPU @ 3.33 GHz****
>>>> 48 GB RAM****
>>>> ** **
>>>> sip-11-ra provided with this package and the sip-b2bua example from
>>>> mobicents as application.****
>>>> JVM is tuned according to our experience with other Jain SLEE platforms.*
>>>> ***
>>>> ** **
>>>> Now here comes the issue:****
>>>> Until 600 CPS the SIPP-Traffic is stable.****
>>>> At 600 CPS the INVITE retransmission counter starts to increase steadily.
>>>> At a rather constant rate INVITEs are simply not answered (with trying) and
>>>> SIPP has to retransmit them � this happens steadily (not only during full
>>>> mobicents as application.****
>>>> JVM is tuned according to our experience with other Jain SLEE platforms.*
>>>> ***
>>>> ** **
>>>> Now here comes the issue:****
>>>> Until 600 CPS the SIPP-Traffic is stable.****
>>>> At 600 CPS the INVITE retransmission counter starts to increase steadily.
>>>> At a rather constant rate INVITEs are simply not answered (with trying) and
>>>> GC).****
>>>> Overall at 600 CPS this happens for ~1 % of calls whereas 99% of calls
>>>> are answered within 100ms � so there are still enough resources, also CPU
>>>> Overall at 600 CPS this happens for ~1 % of calls whereas 99% of calls
>>>> is rather low at this rate.****
>>>> ** **
>>>> A tshark interface trace shows that those INVITEs are received the Jain
>>>> SLEE Server�s IP Interface (so its not a connection or sipp issue)****
>>>> ** **
>>>> A tshark interface trace shows that those INVITEs are received the Jain
>>>> When enabling the Jain SIP Message Logging
>>>> (gov.nist.javax.sip.LOG_MESSAGE_CONTENT) it can be seen that those INVITEs
>>>> are NOT shown in the log � only the retransmissions after are shown and
>>>> (gov.nist.javax.sip.LOG_MESSAGE_CONTENT) it can be seen that those INVITEs
>>>> processed fast.****
>>>> ** **
>>>> So I assume that the SIP stack somehow swallows or ignores those INVITEs
>>>> (even before logging them)****
>>>> ** **
>>>> Also interesting is that when starting the traffic already with 600 CPS
>>>> it takes some time (~20000 messages) until the retransmissions start. ***
>>>> *
>>>> This may indicate that some queue/buffer exceeds its size.****
>>>> ** **
>>>> I already tried to tune some of the following preconfigured
>>>> sipra.properties:****
>>>> THREAD_POOL_SIZE from 8 to 64****
>>>> RECEIVE_UDP_BUFFER_SIZE and SEND_UDP_BUFFER_SIZE to the double and triple
>>>> value.****
>>>> MAX_SERVER_TRANSACTIONS and MAX_CLIENT_TRANSACTIONS to the double value**
>>>> **
>>>> ** **
>>>> None of the changes prevented or delayed the retransmissions.****
>>>> ** **
>>>> So I�m asking if someone experienced the same or similar issue and knows
>>>> ** **
>>>> So I assume that the SIP stack somehow swallows or ignores those INVITEs
>>>> (even before logging them)****
>>>> ** **
>>>> Also interesting is that when starting the traffic already with 600 CPS
>>>> it takes some time (~20000 messages) until the retransmissions start. ***
>>>> *
>>>> This may indicate that some queue/buffer exceeds its size.****
>>>> ** **
>>>> I already tried to tune some of the following preconfigured
>>>> sipra.properties:****
>>>> THREAD_POOL_SIZE from 8 to 64****
>>>> RECEIVE_UDP_BUFFER_SIZE and SEND_UDP_BUFFER_SIZE to the double and triple
>>>> value.****
>>>> MAX_SERVER_TRANSACTIONS and MAX_CLIENT_TRANSACTIONS to the double value**
>>>> **
>>>> ** **
>>>> None of the changes prevented or delayed the retransmissions.****
>>>> ** **
>>>> how to deal with it ? Are there further sip-stack properties which are
>>>> worth tuning ? Maybe someone knows how and what areas are worth
>>>> investigating ? (maybe sip stack provides some possibility to detect buffer
>>>> overflows ?) Is it possible that some mobicents extension of the sip-stack
>>>> interferes here ?****
>>>> ** **
>>>> Thanks in advance for any feedback.****
>>>> ** **
>>>> Best regards,****
>>>> Stephan Klein � Kapsch CarrierCom****
>>>> worth tuning ? Maybe someone knows how and what areas are worth
>>>> investigating ? (maybe sip stack provides some possibility to detect buffer
>>>> overflows ?) Is it possible that some mobicents extension of the sip-stack
>>>> interferes here ?****
>>>> ** **
>>>> Thanks in advance for any feedback.****
>>>> ** **
>>>> Best regards,****
Eduardo Martins
May 3, 2012, 11:02:24 AM5/3/12
to mobicent...@googlegroups.com
Interesting, maybe at 1100 cps you are hitting some component perf limitation, perhaps JBoss Cache, since locally I have pushed pure jain sip stack to around 1500 cps and I'm talking about a macbook pro with intel i5 and 8GB of memory. Anyway can you try my settings but with bigger heap, for instance:
export JAVA_OPTS="-Xmn256m -XX:CMSInitiatingOccupancyFraction=80 -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -XX:MaxPermSize=256M -Xmx24000M -Xms512M -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true"
Also, if it doesn't help perhaps it's time to increase a bit the jsip buffer sizes and/or the jsip stack and mobicents event router threads, and see what happens.
Eduardo Martins
May 3, 2012, 11:03:53 AM5/3/12
to mobicent...@googlegroups.com
The JAIN SIP stack automatically sends the 100 if needed, but perhaps Andrew is right, and if the app sends it the stack performs better, something to try definitely.
I’m new to Mobicents Jain SLEE and currently evaluating how it performs
as a HA AS for SIP based Applications.****
Therefore we are executing High Load tests to find the maximum
performance limits of the platform on a carrier grade hardware:****
** **
HP DL380 G7****
24 Intel Xeon CPU @ 3.33 GHz****
48 GB RAM****
** **
I’m using Mobicents Jain Slee 2.6.0 Final (no cluster – default mode) and
sip-11-ra provided with this package and the sip-b2bua example from
mobicents as application.****
JVM is tuned according to our experience with other Jain SLEE platforms.*
***
** **
Now here comes the issue:****
Until 600 CPS the SIPP-Traffic is stable.****
At 600 CPS the INVITE retransmission counter starts to increase steadily.
At a rather constant rate INVITEs are simply not answered (with trying) and
SIPP has to retransmit them – this happens steadily (not only during full
GC).****
Overall at 600 CPS this happens for ~1 % of calls whereas 99% of calls
are answered within 100ms – so there are still enough resources, also CPU
is rather low at this rate.****
** **
A tshark interface trace shows that those INVITEs are received the Jain
SLEE Server’s IP Interface (so its not a connection or sipp issue)****
When enabling the Jain SIP Message Logging
(gov.nist.javax.sip.LOG_MESSAGE_CONTENT) it can be seen that those INVITEs
are NOT shown in the log – only the retransmissions after are shown and
processed fast.****
** **
So I assume that the SIP stack somehow swallows or ignores those INVITEs
(even before logging them)****
** **
Also interesting is that when starting the traffic already with 600 CPS
it takes some time (~20000 messages) until the retransmissions start. ***
*
This may indicate that some queue/buffer exceeds its size.****
** **
I already tried to tune some of the following preconfigured
sipra.properties:****
THREAD_POOL_SIZE from 8 to 64****
RECEIVE_UDP_BUFFER_SIZE and SEND_UDP_BUFFER_SIZE to the double and triple
value.****
MAX_SERVER_TRANSACTIONS and MAX_CLIENT_TRANSACTIONS to the double value**
**
** **
None of the changes prevented or delayed the retransmissions.****
** **
So I’m asking if someone experienced the same or similar issue and knows
how to deal with it ? Are there further sip-stack properties which are
worth tuning ? Maybe someone knows how and what areas are worth
investigating ? (maybe sip stack provides some possibility to detect buffer
overflows ?) Is it possible that some mobicents extension of the sip-stack
interferes here ?****
** **
Thanks in advance for any feedback.****
** **
Best regards,****
Stephan Klein – Kapsch CarrierCom****
stepha...@kapsch.net
May 3, 2012, 12:18:16 PM5/3/12
to mobicent...@googlegroups.com
Great! - I used your JVM Parameters so that mine now look like this:
-Xms6G -Xmx6G -Xmn256m -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000
-Xms6G -Xmx6G -Xmn256m -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000
Now the limit is at 1600 CPS and i get a very nice SIPP Result screen here:
Call-rate(length) Port Total-time Total-calls Remote-host
1600.0(0 ms)/1.000s 5050 161.72 s 160000 148.198.194.164:5060(UDP)
Call limit reached (-m 160000), 0.000 s period 0 ms scheduler resolution
0 calls (limit 10000000) Peak was 96135 calls, after 100 s
0 Running, 52804 Paused, 0 Woken up
115 dead call msg (discarded) 0 out-of-call msg (discarded)
1 open sockets
Messages Retrans Timeout Unexpected-Msg
INVITE ----------> B-RTD1 160000 11 0
100 <---------- 0 0 0 0
180 <---------- E-RTD1 159071 0 0 0
200 <---------- E-RTD1 160000 5 0 0
Pause [ 150ms] 160000 0
ACK ----------> 160000 5
BYE <---------- 160000 0 0 0
200 ----------> 160000 0
So it really seems my JVM was overtuned and it works a lot better with fewer parameters ;)
Still only 20 % CPU is used - but the resource-usage is quite one-sided in sip-uas scenario - i can imagine that a real life application using other RAs, DB interactions etc. may utilize the CPU better.
However we will continue to tweak the JVM - maybe there is even more potential, will let you know about the outcome.
ty
-- Stephan
Eduardo Martins
May 3, 2012, 12:40:54 PM5/3/12
to mobicent...@googlegroups.com
Once you get a stable limit do a test longer than 1h to ensure it handles such load GC fine, and let us know what's the heap memory usage in the end, it should give us an idea of how hungry the sip stack really is, and if that is truly a prob.
Then, like I said before, try to increase the buffers, this will waste more mem, but may payoff.
If CPU load is low, messing with threads probably won't help.
Finally, that doubt about the 100 trying, if possible can you check it?
Eduardo Martins
May 3, 2012, 12:45:35 PM5/3/12
to mobicent...@googlegroups.com
Btw, b2bua will give you a more complex app picture, it uses some high level code that uses more app state then needed, but mixing the sip uas with jdbc ra and/or diameter should be more interesting :-)
Klein Stephan
May 7, 2012, 11:00:04 AM5/7/12
to mobicent...@googlegroups.com
Hi,
after further tests with sip-uas here are new findings:
- Increased SIP-RA Buffer sizes to double, fourfold, tenfold size
-> No Improvements
- Upgraded to Java 7 U4 (was java 6 before)
-> No Improvements
- Tuned Eduardos JVM Settings: Changed -Xmn256m to -Xmn128m:
-> Improvement from stable 1600 to 1700 CPS
-> Decreasing the Max New Limit of the JVM results in shorter GC Pauses (But more often), which has a positive effect on retransmissions
- Tried New Java 7 U4 Garbage Collector One (GC1) with various options - see http://www.oracle.com/technetwork/java/javase/tech/g1-intro-jsp-135488.html
-> Much worse (limit was 1000 CPS)
-> G1 seems not to achieve very low GC Pauses at high load (target would be <5ms as we have with CMS to avoid retransmissions)
- Updated SIP-UAS SBB with logic for sending 100 TRYING always:
-> Improvement from stable 1700 CPS to 1900 CPS
-> Ran a 1h test with 1900 CPS and it was stable (CPU Load ~30 %)
-> It seems that 100 TRYING can be generated really fast by the stack, and this also has a positive effect on retransmissions
Regarding the memory-greedyness of the sip-stack, find screenshot of the the memory profile for the 1h test attached.
I also calculated from GC-Logging which i enabled (this had no noticeable perf. impact) that the GC collected a total of 3145738,7 MB equal to 873MB per second, or 460K per Call
-> not sure if this can be interpreted as 'greedy' or not ;)
We a rather satisfied with this performance so I will not continue tuning for now, this is a process where you can spend weeks, as JVM tuning options are countless.
However I will let you know if we achieve any further improvements in the future.
Attachments:
- 1900cps_1h_sipp_result.txt: Details about Latency, Nr of Calls etc for the 1h 1900 CPS test
- 1900cps_1h_memory&cpu.png: Screenshot of jvisualvm, shows CPU and Memory Profile
-- Stephan
The information contained in this e-mail message is privileged and
Eduardo Martins
May 7, 2012, 3:56:01 PM5/7/12
to mobicent...@googlegroups.com
Thank you for all your work, I will incorporate the sending of the 100s, and will see the best option to share your findings wrt JVM tunning.
-- Eduardo
Reply all
Reply to author
Forward
0 new messages