Using Kamailio dispatcher for load balancing and failover routing

1,790 views
Skip to first unread message

Sean DiSanti

unread,
Apr 9, 2019, 1:42:38 PM4/9/19
to UniMRCP
I am looking into using Kamailio for load balancing / redundancy across multiple unimrcp server services, but haven't found any specific examples of anyone doing this before, so would appreciate anyone's thoughts or direction towards that end.

The setup I am trying to get working would be to have our mrcp profiles in the media boxes point at a kamailio service, to make use of its dispatcher module to route traffic to unimrcp services running on different boxes. I was think via broadcast / first responder implementation. Set 1 in the dispatch list would be unimrcp services running at the same site, set 2 (only attempted after timeout or 5xx responses from first set) would do the same for the opposite site. Going through the documentation, it looks like it should be possible making use primarily of the dispatcher module https://www.kamailio.org/docs/modules/5.1.x/modules/dispatcher.html#dispatcher.p.db_url and transaction management module https://www.kamailio.org/docs/modules/5.1.x/modules/tm.html

Doug Rylaarsdam

unread,
Apr 10, 2019, 3:53:55 PM4/10/19
to UniMRCP
I've done some initial work with Kamailio with similar goals.  So far I've had good results, routing requests between UniMRCP servers in a dev environment, and doing some basic tests of the dispatcher module round-robin algorithm.   Based on my limited experience your plans sound achievable. It's not much insight, but hopefully the reference helps.

Doug 

Sean DiSanti

unread,
Apr 10, 2019, 7:03:35 PM4/10/19
to uni...@googlegroups.com
Thanks for the feedback!

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Vipul

unread,
Jan 8, 2020, 3:06:28 AM1/8/20
to uni...@googlegroups.com
Hi Sean,

This looks like something which I am also trying do achieve right now.

May I know if you were able to load balance across multiple UniMRCP servers using Kamailio? I also couldn't find any specific examples of anyone doing this so, I would like to know your experience on this.

Thanks.
Vipul

Mickael Hubert

unread,
Jan 8, 2020, 3:37:04 AM1/8/20
to uni...@googlegroups.com
Hi all,
I use opensips to loadbalance between 2 unimrcp servers. It works like a charm.
you can find in attachment my infrastructure.
feel free to ask me your questions ;)

have a good day

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CAMPtzKhCPkNx%2Ba4%3DkG-yK-kaVOY-0c8_pHkhHSyeysZZsS5cqA%40mail.gmail.com.
MRCPv2 prod (forum unimrcp version).png

Sean DiSanti

unread,
Jan 8, 2020, 1:27:01 PM1/8/20
to uni...@googlegroups.com
The short answer is no, we didn't go with Kamailio. We ended up creating our own system for load balancing and license distribution.

The long answer:
Before looking at UniMRCP, we already had a fairly large distributed system comprised of 50+ freeswitch boxes housed in multiple locations, handling hundreds of thousands of calls per day. We were already working on learning one new service with UniMRCP, and didn't want to add more points of failure by adding additional unfamiliar middleware. So we used tools we already had in place to cobble together a pretty reliable redundant system. I think I've diagrammed this out here before, but here's a quick overview of what we came up with.

When a UniMRCP box starts up, it requests ports from the license server at startup, and then a custom service on the box registers with a local Postgres server and reports the ports available as reported by /opt/unimrcp/var/stats/umsgsr-usage.status. Rows are created in a port usage table for each license available to the instance. Our freeswitch boxes use Lua scripts to call a routine on the Postgres box which returns the name of the local unimrcp instance with the least active ports, and stores the m_uuid in the usage table. Each freeswitch box has mrcp profiles for each known UniMRCP instance and a couple of extras in case we need to scale without notice. If Postgres responds that no licenses are available from any instance, the Lua script sends the call to a TT driven version of the call flow. When the call is over, another routine is called from our onhangup script which removes the m_uuid from the usage table. On the UniMRCP instance, I have a heartbeat service that checks in with Postgres every 15 seconds as long as the unimrcpserver process is alive. That way if a node fails, or the service crashes on an instance, we don't try to send it utterances. We also have monitors set up with datadog to alert if a heartbeat or unimrcpserver process goes away, and we have custom metrics from Postgres so we can see port usage per instance on datadog dashboards.

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CA%2BfRVSW16Ekfg%3DH3RPuvjU5--QcOao-WESmPiQ4zgYFP_YuPJw%40mail.gmail.com.

David Villasmil

unread,
Jan 8, 2020, 1:37:40 PM1/8/20
to uni...@googlegroups.com
I just started working on trying to use kamailio to load-balance to a unimrcp box (just one for now).

Here are the issues I’ve found so far:

Directing to kamailio instead of the unimrcp server is fine.
Once the INVITE gets to kamailio I just forward it to the gsr box, but the box simply ignores completely the request.
It may have to do with the fact that kamailio is adding a couple VIA headers. But I’m not sure, everything else looks exactly as if it’s coming from the fs itself.



--
Regards,

David Villasmil

Arsen Chaloyan

unread,
Jan 9, 2020, 8:02:17 PM1/9/20
to UniMRCP
Hi David,

Are you sure the request is sent to the intended SIP port 8060. If you can provide logs and a network capture, I can take a look.

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CAFGRPVruhb3AcMnVzagorpPWwC8S1p-051cFkwix7k6AVn-Gag%40mail.gmail.com.


--
Arsen Chaloyan
Author of UniMRCP
http://www.unimrcp.org

David Villasmil

unread,
Jan 10, 2020, 8:54:38 AM1/10/20
to uni...@googlegroups.com
Hello Arsen,

Positive, i'll get some traces for you.

Regards,

David Villasmil

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CAC5UscLJtPJ_SyOaqmrdzV1ZcWVSSH8wbJLVPG%3DRYPZkH%2B%2B9mQ%40mail.gmail.com.

David Villasmil

unread,
Jan 10, 2020, 9:11:04 AM1/10/20
to uni...@googlegroups.com
Here's unimrcp server's listening:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 1.2.3.55:8060      0.0.0.0:*               LISTEN      15250/./unimrcpserv
tcp        0      0 1.2.3.55:1544      0.0.0.0:*               LISTEN      15250/./unimrcpserv
udp        0      0 1.2.3.55:8060      0.0.0.0:*                           15250/./unimrcpserv

When sending a call to the server:

 84 97.346019763 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
 85 97.844998428 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
 86 98.319247264 1.2.3.54 → 1.2.3.55 SIP/SDP 1122 Request: INVITE sip:1.2.3.55:8060 |
 87 98.782695061 1.2.3.54 → 1.2.3.55 SIP/SDP 1122 Request: INVITE sip:1.2.3.55:8060 |
 88 98.844948746 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
 89 98.845053333 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
 90 99.782470763 1.2.3.54 → 1.2.3.55 SIP/SDP 1122 Request: INVITE sip:1.2.3.55:8060 |
 91 100.844948117 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
 92 101.782521162 1.2.3.54 → 1.2.3.55 SIP/SDP 1122 Request: INVITE sip:1.2.3.55:8060 |
 93 102.844934572 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
 94 104.844942611 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
 95 105.782451594 1.2.3.54 → 1.2.3.55 SIP/SDP 1122 Request: INVITE sip:1.2.3.55:8060 |
 96 106.844959510 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
 97 107.345288307 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
 98 107.845968748 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
 99 108.844951135 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
100 108.845074532 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
101 109.782474158 1.2.3.54 → 1.2.3.55 SIP/SDP 1122 Request: INVITE sip:1.2.3.55:8060 |
102 110.844926886 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
103 110.845044077 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |
104 112.844953632 1.2.3.54 → 1.2.3.55 SIP 388 Request: OPTIONS sip:1.2.3.55:8060 |

Screenshot 2020-01-10 at 14.03.44.png

Unimrcp server:

2020-01-10 14:07:25:453216 [NOTICE] Remove Session <14f351382fc041d9>
2020-01-10 14:07:25:453238 [INFO]   Session Terminated 0x7fbef00047c8 <14f351382fc041d9>
2020-01-10 14:07:25:453257 [NOTICE] Destroy Session <14f351382fc041d9>
2020-01-10 14:07:25:453290 [DEBUG]  Wait for Messages [MRCP Server]

>
>
>
>
>
>

Nothing happens... (i have 2 fs, one configured to send to kamailio, and the other to send straight to the unimrcp. The direct one works, the one via kamailio doesn't)

The INVITE is:

2020/01/10 15:07:49.347814 1.2.3.54:5060 -> 1.2.3.55:8060
INVITE sip:1.2.3.55:8060 SIP/2.0
Record-Route: <sip:1.2.3.54;lr>
Via: SIP/2.0/UDP 1.2.3.54;branch=z9hG4bK4906.98be99d454c0ce66c9c9eee347ed0ab3.0
Via: SIP/2.0/UDP 1.2.3.54:5060;branch=z9hG4bKsr-om1lS8eWyJ96btTBy8ZWypy0SRcFyQfmGRd.GRT0yCuFqYG1EHq1qCm8GFf.y8PWype.SRl2nhbeMYbi4RuFZX9F4Cm.GRT0yCukZstWI2B9NR1db8bkQho0o6LXo8UmbRq3ooV*
Max-Forwards: 69
From: <sip:1.2.3.46:16090>;tag=UN9mDKByjNcZN
To: <sip:1.2.3.55:8060>
Call-ID: 6cdb54ed-ae55-1238-c287-ca5c80b01704
CSeq: 14778842 INVITE
User-Agent: FreeSWITCH
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, PRACK, MESSAGE, SUBSCRIBE, NOTIFY, REFER, UPDATE
Supported: timer, 100rel
Content-Type: application/sdp
Content-Disposition: session
Content-Length: 331

v=0
o=FreeSWITCH 1604687223182261789 5381451631169550039 IN IP4 1.2.3.46
s=-
c=IN IP4 1.2.3.46
t=0 0
m=application 9 TCP/MRCPv2 1
a=setup:active
a=connection:new
a=resource:speechrecog
a=cmid:1
m=audio 14008 RTP/AVP 0 8 96
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:96 L16/8000
a=sendonly
a=mid:1


Thanks!

David Villasmil

Mickael Hubert

unread,
Jan 10, 2020, 9:39:24 AM1/10/20
to uni...@googlegroups.com
Hi,
I have via header too from opensips to unimrcp and it works.
do you have any firewall on your unimrcp server ?
can you put your unimrcp configuration file (unimrcpserver.xml) ?


Doug Rylaarsdam

unread,
Jan 10, 2020, 10:20:43 AM1/10/20
to UniMRCP
Hi David,

Are you seeing any SIP events in the unimrcpserver log (for the OPTION or INVITE requests)? e.g.

[INFO]   Receive SIP Event [nua_i_options] Status 100 Trying [SIP-Agent-1]


Also, the sofia-sip logging may be helpful.

Doug


On Friday, January 10, 2020 at 8:39:24 AM UTC-6, Mickael Hubert wrote:
Hi,
I have via header too from opensips to unimrcp and it works.
do you have any firewall on your unimrcp server ?
can you put your unimrcp configuration file (unimrcpserver.xml) ?



Thanks!

David Villasmil


On Fri, Jan 10, 2020 at 1:54 PM David Villasmil <david.vil...@gmail.com> wrote:
Hello Arsen,

Positive, i'll get some traces for you.

Regards,

David Villasmil


Thanks for the feedback!

To unsubscribe from this group and stop receiving emails from it, send an email to uni...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to uni...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to uni...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to uni...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to uni...@googlegroups.com.
--
Regards,

David Villasmil

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to uni...@googlegroups.com.


--
Arsen Chaloyan
Author of UniMRCP
http://www.unimrcp.org

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to uni...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to uni...@googlegroups.com.

David Villasmil

unread,
Jan 11, 2020, 8:36:58 AM1/11/20
to uni...@googlegroups.com
Hello,

Well it seems i put in some iptables rules that were dropping the INVITE :(
All is working perfectly now.

Thanks!


Regards,

David Villasmil

To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/79da9b3a-ea42-453c-b1ae-b9aba4630bbf%40googlegroups.com.

Michael Levy

unread,
Mar 16, 2020, 6:47:54 PM3/16/20
to UniMRCP
Has anyone written up some information on how to use kamailio for load balancing MRCP?
We seem to be outgrowing our present load balancing solution. I'd really appreciate any "getting started" guidance.

Thanks

Mickael Hubert

unread,
Mar 17, 2020, 2:22:24 AM3/17/20
to uni...@googlegroups.com
Hi
What kind of informations do you need ?
You Can use LB module from kamailio. The sip has to passthrough kamailio. But rtp and mrcp have to arrive directly to your unimrcp servers (like the schema in previous email)


--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/162fe2a4-a84f-4f8d-83cd-d4e4096ada9d%40googlegroups.com.

Michael Levy

unread,
Mar 17, 2020, 11:07:21 AM3/17/20
to UniMRCP
I appreciate your feedback. It looks like we are moving forward with Netscaler. We will review this thread and your suggestions further.

I started a new post for a discussion on using Netscaler in case anyone has advice

David Villasmil

unread,
Mar 17, 2020, 11:10:09 AM3/17/20
to uni...@googlegroups.com
Decided against kamailio? It works PERFECTLY, buddy. You should try it.
I can give you a hand, what do you need to know?

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.
--
Regards,

David Villasmil

Michael Levy

unread,
Mar 17, 2020, 11:40:48 AM3/17/20
to UniMRCP
It wasn't my decision.But thanks for your offer. We have Netscaler in use already. Nobody has the appetite to introduce a new technology right now. I'll let you know how it goes. 

Sean DiSanti

unread,
Mar 18, 2020, 5:14:03 PM3/18/20
to uni...@googlegroups.com
We also went another route (described in an earlier message) but this comes up often enough that if you felt like writing up documentation or a tutorial, it would probably be a welcome addition to both projects.

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CAFGRPVpGnbxwe_QoV8s%3DC%2BVeai2K0Sazu%2BDAN0Ep-EACF0Gxow%40mail.gmail.com.

Michael Levy

unread,
Mar 19, 2020, 11:23:30 AM3/19/20
to UniMRCP
I think I posted the wrong link. Hopefully, this is better: Citrix Netscaler for MRCP Load balancing? - 

Michael Levy

unread,
Mar 19, 2020, 11:48:24 AM3/19/20
to UniMRCP
David and Michael and others on this thread have offered advice. So, I'll ask here.

I'm much more on the development side of our MRCP/ASR project, not the networking/SIP side. But, this task is falling on me. We have an existing deployment of NetScaler. We want to leverage this for MRCPv2 load balancing between voice browsers and unimrcp servers. MRCPv2 is SIP with associated RTP and MRCP application channels, so we can start by looking at Netscaler's SIP loadbalancing - https://docs.citrix.com/en-us/netscaler/12/load-balancing/load-balancing-common-protocols/lb-sip-servers.html


This is what I think we need to do:

Load balanced service: SIP-TCP 

load balanced service: SIP-TCP (this is not listed in https://docs.citrix.com/en-us/netscaler/12/load-balancing/load-balancing-setup.html, only SIP-UDP is listed, but SIP-TCP is mentioned elsewhere in the docs).


Persistence: SIP Call ID

We need SIP Call ID persistence to properly maintain SIP call sessions. We need this because our voice browsers are configured to persist MRCP connections for the life of a call, no matter how many recognitions are performed. See https://docs.citrix.com/en-us/netscaler/12/load-balancing/load-balancing-persistence/callid-persistence.html


Monitoring: SIP-TCP with sipmethod: OPTION and respcode = 200

The load balancer uses the configured monitor to check the health of the ASR server. We will select SIP OPTION (discovery) requests for monitoring and require 200 responses. This should work well with Unimrcp's offline capability (not sure if this is in core, but Arsen accepted a code change from me, so we have SIP Options discovery offline responses - https://github.com/unispeech/unimrcp/issues/242). See https://docs.citrix.com/en-us/netscaler/12/load-balancing/load-balancing-builtin-monitors/monitor-sip-services.html


Use Source IP Address: YES

I'm not sure if this setting is correct for SIP in MRCPv2. I think we'll have to experiment. See https://docs.citrix.com/en-us/netscaler/12/load-balancing/load-balancing-advanced-settings/usip-of-client.html




Am I on the right track? Is this all it needs?

David Villasmil

unread,
Mar 19, 2020, 12:02:26 PM3/19/20
to uni...@googlegroups.com
Hello,

If you’re not using kamailio now for other services, it should be fairly straightforward to implement.

Install kamailio on a Debian Box as per 

This will install a kamailio and configure a database connection to a mariaDB/MySQL. The dB part is not strictly necessary, but nice to have.

Once that’s done, you can use the cfg in 
https://kamailio.org/docs/modules/4.3.x/modules/dispatcher.html that one uses a db as well (where the unimrcp servers IP can be stored  (using kamctl dispatcher” cli utility) for dispatcher to load balance), set the DBURL properly as per your configuration.

Be mindful of setting the right ports in the dispatcher config.



--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.

Mickael Hubert

unread,
Mar 19, 2020, 12:12:49 PM3/19/20
to uni...@googlegroups.com
Hi,
Yes, Kamailio can do all these things:

Load balanced service: SIP-TCP

Persistence: SIP Call ID

OPTIONS


++



To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CAFGRPVrY4e536xi%2BiKN8MXWwtxRLAM-QPhQXKJyGQE_taG0Xdg%40mail.gmail.com.

eduhe...@gmail.com

unread,
Feb 28, 2023, 7:17:59 AM2/28/23
to UniMRCP
Hi there

We have a single UniMRCP server and a backup server, and we are using kamilio to implement failover (but we are no kamailio experts at all...). We have implemented redundancy, using priority algorithm ,so backup server has a lower priority that primary server, and only receives traffic if primary is down. I don't know it that's the better way to go, but it works perfectly

Now we are planning to add a second pair of UniMRCP servers, primary-backup, but we don't know how to implement redundancy. We can't use priority algorithm anymore, because if one primary falls, all the traffic would go to the other primary server...

Has anybody faced or implemented an architecture like that? One kamailio, two primary unimrcp servers, two backup unimrcp servers...

Thanks in advance


Arsen Chaloyan

unread,
Mar 10, 2023, 3:56:38 PM3/10/23
to uni...@googlegroups.com
Hi Eduardo,

You may need to use your current architecture for load balancing and fail-over, and implement redundancy for the entire solution, by replicating all the components you have in the primary site, including IVR and UniMRCP Server instances, in the backup site.

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/2f930308-e544-4f2c-bdf7-7ef34c6eb50dn%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages