Getting 404 or 403 error for SIP INVITE request.

626 views

Skip to first unread message

KazooTester

unread,

Jan 9, 2015, 8:38:32 AM1/9/15

to 2600h...@googlegroups.com

Hello All,

Accidentally I posted this question in users group. Anyway here is the issue.

We have setup kazoo farm across 9 virtual machines.(3 Kazoo + 3 CouchDB + 3 FreeSwitch). All virtual machines are configured with 700MB RAM, 40GB HDD and i3 processor. Everything working fine with few desk phones and mobile soft clients. Problem started when we started testing it with Sipp and Asterisk.

So Here is our setup:

SIPP Tool <------------>KAZOO-FARM<-------------------->Asterisk.

Note: Asterisk is being used as an auto-answering client.

Total 100 SIPP users and 100 Asterisk users are configured with their REGISTER and CALL cycle.

All SIPP users will be making calls, Asterisk users will be answering it, playing a media file (For few RTP packets), let it connected for (120 seconds) and SIPP users finally terminate the call.

Issue:

Starting for 2 calls/second to 5 calls/second everything works fine. But the moment we increase that to say 7calls/second abruptly it starts giving 403 and 404 error for few users. All those time CPU usage was not high neither memory. Do you see any limitation in kazoo itself for processing more than 5 calls/seconds?

Logs:

For SIP ERROR 403

FreeSwitch:

[WARNING] sofia_reg.c:2819 Can't find user [98989...@testfarm1.com] from 172.18.100.212
You must define a domain called 'testfarm1.com' in your directory and add a user with the id="9898900096" attribute
and you must configure your device to use the proper domain in it's authentication credentials.
2014-12-31 17:23:58.796386 [WARNING] sofia_reg.c:1636 SIP auth failure (INVITE) on sofia profile 'sipinterface_1' for [78789...@testfarm1.com] from ip 172.18.100.212
For SIP ERROR 404 NO_ROUTE_DESTINATION

2600hzLogs

Jan 1 14:07:33 farm9 2600hz[3624]: |0000000000|Undefined:Undefined (<0.493.0>) Supervisor whistle_couch_sup had child couch_compactor_fsm started with couch_compactor_fsm:start_link() at <0.12075.1> exit with reason no match of right hand value {error,not_found} in couch_compactor_fsm:get_nodes/0 line 1188 in context child_terminated
Jan 1 14:08:33 farm9 2600hz[3624]: |0000000000|Undefined:Undefined (<0.13519.1>) gen_fsm couch_compactor_fsm in state ready terminated with reason: no match of right hand value {error,not_found} in couch_compactor_fsm:get_nodes/0 line 1188
Jan 1 14:08:33 farm9 2600hz[3624]: |0000000000|Undefined:Undefined (<0.13519.1>) CRASH REPORT Process couch_compactor_fsm with 0 neighbours exited with reason: no match of right hand value {error,not_found} in couch_compactor_fsm:get_nodes/0 line 1188 in gen_fsm:terminate/7 line 611

[NOTICE] kazoo_fetch_agent.c:286 Request for dialplan XML (ba82299e-9192-11e4-9d9e-17bf5339065d) timed-out after 3100ms
2015-01-01 14:16:53.916527 [WARNING] mod_dialplan_xml.c:588 Context context_2 not found
2015-01-01 14:16:53.916527 [INFO] switch_core_state_machine.c:239 No Route, Aborting

Thanks

James Aimonetti

unread,

Jan 9, 2015, 11:56:29 AM1/9/15

to 2600h...@googlegroups.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

You'd have to dig into the calls that were timing out to get an idea
of where the slowdown is occurring.

You haven't described the network topology, what the underlying
hardware is, what VM technology you're using, etc. Load-testing is
very tricky to get right, to make sure you're testing what you think
you are.

Le 2015-01-09 05:38, KazooTester a écrit :
> Hello All, Accidentally I posted this question in users group.
> Anyway here is the issue.
>
> We have setup kazoo farm across 9 virtual machines.(3 Kazoo + 3
> CouchDB + 3 FreeSwitch). All virtual machines are configured with
> 700MB RAM, 40GB HDD and i3 processor. Everything working fine with
> few desk phones and mobile soft clients. Problem started when we
> started testing it with Sipp and Asterisk.
>

> *So Here is our setup:*

>
> SIPP Tool <------------>KAZOO-FARM<-------------------->Asterisk.
>
> Note: Asterisk is being used as an auto-answering client.
>
>
> Total 100 SIPP users and 100 Asterisk users are configured with
> their REGISTER and CALL cycle.
>
> All SIPP users will be making calls, Asterisk users will be
> answering it, playing a media file (For few RTP packets), let it
> connected for (120 seconds) and SIPP users finally terminate the
> call.
>
>

> *Issue:* Starting for 2 calls/second to 5 calls/second everything

> works fine. But the moment we increase that to say 7calls/second
> abruptly it starts giving 403 and 404 error for few users. All
> those time CPU usage was not high neither memory. Do you see any
> limitation in kazoo itself for processing more than 5

> calls/seconds? *Logs:* *For SIP ERROR 403* *FreeSwitch:* [WARNING]

> sofia_reg.c:2819 Can't find user [98989...@testfarm1.com] from
> 172.18.100.212 You must define a domain called 'testfarm1.com

> <http://www.google.com/url?q=http%3A%2F%2Ftestfarm1.com&sa=D&sntz=1&usg=AFQjCNEClGajQcV-Fa1KA7ts1SHHUFRIJA>'

> in your directory and add a user with the id="9898900096"
> attribute and you must configure your device to use the proper
> domain in it's authentication credentials. 2014-12-31
> 17:23:58.796386 [WARNING] sofia_reg.c:1636 SIP auth failure
> (INVITE) on sofia profile 'sipinterface_1' for

> [78789...@testfarm1.com] from ip 172.18.100.212 *For SIP ERROR
> 404 NO_ROUTE_DESTINATION * *2600hzLogs* Jan 1 14:07:33 farm9

> 2600hz[3624]: |0000000000|Undefined:Undefined (<0.493.0>)
> Supervisor whistle_couch_sup had child couch_compactor_fsm started
> with couch_compactor_fsm:start_link() at <0.12075.1> exit with
> reason no match of right hand value {error,not_found} in
> couch_compactor_fsm:get_nodes/0 line 1188 in context
> child_terminated Jan 1 14:08:33 farm9 2600hz[3624]:
> |0000000000|Undefined:Undefined (<0.13519.1>) gen_fsm
> couch_compactor_fsm in state ready terminated with reason: no match
> of right hand value {error,not_found} in
> couch_compactor_fsm:get_nodes/0 line 1188 Jan 1 14:08:33 farm9
> 2600hz[3624]: |0000000000|Undefined:Undefined (<0.13519.1>) CRASH
> REPORT Process couch_compactor_fsm with 0 neighbours exited with
> reason: no match of right hand value {error,not_found} in
> couch_compactor_fsm:get_nodes/0 line 1188 in gen_fsm:terminate/7
> line 611
>
> [NOTICE] kazoo_fetch_agent.c:286 Request for dialplan XML
> (ba82299e-9192-11e4-9d9e-17bf5339065d) timed-out after 3100ms
> 2015-01-01 14:16:53.916527 [WARNING] mod_dialplan_xml.c:588 Context
> context_2 not found 2015-01-01 14:16:53.916527 [INFO]
> switch_core_state_machine.c:239 No Route, Aborting
>
>
> Thanks
>

- --
James Aimonetti
Lead Systems Architect / Impressionable Scallywag
"I thought I fixed that"

2600Hz | http://2600hz.com
sip:ja...@2600hz.com
tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJUsAg5AAoJENTKa+JPXCVgG1YH/368LUwXz1fRHirME/2OIQEt
zdtYPoD/ZnugpPG5StCmIEBM4LmgO+AvnyDz7XjtoKM5UWYUCyI6B74g2ikpWxO0
QSaB+zDWBPygIJF9GDiwZ2jF+zdD1NabhGgK2d9d4WDQPX+YbFg0g1nYuTxKl+YZ
L4Xsda1GYr5OhlHWtViNzEJKTYemSGQiWssgDWjHXY5DUA8ohRK86T0z7ocr4rBF
RsmkkV96U091t+iCY5YCEjDLzNHD/AsQGnq2An5m6eyyxyAVx/GIYPNkDkAfQD8r
+D7BEqmg/rbOrXVF20zQUi3M3aXcdzgjv1kyIe7LgeCaMrnTzir4nPb68WwzJWE=
=7Ngg
-----END PGP SIGNATURE-----

KazooTester

unread,

Jan 24, 2015, 7:28:35 AM1/24/15

to 2600h...@googlegroups.com, ja...@2600hz.com

Hi James,

I am running this on the cloud with 7 instances (2 CouchDB + 3 Freeswitch + 2 kazoo ).

Each of them have a common configuration:

4 CPU, 6 GB RAM, 36 GB HDD and 10GB network connection.

I am seeing only one bottleneck right now, the moment I increase the concurrent calls kamailio unable to ACK to 407 Proxy authorization message send by freeswitch. After around 1 second it sends CANCEL to the current INVITE message and again attempt to other freeswitch node 2 and the same happens. Finally it gives up and send 486 Unable to Comply Error to sipp client.

Here the same I have depicted in the below flow diagram.

Also note that log level for both freeswitch and kamailio to Error level only. System average load is very low and very less than 1 for all 1,5 and 15 mins. Ulimit is also set for 65535 for all machines.

Sipp Client Kazoo(kamailio) freeswitch-1 freeswitch-2 freeswitch-3

| | | | |

|------------->| | | |

| | | | |

|100 Attempting to connect call | |

|<-------------| | | |

| | | | |

| |------------->| | |

| | | | |

| |<-------------| | |

| | | | |

| |407 Proxy Authonization | |

| |<-------------| | |

| | | | |

| |wait for 1 second, than try to other fs and send CANCEL to earlier one.

| | | | |

| |---------------------------->| |

| | | | |

| |<----------------------------| |

| | | | |

| |407 Proxy Authorization | |

| |<----------------------------| |

| | | | |

| |------------->| | |

| | | | |

| |ACK (For 407)(Almost after 1 second) |

| |------------->| | |

| | | | |

| |200 OK (For CANCEL) | |

| |------------->| | |

| | | | |

| |No ACK kamailio will be able to send to sipp client

| | | | |

| |Finally after 1 second it will send CANCEL to freeswitch-2

| | | | |

|486 Unable to Comply | | |

|<-------------| | | |

| | | | |

|ACK | | | |

|------------->| | | |

| | | | |

Let me know if you need anything else.

Thanks,

Niraj

sip:...@2600hz.com

Reply all

Reply to author

Forward

0 new messages