Kazoo {badmatch,'$end_of_table'} Error

[Hummer]

I have just install a 7 server architecture described in the http://www.powerpbx.org/content/kazoo-v3-single-or-multiple-server-voip-telephony-platform-install-guide-v1#all tutorial. All servers seem to be working fine up till the point where l needed to do the flush {/opt/kazoo/utils/sup/sup whapps_config flush}. The BigCouch cluster seemed to have replicated well but don't know why whistle is having trouble getting config data from the database. Please help, l am sitting in a pool of hair on this one.


KAZOO Tail
Jan 19 18:22:02 sip01 2600hz[1342]: |kz_buckets|whapps_config:447 (<0.1094.0>) fetch db config for token_buckets
Jan 19 18:22:02 sip01 2600hz[1342]: |0000000000|Undefined:Undefined (emulator) Error in process <0.1094.0> on node 'whistl...@sip01.hummesh.com' with exit value: {{badmatch,'$end_of_table'},[{wh_couch_connections,get_server,0,[{file,"src/wh_couch_connections.erl"},{line,132}]},{couch_mgr,open_doc,3,[{file,"src/couch_mgr.erl"},{line,575}]},{whapps_config,get_db_config,1,[{file,"src/wh... #012#012


BIGCOUCH Tail
[Mon, 19 Jan 2015 18:29:30 GMT] [info] [<0.19183.0>] [91bf2cb6] 172.16.0.5 undefined GET / 200 ok 0
[Mon, 19 Jan 2015 18:29:31 GMT] [info] [<0.19187.0>] [--------] 172.16.0.12 - - 'GET' / 200
[Mon, 19 Jan 2015 18:29:31 GMT] [info] [<0.19202.0>] [e1338275] undefined - - 'GET' / 200
[Mon, 19 Jan 2015 18:29:31 GMT] [info] [<0.19202.0>] [e1338275] 172.16.0.12 undefined GET / 200 ok 1
[Mon, 19 Jan 2015 18:29:32 GMT] [info] [<0.19217.0>] [--------] 172.16.0.5 - - 'GET' / 200
[Mon, 19 Jan 2015 18:29:32 GMT] [info] [<0.19219.0>] [07e0da13] undefined - - 'GET' / 200
[Mon, 19 Jan 2015 18:29:32 GMT] [info] [<0.19219.0>] [07e0da13] 172.16.0.5 undefined GET / 200 ok 1
[Mon, 19 Jan 2015 18:29:33 GMT] [info] [<0.19223.0>] [--------] 172.16.0.12 - - 'GET' / 200
[Mon, 19 Jan 2015 18:29:33 GMT] [info] [<0.19238.0>] [4bfa01ee] undefined - - 'GET' / 200
[Mon, 19 Jan 2015 18:29:33 GMT] [info] [<0.19238.0>] [4bfa01ee] 172.16.0.12 undefined GET / 200 ok 0

[Hummer]

The actual error message is as follow:-

Connection to service failed!
Failed to connect to service 'whistl...@sip.mydomain.com' with cookie 'acookie'
  Possible fixes:
    * Ensure the whistle service you are trying to connect to is running on the host
    * Ensure that you are using the same cookie as the whistle node, "./sup -c <cookie>"
    * Verify that the hostname being used is a whistle node

Please note that l have checked all of these things and they seem to be in order.

[Darren Schreiber]

This means there were no Couch servers it could find that it could connect to. The bad match “$end_of_table” looks to have occurred after all known Couch servers were tried, but none worked.

Are you sure your Kazoo server can reach your databases? Try curl’ing 5984 from the Kazoo server to the database servers.

--
You received this message because you are subscribed to the Google Groups "2600hz-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 2600hz-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Or 15984 since most often Kazoo talks to a local HAProxy instance.

Le 2015-01-19 11:00, Darren Schreiber a écrit :
> This means there were no Couch servers it could find that it could
> connect to. The bad match “$end_of_table” looks to have occurred
> after all known Couch servers were tried, but none worked.
>
> Are you sure your Kazoo server can reach your databases? Try
> curl’ing 5984 from the Kazoo server to the database servers.
>

> From: Hummer <andre...@gmail.com<mailto:andre...@gmail.com>>
> Reply-To:
> "2600hz...@googlegroups.com<mailto:2600hz...@googlegroups.com>"
> <2600hz...@googlegroups.com<mailto:2600hz...@googlegroups.com>>
>
>
Date: Monday, January 19, 2015 at 1:36 PM
> To:

> "2600hz...@googlegroups.com<mailto:2600hz...@googlegroups.com>"
> <2600hz...@googlegroups.com<mailto:2600hz...@googlegroups.com>>

>
>
Subject: Kazoo {badmatch,'$end_of_table'} Error
>
> I have just install a 7 server architecture described in the
> http://www.powerpbx.org/content/kazoo-v3-single-or-multiple-server-voip-telephony-platform-install-guide-v1#all
> tutorial. All servers seem to be working fine up till the point
> where l needed to do the flush {/opt/kazoo/utils/sup/sup
> whapps_config flush}. The BigCouch cluster seemed to have
> replicated well but don't know why whistle is having trouble
> getting config data from the database. Please help, l am sitting in
> a pool of hair on this one.
>
>
> KAZOO Tail Jan 19 18:22:02 sip01 2600hz[1342]:
> |kz_buckets|whapps_config:447 (<0.1094.0>) fetch db config for
> token_buckets Jan 19 18:22:02 sip01 2600hz[1342]:
> |0000000000|Undefined:Undefined (emulator) Error in process
> <0.1094.0> on node

> 'whistl...@sip01.hummesh.com<mailto:'whistl...@sip01.hummesh.com>'

> with exit value:
> {{badmatch,'$end_of_table'},[{wh_couch_connections,get_server,0,[{file,"src/wh_couch_connections.erl"},{line,132}]},{couch_mgr,open_doc,3,[{file,"src/couch_mgr.erl"},{line,575}]},{whapps_config,get_db_config,1,[{file,"src/wh...
> #012#012
>
>
> BIGCOUCH Tail [Mon, 19 Jan 2015 18:29:30 GMT] [info] [<0.19183.0>]
> [91bf2cb6] 172.16.0.5 undefined GET / 200 ok 0 [Mon, 19 Jan 2015
> 18:29:31 GMT] [info] [<0.19187.0>] [--------] 172.16.0.12 - - 'GET'
> / 200 [Mon, 19 Jan 2015 18:29:31 GMT] [info] [<0.19202.0>]
> [e1338275] undefined - - 'GET' / 200 [Mon, 19 Jan 2015 18:29:31
> GMT] [info] [<0.19202.0>] [e1338275] 172.16.0.12 undefined GET /
> 200 ok 1 [Mon, 19 Jan 2015 18:29:32 GMT] [info] [<0.19217.0>]
> [--------] 172.16.0.5 - - 'GET' / 200 [Mon, 19 Jan 2015 18:29:32
> GMT] [info] [<0.19219.0>] [07e0da13] undefined - - 'GET' / 200
> [Mon, 19 Jan 2015 18:29:32 GMT] [info] [<0.19219.0>] [07e0da13]
> 172.16.0.5 undefined GET / 200 ok 1 [Mon, 19 Jan 2015 18:29:33 GMT]
> [info] [<0.19223.0>] [--------] 172.16.0.12 - - 'GET' / 200 [Mon,
> 19 Jan 2015 18:29:33 GMT] [info] [<0.19238.0>] [4bfa01ee] undefined
> - - 'GET' / 200 [Mon, 19 Jan 2015 18:29:33 GMT] [info]
> [<0.19238.0>] [4bfa01ee] 172.16.0.12 undefined GET / 200 ok 0
>
> -- You received this message because you are subscribed to the
> Google Groups "2600hz-users" group. To unsubscribe from this group
> and stop receiving emails from it, send an email to

> 2600hz-users...@googlegroups.com<mailto:2600hz-users...@googlegroups.com>.

>
>
For more options, visit https://groups.google.com/d/optout.
>

- --
James Aimonetti
Lead Systems Architect / Impressionable Scallywag
"I thought I fixed that"

2600Hz | http://2600hz.com
sip:ja...@2600hz.com
tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJUvVUZAAoJENTKa+JPXCVgPKgH/1hztO/LyEBJqbXBQpNVbqQ6
l/7f4TiEljXv27pIGvN/VItUOP/tfeVJzeGNiBgjGyrPGMFb0FnErzwXz2OteuDO
hLetRtM2b2xP+vmKMWDp7adxlMWgv3TClg+7Eh/si79RKu47cDbTqsO2cQsR/NLN
ClS3DhuKvN9+Y0NC4FkeaTuFldfv1e1HyiTHdsFmoNXBTa1M0Ar3oXbnNHfDEu94
ZplltX81XcTibNZh9OMHSUsgLUPFvgm2VFQHssZ/RQJ2z6LKRJvUM7UyO0W69++t
7lWAzPSpYFzc8rAAkAP9qDCtC+Duu+xRF9/MMRPSI//ln5U3M2r25R7KtIbs4Yk=
=Jp1A
-----END PGP SIGNATURE-----

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Assuming you just filed the ticket about this:


an 19 18:19:06 sip01 2600hz[1342]: |0000000000|Undefined:Undefined
(<0.1086.0>) gen_server <0.1086.0> terminated with reason: no try
clause matching {error,{conn_failed,{error,econnrefused}}} in
wh_couch_connection:maybe_reconnect/1 line 241
Jan 19 18:19:06 sip01 2600hz[1342]: |0000000000|Undefined:Undefined
(<0.1086.0>) CRASH REPORT Process <0.1086.0> with 0 neighbours exited
with reason: no try clause matching
{error,{conn_failed,{error,econnrefused}}} in
wh_couch_connection:maybe_reconnect/1 line 241 in
gen_server:terminate/6 line 747
Jan 19 18:19:06 sip01 2600hz[1342]: |0000000000|Undefined:Undefined
(<0.978.0>) Supervisor wh_couch_connection_sup had child
wh_couch_connection started with
wh_couch_connection:start_link({wh_couch_connection,1,"127.0.0.1",15984,[],[],false,false,false,{server,undefined,undefined,undefined,...}})
at <0.1086.0> exit with reason no try clause matching
{error,{conn_failed,{error,econnrefused}}} in
wh_couch_connection:maybe_reconnect/1 line 241 in context
child_terminated

econnrefused is pretty telling here!

Kazoo tried to connect on 127.0.0.1:15984 but the connection was
refused. Sounds like HAProxy isn't running or isn't configured properly.

iQEcBAEBAgAGBQJUvVlRAAoJENTKa+JPXCVg8+gIAJjVca0kKUQjlP6a3hvYvc/7
R19AvpcchVm8Z2P9lIgJYHMJOQCR/i0TcQ6ZqGfUdHM3gVta64UhVapnQ2tUvFie
yJplGKhYwPVdQTqDN+FIsD/CkhGwowFiUcivnseWjZEVx5b4TlFmRSQ95pM7aWhA
m6EEYeLVVZAQ8zrh/dXReCHQY2Nl6gnhjljhgJ3qynH/jh45dVOuKg6Q2UjQ4mHd
FVKt18+k3zDNrqY7f0Ig3m1Ur2kP6AMkv6g3BBjrgiy6Jf7cfi7KdU9dWSFP7rZv
8gRoSuT+VTcgALRjmvIn+nEI5+x6z0xEZGKC9uzSEpoFXhTeHqNbNYMWQyfjCq0=
=Q/BI
-----END PGP SIGNATURE-----

[Hummer]

Thank you for your replies guys. It really steered me in the right direction. I have solved the problem and the solution are as follows:-

1. kz_whistle_apps refused to start because there was a mismatch in the cookie value.  This one is somewhat elusive because it was not as a result of not following the installation instructions but because erlang creates a file called .erlang.cookie and write an abstract value in there is it cannot find a cookie configuration. Delete or change the value of its content to the value you have configured in the ini file and kz_whistle_apps should start consistently without any problems. NOTE- you must follow the powerpbx installation instruction to the letter for this to work for you.

2. whapps_config was unable to retrieve config data from the database because its db connectivity config requirements differs from that of the other modules. The haproxy.cfg was changed as follows and this problem along with other related or dependent ones disappeared:- NOTE:- This is a change to the powerpbx installation instruction.


listen bigcouch-data 0.0.0.0:5984
  balance roundrobin
    server db01.mydomain.com 172.1.0.1:5984 check
    server db02.mydomain.com 172.1.0.2:5984 check
    server db03.mydomain.com 172.1.0.3:5984 check backup

listen bigcouch-data 127.0.0.1:15984
  balance roundrobin
    server db01.mydomain.com 172.1.0.1:5984 check
    server db02.mydomain.com 172.1.0.2:5984 check
    server db03.mydomain.com 172.1.0.3:5984 check backup

listen bigcouch-mgr 127.0.0.1:15986
  balance roundrobin
    server db01.mydomain.com 172.16.0.1:5986 check
    server db02.mydomain.com 172.16.0.2:5986 check
    server db03.mydomain.com 172.16.0.3:5986 check backup


3. There was another side to the database connectivity issue. The Bigcouch cluster was struggling to replicate, hence had some impact on the ability for kazoo connecting with it. The reason for this was because the max_document_size in the default.ini file on the bigcouch server must be set above 1GB and the default is set to 67MB hence the issue, 1.4GB is what l think the minimum should be. Any less and you would have all sorts of connectivity problems.

I hope it works for anyone else having these problems.

On Monday, January 19, 2015 at 6:36:23 PM UTC, Hummer wrote:

I have just install a 7 server architecture described in the http://www.powerpbx.org/content/kazoo-v3-single-or-multiple-server-voip-telephony-platform-install-guide-v1#all tutorial. All servers seem to be working fine up till the point where l needed to do the flush {/opt/kazoo/utils/sup/sup whapps_config flush}. The BigCouch cluster seemed to have replicated well but don't know why whistle is having trouble getting config data from the database. Please help, l am sitting in a pool of hair on this one.


KAZOO Tail
Jan 19 18:22:02 sip01 2600hz[1342]: |kz_buckets|whapps_config:447 (<0.1094.0>) fetch db config for token_buckets

Jan 19 18:22:02 sip01 2600hz[1342]: |0000000000|Undefined:Undefined (emulator) Error in process <0.1094.0> on node 'whistl...@sip.mydomain.com' with exit value: {{badmatch,'$end_of_table'},[{wh_couch_connections,get_server,0,[{file,"src/wh_couch_connections.erl"},{line,132}]},{couch_mgr,open_doc,3,[{file,"src/couch_mgr.erl"},{line,575}]},{whapps_config,get_db_config,1,[{file,"src/wh... #012#012

[Darren Schreiber]

#3 sounds very, very wrong – 1.4GB for a single document? This can’t be the issue...

From: Hummer <andre...@gmail.com>
Reply-To: "2600hz...@googlegroups.com" <2600hz...@googlegroups.com>
Date: Monday, January 19, 2015 at 4:38 PM
To: "2600hz...@googlegroups.com" <2600hz...@googlegroups.com>

--
You received this message because you are subscribed to the Google Groups "2600hz-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to 2600hz-users...@googlegroups.com.

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Le 2015-01-19 16:38, Hummer a écrit :
> Thank you for your replies guys. It really steered me in the right
> direction. I have solved the problem and the solution are as
> follows:-
>
> 1. kz_whistle_apps refused to start because there was a mismatch in
> the cookie value. This one is somewhat elusive because it was not
> as a result of not following the installation instructions but
> because erlang creates a file called .erlang.cookie and write an
> abstract value in there is it cannot find a cookie configuration.
> Delete or change the value of its content to the value you have
> configured in the ini file and kz_whistle_apps should start
> consistently without any problems. NOTE- you must follow the
> powerpbx installation instruction to the letter for this to work
> for you.

Please prove how you came to this conclusion, as I think you've
mis-interpreted things a bit.

For the record, the cookie value of the whistle_apps VM doesn't matter
for anything other than compaction of databases. Kazoo does not use
the Distributed Erlang protocol, and thus the cookie value of the VM
is relatively meaningless. Kazoo communicates via HTTP (such as
Crossbar) and AMQP (all the backend services); neither of which
require the Erlang VM's cookie.

I suspect you made the HAProxy change at the same time as the cookie
change and when Kazoo could successfully connect to BigCouch, things
started working. I could also be wrong about that!

However, barring logs that prove to the contrary, the cookie just flat
out doesn't matter for 99% of Kazoo-related communication.

>
> 2. whapps_config was unable to retrieve config data from the
> database because its db connectivity config requirements differs
> from that of the other modules. The haproxy.cfg was changed as
> follows and this problem along with other related or dependent ones
> disappeared:- NOTE:- This is a change to the powerpbx installation
> instruction.

False. All Kazoo modules that want to access the database go through
the couch_mgr/couch_util modules.

>
>
> listen bigcouch-data 0.0.0.0:5984 balance roundrobin server
> db01.mydomain.com 172.1.0.1:5984 check server db02.mydomain.com
> 172.1.0.2:5984 check server db03.mydomain.com 172.1.0.3:5984 check
> backup
>
> listen bigcouch-data 127.0.0.1:15984 balance roundrobin server
> db01.mydomain.com 172.1.0.1:5984 check server db02.mydomain.com
> 172.1.0.2:5984 check server db03.mydomain.com 172.1.0.3:5984 check
> backup
>
> listen bigcouch-mgr 127.0.0.1:15986 balance roundrobin server
> db01.mydomain.com 172.16.0.1:5986 check server db02.mydomain.com
> 172.16.0.2:5986 check server db03.mydomain.com 172.16.0.3:5986
> check backup
>

The "normal" (but certainly not required) way to set this up is to
have HAProxy configured to listen on 15984 and 15986 on the
client/Kazoo side, and connect on the standard CouchDB ports of
5984/5986 on the backend/BigCouch side.

You then have to ensure /etc/kazoo/config.ini is also configured to
use the 15984 port.

Perhaps you edited the HAProxy config but didn't restart the service
until you added the 5984 section? I don't know, just speculating here.

Also, did you manually anonymize your BigCouch IP addresses above? How
is the admin port (5986) section going to 172.16.0.X and the api
section (5984) going to 172.1.0.X? Perhaps you made this typo in your
config file too? This would explain why Kazoo couldn't connect to
BigCouch and you saw the '$end_of_table' errors in the logs.

>
> 3. There was another side to the database connectivity issue. The
> Bigcouch cluster was struggling to replicate, hence had some impact
> on the ability for kazoo connecting with it. The reason for this
> was because the max_document_size in the default.ini file on the
> bigcouch server must be set above 1GB and the default is set to
> 67MB hence the issue, 1.4GB is what l think the minimum should be.
> Any less and you would have all sorts of connectivity problems.

What document did you create over 1GB? The only documents I can think
of that might even approach this is the voicemail box doc or maybe the
services doc. Or did you try to upload a huge media file? Seriously
confused and curious how this limit was hit for you.

>
> I hope it works for anyone else having these problems.
>
> On Monday, January 19, 2015 at 6:36:23 PM UTC, Hummer wrote:
>>
>> I have just install a 7 server architecture described in the
>> http://www.powerpbx.org/content/kazoo-v3-single-or-multiple-server-voip-telephony-platform-install-guide-v1#all
>> tutorial. All servers seem to be working fine up till the point
>> where l needed to do the flush {/opt/kazoo/utils/sup/sup
>> whapps_config flush}. The BigCouch cluster seemed to have
>> replicated well but don't know why whistle is having trouble
>> getting config data from the database. Please help, l am sitting
>> in a pool of hair on this one.
>>
>>
>> KAZOO Tail Jan 19 18:22:02 sip01 2600hz[1342]:
>> |kz_buckets|whapps_config:447 (<0.1094.0>) fetch db config for
>> token_buckets Jan 19 18:22:02 sip01 2600hz[1342]:
>> |0000000000|Undefined:Undefined (emulator) Error in process
>> <0.1094.0> on node ' whistl...@sip.mydomain.com

>> <whistl...@sip01.hummesh.com>' with exit value:

>> {{badmatch,'$end_of_table'},[{wh_couch_connections,get_server,0,[{file,"src/wh_couch_connections.erl"},{line,132}]},{couch_mgr,open_doc,3,[{file,"src/couch_mgr.erl"},{line,575}]},{whapps_config,get_db_config,1,[{file,"src/wh...
>> #012#012
>>
>>
>> BIGCOUCH Tail [Mon, 19 Jan 2015 18:29:30 GMT] [info]
>> [<0.19183.0>] [91bf2cb6] 172.16.0.5 undefined GET / 200 ok 0
>> [Mon, 19 Jan 2015 18:29:31 GMT] [info] [<0.19187.0>] [--------]
>> 172.16.0.12 - - 'GET' / 200 [Mon, 19 Jan 2015 18:29:31 GMT]
>> [info] [<0.19202.0>] [e1338275] undefined - - 'GET' / 200 [Mon,
>> 19 Jan 2015 18:29:31 GMT] [info] [<0.19202.0>] [e1338275]
>> 172.16.0.12 undefined GET / 200 ok 1 [Mon, 19 Jan 2015 18:29:32
>> GMT] [info] [<0.19217.0>] [--------] 172.16.0.5 - - 'GET' / 200
>> [Mon, 19 Jan 2015 18:29:32 GMT] [info] [<0.19219.0>] [07e0da13]
>> undefined - - 'GET' / 200 [Mon, 19 Jan 2015 18:29:32 GMT] [info]
>> [<0.19219.0>] [07e0da13] 172.16.0.5 undefined GET / 200 ok 1
>> [Mon, 19 Jan 2015 18:29:33 GMT] [info] [<0.19223.0>] [--------]
>> 172.16.0.12 - - 'GET' / 200 [Mon, 19 Jan 2015 18:29:33 GMT]
>> [info] [<0.19238.0>] [4bfa01ee] undefined - - 'GET' / 200 [Mon,
>> 19 Jan 2015 18:29:33 GMT] [info] [<0.19238.0>] [4bfa01ee]
>> 172.16.0.12 undefined GET / 200 ok 0
>>
>

- --
James Aimonetti
Lead Systems Architect / Impressionable Scallywag
"I thought I fixed that"

2600Hz | http://2600hz.com
sip:ja...@2600hz.com
tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJUvfMtAAoJENTKa+JPXCVgm0sIAKP2ZdVtKpm4xSbCqLZMcJp3
/Lkodka04Trhf9a0VetOZA+Yn5E2OTci0iHlQuvW95/AMOLeQkNpDXDmibTZvbUV
56Jv3phJvd3tkd1IXTk6URlLZMTX8vR1sOiyg/vo3UDEjTS1cOCQvqOSdWTi6Xju
JM4nAeo2mIV019ztw6uAd/LfNukIyrDUXa3FASi8npxgXY8sXamaguLu2A5bOEJ8
zPv7/5llOOAxp7PKBVqHEzwqWxi/YwZDRpZtcjodycIWm55GFTCUZ4ur1NMe27EY
xw1bjz9oMz9EjpRFYUyrl0/xNvbnsYDPvMHfZfdh8iabaJRA6IEgVHW3vHMvguU=
=jgmG
-----END PGP SIGNATURE-----

[Hummer]

Like l pointed out in my initial comment, my POC env has 7 1GB RAM VMs hosting the Kazoo servers. The bigcouch server 1 has the following in its log as it struggles with 1GB ram. It seem the HTTP response data is larger than 1024MB, hence the uncaught error below. It may well be that bigcouch need to be configured to respond with smaller size pages, but l have not figured out how to do that yet.


[Tue, 20 Jan 2015 08:56:27 GMT] [info] [<0.4130.20>] [--------] 172.16.0.5 - - 'GET' / 200
[Tue, 20 Jan 2015 08:56:29 GMT] [info] [<0.4149.20>] [753b2085] undefined - - 'GET' / 200
[Tue, 20 Jan 2015 08:56:29 GMT] [error] [<0.4149.20>] [753b2085] Uncaught error in HTTP request: {exit,normal}
[Tue, 20 Jan 2015 08:56:29 GMT] [info] [<0.4149.20>] [753b2085] Stacktrace: [{mochiweb_request,send,2,
                                              [{file,
                                                "src/mochiweb_request.erl"},
                                               {line,146}]},
                                             {mochiweb_request,respond,2,
                                              [{file,
                                                "src/mochiweb_request.erl"},
                                               {line,330}]},
                                             {couch_httpd,send_response,4,
                                              [{file,"src/couch_httpd.erl"},
                                               {line,618}]},
                                             {chttpd,handle_request,1,
                                              [{file,"src/chttpd.erl"},
                                               {line,198}]},
                                             {mochiweb_http,headers,5,
                                              [{file,"src/mochiweb_http.erl"},
                                               {line,126}]},
                                             {proc_lib,init_p_do_apply,3,
                                              [{file,"proc_lib.erl"},
                                               {line,227}]}]
[Tue, 20 Jan 2015 08:56:29 GMT] [info] [<0.4149.20>] [753b2085] undefined - - 'GET' / 500
[Tue, 20 Jan 2015 08:56:30 GMT] [info] [<0.4166.20>] [--------] 172.16.0.5 - - 'GET' / 200
[Tue, 20 Jan 2015 08:56:31 GMT] [info] [<0.4165.20>] [6fddc01d] undefined - - 'GET' / 200
[Tue, 20 Jan 2015 08:56:31 GMT] [error] [<0.4165.20>] [6fddc01d] Uncaught error in HTTP request: {exit,normal}
[Tue, 20 Jan 2015 08:56:31 GMT] [info] [<0.4165.20>] [6fddc01d] Stacktrace: [{mochiweb_request,send,2,
                                              [{file,
                                                "src/mochiweb_request.erl"},
                                               {line,146}]},
                                             {mochiweb_request,respond,2,
                                              [{file,
                                                "src/mochiweb_request.erl"},
                                               {line,330}]},
                                             {couch_httpd,send_response,4,
                                              [{file,"src/couch_httpd.erl"},
                                               {line,618}]},
                                             {chttpd,handle_request,1,
                                              [{file,"src/chttpd.erl"},
                                               {line,198}]},
                                             {mochiweb_http,headers,5,
                                              [{file,"src/mochiweb_http.erl"},
                                               {line,126}]},
                                             {proc_lib,init_p_do_apply,3,
                                              [{file,"proc_lib.erl"},
                                               {line,227}]}]
[Tue, 20 Jan 2015 08:56:31 GMT] [info] [<0.4165.20>] [6fddc01d] undefined - - 'GET' / 500
[Tue, 20 Jan 2015 08:56:32 GMT] [info] [<0.4201.20>] [--------] 172.16.0.5 - - 'GET' / 200
[Tue, 20 Jan 2015 08:56:33 GMT] [info] [<0.4220.20>] [aaf4d880] undefined - - 'GET' / 200
[Tue, 20 Jan 2015 08:56:33 GMT] [info] [<0.4220.20>] [aaf4d880] 172.16.0.5 undefined GET / 200 ok 1
[Tue, 20 Jan 2015 08:56:34 GMT] [info] [<0.4217.20>] [--------] 172.16.0.5 - - 'GET' / 200
^Z

[Hummer]

RESPONSE
It is a good question. As it goes, l  had the haproxy.cfg set to listen bigcouch-data 0.0.0.0:5984 when l made the change to the .erlang.cookie file and restarted kz_whistle_apps. It started successfully but the log started to show other modules were unable to connect to bigcouch hence the need for 127.0.0.1:15984 and every kazoo module restarted successfully without errors.
This was on a vanilla VM Build, hence there could not have been any other influences.

RESPONSE

Based on the Powerpbx instructions, Kazoo, Kamailio and haproxy are running on the same server.  As a result, kazoo is an on server haproxy client calling the 15984 port. This works for me. Some old problems return when l removed the 127.0.0.1:15984.
 

Perhaps you edited the HAProxy config but didn't restart the service
until you added the 5984 section? I don't know, just speculating here.

RESPONSE

Actually, l reboot the server just in case and this is my finding. But l would double check later on today and report back.
 

Also, did you manually anonymize your BigCouch IP addresses above? How
is the admin port (5986) section going to 172.16.0.X and the api
section (5984) going to 172.1.0.X? Perhaps you made this typo in your
config file too? This would explain why Kazoo couldn't connect to
BigCouch and you saw the '$end_of_table' errors in the logs. 

 

RESPONSE

Yes it was a poor attempt at anonymizing the IP addresses. But this is irrelevant to the three points listed.

 

sip:...@2600hz.com

[Hummer]

I just want point out that the second step stated below is wrong. PLEASE DO NOT DO BULLET POINT 2 below. Item 1 is correct and item 2 is still found to be repeatably correct. Someone in the Couchdb forum suggested that it is because beam.smb requires 1.4GB to function properly. I now have my basic Bigcouch installation running fine with 2GB ram in VMs.

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Can you link that thread?

>>> <whistl...@sip01.hummesh.com>' with exit value:

>>> {{badmatch,'$end_of_table'},[{wh_couch_connections,get_server,0,[{file,"src/wh_couch_connections.erl"},{line,132}]},{couch_mgr,open_doc,3,[{file,"src/couch_mgr.erl"},{line,575}]},{whapps_config,get_db_config,1,[{file,"src/wh...
>>> #012#012
>>>
>>>
>>> BIGCOUCH Tail [Mon, 19 Jan 2015 18:29:30 GMT] [info]
>>> [<0.19183.0>] [91bf2cb6] 172.16.0.5 undefined GET / 200 ok 0
>>> [Mon, 19 Jan 2015 18:29:31 GMT] [info] [<0.19187.0>] [--------]
>>> 172.16.0.12 - - 'GET' / 200 [Mon, 19 Jan 2015 18:29:31 GMT]
>>> [info] [<0.19202.0>] [e1338275] undefined - - 'GET' / 200 [Mon,
>>> 19 Jan 2015 18:29:31 GMT] [info] [<0.19202.0>] [e1338275]
>>> 172.16.0.12 undefined GET / 200 ok 1 [Mon, 19 Jan 2015 18:29:32
>>> GMT] [info] [<0.19217.0>] [--------] 172.16.0.5 - - 'GET' /
>>> 200 [Mon, 19 Jan 2015 18:29:32 GMT] [info] [<0.19219.0>]
>>> [07e0da13] undefined - - 'GET' / 200 [Mon, 19 Jan 2015 18:29:32
>>> GMT] [info] [<0.19219.0>] [07e0da13] 172.16.0.5 undefined GET /
>>> 200 ok 1 [Mon, 19 Jan 2015 18:29:33 GMT] [info] [<0.19223.0>]
>>> [--------] 172.16.0.12 - - 'GET' / 200 [Mon, 19 Jan 2015
>>> 18:29:33 GMT] [info] [<0.19238.0>] [4bfa01ee] undefined - -
>>> 'GET' / 200 [Mon, 19 Jan 2015 18:29:33 GMT] [info]
>>> [<0.19238.0>] [4bfa01ee] 172.16.0.12 undefined GET / 200 ok 0
>>>
>>
>

- --
James Aimonetti
Lead Systems Architect / Impressionable Scallywag
"I thought I fixed that"

2600Hz | http://2600hz.com

sip:ja...@2600hz.com

tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJU0qgwAAoJENTKa+JPXCVgZOwH/1wWOlaD2kvauurHzLDEthst
mdrI3A7mLoV/x6WTv5MyPHWwEOLqpOFS2sPK8LsRRtA8Cteo19nFaDYB9Y+aBXSJ
lNZhs4VBMAnGA2KRBD/ZsVsKzSoRClsnjosV9eMvrSBSVeYe5kyO3Tp9li7hs2Ke
n21nWEw72S0laslgUdYrFNQxj3eaQmDWYti683/ENOA7HdYLJzHYhMSWM3uk6oLe
rGfuqW2ahYCQk839plbpnHeK8QegwPbL0DCO1Nucvugznfv2gaJ/gW2e5SXkA1r9
1WbPr7Uh5JAqRGHBLd9wsdiV/ZaOkJZIdf/auWso2PD9e4oMbozT58CyUkdCVUM=
=wPA1
-----END PGP SIGNATURE-----