SAA / RLS not running

[Todd Hodgen]

I installed a new 15.06 from ISO  and restored a backup from a 4.4 system.   I’m running into a problem with SAA/RLS not running.   I’ve sent all profiles.  I’ve restarted these services specifically as well.   I see there was an issue related to saa/rls resolve din 15.06, not sure if this might be related to that fix or not – but will to dig and test to see.   Hoping someone has some suggestions.

 

I find the following in the job status –

restart services           7/4/15 4:47 PM   7/4/15 4:47 PM   Failed Agent run on phone.domain.local finshed but returned error code 143 null

IMDB regeneration              7/4/15 5:40 PM   7/4/15 5:40 PM   Failed

 

In RLS log I find this –

"2015-07-05T00:12:06.001699Z":1:KERNEL:NOTICE:phone.domain.local::7fede1005860:sipxrls:"OsResourceLimit::setCurrentLimit: Resource RLIMIT_NOFILE set to soft: 32768 hard: 65536"

"2015-07-05T00:12:06.001876Z":2:KERNEL:NOTICE:phone.domain.local::7fede1005860:sipxrls:"Maximum file descriptors set to 32768"

"2015-07-05T00:12:06.004405Z":3:SIP:NOTICE:phone.domain.local::7fede1005860:sipxrls:"sipxrls initialized"

"2015-07-05T00:12:06.005508Z":4:RLS:CRIT:phone.domain.local::7fede1005860:sipxrls:"No credential found for 'sip:~~id~sip...@domain.local' in realm 'domain.local'"

"2015-07-05T00:12:06.005573Z":5:KERNEL:NOTICE:phone.domain.local::7fede1005860:sipxrls:"Exiting sipxrls"

 

In SAA log I find similar –

"2015-07-04T17:04:18.894363Z":1:KERNEL:NOTICE:phone.domain.local::7f77f6bd6800:sipxsaa:"OsResourceLimit::setCurrentLimit: Resource RLIMIT_NOFILE set to soft: 32768 hard: 65536"

"2015-07-04T17:04:18.894535Z":2:KERNEL:NOTICE:phone.domain.local::7f77f6bd6800:sipxsaa:"Maximum file descriptors set to 32768"

"2015-07-04T17:04:18.896899Z":3:SIP:NOTICE:phone.domain.local::7f77f6bd6800:sipxsaa:"sipxsaa initialized"

"2015-07-04T17:04:18.898211Z":4:SAA:CRIT:phone.domain.local::7f77f6bd6800:sipxsaa:"No credential found for 'sip:~~id~sip...@domain.local' in realm 'domain.local'"

"2015-07-04T17:04:18.898250Z":5:KERNEL:NOTICE:phone.domain.local::7f77f6bd6800:sipxsaa:"Exiting sipxsaa"

 

This may be useless, but I find this in sipxagent log –

Skipping whole next promise (${g.pf} Service sipxsaa on is enabled), as context all is not relevant

cf3> . . . . . . . . . . . . . . . . . . . . . . . . . . . .

cf3>

cf3> . . . . . . . . . . . . . . . . . . . . . . . . . . . .

cf3> Skipping whole next promise (${g.pk} Service sipxsaa on is not enabled), as context all is not relevant

cf3> . . . . . . . . . . . . . . . . . . . . . . . . . . . .

cf3>

cf3>    =========================================================

cf3>    classes in bundle rh_chkconfig_status (2)

cf3>    =========================================================

cf3>

cf3> Class identifier "sipxsaa on_enabled" contains illegal characters - canonifying

cf3> Initiate variable convergence...

cf3>

cf3>      +  Private classes augmented:

cf3>      +       sipxsaa_on_enabled

cf3>

cf3>      -  Private classes diminished:

cf3>      -       unmanaged_dhcpd

cf3>      -       unmanaged_ntpd

cf3>      -       security_unmanaged

cf3>      -       sipxbridge

cf3>      -       mongod_arbiter

cf3>      -       mongo_local

cf3>      -       mongo_local_arbiter

cf3>      -       callqueue

cf3>      -       sipxdns_unmanaged

cf3>      -       sipxopenfire

cf3>      -       sipxofconfig

cf3>      -       sipximbot

cf3>      -       firewall_unmanaged

cf3>      -       dhcpd

 

 

 

 

Todd R. Hodgen

President / Founder

Sound IP Telecom

http://SIPTelecom.systems

206-432-4344 - Direct

206-390-4689 - Cell

 

Your Puget Sound Telcom Partner

 

Do you Like Snow, Rain, Sleet, Hail, Thunder, Lightning?

If you do, then you will love the Cloud!

 

A Division of Misiu Systems LLC

 

[Michael Picher]

Did you set the firewall settings?

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/DM2PR0701MB105099C260AC5F130D7EA7C8FD940%40DM2PR0701MB1050.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

[Todd Hodgen]

Set or Check?    Is that something new with RLS/SAA.  They were both set to Cluster.  I changed them to Public and no different. 

 

Still receiving the errors from Alarms as well - :  failed to start; exit code: 5

 

I’ll try turning Firewall off to see if it makes a difference.

 

Thanks Mike.

 

 

Todd R. Hodgen

President / Founder

Sound IP Telecom

http://SIPTelecom.systems

206-432-4344 - Direct

206-390-4689 - Cell

 

Your Puget Sound Telcom Partner

 

Do you Like Snow, Rain, Sleet, Hail, Thunder, Lightning?

If you do, then you will love the Cloud!

 

We appreciate your business referrals!

A Division of Misiu Systems LLC

 

[Tony Graziano]

Disable the service. Send the server it's profiles. Wait 10 minutes then enable the service and send the server it's profiles.

[Michael Picher]

Sorry, was away from computer and couldn't remember the jira number...  see comments section: http://jira.sipxcom.org/browse/SIPX-25

On Sat, Jul 4, 2015 at 10:46 PM Tony Graziano <tgra...@myitdepartment.net> wrote:

Disable the service. Send the server it's profiles. Wait 10 minutes then enable the service and send the server it's profiles.

--
LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: help...@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/cd92fc1a-52c4-4f19-aaa0-e1bef07055de%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,

   Mike

-----------------------------------------------------------------------------------------

There are 10 types of people in this world...  Those who understand binary and those who don't.

[Todd Hodgen]

I keep coming back to this particular error - ~~id~sip...@domain.local is NOT present in namespace imdb.entity

 

Seems off that it isn’t being found in the database – is it maybe looking for domain when it should be looking for FQDN?  Or did it get written as FQDN in error?  Could it be a corrupted imdb – my thought is there would be other issues if that was the case.  

 

 

 

Todd R. Hodgen

President / Founder

Sound IP Telecom

http://SIPTelecom.systems

206-432-4344 - Direct

206-390-4689 - Cell

 

Your Puget Sound Telcom Partner

 

Do you Like Snow, Rain, Sleet, Hail, Thunder, Lightning?

If you do, then you will love the Cloud!

 

We appreciate your business referrals!

A Division of Misiu Systems LLC

 

From: sipxco...@googlegroups.com [mailto:sipxco...@googlegroups.com] On Behalf Of Michael Picher
Sent: Sunday, July 05, 2015 12:11 AM
To: Tony Graziano; sipxco...@googlegroups.com
Subject: Re: [sipxcom-users] SAA / RLS not running

 

Sorry, was away from computer and couldn't remember the jira number...  see comments section: http://jira.sipxcom.org/browse/SIPX-25

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAAfvEN7a5nzGzQEbo4VSA%3Di2nT0694VjwSWTtjwbqd%2BgV_31%3DQ%40mail.gmail.com.

[Tony Graziano]

Has the server been rebooted since the restore process?

[Joegen Baclor]

Take a snapshot and look at the content of the identity database.  See if ~~id~sipXsaa is found.  Also take a look at domain-config file and check what's the actual domain that was assigned.  

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/BLUPR0701MB104268675C2CD7460822D6B8FD930%40BLUPR0701MB1042.namprd07.prod.outlook.com.

[George Niculae]

Try also sending profiles to server

George

On Monday, July 6, 2015, Tony Graziano <tgra...@myitdepartment.net> wrote:

Has the server been rebooted since the restore process?

--
LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: help...@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/10897415-4cda-4438-9391-56a803fcb8de%40googlegroups.com.

[Todd Hodgen]

Thanks for the tips Joegen.   I’ve looked at the domain config file, and it seems to be okay.  Matches what is displayed on the domain page.

 

I’m not finding the IMDB file – that you reference.  Can you tell me where in a snapshot that file is located?

 

Todd R. Hodgen

President / Founder

Sound IP Telecom

http://SIPTelecom.systems

206-432-4344 - Direct

206-390-4689 - Cell

 

Your Puget Sound Telcom Partner

 

Do you Like Snow, Rain, Sleet, Hail, Thunder, Lightning?

If you do, then you will love the Cloud!

 

We appreciate your business referrals!

A Division of Misiu Systems LLC

 

[Todd Hodgen]

George,  Thought I saw a response from you as well, but I think I deleted by mistake.     I have tried sending profiles several times, it doesn’t seem to make a difference.

 

Regards,

 

Todd R. Hodgen

President / Founder

Sound IP Telecom

http://SIPTelecom.systems

206-432-4344 - Direct

206-390-4689 - Cell

 

Your Puget Sound Telcom Partner

 

Do you Like Snow, Rain, Sleet, Hail, Thunder, Lightning?

If you do, then you will love the Cloud!

 

We appreciate your business referrals!

A Division of Misiu Systems LLC

 

From: Joegen Baclor [mailto:jba...@ezuce.com]

Sent: Sunday, July 05, 2015 6:57 PM

[Joegen Baclor]

mongo.imdb.entity.json

[Todd Hodgen]

Opened the json file, and did a search for ~~id~sipXsaa and it did not find it.

[Joegen Baclor]

So that is where your problem lies.  Replication of profiles failed somehow.  This is the end of how much I could help here.   Surely, this is replication related.  Though I really doubt having the domain.local root domain as being valid since that is almost the equivalent of having your domain as localhost.  I havent tried it but if I were you, i'd start from scratch and give it a valid domain that is not ending with a .local.

[George Niculae]

Todd, check if any failed jobs when you send profiles to server.

George

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/5599F293.3080904%40ezuce.com.

[Joegen Baclor]

Too much chocolates again George or you woke up really early?

[Todd Hodgen]

This is the error I get after sending profiles –

IMDB regeneration                 7/5/15 8:21 PM           7/5/15 8:21 PM           Failed

 

Joegen – he might be either celebrating or cursing the winner of the Women’s Cup this evening.  We are celebrating in this house.    Or, he is eating to much chocolate covered Coffee beans……

 

 

Todd R. Hodgen

President / Founder

Sound IP Telecom

http://SIPTelecom.systems

206-432-4344 - Direct

206-390-4689 - Cell

 

Your Puget Sound Telcom Partner

 

Do you Like Snow, Rain, Sleet, Hail, Thunder, Lightning?

If you do, then you will love the Cloud!

 

We appreciate your business referrals!

A Division of Misiu Systems LLC

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAPMdFuggqEUPPbxYXDKpJWwDTtLY-_ajmOPrCVvXNvjE%2BQBf1Q%40mail.gmail.com.

[George Niculae]

Chocolate for sure :)

Todd, check sipxconfig.log for errors and paste it here.

Thanks,

George

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/BLUPR0701MB1042E519875CE96FB82E0DB0FD930%40BLUPR0701MB1042.namprd07.prod.outlook.com.

[Todd Hodgen]

Tony had suggested that it might be related to call park.  I disabled call park earlier and had this issue still.  However, I now see that the error I see in Job status is related to sending profiles when I send profiles while call park is enabled.   When I try to go to call park, I get an error message on the screen – standard internal error has occurred – click here to continue

 

 with Park disabled, this is what I see in the sipxconfig.log file

"2015-07-06T06:21:34.021000Z":173:JAVA:INFO:phone.domain.local:pool-31-thread-1:00000000:SipxReplicationContextImpl:"Start replication: IMDB regeneration"

"2015-07-06T06:21:34.027000Z":174:JAVA:INFO:phone.domain.local:pool-31-thread-1:00000000:ReplicationManagerImpl:"Starting parallel regeneration of mongo group of 127 entities on 2 threads using chunks of 1000 users"

"2015-07-06T06:21:38.913000Z":175:JAVA:INFO:phone.domain.local:pool-31-thread-1:00000000:ReplicationManagerImpl:"Regeneration of entities finished in 4s | 0m."

"2015-07-06T06:21:38.914000Z":176:JAVA:INFO:phone.domain.local:pool-31-thread-1:00000000:ReplicationManagerImpl:"Starting parallel regeneration of mongo group of 34 entities on 2 threads using chunks of 1000 users"

"2015-07-06T06:21:39.484000Z":177:JAVA:INFO:phone.domain.local:pool-31-thread-1:00000000:ReplicationManagerImpl:"Regeneration of entities finished in 0s | 0m."

"2015-07-06T06:21:39.492000Z":178:JAVA:INFO:phone.domain.local:pool-31-thread-1:00000000:ReplicationManagerImpl:"Replication: removed Voip.ms"

"2015-07-06T06:21:39.670000Z":179:JAVA:INFO:phone.domain.local:pool-31-thread-1:00000000:ReplicationManagerImpl:"Replication: removed AutoAttendant"

"2015-07-06T06:21:39.671000Z":180:JAVA:INFO:phone.domain.local:pool-31-thread-1:00000000:ReplicationManagerImpl:"Replication: removed afterhours"

"2015-07-06T06:21:39.707000Z":182:JAVA:INFO:phone.domain.local:pool-31-thread-1:00000000:auditlog:"Replicated in Mongo DBDatabase regeneration to phone.domain.local"

"2015-07-06T06:21:41.019000Z":183:JAVA:INFO:phone.domain.local:Replication worker thread:00000000:ConfigManagerImpl:"Configuration work to do. Notifying providers."

"2015-07-06T06:21:41.023000Z":184:JAVA:INFO:phone.domain.local:Replication worker thread:00000000:ResLimitsConfiguration:"Write FEATURED resource limits"

"2015-07-06T06:21:41.757000Z":185:JAVA:WARNING:phone.domain.local:Replication worker thread:00000000:RulesXmlFile:"Cannot read from external mapping rules file: "

"2015-07-06T06:21:41.757000Z":186:JAVA:WARNING:phone.domain.local:Replication worker thread:00000000:RulesXmlFile:"Cannot read from external mapping rules file: "

"2015-07-06T06:21:41.981000Z":187:JAVA:INFO:phone.domain.local:Replication worker thread:00000000:ConfigManagerImpl:"Running Signal XMPP configuration change"

"2015-07-06T06:21:42.012000Z":188:JAVA:INFO:phone.domain.local:Replication worker thread:00000000:AgentRunner:"Starting agent run /usr/bin/sipxagent --host 127.0.0.1 "

"2015-07-06T06:21:45.484000Z":190:JAVA:INFO:phone.domain.local:pool-1-thread-1:00000000:auditlog:"Replicated: 1 - Publish finished replication event ..."

"2015-07-06T06:25:52.675000Z":191:JAVA:INFO:phone.domain.local:qtp1658543772-26:00000000:ProcessSnmpReader:"Connecting to 10.1.38.30 to get SNMP information"

"2015-07-06T06:25:52.708000Z":192:JAVA:INFO:phone.domain.local:DefaultUDPTransportMapping_10.1.38.30/0:00000000:SnmpTableResultsParser:"Table walk completed with status 0. Received 252 rows."

"2015-07-06T06:25:52.708000Z":193:JAVA:INFO:phone.domain.local:qtp1658543772-26:00000000:ProcessSnmpReader:"Table received in 30 milliseconds."

"2015-07-06T06:25:59.992000Z":194:JAVA:INFO:phone.domain.local:qtp1658543772-28:00000000:ProcessSnmpReader:"Connecting to 10.1.38.30 to get SNMP information"

"2015-07-06T06:26:00.014000Z":195:JAVA:INFO:phone.domain.local:DefaultUDPTransportMapping_10.1.38.30/0:00000000:SnmpTableResultsParser:"Table walk completed with status 0. Received 252 rows."

"2015-07-06T06:26:00.014000Z":196:JAVA:INFO:phone.domain.local:qtp1658543772-28:00000000:ProcessSnmpReader:"Table received in 20 milliseconds."

"2015-07-06T06:26:00.069000Z":197:JAVA:INFO:phone.domain.local:qtp1658543772-28:00000000:ConfigManagerImpl:"Running restart services"

"2015-07-06T06:26:00.069000Z":198:JAVA:INFO:phone.domain.local:qtp1658543772-28:00000000:AgentRunner:"Starting agent run /usr/bin/sipxagent --host 127.0.0.1   --define restart_sipxsaa --define restart_sipxrls"

"2015-07-06T06:26:23.027000Z":199:JAVA:INFO:phone.domain.local:qtp1658543772-29:00000000:ProcessSnmpReader:"Connecting to 10.1.38.30 to get SNMP information"

"2015-07-06T06:26:23.062000Z":200:JAVA:INFO:phone.domain.local:DefaultUDPTransportMapping_10.1.38.30/0:00000000:SnmpTableResultsParser:"Table walk completed with status 0. Received 252 rows."

"2015-07-06T06:26:23.063000Z":201:JAVA:INFO:phone.domain.local:qtp1658543772-29:00000000:ProcessSnmpReader:"Table received in 32 milliseconds."

 

Todd R. Hodgen

President / Founder

Sound IP Telecom

http://SIPTelecom.systems

206-432-4344 - Direct

206-390-4689 - Cell

 

Your Puget Sound Telcom Partner

 

Do you Like Snow, Rain, Sleet, Hail, Thunder, Lightning?

If you do, then you will love the Cloud!

 

We appreciate your business referrals!

A Division of Misiu Systems LLC

 

From: George Niculae [mailto:geo...@ezuce.com]

Sent: Sunday, July 05, 2015 8:28 PM
To: Todd Hodgen

[Todd Hodgen]

Not sure if my response went out.

HEre are the logs after doing a send all profiles, and then doing a restart on saa and rls from server/services

Turns out the error I am seeing for the failure to replicate is when I have call park enabled.  I've disabled it for now - although I can't seem to access call park when enable - goes to an error screen.

---

From: George Niculae <geo...@ezuce.com>
Sent: Sunday, July 5, 2015 8:27 PM
To: Todd Hodgen
Cc: Joegen Baclor; Michael Picher; Tony Graziano; sipxco...@googlegroups.com

[George Niculae]

Would be good to attach entire sipxconfig.log file, I don't see the exception that makes replication fail in lines you pasted.

[George Niculae]

I committed fix for park orbits issue (it is part of 15.08). The problem manifest only when restoring 4.4 config backup on versions > than 14.10 and can be workarounded by rebuilding orbits (that is delete them and recreate them through UI) or by manually running SQL command

update park_orbit set location_id=(select location_id from location where primary_location=true) WHERE location_id IS NULL;

[Todd Hodgen]

Resending to the entire list…

 

In the scenario I had, I wasn’t able to access the park orbits, as the system threw an error message when attempting to go to that screen.

 

A quick work around could be to delete park orbits prior to backing up and upgrading a 4.4 system to 1410 or greater…….

 

Thanks for finding this George!   George to the rescue again!

 

BTW, fixing this fixed my issue with SAA and RLS not running as well.

 

 

Todd R. Hodgen

President / Founder

Sound IP Telecom

http://SIPTelecom.systems

206-432-4344 - Direct

206-390-4689 - Cell

 

Your Puget Sound Telcom Partner

 

Do you Like Snow, Rain, Sleet, Hail, Thunder, Lightning?

If you do, then you will love the Cloud!

 

We appreciate your business referrals!

A Division of Misiu Systems LLC

 

[Tony Graziano]

I just went through this. If the park orbits are on the primary server...

psql -U postgres SIPXCONFIG
update park_orbit set location_id=1;

Then send the server it's profiles. This pretty much clears that up in a few minutes. No IMDB issues. No services shut down.