SipxCom 15.05 System Loses Phone Registrations

167 views
Skip to first unread message

pmkr...@gmail.com

unread,
Jul 19, 2015, 10:04:02 AM7/19/15
to sipxco...@googlegroups.com
We have a production 15.05 update 0 system where all phones lose registrations completely every several days and the system cannot be access via the web browser. When doing a service sipxecs status command from root, the sipxregistrar and sipxsaa processes are not running - these processes return errors when trying to restart manually.  Everything comes back normally when Sipxcom is rebooted.

I took a closer look at the logs this morning, and it appears that the weekly log rotation for mongo appears to be the trigger - the same behavior has appeared the last 3 Sunday mornings. Snapshots of the Sipxregistrar, sipxsaa, and mongodb logs from this morning are at the end of this post.

Has anyone else seen this issue - any insights appreciated.

All the best
Peter

-------------------------------------------------------------------
SipRegistry.log
sipxregistry:"ReplicaSetMonitorWatcher - Failed to connect to 10.20.2.15:27017, reason: errno:111 Connection refused
sipxregistry:"ReplicaSetMonitorWatcher - No primary detected for set sipxecs"

Sipxsaa.log
"2015-07-19T07:18:09.485730Z":298:MONGO_CLIENT:WARNING:pbx.abcd.com::7f2560bb6700:sipxsaa:"ReplicaSetMonitorWatcher - No primary detected for set sipxecs"
"2015-07-19T07:18:19.486299Z":299:MONGO_CLIENT:WARNING:pbx.abcd.com::7f2560bb6700:sipxsaa:"ReplicaSetMonitorWatcher - Failed to connect to 10.20.2.15:27017, reason: errno:111 Connection refused\n"

Mongodb.log
2015-07-19T03:18:02.255-0400 [signalProcessingThread] pid=1781 port=27017 64-bit host=pbx.abcd.com
2015-07-19T03:18:02.255-0400 [signalProcessingThread] db version v2.6.7
2015-07-19T03:18:02.255-0400 [signalProcessingThread] git version: nogitversion
2015-07-19T03:18:02.255-0400 [signalProcessingThread] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2015-07-19T03:18:02.255-0400 [signalProcessingThread] build info: Linux localhost.localdomain 3.3.4-5.fc17.x86_64 #1 SMP Mon May 7 17:29:34 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49
2015-07-19T03:18:02.255-0400 [signalProcessingThread] allocator: tcmalloc
2015-07-19T03:18:02.255-0400 [signalProcessingThread] options: { command: [ "run" ], config: "/etc/mongodb.conf", net: { bindIp: "0.0.0.0", http: { enabled: false }, port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongodb.pid" }, replication: { replSet: "sipxecs" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true }, smallFiles: true }, systemLog: { destination: "file", path: "/var/log/mongodb/mongodb.log", quiet: true } }
2015-07-19T03:18:02.265-0400 [signalProcessingThread] Fatal Assertion 16782
2015-07-19T03:18:02.356-0400 [signalProcessingThread] 0xf61266 0xf0ecb2 0xef7ee3 0xf60dcf 0xf99c5c 0x7f65e30a29d1 0x7f65e1df88fd
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf61266]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x1a2) [0xf0ecb2]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xef7ee3]
 /usr/bin/mongod() [0xf60dcf]
 /usr/bin/mongod() [0xf99c5c]
 /lib64/libpthread.so.0(+0x79d1) [0x7f65e30a29d1]
 /lib64/libc.so.6(clone+0x6d) [0x7f65e1df88fd]
2015-07-19T03:18:02.356-0400 [signalProcessingThread]

***aborting after fassert() failure


2015-07-19T03:18:02.368-0400 [signalProcessingThread] SEVERE: Got signal: 6 (Aborted).
Backtrace:0xf61266 0xf61040 0x7f65e1d426a0 0x7f65e1d42625 0x7f65e1d43e05 0xef7f7c 0xf60dcf 0xf99c5c 0x7f65e30a29d1 0x7f65e1df88fd
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf61266]
 /usr/bin/mongod() [0xf61040]
 /lib64/libc.so.6(+0x326a0) [0x7f65e1d426a0]
 /lib64/libc.so.6(gsignal+0x35) [0x7f65e1d42625]
 /lib64/libc.so.6(abort+0x175) [0x7f65e1d43e05]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x13c) [0xef7f7c]
 /usr/bin/mongod() [0xf60dcf]
 /usr/bin/mongod() [0xf99c5c]
 /lib64/libpthread.so.0(+0x79d1) [0x7f65e30a29d1]
 /lib64/libc.so.6(clone+0x6d) [0x7f65e1df88fd]
warning: bind_ip of 0.0.0.0 is unnecessary; listens on all ips by default
about to fork child process, waiting until server is ready for connections.
forked process: 1783
child process started successfully, parent exiting

Michael Picher

unread,
Jul 19, 2015, 10:14:37 AM7/19/15
to pmkr...@gmail.com, sipxco...@googlegroups.com
Connections look to be getting used up somehow.  Could be the syslog reverse lookup issue that was fixed in 15.06.  I'd suggest upgrading or check the workaround for sipx-38.

Mike


Michael Picher, VP of Product Innovation
eZuce, Inc.

300 Brickstone Square

Suite 104

Andover, MA. 01810


Notice: This transmittal and/or attachments may be privileged or confidential. It is intended solely for the addressee(s) named above. Any dissemination, or copying is strictly prohibited. If you received this transmittal in error, please notify us immediately by reply and immediately delete this message and all its attachments. Thank you. FMS

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/f3e6bffa-82bf-4fbf-b359-1aa11e0e68e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

George Niculae

unread,
Jul 19, 2015, 11:03:07 AM7/19/15
to Michael Picher, pmkr...@gmail.com, sipxco...@googlegroups.com
You need this fix, upgrade to 15.06 and make sure there's only mongodb.log under /var/log/mongodb directory

pmkr...@gmail.com

unread,
Jul 20, 2015, 6:56:45 AM7/20/15
to sipxco...@googlegroups.com, pmkr...@gmail.com, mpi...@ezuce.com
Thanks George/Mike for the responses. By manually doing a log rotation on the mongodb yesterday afternoon, the issue appeared again overnight.

Are the rsyslog -x workaround (SIPX-38) and UC-3618 fix separate issues? We have another similarly configured 15.04 production system where this problem does not appear.

I also notice that UC-3618 is in the 15.05 change log - does this fix get applied if the 15.05 system is updated?

btw great work on the change logs - much appreciated.

Peter

George Niculae

unread,
Jul 20, 2015, 7:12:10 AM7/20/15
to pmkr...@gmail.com, sipxco...@googlegroups.com, Michael Picher
On Mon, Jul 20, 2015 at 12:56 PM, <pmkr...@gmail.com> wrote:
Thanks George/Mike for the responses. By manually doing a log rotation on the mongodb yesterday afternoon, the issue appeared again overnight.

Are the rsyslog -x workaround (SIPX-38) and UC-3618 fix separate issues? We have another similarly configured 15.04 production system where this problem does not appear.

Yes, they are different, your problem though is related to mongodb not rsyslog IMO. Do you have regions configured on the systems you see the issue?
 

I also notice that UC-3618 is in the 15.05 change log - does this fix get applied if the 15.05 system is updated?

Yes, looks like, though you have to make sure there's no other *.log file in /var/log/mongodb/ (e.g. local-mongo.log). The mongodb logrotate directive checks for *.log, so in case any other file beside mongodb.log match pattern then logrotate will be called twice.
 

pmkr...@gmail.com

unread,
Jul 20, 2015, 2:54:51 PM7/20/15
to sipxco...@googlegroups.com, mpi...@ezuce.com, pmkr...@gmail.com
I did a yum update on the 15.05 system to pick up the UC-3618 fix, and temporarily changed the MongoDB lograte config file from weekly to rotate nightly. There are no regions defined on either system, although on the problem system, there were arbiter.log files from 2013 - don't know how they got there, but I removed them.

Should get results tomorrow morning.

Thanks again.

Peter

George Niculae

unread,
Jul 20, 2015, 3:21:19 PM7/20/15
to pmkr...@gmail.com, sipxco...@googlegroups.com, mpi...@ezuce.com
Yeah, that arbiter.log is causing the issue
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/6b244312-ffee-4b92-b0b4-70b505773411%40googlegroups.com.

pmkr...@gmail.com

unread,
Jul 21, 2015, 7:46:03 AM7/21/15
to sipxco...@googlegroups.com, pmkr...@gmail.com, mpi...@ezuce.com
The Mongo logrotation ran successfully last evening, and everything is normal. I'll keep the mongo log rotation set to nightly for 2-3 more days before switching back to the standard weekly rotation.

Appreciate the support - Peter

jole...@gmail.com

unread,
Nov 2, 2015, 4:11:52 AM11/2/15
to sipxcom-users, pmkr...@gmail.com, mpi...@ezuce.com
Hello

I had a same problem last friday night, but i'm running sipx 15.06 and the /var/log/mongodb  folder only contains mongo logs (mongodb.log, mongodb1.log etc...)
The sip register service was stop and I had to start it to have my phones registered again.

this sipxecs is running for a month now and never had the problem until friday

do you think it is the same resolution way ? something to do with log rotate ?

thanks
best regards



To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
 To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

George Niculae

unread,
Nov 2, 2015, 4:14:01 AM11/2/15
to jole...@gmail.com, sipxcom-users, pmkr...@gmail.com, Michael Picher
Depends, what's the problem you see? Is mongo exiting? If so, do you have multiple .log files in /var/log/mongodb/ directory?

George

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAAHujP7B3V%2BBZ_ncpGvQXXPMnkYJFhy94KAN%2Bv5BeK9v6-r4pA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
 To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e7d001d9-5eab-4620-87d5-63fdfc96ca05%40googlegroups.com.

jole...@gmail.com

unread,
Nov 2, 2015, 4:27:34 AM11/2/15
to sipxcom-users, jole...@gmail.com, pmkr...@gmail.com, mpi...@ezuce.com

This morning, all the phones were not registered, on sipX, the registration was expired and the sipregistrar service was stop

here is the file list in my mongod log folder

ls /var/log/mongodb/    
mongodb.log                      mongodb.log.2015-09-28T07-32-50  mongodb.log.2015-10-01T19-16-09  mongodb.log.2015-10-11T01-31-01  mongodb.log.3.gz
mongodb.log.1                    mongodb.log.2015-09-28T08-55-15  mongodb.log.2015-10-01T19-25-12  mongodb.log.2015-10-18T01-35-01  mongodb.log.4.gz
mongodb.log.2.gz                 mongodb.log.2015-10-01T18-47-29  mongodb.log.2015-10-01T19-42-45  mongodb.log.2015-10-25T02-17-01  mongodb.log.5.gz
mongodb.log.2015-09-28T07-21-59  mongodb.log.2015-10-01T18-53-54  mongodb.log.2015-10-04T01-49-01  mongodb.log.2015-11-01T02-18-02

I found this log at the time it crashed

"2015-11-01T02:24:51.773172Z":1102958:SIP:EMERG:vm1.sipx.local::7f0d9be2e700:sipxregistry:"ALARM_MONGODB_SLOW_READ Last Mongo read took a long time: document: node.registrar delay: 
353 milliseconds"
"2015-11-01T18:58:49.857963Z":1120956:SIP:EMERG:vm1.sipx.local::7f0d9b42d700:sipxregistry:"ALARM_MONGODB_SLOW_UPDATE Last Mongo update took a long time: document: node.registrar del
ay: 867 milliseconds"
"2015-11-01T19:04:49.954596Z":1121061:LOG:CRIT:vm1.sipx.local:SipSubscribeServer-29:7f0d8e2e8700:sipxregistry:"mongo query returned null cursor\n/usr/lib64/libsipXcommserver.so.2(_Z
N18OsExceptionHandler33defaultStdGeneralExceptionHandlerERSt9exception+0x86) [0x7f0da45f2aa6]\n/usr/lib64/libsipXcommserver.so.2(_ZN18OsExceptionHandler12callHandlersESt6vectorIN5boost
8functionIFvRSt9exceptionEEESaIS6_EES4_+0x51) [0x7f0da45e7321]\n/usr/lib64/libsipXcommserver.so.2(_ZN18OsExceptionHandler14treatExceptionE13ExceptionTypeRKSsRSt3mapIS1_St6vectorIN5boos
t8functionIFvRSt9exceptionEEESaISA_EESt4lessIS1_ESaISt4pairIS1_SC_EEES8_+0x1f1) [0x7f0da45ea061]\n/usr/lib64/libsipXcommserver.so.2(_ZN18OsExceptionHandler12catch_globalEv+0x275) [0x7f
0da45ea4b5]\n/usr/lib64/libstdc++.so.6(+0xbcbd6) [0x7f0da0f45bd6]\n/usr/lib64/libstdc++.so.6(+0xbcc03) [0x7f0da0f45c03]\n/usr/lib64/libstdc++.so.6(+0xbcd22) [0x7f0da0f45d22]\n/usr/lib6
4/libsipXcommserver.so.2(_ZNK11SubscribeDB11updateToTagERK9UtlStringS2_S2_+0x16b9) [0x7f0da4638de9]\n/usr/lib64/libsipXcommserver.so.2(_ZN28SipPersistentSubscriptionMgr19getNotifyDialo
gInfoERK9UtlStringR10SipMessagePKcPS0_S7_S7_S7_Pb+0x1d5) [0x7f0da45ddfe5]\n/usr/lib64/libsipXtack.so.2(_ZN18SipSubscribeServer15handleSubscribeERK10SipMessage+0x1ea) [0x7f0da4056f0a]\n
/usr/lib64/libsipXtack.so.2(_ZN18SipSubscribeServer13handleMessageER5OsMsg+0x1c2) [0x7f0da4057722]\n/usr/lib64/libsipXport.so.2(_ZN12OsServerTask3runEPv+0x67) [0x7f0da3cdb647]\n/usr/li
b64/libsipXport.so.2(_ZN11OsTaskLinux9taskEntryEPv+0x1c0) [0x7f0da3d088b0]\n/lib64/libpthread.so.0(+0x79d1) [0x7f0da359b9d1]\n/lib64/libc.so.6(clone+0x6d) [0x7f0da07438fd]\n"
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
 To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

George Niculae

unread,
Nov 2, 2015, 5:12:13 AM11/2/15
to jole...@gmail.com, sipxcom-users, pmkr...@gmail.com, Michael Picher
That's a different issue than the one solved in this topic

George

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAAHujP7B3V%2BBZ_ncpGvQXXPMnkYJFhy94KAN%2Bv5BeK9v6-r4pA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
 To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/4d679cc3-9921-4721-a3bc-8f0a5b97187d%40googlegroups.com.

jole...@gmail.com

unread,
Nov 4, 2015, 3:34:59 AM11/4/15
to sipxcom-users, jole...@gmail.com, pmkr...@gmail.com, mpi...@ezuce.com
ok, I open a new topic
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
 To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
Reply all
Reply to author
Forward
0 new messages