SipxCom 15.05 System Loses Phone Registrations

[pmkr...@gmail.com]

We have a production 15.05 update 0 system where all phones lose registrations completely every several days and the system cannot be access via the web browser. When doing a service sipxecs status command from root, the sipxregistrar and sipxsaa processes are not running - these processes return errors when trying to restart manually.  Everything comes back normally when Sipxcom is rebooted.

I took a closer look at the logs this morning, and it appears that the weekly log rotation for mongo appears to be the trigger - the same behavior has appeared the last 3 Sunday mornings. Snapshots of the Sipxregistrar, sipxsaa, and mongodb logs from this morning are at the end of this post.

Has anyone else seen this issue - any insights appreciated.

All the best
Peter

-------------------------------------------------------------------
SipRegistry.log
sipxregistry:"ReplicaSetMonitorWatcher - Failed to connect to 10.20.2.15:27017, reason: errno:111 Connection refused
sipxregistry:"ReplicaSetMonitorWatcher - No primary detected for set sipxecs"

Sipxsaa.log
"2015-07-19T07:18:09.485730Z":298:MONGO_CLIENT:WARNING:pbx.abcd.com::7f2560bb6700:sipxsaa:"ReplicaSetMonitorWatcher - No primary detected for set sipxecs"
"2015-07-19T07:18:19.486299Z":299:MONGO_CLIENT:WARNING:pbx.abcd.com::7f2560bb6700:sipxsaa:"ReplicaSetMonitorWatcher - Failed to connect to 10.20.2.15:27017, reason: errno:111 Connection refused\n"

Mongodb.log
2015-07-19T03:18:02.255-0400 [signalProcessingThread] pid=1781 port=27017 64-bit host=pbx.abcd.com
2015-07-19T03:18:02.255-0400 [signalProcessingThread] db version v2.6.7
2015-07-19T03:18:02.255-0400 [signalProcessingThread] git version: nogitversion
2015-07-19T03:18:02.255-0400 [signalProcessingThread] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2015-07-19T03:18:02.255-0400 [signalProcessingThread] build info: Linux localhost.localdomain 3.3.4-5.fc17.x86_64 #1 SMP Mon May 7 17:29:34 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49
2015-07-19T03:18:02.255-0400 [signalProcessingThread] allocator: tcmalloc
2015-07-19T03:18:02.255-0400 [signalProcessingThread] options: { command: [ "run" ], config: "/etc/mongodb.conf", net: { bindIp: "0.0.0.0", http: { enabled: false }, port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongodb.pid" }, replication: { replSet: "sipxecs" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true }, smallFiles: true }, systemLog: { destination: "file", path: "/var/log/mongodb/mongodb.log", quiet: true } }
2015-07-19T03:18:02.265-0400 [signalProcessingThread] Fatal Assertion 16782
2015-07-19T03:18:02.356-0400 [signalProcessingThread] 0xf61266 0xf0ecb2 0xef7ee3 0xf60dcf 0xf99c5c 0x7f65e30a29d1 0x7f65e1df88fd
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf61266]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x1a2) [0xf0ecb2]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xef7ee3]
 /usr/bin/mongod() [0xf60dcf]
 /usr/bin/mongod() [0xf99c5c]
 /lib64/libpthread.so.0(+0x79d1) [0x7f65e30a29d1]
 /lib64/libc.so.6(clone+0x6d) [0x7f65e1df88fd]
2015-07-19T03:18:02.356-0400 [signalProcessingThread]

***aborting after fassert() failure


2015-07-19T03:18:02.368-0400 [signalProcessingThread] SEVERE: Got signal: 6 (Aborted).
Backtrace:0xf61266 0xf61040 0x7f65e1d426a0 0x7f65e1d42625 0x7f65e1d43e05 0xef7f7c 0xf60dcf 0xf99c5c 0x7f65e30a29d1 0x7f65e1df88fd
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf61266]
 /usr/bin/mongod() [0xf61040]
 /lib64/libc.so.6(+0x326a0) [0x7f65e1d426a0]
 /lib64/libc.so.6(gsignal+0x35) [0x7f65e1d42625]
 /lib64/libc.so.6(abort+0x175) [0x7f65e1d43e05]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x13c) [0xef7f7c]
 /usr/bin/mongod() [0xf60dcf]
 /usr/bin/mongod() [0xf99c5c]
 /lib64/libpthread.so.0(+0x79d1) [0x7f65e30a29d1]
 /lib64/libc.so.6(clone+0x6d) [0x7f65e1df88fd]
warning: bind_ip of 0.0.0.0 is unnecessary; listens on all ips by default
about to fork child process, waiting until server is ready for connections.
forked process: 1783
child process started successfully, parent exiting

[Michael Picher]

Connections look to be getting used up somehow.  Could be the syslog reverse lookup issue that was fixed in 15.06.  I'd suggest upgrading or check the workaround for sipx-38.

Mike

Michael Picher, VP of Product Innovation
eZuce, Inc.

300 Brickstone Square

Suite 104

Andover, MA. 01810

O.978-296-1005 X2015 
M.207-956-0262
@mpicher <http://twitter.com/mpicher> 

linkedin
www.ezuce.com

Notice: This transmittal and/or attachments may be privileged or confidential. It is intended solely for the addressee(s) named above. Any dissemination, or copying is strictly prohibited. If you received this transmittal in error, please notify us immediately by reply and immediately delete this message and all its attachments. Thank you. FMS

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/f3e6bffa-82bf-4fbf-b359-1aa11e0e68e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George Niculae]

You need this fix, upgrade to 15.06 and make sure there's only mongodb.log under /var/log/mongodb directory

https://github.com/sipXcom/sipxecs/commit/b96150888433eebf00a751667dbd1c62bcdce31c

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAAHujP7B3V%2BBZ_ncpGvQXXPMnkYJFhy94KAN%2Bv5BeK9v6-r4pA%40mail.gmail.com.

[pmkr...@gmail.com]

Thanks George/Mike for the responses. By manually doing a log rotation on the mongodb yesterday afternoon, the issue appeared again overnight.

Are the rsyslog -x workaround (SIPX-38) and UC-3618 fix separate issues? We have another similarly configured 15.04 production system where this problem does not appear.

I also notice that UC-3618 is in the 15.05 change log - does this fix get applied if the 15.05 system is updated?

btw great work on the change logs - much appreciated.

Peter

[George Niculae]

On Mon, Jul 20, 2015 at 12:56 PM, <pmkr...@gmail.com> wrote:

Thanks George/Mike for the responses. By manually doing a log rotation on the mongodb yesterday afternoon, the issue appeared again overnight.

Are the rsyslog -x workaround (SIPX-38) and UC-3618 fix separate issues? We have another similarly configured 15.04 production system where this problem does not appear.

Yes, they are different, your problem though is related to mongodb not rsyslog IMO. Do you have regions configured on the systems you see the issue?

 

I also notice that UC-3618 is in the 15.05 change log - does this fix get applied if the 15.05 system is updated?

Yes, looks like, though you have to make sure there's no other *.log file in /var/log/mongodb/ (e.g. local-mongo.log). The mongodb logrotate directive checks for *.log, so in case any other file beside mongodb.log match pattern then logrotate will be called twice.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

[pmkr...@gmail.com]

I did a yum update on the 15.05 system to pick up the UC-3618 fix, and temporarily changed the MongoDB lograte config file from weekly to rotate nightly. There are no regions defined on either system, although on the problem system, there were arbiter.log files from 2013 - don't know how they got there, but I removed them.

Should get results tomorrow morning.

Thanks again.

Peter

[George Niculae]

Yeah, that arbiter.log is causing the issue

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/6b244312-ffee-4b92-b0b4-70b505773411%40googlegroups.com.

[pmkr...@gmail.com]

The Mongo logrotation ran successfully last evening, and everything is normal. I'll keep the mongo log rotation set to nightly for 2-3 more days before switching back to the standard weekly rotation.

Appreciate the support - Peter

[jole...@gmail.com]

Hello

I had a same problem last friday night, but i'm running sipx 15.06 and the /var/log/mongodb  folder only contains mongo logs (mongodb.log, mongodb1.log etc...)

The sip register service was stop and I had to start it to have my phones registered again.

this sipxecs is running for a month now and never had the problem until friday

do you think it is the same resolution way ? something to do with log rotate ?

thanks

best regards

https://github.com/sipXcom/sipxecs/commit/b96150888433eebf00a751667dbd1c62bcdce31c

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/f3e6bffa-82bf-4fbf-b359-1aa11e0e68e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAAHujP7B3V%2BBZ_ncpGvQXXPMnkYJFhy94KAN%2Bv5BeK9v6-r4pA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

[George Niculae]

Depends, what's the problem you see? Is mongo exiting? If so, do you have multiple .log files in /var/log/mongodb/ directory?

George

https://github.com/sipXcom/sipxecs/commit/b96150888433eebf00a751667dbd1c62bcdce31c

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/f3e6bffa-82bf-4fbf-b359-1aa11e0e68e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAAHujP7B3V%2BBZ_ncpGvQXXPMnkYJFhy94KAN%2Bv5BeK9v6-r4pA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/6b244312-ffee-4b92-b0b4-70b505773411%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e7d001d9-5eab-4620-87d5-63fdfc96ca05%40googlegroups.com.

[jole...@gmail.com]

This morning, all the phones were not registered, on sipX, the registration was expired and the sipregistrar service was stop

here is the file list in my mongod log folder

ls /var/log/mongodb/    

mongodb.log                      mongodb.log.2015-09-28T07-32-50  mongodb.log.2015-10-01T19-16-09  mongodb.log.2015-10-11T01-31-01  mongodb.log.3.gz

mongodb.log.1                    mongodb.log.2015-09-28T08-55-15  mongodb.log.2015-10-01T19-25-12  mongodb.log.2015-10-18T01-35-01  mongodb.log.4.gz

mongodb.log.2.gz                 mongodb.log.2015-10-01T18-47-29  mongodb.log.2015-10-01T19-42-45  mongodb.log.2015-10-25T02-17-01  mongodb.log.5.gz

mongodb.log.2015-09-28T07-21-59  mongodb.log.2015-10-01T18-53-54  mongodb.log.2015-10-04T01-49-01  mongodb.log.2015-11-01T02-18-02

I found this log at the time it crashed

"2015-11-01T02:24:51.773172Z":1102958:SIP:EMERG:vm1.sipx.local::7f0d9be2e700:sipxregistry:"ALARM_MONGODB_SLOW_READ Last Mongo read took a long time: document: node.registrar delay: 

353 milliseconds"

"2015-11-01T18:58:49.857963Z":1120956:SIP:EMERG:vm1.sipx.local::7f0d9b42d700:sipxregistry:"ALARM_MONGODB_SLOW_UPDATE Last Mongo update took a long time: document: node.registrar del

ay: 867 milliseconds"

"2015-11-01T19:04:49.954596Z":1121061:LOG:CRIT:vm1.sipx.local:SipSubscribeServer-29:7f0d8e2e8700:sipxregistry:"mongo query returned null cursor\n/usr/lib64/libsipXcommserver.so.2(_Z

N18OsExceptionHandler33defaultStdGeneralExceptionHandlerERSt9exception+0x86) [0x7f0da45f2aa6]\n/usr/lib64/libsipXcommserver.so.2(_ZN18OsExceptionHandler12callHandlersESt6vectorIN5boost

8functionIFvRSt9exceptionEEESaIS6_EES4_+0x51) [0x7f0da45e7321]\n/usr/lib64/libsipXcommserver.so.2(_ZN18OsExceptionHandler14treatExceptionE13ExceptionTypeRKSsRSt3mapIS1_St6vectorIN5boos

t8functionIFvRSt9exceptionEEESaISA_EESt4lessIS1_ESaISt4pairIS1_SC_EEES8_+0x1f1) [0x7f0da45ea061]\n/usr/lib64/libsipXcommserver.so.2(_ZN18OsExceptionHandler12catch_globalEv+0x275) [0x7f

0da45ea4b5]\n/usr/lib64/libstdc++.so.6(+0xbcbd6) [0x7f0da0f45bd6]\n/usr/lib64/libstdc++.so.6(+0xbcc03) [0x7f0da0f45c03]\n/usr/lib64/libstdc++.so.6(+0xbcd22) [0x7f0da0f45d22]\n/usr/lib6

4/libsipXcommserver.so.2(_ZNK11SubscribeDB11updateToTagERK9UtlStringS2_S2_+0x16b9) [0x7f0da4638de9]\n/usr/lib64/libsipXcommserver.so.2(_ZN28SipPersistentSubscriptionMgr19getNotifyDialo

gInfoERK9UtlStringR10SipMessagePKcPS0_S7_S7_S7_Pb+0x1d5) [0x7f0da45ddfe5]\n/usr/lib64/libsipXtack.so.2(_ZN18SipSubscribeServer15handleSubscribeERK10SipMessage+0x1ea) [0x7f0da4056f0a]\n

/usr/lib64/libsipXtack.so.2(_ZN18SipSubscribeServer13handleMessageER5OsMsg+0x1c2) [0x7f0da4057722]\n/usr/lib64/libsipXport.so.2(_ZN12OsServerTask3runEPv+0x67) [0x7f0da3cdb647]\n/usr/li

b64/libsipXport.so.2(_ZN11OsTaskLinux9taskEntryEPv+0x1c0) [0x7f0da3d088b0]\n/lib64/libpthread.so.0(+0x79d1) [0x7f0da359b9d1]\n/lib64/libc.so.6(clone+0x6d) [0x7f0da07438fd]\n"

https://github.com/sipXcom/sipxecs/commit/b96150888433eebf00a751667dbd1c62bcdce31c

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/f3e6bffa-82bf-4fbf-b359-1aa11e0e68e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAAHujP7B3V%2BBZ_ncpGvQXXPMnkYJFhy94KAN%2Bv5BeK9v6-r4pA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/6b244312-ffee-4b92-b0b4-70b505773411%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George Niculae]

That's a different issue than the one solved in this topic

George

https://github.com/sipXcom/sipxecs/commit/b96150888433eebf00a751667dbd1c62bcdce31c

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/f3e6bffa-82bf-4fbf-b359-1aa11e0e68e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAAHujP7B3V%2BBZ_ncpGvQXXPMnkYJFhy94KAN%2Bv5BeK9v6-r4pA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/6b244312-ffee-4b92-b0b4-70b505773411%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e7d001d9-5eab-4620-87d5-63fdfc96ca05%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/4d679cc3-9921-4721-a3bc-8f0a5b97187d%40googlegroups.com.

[jole...@gmail.com]

ok, I open a new topic

https://github.com/sipXcom/sipxecs/commit/b96150888433eebf00a751667dbd1c62bcdce31c

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/f3e6bffa-82bf-4fbf-b359-1aa11e0e68e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAAHujP7B3V%2BBZ_ncpGvQXXPMnkYJFhy94KAN%2Bv5BeK9v6-r4pA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e601b396-e4ef-416a-a6ce-f7c624545f44%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/6b244312-ffee-4b92-b0b4-70b505773411%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at http://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/e7d001d9-5eab-4620-87d5-63fdfc96ca05%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.