SipXcom 16.12.1 IVR Down

41 views
Skip to first unread message

Tommy Dome

unread,
Mar 22, 2017, 2:01:32 PM3/22/17
to sipxcom-users
I have a SipXcom 16.12.1. Currently the auto attendant and all IVR functionality is down. Checking the sipivr.log file I see the following

"2017-03-22T17:47:17.292000Z":36:sipXivr:INFO:www.oympbx.com:SocketListener0-3:00000000:sipxivr:"Mwistatus::doGet 50...@xxxxx.com"
"2017-03-22T17:47:26.011000Z":1:sipXivr:INFO:www.oympbx.com:main:00000000:sipxivr:"Starting Hazelcast instance"
"2017-03-22T17:47:27.355000Z":2:sipXivr:WARNING:www.oympbx.com:hz._hzInstance_1_hazelcast-sipxecs.IO.thread-in-1:00000000:ReadHandler:"[192.168.9.222]:6702 [hazelcast-$
"2017-03-22T17:47:34.501000Z":3:sipXivr:INFO:www.oympbx.com:main:00000000:sipxivr:"Configuration::loadXML Loading autoattendants.xml configuration"
"2017-03-22T17:47:34.559000Z":4:sipXivr:WARNING:www.oympbx.com:main:00000000:ServletHandler:"pathSpec should start with '/' or '*' : mwi"
"2017-03-22T17:47:34.560000Z":5:sipXivr:INFO:www.oympbx.com:main:00000000:sipxivr:"Adding Servlet mwistatus on mwi"
"2017-03-22T17:47:34.561000Z":6:sipXivr:INFO:www.oympbx.com:main:00000000:sipxivr:"Adding Servlet mailbox on /mailbox/*"
"2017-03-22T17:47:34.561000Z":7:sipXivr:INFO:www.oympbx.com:main:00000000:sipxivr:"Adding Servlet media on /media/*"
"2017-03-22T17:47:34.561000Z":8:sipXivr:INFO:www.oympbx.com:main:00000000:sipxivr:"Adding Servlet recording on /recording/*"
"2017-03-22T17:47:34.562000Z":9:sipXivr:INFO:www.oympbx.com:main:00000000:sipxivr:"Adding Servlet manage on /manage/*"

and

"2017-03-22T17:28:39.759000Z":89:sipXivr:WARNING:www.oympbx.com:SocketListener0-3:00000000:ServletHandler:"Error for /mwi?identity=5300%40oympbx.com&eventtype=message-$
java.lang.InternalError
        at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:784)
        at sun.misc.URLClassPath.getResource(URLClassPath.java:189)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:209)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:316)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
        at java.util.ResourceBundle$RBClassLoader.loadClass(ResourceBundle.java:459)
        at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2402)
        at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1424)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1383)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1310)
        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1252)
        at java.util.ResourceBundle.getBundle(ResourceBundle.java:730)
        at java.util.logging.Level.getLocalizedLevelName(Level.java:259)
        at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:86)
        at java.util.logging.StreamHandler.publish(StreamHandler.java:196)
        at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
        at java.util.logging.Logger.log(Logger.java:573)
        at java.util.logging.Logger.doLog(Logger.java:598)
        at java.util.logging.Logger.log(Logger.java:687)
        at com.mongodb.DBPortPool.gotError(DBPortPool.java:240)
        at com.mongodb.DBTCPConnector$MyPort.error(DBTCPConnector.java:443)
        at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:249)
        at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216)
        at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:288)
        at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:273)
        at com.mongodb.DBCollection.findOne(DBCollection.java:728)
    at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2402)
        at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1424)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1383)
        at java.util.ResourceBundle.findBundle(ResourceBundle.java:1310)
        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1252)
        at java.util.ResourceBundle.getBundle(ResourceBundle.java:730)
        at java.util.logging.Level.getLocalizedLevelName(Level.java:259)
        at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:86)
        at java.util.logging.StreamHandler.publish(StreamHandler.java:196)
        at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
        at java.util.logging.Logger.log(Logger.java:573)
        at java.util.logging.Logger.doLog(Logger.java:598)
        at java.util.logging.Logger.log(Logger.java:687)
        at com.mongodb.DBPortPool.gotError(DBPortPool.java:240)
        at com.mongodb.DBTCPConnector$MyPort.error(DBTCPConnector.java:443)
        at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:249)
        at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216)
        at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:288)
        at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:273)
        at com.mongodb.DBCollection.findOne(DBCollection.java:728)
        at com.mongodb.DBCollection.findOne(DBCollection.java:670)
        at org.sipfoundry.commons.userdb.ValidUsers.getUser(ValidUsers.java:275)
        at org.sipfoundry.voicemail.Mwistatus.doGet(Mwistatus.java:62)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:596)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
        at org.mortbay.jetty.servlet.ServletHandler.dispatch(ServletHandler.java:665)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
        at org.sipfoundry.sipxivr.rest.SipxIvrServletHandler.handle(SipxIvrServletHandler.java:52)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
        at org.mortbay.http.HttpServer.service(HttpServer.java:954)
        at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
        at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
        at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
        at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
        at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
Caused by: java.io.FileNotFoundException: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.39.x86_64/jre/lib/ext/localedata.jar
 at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:672)
        at sun.misc.URLClassPath$JarLoader.access$600(URLClassPath.java:559)
        at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:628)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:620)
        at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:782)
        ... 47 more


I have the voicemail set to DEBUG for logs but I am not getting anything beyond what I am seeing other than normal logging for MWI. Looks like an issue with Java or Mongo but I am not sure where to start. Any help would be greatly appreciated

George Niculae

unread,
Mar 22, 2017, 2:18:26 PM3/22/17
to Tommy Dome, sipxcom-users
Here's the problem 

Caused by: java.io.FileNotFoundException: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.39.x86_64/jre/lib/ext/localedata.jar

You should have java 7, probably you changed something on system if it previously worked

George

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.
To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at https://groups.google.com/group/sipxcom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/29a8a61a-813f-4095-a92b-34abdd337055%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tommy Laino

unread,
Mar 22, 2017, 2:19:47 PM3/22/17
to George Niculae, Dome, sipxcom-users
I actually haven't changed anything. Got a call this morning and have been working on it since. Did a update to 16.12.1 but to no avail. How to update to Java 7?

Tommy Laino

unread,
Mar 22, 2017, 2:21:06 PM3/22/17
to George Niculae, Tommy Dome, sipxcom-users

I also see that sipxfreeswitch is stopped and I cant get it restarted. It fails when I run service sipxfreeswitch start or restart

 

Thank you,

Tommy Laino

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.


To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at https://groups.google.com/group/sipxcom-users.


For more options, visit https://groups.google.com/d/optout.


No virus found in this message.
Checked by AVG - www.avg.com
Version: 2016.0.8007 / Virus Database: 4756/14124 - Release Date: 03/16/17

George Niculae

unread,
Mar 22, 2017, 2:26:08 PM3/22/17
to Tommy Laino, Tommy Dome, sipxco...@googlegroups.com
Check output for

java -version

To change it, use

sudo update-alternatives --config java

and pick java 7 if installed, then restart IVR. Freeswitch probably is a different issue, you could check fs logs to see why it fails

George

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at https://groups.google.com/group/sipxcom-users.

Tommy Laino

unread,
Mar 22, 2017, 2:28:48 PM3/22/17
to George Niculae, Tommy Dome, sipxco...@googlegroups.com

This is the output

 

  1           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java

*+ 2           /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java

 

Looks like Java 7 is selected. Still not working after selecting and restarting sipxivr

 

Thank you,

Tommy Laino

 

From: George Niculae [mailto:geo...@ezuce.com]
Sent: Wednesday, March 22, 2017 2:26 PM
To: Tommy Laino
Cc: Tommy Dome; sipxco...@googlegroups.com
Subject: RE: [sipxcom-users] SipXcom 16.12.1 IVR Down

 

Check output for

 

java -version

 

To change it, use

 

sudo update-alternatives --config java

 

and pick java 7 if installed, then restart IVR. Freeswitch probably is a different issue, you could check fs logs to see why it fails

 

George

On Mar 22, 2017 8:21 PM, "Tommy Laino" <toml...@gmail.com> wrote:

I also see that sipxfreeswitch is stopped and I cant get it restarted. It fails when I run service sipxfreeswitch start or restart

 

Thank you,

Tommy Laino

 

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.


To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at https://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAPMdFuj15Vv2uSpsnUoijdqF2Et%3DFZ-EMcYzwzKKix3oEtYfnA%40mail.gmail.com.


For more options, visit https://groups.google.com/d/optout.


No virus found in this message.
Checked by AVG - www.avg.com
Version: 2016.0.8007 / Virus Database: 4756/14124 - Release Date: 03/16/17

Tommy Laino

unread,
Mar 22, 2017, 2:30:55 PM3/22/17
to George Niculae, Tommy Dome, sipxco...@googlegroups.com

This is the output from freeswitch.log

 

2017-03-21 20:21:39.871760 [ERR] switch_core_sqldb.c:789 SQL ERR [database disk image is malformed]

2017-03-21 20:21:39.971758 [CRIT] switch_core_sqldb.c:1085 SQL thread unable to commit transaction, records lost!

2017-03-21 20:22:02.831749 [NOTICE] sofia.c:628 Hangup sofia/oympbx.com/53...@oympbx.com [CS_EXECUTE] [NORMAL_CLEARING]

2017-03-21 20:22:02.851746 [NOTICE] switch_ivr_bridge.c:669 Hangup sofia/oympbx.com/sip:I...@192.168.9.222:15060 [CS_EXCHANGE_MEDIA] [NORMAL_CLEARING]

2017-03-21 20:22:02.851746 [NOTICE] switch_core_session.c:1400 Session 63697 (sofia/oympbx.com/sip:I...@192.168.9.222:15060) Ended

2017-03-21 20:22:02.851746 [NOTICE] switch_core_session.c:1402 Close Channel sofia/oympbx.com/sip:I...@192.168.9.222:15060 [CS_DESTROY]

2017-03-21 20:22:02.851746 [NOTICE] switch_core_session.c:1400 Session 63696 (sofia/oympbx.com/53...@oympbx.com) Ended

2017-03-21 20:22:02.851746 [NOTICE] switch_core_session.c:1402 Close Channel sofia/oympbx.com/53...@oympbx.com [CS_DESTROY]

2017-03-21 20:22:02.871745 [NOTICE] sofia.c:628 Hangup sofia/oympbx.com/53...@192.168.9.222 [CS_EXECUTE] [NORMAL_CLEARING]

2017-03-21 20:22:02.871745 [NOTICE] switch_core_session.c:1400 Session 63698 (sofia/oympbx.com/53...@192.168.9.222) Ended

2017-03-21 20:22:02.871745 [NOTICE] switch_core_session.c:1402 Close Channel sofia/oympbx.com/53...@192.168.9.222 [CS_DESTROY]

 

Thank you,

Tommy Laino

 

From: George Niculae [mailto:geo...@ezuce.com]

Sent: Wednesday, March 22, 2017 2:26 PM
To: Tommy Laino
Cc: Tommy Dome; sipxco...@googlegroups.com

Subject: RE: [sipxcom-users] SipXcom 16.12.1 IVR Down

Check output for

 

java -version

 

To change it, use

 

sudo update-alternatives --config java

 

and pick java 7 if installed, then restart IVR. Freeswitch probably is a different issue, you could check fs logs to see why it fails

 

George

On Mar 22, 2017 8:21 PM, "Tommy Laino" <toml...@gmail.com> wrote:

I also see that sipxfreeswitch is stopped and I cant get it restarted. It fails when I run service sipxfreeswitch start or restart

 

Thank you,

Tommy Laino

 

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.


To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at https://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAPMdFuj15Vv2uSpsnUoijdqF2Et%3DFZ-EMcYzwzKKix3oEtYfnA%40mail.gmail.com.


For more options, visit https://groups.google.com/d/optout.


No virus found in this message.
Checked by AVG - www.avg.com
Version: 2016.0.8007 / Virus Database: 4756/14124 - Release Date: 03/16/17

George Niculae

unread,
Mar 22, 2017, 2:32:07 PM3/22/17
to Tommy Laino, sipxco...@googlegroups.com, Tommy Dome
Does java -version show the correct version?

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-users+unsubscribe@googlegroups.com.


To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at https://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAPMdFuj15Vv2uSpsnUoijdqF2Et%3DFZ-EMcYzwzKKix3oEtYfnA%40mail.gmail.com.


For more options, visit https://groups.google.com/d/optout.


No virus found in this message.
Checked by AVG - www.avg.com
Version: 2016.0.8007 / Virus Database: 4756/14124 - Release Date: 03/16/17

Dome Industries

unread,
Mar 22, 2017, 2:34:13 PM3/22/17
to George Niculae, sipxcom-users, Tommy Laino
Yes it shows java version 1.7.0_131

George Niculae

unread,
Mar 22, 2017, 2:34:15 PM3/22/17
to Tommy Laino, sipxco...@googlegroups.com, Tommy Dome
Re FS  - probably you have corrupt disk, try deleting /var/sipxdata/tmp/freeswitch  restart service and see if you get rid of errors

George

Tommy Laino

unread,
Mar 22, 2017, 2:49:19 PM3/22/17
to George Niculae, sipxco...@googlegroups.com, Tommy Dome

That did it. Fixed my IVR issue as well.

 

Thanks as always for all your help George

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.


To post to this group, send email to sipxco...@googlegroups.com.
Visit this group at https://groups.google.com/group/sipxcom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/CAPMdFuj15Vv2uSpsnUoijdqF2Et%3DFZ-EMcYzwzKKix3oEtYfnA%40mail.gmail.com.


For more options, visit https://groups.google.com/d/optout.


No virus found in this message.
Checked by AVG - www.avg.com
Version: 2016.0.8007 / Virus Database: 4756/14124 - Release Date: 03/16/17


No virus found in this message.
Checked by AVG - www.avg.com
Version: 2016.0.8007 / Virus Database: 4756/14124 - Release Date: 03/16/17

Reply all
Reply to author
Forward
0 new messages