sipxproxy down

Pedro Candel

unread,

Jul 31, 2020, 4:00:49 AM7/31/20

to sipxcom-users

Hi all,

I install a sipxecs in CentOS7 but the sipproxy process goes down at 1 hour. I can see the following error:

time->Thu Jul 30 21:05:44 2020

type=ANOM_ABEND msg=audit(1596135944.785:6001): auid=4294967295 uid=992 gid=990 ses=4294967295 pid=18881 comm="sipXproxy" reason="memory violation" sig=6

----

time->Thu Jul 30 23:11:05 2020

type=ANOM_ABEND msg=audit(1596143465.154:6541): auid=4294967295 uid=992 gid=990 ses=4294967295 pid=29998 comm="sipXproxy" reason="memory violation" sig=6

[‎31/‎07/‎2020 9:05] Lopez Moreno, Manuel:

Sin título

[root@ipt01-com-mad sipxpbx]# cat sipXproxy.log-20200731 | grep Err

"2020-07-30T09:49:03.641865Z":265:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-3 protocol 1 framed 1"

"2020-07-30T09:49:03.742392Z":268:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-4 protocol 1 framed 1"

"2020-07-30T09:54:04.993989Z":600:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-5 protocol 1 framed 1"

"2020-07-30T09:54:05.094811Z":604:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-6 protocol 1 framed 1"

"2020-07-30T09:59:04.912836Z":933:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-7 protocol 1 framed 1"

"2020-07-30T09:59:05.013718Z":936:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-8 protocol 1 framed 1"

"2020-07-30T10:04:05.760758Z":1267:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-9 protocol 1 framed 1"

"2020-07-30T10:04:05.861580Z":1270:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-10 protocol 1 framed 1"

"2020-07-30T10:09:07.695186Z":1603:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-11 protocol 1 framed 1"

"2020-07-30T10:09:07.795587Z":1606:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-12 protocol 1 framed 1"

"2020-07-30T10:14:08.490222Z":1938:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-13 protocol 1 framed 1"

"2020-07-30T10:14:08.591020Z":1941:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-14 protocol 1 framed 1"

"2020-07-30T10:19:09.163942Z":2271:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-15 protocol 1 framed 1"

"2020-07-30T10:19:09.264583Z":2275:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-16 protocol 1 framed 1"

"2020-07-30T10:24:13.997172Z":2610:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-17 protocol 1 framed 1"

"2020-07-30T10:24:14.098868Z":2613:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-18 protocol 1 framed 1"

"2020-07-30T10:29:14.283396Z":2944:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-19 protocol 1 framed 1"

"2020-07-30T10:29:14.383997Z":2947:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f84ef2ae700:sipxproxy:"SipClient[SipClientUdp-11]::run error wait status tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-20 protocol 1 framed 1"

"2020-07-30T18:18:09.220349Z":80:SIP:DEBUG:ipt01-com-mad.prosodie.es:SipClientUdp-11:7f38606b5700:sipxproxy:"SipClient[SipClientUdp-11]::run start tcpOnErrWaitForSend-1 waitingToReportErr-0 mbTcpOnErrWaitForSend-1 repeatedEOFs-0"

Any idea that what is happens?

Thanks in advanced

Michael Picher

unread,

Jul 31, 2020, 4:46:26 AM7/31/20

to Pedro Candel, sipxcom-users

What are the specs for the server?

What are you running for devices?

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/f131c0d5-9377-4e06-a5c5-f557c1f591e6n%40googlegroups.com.

Manuel López

unread,

Jul 31, 2020, 7:42:18 AM7/31/20

to sipxcom-users

Hi @Michael Picher!,

First, thank you for answering!.

I'm Rockisa's workmate. I'll answer you.

The server specifications are as follows:

8GB RAM.
2G SWAP
4 CPU
120GB DISK

Server status:

[root@ipt ~]# free -h
total used free shared buff/cache available
Mem: 7.6G 1.4G 5.5G 11M 761M 6.0G
Swap: 2.0G 0B 2.0G

[root@ipt ~]# vgs -a
VG #PV #LV #SN Attr VSize VFree
cents 1 5 0 wz--n- <119.00g 100.00g
[root@ipt ~]# w
13:33:32 up 31 min, 2 users, load average: 0.02, 0.04, 0.11

Services status:

[root@ipt ~]# /etc/init.d/sipxecs status
sipxconfig ( pid 4897 ) is running and listening on port
● sipxlogwatcher.service - SYSV: Simple Event Correlator script to filter log file entries
   Loaded: loaded (/etc/rc.d/init.d/sipxlogwatcher; bad; vendor preset: disabled)
   Active: active (running) since vie 2020-07-31 13:02:13 CEST; 33min ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/sipxlogwatcher.service
           └─1599 /usr/bin/perl -w /usr/bin/sec --log=/var/log/sipxpbx/sipxlogwatcher.log --debug=1 --detach --conf=/etc/sipxpbx/si...

jul 31 13:02:09 ipt systemd[1]: Starting SYSV: Simple Event Correlator script to filter log file entries...
jul 31 13:02:13 ipt sipxlogwatcher[1160]: Starting sipxlogwatcher instance 1: [ OK ]
jul 31 13:02:13 ipt systemd[1]: Started SYSV: Simple Event Correlator script to filter log file entries.
sipxrelay ( pid 2325 ) is running and listening on port
sipxcdr ( pid 3040 ) is running and listening on port
sipXproxy ( pid 1674 ) is running and listening on port
cf-serverd ( pid 1328 ) is running and listening on port
● sipxtcpdumplog.service - sipxtcpdumplog
   Loaded: loaded (/usr/lib/systemd/system/sipxtcpdumplog.service; enabled; vendor preset: disabled)
   Active: active (exited) since vie 2020-07-31 13:02:09 CEST; 34min ago
     Docs: http://sipxcom.org
Main PID: 1050 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/sipxtcpdumplog.service
           └─1092 /usr/sbin/tcpdump -Z root -i any -s 0 -C 50 -W 100 -w /var/log/sipxpbx/tcpdump/sipxecs-tcpdump-log.pcap

jul 31 13:02:08 ipt systemd[1]: Starting sipxtcpdumplog...
jul 31 13:02:09 ipt systemd[1]: Started sipxtcpdumplog.
sipregistrar ( pid 6155 ) is running and listening on port
/etc/init.d/sipxecs: línea 21: /etc/init.d/sipxfreeswitch: No existe el fichero o el directorio

The processes that are being executed are the following:

[root@ipt ~]# ps auxf | grep -i sipxproxy | grep -v grep
sipx      1674 0.1 0.2 401340 16780 ?        Ssl 13:02   0:02 /usr/bin/sipXproxy --pid-file /var/run/sipxpbx/sipxproxy/sipxproxy.pid

It's strange because the "sipXproxy" binary dies from a system signal (sig 6)
In the audit, it shows this message:

time->Thu Jul 30 23:11:05 2020
type=ANOM_ABEND msg=audit(1596143465.154:6541): auid=4294967295 uid=992 gid=990 ses=4294967295 pid=29998 comm="sipXproxy" reason="memory violation" sig=6

But we have NO memory problems.

BR,

Manu

Manuel López

unread,

Jul 31, 2020, 7:46:50 AM7/31/20

to sipxcom-users

Hi, I can't edit the message.

Centos 7.8

sipX version 18.12.

BR,

Manu

Manuel López

unread,

Sep 8, 2020, 10:37:01 AM9/8/20

to sipxcom-users

Hi @Michael Picher ,

can you help us with this case?

BR,

Gerald Drouillard

unread,

Sep 8, 2020, 10:48:45 AM9/8/20

to Manuel López, sipxcom-users

Have you tried:

Sometimes unchecking all the services under “servers” and then rechecking them will bring a machine back. Especially SIP Trunking - uncheck - send profiles - wait - check Trunking - send profiles.

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/d9c84fcb-90f1-407d-b388-6ef1bc4c565bn%40googlegroups.com.

Matt Keys

unread,

Sep 9, 2020, 9:07:22 AM9/9/20

to sipxcom-users

Memory violation could indicate NUMA architecture in use on the hardware, or perhaps there is a device sending malformed sip messages to the proxy. On numa see: https://docs.mongodb.com/manual/administration/production-notes/#mongodb-and-numa-hardware . There is a JIRA currently open on the cfengine/sipxpromise to disable NUMA (SIPX-818), so that may not be working correctly. If NUMA is in use mongodb will probably complain at the top of /var/log/mongodb/mongod.log like ..

2017-03-28T01:52:54.910-0700 [initandlisten]

2017-03-28T01:52:54.910-0700 [initandlisten] ** WARNING: You are running on a NUMA machine.

2017-03-28T01:52:54.911-0700 [initandlisten] ** We suggest launching mongod like this to avoid performance problems:

2017-03-28T01:52:54.911-0700 [initandlisten] ** numactl --interleave=all mongod [other options]

If it is malformed SIP messages the output of /usr/bin/sipcodes.sh may provide some clues as to which user/device to investigate.

Message has been deleted

Pedro Candel

unread,

Sep 21, 2020, 5:28:15 AM9/21/20

to sipxcom-users

Hi all,

We try to install the machine in a CentOS 6.7 and we still having the same issue, sip proxy goes down. Looking with more detail we can see the following:

--

type=PATH msg=audit(18/09/20 01:18:25.153:5194292) : item=1 name=/var/log/sipxpbx/sipxrelay.log inode=3043 dev=fd:03 mode=file,644 ouid=sipx ogid=sipx rdev=00:00 nametype=NORMAL
type=PATH msg=audit(18/09/20 01:18:25.153:5194292) : item=0 name=/var/log/sipxpbx/ inode=553 dev=fd:03 mode=dir,755 ouid=sipx ogid=sipx rdev=00:00 nametype=PARENT
type=SYSCALL msg=audit(18/09/20 01:18:25.153:5194292) : arch=x86_64 syscall=open success=yes exit=25 a0=0x7f389401ed20 a1=O_WRONLY|O_CREAT|O_APPEND|O_CLOEXEC a2=0666 a3=0x1f items=2 ppid=11578 pid=11624 auid=operador uid=sipx gid=sipx euid=sipx suid=sipx fsuid=sipx egid=sipx sgid=sipx fsgid=sipx tty=(none) ses=49367 comm=java exe=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261.x86_64/jre/bin/java key=logs
----
type=ANOM_ABEND msg=audit(18/09/20 01:19:09.155:5194348) : auid=unset uid=sipx gid=sipx ses=unset pid=23753 comm=sipXproxy sig=SIGABRT

--
type=PATH msg=audit(18/09/20 02:08:20.160:5199449) : item=1 name=/var/log/sipxpbx/sipxrelay.log inode=3043 dev=fd:03 mode=file,644 ouid=sipx ogid=sipx rdev=00:00 nametype=NORMAL
type=PATH msg=audit(18/09/20 02:08:20.160:5199449) : item=0 name=/var/log/sipxpbx/ inode=553 dev=fd:03 mode=dir,755 ouid=sipx ogid=sipx rdev=00:00 nametype=PARENT
type=SYSCALL msg=audit(18/09/20 02:08:20.160:5199449) : arch=x86_64 syscall=open success=yes exit=25 a0=0x7f3894014090 a1=O_WRONLY|O_CREAT|O_APPEND|O_CLOEXEC a2=0666 a3=0x1f items=2 ppid=11578 pid=11624 auid=operador uid=sipx gid=sipx euid=sipx suid=sipx fsuid=sipx egid=sipx sgid=sipx fsgid=sipx tty=(none) ses=49367 comm=java exe=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261.x86_64/jre/bin/java key=logs
----
type=ANOM_ABEND msg=audit(18/09/20 02:09:15.031:5199515) : auid=unset uid=sipx gid=sipx ses=unset pid=26618 comm=sipXproxy sig=SIGABRT

Here the sipxrelay log:

root@ sipxpbx]# tail -19 /var/log/sipxpbx/sipxrelay.log
"2020-09-21T09:21:58.955000Z":4189:sipXrelay:ERR:ipt02-com-mad.prosodie.es:Timer-0:00000000:sipxrelay:"Error discovering address -- Check Stun Server"
net.java.stun4j.StunException: An IOException occurred while starting access point: StunAddress=/X.X.X.X:3479
at net.java.stun4j.stack.NetAccessManager.installNetAccessPoint(NetAccessManager.java:234)
at net.java.stun4j.stack.StunStack.installNetAccessPoint(StunStack.java:108)
at net.java.stun4j.client.NetworkConfigurationDiscoveryProcess.start(NetworkConfigurationDiscoveryProcess.java:173)
at org.sipfoundry.sipxrelay.SymmitronServer.discoverAddress(SymmitronServer.java:424)
at org.sipfoundry.sipxrelay.SymmitronServer.tryDiscoverAddress(SymmitronServer.java:1694)
at org.sipfoundry.sipxrelay.SymmitronServer.access$300(SymmitronServer.java:56)
at org.sipfoundry.sipxrelay.SymmitronServer$2.run(SymmitronServer.java:1766)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: java.net.BindException: La dirección ya se está usando (Bind failed)
at java.net.PlainDatagramSocketImpl.bind0(Native Method)
at java.net.AbstractPlainDatagramSocketImpl.bind(AbstractPlainDatagramSocketImpl.java:95)
at java.net.DatagramSocket.bind(DatagramSocket.java:395)
at java.net.DatagramSocket.<init>(DatagramSocket.java:249)
at net.java.stun4j.stack.NetAccessPoint.start(NetAccessPoint.java:114)
at net.java.stun4j.stack.NetAccessManager.installNetAccessPoint(NetAccessManager.java:229)
... 8 more

Thanks in advanced for your help!

Pedro Candel

unread,

Oct 29, 2020, 5:55:24 AM10/29/20

to sipxcom-users

Hi all,

The issue was solved disabling NAT transversal y el NAT behind NAT. Sipproxy process neves goes down.

Thanks a lot for your good help.

Regards,

Pedro.

Reply all

Reply to author

Forward