/dev/mapper/centos-root Disk Drive Fills Up in 20.08

[Peter Krautle]

We have a Sipxcom 20.08 system that fills up nightly and runs out of disk space. The issue happens around log rotation at ~3:30 AM. The Sipxbridge.log file is the main file that creates the disk full condition, and when this file is erased, the system is rebooted, and sipxbridge is restarted, everything returns to normal.

The system talks to an Edgemarc 4552 SBC that is managed by Optimum, and has been in place since 2012. The voice server was upgraded recently from 4.6.11 where this issue never appeared. We connect to the SBC via a registered SIP trunk.

Any thoughts on how to troubleshoot this issue is appreciated. When the issue reappears, I'll try and look at the Sipxbridge.log file to gather more information on what is triggering the issue.

All the best

Peter

[Todd Hodgen]

Peter,  that file might be taking you over the deep edge.  However, I had a similar issue, but it turned out the system wasn't deleting the old backup file.  In their case, nobody was deleting old voicemails and relied on them forwarded to email.  I ended up clearing all but a couple backups and cleared old emails out to fix things.

sent using my two left twiddling thumbs

---

From: sipxco...@googlegroups.com <sipxco...@googlegroups.com> on behalf of Peter Krautle <pmkr...@gmail.com>
Sent: Wednesday, March 9, 2022 9:31:19 AM
To: sipxcom-users <sipxco...@googlegroups.com>
Subject: [sipxcom-users] /dev/mapper/centos-root Disk Drive Fills Up in 20.08

 

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/b4dcb41c-f381-46e7-a264-7ed14cb015bbn%40googlegroups.com.

[Michael Picher]

Backups require 3x the size of all voicemails...

To view this discussion on the web visit https://groups.google.com/d/msgid/sipxcom-users/SJ0P222MB050484518EB5FA35C6F5811BFD0A9%40SJ0P222MB0504.NAMP222.PROD.OUTLOOK.COM.

--

 

Michael Picher Product Manager, UCaaS 215.297.4400 x.445 751 Arbor Way, Arbor Crest I, Suite 150, Blue Bell, PA 19422 coredial.com

The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

[Peter Krautle]

Thanks for the response. What I'm seeing is that sipxbridge is running but lots of Java errors in the log file. I stand to be corrected, but suspect the Edgemarc is intermittently misbehaving and sending a SIP exchange that triggers Sipxbridge and causes the log file behavior. Optimum regularly applies updates their cable modems and SIP gateways around the same time as log rotations - the SIP gateways are probably updated less frequently and hence, the reason the issue does not occur regularly. A disk full probe is being enabled which will generate an alarm when the disk is 90 percent full. I'll go in do an analysis of Sipxbridge log file to see whether a trigger can be found. I'll also cal Optimum support to see whether any firmware updates or patches were applied around the time that the issue took place. Peter

[Matt Keys]

sipcodes.sh can be used against sipxbridge.log as well as sipXproxy.log. The only catch is it must be at least INFO verbosity. Typically if bridge is large proxy will also be large. Bridge is just the ITSP facing side of the conversation and proxy has all the rest. 

[Matt Keys]

I should also add you've pointed out it's a new install and using LVM given the subject line. In Centos 7 if the disk is larger than 50G and you used automatic partitioning, most of the space will be allocated to /home rather than /. 

Check the 'df -h' command to see if that's the case. If it is it'd be a good idea to remove /home and allocate that to / instead. There's a good guide here: https://gist.github.com/troyfontaine/87091bd6a5c68f45dd62ced3d12bc377

[Peter Krautle]

Thanks Matt - info level verbosity has been enabled on Sipxbridge - think the issue will be pinpointed there. The Sipxcom disk space is 50G -will look at the document (and thank you!). Besides the PRTG alarming of disk space on this system, I also manually check the system every morning before the office opens. To get this voice server up and running quickly, I also built a Windows desktop on the same server - we built the system with Vmware and Windows to get remote access to the Edgemarc which does not have a default gateway in their configuration. When the issue occurs again, I'll FTP the sipxbridge.log filer to the Windows desktop, erase the file on Sipxcom to get the voice server running again, and then transmit overnight to analyze for analysis. Peter

[Matt Keys]

I'd also recommend upgrade to 21.04 also so you get the DNS fix that disables AAAA on IPv4 (UC-4850 - https://wiki.sipxcom.org/display/sipXcom/sipXcom+21.04 ).