Upcoming Update 9 / Garbage Collection Fix

[OnRelay Support]

If you are testing 25.01 on RL9 beware of this issue: https://github.com/onrelay/sipxecs/issues/67

We found over time the SIP transaction garbage collection stopped working, so eventually the number of active SIP transactions exceeds the max, causing 408 timeouts.

The issue has been diagnosed, and a fix is being tested for update 9. We will however just monitor for a couple of days before publishing to ensure no unexpected side effects.

[OnRelay Support]

Just confirming update 9 has now been published with this fix only.

[J.T. D]

Thank you for this.  I'm happy/sad to report that for me, it works better, but still has issues.

Before this fix, I'd see this error shortly after starting the services.  After the fix/update, everything is stable for a good number of hours, but shows itself again in under 24 hours since last service restart.  I see in the sipxproxy-alarms.log file the error about proxy filled queue and transaction count is 28000+/20000.  (I did update to 9.)

[Support]

Thanks,

There were actually 2 different issues we fixed that both blocked the transaction garbage collector, both similar older C++ delete errors that previously did not cause any damage, but now blocked or hung the garbage collection thread.

After the fix garbage collection has been stable here now, but there could be something different in your config that triggers a similar error.  Would you mind looking for the last occurrence of removeOldTransactions in the sipXproxy.log, if need be turn on DEBUG for sipXproxy first, and send us the log immediately surrounding that last occurrence?

Thank you;

--
You received this message because you are subscribed to the Google Groups "sipxcom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sipxcom-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sipxcom-users/6e1d8562-74a6-4144-8df2-a2905a3fa7den%40googlegroups.com.

[J.T. D]

As requested.  Thank you!

[Support]

We are struggling to reproduce this as everything is stable here now including transaction garbage collection.

As you can see from your logs the garbage collector stops after destruction of SipClientTcp's. This is also where the previous error was, where flushing its messages in the destructor could cause a garbage collector thread crash or block.

So we have now reverted one strictly unnecessary change in the destructor making the SipClientTcp wait longer for shutdown as in the previous implementation, in a hope that may resolve it. 

This fix attempt will be included in update 10, but we continue to keep the issue open until fix is confirmed.

<sipxproxy_debug.log>

[J.T. D]

Updated to update 10.  Experienced the same behavior, though it isn't as routine as it had been: a few days elapsed between crashes, not every day as it had been previously.

[Ivar Plahte]

Hmm, tricky one, not seeing it here. A difference is we use TLS to register SIP terminals, you probably use TCP?  

Previous garbage collection errors were with the SIP TCP client C++ class in sipXproxy.

Would you mind or are you able to switch between TCP and UDP to check if the behavior persists with UDP? Not as a permanent change, just to see if we can isolate the error. It is a bit of a PIA to change to / from secure SIP end-to-end here. 

To view this discussion visit https://groups.google.com/d/msgid/sipxcom-users/3b51bec3-4162-405f-9a2b-badefa3837a1n%40googlegroups.com.

[J.T. D]

Sure, I'll make the change and report back.

[J.T. D]

I regret to report: same behavior.  Happened 6 hours after a restart of services. I enabled debug logging for sipxproxy, if that helps.

[Support]

Not to worry, the TCP client is still used for internal signaling, so doesn’t tell us much except the issue is still open.

Yes. the debug log immediately following your last RemoveOldTransactions entry is useful.

To view this discussion visit https://groups.google.com/d/msgid/sipxcom-users/834f18fd-9eec-4ee0-893e-a2020debfb5an%40googlegroups.com.

[J.T. D]

Attached.

[Support]

Yes, pretty clear it is the 2nd of these Tcp SIP clients that is causing the lock, hang or thread crash when being deleted, as the 3rd deletion is not executed. We will figure it out.

"2026-08-21T18:34:31.283000Z":395386:SIP:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"SipProtocolServerBase[SipTcpServer-6]::removeOldClients deleting 3 of 6 SipClients"

"2026-08-21T18:34:31.283009Z":395387:KERNEL:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"OsTaskBase::requestShutdown on task 'SipClientTcp-589', transition RUNNING -> SHUTTING_DOWN"

"2026-08-21T18:34:31.283196Z":395396:SIP:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"SipClient[SipClientTcp-589]::~ called"

"2026-08-21T18:34:31.283206Z":395397:SIP:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"SipClient[SipClientTcp-589]::~ 0x7f079001ff80 socket 0x7f079000cac0 closing TCP socket"

"2026-08-21T18:34:31.283250Z":395398:KERNEL:INFO:***removed sip domain***::7f07c2b42640:sipxproxy:"OsConnectionSocket::~"

"2026-08-21T18:34:31.283275Z":395400:KERNEL:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"OsSocket::~ close socket 27"

"2026-08-21T18:34:31.283289Z":395402:KERNEL:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"OsServerTaskWaitable::~ closed 30 -> 29"

"2026-08-21T18:34:31.283302Z":395403:KERNEL:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"OsServerTask::~ 'SipClientTcp-589' TERMINATED"

"2026-08-21T18:34:31.283314Z":395404:KERNEL:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"OsTaskBase::requestShutdown on task 'SipClientTcp-590', transition RUNNING -> SHUTTING_DOWN"

"2026-08-21T18:34:31.283521Z":395409:SIP:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"SipClient[SipClientTcp-590]::~ called"

"2026-08-21T18:34:31.283532Z":395410:SIP:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"SipClient[SipClientTcp-590]::~ 0x7f0784020c30 socket 0x5565709f1180 closing TCP socket"

"2026-08-21T18:34:31.283538Z":395411:KERNEL:INFO:***removed sip domain***::7f07c2b42640:sipxproxy:"OsConnectionSocket::~"

"2026-08-21T18:34:31.283550Z":395412:KERNEL:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"OsSocket::~ close socket 11"

"2026-08-21T18:34:31.283560Z":395413:KERNEL:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"OsServerTaskWaitable::~ closed 33 -> 32"

"2026-08-21T18:34:31.283578Z":395414:KERNEL:DEBUG:***removed sip domain***::7f07c2b42640:sipxproxy:"OsServerTask::~ 'SipClientTcp-590' TERMINATED"

<sipXproxy-debug.log>

[OnRelay Support]

We have issued another update 11 with only another fix attempt for this issue. 

This update 11 fix changes the shutdown and destructor sequence for the SIP TCP client, where there was a clear difference vs the SIP UDP client that could cause potentially a crash with the former. 

The issue is still not reproducible in our environment, and we are crossing fingers this fix does the trick elsewhere. 

[J.T. D]

Thank you! I've updated to u11 and am monitoring.

[J.T. D]

Unfortunately same behavior.  You said internally you use TLS.  Would switching from TCP or UDP help? Or is this apparent issue transport protocol agnostic?

[OnRelay Support]

Thanks. We use TLS towards external SIP extensions, but internally between services all connections are UDP or TCP.  And the SipClientTcp destructor doesn’t cause any similar problems here.

So the difference is likely just related to timing on the different HW platforms, where yours is either slower or faster than what we use and test on. It is a typical C++ issue where there is too much going on and too many dependencies between different threads in the destructor logic, and if things get a bit out of sequence something may get accessed after it has been deleted or forever hang on an old lock. So it is likely timing dependent, which is also why you hit the landmine only after a few hours and we don't.

As mentioned we already fixed a couple of other similar issues with SipClientTCP that we were able to reproduce consistently, and it is just a matter of time before we get to this one as well. It certainly complicates things that we are not able to reproduce here, as if so it is very straightforward to resolve these type errors.

So we will just keep at it as it is an important issue.  We may need to add some more logging in a next upgrade to see if we can catch it that way, notwithstanding we hate adding such debug info on production releases.

[Support]

Forgot to mention, could you please share sipXproxy debug logs again directly following your last occurrence of removeOldTransactions, and also double check that this mechanism is actually blocked?

Just to ensure we are looking at the right thing and verify the update 10 changes there. 

On Aug 26, 2026, at 2:33 PM, Support <sup...@onrelay.net> wrote:

Thanks. We use TLS towards external SIP extensions, but internally between services all connections are UDP or TCP.  And the SipClientTcp destructor doesn’t cause any similar problems here.

So the difference is likely just related to timing on the different HW platforms, where yours is either slower or faster than what we use and test on. It is a typical C++ issue where there is too much going on and too many dependencies between different threads in the destructor logic, and if things get a bit out of sequence something may get accessed after it has been deleted or forever hang on an old lock. So it is likely timing dependent, which is also why you hit the landmine only after a few hours and we don't.

As mentioned we already fixed a couple of other similar issues with SipClientTCP that we were able to reproduce consistently, and it is just a matter of time before we get to this one as well. It certainly complicates things that we are not able to reproduce here, as if so it is very straightforward to resolve these type errors.

So we will just keep at it as it is an important issue.  We may need to add some more logging in a next upgrade to see if we can catch it that way, notwithstanding we hate adding such debug info on production releases.

To view this discussion visit https://groups.google.com/d/msgid/sipxcom-users/78d80a14-2298-4f63-bdd9-3fc3b5a6625en%40googlegroups.com.

[J.T. D]

See attached.  Thank you!

[Support]

Thanks, we may have caught the root cause now. By splitting the SIP client shutdown from the SIP client destructor between two garbage collection iterations the error was consistently reproduced. And the SIP client isOK() check ironically had a missing null pointer check for its closed socket. So there was a race between the TCP socket shutting down and the isOK() check in the garbage collection handler, causing the garbage collection to break. I.e. one of those typical C++ landmines that can be hard to pin down but trivial to fix.

We will ship the fix in update 12 with perhaps a couple of other issues, and monitor this issue until confirmed OK.

<sipxproxy-debug.log>

[J.T. D]

Wonderful, thank you.  I greatly appreciate the efforts to track this down.

[OnRelay Support]

Update 12 is out with just another fix attempt for this issue.  

As mentioned above we hope a missing NULL pointer check in the SIP client isOK() method is / was the root cause, but continue to monitor issue as open until confirmed.

[J.T. D]

After over 48 hours of running on u12, no issues with the garbage cleanup process have been observed. Tailing sipXproxy.log for 'removeOldTransactions' streams log entries of deletions.  I'm thinking if no further crashes appear come Monday, this issue can be happily resolved!

[Support]

Glad to hear it has improved, but reproduced here now after approx. 5 days, so we will keep it open.

Next, for update 13 we are improving some weaknesses with ServerTaskWaitable, which is a common base class for the SIP TCP and TLS clients. Its socket file descriptors are not thread safe, but still accessed from multiple threads. The garbage collection thread stops immediately following its destructor, which also closes its sockets, and the associated race condition can explain these rare crashes.

So looks like a bit of whack-a-mole here, but it is too heavy lifting and risky to refactor the entire garbage collection synchronization logic, so we will just keep at these pinpoint conservative fixes until 100% stable. 

To view this discussion visit https://groups.google.com/d/msgid/sipxcom-users/b697901d-b049-4d94-b93c-bc2e277b0fddn%40googlegroups.com.

[J.T. D]

I've noticed an oddity (maybe?) related to this (or it's my lack of understanding of how proxy works.)  I've observed the garbage collectors stop processing (indicated by increasing active transaction counts) after awhile.  When I take a look at the sipXproxy log, I do see activity being recorded, but my phones lose registration. The active transaction count is around ~10,000.  I can kill sipxproxy and let the service/systemctl restart it (or do a full service/systemctl stop/start.) Once sipxproxy restarts, the phones then register.

If sipxproxy appears to be functioning based on logging, why do phones lose registration until a restart?

[Support]

It is the transaction count that goes above a congestion threshold that causes registrations to be actively rejected with a time out. We could just set the transaction threshold much higher as a workaround, but that would cause a memory leak. We want to ensure this pestering problem is fixed once and for all, we are still working on it at prio 1 and expect to update shortly. 

You received this message because you are subscribed to a topic in the Google Groups "sipxcom-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sipxcom-users/A824eQKDpiQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sipxcom-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sipxcom-users/1046acb5-6ef5-45fb-a80c-316abcd6b381n%40googlegroups.com.

[Support]

Update 13 is looking good for a few days now. So this issue is now considered closed unless / until it should recur. 

Two fixes were deployed to the garbage collection logic for update 13, (1) the root cause appeared to be some old style pointer arithmetic that could cause a heap corruption which was a landmine for a subsequent SIP client deletes, and (2) we strengthened the surrounding garbage collection controller where a new independent worker thread now is created and dispatched for each iteration to better tolerate such errors.