`ActionPlan_Daily_Toll_Free_Topup_Bulk` only processes ~12,000-12,500 out of 49,920 accounts, then stops silently with no errors.

28 views
Skip to first unread message

M B

unread,
Jan 25, 2026, 10:58:50 AM (8 days ago) Jan 25
to CGRateS
Environment:
- CGRates version: CGR...@v0.11.0~dev-20250526124343-ccdf3ef1f10d
- Redis: 10,000 max clients, 20 connection pool
- Action plan: 49,920 accounts, runs daily at 09:06:11

What we've tried:
1. System optimizations:
   - Disabled transparent huge pages
   - Set `vm.overcommit_memory=1`
   - Set `vm.swappiness=1`
   - Disabled Redis RDB snapshots (`save ""`)

2. Investigated Redis performance:
   - No connection exhaustion (20/10,000 used)
   - No errors in Redis logs
   - No memory issues
   - No swap usage

3. Investigated scheduler behavior:
   - Found TWO action plans appearing in scheduler queue together
   - `ActionPlan_Daily_Toll_Free_Topup` (1 account, scheduled 06:48)
   - `ActionPlan_Daily_Toll_Free_Topup_Bulk` (49,920 accounts, scheduled 09:06)
   - Both actions fire at BOTH times (06:48 and 09:06)
   - Processing stops at 09:08:17 (2 minutes after start)
   - Scheduler queue length stays at "2" constantly

4. Key findings:
   - Processing rate: ~6,000 accounts/minute when running
   - Stops after exactly 2 minutes of execution
   - No timeout/limit settings found in configuration
   - Manual execution (`scheduler_execute`) also stops at ~11,750-12,494 accounts

**Logs show:**
```
09:06:11 - Action: Action_Daily_Toll_Free_Topup_Bulk (starts)
09:06:11 - Scheduler queue length: 2
09:06:11 - Action: Action_Daily_Toll_Free_Topup (appears immediately after)
09:08:17 - Last account processed (silent stop)

M B

unread,
Jan 26, 2026, 2:42:30 AM (7 days ago) Jan 26
to CGRateS
Quick guess:
It takes 2-3 minutes for the whole process to run. So we start at 09:16 and end at 09:18 for example. So there's an account that has 2 minutes remaining when we start the next daye at 09:16 and the scheduler does not update it. Is that a possible explanation? How do we fix? This is the Action_Plan:
{ "Id": "ActionPlan_Daily_Toll_Free_Topup_Bulk", "AccountIDs": { ... 49,920 accounts ... }, "ActionTimings": [{ "Timing": { "ID": "*daily", "StartTime": "09:06:11" }, "ActionsID": "Action_Daily_Toll_Free_Topup_Bulk" }] }

Armir Veliaj

unread,
Jan 30, 2026, 9:51:01 AM (3 days ago) Jan 30
to CGRateS
Hi,

Working with 49,920 accounts should not be a problem, as we have cases where millions of accounts are processed without any issues.
However, please open the issue on GitHub and attach all your configuration files there, together with the APIs you used and some sample accounts, just as you did in your scenario, so we can try to replicate your issue.

Thanks,
Armir

Reply all
Reply to author
Forward
0 new messages