M B
unread,Jan 25, 2026, 10:58:50 AM (8 days ago) Jan 25Sign in to reply to author
Sign in to forward
You do not have permission to delete messages in this group
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to CGRateS
Environment:
- CGRates version: CGR...@v0.11.0~dev-20250526124343-ccdf3ef1f10d
- Redis: 10,000 max clients, 20 connection pool
- Action plan: 49,920 accounts, runs daily at 09:06:11
What we've tried:
1. System optimizations:
- Disabled transparent huge pages
- Set `vm.overcommit_memory=1`
- Set `vm.swappiness=1`
- Disabled Redis RDB snapshots (`save ""`)
2. Investigated Redis performance:
- No connection exhaustion (20/10,000 used)
- No errors in Redis logs
- No memory issues
- No swap usage
3. Investigated scheduler behavior:
- Found TWO action plans appearing in scheduler queue together
- `ActionPlan_Daily_Toll_Free_Topup` (1 account, scheduled 06:48)
- `ActionPlan_Daily_Toll_Free_Topup_Bulk` (49,920 accounts, scheduled 09:06)
- Both actions fire at BOTH times (06:48 and 09:06)
- Processing stops at 09:08:17 (2 minutes after start)
- Scheduler queue length stays at "2" constantly
4. Key findings:
- Processing rate: ~6,000 accounts/minute when running
- Stops after exactly 2 minutes of execution
- No timeout/limit settings found in configuration
- Manual execution (`scheduler_execute`) also stops at ~11,750-12,494 accounts
**Logs show:**
```
09:06:11 - Action: Action_Daily_Toll_Free_Topup_Bulk (starts)
09:06:11 - Scheduler queue length: 2
09:06:11 - Action: Action_Daily_Toll_Free_Topup (appears immediately after)
09:08:17 - Last account processed (silent stop)