Cluster load exceed

[HeinHtet Aung]

I got this log multiple times in my tinode server, I am running with cluster mode with 3 nodes. Is it because of the wrong configuration?  I saw buffer size limit is 64 in code. 

proxy topic[p2pOOZ1n2-Ohy4_p3pei3WJWQ]: route join request from proxy to master failed - cluster: load exceeded

[Gene]

I've never seen this before, even under heavy load. Can you show the logs?
Is this an unmodified code?

[HeinHtet Aung]

I do some updates in ios notification payload part, others will be the same code base. I notice that " cluster: load exceeded" logs happen in the master node. I am using tinode version 0.23.2.

Here is some the log, 

I2025/10/08 01:44:33 in: '{"login":{"id":"129965","scheme":"token","secret":"waLssYXyZo05k\/NoFAABAAEA7Fi+FZRrQRkM0GbuxgRRKgrQFJQquGrHyGpI\/MroLvQ="}}' sid='zxBSIDc2wFU' uid=''
I2025/10/08 01:44:33 cluster: multiplexing session started fnd2CnTbg6DWPU-tinode-1 229
I2025/10/08 01:44:33 in: '{"sub":{"get":{"desc":{"ims":"2025-05-17T01:49:27.238Z"},"sub":{"ims":"2025-10-03T13:29:11.951Z"},"what":"desc sub tags cred"},"id":"129966","topic":"me"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:33 in: '{"sub":{"get":{"desc":{"ims":"2025-05-17T01:49:27.238Z"},"sub":{"ims":"2025-10-03T13:29:11.951Z"},"what":"desc sub tags cred"},"id":"129967","topic":"me"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
W2025/10/08 01:44:33 proxy topic[p2pOOZ1n2-Ohy49xpBk4bawYg]: session WuvDj9Hr6Uo not found; already terminated?
I2025/10/08 01:44:33 in: '{"sub":{"id":"129968","topic":"fnd"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:33 media serve: completed with status 307 uid= 2CnTbg6DWPU
I2025/10/08 01:44:34 in: '{"set":{"id":"129969","topic":"fnd","desc":{"public":"channel"}}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:34 in: '{"get":{"id":"129970","topic":"fnd","what":"sub"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:34 in: '{"set":{"id":"129971","topic":"fnd","desc":{"public":"default"}}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:34 in: '{"get":{"id":"129972","topic":"fnd","what":"sub"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:34 in: '{"set":{"id":"129973","topic":"fnd","desc":{"public":"bot"}}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:34 in: '{"get":{"id":"129974","topic":"fnd","what":"sub"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:34 in: '{"sub":{"get":{"data":{"limit":24,"since":0},"desc":{"ims":"2025-06-04T09:37:59.893Z"},"sub":{"ims":"2025-05-17T01:49:28.782Z"},"what":"desc sub data del"},"id":"129976","topic":"usrOOZ1n2-Ohy4"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
W2025/10/08 01:44:34 proxy topic[p2pOOZ1n2-Ohy7BouyxhfJmjQ]: route join request from proxy to master failed - cluster: load exceeded
I2025/10/08 01:44:34 in: '{"sub":{"get":{"data":{"limit":24,"since":0},"desc":{"ims":"2025-06-04T09:38:26.938Z"},"sub":{"ims":"2025-05-17T01:49:28.772Z"},"what":"desc sub data del"},"id":"129977","topic":"usrIMLZMerJoQY"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:34 in: '{"sub":{"get":{"data":{"limit":24,"since":0},"del":{"since":2},"desc":{"ims":"2025-10-03T13:29:11.951Z"},"sub":{"ims":"2025-10-03T13:29:11.951Z"},"what":"desc sub data del"},"id":"129975","topic":"chnnXJ_23nlRbA"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
W2025/10/08 01:44:34 proxy topic[p2pIMLZMerJoQbBouyxhfJmjQ]: route join request from proxy to master failed - cluster: load exceeded
I2025/10/08 01:44:34 in: '{"sub":{"get":{"data":{"limit":24,"since":0},"desc":{"ims":"2025-10-01T06:32:01.634Z"},"sub":{"ims":"2025-09-29T13:32:31.336Z"},"what":"desc sub data del"},"id":"129978","topic":"usrd8W5rhGOVRs"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
W2025/10/08 01:44:34 proxy topic[grpnXJ_23nlRbA]: route join request from proxy to master failed - cluster: load exceeded
W2025/10/08 01:44:34 proxy topic[p2pd8W5rhGOVRvBouyxhfJmjQ]: route join request from proxy to master failed - cluster: load exceeded
I2025/10/08 01:44:34 in: '{"set":{"id":"129979","topic":"fnd","desc":{"public":"channel"}}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:34 in: '{"get":{"id":"129980","topic":"fnd","what":"sub"}}' sid='zxBSIDc2wFU' uid='waLssYXyZo0'
I2025/10/08 01:44:34 ws: session started tyNnO_Zn3xU 13.212.61.160 229
I2025/10/08 01:44:34 in: '{"hi":{"id":"109285","ver":"0.22","ua":"lugo (Android 5.10.234-225.895.amzn2.x86_64; en_US); tindroid/"}}' sid='tyNnO_Zn3xU' uid=''
I2025/10/08 01:44:34 in: '{"login":{"id":"109286","scheme":"basic","secret":"OHR4eGc3bTI6N21xeGZhbDB1bnky"}}' sid='tyNnO_Zn3xU' uid=''
I2025/10/08 01:44:34 cluster: multiplexing session started p2p7Cz_28_VwQk45nWfb46HLg-tinode-2 230
I2025/10/08 01:44:34 cluster: multiplexing session started fnd7Cz_28_VwQk-tinode-2 230
I2025/10/08 01:44:34 cluster: session proxy closed p2pOOZ1n2-Ohy7EFTeF0-Y5gA-tinode-2
I2025/10/08 01:44:34 cluster: session proxy closed p2pIMLZMerJoQbEFTeF0-Y5gA-tinode-1
I2025/10/08 01:44:34 ws: session started dr18wbh3l6M 113.211.115.88 229
I2025/10/08 01:44:34 cluster: session proxy closed p2pKhM39CkCNyA45nWfb46HLg-tinode-2
I2025/10/08 01:44:35 cluster: multiplexing session started p2pd8W5rhGOVRsgGVTaPkfOdA-tinode-2 229
I2025/10/08 01:44:35 cluster: multiplexing session started usrH99J2XEqEjU-tinode-2 230
I2025/10/08 01:44:35 media serve: completed with status 307 uid= 7Cz_28_VwQk
I2025/10/08 01:44:35 cluster: multiplexing session started fndH99J2XEqEjU-tinode-2 231
I2025/10/08 01:44:35 cluster: multiplexing session started p2pd8W5rhGOVRsf30nZcSoSNQ-tinode-2 232
I2025/10/08 01:44:35 ws: session started eaEyptWpldI 182.62.205.16 233
I2025/10/08 01:44:35 media serve: completed with status 307 uid= cl6pLlMtp7Y
I2025/10/08 01:44:35 media serve: completed with status 307 uid= cl6pLlMtp7Y
I2025/10/08 01:44:35 media serve: completed with status 307 uid= cl6pLlMtp7Y
I2025/10/08 01:44:35 media serve: completed with status 307 uid= H99J2XEqEjU
I2025/10/08 01:44:35 media serve: completed with status 307 uid= H99J2XEqEjU
I2025/10/08 01:44:35 media serve: completed with status 307 uid= H99J2XEqEjU
I2025/10/08 01:44:36 cluster: session proxy closed p2pIMLZMerJoQaPY8flqE4hiA-tinode-1
I2025/10/08 01:44:36 in: '{"sub":{"get":{"desc":{"ims":"2025-05-28T01:44:38.243Z"},"sub":{"ims":"2025-10-03T11:12:40.849Z"},"what":"desc sub tags cred"},"id":"78409","topic":"me"}}' sid='goakZRKLC9M' uid='ab6W0xzZsjg'
W2025/10/08 01:44:36 proxy topic[usrab6W0xzZsjg]: route join request from proxy to master failed - cluster: load exceeded
I2025/10/08 01:44:36 ws: session started Fqt8v8EmLj0 113.211.131.134 233
I2025/10/08 01:44:36 cluster: session proxy closed p2pRLPB96-tGwE45nWfb46HLg-tinode-1
I2025/10/08 01:44:36 cluster: multiplexing session started usrdW1bBW-DIKA-tinode-2 233
I2025/10/08 01:44:36 cluster: multiplexing session started p2pIMLZMerJoQZ1bVsFb4MgoA-tinode-1 234
I2025/10/08 01:44:36 cluster: multiplexing session started p2pOOZ1n2-Ohy58bqBAcCCrPg-tinode-1 235
I2025/10/08 01:44:36 cluster: multiplexing session started usrOlVmRZtHUkE-tinode-2 236
I2025/10/08 01:44:36 ws: session started OzoxGbPnASg 115.164.212.41 237
I2025/10/08 01:44:36 in: '{"hi":{"dev":"cSV_AuwaSUa61bzN06KkqY:APA91bHAVej3o3N4T_OMlvmmwoqBJbR5lFIPMkjaKFDzzMpwDdtB0yO-7jiXVqJ_4IN-JAtoshUYpzuJEUA8-FnKg0Iva5ITPI3T-85IQzOpoIqPC9wqChg","id":"95482","lang":"en_MY","platf":"android","ua":"1.2.4 (Android 15; en_MY); tindroid\/1.0","ver":"0.22"}}' sid='OzoxGbPnASg' uid=''
I2025/10/08 01:44:36 in: '{"login":{"id":"95483","scheme":"token","secret":"0f8UCG7MvuvTNvhoFAABAAEACTn1O2XnH4n+go9jYZGPXGIRXXMqnphg0BrB1E4YGsw="}}' sid='OzoxGbPnASg' uid=''
I2025/10/08 01:44:36 cluster: session proxy closed p2pIMLZMerJoQZR-0D51a_pFg-tinode-1
I2025/10/08 01:44:37 in: '{"sub":{"get":{"desc":{"ims":"2025-09-30T07:04:07.826Z"},"sub":{"ims":"2025-10-03T17:53:12.195Z"},"what":"desc sub tags cred"},"id":"95484","topic":"me"}}' sid='OzoxGbPnASg' uid='0f8UCG7Mvus'
I2025/10/08 01:44:37 cluster: multiplexing session started p2pIMLZMerJoQY6VWZFm0dSQQ-tinode-1 237
I2025/10/08 01:44:37 cluster: multiplexing session started p2pOOZ1n2-Ohy46VWZFm0dSQQ-tinode-2 238
I2025/10/08 01:44:37 cluster: session proxy closed p2pOOZ1n2-Ohy4NanQhDbeXyA-tinode-2
I2025/10/08 01:44:37 cluster: multiplexing session started p2pOOZ1n2-Ohy7R_xQIbsy-6w-tinode-2 238
I2025/10/08 01:44:37 cluster: multiplexing session started p2pIMLZMerJoQbR_xQIbsy-6w-tinode-1 239

[Gene]

On Wednesday, October 8, 2025 at 5:45:26 AM UTC+3 HeinHtet Aung wrote:

I do some updates in ios notification payload part, others will be the same code base. I notice that " cluster: load exceeded" logs happen in the master node. I am using tinode version 0.23.2.

No, they are happening in the proxy node for that topic. Apparently the calls at https://github.com/tinode/chat/blob/6f18ec8814e4c9a3316e6bfd8b4c05757352f235/server/cluster.go#L385 are taking a long time to complete. There are no errors in the log to indicate that the calls are failing though. That means your master node for that topic is slow to respond or maybe not responding at all (more likely). Take a look at the logs there. Or you may have networking problems. I suspect one of the nodes in the cluster is not responding/locked up.

 

Take a look at the master node log for that topic. And in general look at the logs in all nodes.