stream-perf-tool - rate limiter latency problem
120 views
Skip to first unread message
Philipp Chlebicki
Jun 12, 2023, 7:15:56 AM6/12/23
to rabbitmq-users
Hey,
Once I limit the rate to ~100k with --rate 100000 the latency increases constantly to multiple seconds (the consumed messages are much lower than with an unlimited rate test).
Am I doing something wrong or is this a bug of the tool / metric calculation?
The same is reported to me via stdout of stream-perf-tool:
Monitoring endpoints started on http://localhost:8080
Starting producer
1, published 115540 msg/s, confirmed 113997 msg/s, consumed 88732 msg/s, latency median/75th/95th/99th 174/191/250/258 ms, chunk size 117
2, published 99822 msg/s, confirmed 101233 msg/s, consumed 48435 msg/s, latency median/75th/95th/99th 503/654/755/789 ms, chunk size 80
3, published 100089 msg/s, confirmed 99948 msg/s, consumed 62863 msg/s, latency median/75th/95th/99th 1007/1040/1107/1174 ms, chunk size 70
4, published 99975 msg/s, confirmed 100075 msg/s, consumed 59269 msg/s, latency median/75th/95th/99th 1342/1409/1476/1544 ms, chunk size 64
5, published 100154 msg/s, confirmed 100058 msg/s, consumed 58844 msg/s, latency median/75th/95th/99th 1678/1812/1946/1946 ms, chunk size 60
6, published 100100 msg/s, confirmed 100113 msg/s, consumed 54144 msg/s, latency median/75th/95th/99th 2215/2349/2483/2483 ms, chunk size 58
7, published 100044 msg/s, confirmed 100042 msg/s, consumed 57335 msg/s, latency median/75th/95th/99th 2684/2684/2819/2819 ms, chunk size 57
8, published 100122 msg/s, confirmed 100011 msg/s, consumed 57509 msg/s, latency median/75th/95th/99th 3087/3087/3221/3221 ms, chunk size 56
9, published 100037 msg/s, confirmed 99938 msg/s, consumed 54945 msg/s, latency median/75th/95th/99th 3490/3624/3624/3624 ms, chunk size 55
10, published 99989 msg/s, confirmed 100190 msg/s, consumed 61432 msg/s, latency median/75th/95th/99th 3892/4027/4027/4027 ms, chunk size 55
11, published 100042 msg/s, confirmed 99863 msg/s, consumed 63343 msg/s, latency median/75th/95th/99th 4429/4429/4429/4429 ms, chunk size 54
12, published 100014 msg/s, confirmed 100293 msg/s, consumed 56204 msg/s, latency median/75th/95th/99th 4563/4563/4832/4832 ms, chunk size 54
13, published 100094 msg/s, confirmed 99726 msg/s, consumed 44477 msg/s, latency median/75th/95th/99th 5100/5100/5369/5369 ms, chunk size 54
14, published 99999 msg/s, confirmed 100267 msg/s, consumed 48538 msg/s, latency median/75th/95th/99th 5637/5637/5906/5906 ms, chunk size 53
15, published 100075 msg/s, confirmed 100125 msg/s, consumed 53887 msg/s, latency median/75th/95th/99th 6174/6174/6174/6442 ms, chunk size 53
16, published 99924 msg/s, confirmed 99792 msg/s, consumed 45812 msg/s, latency median/75th/95th/99th 6711/6711/6711/6979 ms, chunk size 53
...
182, published 100009 msg/s, confirmed 99940 msg/s, consumed 53115 msg/s, latency median/75th/95th/99th 66572/66572/66572/66572 ms, chunk size 56
183, published 100026 msg/s, confirmed 100130 msg/s, consumed 58306 msg/s, latency median/75th/95th/99th 66572/66572/66572/66572 ms, chunk size 56
184, published 100126 msg/s, confirmed 100158 msg/s, consumed 59981 msg/s, latency median/75th/95th/99th 66572/66572/66572/66572 ms, chunk size 56
185, published 100096 msg/s, confirmed 100164 msg/s, consumed 55052 msg/s, latency median/75th/95th/99th 66572/66572/66572/66572 ms, chunk size 56
186, published 99917 msg/s, confirmed 99741 msg/s, consumed 53589 msg/s, latency median/75th/95th/99th 66572/66572/66572/66572 ms, chunk size 56
I am currently trying to evaluate on whether RabbitMQ could be a replacement for our current message broking solution and try to simulate the expected load of our system with the stream-perf-tool. Without the rate parameter and the simulation of multiple producers / consumer it will be quite hard to do so.
Thanks and BR,
Philipp
what is the difference in the stream-perf-tool when adding a rate limiter? I see a massive decrease in performance when using it.
When triggering stream-perf-tool with the following parameters (--streams stream-queue --delete-streams --prometheus --max-length-bytes 1000000000), I have a throughput of ~ 450k messages / second with an e2e latency of ~56ms (mean - 0.99 percentile).
Once I limit the rate to ~100k with --rate 100000 the latency increases constantly to multiple seconds (the consumed messages are much lower than with an unlimited rate test).
Am I doing something wrong or is this a bug of the tool / metric calculation?
The same is reported to me via stdout of stream-perf-tool:
Monitoring endpoints started on http://localhost:8080
Starting producer
1, published 115540 msg/s, confirmed 113997 msg/s, consumed 88732 msg/s, latency median/75th/95th/99th 174/191/250/258 ms, chunk size 117
2, published 99822 msg/s, confirmed 101233 msg/s, consumed 48435 msg/s, latency median/75th/95th/99th 503/654/755/789 ms, chunk size 80
3, published 100089 msg/s, confirmed 99948 msg/s, consumed 62863 msg/s, latency median/75th/95th/99th 1007/1040/1107/1174 ms, chunk size 70
4, published 99975 msg/s, confirmed 100075 msg/s, consumed 59269 msg/s, latency median/75th/95th/99th 1342/1409/1476/1544 ms, chunk size 64
5, published 100154 msg/s, confirmed 100058 msg/s, consumed 58844 msg/s, latency median/75th/95th/99th 1678/1812/1946/1946 ms, chunk size 60
6, published 100100 msg/s, confirmed 100113 msg/s, consumed 54144 msg/s, latency median/75th/95th/99th 2215/2349/2483/2483 ms, chunk size 58
7, published 100044 msg/s, confirmed 100042 msg/s, consumed 57335 msg/s, latency median/75th/95th/99th 2684/2684/2819/2819 ms, chunk size 57
8, published 100122 msg/s, confirmed 100011 msg/s, consumed 57509 msg/s, latency median/75th/95th/99th 3087/3087/3221/3221 ms, chunk size 56
9, published 100037 msg/s, confirmed 99938 msg/s, consumed 54945 msg/s, latency median/75th/95th/99th 3490/3624/3624/3624 ms, chunk size 55
10, published 99989 msg/s, confirmed 100190 msg/s, consumed 61432 msg/s, latency median/75th/95th/99th 3892/4027/4027/4027 ms, chunk size 55
11, published 100042 msg/s, confirmed 99863 msg/s, consumed 63343 msg/s, latency median/75th/95th/99th 4429/4429/4429/4429 ms, chunk size 54
12, published 100014 msg/s, confirmed 100293 msg/s, consumed 56204 msg/s, latency median/75th/95th/99th 4563/4563/4832/4832 ms, chunk size 54
13, published 100094 msg/s, confirmed 99726 msg/s, consumed 44477 msg/s, latency median/75th/95th/99th 5100/5100/5369/5369 ms, chunk size 54
14, published 99999 msg/s, confirmed 100267 msg/s, consumed 48538 msg/s, latency median/75th/95th/99th 5637/5637/5906/5906 ms, chunk size 53
15, published 100075 msg/s, confirmed 100125 msg/s, consumed 53887 msg/s, latency median/75th/95th/99th 6174/6174/6174/6442 ms, chunk size 53
16, published 99924 msg/s, confirmed 99792 msg/s, consumed 45812 msg/s, latency median/75th/95th/99th 6711/6711/6711/6979 ms, chunk size 53
...
182, published 100009 msg/s, confirmed 99940 msg/s, consumed 53115 msg/s, latency median/75th/95th/99th 66572/66572/66572/66572 ms, chunk size 56
183, published 100026 msg/s, confirmed 100130 msg/s, consumed 58306 msg/s, latency median/75th/95th/99th 66572/66572/66572/66572 ms, chunk size 56
184, published 100126 msg/s, confirmed 100158 msg/s, consumed 59981 msg/s, latency median/75th/95th/99th 66572/66572/66572/66572 ms, chunk size 56
185, published 100096 msg/s, confirmed 100164 msg/s, consumed 55052 msg/s, latency median/75th/95th/99th 66572/66572/66572/66572 ms, chunk size 56
186, published 99917 msg/s, confirmed 99741 msg/s, consumed 53589 msg/s, latency median/75th/95th/99th 66572/66572/66572/66572 ms, chunk size 56
I am currently trying to evaluate on whether RabbitMQ could be a replacement for our current message broking solution and try to simulate the expected load of our system with the stream-perf-tool. Without the rate parameter and the simulation of multiple producers / consumer it will be quite hard to do so.
Thanks and BR,
Philipp
Michal Kuratczyk
Jun 12, 2023, 8:23:34 AM6/12/23
to rabbitm...@googlegroups.com
I don't see such behaviour. I've tried a few combinations with tiny messages and 1kb message, single node and cluster, everything local and on Kubernetes (with stream-perf-test on a separate node). In all cases, --rate 100000 behaved as expected. For example, single-node, everything local:
> java -jar stream-perf-test-latest.jar --max-length-bytes 1000000000 -z 10
Starting producer
1, published 871210 msg/s, confirmed 861348 msg/s, consumed 857301 msg/s, latency median/75th/95th/99th 10/14/50/75 ms, chunk size 1499
2, published 913132 msg/s, confirmed 915558 msg/s, consumed 919658 msg/s, latency median/75th/95th/99th 10/15/24/29 ms, chunk size 1874
3, published 1016344 msg/s, confirmed 1017901 msg/s, consumed 1007959 msg/s, latency median/75th/95th/99th 9/13/23/30 ms, chunk size 1893
4, published 1215694 msg/s, confirmed 1218995 msg/s, consumed 1217315 msg/s, latency median/75th/95th/99th 9/11/14/18 ms, chunk size 2070
5, published 961930 msg/s, confirmed 960976 msg/s, consumed 967091 msg/s, latency median/75th/95th/99th 9/14/24/27 ms, chunk size 2010
6, published 850836 msg/s, confirmed 844584 msg/s, consumed 845992 msg/s, latency median/75th/95th/99th 11/16/23/28 ms, chunk size 2027
7, published 864251 msg/s, confirmed 869032 msg/s, consumed 871082 msg/s, latency median/75th/95th/99th 11/16/24/32 ms, chunk size 2066
8, published 789121 msg/s, confirmed 787530 msg/s, consumed 787530 msg/s, latency median/75th/95th/99th 14/21/25/30 ms, chunk size 2047
9, published 1116097 msg/s, confirmed 1116399 msg/s, consumed 1114734 msg/s, latency median/75th/95th/99th 9/12/19/25 ms, chunk size 2081
10, published 923390 msg/s, confirmed 922524 msg/s, consumed 926239 msg/s, latency median/75th/95th/99th 11/16/25/27 ms, chunk size 2067
Summary: published 951918 msg/s, confirmed 950921 msg/s, consumed 950512 msg/s, latency 95th 25 ms, chunk size 2067
> java -jar stream-perf-test-latest.jar --max-length-bytes 1000000000 -z 10 --rate 100000
Starting producer
1, published 107771 msg/s, confirmed 107672 msg/s, consumed 107474 msg/s, latency median/75th/95th/99th 2/5/31/32 ms, chunk size 72
2, published 99931 msg/s, confirmed 99931 msg/s, consumed 99995 msg/s, latency median/75th/95th/99th 2/3/3/4 ms, chunk size 68
3, published 100039 msg/s, confirmed 100139 msg/s, consumed 100075 msg/s, latency median/75th/95th/99th 2/3/4/4 ms, chunk size 68
4, published 100008 msg/s, confirmed 99872 msg/s, consumed 100008 msg/s, latency median/75th/95th/99th 2/3/4/5 ms, chunk size 69
5, published 100129 msg/s, confirmed 100165 msg/s, consumed 100229 msg/s, latency median/75th/95th/99th 2/3/3/4 ms, chunk size 68
6, published 100090 msg/s, confirmed 99990 msg/s, consumed 99890 msg/s, latency median/75th/95th/99th 2/3/3/4 ms, chunk size 68
7, published 100000 msg/s, confirmed 100064 msg/s, consumed 100100 msg/s, latency median/75th/95th/99th 2/3/4/4 ms, chunk size 68
8, published 100002 msg/s, confirmed 100038 msg/s, consumed 100066 msg/s, latency median/75th/95th/99th 2/3/3/5 ms, chunk size 69
9, published 99929 msg/s, confirmed 100029 msg/s, consumed 100065 msg/s, latency median/75th/95th/99th 2/3/4/4 ms, chunk size 69
10, published 100140 msg/s, confirmed 100140 msg/s, consumed 100040 msg/s, latency median/75th/95th/99th 2/3/4/4 ms, chunk size 68
Summary: published 100789 msg/s, confirmed 100779 msg/s, consumed 100759 msg/s, latency 95th 4 ms, chunk size 68
Starting producer
1, published 871210 msg/s, confirmed 861348 msg/s, consumed 857301 msg/s, latency median/75th/95th/99th 10/14/50/75 ms, chunk size 1499
2, published 913132 msg/s, confirmed 915558 msg/s, consumed 919658 msg/s, latency median/75th/95th/99th 10/15/24/29 ms, chunk size 1874
3, published 1016344 msg/s, confirmed 1017901 msg/s, consumed 1007959 msg/s, latency median/75th/95th/99th 9/13/23/30 ms, chunk size 1893
4, published 1215694 msg/s, confirmed 1218995 msg/s, consumed 1217315 msg/s, latency median/75th/95th/99th 9/11/14/18 ms, chunk size 2070
5, published 961930 msg/s, confirmed 960976 msg/s, consumed 967091 msg/s, latency median/75th/95th/99th 9/14/24/27 ms, chunk size 2010
6, published 850836 msg/s, confirmed 844584 msg/s, consumed 845992 msg/s, latency median/75th/95th/99th 11/16/23/28 ms, chunk size 2027
7, published 864251 msg/s, confirmed 869032 msg/s, consumed 871082 msg/s, latency median/75th/95th/99th 11/16/24/32 ms, chunk size 2066
8, published 789121 msg/s, confirmed 787530 msg/s, consumed 787530 msg/s, latency median/75th/95th/99th 14/21/25/30 ms, chunk size 2047
9, published 1116097 msg/s, confirmed 1116399 msg/s, consumed 1114734 msg/s, latency median/75th/95th/99th 9/12/19/25 ms, chunk size 2081
10, published 923390 msg/s, confirmed 922524 msg/s, consumed 926239 msg/s, latency median/75th/95th/99th 11/16/25/27 ms, chunk size 2067
Summary: published 951918 msg/s, confirmed 950921 msg/s, consumed 950512 msg/s, latency 95th 25 ms, chunk size 2067
> java -jar stream-perf-test-latest.jar --max-length-bytes 1000000000 -z 10 --rate 100000
Starting producer
1, published 107771 msg/s, confirmed 107672 msg/s, consumed 107474 msg/s, latency median/75th/95th/99th 2/5/31/32 ms, chunk size 72
2, published 99931 msg/s, confirmed 99931 msg/s, consumed 99995 msg/s, latency median/75th/95th/99th 2/3/3/4 ms, chunk size 68
3, published 100039 msg/s, confirmed 100139 msg/s, consumed 100075 msg/s, latency median/75th/95th/99th 2/3/4/4 ms, chunk size 68
4, published 100008 msg/s, confirmed 99872 msg/s, consumed 100008 msg/s, latency median/75th/95th/99th 2/3/4/5 ms, chunk size 69
5, published 100129 msg/s, confirmed 100165 msg/s, consumed 100229 msg/s, latency median/75th/95th/99th 2/3/3/4 ms, chunk size 68
6, published 100090 msg/s, confirmed 99990 msg/s, consumed 99890 msg/s, latency median/75th/95th/99th 2/3/3/4 ms, chunk size 68
7, published 100000 msg/s, confirmed 100064 msg/s, consumed 100100 msg/s, latency median/75th/95th/99th 2/3/4/4 ms, chunk size 68
8, published 100002 msg/s, confirmed 100038 msg/s, consumed 100066 msg/s, latency median/75th/95th/99th 2/3/3/5 ms, chunk size 69
9, published 99929 msg/s, confirmed 100029 msg/s, consumed 100065 msg/s, latency median/75th/95th/99th 2/3/4/4 ms, chunk size 69
10, published 100140 msg/s, confirmed 100140 msg/s, consumed 100040 msg/s, latency median/75th/95th/99th 2/3/4/4 ms, chunk size 68
Summary: published 100789 msg/s, confirmed 100779 msg/s, consumed 100759 msg/s, latency 95th 4 ms, chunk size 68
```
There is a significant difference in the chunk size and small chunks may lead to poor performance (https://github.com/rabbitmq/osiris/pull/115 addressed this to some extent but there's only so much we can do when the chunks are very small).
Given I can't reproduce this, please provide full details of how you run these tests. It might be easier through github.
Best,
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/27edad4d-f41d-47d6-93ba-17b18e4afd8fn%40googlegroups.com.
Michał
RabbitMQ teamPhilipp Chlebicki
Jun 13, 2023, 1:35:17 AM6/13/23
to rabbitmq-users
Cześć Michał,
here are the details:
here are the details:
- RabbitMQ Cluster Operator Helm Chart: 3.4.1
- RabbitMQ 3.11.16
- Erlang 25.3.2
- RabbitMQ Cluster (running on Kubernetes clusteR):
- "Production Ready" as defined on https://github.com/rabbitmq/cluster-operator/blob/main/docs/examples/production-ready/rabbitmq.yaml
- Added rabbitmq_stream plugin
- Changed persistence.storage to 0 to disable persistent volume creation (as I saw some performance hickups / latency 1-2 times per minute > need to investigate further later).
- I see the same problematic behaviour when having enabled the persistency storage
- "Production Ready" as defined on https://github.com/rabbitmq/cluster-operator/blob/main/docs/examples/production-ready/rabbitmq.yaml
- Stream Perf Test Client: 0.10.0 / latest (both have the same behaviour)
- Behaviour is the same for access from inside (without --loadbalancer) AND outside cluster (with --loadbalancer)
I simplified the stream perf test client call a bit:
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-latest.jar --uris rabbitmq-stream://<user>:<pass>@<ip>:5552 --load-balancer --delete-streams -z 10 --rate 100000
Starting producer
1, published 178735 msg/s, confirmed 178473 msg/s, consumed 145493 msg/s, latency median/75th/95th/99th 331/348/348/365 ms, chunk size 147
2, published 100007 msg/s, confirmed 99874 msg/s, consumed 56976 msg/s, latency median/75th/95th/99th 587/654/755/755 ms, chunk size 88
3, published 100120 msg/s, confirmed 100252 msg/s, consumed 58738 msg/s, latency median/75th/95th/99th 1007/1107/1174/1174 ms, chunk size 71
4, published 100033 msg/s, confirmed 100133 msg/s, consumed 63197 msg/s, latency median/75th/95th/99th 1409/1476/1544/1544 ms, chunk size 64
5, published 99965 msg/s, confirmed 99965 msg/s, consumed 58218 msg/s, latency median/75th/95th/99th 1812/1879/1946/1946 ms, chunk size 59
6, published 100006 msg/s, confirmed 99974 msg/s, consumed 52717 msg/s, latency median/75th/95th/99th 2349/2349/2483/2483 ms, chunk size 56
7, published 100040 msg/s, confirmed 100040 msg/s, consumed 50118 msg/s, latency median/75th/95th/99th 2684/2819/2819/2819 ms, chunk size 54
8, published 100145 msg/s, confirmed 100077 msg/s, consumed 60390 msg/s, latency median/75th/95th/99th 3087/3221/3221/3221 ms, chunk size 53
9, published 99945 msg/s, confirmed 100046 msg/s, consumed 62386 msg/s, latency median/75th/95th/99th 3490/3624/3624/3624 ms, chunk size 51
10, published 100144 msg/s, confirmed 100012 msg/s, consumed 67089 msg/s, latency median/75th/95th/99th 3892/3892/3892/4027 ms, chunk size 51
Summary: published 108029 msg/s, confirmed 108002 msg/s, consumed 67672 msg/s, latency 95th 4027 ms, chunk size 51
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-latest.jar --uris rabbitmq-stream:// <user>:<pass>@<ip> :5552 --load-balancer --delete-streams -z 10
Starting producer
1, published 393240 msg/s, confirmed 386377 msg/s, consumed 75833 msg/s, latency median/75th/95th/99th 518/568/568/568 ms, chunk size 113
2, published 628669 msg/s, confirmed 625715 msg/s, consumed 945973 msg/s, latency median/75th/95th/99th 285/385/486/520 ms, chunk size 627
3, published 692770 msg/s, confirmed 692743 msg/s, consumed 555839 msg/s, latency median/75th/95th/99th 192/192/226/234 ms, chunk size 749
4, published 670996 msg/s, confirmed 673364 msg/s, consumed 804622 msg/s, latency median/75th/95th/99th 31/143/210/226 ms, chunk size 842
5, published 679467 msg/s, confirmed 682267 msg/s, consumed 677661 msg/s, latency median/75th/95th/99th 16/19/31/32 ms, chunk size 893
6, published 645046 msg/s, confirmed 640210 msg/s, consumed 651705 msg/s, latency median/75th/95th/99th 16/20/32/32 ms, chunk size 920
7, published 669113 msg/s, confirmed 668754 msg/s, consumed 668243 msg/s, latency median/75th/95th/99th 16/31/36/48 ms, chunk size 938
8, published 668922 msg/s, confirmed 669921 msg/s, consumed 660231 msg/s, latency median/75th/95th/99th 16/18/31/38 ms, chunk size 960
9, published 616743 msg/s, confirmed 621738 msg/s, consumed 621275 msg/s, latency median/75th/95th/99th 16/30/32/32 ms, chunk size 970
10, published 630167 msg/s, confirmed 624161 msg/s, consumed 625939 msg/s, latency median/75th/95th/99th 16/20/32/32 ms, chunk size 977
Summary: published 628671 msg/s, confirmed 627676 msg/s, consumed 626991 msg/s, latency 95th 32 ms, chunk size 978
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-latest.jar --uris rabbitmq-stream://<user>:<pass>@<ip>:5552 --load-balancer --delete-streams -z 10 --rate 100000
Starting producer
1, published 178735 msg/s, confirmed 178473 msg/s, consumed 145493 msg/s, latency median/75th/95th/99th 331/348/348/365 ms, chunk size 147
2, published 100007 msg/s, confirmed 99874 msg/s, consumed 56976 msg/s, latency median/75th/95th/99th 587/654/755/755 ms, chunk size 88
3, published 100120 msg/s, confirmed 100252 msg/s, consumed 58738 msg/s, latency median/75th/95th/99th 1007/1107/1174/1174 ms, chunk size 71
4, published 100033 msg/s, confirmed 100133 msg/s, consumed 63197 msg/s, latency median/75th/95th/99th 1409/1476/1544/1544 ms, chunk size 64
5, published 99965 msg/s, confirmed 99965 msg/s, consumed 58218 msg/s, latency median/75th/95th/99th 1812/1879/1946/1946 ms, chunk size 59
6, published 100006 msg/s, confirmed 99974 msg/s, consumed 52717 msg/s, latency median/75th/95th/99th 2349/2349/2483/2483 ms, chunk size 56
7, published 100040 msg/s, confirmed 100040 msg/s, consumed 50118 msg/s, latency median/75th/95th/99th 2684/2819/2819/2819 ms, chunk size 54
8, published 100145 msg/s, confirmed 100077 msg/s, consumed 60390 msg/s, latency median/75th/95th/99th 3087/3221/3221/3221 ms, chunk size 53
9, published 99945 msg/s, confirmed 100046 msg/s, consumed 62386 msg/s, latency median/75th/95th/99th 3490/3624/3624/3624 ms, chunk size 51
10, published 100144 msg/s, confirmed 100012 msg/s, consumed 67089 msg/s, latency median/75th/95th/99th 3892/3892/3892/4027 ms, chunk size 51
Summary: published 108029 msg/s, confirmed 108002 msg/s, consumed 67672 msg/s, latency 95th 4027 ms, chunk size 51
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-latest.jar --uris rabbitmq-stream:// <user>:<pass>@<ip> :5552 --load-balancer --delete-streams -z 10
Starting producer
1, published 393240 msg/s, confirmed 386377 msg/s, consumed 75833 msg/s, latency median/75th/95th/99th 518/568/568/568 ms, chunk size 113
2, published 628669 msg/s, confirmed 625715 msg/s, consumed 945973 msg/s, latency median/75th/95th/99th 285/385/486/520 ms, chunk size 627
3, published 692770 msg/s, confirmed 692743 msg/s, consumed 555839 msg/s, latency median/75th/95th/99th 192/192/226/234 ms, chunk size 749
4, published 670996 msg/s, confirmed 673364 msg/s, consumed 804622 msg/s, latency median/75th/95th/99th 31/143/210/226 ms, chunk size 842
5, published 679467 msg/s, confirmed 682267 msg/s, consumed 677661 msg/s, latency median/75th/95th/99th 16/19/31/32 ms, chunk size 893
6, published 645046 msg/s, confirmed 640210 msg/s, consumed 651705 msg/s, latency median/75th/95th/99th 16/20/32/32 ms, chunk size 920
7, published 669113 msg/s, confirmed 668754 msg/s, consumed 668243 msg/s, latency median/75th/95th/99th 16/31/36/48 ms, chunk size 938
8, published 668922 msg/s, confirmed 669921 msg/s, consumed 660231 msg/s, latency median/75th/95th/99th 16/18/31/38 ms, chunk size 960
9, published 616743 msg/s, confirmed 621738 msg/s, consumed 621275 msg/s, latency median/75th/95th/99th 16/30/32/32 ms, chunk size 970
10, published 630167 msg/s, confirmed 624161 msg/s, consumed 625939 msg/s, latency median/75th/95th/99th 16/20/32/32 ms, chunk size 977
Summary: published 628671 msg/s, confirmed 627676 msg/s, consumed 626991 msg/s, latency 95th 32 ms, chunk size 978
This was a short test from outside the Kubernetes cluster (using --loadbalancer), but the same happens also when I run the tool in the container (without --loadbalancer) and directly access to the cluster.
Any tips are welcome here :(.
Greetings from Vienna,
Any tips are welcome here :(.
Greetings from Vienna,
Philipp
Philipp Chlebicki
Jun 13, 2023, 1:48:59 AM6/13/23
to rabbitmq-users
Hey.
me again. I did a few additional tests with a single node cluster (set the above cluster definition to replica 1) and have still the same results:
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-latest.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --delete-streams -z 10
Starting producer
1, published 430398 msg/s, confirmed 421144 msg/s, consumed 196918 msg/s, latency median/75th/95th/99th 284/318/351/351 ms, chunk size 276
2, published 774687 msg/s, confirmed 774191 msg/s, consumed 1001626 msg/s, latency median/75th/95th/99th 84/193/285/319 ms, chunk size 1016
3, published 835602 msg/s, confirmed 835602 msg/s, consumed 838674 msg/s, latency median/75th/95th/99th 15/16/21/32 ms, chunk size 1376
4, published 831931 msg/s, confirmed 834834 msg/s, consumed 828328 msg/s, latency median/75th/95th/99th 15/16/31/32 ms, chunk size 1616
5, published 805782 msg/s, confirmed 804417 msg/s, consumed 808914 msg/s, latency median/75th/95th/99th 15/16/25/29 ms, chunk size 1791
6, published 831319 msg/s, confirmed 835632 msg/s, consumed 831857 msg/s, latency median/75th/95th/99th 15/16/31/32 ms, chunk size 1950
7, published 870907 msg/s, confirmed 866387 msg/s, consumed 868404 msg/s, latency median/75th/95th/99th 15/16/20/32 ms, chunk size 2069
8, published 893488 msg/s, confirmed 893920 msg/s, consumed 895055 msg/s, latency median/75th/95th/99th 15/16/22/32 ms, chunk size 2177
9, published 829373 msg/s, confirmed 834843 msg/s, consumed 837549 msg/s, latency median/75th/95th/99th 15/16/31/32 ms, chunk size 2236
10, published 826515 msg/s, confirmed 819212 msg/s, consumed 813298 msg/s, latency median/75th/95th/99th 16/16/20/32 ms, chunk size 2274
Summary: published 791615 msg/s, confirmed 791017 msg/s, consumed 790029 msg/s, latency 95th 20 ms, chunk size 2275
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-latest.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --delete-streams -z 10 --rate 100000
Starting producer
1, published 108704 msg/s, confirmed 108663 msg/s, consumed 85373 msg/s, latency median/75th/95th/99th 192/217/225/242 ms, chunk size 89
2, published 99916 msg/s, confirmed 99721 msg/s, consumed 85885 msg/s, latency median/75th/95th/99th 277/327/361/377 ms, chunk size 77
3, published 100056 msg/s, confirmed 100192 msg/s, consumed 76149 msg/s, latency median/75th/95th/99th 470/554/621/621 ms, chunk size 73
4, published 100055 msg/s, confirmed 99955 msg/s, consumed 85377 msg/s, latency median/75th/95th/99th 671/705/738/738 ms, chunk size 71
5, published 99968 msg/s, confirmed 99968 msg/s, consumed 72135 msg/s, latency median/75th/95th/99th 906/973/1007/1007 ms, chunk size 69
6, published 99998 msg/s, confirmed 100099 msg/s, consumed 72553 msg/s, latency median/75th/95th/99th 1208/1208/1275/1275 ms, chunk size 68
7, published 100172 msg/s, confirmed 100172 msg/s, consumed 85185 msg/s, latency median/75th/95th/99th 1409/1409/1409/1409 ms, chunk size 68
8, published 100163 msg/s, confirmed 100063 msg/s, consumed 85184 msg/s, latency median/75th/95th/99th 1476/1544/1544/1611 ms, chunk size 68
9, published 99991 msg/s, confirmed 99991 msg/s, consumed 78359 msg/s, latency median/75th/95th/99th 1678/1745/1812/1812 ms, chunk size 67
10, published 100081 msg/s, confirmed 100081 msg/s, consumed 82645 msg/s, latency median/75th/95th/99th 1879/1946/1946/1946 ms, chunk size 67
Summary: published 100881 msg/s, confirmed 100871 msg/s, consumed 80870 msg/s, latency 95th 1946 ms, chunk size 67
me again. I did a few additional tests with a single node cluster (set the above cluster definition to replica 1) and have still the same results:
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-latest.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --delete-streams -z 10
Starting producer
1, published 430398 msg/s, confirmed 421144 msg/s, consumed 196918 msg/s, latency median/75th/95th/99th 284/318/351/351 ms, chunk size 276
2, published 774687 msg/s, confirmed 774191 msg/s, consumed 1001626 msg/s, latency median/75th/95th/99th 84/193/285/319 ms, chunk size 1016
3, published 835602 msg/s, confirmed 835602 msg/s, consumed 838674 msg/s, latency median/75th/95th/99th 15/16/21/32 ms, chunk size 1376
4, published 831931 msg/s, confirmed 834834 msg/s, consumed 828328 msg/s, latency median/75th/95th/99th 15/16/31/32 ms, chunk size 1616
5, published 805782 msg/s, confirmed 804417 msg/s, consumed 808914 msg/s, latency median/75th/95th/99th 15/16/25/29 ms, chunk size 1791
6, published 831319 msg/s, confirmed 835632 msg/s, consumed 831857 msg/s, latency median/75th/95th/99th 15/16/31/32 ms, chunk size 1950
7, published 870907 msg/s, confirmed 866387 msg/s, consumed 868404 msg/s, latency median/75th/95th/99th 15/16/20/32 ms, chunk size 2069
8, published 893488 msg/s, confirmed 893920 msg/s, consumed 895055 msg/s, latency median/75th/95th/99th 15/16/22/32 ms, chunk size 2177
9, published 829373 msg/s, confirmed 834843 msg/s, consumed 837549 msg/s, latency median/75th/95th/99th 15/16/31/32 ms, chunk size 2236
10, published 826515 msg/s, confirmed 819212 msg/s, consumed 813298 msg/s, latency median/75th/95th/99th 16/16/20/32 ms, chunk size 2274
Summary: published 791615 msg/s, confirmed 791017 msg/s, consumed 790029 msg/s, latency 95th 20 ms, chunk size 2275
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-latest.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --delete-streams -z 10 --rate 100000
Starting producer
1, published 108704 msg/s, confirmed 108663 msg/s, consumed 85373 msg/s, latency median/75th/95th/99th 192/217/225/242 ms, chunk size 89
2, published 99916 msg/s, confirmed 99721 msg/s, consumed 85885 msg/s, latency median/75th/95th/99th 277/327/361/377 ms, chunk size 77
3, published 100056 msg/s, confirmed 100192 msg/s, consumed 76149 msg/s, latency median/75th/95th/99th 470/554/621/621 ms, chunk size 73
4, published 100055 msg/s, confirmed 99955 msg/s, consumed 85377 msg/s, latency median/75th/95th/99th 671/705/738/738 ms, chunk size 71
5, published 99968 msg/s, confirmed 99968 msg/s, consumed 72135 msg/s, latency median/75th/95th/99th 906/973/1007/1007 ms, chunk size 69
6, published 99998 msg/s, confirmed 100099 msg/s, consumed 72553 msg/s, latency median/75th/95th/99th 1208/1208/1275/1275 ms, chunk size 68
7, published 100172 msg/s, confirmed 100172 msg/s, consumed 85185 msg/s, latency median/75th/95th/99th 1409/1409/1409/1409 ms, chunk size 68
8, published 100163 msg/s, confirmed 100063 msg/s, consumed 85184 msg/s, latency median/75th/95th/99th 1476/1544/1544/1611 ms, chunk size 68
9, published 99991 msg/s, confirmed 99991 msg/s, consumed 78359 msg/s, latency median/75th/95th/99th 1678/1745/1812/1812 ms, chunk size 67
10, published 100081 msg/s, confirmed 100081 msg/s, consumed 82645 msg/s, latency median/75th/95th/99th 1879/1946/1946/1946 ms, chunk size 67
Summary: published 100881 msg/s, confirmed 100871 msg/s, consumed 80870 msg/s, latency 95th 1946 ms, chunk size 67
Greetings,
Philipp
Philipp
Philipp Chlebicki
Jun 13, 2023, 3:43:53 AM6/13/23
to rabbitmq-users
Hey,
one thing I recognized that my limit on a single node cluster is ~80k messages with enabled rate limiter (depending on the used infrastructure this might higher). So you might be able to reproduce it with a rate limit of 500k (if your cluster can handle up to 900k without the rate limiter).
Would you mind to try this out for me?
BR,
Philipp
one thing I recognized that my limit on a single node cluster is ~80k messages with enabled rate limiter (depending on the used infrastructure this might higher). So you might be able to reproduce it with a rate limit of 500k (if your cluster can handle up to 900k without the rate limiter).
Would you mind to try this out for me?
BR,
Philipp
Arnaud Cogoluègnes
Jun 13, 2023, 4:28:10 AM6/13/23
to rabbitmq-users
We cannot reproduce this latency issue.
Can you try to only publish and then only consume?
stream-perf-test --producers 1 --consumers 0 --load-balancer -z 10
stream-perf-test --producers 0 --consumers 1 --load-balancer -z 10 --delete-streams --offset first
Adapt the publishing rate: no limitation the first time, limitation the second time.
We would like to see if the change in chunk could be the cause of the issue.
Could you also try to run the performance tool on a Linux machine? We don't use Windows and we don't know if stream-perf-test behaves as expected on it.
Philipp Chlebicki
Jun 13, 2023, 5:46:08 AM6/13/23
to rabbitmq-users
Hey,
as mentioned upfront I have the same behaviour, when running the stream-perf-tool in the kubernetes cluster as a Linux pod / container using the official kubectl rabbitmq plugin without the loadbalancer flag (kubectl rabbitmq stream-perf-test <clustername>)
Here are the results from an external client (both scenarios from the identical host).
--- WITHOUT RATE OPTION ---
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-0.10.0.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --producers 1 --consumers 0 -z 10
Starting producer
1, published 444130 msg/s, confirmed 434850 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
2, published 616613 msg/s, confirmed 617810 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
3, published 644634 msg/s, confirmed 647799 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
4, published 610799 msg/s, confirmed 605994 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
5, published 596008 msg/s, confirmed 598027 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
6, published 631771 msg/s, confirmed 629756 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
7, published 587405 msg/s, confirmed 587787 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
8, published 631647 msg/s, confirmed 631267 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
9, published 615286 msg/s, confirmed 615267 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
10, published 607147 msg/s, confirmed 607108 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
Summary: published 597862 msg/s, confirmed 596938 msg/s, consumed 0 msg/s, latency 95th 0 ms, chunk size 0
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-0.10.0.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --producers 0 --consumers 1 -z 10 --delete-streams --offset first
1, published 0 msg/s, confirmed 0 msg/s, consumed 430903 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 511
2, published 0 msg/s, confirmed 0 msg/s, consumed 593150 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 743
3, published 0 msg/s, confirmed 0 msg/s, consumed 653401 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 859
4, published 0 msg/s, confirmed 0 msg/s, consumed 568204 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 911
5, published 0 msg/s, confirmed 0 msg/s, consumed 818571 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 964
6, published 0 msg/s, confirmed 0 msg/s, consumed 918209 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 1011
7, published 0 msg/s, confirmed 0 msg/s, consumed 751244 msg/s, latency median/75th/95th/99th 22549/22549/22549/22549 ms, chunk size 1031
8, published 0 msg/s, confirmed 0 msg/s, consumed 614848 msg/s, latency median/75th/95th/99th 22549/23622/23622/23622 ms, chunk size 1034
9, published 0 msg/s, confirmed 0 msg/s, consumed 644860 msg/s, latency median/75th/95th/99th 22549/22549/22549/22549 ms, chunk size 1049
Summary: published 0 msg/s, confirmed 0 msg/s, consumed 598668 msg/s, latency 95th 0 ms, chunk size 1049
--- WITH RATE OPTION ---
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-0.10.0.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --producers 1 --consumers 0 -z 10 --rate 100000
Starting producer
1, published 104575 msg/s, confirmed 104311 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
2, published 100026 msg/s, confirmed 100094 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
3, published 100042 msg/s, confirmed 99973 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
4, published 100116 msg/s, confirmed 100116 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
5, published 99975 msg/s, confirmed 99875 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
6, published 100013 msg/s, confirmed 100002 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
7, published 100095 msg/s, confirmed 100128 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
8, published 99983 msg/s, confirmed 100062 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
9, published 100109 msg/s, confirmed 100077 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
10, published 100063 msg/s, confirmed 100056 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
Summary: published 100475 msg/s, confirmed 100448 msg/s, consumed 0 msg/s, latency 95th 0 ms, chunk size 0
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-0.10.0.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --producers 0 --consumers 1 -z 10 --delete-streams --offset first --rate 100000
1, published 0 msg/s, confirmed 0 msg/s, consumed 76632 msg/s, latency median/75th/95th/99th 12885/12885/12885/12885 ms, chunk size 68
2, published 0 msg/s, confirmed 0 msg/s, consumed 72799 msg/s, latency median/75th/95th/99th 13422/13422/13422/13422 ms, chunk size 56
3, published 0 msg/s, confirmed 0 msg/s, consumed 73062 msg/s, latency median/75th/95th/99th 13422/13422/13422/13422 ms, chunk size 52
4, published 0 msg/s, confirmed 0 msg/s, consumed 62916 msg/s, latency median/75th/95th/99th 13959/13959/13959/13959 ms, chunk size 49
5, published 0 msg/s, confirmed 0 msg/s, consumed 65632 msg/s, latency median/75th/95th/99th 13959/14496/14496/14496 ms, chunk size 48
6, published 0 msg/s, confirmed 0 msg/s, consumed 69298 msg/s, latency median/75th/95th/99th 14496/14496/14496/14496 ms, chunk size 47
7, published 0 msg/s, confirmed 0 msg/s, consumed 72004 msg/s, latency median/75th/95th/99th 15032/15032/15032/15032 ms, chunk size 47
8, published 0 msg/s, confirmed 0 msg/s, consumed 65412 msg/s, latency median/75th/95th/99th 15032/15032/15032/15032 ms, chunk size 47
9, published 0 msg/s, confirmed 0 msg/s, consumed 67139 msg/s, latency median/75th/95th/99th 15569/15569/15569/15569 ms, chunk size 46
10, published 0 msg/s, confirmed 0 msg/s, consumed 67421 msg/s, latency median/75th/95th/99th 15569/16106/16106/16106 ms, chunk size 46
Summary: published 0 msg/s, confirmed 0 msg/s, consumed 69226 msg/s, latency 95th 16106 ms, chunk size 46
To my understanding, the delay is still constantly increasing as the consume rate is slower, then the initial publishing rate. So also here the rate parameter influences slow consumption.
Thanks for all your efforts! I have to say that I am buzzed by all the support in this user group. Will try to give a bit back here in the future!
BR,
Philipp
as mentioned upfront I have the same behaviour, when running the stream-perf-tool in the kubernetes cluster as a Linux pod / container using the official kubectl rabbitmq plugin without the loadbalancer flag (kubectl rabbitmq stream-perf-test <clustername>)
Here are the results from an external client (both scenarios from the identical host).
--- WITHOUT RATE OPTION ---
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-0.10.0.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --producers 1 --consumers 0 -z 10
Starting producer
1, published 444130 msg/s, confirmed 434850 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
2, published 616613 msg/s, confirmed 617810 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
3, published 644634 msg/s, confirmed 647799 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
4, published 610799 msg/s, confirmed 605994 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
5, published 596008 msg/s, confirmed 598027 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
6, published 631771 msg/s, confirmed 629756 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
7, published 587405 msg/s, confirmed 587787 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
8, published 631647 msg/s, confirmed 631267 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
9, published 615286 msg/s, confirmed 615267 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
10, published 607147 msg/s, confirmed 607108 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
Summary: published 597862 msg/s, confirmed 596938 msg/s, consumed 0 msg/s, latency 95th 0 ms, chunk size 0
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-0.10.0.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --producers 0 --consumers 1 -z 10 --delete-streams --offset first
1, published 0 msg/s, confirmed 0 msg/s, consumed 430903 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 511
2, published 0 msg/s, confirmed 0 msg/s, consumed 593150 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 743
3, published 0 msg/s, confirmed 0 msg/s, consumed 653401 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 859
4, published 0 msg/s, confirmed 0 msg/s, consumed 568204 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 911
5, published 0 msg/s, confirmed 0 msg/s, consumed 818571 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 964
6, published 0 msg/s, confirmed 0 msg/s, consumed 918209 msg/s, latency median/75th/95th/99th 23622/23622/23622/23622 ms, chunk size 1011
7, published 0 msg/s, confirmed 0 msg/s, consumed 751244 msg/s, latency median/75th/95th/99th 22549/22549/22549/22549 ms, chunk size 1031
8, published 0 msg/s, confirmed 0 msg/s, consumed 614848 msg/s, latency median/75th/95th/99th 22549/23622/23622/23622 ms, chunk size 1034
9, published 0 msg/s, confirmed 0 msg/s, consumed 644860 msg/s, latency median/75th/95th/99th 22549/22549/22549/22549 ms, chunk size 1049
Summary: published 0 msg/s, confirmed 0 msg/s, consumed 598668 msg/s, latency 95th 0 ms, chunk size 1049
--- WITH RATE OPTION ---
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-0.10.0.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --producers 1 --consumers 0 -z 10 --rate 100000
Starting producer
1, published 104575 msg/s, confirmed 104311 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
2, published 100026 msg/s, confirmed 100094 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
3, published 100042 msg/s, confirmed 99973 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
4, published 100116 msg/s, confirmed 100116 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
5, published 99975 msg/s, confirmed 99875 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
6, published 100013 msg/s, confirmed 100002 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
7, published 100095 msg/s, confirmed 100128 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
8, published 99983 msg/s, confirmed 100062 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
9, published 100109 msg/s, confirmed 100077 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
10, published 100063 msg/s, confirmed 100056 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
Summary: published 100475 msg/s, confirmed 100448 msg/s, consumed 0 msg/s, latency 95th 0 ms, chunk size 0
C:\Users\pch\Desktop\tools\stream-perf-test>java -jar stream-perf-test-0.10.0.jar --uris rabbitmq-stream://<user>:<password>@<ip>:5552 --load-balancer --producers 0 --consumers 1 -z 10 --delete-streams --offset first --rate 100000
1, published 0 msg/s, confirmed 0 msg/s, consumed 76632 msg/s, latency median/75th/95th/99th 12885/12885/12885/12885 ms, chunk size 68
2, published 0 msg/s, confirmed 0 msg/s, consumed 72799 msg/s, latency median/75th/95th/99th 13422/13422/13422/13422 ms, chunk size 56
3, published 0 msg/s, confirmed 0 msg/s, consumed 73062 msg/s, latency median/75th/95th/99th 13422/13422/13422/13422 ms, chunk size 52
4, published 0 msg/s, confirmed 0 msg/s, consumed 62916 msg/s, latency median/75th/95th/99th 13959/13959/13959/13959 ms, chunk size 49
5, published 0 msg/s, confirmed 0 msg/s, consumed 65632 msg/s, latency median/75th/95th/99th 13959/14496/14496/14496 ms, chunk size 48
6, published 0 msg/s, confirmed 0 msg/s, consumed 69298 msg/s, latency median/75th/95th/99th 14496/14496/14496/14496 ms, chunk size 47
7, published 0 msg/s, confirmed 0 msg/s, consumed 72004 msg/s, latency median/75th/95th/99th 15032/15032/15032/15032 ms, chunk size 47
8, published 0 msg/s, confirmed 0 msg/s, consumed 65412 msg/s, latency median/75th/95th/99th 15032/15032/15032/15032 ms, chunk size 47
9, published 0 msg/s, confirmed 0 msg/s, consumed 67139 msg/s, latency median/75th/95th/99th 15569/15569/15569/15569 ms, chunk size 46
10, published 0 msg/s, confirmed 0 msg/s, consumed 67421 msg/s, latency median/75th/95th/99th 15569/16106/16106/16106 ms, chunk size 46
Summary: published 0 msg/s, confirmed 0 msg/s, consumed 69226 msg/s, latency 95th 16106 ms, chunk size 46
To my understanding, the delay is still constantly increasing as the consume rate is slower, then the initial publishing rate. So also here the rate parameter influences slow consumption.
Thanks for all your efforts! I have to say that I am buzzed by all the support in this user group. Will try to give a bit back here in the future!
BR,
Philipp
Arnaud Cogoluègnes
Jun 13, 2023, 10:08:21 AM6/13/23
to rabbitmq-users
Latency numbers don't mean much in this case, because the performance tool uses a timestamp in the message, so if you publish a message and consume it 10 minutes later, the latency will be 10 minutes. But that was expected.
I wanted to check the impact of chunk size (a chunk is the unit of storage, replication, and transfer, it's a batch of messages). Batch size mainly depends on ingress, so the lower the publishing rate, the smaller the chunk size. The chunk size is quite small when publishing at 100 K (around 46), but the consuming rate should not be that low still. I'm running the same tests on a local box, and the consuming rate is 500 K for the same chunk size (~ 50). The 1 M published messages are consumed in a couple of seconds.
I'm running a bit dry. Can you beef up the K8S instances (faster CPU and storage) and do more testing?
Philipp Chlebicki
Jun 13, 2023, 3:43:45 PM6/13/23
to rabbitmq-users
Hey,
I am aware that the latency is high in that case, but what I wanted to emphasize here is that the latency is steadily increasing on consumption (from 12-l to 16 seconds). Similar to what we see when publishing and consuming in parallel.
I do not think it is problem of insufficient resources. CPU and Disk should be fine (as seen without the rate limiter option). Or am I wrong?
We have here more then enough CPU available for assignment. Will double that tomorrow and do additional testing
Regarding the disk it's a bit more complex. Will recheck.
Side questions:
- Which RabbitMQ Version are you using?
- How does your infrastructure looks like (Kubernetes, Linux VM, Bare Metal, ...)?
- Which stream perf tool version are you using?
Thanks and BR,
Philipp
Michal Kuratczyk
Jun 14, 2023, 1:56:21 AM6/14/23
to rabbitm...@googlegroups.com
Hi,
In my case I was testing with stream-perf-test-latest.jar from the day I was testing a few days ago.
I tried on a bare metal Linux box and a GKE cluster. In both cases, I was using RabbitMQ from the main branch.
Best,
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/0756f33f-c889-4ec8-b903-f988c025ef34n%40googlegroups.com.
Michał
RabbitMQ teamArnaud Cogoluègnes
Jun 14, 2023, 3:52:13 AM6/14/23
to rabbitmq-users
I used the latest stream-perf-test, latest stable Docker image and build from main, all on Linux.
Philipp Chlebicki
Jun 15, 2023, 6:47:45 AM6/15/23
to rabbitmq-users
Hey,
would like to update you in this case. I updated to the latest rabbitmq version + operator and triggered the test in a different environment, which uses a more performant data store with double the amount of CPU assigned to RabbitMQCluster.
Unfortunately I still have here the same problems, when using the stream perf tool in combination with the rate limiter. Without the rate limitation the throughput is okish.
Any further ideas from anyone? Where can I dig into?
Thanks and BR,
Philipp
---
kubectl rabbitmq stream-perf-test message-broker --producers 1 --consumers 0 -z 10 --rate 100000
Starting producer
1, published 104920 msg/s, confirmed 104754 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
2, published 99497 msg/s, confirmed 99397 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
3, published 100005 msg/s, confirmed 100169 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
4, published 100095 msg/s, confirmed 100031 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
5, published 100022 msg/s, confirmed 100022 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
6, published 100123 msg/s, confirmed 100039 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
7, published 100075 msg/s, confirmed 100158 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
8, published 99924 msg/s, confirmed 100025 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
9, published 100117 msg/s, confirmed 100085 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
10, published 100102 msg/s, confirmed 99933 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
Summary: published 100432 msg/s, confirmed 100432 msg/s, consumed 0 msg/s, latency 95th 0 ms, chunk size 0
---
kubectl rabbitmq stream-perf-test message-broker --producers 0 --consumers 1 --offset first --delete-streams -z 10 --rate 100000
1, published 0 msg/s, confirmed 0 msg/s, consumed 84667 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 57
2, published 0 msg/s, confirmed 0 msg/s, consumed 81970 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 48
3, published 0 msg/s, confirmed 0 msg/s, consumed 85234 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 44
4, published 0 msg/s, confirmed 0 msg/s, consumed 86648 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 43
5, published 0 msg/s, confirmed 0 msg/s, consumed 79006 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 43
6, published 0 msg/s, confirmed 0 msg/s, consumed 78186 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 42
7, published 0 msg/s, confirmed 0 msg/s, consumed 80936 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 42
8, published 0 msg/s, confirmed 0 msg/s, consumed 82620 msg/s, latency median/75th/95th/99th 57982/60130/60130/60130 ms, chunk size 42
9, published 0 msg/s, confirmed 0 msg/s, consumed 75581 msg/s, latency median/75th/95th/99th 60130/60130/60130/60130 ms, chunk size 41
10, published 0 msg/s, confirmed 0 msg/s, consumed 77440 msg/s, latency median/75th/95th/99th 60130/60130/60130/60130 ms, chunk size 41
Summary: published 0 msg/s, confirmed 0 msg/s, consumed 81205 msg/s, latency 95th 60130 ms, chunk size 41
---
Here a test without the rate limtier in this environment.
kubectl rabbitmq stream-perf-test message-broker --producers 1 --consumers 1 -z 10 --delete-streams
Starting producer
1, published 617375 msg/s, confirmed 610237 msg/s, consumed 608966 msg/s, latency median/75th/95th/99th 142/201/218/235 ms, chunk size 692
2, published 747928 msg/s, confirmed 745122 msg/s, consumed 746404 msg/s, latency median/75th/95th/99th 15/19/23/27 ms, chunk size 890
3, published 739891 msg/s, confirmed 739981 msg/s, consumed 739469 msg/s, latency median/75th/95th/99th 15/18/21/22 ms, chunk size 976
4, published 750338 msg/s, confirmed 750248 msg/s, consumed 748990 msg/s, latency median/75th/95th/99th 15/16/20/23 ms, chunk size 1031
5, published 881988 msg/s, confirmed 882889 msg/s, consumed 884663 msg/s, latency median/75th/95th/99th 13/14/16/19 ms, chunk size 1082
6, published 850584 msg/s, confirmed 849707 msg/s, consumed 849707 msg/s, latency median/75th/95th/99th 13/15/19/22 ms, chunk size 1097
7, published 790470 msg/s, confirmed 794218 msg/s, consumed 794218 msg/s, latency median/75th/95th/99th 14/16/19/24 ms, chunk size 1118
8, published 781487 msg/s, confirmed 779115 msg/s, consumed 777502 msg/s, latency median/75th/95th/99th 14/16/20/22 ms, chunk size 1129
9, published 696936 msg/s, confirmed 695536 msg/s, consumed 697149 msg/s, latency median/75th/95th/99th 16/19/22/22 ms, chunk size 1138
10, published 766582 msg/s, confirmed 766625 msg/s, consumed 764580 msg/s, latency median/75th/95th/99th 14/18/21/22 ms, chunk size 1146
Summary: published 761766 msg/s, confirmed 760772 msg/s, consumed 760772 msg/s, latency 95th 21 ms, chunk size 1146
would like to update you in this case. I updated to the latest rabbitmq version + operator and triggered the test in a different environment, which uses a more performant data store with double the amount of CPU assigned to RabbitMQCluster.
Unfortunately I still have here the same problems, when using the stream perf tool in combination with the rate limiter. Without the rate limitation the throughput is okish.
Any further ideas from anyone? Where can I dig into?
Thanks and BR,
Philipp
---
kubectl rabbitmq stream-perf-test message-broker --producers 1 --consumers 0 -z 10 --rate 100000
Starting producer
1, published 104920 msg/s, confirmed 104754 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
2, published 99497 msg/s, confirmed 99397 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
3, published 100005 msg/s, confirmed 100169 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
4, published 100095 msg/s, confirmed 100031 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
5, published 100022 msg/s, confirmed 100022 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
6, published 100123 msg/s, confirmed 100039 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
7, published 100075 msg/s, confirmed 100158 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
8, published 99924 msg/s, confirmed 100025 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
9, published 100117 msg/s, confirmed 100085 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
10, published 100102 msg/s, confirmed 99933 msg/s, consumed 0 msg/s, latency median/75th/95th/99th 0/0/0/0 ms, chunk size 0
Summary: published 100432 msg/s, confirmed 100432 msg/s, consumed 0 msg/s, latency 95th 0 ms, chunk size 0
---
kubectl rabbitmq stream-perf-test message-broker --producers 0 --consumers 1 --offset first --delete-streams -z 10 --rate 100000
1, published 0 msg/s, confirmed 0 msg/s, consumed 84667 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 57
2, published 0 msg/s, confirmed 0 msg/s, consumed 81970 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 48
3, published 0 msg/s, confirmed 0 msg/s, consumed 85234 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 44
4, published 0 msg/s, confirmed 0 msg/s, consumed 86648 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 43
5, published 0 msg/s, confirmed 0 msg/s, consumed 79006 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 43
6, published 0 msg/s, confirmed 0 msg/s, consumed 78186 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 42
7, published 0 msg/s, confirmed 0 msg/s, consumed 80936 msg/s, latency median/75th/95th/99th 57982/57982/57982/57982 ms, chunk size 42
8, published 0 msg/s, confirmed 0 msg/s, consumed 82620 msg/s, latency median/75th/95th/99th 57982/60130/60130/60130 ms, chunk size 42
9, published 0 msg/s, confirmed 0 msg/s, consumed 75581 msg/s, latency median/75th/95th/99th 60130/60130/60130/60130 ms, chunk size 41
10, published 0 msg/s, confirmed 0 msg/s, consumed 77440 msg/s, latency median/75th/95th/99th 60130/60130/60130/60130 ms, chunk size 41
Summary: published 0 msg/s, confirmed 0 msg/s, consumed 81205 msg/s, latency 95th 60130 ms, chunk size 41
---
Here a test without the rate limtier in this environment.
kubectl rabbitmq stream-perf-test message-broker --producers 1 --consumers 1 -z 10 --delete-streams
Starting producer
1, published 617375 msg/s, confirmed 610237 msg/s, consumed 608966 msg/s, latency median/75th/95th/99th 142/201/218/235 ms, chunk size 692
2, published 747928 msg/s, confirmed 745122 msg/s, consumed 746404 msg/s, latency median/75th/95th/99th 15/19/23/27 ms, chunk size 890
3, published 739891 msg/s, confirmed 739981 msg/s, consumed 739469 msg/s, latency median/75th/95th/99th 15/18/21/22 ms, chunk size 976
4, published 750338 msg/s, confirmed 750248 msg/s, consumed 748990 msg/s, latency median/75th/95th/99th 15/16/20/23 ms, chunk size 1031
5, published 881988 msg/s, confirmed 882889 msg/s, consumed 884663 msg/s, latency median/75th/95th/99th 13/14/16/19 ms, chunk size 1082
6, published 850584 msg/s, confirmed 849707 msg/s, consumed 849707 msg/s, latency median/75th/95th/99th 13/15/19/22 ms, chunk size 1097
7, published 790470 msg/s, confirmed 794218 msg/s, consumed 794218 msg/s, latency median/75th/95th/99th 14/16/19/24 ms, chunk size 1118
8, published 781487 msg/s, confirmed 779115 msg/s, consumed 777502 msg/s, latency median/75th/95th/99th 14/16/20/22 ms, chunk size 1129
9, published 696936 msg/s, confirmed 695536 msg/s, consumed 697149 msg/s, latency median/75th/95th/99th 16/19/22/22 ms, chunk size 1138
10, published 766582 msg/s, confirmed 766625 msg/s, consumed 764580 msg/s, latency median/75th/95th/99th 14/18/21/22 ms, chunk size 1146
Summary: published 761766 msg/s, confirmed 760772 msg/s, consumed 760772 msg/s, latency 95th 21 ms, chunk size 1146
Arnaud Cogoluègnes
Jun 15, 2023, 7:45:58 AM6/15/23
to rabbitmq-users
I'm dry here. The difference is the chunk size, which is a batch of messages. Chunks are the unit of storage, replication, and dispatching (to consumers). The number of messages in a chunk depends on the ingress, so if you publish fast you're likely to have large chunks. This is what you observe: about 1 K messages without rate limitation and about 40 with rate limitation. So with smaller chunks, there's less batching, so more dispatching frames, which can slow down consumption. Still, 40 is not a large chunk size, but you should be able to make better than 75 K messages / second, especially when you only consume (hence the tests I suggested above).
Can you try again with stream-perf-test 0.9.0 [1]? It has a more aggressive flow control strategy than the current version.
Philipp Chlebicki
Jun 19, 2023, 1:40:26 AM6/19/23
to rabbitmq-users
Hi,
thanks for getting back to me and generally I understand the chunks topic.
I did a small test with stream-perf-test 0.9.0 and compared it against latest using the official container images (https://hub.docker.com/r/pivotalrabbitmq/stream-perf-test) to omit network troubles from outside my Kubernetes cluster.
While the chunk size stayed the same, the message rate was finally at an acceptable state. So not sure what the difference between both version is, but from my standpoint it does not seem to be the calculation of chunks.
Latest dockerhub container image
I did a small test with stream-perf-test 0.9.0 and compared it against latest using the official container images (https://hub.docker.com/r/pivotalrabbitmq/stream-perf-test) to omit network troubles from outside my Kubernetes cluster.
While the chunk size stayed the same, the message rate was finally at an acceptable state. So not sure what the difference between both version is, but from my standpoint it does not seem to be the calculation of chunks.
Latest dockerhub container image
1, published 111251 msg/s, confirmed 110951 msg/s, consumed 67245 msg/s, latency median/75th/95th/99th 266/350/417/451 ms, chunk size 110
2, published 99807 msg/s, confirmed 99808 msg/s, consumed 48114 msg/s, latency median/75th/95th/99th 721/822/923/956 ms, chunk size 75
3, published 100012 msg/s, confirmed 100016 msg/s, consumed 53345 msg/s, latency median/75th/95th/99th 1242/1309/1376/1443 ms, chunk size 65
4, published 99958 msg/s, confirmed 99954 msg/s, consumed 56880 msg/s, latency median/75th/95th/99th 1611/1745/1812/1812 ms, chunk size 59
5, published 100052 msg/s, confirmed 100150 msg/s, consumed 50664 msg/s, latency median/75th/95th/99th 2080/2215/2349/2349 ms, chunk size 56
6, published 100049 msg/s, confirmed 99279 msg/s, consumed 47428 msg/s, latency median/75th/95th/99th 2684/2684/2819/2819 ms, chunk size 54
7, published 99320 msg/s, confirmed 100197 msg/s, consumed 47506 msg/s, latency median/75th/95th/99th 3087/3221/3355/3355 ms, chunk size 52
8, published 100666 msg/s, confirmed 100529 msg/s, consumed 51233 msg/s, latency median/75th/95th/99th 3624/3758/3758/3758 ms, chunk size 51
9, published 100076 msg/s, confirmed 99945 msg/s, consumed 49237 msg/s, latency median/75th/95th/99th 4027/4161/4429/4429 ms, chunk size 51
10, published 100115 msg/s, confirmed 100131 msg/s, consumed 51706 msg/s, latency median/75th/95th/99th 4563/4563/4832/4832 ms, chunk size 50
0.9.0
2, published 99807 msg/s, confirmed 99808 msg/s, consumed 48114 msg/s, latency median/75th/95th/99th 721/822/923/956 ms, chunk size 75
3, published 100012 msg/s, confirmed 100016 msg/s, consumed 53345 msg/s, latency median/75th/95th/99th 1242/1309/1376/1443 ms, chunk size 65
4, published 99958 msg/s, confirmed 99954 msg/s, consumed 56880 msg/s, latency median/75th/95th/99th 1611/1745/1812/1812 ms, chunk size 59
5, published 100052 msg/s, confirmed 100150 msg/s, consumed 50664 msg/s, latency median/75th/95th/99th 2080/2215/2349/2349 ms, chunk size 56
6, published 100049 msg/s, confirmed 99279 msg/s, consumed 47428 msg/s, latency median/75th/95th/99th 2684/2684/2819/2819 ms, chunk size 54
7, published 99320 msg/s, confirmed 100197 msg/s, consumed 47506 msg/s, latency median/75th/95th/99th 3087/3221/3355/3355 ms, chunk size 52
8, published 100666 msg/s, confirmed 100529 msg/s, consumed 51233 msg/s, latency median/75th/95th/99th 3624/3758/3758/3758 ms, chunk size 51
9, published 100076 msg/s, confirmed 99945 msg/s, consumed 49237 msg/s, latency median/75th/95th/99th 4027/4161/4429/4429 ms, chunk size 51
10, published 100115 msg/s, confirmed 100131 msg/s, consumed 51706 msg/s, latency median/75th/95th/99th 4563/4563/4832/4832 ms, chunk size 50
0.9.0
1, published 110983 msg/s, confirmed 110786 msg/s, consumed 110491 msg/s, latency median/75th/95th/99th 9/46/54/56 ms, chunk size 79
2, published 99981 msg/s, confirmed 99843 msg/s, consumed 99981 msg/s, latency median/75th/95th/99th 6/7/10/13 ms, chunk size 63
3, published 100014 msg/s, confirmed 100146 msg/s, consumed 99645 msg/s, latency median/75th/95th/99th 6/7/11/15 ms, chunk size 57
4, published 100150 msg/s, confirmed 99875 msg/s, consumed 100443 msg/s, latency median/75th/95th/99th 7/8/11/19 ms, chunk size 55
5, published 99891 msg/s, confirmed 100197 msg/s, consumed 99898 msg/s, latency median/75th/95th/99th 6/7/11/15 ms, chunk size 53
6, published 100130 msg/s, confirmed 98800 msg/s, consumed 99197 msg/s, latency median/75th/95th/99th 6/7/9/11 ms, chunk size 52
7, published 100050 msg/s, confirmed 101147 msg/s, consumed 100950 msg/s, latency median/75th/95th/99th 6/7/10/13 ms, chunk size 51
8, published 100046 msg/s, confirmed 100250 msg/s, consumed 99941 msg/s, latency median/75th/95th/99th 6/7/9/12 ms, chunk size 50
9, published 100077 msg/s, confirmed 99977 msg/s, consumed 100105 msg/s, latency median/75th/95th/99th 7/8/10/12 ms, chunk size 50
10, published 100039 msg/s, confirmed 99938 msg/s, consumed 100247 msg/s, latency median/75th/95th/99th 6/7/8/10 ms, chunk size 49
The good thing is that I now have the tools to simulate different publishers at different rates and that I can outrule the infrastructure and blame at least the tooling here.
If you need additional tests or further debugging, feel free to contact me at any point in time.
Thanks a lot!
BR,
Philipp
2, published 99981 msg/s, confirmed 99843 msg/s, consumed 99981 msg/s, latency median/75th/95th/99th 6/7/10/13 ms, chunk size 63
3, published 100014 msg/s, confirmed 100146 msg/s, consumed 99645 msg/s, latency median/75th/95th/99th 6/7/11/15 ms, chunk size 57
4, published 100150 msg/s, confirmed 99875 msg/s, consumed 100443 msg/s, latency median/75th/95th/99th 7/8/11/19 ms, chunk size 55
5, published 99891 msg/s, confirmed 100197 msg/s, consumed 99898 msg/s, latency median/75th/95th/99th 6/7/11/15 ms, chunk size 53
6, published 100130 msg/s, confirmed 98800 msg/s, consumed 99197 msg/s, latency median/75th/95th/99th 6/7/9/11 ms, chunk size 52
7, published 100050 msg/s, confirmed 101147 msg/s, consumed 100950 msg/s, latency median/75th/95th/99th 6/7/10/13 ms, chunk size 51
8, published 100046 msg/s, confirmed 100250 msg/s, consumed 99941 msg/s, latency median/75th/95th/99th 6/7/9/12 ms, chunk size 50
9, published 100077 msg/s, confirmed 99977 msg/s, consumed 100105 msg/s, latency median/75th/95th/99th 7/8/10/12 ms, chunk size 50
10, published 100039 msg/s, confirmed 99938 msg/s, consumed 100247 msg/s, latency median/75th/95th/99th 6/7/8/10 ms, chunk size 49
The good thing is that I now have the tools to simulate different publishers at different rates and that I can outrule the infrastructure and blame at least the tooling here.
If you need additional tests or further debugging, feel free to contact me at any point in time.
Thanks a lot!
BR,
Philipp
Arnaud Cogoluègnes
Jun 19, 2023, 2:55:47 AM6/19/23
to rabbitmq-users
Thanks for testing. We'll revert the change we made between 0.9 and 0.10 to avoid this kind of problem.
Message has been deleted
Arnaud Cogoluègnes
Jul 6, 2023, 10:17:13 AM7/6/23
to rabbitmq-users
stream-perf-tool 0.11.0 is out [1] with a new flag: --initial-credits
You can set this flag to e.g. 10 to have more in-flight chunks of messages. We set this value from 10 to 1 between 0.9.0 and 0.10.0, this is a conservative change, but apparently it was not working well in your case.
Reply all
Reply to author
Forward
0 new messages