Rabbit MQ Super Stream Partitions badrpc

[Viet Nguyen]

Hi All, I have issues while trying to create a super stream with 5000 partitions, and have some question

1. Is there a limit to how many partitions a stream can have?
2. We want to create a partition for each individual user in our system. We have upwards to 100k users. Can the super streams handle this correctly or should we instead have only 10 streams and use the code to shard the users across the streams?
3. Is the error listed here from the server running out of memory? Even though we get that error, it does still seem to create the partitions anyway.

docker exec ${RABBIT_CONTAINER} rabbitmq-streams add_super_stream test --partitions 5000
Adding a super stream...
Stack trace:
** (FunctionClauseError) no function clause matching in RabbitMQ.CLI.Ctl.Commands.AddSuperStreamCommand.output/2
    (rabbitmq_stream 3.11.5) Elixir.RabbitMQ.CLI.Ctl.Commands.AddSuperStreamCommand.erl:285: RabbitMQ.CLI.Ctl.Commands.AddSuperStreamCommand.output({:badrpc, {:EXIT, {:timeout, {:gen_server, :call, [:rabbit_stream_manager, {:create_super_stream, "/", "test", ["test-0", "test-1", "test-2", "test-3", "test-4", "test-5", "test-6", "test-7", "test-8", "test-9", "test-10", "test-11", "test-12", "test-13", "test-14", "test-15", "test-16", "test-17", "test-18", "test-19", "test-20", "test-21", "test-22", "test-23", "test-24", "test-25", "test-26", "test-27", "test-28", "test-29", "test-30", "test-31", "test-32", "test-33", "test-34", ...], %{}, ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31", "32", ...], "rmq-cli"}]}}}}, %{longnames: false, node: :rabbit@cb5a2807e0a5, partitions: 5000, timeout: :infinity, vhost: "/"})
    (rabbitmqctl 3.11.0) lib/rabbitmqctl.ex:165: RabbitMQCtl.maybe_run_command/3
    (rabbitmqctl 3.11.0) lib/rabbitmqctl.ex:133: anonymous fn/5 in RabbitMQCtl.do_exec_parsed_command/5
    (rabbitmqctl 3.11.0) lib/rabbitmqctl.ex:613: RabbitMQCtl.maybe_with_distribution/3
    (rabbitmqctl 3.11.0) lib/rabbitmqctl.ex:98: RabbitMQCtl.exec_command/2
    (rabbitmqctl 3.11.0) lib/rabbitmqctl.ex:32: RabbitMQCtl.main/1
    (elixir 1.14.0) lib/kernel/cli.ex:131: anonymous fn/3 in Kernel.CLI.exec_fun/2

Error:
:function_clause

[Salvatore Garbesi]

I've been wondering this same thing myself.

[Trong Pham]

I've had the same problem.

[kjnilsson]

Hi,

Do not create super streams with thousands of partitions. "Partitions" are first class streams inside RabbitMQ and use some resources even if idle. Instead ask yourself why you need to partition your streams and work from there to determine the number of partitions you need. 

Mostly users need to partition streams due to the lack of competing consumers over streams. So work out your expected ingress rate and the approximate average time it takes for a consumer to process one of these events. If a single consumer can't keep up with the ingress rate you need to use a super stream with enough partitions to meet the ingress rate. The perhaps add 1 or two partitions for future expansion.

Hope this helps.

Cheers

Karl