federation exchange mode has low throughput about 3MB/s

[zou tao]

Hi,

I have three brokers to test federation exchange performance, HW 2VCPU, 8G RAM, 40G disk.

One is upstream, one is downstream server, the other run PT tool.

and I found the speed upstream broker propagate to downstream broker is slow, and  'consumer utilisation' is low.

I tried to increase the Prefetch count from default value to 30000. it didn't help.   CPU usage is around 120~140%.

any suggestion will be appreciated.

B.R,

Tao

First I tested the upstream broker by p1_c1_spec.js, the  througput is about 50K message/s, CPU usage is around 180~190%.

          [{"name":      "producer/consume",

  "uri":       "amqp://10.75.217.27:5672",
  "type":      "simple",
  "params":    [{"time-limit":     200,
                 "producer-count": 1,
                 "consumer-count": 1,
                 "min-msg-size": 1024,
                 "producer-rate-limit": 50000,
                 "consumer-rate-limit": 50000,
                 "auto-ack": true}],
  "flag":        [{"persistent": false}]
}]

 1.  upstream

   bin/runjava com.rabbitmq.perf.PerfTestMulti p1_spec.js p1_result.js

more p1_spec.js:

   

[{"name":      "one producer to publish",
  "uri":       "amqp://10.75.217.27:5672",
  "type":      "simple",
  "params":    [{"time-limit":     86400,
                 "producer-count": 1,
                 "routing-key": "",
                 "consumer-count": 1,
                 "min-msg-size": 1024,
                 "exchange-name": "fruit",
                 "predeclared": true,
                 "producer-rate-limit": 8000,
                 "auto-ack": true}],
  "flag":        [{"persistent": false}]
}]

   2. downstream :

  bin/runjava com.rabbitmq.perf.PerfTestMulti c1_spec.js c1_result.js

 Running scenario 'consume' more c1[root@crmq-autopt-lab-3 pt_tool]# more c1_spec.js
[{"name":      "consume",
  "uri":       "amqp://10.75.217.41:5672",
  "type":      "simple",
  "params":    [{"time-limit":     86400,
                 "producer-count": 1,
                 "consumer-count": 1,
                 "min-msg-size": 1024,
                 "queue-names": ["apple"],
                 "predeclared": true,
                 "consumer-rate-limit": 8000,
                 "auto-ack": true}],
  "flag":        [{"persistent": false}]
}]

[Michael Klishin]

30K is excessive and won't be meaningfully different from 3-5K in practice.

What are the federation link settings used? E.g. acknowledgement mode will have a significant effect

on throughput [1]. We do not recommend using automatic acknowledgements [2] unless you understand the risk

associated with it but it would likely make a difference.

2 vCPU isn't necessarily a lot for all workloads. See rabbitmq-top [3] reductions (relative units of work in the runtime) data

and as I suggested in a different thread, make sure you are comparing apples to apples. A Java client consumer with automatic

acknowledgements *will* demonstrate the kind of throughput a federation link, in particular with manual acknowledgement and potentially

using publisher confirms [2] on the republishing end can never match.

1. http://www.rabbitmq.com/federation-reference.html

2. http://www.rabbitmq.com/confirms.html

3. http://github.com/rabbitmq/rabbitmq-top

4. http://www.rabbitmq.com/queues.html#runtime-characteristics

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[zou tao]

hi MK,

Thanks for your reply.

the federation setting like:

rabbitmqctl set_parameter federation-upstream my-upstream \'{"uri":"amqp://%s","expires":3600000}\'

rabbitmqctl set_parameter federation-upstream my-upstream1 \'{"uri":"amqp://%s","expires":3600000}\'

policy:

rabbitmqctl set_policy --apply-to exchanges fruit "fruit" \'{"federation-upstream-set":"all"}\'

exchange:

 fruit

I used two federation upstream and hope it could speed it up.

I also tried to set the ack mode to on-publish, but it didn't help.

I just don't understand why 'consumer utilization' is so slow here.

The output for rabbitmq-top is ,

the CPU top for upstream broker is about 140%(2v CPU), for downstream broker is about 180%.

if I run one producer/consumer for local exchange/queue, the top speed will be 50K(message size is 1K),

but the federation exchange is about 3K when I set the speed limit of producer to 8K messages per seconds.

since there is no benchmark for federation plugin. I'm not sure if it's normal.

do you think the performance is normal?

B.R,

Tao

在 2018年8月6日星期一 UTC+8下午3:00:37，Michael Klishin写道：

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Each [exchange] federation upstream gets a link, which in turn carries replicated stream of messages.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[zou tao]

Hi MK,

I setup other enviroment in order to increase the VCPU number of downstream broker.

downstream broker: 4VCPU,   10.76.6.181

upstream broker:    2VCPU,    10.76.6.215

both have 8G ram, 80G disk.

and set of federation exchange like:

 

the federation setting like:

rabbitmqctl set_parameter federation-upstream a1 \'{"uri":"amqp:10.76.6.215//"}\'

    .

rabbitmqctl set_parameter federation-upstream a10 \'{"uri":"amqp:10.76.6.215//"}\'

I used 10 federation -upstream to increase consumer number.

policy:

rabbitmqctl set_policy --apply-to exchanges animal "animal" \'{"federation-upstream-set":"all"}\'

and using the # ./bin/runjava com.rabbitmq.perf.PerfTestMulti  p1_spec.js p1_result.js   to simulate the streams.

The upper speed limit is 8K, message size is 1K,

consumer setting is the similiar.

  
[root@taozou-rabbitmq-4 pt_tool]# more p1_spec.js
[{"name":      "publish",
  "uri":       "amqp://10.76.6.215:5672",

  "type":      "simple",
  "params":    [{"time-limit":     86400,
                 "producer-count": 1,

                 "consumer-count": 0,
                 "min-msg-size": 1024,
                         "exchange-name": "animal",
                 "routing-key": "",

                 "predeclared": true,
                 "producer-rate-limit": 8000,
                 "auto-ack": true}],
  "flag":        [{"persistent": false}]
}]

and #[root@taozou-rabbitmq-4 pt_tool]# more c1_spec.js
[{"name":      "consume",
  "uri":       "amqp://10.76.6.181:5672",

  "type":      "simple",
  "params":    [{"time-limit":     86400,

                 "producer-count": 0,
                 "consumer-count": 1,
                 "min-msg-size": 1024,
                     "queue-names": ["cat"],

                 "predeclared": true,
                 "consumer-rate-limit": 8000,
                 "auto-ack": true}],
  "flag":        [{"persistent": false}]
}]

while running, the downstream broker only got about  3K.

the CPU usage of downstream broker is:

  

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
15498 rabbitmq  20   0 3958984 242280   4460 S 145.5  3.0  83:17.95 beam.smp
18471 rabbitmq  20   0   41504   1268    972 S  18.3  0.0   6:15.41 inet_gethost
15930 rabbitmq  20   0   11588    636    520 S   1.3  0.0   0:42.14 inet_gethost
15649 rabbitmq  20   0   11756    608    304 S   0.0  0.0   0:00.07 epmd
15837 rabbitmq  20   0    4352    532    444 S   0.0  0.0   0:00.17 erl_child_setup
15931 rabbitmq  20   0   41504   1268    968 S   0.0  0.0   3:09.43 inet_gethost
18470 rabbitmq  20   0   41504   1268    972 S   0.0  0.0   1:16.28 inet_gethost

since the broker has 4VCPU, i can't image why the speed is so slow.

for upstream broker CPU is about 70. the network bandwith between these two nodes are more than 100MB.

I wanna use rabbitmq_federation plugin to backup between two clusters . but the  performance is so poor that I can't

meet the peformance requirement.

Any suggestions will be appreciated.

B.R,

Tao

在 2018年8月6日星期一 UTC+8下午6:32:31，Michael Klishin写道：

[zou tao]

as I understand, federation plugin will create a federation-exchange and a queue.

plugin will bind the federation-exchange to target exchange and queue.

The downstream broker will run a consumer to retrieve the messages and publish them to target queue.

In my scenario, there are 2 consumers and 1 producer.

why the thoughput is so low? and any benchmark for federatoin plugins?

在 2018年8月9日星期四 UTC+8下午5:09:26，zou tao写道：

[zou tao]

在 2018年8月9日星期四 UTC+8下午5:27:51，zou tao写道：

[Michael Klishin]

Your limiting factor — even if not today but eventually — will be the throughput a single queue can sustain.

As you correctly found out, each exchange federation link uses a queue internally.

3K is far from the limits of a single queue (you can see for yourself with PerfTest agaisnt a standalone node, for example)

but the fundamental issue still remains. Is a single link test realistic? Do you plan to have a workload

that pushes through many thousands of messages per second using a single exchange? Most workloads use more than one

exchange and definitely more than one queue. That's what most parts of RabbitMQ were designed with in mind.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

So the top process is the single queue. That's expected on a benchmark that uses a single publisher and consumer.

There's also a bunch of links that have done a comparable number of reeductions and spent most of them (reductions) in

`init/1`, a constructor function [1]. Moreover, they have a backlog of gen_server2 messages (internal buffer for Erlang messages, not RabbitMQ ones).

On top of that, those links somehow do as much work as a client connection writer and channel.

That's pretty surprising. Can definitions be exported from both ends of the federation link and shared here? I won't be able to get to an environment with 2 physical

machines until mid-next week but this is something worth profiling further.

1. https://github.com/rabbitmq/rabbitmq-federation/blob/master/src/rabbit_federation_exchange_link.erl#L71

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[zou tao]

Hi MK,

thanks for your reply.

The point is no matter how many exchanges and queues I'll use, I want to know the capacity of one exchange with one queue,

so I could figure out how many exchanges and queues I'll need to satisfy my requirement.

I have tested on the same 2VCPU node, one produce/one consumer, the top speed should be 50K(message size is 1024).

even I use one federation link(or more) to pull messages from just one upstream federation queue, i hope the speed could reach 50K,

currently the bottleneck is the consumer which inside downstream federation pulling messages from upstream federation queue.

so the question here is, why the speed in federation consumer is so slow?

from code, handle_info(#'basic.deliver') just change header and forward message to local queue. on theory,two consumer should enough to reach

the 50K limit.

Let me know if you need any other info.

downstream, no upstream federation link

[root@taozou-rabbitmq-4 ~]# rabbitmqctl  federation_status
Listing federation links of node rabbit@taozou-rabbitmq-4...
[#{error => <<>>,exchange => <<"animal">>,id => <<"f9054ffe">>,
   last_changed => <<"2018-08-10 04:26:24">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5728.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"a7">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"8c7df9a4">>,
   last_changed => <<"2018-08-10 04:26:27">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5725.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"my4">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"3d9759ad">>,
   last_changed => <<"2018-08-10 04:26:24">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5705.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"a2">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"9915c548">>,
   last_changed => <<"2018-08-10 04:26:24">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5722.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"a4">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"1e1324c1">>,
   last_changed => <<"2018-08-10 04:26:27">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5687.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"my2">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"90adfcfe">>,
   last_changed => <<"2018-08-10 04:26:24">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5697.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"a8">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"10143b12">>,
   last_changed => <<"2018-08-10 04:26:24">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5710.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"a3">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"6e1b943c">>,
   last_changed => <<"2018-08-10 04:26:27">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5681.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"my3">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"2a2a65b1">>,
   last_changed => <<"2018-08-10 04:26:24">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5702.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"a1">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"adc3c6e8">>,
   last_changed => <<"2018-08-10 04:26:24">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5694.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"a6">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"846eacce">>,
   last_changed => <<"2018-08-10 04:26:24">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5684.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"a9">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"86329bae">>,
   last_changed => <<"2018-08-10 04:26:24">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5719.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"a5">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"5c6b411b">>,
   last_changed => <<"2018-08-10 04:26:27">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5714.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"my6">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"68131859">>,
   last_changed => <<"2018-08-10 04:26:24">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5678.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"a10">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"fc06dad9">>,
   last_changed => <<"2018-08-10 04:26:27">>,
   local_connection => <<"<rab...@taozou-rabbitmq-4.3.5691.0>">>,
   queue => <<>>,status => running,type => exchange,upstream => <<"my5">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>}]

 

在 2018年8月9日星期四 UTC+8下午9:18:49，Michael Klishin写道：

[zou tao]

Hi MK,

I tested with perf tool on  2VCPU node, and one producer, one consumer, the top speed is more than 42.5K(message size is 1024).

CPU usage：

21780 rabbitmq  20   0 2862264 109436   4296 S 200.0  2.8  34:52.17 beam.smp
22166 rabbitmq  20   0    4352    532    444 S   0.0  0.0   0:00.01 erl_child_setup
22213 rabbitmq  20   0   11588    444    344 S   0.0  0.0   0:00.01 inet_gethost
22214 rabbitmq  20   0   39440   1240    948 S   0.0  0.0   0:00.26 inet_gethost

so may i conclude that one queue could support up to 1million reductions/sec?

but federation exchange queue is about

the rabbitmq top :

It seems that producer used more CPU than consumer.  why producer process type is 'reader'? I don't know what reader/channel/writer mean  here.

anyway, the reductions of consumer write/channel about 800K. which is ten times than federation consumer.

在 2018年8月9日星期四 UTC+8下午9:18:49，Michael Klishin写道：

So the top process is the single queue. That's expected on a benchmark that uses a single publisher and consumer.

[Michael Klishin]

I already told you that a federation link cannot match RabbitMQ Java client in terms of throughput.

"reader" is the "reading" part of a connection ("writer" is pretty self-explanatory then). Channels are, well, channel processes

and they perform routing and do most of other work that's not protocol parsing/authentication and not queues.

On a 2 vCPU machine they all + the link will be competing for resources.

A profiling session would tell more.

On Fri, Aug 10, 2018 at 4:33 AM, zou tao <zout...@gmail.com> wrote:

Hi MK,

thanks for your reply.

The point is no matter how many exchanges and queues I'll use, I want to know the capacity of one exchange with one queue,

so I could figure out how many exchanges and queues I'll need to satisfy my requirement.

I have tested on the same 2VCPU node, one produce/one consumer, the top speed should be 50K(message size is 1024).

even I use one federation link(or more) to pull messages from just one upstream federation queue, i hope the speed could reach 50K,

currently the bottleneck is the consumer which inside downstream federation pulling messages from upstream federation queue.

so the question here is, why the speed in federation consumer is so slow?

from code, handle_info(#'basic.deliver') just change header and forward message to local queue. on theory,two consumer should enough to reach

the 50K limit.

Let me know if you need any other info.

downstream, no upstream federation link

[root@taozou-rabbitmq-4 ~]# rabbitmqctl  federation_status
Listing federation links of node rabbit@taozou-rabbitmq-4...
[#{error => <<>>,exchange => <<"animal">>,id => <<"f9054ffe">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5728.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a7">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"8c7df9a4">>,
   last_changed => <<"2018-08-10 04:26:27">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5725.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"my4">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"3d9759ad">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5705.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a2">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"9915c548">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5722.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a4">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"1e1324c1">>,
   last_changed => <<"2018-08-10 04:26:27">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5687.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"my2">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"90adfcfe">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5697.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a8">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"10143b12">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5710.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a3">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"6e1b943c">>,
   last_changed => <<"2018-08-10 04:26:27">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5681.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"my3">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"2a2a65b1">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5702.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a1">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"adc3c6e8">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5694.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a6">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"846eacce">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5684.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a9">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"86329bae">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5719.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a5">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"5c6b411b">>,
   last_changed => <<"2018-08-10 04:26:27">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5714.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"my6">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"68131859">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5678.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a10">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"fc06dad9">>,
   last_changed => <<"2018-08-10 04:26:27">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5691.0>">>,

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

I think we've already established that for a single exchange adding more links means

*more replica streams* (copies of the same stream of messages), NOT more competing consumers on one federated exchange's queue.

On Fri, Aug 10, 2018 at 4:33 AM, zou tao <zout...@gmail.com> wrote:

Hi MK,

thanks for your reply.

The point is no matter how many exchanges and queues I'll use, I want to know the capacity of one exchange with one queue,

so I could figure out how many exchanges and queues I'll need to satisfy my requirement.

I have tested on the same 2VCPU node, one produce/one consumer, the top speed should be 50K(message size is 1024).

even I use one federation link(or more) to pull messages from just one upstream federation queue, i hope the speed could reach 50K,

currently the bottleneck is the consumer which inside downstream federation pulling messages from upstream federation queue.

so the question here is, why the speed in federation consumer is so slow?

from code, handle_info(#'basic.deliver') just change header and forward message to local queue. on theory,two consumer should enough to reach

the 50K limit.

Let me know if you need any other info.

downstream, no upstream federation link

[root@taozou-rabbitmq-4 ~]# rabbitmqctl  federation_status
Listing federation links of node rabbit@taozou-rabbitmq-4...
[#{error => <<>>,exchange => <<"animal">>,id => <<"f9054ffe">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5728.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a7">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"8c7df9a4">>,
   last_changed => <<"2018-08-10 04:26:27">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5725.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"my4">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"3d9759ad">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5705.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a2">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"9915c548">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5722.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a4">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"1e1324c1">>,
   last_changed => <<"2018-08-10 04:26:27">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5687.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"my2">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"90adfcfe">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5697.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a8">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"10143b12">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5710.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a3">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"6e1b943c">>,
   last_changed => <<"2018-08-10 04:26:27">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5681.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"my3">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"2a2a65b1">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5702.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a1">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"adc3c6e8">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5694.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a6">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"846eacce">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5684.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a9">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"86329bae">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5719.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a5">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"5c6b411b">>,
   last_changed => <<"2018-08-10 04:26:27">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5714.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"my6">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.75.217.27">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"68131859">>,
   last_changed => <<"2018-08-10 04:26:24">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5678.0>">>,

   queue => <<>>,status => running,type => exchange,upstream => <<"a10">>,
   upstream_exchange => <<"animal">>,upstream_queue => <<>>,
   uri => <<"amqp://10.76.6.215">>,vhost => <<"/">>},
 #{error => <<>>,exchange => <<"animal">>,id => <<"fc06dad9">>,
   last_changed => <<"2018-08-10 04:26:27">>,

   local_connection => <<"<rabbit@taozou-rabbitmq-4.3.5691.0>">>,

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[zou tao]

Hi MK,

Java performance client using no-ack mode to do performance.

but federation link try to ack for each message if not ack mode using.

The speed I have tested even low than 900 package/s. I also tried 4VCPU , i think

CPU resource is not the bottleneck here.

so the problem is even I could use multiple exchange upstream, but each exchange could

have only one upstream (federation) queue, right? so the bottleneck is that queue. if the target

exchange has more throughput than the queue,  it's hard to use the federation plugin to duplicate

the upstream exchange, right?  if one federation link could only support 3MB, should we configure

the capacity of one exchange no more than 3MB?

B.R,

Tao

在 2018年8月10日星期五 UTC+8下午4:57:50，Michael Klishin写道：

[zou tao]

I tried to send message from upstream exchange which has two federation link,  and the downstream queue only got one message,

so i don't understand why two federation link got replica streams. if that, the downstream queue should have two messages, right?

If two consumers subsribe one queue, the queue should use RR policy to deliver the message.

在 2018年8月10日星期五 UTC+8下午4:58:49，Michael Klishin写道：

[zou tao]

Hi MK,

I tried federation plugin on two different openstack enviroment. both 2VCPU, 8G RAM

and also the same configuration for federation plugin. and got

but  others are:

one federation link speed is about 5K, the other is about 500. any way to debug that or what kind of profile job I could do？

B.R,

Tao

在 2018年8月13日星期一 UTC+8下午4:09:23，zou tao写道：

[Michael Klishin]

I'm not sure what to say: automatic acknowledgements will in practice offer significantly higher throughput

than manual ones, especially with a low prefetch value.

Erlang has a bunch of profilers to choose from but when communicating over the network they are less useful as they only

show you a part of the overall picture. fprof, our own Looking Glass (currently incompatible with OTP 21) and more.

A traffic capture might reveal a comparable amount of interesting information.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[汤剑超]

您好在吗?我这边也遇到相似的问题,看了您这边的讨论,最后有比较好的解决方案吗?