TCP buffer size configuration option is not work as expected.

[Jeongwook Park]

Hello, I'm trying to optimize RabbitMQ instance for our environment (Large number of connections, few messages).

Thus, I tested various tcp_listen_options.sndbuf & recbuf.

For instance, I set snd & rec buffer size very low for the test.

- mqtt.tcp_listen_options.sndbuf = 1

- mqtt.tcp_listen_options.recbuf = 1

Also, I set the & rec buffer size very high for the test.

- mqtt.tcp_listen_options.sndbuf = 212992000 (~ 200MB)

- mqtt.tcp_listen_options.recbuf = 212992000

For both setups, I connect 100 concurrent MQTT clients & check the memory consumed.

However, consumed total memory (calculated by RSS strategy) was almost the same for both options!

I notice that the memory consumption was different in both cases when the calculation strategy was 'allocated' (1MB vs 20GB memory consumption for the above cases).

However, that does not affect the system performance since it was only virtually allocated memory not physically used by the system (I checked with 'top' command).

In conclusion, I can't see the effect of resizing TCP buffer size for reducing memory consumption. 

Are there any configurable options that I forgot to use? Then, please let me know.

[Luke Bakken]

Hello,

Always include the version of RabbitMQ and Erlang you're using, and on what operating system.

Please attach your complete configuration files, and not just excerpts from them.

[Jeongwook Park]

Thank you for the response.

I attached my system configuration below.

OS: Linux version 4.14.77-70.59.amzn1.x86_64 (mockbuild@gobi-build-64001) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC))

I ran RabbiMQ using docker (rabbit:3.7-management).

- Erlang version: 22

- RabbitMQ version: 3.7.15

<rabbitmq.conf>

loopback_users.guest = false

listeners.ssl.default = 5671

ssl_options.cacertfile = pem_file_path

ssl_options.certfile = pem_file_path

ssl_options.keyfile = pem_file_path

ssl_options.fail_if_no_peer_cert = true

ssl_options.verify = verify_none

default_pass = ****

default_user = ****

hipe_compile = false

management.listener.port = 15672

mqtt.allow_anonymous = false

mqtt.listeners.tcp.default = 1883

mqtt.listeners.ssl.default = 8883

mqtt.tcp_listen_options.sndbuf = 1

mqtt.tcp_listen_options.recbuf = 1

log.file.level = debug

vm_memory_high_watermark.absolute = 512MB

2019년 6월 27일 목요일 오후 11시 46분 58초 UTC+9, Luke Bakken 님의 말:

[Luke Bakken]

Hello,

I'm assuming you're running the MQTT clients on different servers than those running RabbitMQ. Is that correct?

sndbuf and recbuf set the minimum size of a TCP buffer but the operating system is allowed to choose one larger. Using a value like 1024 might make more sense as you've chosen an unrealistically small value.

Finally, please try the attached rabbitmq.config file instead of the configuration file you are using. It sets one more socket option that may be useful in your case. I have used it to test scenarios like yours.

If the above doesn't work the next step is to look at sysctl tuning

Thanks -

Luke