The broker has blocked the connection: low on disk

[Margin Hu]

Hi Guys,

I met an issue that cinder-volume serivce was reporting as following

2017-06-28 16:58:48.552 7 INFO oslo.messaging._drivers.impl_rabbit [req-45ffcbf7-0a73-433f-8559-8408bff3e3c5 - - - - -] [189f008b-f46a-4fff-9397-6a06dbe093d0] Reconnected to AMQP server on 10.53.16.10:5672 via [amqp] client with port 37760.
2017-06-28 16:58:48.554 7 ERROR oslo.messaging._drivers.impl_rabbit [req-45ffcbf7-0a73-433f-8559-8408bff3e3c5 - - - - -] The broker has blocked the connection: low on disk

but the disk free capability is very large on rabbitmq nodes.

[root@cloud-production-01 cinder]# df -h  | head -n 10
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/cl-root             50G  5.5G   45G  11% /
devtmpfs                       126G     0  126G   0% /dev
tmpfs                          126G     0  126G   0% /dev/shm
tmpfs                          126G   21M  126G   1% /run
tmpfs                          126G     0  126G   0% /sys/fs/cgroup
/dev/sda3                      100G   19G   81G  19% /var
/dev/mapper/cl-home            100G  2.8G   98G   3% /home
/dev/sda1                      497M  182M  316M  37% /boot

version:
rabbitmq-server-3.6.5-1.el7.noarch


rabbit@cloud-production-b12-01 /]# rabbitmqctl  status
Status of node 'rabbit@cloud-production-b12-01' ...
[{pid,261},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.6.5"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.5"},
      {rabbit,"RabbitMQ","3.6.5"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.5"},
      {webmachine,"webmachine","1.10.3"},
      {mochiweb,"MochiMedia Web Server","2.13.1"},
      {ssl,"Erlang/OTP SSL application","7.3.3.1"},
      {amqp_client,"RabbitMQ AMQP Client","3.6.5"},
      {rabbit_common,[],"3.6.5"},
      {ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},
      {public_key,"Public key infrastructure","1.1.1"},
      {inets,"INETS  CXC 138 49","6.2.4"},
      {asn1,"The Erlang ASN1 compiler version 4.0.2","4.0.2"},
      {os_mon,"CPO  CXC 138 46","2.4"},
      {mnesia,"MNESIA  CXC 138 12","4.13.4"},
      {xmerl,"XML parser","1.3.10"},
      {syntax_tools,"Syntax tools","1.7"},
      {compiler,"ERTS  CXC 138 10","6.0.3"},
      {crypto,"CRYPTO","3.6.3"},
      {rabbitmq_clusterer,"Declarative RabbitMQ clustering",[]},
      {sasl,"SASL  CXC 138 11","2.7"},
      {stdlib,"ERTS  CXC 138 10","2.8"},
      {kernel,"ERTS  CXC 138 10","4.2"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang/OTP 18 [erts-7.3.1.2] [source] [64-bit] [smp:56:56] [async-threads:896] [hipe] [kernel-poll:true]\n"},
 {memory,
     [{total,284303752},
      {connection_readers,7150720},
      {connection_writers,2166552},
      {connection_channels,5411552},
      {connection_other,23567800},

rabbitmq log can be found in attachement.
      {queue_procs,8549800},
      {queue_slave_procs,959176},
      {plugins,998584},
      {other_proc,24990176},
      {mnesia,1673456},
      {mgmt_db,2808},
      {msg_index,209600},
      {other_ets,2416664},
      {binary,128509296},
      {code,27777792},
      {atom,1000601},
      {other_system,48919175}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"10.53.16.12"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,107519852544},
 {disk_free_limit,50000000},
 {disk_free,86859206656},
 {file_descriptors,
     [{total_limit,65436},
      {total_used,782},
      {sockets_limit,58890},
      {sockets_used,780}]},
 {processes,[{limit,1048576},{used,10087}]},
 {run_queue,0},
 {uptime,66970},
 {kernel,{net_ticktime,60}}]
rabbit@cloud-production-b12-01 /]# rabbitmqctl  cluster_status
Cluster status of node 'rabbit@cloud-production-b12-01' ...
[{nodes,[{disc,['rabbit@cloud-production-b12-01',
                'rabbit@cloud-production-b12-02',
                'rabbit@cloud-production-b12-03']}]},
 {running_nodes,['rabbit@cloud-production-b12-02',
                 'rabbit@cloud-production-b12-01']},
 {cluster_name,<<"rabbit@cloud-production-b12-01">>},
 {partitions,[]},
 {alarms,[{'rabbit@cloud-production-b12-02',[]},
          {'rabbit@cloud-production-b12-01',[]}]}]

logs can be found in attachement.

[Michael Klishin]

Free disk space monitoring can tell you more.

I don't think I have ever seen a false positive around free disk space, it is parsed from df output:

df -kP /path/to/node/directory

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Staff Software Engineer, Pivotal/RabbitMQ

[Margin Hu]

I haven't seen any full info.

[Margin Hu]

I found one node within 3 nodes  has exception.



/var is a btrfs partition.

# mount | grep sda
/dev/sda1 on /boot type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sda2 on /var type btrfs (rw,relatime,ssd,space_cache,subvolid=257,subvol=/var)
/dev/sda2 on /var/lib/docker/btrfs type btrfs (rw,relatime,ssd,space_cache,subvolid=257,subvol=/var/lib/docker/btrfs)

 #btrfs fi show
Label: 'cl00'  uuid: 3d7edf01-42a5-4f7e-b7a2-581d60f7d04b
        Total devices 1 FS bytes used 16.69GiB
        devid    1 size 100.00GiB used 100.00GiB path /dev/sda2

I will fix the file system first.

On Thursday, June 29, 2017 at 11:49:54 AM UTC+8, Michael Klishin wrote:

[Michael Klishin]

RabbitMQ node data on *nix systems is often stored under /var/lib/ by default.

"Disk is full alarms" are very visible in node logs.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ