why rabbitmq generate lots of idx file in queue diretory

[Tony Wang]

Hi all

how to stop these large file generated automatically or keeping in a small size?

Details of question shows here:

http://stackoverflow.com/questions/42829641/why-rabbitmq-generate-lots-of-idx-file-in-queue-diretory

[Michael Klishin]

Consume messages so that they are deleted from queue indices. 

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

You can also purge or delete queues, of course. 

[Tony Wang]

Hi Michael 

Thanks for your reply first!

Lots of tutorials in celery or some other blogs shows that, the tasks script is the right way to consume task. But you said I should consume the tasks , so confusing here, what's the right way to "consume" task?

--

Tony Wang 

Warm Regards

[Michael Klishin]

You need to consume messages (with Celery or any other way).

See http://www.rabbitmq.com/getstarted.html and http://www.rabbitmq.com/tutorials/amqp-concepts.html if
the difference isn't clear.

I suspect that there may be a Celery worker that enqueues messages but they are never consumed.
It should be quite easy to see in the management UI. 

On 16 March 2017 at 12:49:30, Tony Wang (plantp...@gmail.com) wrote:
> Hi Michael
>
> Thanks for your reply first!
>
> Lots of tutorials in celery or some other blogs shows that, the tasks
> script is the right way to consume task. But you said I should consume the
> tasks , so confusing here, what's the right way to "consume" task?
>

[Tony Wang]

The code in tasks file should be 'consuming' the tasks, because lots of items has been inserted into database. I've checked the idx file, there are about 6 million tasks there and the size of each task is 260 bytes, So the total size should be 1487M not 7G, right?

def longtime_add(self,i):

     post.insert({'status':r.status_code,"creat_time":time.time()})   

[Michael Klishin]

I don't think that QI segment structure is worth discussing here. Your question is how to free up disk space.

The answer is:

 * Consume messages and acknowledge them

OR

 * Purge or delete queues 

[Tony Wang]

Hi Michael 

Sorry for that, Maybe it's not good description for such question.

More details here. 

1. Screenshot of celery flower, about 15k tasks has been processed sucessfully.

2. Status of RabbitMQ server

root@rabbit:/# rabbitmqctl status

Status of node rabbit@rabbit ...

[{pid,232},

 {running_applications,[{rabbit,"RabbitMQ","3.6.6"},

                        {rabbit_common,[],"3.6.6"},

                        {mnesia,"MNESIA  CXC 138 12","4.14.2"},

                        {os_mon,"CPO  CXC 138 46","2.4.1"},

                        {xmerl,"XML parser","1.3.12"},

                        {ranch,"Socket acceptor pool for TCP protocols.",

                               "1.2.1"},

                        {sasl,"SASL  CXC 138 11","3.0.2"},

                        {stdlib,"ERTS  CXC 138 10","3.2"},

                        {kernel,"ERTS  CXC 138 10","5.1.1"}]},

 {os,{unix,linux}},

 {erlang_version,"Erlang/OTP 19 [erts-8.2] [source] [64-bit] [smp:2:2] [async-threads:64] [hipe] [kernel-poll:true]\n"},

 {memory,[{total,294747360},

          {connection_readers,2776264},

          {connection_writers,509368},

          {connection_channels,6532480},

          {connection_other,2148656},

          {queue_procs,24164688},

          {queue_slave_procs,0},

          {plugins,0},

          {other_proc,9793744},

          {mnesia,77640},

          {mgmt_db,0},

          {msg_index,52488},

          {other_ets,1098736},

          {binary,208044976},

          {code,17814931},

          {atom,752561},

          {other_system,20980828}]},

 {alarms,[]},

 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,838957465},

 {disk_free_limit,50000000},

 {disk_free,10583482368},

 {file_descriptors,[{total_limit,1048476},

                    {total_used,121},

                    {sockets_limit,943626},

                    {sockets_used,118}]},

 {processes,[{limit,1048576},{used,1231}]},

 {run_queue,2},

 {uptime,1832},

 {kernel,{net_ticktime,60}}]

My question is not just about how to free up disk, it's about how to figure it out. You said that , the celery worker isn't consuming tasks, But the screenshot of celery flowers shows that it does consume. 

I'm new to RabbitMQ, so could help me how to figure out whether the worker is consuming tasks or not ? What else details you need for such case ?

Thanks!

[Michael Klishin]

This is not a Celery forum. I personally know very little about Celery.

I've already mentioned that you can inspect actual queues using RabbitMQ management UI:

http://rabbitmq.com/management.html

You can also navigate to individual queue pages and delete or purge them.

`rabbitmqctl` can be used to list queues with their various stats:

https://www.rabbitmq.com/man/rabbitmqctl.1.man.html

> > > > an email to rabbitmq-users+unsubscribe@googlegroups.com.

> > > > > > To post to this group, send an email to

> > > > rabbitmq-users@googlegroups.com.

[Michael Klishin]

I said that I *suspect* that there are queues that have messages routed to them but

never consumed (and that's how you've accumulated many gigabytes of data on disk).

This is trivial to inspect using `rabbitmqctl list_queues` for a single vhost and management UI

for all of them (provided your user has sufficient permissions).

> > > > rabbitm...@googlegroups.com.

[Tony Wang]

The RabbitMQ is running in a docker container. I can't stop it and port it to another port. So management UI is not a good idea for this case, but still thanks for that , I will configure it next time.

That's what I got by rabbitmqctl

root@rabbit:/# rabbitmqctl -q list_queues name messages messages_ready messages_unacknowledged                      

cel...@i-atbxncfv.celery.pidbox 0 0 0

celery 701918 699887 2031

celeryev.88e4b508-6559-4194-98a2-36874e9741c2 0 0 0

celeryev.10f9ede2-e0fb-4df9-9b95-227c6983b709 0 0 0

celeryev.073840d4-dbb2-4e1c-b0ad-7f2ae0b8cb50 0 0 0

cel...@1e35e1104315.celery.pidbox 0 0 0

> > > > an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

You have 699887 messages ready for delivery and another 2031 unacknowledged in the queue named `celery`.

> > > > an email to rabbitmq-users+unsubscribe@googlegroups.com.

> > > > > > To post to this group, send an email to

> > > > rabbitmq-users@googlegroups.com.

[Tony Wang]

So you mean that the 699887 messages need to be consumed by worker?  Is there any cmds to inspect how many tasks has been consumed? Perhaps the speed of producing is much faster than consuming.

> > > > an email to rabbitmq-user...@googlegroups.com.

> > > > > > To post to this group, send an email to

> > > > rabbitm...@googlegroups.com.

[Michael Klishin]

Tony,

Please understand that RabbitMQ knows absolutely nothing about Celery or its tasks.

Celery is just one tool that uses RabbitMQ as a temporary storage, routing, delivery service.

RabbitMQ operates in terms of queues and messages.

Yes, there are 700K messages that are not consumed. I don't know why. You either can purge

the queue, delete it (using any RabbitMQ client or HTTP API) or consume them.

In case Celery keeps a log, it's worth inspecting to see why you have 700K messages

enqueued and not consumed (or not consumed fast enough)

> > > > an email to rabbitmq-users+unsubscribe@googlegroups.com.

> > > > > > To post to this group, send an email to

> > > > rabbitmq-users@googlegroups.com.

[Tony Wang]

Thanks Michael for you patient answer. I will check it now as you said.

> > > > an email to rabbitmq-user...@googlegroups.com.

> > > > > > To post to this group, send an email to

> > > > rabbitm...@googlegroups.com.

> > > > > > For more options, visit https://groups.google.com/d/optout.
> > > > > >
> > > > >
> > > > > --
> > > > > MK
> > > > >
> > > > > Staff Software Engineer, Pivotal/RabbitMQ
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > MK
> > > >
> > > > Staff Software Engineer, Pivotal/RabbitMQ
> > > >
> > > >
> > > > --
> > >
> > > Tony Wang
> > >
> > > Warm Regards
> > >
> >
> > --
> > MK
> >
> > Staff Software Engineer, Pivotal/RabbitMQ
> >
> >
> > --
>
> Tony Wang
>
> Warm Regards
>

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

--

Tony Wang