Celery Issue. Unable to enroll candidates, bulk mail or perform other queue bask tasks.

[Siddhant Singh]

Hi, 

    I have been trying to add some email ids from instructor panel repeatedly however it fails with error enrolling in users. The error log shows the below mentioned error. 

 Oct 29 05:55:52 ip-172-31-5-178 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-172-31-5-178  1330] [consumer.py:796] - consumer: Cannot connect to amqp://cel...@127.0.0.1:5672//: [Errno 104] Connection reset by peer.

Trying again in 32.00 seconds...

I have also tried telnet localhost 5672

which yields:

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

Connection closed by foreign host.

I have also tried updating edx-ora, however it has no effect. Please guide.

	P:+91-9555-192-192 www.nimt.ac.in
RANKED 11TH IN INDIA IN AIMA B SCHOOL SURVEY BY BUSINESS INDIA
SCHOOLS   HIGHER EDUCATION  DISTANCE & ONLINE EDUCATION

[Carlos Andrés Rocha]

Seems like a problem with RabbitMQ. What configuration/deployment strategy are you using? Can you verify if the RabbitMQ service is running?

--

Carlos Andrés Rocha
www.edx.org

[Siddhant Singh]

RabbitMQ seems to be running... I originally deployed it using the kifli ami on ubuntu 12.04. After this I have update the code base several times including one time updating the code base to aspen release across all repos including edx-platform, ora, xqueue etc..

Here is the response :

root@ip-172-31-5-178:/edx/app/edxapp/blocks# sudo service rabbitmq-server status

Status of node 'rabbit@ip-172-31-5-178' ...

[{pid,1190},

 {running_applications,

     [{rabbitmq_management,"RabbitMQ Management Console","3.2.3"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.3"},

      {rabbit,"RabbitMQ","3.2.3"},

      {os_mon,"CPO  CXC 138 46","2.2.7"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.3"},

      {webmachine,"webmachine","1.10.3-rmq3.2.3-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.3-git680dba8"},

      {xmerl,"XML parser","1.2.10"},

      {inets,"INETS  CXC 138 49","5.7.1"},

      {mnesia,"MNESIA  CXC 138 12","4.5"},

      {amqp_client,"RabbitMQ AMQP Client","3.2.3"},

      {sasl,"SASL  CXC 138 11","2.1.10"},

      {stdlib,"ERTS  CXC 138 10","1.17.5"},

      {kernel,"ERTS  CXC 138 10","2.14.5"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},

 {memory,

     [{total,32281512},

      {connection_procs,95416},

      {queue_procs,5408},

      {plugins,330288},

      {other_proc,9341976},

      {mnesia,59584},

      {mgmt_db,48896},

      {msg_index,34160},

      {other_ets,1067400},

      {binary,524984},

      {code,17199673},

      {atom,1778401},

      {other_system,1795326}]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,1550883225},

 {disk_free_limit,50000000},

 {disk_free,36326428672},

 {file_descriptors,

     [{total_limit,924},{total_used,6},{sockets_limit,829},{sockets_used,4}]},

 {processes,[{limit,1048576},{used,201}]},

 {run_queue,0},

 {uptime,77836}]

...done.

[Carlos Andrés Rocha]

Mmmm, weird. I am not sure what else you can do besides checking the obvious places. Have you tried using the RabbitMQ web interface, or looking at the RabbitMQ logs? There could be an indication of an error there.

[Siddhant Singh]

I am not sure about the web interface for rabbitmq https://www.rabbitmq.com/management.html mentions checking on http://localhost:15672. I will need to tunnel the localhost to check it. I will try that.

 However I checked the logs. There are many log files in rabbitmq folder:

root@ip-172-31-5-178:/edx/var/log/rabbitmq# ls
certificates.log
edX-Open_DemoX.log

edx.cms.core.default.%h.celery.pidbox.log

edx.cms.core.default.log

edx.cms.core.high.%h.celery.pidbox.log

edx.cms.core.high.log

edx.cms.core.low.%h.celery.pidbox.log

edx.cms.core.low.log

edx.lms.core.default.%h.celery.pidbox.log

edx.lms.core.default.log

edx.lms.core.high.%h.celery.pidbox.log

edx.lms.core.high.log

edx.lms.core.high_mem.%h.celery.pidbox.log

edx.lms.core.high_mem.log

edx.lms.core.low.%h.celery.pidbox.log

edx.lms.core.low.log

open-ended.log

I have checked all the logs. Nothing seemed out of place. 

All the logs had this line (with the queue name changed to respective queues). All the log files contained logs dating back to 23rd September. 

date='Tue Sep 23 14:06:02 UTC 2014' vhost='/' queue='edx.lms.core.low' length=0

[Carlos Andrés Rocha]

Hi Siddhant,

I am not sure what else to check. The connection error indicates to me that there is a RabbitMQ or a network configuration problem. I would start by trying to communicate directly to Rabbit from your LMS server. The RabbitMQ console tools are handy for this purpose: https://www.rabbitmq.com/management-cli.html 

What type of environment are you using? Production/Development stack? Are you running your own deployment?

[Siddhant Singh]

I am using the production stack. The rabbitmq plugin is to be used after sshing into the edx instance?