Celery problems

[Ethan Kiczek]

Hi Open edX operations,

I am running a working single instance full stack on an Amazon Web Services EC2.

Recently, I took an AMI snapshot of the image and spun up a new EC2 with this AMI image.

On this new, cloned machine, I get errors in /edx/var/log/lms/edx.log like:

May 12 20:52:01 ip-10-39-83-123 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-10-39-83-123  1347] [consumer.py:796] - consumer: Cannot connect to amqp://cel...@127.0.0.1:5672//: [Errno 104] Connection reset by peer.

Trying again in 2.00 seconds...

So I restarted services by following instructions at https://github.com/edx/configuration/wiki/edX-Managing-the-Full-Stack, specifically running the commands specified at https://github.com/edx/configuration/wiki/edX-Managing-the-Full-Stack#restarting-services.

After restart, I get a variation on the above error:

May 13 14:01:01 ip-10-39-83-123 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-10-39-83-123  1297] [consumer.py:364] - consumer: Cannot connect to amqp://celery:**@127.0.0.1:5672//: [Errno 104] Connection reset by peer.

Trying again in 32.00 seconds...

May 13 14:01:02 ip-10-39-83-123 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-10-39-83-123  1266] [consumer.py:364] - consumer: Cannot connect to amqp://celery:**@127.0.0.1:5672//: [Errno 104] Connection reset by peer.

Trying again in 32.00 seconds...

May 13 14:01:02 ip-10-39-83-123 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-10-39-83-123  1264] [consumer.py:364] - consumer: Cannot connect to amqp://celery:**@127.0.0.1:5672//: [Errno 104] Connection reset by peer.

Trying again in 32.00 seconds...

May 13 14:01:02 ip-10-39-83-123 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-10-39-83-123  1253] [consumer.py:364] - consumer: Cannot connect to amqp://celery:**@127.0.0.1:5672//: [Errno 104] Connection reset by peer.

Trying again in 32.00 seconds...

Can you help me troubleshoot and fix this? I am fairly new to both Open edX and celery.

Thanks,

Ethan

[Pierre Mailhot]

Hi Ethan,

I ran into this problem a while back. Celery / RabbitMQ doesn't like to have its IP changed.

Ed Zarecor from edX previously sent me these instructions:

sudo rabbitmqctl status

Let's try this and see if it gets rabbitmq happy again.

sudo bash
. /edx/app/edx_ansible/venvs/edx_ansible/bin/activate
cd /edx/app/edx_ansible/edx_ansible/playbooks/
ansible-playbook -c local -i 'localhost,' ./run_role.yml -e "role=rabbitmq" -e@/edx/app/edx_ansible/server-vars.yml

Afterwards, I found I needed to do this in order to complete the patch:

The problem was that when rabbitmq was reinstalled with the ansible-playbook, it kept the IP address of the server in /etc/rabbitmq/rabbitmq-env.conf 

I simply had to change the value of 

RABBITMQ_NODE_IP_ADDRESS from 10.0.0.35 to 127.0.0.1

I was wondering why I couldn't connect on localhost and why my error messages in /edx/var/log/lms were of the type

Jan 15 20:09:10 ip-10-0-0-35 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-10-0-0-35  1508] [consumer.py:796] - consumer: Cannot connect to amqp://cel...@127.0.0.1:5672//: [Errno 111] Connection refused.

[Ethan Kiczek]

Hi Pierre,

I tried this before but I don't think I performed that crucial last step of modifying the RABBITMQ_NODE_IP_ADDRESS in /etc/rabbitmq/rabbitmq-env.conf. I just did that, rebooted the machine and it worked!

Thanks,

E-

On Wednesday, May 13, 2015 at 10:09:33 AM UTC-4, Ethan Kiczek wrote:

[Pierre Mailhot]

I am happy I was able to help. 

Regards,

Pierre

[Apssdc Gov]

am getting the bellow error can any one help me out , am unable to get discussion forum after diving in to database and lms 

Aug 25 17:22:41 ip-172-31-17-126 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-172-31-17-126  1656] [consumer.py:364] - consumer: Cannot connect to amqp://celery:**@lms.apssdc.in:5672//: Socket closed.

Trying again in 32.00 seconds...

Aug 25 17:22:41 ip-172-31-17-126 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-172-31-17-126  1646] [consumer.py:364] - consumer: Cannot connect to amqp://celery:**@lms.apssdc.in:5672//: Socket closed.

Trying again in 32.00 seconds...

Aug 25 17:22:42 ip-172-31-17-126 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-172-31-17-126  1651] [consumer.py:364] - consumer: Cannot connect to amqp://celery:**@lms.apssdc.in:5672//: Socket closed.

Trying again in 32.00 seconds...

Aug 25 17:22:42 ip-172-31-17-126 [service_variant=lms][celery.worker.consumer][env:sandbox] ERROR [ip-172-31-17-126  1642] [consumer.py:364] - consumer: Cannot connect to amqp://celery:**@lms.apssdc.in:5672//: Socket closed.

Trying again in 32.00 seconds...

[gaurav k]

i am having isssue with creating and editing the courses .  Below error is coming -

OpenEDXv2 [service_variant=cms][django.request][env:sandbox] ERROR [OpenEDXv2  25659] [base.py:256] - Internal Server Error: /course/course-v1:edX+DemoX+Demo_Course

Traceback (most recent call last):

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 145, in inner

    return func(*args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view

    return view_func(request, *args, **kwargs)

  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/views/course.py", line 266, in course_handler

    return course_index(request, CourseKey.from_string(course_key_string))

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view

    return view_func(request, *args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 110, in _wrapped_view

    response = view_func(request, *args, **kwargs)

  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/views/course.py", line 609, in course_index

    deprecated_blocks_info = _deprecated_blocks_info(course_module, deprecated_block_names)

  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/views/course.py", line 569, in _deprecated_blocks_info

    structure_data = api.course_structure(course_module.id, block_types=deprecated_block_types)

  File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/content/course_structures/api/v0/api.py", line 104, in course_structure

    tasks.update_course_structure.delay(unicode(course_key))

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/celery/app/task.py", line 453, in delay

    return self.apply_async(args, kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/celery/app/task.py", line 559, in apply_async

    **dict(self._get_exec_options(), **options)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/celery/app/base.py", line 353, in send_task

    reply_to=reply_to or self.oid, **options

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/celery/app/amqp.py", line 305, in publish_task

    **kwargs

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/messaging.py", line 172, in publish

    routing_key, mandatory, immediate, exchange, declare)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/connection.py", line 470, in _ensured

    interval_max)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/connection.py", line 382, in ensure_connection

    interval_start, interval_step, interval_max, callback)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time

    return fun(*args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/connection.py", line 250, in connect

    return self.connection

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/connection.py", line 756, in connection

    self._connection = self._establish_connection()

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/connection.py", line 711, in _establish_connection

    conn = self.transport.establish_connection()

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection

    conn = self.Connection(**opts)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/amqp/connection.py", line 180, in __init__

    (10, 30),  # tune

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait

    self.channel_id, allowed_methods, timeout)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/amqp/connection.py", line 241, in _wait_method

    channel, method_sig, args, content = read_timeout(timeout)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/amqp/connection.py", line 330, in read_timeout

    return self.method_reader.read_method()

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method

    raise m

error: [Errno 104] Connection reset by peer

On Wednesday, May 13, 2015 at 10:09:33 AM UTC-4, Ethan Kiczek wrote:

[Trinh Nguyen]

Check your celery connection. Make sure the user and password are correct. That's what the error says.

Trinh Nguyen | Founder & CTO

E: dangt...@gmail.com | W: www.edlab.xyz | T: +84916298737

--
You received this message because you are subscribed to the Google Groups "Open edX operations" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openedx-ops+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openedx-ops/6bc65028-9515-498d-833e-b730939f5473%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[gaurav k]

Thanks Trinh. i am very new to this. can u pls tell me where to check celery connection - in lms/cms json files?  

i just checked user register also not working and giving me below errror, it looks same error coming every where -

Failed to load resource: the server responded with a status of 500 (INTERNAL SERVER ERROR)

register:9 Uncaught SyntaxError: Unexpected token < in JSON at position 8

    at JSON.parse (<anonymous>)

    at child.saveError (logistration_factory.39b8b2b9e9a9.js?raw:1)

    at triggerEvents (lms-base-vendor.15f98b938c5a.js:10)

    at triggerApi (lms-base-vendor.15f98b938c5a.js:10)

    at eventsApi (lms-base-vendor.15f98b938c5a.js:10)

    at child.Events.trigger (lms-base-vendor.15f98b938c5a.js:10)

    at Object.error (logistration_factory.39b8b2b9e9a9.js?raw:1)

    at fire (lms-base-vendor.15f98b938c5a.js:2)

    at Object.fireWith [as rejectWith] (lms-base-vendor.15f98b938c5a.js:2)

    at done (lms-base-vendor.15f98b938c5a.js:4)

saveError @ logistration_factory.39b8b2b9e9a9.js?raw:1

triggerEvents @ lms-base-vendor.15f98b938c5a.js:10

triggerApi @ lms-base-vendor.15f98b938c5a.js:10

eventsApi @ lms-base-vendor.15f98b938c5a.js:10

Events.trigger @ lms-base-vendor.15f98b938c5a.js:10

error @ logistration_factory.39b8b2b9e9a9.js?raw:1

fire @ lms-base-vendor.15f98b938c5a.js:2

fireWith @ lms-base-vendor.15f98b938c5a.js:2

done @ lms-base-vendor.15f98b938c5a.js:4

(anonymous) @ lms-base-vendor.15f98b938c5a.js:4

To unsubscribe from this group and stop receiving emails from it, send an email to openedx-ops...@googlegroups.com.

[gaurav k]

Thanks, i fixed it.

[Trinh Nguyen]

How did you fix it?

Trinh Nguyen | Founder & CTO

E: dangt...@gmail.com | W: www.edlab.xyz | T: +84916298737

To unsubscribe from this group and stop receiving emails from it, send an email to openedx-ops+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openedx-ops/4c61cbd3-9206-4b5d-9faa-962293738331%40googlegroups.com.

[gaurav k]

i deleted celery user and created again , then it worked 

[Trinh Nguyen]

Cool.

Trinh Nguyen | Founder & CTO

E: dangt...@gmail.com | W: www.edlab.xyz | T: +84916298737

To unsubscribe from this group and stop receiving emails from it, send an email to openedx-ops+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openedx-ops/85551492-9c0f-42b1-897a-07fa98fd3657%40googlegroups.com.

[gaurav k]

Hi Trinh,

I am facing another issue, after course enrollment when user login in to  his  account then below error comes. i  need help..

OpenEDXv2 [service_variant=lms][django.request][env:sandbox] ERROR [OpenEDXv2  50154] [base.py:256] - Internal Server Error: /course_modes/choose/course-v1:Microsoft+DAT207x+20170622/

Traceback (most recent call last):

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view

    return self.dispatch(request, *args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 34, in _wrapper

    return bound_func(*args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 30, in bound_func

    return func.__get__(self, type(self))(*args2, **kwargs2)

  File "/edx/app/edxapp/edx-platform/common/djangoapps/course_modes/views.py", line 56, in dispatch

    return super(ChooseModeView, self).dispatch(*args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch

    return handler(request, *args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 34, in _wrapper

    return bound_func(*args, **kwargs)

  File "/edx/app/edxapp/edx-platform/common/djangoapps/third_party_auth/decorators.py", line 87, in inner

    return func(request, *args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 30, in bound_func

    return func.__get__(self, type(self))(*args2, **kwargs2)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 34, in _wrapper

    return bound_func(*args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view

    return view_func(request, *args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 30, in bound_func

    return func.__get__(self, type(self))(*args2, **kwargs2)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 34, in _wrapper

    return bound_func(*args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 145, in inner

    return func(*args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/decorators.py", line 30, in bound_func

    return func.__get__(self, type(self))(*args2, **kwargs2)

  File "/edx/app/edxapp/edx-platform/common/djangoapps/course_modes/views.py", line 120, in get

    return redirect(self._get_redirect_url_for_audit_enrollment(request, course_id))

  File "/edx/app/edxapp/edx-platform/common/djangoapps/course_modes/views.py", line 298, in _get_redirect_url_for_audit_enrollment

    enterprise_learner_data = enterprise_api.get_enterprise_learner_data(site=request.site, user=request.user)

  File "/edx/app/edxapp/edx-platform/openedx/features/enterprise_support/api.py", line 391, in get_enterprise_learner_data

    enterprise_learner_data = EnterpriseApiClient().fetch_enterprise_learner_data(site=site, user=user)

  File "/edx/app/edxapp/edx-platform/openedx/features/enterprise_support/api.py", line 54, in __init__

    self.user = User.objects.get(username=settings.ENTERPRISE_SERVICE_WORKER_USERNAME)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method

    return getattr(self.get_queryset(), name)(*args, **kwargs)

  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/models/query.py", line 334, in get

    self.model._meta.object_name

DoesNotExist: User matching query does not exist.

 OpenEDXv2 [service_variant=lms][openedx.core.lib.log_utils][env:sandbox] INFO [OpenEDXv2  50149] [log_utils.py:36] - ApiKeyHeaderPermission used: ip="127.0.0.1", path="/notifier_api/v1/users/"

Cool.

[Trinh Nguyen]

Properly related to the userprofile.

To view this discussion on the web visit https://groups.google.com/d/msgid/openedx-ops/cc4a7c0b-dc5d-4cd4-a9dc-0ed27121f8d2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[Trinh Nguyen]

Try register a new user through the LMS's interface and test.

[gaurav k]

I created through LMS but still same issue -

2017-08-03 04:57:53,999 ERROR 17357 [django.request] base.py:256 - Internal Server Error: /course_modes/choose/course-v1:Microsoft+DAT207x+20170622/

[Trinh Nguyen]

Hi,

Reading your logs carefully, I can see it related to your Enterprise Service settings. Please read these documents and check your configuration:

https://media.readthedocs.org/pdf/open-edx-enterprise-service-documentation/latest/open-edx-enterprise-service-documentation.pdf

http://open-edx-enterprise-service-documentation.readthedocs.io/en/latest/configuration_and_usage.html

I'm not familiar with the service myself but it may give you some clues.

Cheers,

Trinh Nguyen | Founder & CTO

E: dangt...@gmail.com | W: www.edlab.xyz | T: +84916298737

To unsubscribe from this group and stop receiving emails from it, send an email to openedx-ops+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openedx-ops/695f2cec-9cc6-4f41-9e0b-3a994ed0a20b%40googlegroups.com.

[gaurav k]

Thanks, let me try this. 

[YusufSalahAdDin]

How did you it?

[gaurav k]

i deleted celery user and created again , then it worked 

[wasim...@dfmail.org]

Hi we are facing one issue while registering a new user, we are unable to register a new user in console it is showing 500 internal server error.

Everything was working fine the user registeration and all but after restoring we are unable to register.

If any could help it will be great.

For reference i have attached screen shot 

[René Rolando Elizalde Solano]

This solution worked for me

Good!