Error when trying to start celery

[Angela Labrador]

Hello all,

I'm testing an upgrade to Arches 5.1 and have hit a stumbling block with trying to get celery running. I've followed the directions on the doc, and have RabbitMQ installed and my settings.py file updated. I did not create a tasks.py file within my project folder because the docs imply this is only needed if I need to add additional tasks for my project (I just need the default Arches tasks). 

When I run python manage.py celery start from within my virtualenv, I get 

Error:
Unable to load celery application.
The module XXX was not found.  

where XXX = my settings.ELASTICSEARCH_PREFIX

However, when I edit the celery.py management command to use 

cmd = f"celery -A arches worker -B -l info"  

instead of settings.ELASTICSEARCH_PREFIX, celery starts up and connects to rabbitmq. 

I'm wondering why that management command is using the ES prefix and what I could be missing in my own project config that is causing this.

Thanks,

Angela

[Lindsey Gant]

Hi Angela,

Were you able to find a solution to your celery error? If not, let me see if someone in the group can follow up.

Thanks,

Lindsey

GCI Arches Project Team

[Angela Labrador]

Hi Lindsay,

Thanks for checking in -- no, I haven't really solved it. I've got a workaround, but would appreciate any ideas or explanations so I can understand how celery is supposed to be configured (or so that we can update the docs if they need to be edited).

Angela

[Alexei Peters]

Hi Angela,

I assume that you're running the "python manage.py celery start" command from your project and not from the root of the Arches code base correct?
The command also assumes that you haven't rename your folders in your project directory.

-Alexei

Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

--
-- To post, send email to arches...@googlegroups.com. To unsubscribe, send email to archesprojec...@googlegroups.com. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups "Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archesprojec...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/17891260-c650-469a-bdff-9e68f9b5b462n%40googlegroups.com.

[Angela Labrador]

Hi Alexei,

Correct. I am running python manage.py celery start from within [my_project] folder, as per docs, which results in the 

Error:

Unable to load celery application.

The module [ my settings.ELASTICSEARCH_PREFIX ] was not found.

 

My project folder structure is standard -- no renaming of anything. 

Thanks,

Angela

[Alexei Peters]

Hi Angela,

Just to be clear the command should be run from the directory where the "manage.py" file is located.

EG: you CAN'T do this --> python somedir/manage.py celery start

Also, does your celery file look something like this?

from __future__ import absolute_import, unicode_literals

import os

from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', '{project_name}.settings')

app = Celery('{project_name}')

app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()

where {project_name} is the name of the directory where your project exists

Cheers,

Alexei

Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/4a95537c-db9b-4935-8144-37e930706511n%40googlegroups.com.

[Angela Labrador]

Hi Alexei,

Yep, I'm running the command local to where the manage.py file is located (running the command just like any other python manage.py command with no dir in front of the command).

But -- I think you just identified the problem! I do not have a celery.py file within my-project folder besides the celery.py files in media/packages/arches. This is a project that has been migrated from 4.0 through the major versions over some time. So, maybe it never created the celery.py file during one of those migrations? 

I jumped on another virtual machine and installed a brand new arches 5.1 project, and it did create a celery.py file with my project name as you pasted. 

Just for reference, on my migrated project, running 

find -iname celery.py

results in:

./[my-project]/[my-project]/media/packages/arches/arches/management/commands/celery.py

./[my-project]/[my-project]/media/packages/arches/arches/celery.py

./[my-project]/static/packages/arches/tests/fixtures/testing_prj/testing_prj/celery.py

./[my-project]/static/packages/arches/arches/management/commands/celery.py

./[my-project]/static/packages/arches/arches/celery.py

./env/lib/python3.7/site-packages/celery/bin/celery.py

./arches/tests/fixtures/testing_prj/testing_prj/celery.py

./arches/arches/management/commands/celery.py

./arches/arches/celery.py

So, should I just copy the contents of the celery.py file from the fresh project to my migrated project? And do you think this is an anomaly or a "feature" of migrating from much earlier versions? If the latter, should we add something to the docs to cover this?

Thank you for sticking with this,

Angela

[Alexei Peters]

Hi Angela,

It looks like you identified the issue.  Older projects weren't created with that celery file and that explains the problem you're seeing.

Copying that file over to your project should solve the problem.  

This was most likely an oversight in our upgrade announcement (although I can't be sure).  I'm actually not sure how/if this should be documented.  Suggestions are welcome there!  :)

Glad we solved this!

Cheers,

Alexei

Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/78043179-6107-4ac3-b597-7184cb61a571n%40googlegroups.com.

[Angela Labrador]

Hi Alexei,

I'm back! I have copied the new celery.py file over into my project directory as suggested above, however, I still get the same error. 

Error:

Unable to load celery application.
The module XXX was not found.  

where XXX = {settings.ELASTICSEARCH_PREFIX}

The only celery management command that I have is in the arches/arches/management/commands/ directory, and I see that is using the elasticsearch_prefix variable on lines 41 and 43 and not the project name. When I edit those lines manually and change them to use my project's name, I can run the command fine. 

Am I supposed to have a separate celery management command within my project folder or is this a bug with the arches management command?

Thanks,

Angela

[Alexei Peters]

Hi Angela,

After you copy that file over, you'll need to edit a couple of lines in that file:

THESE LINES:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
app = Celery("arches")

SHOULD CHANGE TO THIS:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "{my project name}.settings")
app = Celery("{my project name}")

Hope that helps!

-Alexei

Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/a116bc89-5359-4c30-a832-e58701240284n%40googlegroups.com.

[Angela Labrador]

Hi Alexei,

Thanks but I copied the celery.py file from the Arches template folder so it already had those lines in it. My file now looks like yours from the earlier reply. 

The issue is that the actual management command that starts celery never uses that app variable. Instead it uses the elastic search prefix — see lines 41 and 43. Because my elastic search prefix is different from my app name, I get an error.  Can you please try retesting by setting your elastic search prefix setting to something that doesn’t equal your project name to see if you hit the same error as me?

Thanks, 

Angela

[Alexei Peters]

Hi Angela,

Sorry, I see now what you're talking about.  I wasn't looking at the management command.  So, having that set to use the ELASTICSEARCH_PREFIX is probably a mistake.

Instead of running the command you can run what the command runs.

"celery -A {your project name} worker -l info"

That should work.

Cheers,

-Alexei

Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

[Angela Labrador]

Hi Alexei,

Ok that workaround makes sense. Thanks for confirming! I will file an issue on GitHub for this too (unless you beat me to it). 

Enjoy your weekend,

Angela