Running Celery 4+ / Outdated Installation Guide

[Chris Ward]

Hi Everyone,

I have just started to perform some testing for my deployed OTM instance, I noticed that the celeryd service isn't running on my server. Trying to run python /usr/local/otm/app/opentreemap/manage.py celeryd always returned with an error saying that celeryd was not a valid option in manage.py. I looked in the requirements.txt file and noticed that django-celery isn't included. After some digging, it looks like the installation guide may be out of date. 

It looks like a PR from August 17, 2017 (https://github.com/OpenTreeMap/otm-core/pull/3184/commits/1147a7e3a8ad5d4f60a7f4b5e5c80857795f7df2) upgraded celery to version 4+ from 3.1.x and as part of this upgrade, dropped the django-celery package as well. 

Looking at the difference between the versions, the most notable is the built-in support for django with celery, which allowed the team to drop the django-celery package, meaning the 'celeryd' option went away from manage.py. The Installation Guide still references using a service calling manage.py, but it seems like the proper way would be to call celery itself with something similar to  celery -A opentreemap worker -l info. 

Has anyone determined the proper way to start up celery using this new method?

I've tried running  celery -A opentreemap -b redis://$BROKER_URL/2 worker -l info, where BROKER_URL is the same value as from the opentreemap/settings/local_settings.py file. 

This seems to start celery up fine: 

 -------------- celery@<hostname> v4.1.0 (latentcall)

---- **** -----

--- * ***  * -- Linux-4.4.0-138-generic-x86_64-with-Ubuntu-16.04-xenial 2018-11-25 03:53:45

-- * - **** ---

- ** ---------- [config]

- ** ---------- .> app:         opentreemap:0x7f38a1fa6fd0

- ** ---------- .> transport:   redis://$BROKER_URL/2

- ** ---------- .> results:     redis://$BROKER_URL/

- *** --- * --- .> concurrency: 2 (prefork)

-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)

--- ***** -----

 -------------- [queues]

                .> Black            exchange=Black(direct) key=task.Black

[tasks]

  . appevents.tasks.handle_event

  . appevents.tasks.queue_events_to_be_handled

  . exporter.tasks.async_csv_export

  . exporter.tasks.async_users_export

  . exporter.tasks.custom_async_csv

  . exporter.tasks.simple_async_csv

  . importer.tasks._commit_rows

  . importer.tasks._finalize_commit

  . importer.tasks._finalize_validation

  . importer.tasks._validate_rows

  . importer.tasks.commit_import_event

  . importer.tasks.get_import_export

  . importer.tasks.run_import_event_validation

[2018-11-25 03:53:45,374: INFO/MainProcess] Connected to redis://$BROKER_URL/2

[2018-11-25 03:53:45,379: INFO/MainProcess] mingle: searching for neighbors

[2018-11-25 03:53:46,393: INFO/MainProcess] mingle: all alone

[2018-11-25 03:53:46,402: INFO/MainProcess] celery@<hostname>  ready.

However, when trying to perform an import or an export through the OTM site, the operation hangs and eventually returns with a 504 gateway timeout error. Using rollbar, I've been able to trace this to the  run_import_event_validation function call located at opentreemap/importer/views.py:722. This makes me think that something may be wrong with my celery command parameters.

tail /var/log/syslog shows:

Nov 25 05:01:40 otm-web1 gunicorn[901]: [2018-11-25 04:01:40 +0000] [1666] [DEBUG] POST /map/importer/start_import/

Nov 25 05:06:40 otm-web1 gunicorn[901]: [2018-11-25 05:06:40 +0000] [901] [CRITICAL] WORKER TIMEOUT (pid:1666)

Nov 25 05:06:40 otm-web1 gunicorn[901]: [2018-11-25 04:06:40 +0000] [1666] [INFO] Worker exiting (pid: 1666)

Nov 25 05:06:40 otm-web1 gunicorn[901]: [2018-11-25 05:06:40 +0000] [3595] [INFO] Booting worker with pid: 3595

Any advice is appreciated!

Regards

Chris Ward

[Justin Walgran]

Sorry that you have run into an problem with outdated documentation. Thanks for including such a detailed description of your process and the failure.

When running our celery workers, we chdir to the directory containing manage.py and then run a command much like the one you posted. On difference, is the value of the -A option. We use "opentreemap.celery.app" which refers to this module https://github.com/OpenTreeMap/otm-core/blob/c994b3b44f12a506fe01380428ecbba453e711dd/opentreemap/opentreemap/celery.py

Please try this launch command and let us know if it works.

 celery -A "opentreemap.celery.app" -b redis://$BROKER_URL/2 worker -l info

--
You received this message because you are subscribed to the Google Groups "opentreemap-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opentreemap-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Ward]

Unfortunately, no, that didn't work. I'm experiencing the same results.

[KenQiProductions]

I think that I have exactly the same problem that you have but i don't have a solution to it. hopefully someone can figure out how to fix it.

[Chris Ward]

I'm hoping so as well. I'm preparing to import an entire campus worth of tree information and have spent about a day now trying to debug the issue with no luck. 

--

[Scott Lusebrink]

Did any of you guys run into a missing FreeThreadedComment library issue?  I also found the directions were not kept up with the times. Lets see if we can work this out together.

Scott

Request Method:	GET
Request URL:	****
Django Version:	1.11.3
Exception Type:	InvalidTemplateLibrary
Exception Value:	Invalid template library specified. ImportError raised when trying to load 'threadedcomments.templatetags.threadedcommentstags': cannot import name FreeThreadedComment
Exception Location:	/usr/local/lib/python2.7/dist-packages/django/template/backends/django.py in get_package_libraries, line 130
Python Executable:	/usr/bin/python
Python Version:	2.7.6
Python Path:	['/usr/local/otm/app/opentreemap', '/usr/local/bin', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

[Chris Ward]

I didn't run into that particular issue. You have ran sudo pip install -r requirements.txt from /usr/local/otm/app, correct? Which distro/version are you running OTM on? I was originally on 14.04 but after I had everything (except celery) running upgraded to 16.04. I did try installing it on RHEL as well but didn't have much luck.

[Chris Ward]

@Justin Do you any other advice or suggestions? I'm spinning my tires trying to get this thing running.

[kel...@urban-ecos.com]

@ Justin: Any movement on this? We're having the same problem in SF and are otherwise ready to be up and running.

@Chris: Have you had any luck?

[Chris Ward]

@Kelaine I kept trying all kinds of variations to get it working, using different configs, etc. without any luck. I eventually re-evaluated my use case and realized that I don't really need what celery offers, so I completely removed its functionality from the import process. My use case is that one or two people will periodically import new data, so the requirement for distributed processing or task scheduling is overkill. It's completely understandable that it would be included generally, though. 

I know this is not ideal, but given the circumstances and the amount of time I spent researching and rubber ducking the issue, this was my last resort and it ended up working great. I was able to import 37 species and 910 trees without issue.

The latest commit to my fork contains these changes. In opentreemap/importer/views.py I removed the .delay() from each call, and several other celery calls from the tasks.py file. I decided to leave all of my rollbar integrations so that they may assist with debugging any import errors in the future. I did take an initial shot at decoupling celery from the export service, but haven't had a chance to finish. 

If you would like to view my changes or try them for yourself, my repo may be found here: 

https://github.com/cwardcode/otm-core/commit/4bbdee6ffe09086cc7cb7e8ce7d2a57e636c4f8a#diff-740e0196a0077aab9488be62334f2a26 

If interested, my repo has other modifications to the service -- I have made some minor customizations to the UI so that 'edit', 'delete', 'add tree' only show up when a user is logged in. I also removed the list users API to button up security so that just anyone can't see the usernames within the project.

[kel...@urban-ecos.com]

@Chris Thanks for the quick and thorough reply and for the link to the github repo. I confess-- I'm not the tech person on this project. I was the urban forestry expert for the very first urban forest map in SF. Ken Qi (also here in the forum) is our programmer. Our use case is similar to yours so maybe your solution will get us through too. And maybe with this collaboration we're getting more back into the spirit of open source! Thanks again.

[KenQiProductions]

@Chris, this is great. I tried to roll back on the celery part but to no success. i'll definitely check out your repo and try to replicate what you did, hopefully that will work. like Kelaine said, i think our use scenario is very similar. thanks for the work. 

[KenQiProductions]

@Chris Do you have this issue when importing a few hundreds or thousand of trees, that it didn't everything and sorta stop half way? it seems to be doing this to me. I don't know if this is related but I didn't change the default_settings.py yet.

[Chris Ward]

@KenQi, Sorry for the delayed response. Unfortunately I have not. I was able to import around 700 trees in one go, mainly because the other ~200 did not have a species defined.

One thing I did to help the debugging process was to create a free account on https://rollbar.com/ and configure the django app to use my api key. I then set up various rollbar reporting during the import/verification methods in 'opentreemap/importer/tasks.py' and 'opentreemap/importer/views.py'. One advantage is that it lets you track after which call the process is breaking, and if there is a server error generated, rollbar will automatically report it as well. It did wonders for me, you can see the level of detail rollbar provides in my attachment. 

[KenQiProductions]

@Christ, no problem, thanks for the reply. i'll try that to see what's going on.

[KenQiProductions]

@Chris,

what version of Celergy are you running? did you have to roll back to pre v4?

[Justin Walgran]

Sorry to everyone who has been blocked by Celery issues. I managed to reproduce the problem using the otm-vagrant setup and found a solution. 

TL;DR The setting BROKER_URL needs to be renamed CELERY_BROKER_URL. See this pull request for details https://github.com/OpenTreeMap/otm-vagrant/pull/55/

In my experience, the [CRITICAL] WORKER TIMEOUT errors were appearing because all CSV upload requests were hanging on this line in importer/views.py 

These hangs can occur if the application cannot make a connection to the Celery broker (Redis). I verified that Redis was running, and that the background worker process was also running. 

The problem was now isolated to the application itself, so I opened a Django shell and ran the code in celery.py one line at a time. It was through this process that I discovered that the call to app.config_from_object('django.conf:settings', namespace='CELERY') was not correctly configuring the Celery "app." Since the broker URL was set to an incorrect, default value, all future attempts to connect would to the broker would hang.

It turns out that app.config_from_object('django.conf:settings', namespace='CELERY') was not working correctly specifically because of the namespace keyword argument. When we specify namespace='CELERY' we are declaring that all Celery related settings must be prefixed with CELERY_.  The out-of-date installation instructions incorrectly specify that the settings file should contain a BROKER_URL. As of our update to Celery 4 and our use of namespace='CELERY' that setting must be named CELERY_BROKER_URL instead.

[KenQiProductions]

@Justin, thanks for putting the time to look into this. I just made an installation on your solution and everything works now.

[Justin Walgran]

That’s great to hear. I’m glad the fix worked.