Not all Celery tasks registering?

[ben.e...@gmail.com]

I'm trying out the development branch of OpenREM (I try to update it every once in a while to try it out).

I seem to have an issue with the registering of some of the tasks in Celery which prevents them from running, specifically remapp.netdicom.qrscu is not being discovered.

If I run:

celery worker -A openremproject --loglevel=INFO

Then the tasks listed are:

[tasks]
. openrem.remapp.extractors.ct_philips.ct_philips
. openrem.remapp.extractors.dx.dx
. openrem.remapp.extractors.mam.mam
. openrem.remapp.extractors.rdsr.rdsr
. openremproject.celery.debug_task
. remapp.exports.ct_export.ct_csv
. remapp.exports.ct_export.ctxlsx
. remapp.exports.dx_export.dxxlsx
. remapp.exports.dx_export.exportDX2excel
. remapp.exports.mg_csv_nhsbsp.mg_csv_nhsbsp
. remapp.exports.mg_export.exportMG2excel
. remapp.exports.rf_export.exportFL2excel
. remapp.exports.rf_export.rfopenskin
. remapp.exports.rf_export.rfxlsx
. remapp.extractors.ct_philips.ct_philips
. remapp.extractors.dx.dx
. remapp.extractors.mam.mam
. remapp.extractors.rdsr.rdsr

I can load remapp.netdicom.qrscu by editing openremproject/celery.py and adding the line:

from remapp.netdicom.qrscu import qrscu

The tasks listed are then:

[tasks]
. openrem.remapp.extractors.ct_philips.ct_philips
. openrem.remapp.extractors.dx.dx
. openrem.remapp.extractors.mam.mam
. openrem.remapp.extractors.rdsr.rdsr
. openremproject.celery.debug_task
. remapp.exports.ct_export.ct_csv
. remapp.exports.ct_export.ctxlsx
. remapp.exports.dx_export.dxxlsx
. remapp.exports.dx_export.exportDX2excel
. remapp.exports.mg_csv_nhsbsp.mg_csv_nhsbsp
. remapp.exports.mg_export.exportMG2excel
. remapp.exports.rf_export.exportFL2excel
. remapp.exports.rf_export.rfopenskin
. remapp.exports.rf_export.rfxlsx
. remapp.extractors.ct_philips.ct_philips
. remapp.extractors.dx.dx
. remapp.extractors.mam.mam
. remapp.extractors.rdsr.rdsr
. remapp.netdicom.qrscu.movescu
. remapp.netdicom.qrscu.qrscu

It seems strange to me that some but not all of the tasks are registered? It makes me think it may be more than some local config setting I've messed up.

I'm using celery==4.0.2.

Thanks,

Ben

[Ed McDonagh]

Hi Ben, thanks for the feedback - always good to hear!

Can you have a look at your remapp/tasks.py file and compare it to https://bitbucket.org/openrem/openrem/src/5bfe8f63aa74ec8686cc66de204b586a9277463f/openrem/remapp/tasks.py?at=develop&fileviewer=file-view-default

With my development enviroment, I get the following:

[tasks]

  . openrem.remapp.extractors.ct_philips.ct_philips

  . openrem.remapp.extractors.dx.dx

  . openrem.remapp.extractors.mam.mam

  . openrem.remapp.extractors.rdsr.rdsr

  . openrem.remapp.extractors.rdsr_toshiba_ct_from_dose_images.rdsr_toshiba_ct_from_dose_images

  . openremproject.celery.debug_task

  . remapp.exports.ct_export.ct_csv

  . remapp.exports.ct_export.ctxlsx

  . remapp.exports.dx_export.dxxlsx

  . remapp.exports.dx_export.exportDX2excel

  . remapp.exports.mg_csv_nhsbsp.mg_csv_nhsbsp

  . remapp.exports.mg_export.exportMG2excel

  . remapp.exports.rf_export.exportFL2excel

  . remapp.exports.rf_export.rfopenskin

  . remapp.exports.rf_export.rfxlsx

  . remapp.extractors.ct_philips.ct_philips

  . remapp.extractors.dx.dx

  . remapp.extractors.mam.mam

  . remapp.extractors.ptsizecsv2db.websizeimport

  . remapp.extractors.rdsr.rdsr

  . remapp.extractors.rdsr_toshiba_ct_from_dose_images.rdsr_toshiba_ct_from_dose_images

  . remapp.netdicom.keepalive.keep_alive

  . remapp.netdicom.qrscu.movescu

  . remapp.netdicom.qrscu.qrscu

  . remapp.tools.make_skin_map

What I can see from that though is that the extractors are being brought in twice, with two different namespaces. I'll have to look at that. See https://bitbucket.org/openrem/openrem/issues/567/celery-extractor-tasks-imported-with-two

Ed

--
You received this message because you are subscribed to the Google Groups "OpenREM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrem+unsubscribe@googlegroups.com.
To post to this group, send an email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/openrem.
For more options, visit https://groups.google.com/d/optout.

[ben.e...@gmail.com]

> To unsubscribe from this group and stop receiving emails from it, send an email to openrem+u...@googlegroups.com.

>
> To post to this group, send an email to ope...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/openrem.
>
> For more options, visit https://groups.google.com/d/optout.

Hi Ed,

My tasks.py seems normal from what I can tell:

"""
.. module:: tasks.py
:synopsis: Module to import all the functions that run as Celery tasks.
"""
from __future__ import absolute_import

from celery import shared_task # pylint: disable=unused-import

from remapp.exports.mg_export import exportMG2excel # pylint: disable=unused-import
from remapp.exports.ct_export import ctxlsx, ct_csv # pylint: disable=unused-import
from remapp.exports.mg_csv_nhsbsp import mg_csv_nhsbsp # pylint: disable=unused-import
from remapp.exports.dx_export import exportDX2excel, dxxlsx # pylint: disable=unused-import
from remapp.exports.rf_export import exportFL2excel, rfxlsx # pylint: disable=unused-import
from remapp.extractors.ptsizecsv2db import websizeimport # pylint: disable=unused-import
from remapp.extractors.dx import dx # pylint: disable=unused-import
from remapp.extractors.rdsr import rdsr # pylint: disable=unused-import
from remapp.extractors.mam import mam # pylint: disable=unused-import
from remapp.extractors.ct_philips import ct_philips # pylint: disable=unused-import
from remapp.netdicom.qrscu import qrscu, movescu # pylint: disable=unused-import
from remapp.netdicom.keepalive import keep_alive # pylint: disable=unused-import
from remapp.tools.make_skin_map import make_skin_map # pylint: disable=unused-import

I tried deleting the .pyc version too but that made no difference. From my understanding, these tasks are imported through the django apps in settings.py which has remapp as an app. They're imported through the line:

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

in celery.py. The celery docs (http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html) seem to suggest from version 4.0+ only:

app.autodiscover_tasks()

Should be needed to discover tasks, although this doesn't make any difference to my issue.

Thanks,

Ben

[ben.e...@gmail.com]

I think I might have found the issue. In https://bitbucket.org/openrem/openrem/src/5bfe8f63aa74ec8686cc66de204b586a9277463f/openrem/remapp/extractors/__init__.py?at=develop&fileviewer=file-view-default

The extractor stuff is getting imported, which doubles them up in the task list like you said. If I comment these lines out then the netdicom imports suddenly appear on the celery tasks list. Not sure I understand why that has that effect but seems to be the cause...

Ben

[Tim de Wit]

After upgrading openrem to the development branch, I'm having similar problems...

In my case only the following tasks are registred:

    * openremproject.celery.debug_task

    * remapp.exports.ct_export.ct_csv

    * remapp.exports.ct_export.ctxlsx

    * remapp.exports.dx_export.dxxlsx

    * remapp.exports.dx_export.exportDX2excel

    * remapp.exports.mg_csv_nhsbsp.mg_csv_nhsbsp

    * remapp.exports.mg_export.exportMG2excel

    * remapp.exports.rf_export.exportFL2excel

    * remapp.exports.rf_export.rfopenskin

    * remapp.exports.rf_export.rfxlsx

    * remapp.extractors.ptsizecsv2db.websizeimport

Any clue what to try next?

regards, Tim

[Ed McDonagh]

Sorry Tim I haven't addressed this yet - see issues #567.

I'll try and have a look at it later.

To unsubscribe from this group and stop receiving emails from it, send an email to openrem+unsubscribe@googlegroups.com.

To post to this group, send email to ope...@googlegroups.com.