Pickling of task function

22 views
Skip to first unread message

Ayush Bisht

unread,
May 12, 2024, 11:13:47 AMMay 12
to celery-users
Hey guys
I'm new in this channel, I hope I'll get to learn something about celery here.

Since few days, I'm trying to understand how celery works and was trying to recreate it . I was successfully managed to create broker, workers and celery app interface to register the task. But I stumbled in serializing the task. 

So here is the application directory

mycelery
|     ---- mycelery
|            apps
|                   broker, worker, celery
|            __main__.py
|     task.py 

The issue is if my task is outside the celery directory then pickle is failing to dump the task function. Right now I'm doing class based registration of task. This is happening because of interrupter not knowing from where the function came. It works by appending the absolute path of task.py file inside __main__.py, But I want it to be auto managed by module itself just like celery is doing. 

Can someone please share that part of celery codebase, where such things are handled, I've checked that we are importing the module during unpickling of Task object. But the way it was done its quite complicated to me. 

# celery.app.registry def _unpickle_task(name):
return get_current_app().tasks[name]


def _unpickle_task_v2(name, module=None):
if module:
import_module(module)
return get_current_app().tasks[name]


Please, do comment if you guys need more context about what am I doing. 
Thanks
Reply all
Reply to author
Forward
0 new messages