Help with logging

1,638 views
Skip to first unread message

Amit Saha

unread,
Jul 17, 2012, 4:03:31 AM7/17/12
to celery...@googlegroups.com
Hello,

I have a package level logging (to a file) enabled in my application.
I use celery to execute this code on different nodes. For eg. I have a
tasks.py with a single task which basically calls the application.
Now, i see that the applications log messages are appended to the
celery log itself. I would like to keep them separate.

What would be the best/easiest way to go about it?

Best,
Amit

--
http://echorand.me

Amit Saha

unread,
Jul 18, 2012, 10:12:22 AM7/18/12
to celery...@googlegroups.com
Hi,
Just to be clearer. Here is my tasks.py:

from __future__ import absolute_import
import json
from celery.task import task

from image_builder.imagebuilder import ImageBuilder

@task
def build(buildconfig, kickstart):

builder = ImageBuilder(json.loads(buildconfig), kickstart)
status = builder.build()

return

As can be seen, it basically calls the code in
image_builder.imagebuilder. The image_builder package uses the Python
standard logging. I would like to keep this as it is, and not let
Celery hijack it. I have tried setting the
CELERYD_HIJACK_ROOT_LOGGER=False in my celeryconfig.py. But that
doesn't help.

Any ideas or I might be doing something wrong?

Thanks a lot in advance.

Best,
Amit

>
> Best,
> Amit
>
> --
> http://echorand.me



--
http://echorand.me

Amit Saha

unread,
Jul 18, 2012, 11:41:44 PM7/18/12
to celery...@googlegroups.com
Hello,
Found a solution for this. So that it is useful, I shall be a little
descriptive.

Problem: You have an existing Python application with standard logging
to a file. You want to keep it as it is when using celery instead of
allowing celery to "hijack" it to its task log.

Solution: Using celery.signals.worker_process_init (from
http://stackoverflow.com/a/6193082/59634)

Basically, set this up in the top-level package and use away!

I did not need to do anything else.

Cheers,
-Amit


>
> Best,
> Amit
>
>>
>> Best,
>> Amit
>>
>> --
>> http://echorand.me
>
>
>
> --
> http://echorand.me



--
http://echorand.me

Ask Solem

unread,
Jul 20, 2012, 7:00:36 AM7/20/12
to celery...@googlegroups.com

On 19 Jul 2012, at 04:41, Amit Saha wrote:
>
> Found a solution for this. So that it is useful, I shall be a little
> descriptive.
>
> Problem: You have an existing Python application with standard logging
> to a file. You want to keep it as it is when using celery instead of
> allowing celery to "hijack" it to its task log.
>
> Solution: Using celery.signals.worker_process_init (from
> http://stackoverflow.com/a/6193082/59634)
>
> Basically, set this up in the top-level package and use away!
>
> I did not need to do anything else.
>


Note that the answer is slightly outdated for 3.0, as
in 3.0 you're discouraged from using task.get_logger.

Instead the best practice is to add a module-level logger:

from celery import task
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)


@task
def add(x, y):
logger.debug('add: %r %r', x, y)
return x + y


In 3.0 all task logger inherit from a common logger called 'celery.task',
which you can get from 'celery.utils.task_logger',
so if you want to agument the configuration you can do:

from celery.utils.log import task_logger
from celery.signals import after_setup_task_logger


@after_setup_task_logger.connect
def augment_task_logging(**kwargs):
logger = logging.getLogger('myapp.tasks')
if not logger.handlers:
formatter = logging.Formatter(logging.BASIC_FORMAT)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.propagate = 0


(setting propagate to 0 there, because some versions seems to have
problems if it's set to True or False).


Also the worker_process_init signal is only sent for each pool child process
when using the multiprocessing pool, so it won't be setup in the mainprocess
(which would be a problem if using other pools like eventlet).

--
Ask Solem
twitter.com/asksol | +44 (0)7713357179

signature.asc

Amit Saha

unread,
Aug 14, 2012, 4:11:37 AM8/14/12
to celery...@googlegroups.com
Hi,
Thanks, I implemented your solution. And it works fine. Infact, i was
observing something weird with my earlier approach, but it seems all
fine now.

Thanks.
-Amit


--
http://echorand.me

José de Zárate

unread,
Nov 2, 2012, 10:41:37 AM11/2/12
to celery...@googlegroups.com
ask

I have a super-simple tasks file, and still don't get log ok

here it is

import celery

from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)

@celery.task()
def add(x, y):
    logger.error("Adding %s + %s" % (x, y))
    print logger
    return x + y


from celery.signals import after_setup_task_logger
import logging

def foo_tasks_setup_logging(**kw):
    logger = logging.getLogger('helpck.tasks')
    if not logger.handlers:
        print logger
        handler = logging.FileHandler('tasks.log')
        formatter = logging.Formatter(logging.BASIC_FORMAT) # you may want to customize this.
        handler.setFormatter(formatter)
        logger.addHandler(handler)
        logger.propagate = False

after_setup_task_logger.connect(foo_tasks_setup_logging)



all I can tell is that the logger that is inside the task is a "processawarelog", and the logger you manipulate in the signal handler is "sigsafelog"

???

any help, would be greatly appreciated.

José de Zárate

unread,
Nov 2, 2012, 2:38:02 PM11/2/12
to celery...@googlegroups.com
The weirdest thing....

I got it working, but don't understand it very well. No need for signal handling, just manipulate the logger object at the outside scope:

import logging


import celery
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
handler = logging.FileHandler('tasks.log')
logger.addHandler(handler)

@celery.task()
def add(x, y):
    logger.info("Adding %s + %s" % (x, y))
    return x + y


and it works. my guess is : when you set code like in the previous example , using the signal handler, at the time that handler is called, celery hasn't yet built the specific "ProcessAwareLog" object that is going to use in the tasks. Also the logger defined in the outside scope (the one I just wrote about) is this not-yet captured log called "SigSafeLog" or something similar.

when the worker actually starts, that file is "read" again and those instructions are processed again, only this time the logs are already the "ProcessAwareLog" which we'll be using inside the task.

Looks like the after_setup_logger signal is sent before the worker's log is built. in consequence all the manipulations made inside that handler, are manipulations to a logger (SigSafeLog) which is not the one the task will use.
???

However, w

Andrew Pashkin

unread,
Jun 26, 2015, 9:02:57 AM6/26/15
to celery...@googlegroups.com
If all task loggers inherit from 'celery.task', why in your example, you configuring 'myapp.tasks'?

José de Zárate

unread,
Jun 26, 2015, 9:27:59 AM6/26/15
to celery...@googlegroups.com
andrew. I'm sorry I can't answer that, I barely remember that code, and I'm positive the logging mechanism in celery has changed quite a lot since this, so I'm dubious this thread is of any help :)

--
You received this message because you are subscribed to a topic in the Google Groups "celery-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/celery-users/xNPYTobJ5Rg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to celery-users...@googlegroups.com.
To post to this group, send email to celery...@googlegroups.com.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.



--
uh, oh.


Reply all
Reply to author
Forward
0 new messages