event handlers, how should they really behave?

[Bill Tucker]

Hi There,

I'm new to Luigi, and I have a question:


When I decorate a class function with an event_handler, e.g.

 @luigi.Task.event_handler(luigi.Event.SUCCESS)

it seems to fire for every Task, not just for the class that is decorated.  

According to the docs for event_handler, the event will be triggered only from that class or subclass, implying that it won't call the one above it.

I'm seeing that it's trigger for other classes.   Here's an example (luex.py)  I cobbled together of a ComplexTask, calling a SimpleTask, only the latter of which has an event_handler.

import luigi


class ComplexTask(luigi.Task):
    """ This call the simple task before performing it's task"""

    x = luigi.IntParameter()
    y = luigi.IntParameter(default=32)

    def requires(self):
        return SimpleTask(self.x, self.y)

    def run(self):
        print self.x + self.y


class SimpleTask(luigi.Task):

    x = luigi.IntParameter()
    y = luigi.IntParameter(default=45)

    def run(self):
        print self.x * self.y

    @luigi.Task.event_handler(luigi.Event.SUCCESS)
    def celebrate_success(self):
        """
        Report success event
        Uses "-+--+--+--+--+--+--+--+--+--+-" as indicator
        """

        print(26 * '-+-')
        print("Yay!, {c} succeeded. :)".format(c=self.__class__.__name__))
        print(26 * '-+-')

    @luigi.Task.event_handler(luigi.Event.FAILURE)
    def mourn_failure(self, exception):
        """
        Report failure event
        Uses "-!--!--!--!--!--!--!--!--!--!-"  as indicator
        """

        print(26 * '-!-')
        print("Boo!, {c} failed.  :(".format(c=self.__class__.__name__))
        print(".. with this exception: '{e}'".format(e=str(exception)))
        print(26 * '-!-')


def main():
    """ This task can be run by: python <filename.py> <classname> <args>
        For example:
            python luex.py ComplexTask --x=10
    """
    luigi.run()




if __name__ == "__main__":
    main()

When I run this code I get output containing the following:

ERROR: [pid 5825] Worker Worker(salt=182110569, workers=1, host=luigi1, username=bill, pid=5825) failed    ComplexTask(x=42, y=32)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/luigi/worker.py", line 136, in run
    raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: SimpleTask(x=42, y=32)
-!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!-
Boo!, ComplexTask failed.  :(
.. with this exception: 'Unfulfilled dependency at run time: SimpleTask(x=42, y=32)'
-!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!--!-

I'm not expecting to see this failure print out in this way.
It's seems that the ComplexTask failure is calling the SimpleTask's mourn_failure function, and that seems.. undesirable behavior.

Should I file a bug on github, or am I using this decorator incorrectly or differently than intended?


Cheers,

-Bill Tucker

[Erik Bernhardsson]

Does someone have an answer to this question? I haven't used event handlers

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Elias Freider]

Hi!

The docs aren't super clear about how to implement event handlers for specific classes (in the example it uses the base Task class).

The event_handler decorator/class method triggers for all classes of the same or in a subclass of the class that the decorator is bound to. In your example you decorate using the Task base class, which will make it trigger for all tasks.

@luigi.Task.event_handler(luigi.Event.SUCCESS)

What you instead probably want to be doing is to use your own class name, typically in module scope after the class declaration or somewhere else:

@SimpleTask.event_handler(luigi.Event.SUCCESS)

def celebrate_success(simple_task_instance):

    ...

Then the decorated method should just trigger when SimpleTask (or subclasses thereof) finish successfully (similarly can be done for failures).

/Elias

[annur...@gmail.com]

Did you by any chance get this to work?

[Grant Watts]

I had a simular issue and solved it by moving the function outside the class and using that classes event handler call instead of luigi Task class. Sorry first time posting to the place so wasn't sure how to format the code correctly.

```

python

import luigi


class ComplexTask(luigi.Task):
    """ This call the simple task before performing it's task"""

    x = luigi.IntParameter()
    y = luigi.IntParameter(default=32)

    def requires(self):
        return SimpleTask(self.x, self.y)

    def run(self):
        print self.x + self.y


class SimpleTask(luigi.Task):

    x = luigi.IntParameter()
    y = luigi.IntParameter(default=45)

    def run(self):
        print self.x * self.y

@SimpleTask.event_handler(luigi.Event.SUCCESS)

def celebrate_success(self):
    """
    Report success event
    Uses "-+--+--+--+--+--+--+--+--+--+-" as indicator
    """

    print(26 * '-+-')
    print("Yay!, {c} succeeded. :)".format(c=self.__class__.__name__))
    print(26 * '-+-')

@SimpleTask.event_handler(luigi.Event.FAILURE)

```

[Adrien Berchet]

It is possible to keep the functions inside the class and setup the event handler in the __init__ method:

```python

import luigi


class ComplexTask(luigi.Task):
    """ This call the simple task before performing it's task"""

    x = luigi.IntParameter()
    y = luigi.IntParameter(default=32)

    def requires(self):
        return SimpleTask(self.x, self.y)

    def run(self):

        print(self.x + self.y)

class SimpleTask(luigi.Task):

    x = luigi.IntParameter()
    y = luigi.IntParameter(default=45)

    def __init__(self, *args, **kwargs):
        event_handler = super().event_handler

        @event_handler(luigi.Event.SUCCESS)

        def celebrate_success(self):
            """
            Report success event
            Uses "-+--+--+--+--+--+--+--+--+--+-" as indicator
            """

            print(26 * '-+-')
            print("Yay!, {c} succeeded. :)".format(c=self.__class__.__name__))
            print(26 * '-+-')

        @event_handler(luigi.Event.FAILURE)

        def mourn_failure(self, exception):
            """
            Report failure event
            Uses "-!--!--!--!--!--!--!--!--!--!-"  as indicator
            """

            print(26 * '-!-')
            print("Boo!, {c} failed.  :(".format(c=self.__class__.__name__))
            print(".. with this exception: '{e}'".format(e=str(exception)))
            print(26 * '-!-')

        super().__init__(*args, **kwargs)

    def run(self):
        print(self.x * self.y)

def main():
    """ This task can be run by: python <filename.py> <classname> <args>
        For example:
            python luex.py ComplexTask --x=10
    """
    luigi.run()


if __name__ == "__main__":
    main()
```