Multi-step jobs from multiple classes?

[Michael Armida]

Is it possible for a job to span multiple classes? The definition returned by steps is a list - is it possible for it to include a second entry, and for that entry to be another MRStep referencing another class, e.g.:

class AJob(MRJob):

    ...

    def steps(self):

        return [MRStep(mapper=self.mapper), MRStep(reducer=OtherClass.reducer)]

[dm]

Yes, but you may be asking the wrong question?

mappers, reducers, etc. can be anything that matches the function signature. They don't have to be attached to a class at all; they can be plain old functions! In your example, OtherClass.reducer appears to be a class or static method (though nothing would prevent you from instantiating another job class and getting an instance method from it: OtherClass().reducer).

There are two things that making your mappers/reducers instance methods in your job class buy you:

- the module that the job class is in will be automatically uploaded, so you don't need to worry about setup options, PYTHONPATH, etc.

- self.stderr can be redirected to a StringIO for unit testing (self.increment_counter() and self.set_status() use it).

-Dave