Classic worker being killed off?

[Ryan Nowakowski]

Jared has mentioned[1] that the classic worker is slowly being killed off.
What should I use instead to construct my map reduce jobs?

Thanks!

Ryan

1. https://github.com/discoproject/disco/issues/298#issuecomment-5366806

[ryan]

Sorry everyone.  My previous post never hit my inbox so I resent.  Ignore this.

[Jared Flatow]

Hi Ryan,

The case where you are using the Job class is preferred because it actually sends the source whereas the classic worker pickles the byte code.

jared

> --
> You received this message because you are subscribed to the Google Groups "Disco-development" group.
> To post to this group, send email to disc...@googlegroups.com.
> To unsubscribe from this group, send email to disco-dev+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/disco-dev?hl=en.
>

[Ryan Nowakowski]

I think I understand:

=== Classic Worker(deprecated) ===
# Here only the byte-code is pickle and sent over the wire

def map(line, params):
for word in line.split():
yield word, 1

def reduce(iter, params):
from disco.util import kvgroup
for word, counts in kvgroup(sorted(iter)):
yield word, sum(counts)


if __name__ == '__main__':
job = Job().run(input=["http://discoproject.org/media/text/chekhov.txt"],
map=map,
reduce=reduce)


=== New Hotness(preferred) ===
# Here the entire source file is sent over the wire

class WordCountJob(Job):
@staticmethod
def map(line, params):
for word in line.split():
yield word, 1

@staticmethod
def reduce(iter, params):
from disco.util import kvgroup
for word, counts in kvgroup(sorted(iter)):
yield word, sum(counts)


if __name__ == '__main__':
job = WordCountJob().run(input=["http://discoproject.org/media/text/chekhov.txt"])