Specifying libjars to mrjob in local hadoop mode

155 views
Skip to first unread message

Saravana Kumar

unread,
Apr 16, 2012, 5:59:59 PM4/16/12
to mr...@googlegroups.com
How can I specify the libjar argument when running in local hadoop mode?

Steve Johnson

unread,
Apr 16, 2012, 6:12:10 PM4/16/12
to mr...@googlegroups.com
This should answer your question: https://github.com/Yelp/mrjob/issues/198

It's included in the 0.3.4 milestone. We'll see if it actually happens.

Currently, using `-libjars` requires something like this:
    --bootstrap-file myjar.jar --bootstrap-cmd 'cp myjar.jar /home/hadoop/myjar.jar' --hadoop-arg -libjars --hadoop-arg /home/hadoop/myjar.jar
It would be nice if we could do this instead:
    --libjars myjar.jar
mrjob would:
* Upload the file like the other bootstrap files
* Copy it to a unique location like /tmp/myjar-235324.jar
* Add the appropriate `-libjars` arguments to `StreamingStep.step_args`

Saravana Kumar

unread,
Apr 16, 2012, 6:16:40 PM4/16/12
to mr...@googlegroups.com
Great, thanks.
Reply all
Reply to author
Forward
0 new messages