mrjob v0.6.6 is released. This is mostly a series of small changes that make your life easier:
- you can safely use booleans as jobconf values in mrjob.conf
- faster probable cause of error on EMR if you have SSH set up
- you can use --spark-args=‘...’ instead of --spark-arg with one argument at a time
- same with --hadoop-args
- you can use -D instead of --jobconf
- --local-tmp-dir option can override your config, or just use the default temp dir (--local-tmp-dir ‘')
mrjob can also now run JarSteps that don’t understand Hadoop generic options (e.g. -D, -libjar)
For the full list of changes, see:
https://mrjob.readthedocs.io/en/latest/whats-new.html#v0-6-6
-Dave