mrjob v0.7.4 is out!
62 views
Skip to first unread message
Coyote Codornices Marin
Sep 17, 2020, 6:42:22 PM9/17/20
to mr...@googlegroups.com
mrjob v0.7.4 is out!
mrjob now supports Docker on EMR. Using this feature can be as simple as pointing docker_image to your docker image on DockerHub
mrjob now also supports running steps concurrently on the same cluster on EMR with the max_concurrent_steps option.
Finally, there are now better tools for managing a pool of clusters. You can limit the number of clusters in the pool with max_clusters_in_pool. If you like, a job that is unable to join or create a cluster can now time out with the pool_timeout_minutes option. pool_wait_minutes is also a bit smarter; it won’t wait when there isn’t actually a matching cluster to wait for.
For more details, see https://mrjob.readthedocs.io/en/latest/whats-new.html#v0-7-4
-Coyote
P.S. In case you’re confused about who Coyote is, since the last release I came out as non-binary, and this is my new name (I use they/them pronouns). Same person who’s been maintaining mrjob for the last decade. ^.^
mrjob now supports Docker on EMR. Using this feature can be as simple as pointing docker_image to your docker image on DockerHub
mrjob now also supports running steps concurrently on the same cluster on EMR with the max_concurrent_steps option.
Finally, there are now better tools for managing a pool of clusters. You can limit the number of clusters in the pool with max_clusters_in_pool. If you like, a job that is unable to join or create a cluster can now time out with the pool_timeout_minutes option. pool_wait_minutes is also a bit smarter; it won’t wait when there isn’t actually a matching cluster to wait for.
For more details, see https://mrjob.readthedocs.io/en/latest/whats-new.html#v0-7-4
-Coyote
P.S. In case you’re confused about who Coyote is, since the last release I came out as non-binary, and this is my new name (I use they/them pronouns). Same person who’s been maintaining mrjob for the last decade. ^.^
Reply all
Reply to author
Forward
0 new messages