How to edit mrjob.conf to submit to an existing emr cluster?

[newb1er]

Kind of an EMR noob.  I know the value of EMR is you can start them up only as needed, but if I've set something up through the GUI and want to send my job to that cluster, how would I do that?

From the CLI tools, I know I can:

aws emr describe-cluster --cluster-id j-1ZXXXXXXXXXXX

What could I change in:

runners:

  emr:

(

like maybe:

    cluster-id: j-1ZXXXXXXXXXXX 

?

)

that instead of starting a new cluster in my emr control panel at:

my_analysis.username.20151207.022413.797727

...With ID: j-YYYYYYYYYYYYYY

Would send this job to the existing one?

[David Marin]

You wouldn't put the cluster in mrjob.conf, but you can specify a cluster by running your job with --emr-job-flow-id <cluster ID>.

But... if you started a cluster with the AWS CLI, it won't have the mrjob library installed. Instead, I recommend starting a cluster with `mrjob create-job-flow`, which does the relevant bootstrapping so that you can run mrjobs on the cluster.

You might also want to look into job flow pooling, which will either start a new cluster or reuse an existing one as appropriate:

https://pythonhosted.org/mrjob/guides/emr-opts.html#option-pool_emr_job_flows

-Dave

P.S. Always make sure to terminate your clusters when you're done with them; the hourly charges for idle clusters can really add up.

P.P.S. mrjob has a tool that does this as well, check out `mrjob terminate-idle-job-flows`:

https://pythonhosted.org/mrjob/guides/emr-tools.html#module-mrjob.tools.emr.terminate_idle_job_flows

> --
> You received this message because you are subscribed to the Google Groups "mrjob" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mrjob+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.