detach terminal after submitting a long running job on EMR

[Xuchen Yao]

Hi,

I use `python mrjob_script.py -r emr` to submit jobs to EMR. The script doesn't exit until all steps are executed. However the job takes days to finish, during which if I turn off my computer it'll stop the job. Is there a way to get around this? Thank you.

Xuchen

[Ash]

Use tmux

--
You received this message because you are subscribed to the Google Groups "mrjob" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mrjob+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mrjob/7eef4e62-4dd2-46c4-a6ed-ccf65e0639e1%40googlegroups.com.

[Kostikey Mustakas]

Classic use for screen or tmux... 

--

[Xuchen Yao]

My understanding is that I can use screen or tmux or nohup if I submitted the job on a long running server and only detach the terminal.

However in my scenario I'd submit the job from my laptop. Once I shutdown or put my laptop to sleep and come to work the next day: I can still resume the previous session, but the connection from mrjob to EMR ssh tunnel would be lost, and possibly on the EMR server the jobs will be stopped. Please correct me if my understanding is wrong.

To view this discussion on the web visit https://groups.google.com/d/msgid/mrjob/CAETNksc7UN2%2BCk8_PpDmo_RkmCVy-T8zLnoB7wbNjQnWuShsZA%40mail.gmail.com.

[Kostikey Mustakas]

How are you submitting the job? Are you just calling the script on an EMR edge node that you are sshed into?

To view this discussion on the web visit https://groups.google.com/d/msgid/mrjob/CAEq4mkY3V1AV7%3DhPBqZRGWN8zMiMje5dduM2an0LOUVkmwA9XA%40mail.gmail.com.

[Xuchen Yao]

Nah I'm submitting from my laptop...

What's an EMR edge node? I can ssh to a master node. Should I submit on a master or an edge node?

To view this discussion on the web visit https://groups.google.com/d/msgid/mrjob/CAETNksdkxUfNQB2WM7dEU14d6j9_2U67JYge_bcwJTYv%2BE0w-Q%40mail.gmail.com.

[Ash]

If its an MR job that you submitted to AWS's EMR cluster (--runer=emr) from your laptop's terminal then you would just lose the ssh connection that mrjob maintains to the master node when you shut down. The actual MRJob would still be running if you look at the EMR console. 

But if its an MRJob that you are running locally (--runner=local) then it will stop when you shutdown.

To view this discussion on the web visit https://groups.google.com/d/msgid/mrjob/CAEq4mkZUJWHY%2BCddFQbL617sLa3K%3DmNXT9x7A2sBF8AknHK9Bw%40mail.gmail.com.