Checkpointable preemption for Disco

[Augusto Souza]

Hello everyone, how are you doing?

My name is Augusto Souza (http://github.com/augustorsouza) and I am a computer science student in University of Campinas (Brazil). My area of research is Distributed Systems and I have been trying to work with the Hadoop community to make some contributions to it:

- https://issues.apache.org/jira/browse/MAPREDUCE-5269

- https://issues.apache.org/jira/browse/MAPREDUCE-6434

- https://issues.apache.org/jira/browse/MAPREDUCE-6444

I have been trying to help an Apache developer to make changes related to checkpointable preemption of jobs accepted (as you can see by my activity in Hadoop's Jira). Also, I have been looking into alternatives distributed systems frameworks that could have the benefit of checkpointable preemption of the mappers and reducers when a job with more priority gets scheduled, I found Disco on Github, and then I got into your profile.

I am curious if a checkpointing feature would be useful for this project, and if so I would like to contribute in some way to this project and measure some results to help me with my dissertation. 

I think the first steps are to implement a preemption policy for the Disco scheduler (a killing one to start) and then another policy based on checkpointing before killing, and then when rescheduled the job gets the benefit of restoring its status from the DDFS checkpoint (that might be saved before killing the job).

Also, I am able to help in any other need Disco might have and I might perform some research in order to complete my Masters.

Thank you in advance!

Best regards,

Augusto Souza