Docker image with CUDA/Pytorch
208 views
Skip to first unread message
Sara Summerton
Apr 3, 2023, 11:30:34 PM4/3/23
to physionet-challenges
Hi Challenge team and fellow participants,
When using a newer version of Docker with a CUDA or Pytorch base image, it seems that you must now pass the flag --gpus all with the docker run command in order to allow the container code to access the system GPU. However, in the challenge FAQ we are instructed that the team is unable to refer to our README file to change how they build and run our code.
So, to the Challenge team: are we allowed to request runtime configuration flags to be passed with the docker run command? If so, what is the best way to specify this with our submission?
To fellow challenge participants: Has anyone found a way around this? Appreciate any and all Docker advice.
Thanks for your help!
Sara
PhysioNet Challenge
Apr 3, 2023, 11:38:16 PM4/3/23
to physionet-challenges
Dear Sara,
Thanks for the excellent comments and questions.
We do add a few command-line arguments to the "docker run" command to support various base images and libraries. For example, we use "--gpus all" to provide GPU support and "--ipc=host" or "--shm-size X" to provide multiprocessing support for PyTorch. For the sake of reproducibility, we should (and will) share these full commands on the Challenge website. (The Challenge forum has some of this information, but they are incomplete, and the posts can be difficult to find.) The arguments may not work with future Docker versions or every host machine, but, in general, they should be helpful.
To request a change in how we build and run your (and others') code, please simply email us to ask. Unfortunately, we cannot make every change that we receive: technical restrictions, capacity constraints, looming deadlines, and problem definitions limit the changes that we are able or willing to make. However, please do ask, especially during the unofficial phase and hiatus, and we will do what we can. In many cases, we have been unable to run someone's code, but we thought that we should have been able to run it, so we troubleshooted the issue and made a change to our scoring pipeline on our own. In other cases, we worked with the team to find a solution. In some cases, the request would have affected the Challenge in an unintended or undesirable way, such as changing the problem definition, so we tried to find an alternative. In all cases, we want to be able to support as much and as diverse code as possible to address the problem posed by the Challenge.
We do ask teams not to use the README to tell us how to run their code for a number of reasons, including (1) we may interpret a team's instructions incorrectly, inconsistently, or differently from how the team wanted us to interpret them; (2) we may interpret a team's instructions correctly, but the instructions may have missing or implied steps; (3) human interpretation takes time, we receive many submissions near deadlines, and we want to be able to run the code as quickly as possible; (4) we want a team's released code to be as accessible as possible; and (5) we may be unable or unwilling to follow the instructions because of one or more of the reasons in the previous paragraph. For context, we are unable to run roughly half of the submissions that we receive as-is, so even minor troubleshooting does not scale well near deadlines.
You are welcome (and encouraged) to use the README to help others to use your code. However, in many cases, a team may be able to include instructions as explicit commands that we run when building their Docker image, which makes the instructions unambiguous. In other cases, such as the "--gpus all" argument, a team may not be able to make the change on their own, so please feel free to include it in your README and to email us to request a change.
In short, please ask us directly. Both individual requests and community feedback are welcome.
Best,
Matt
(On behalf of the Challenge team.)
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.
Thanks for the excellent comments and questions.
We do add a few command-line arguments to the "docker run" command to support various base images and libraries. For example, we use "--gpus all" to provide GPU support and "--ipc=host" or "--shm-size X" to provide multiprocessing support for PyTorch. For the sake of reproducibility, we should (and will) share these full commands on the Challenge website. (The Challenge forum has some of this information, but they are incomplete, and the posts can be difficult to find.) The arguments may not work with future Docker versions or every host machine, but, in general, they should be helpful.
To request a change in how we build and run your (and others') code, please simply email us to ask. Unfortunately, we cannot make every change that we receive: technical restrictions, capacity constraints, looming deadlines, and problem definitions limit the changes that we are able or willing to make. However, please do ask, especially during the unofficial phase and hiatus, and we will do what we can. In many cases, we have been unable to run someone's code, but we thought that we should have been able to run it, so we troubleshooted the issue and made a change to our scoring pipeline on our own. In other cases, we worked with the team to find a solution. In some cases, the request would have affected the Challenge in an unintended or undesirable way, such as changing the problem definition, so we tried to find an alternative. In all cases, we want to be able to support as much and as diverse code as possible to address the problem posed by the Challenge.
We do ask teams not to use the README to tell us how to run their code for a number of reasons, including (1) we may interpret a team's instructions incorrectly, inconsistently, or differently from how the team wanted us to interpret them; (2) we may interpret a team's instructions correctly, but the instructions may have missing or implied steps; (3) human interpretation takes time, we receive many submissions near deadlines, and we want to be able to run the code as quickly as possible; (4) we want a team's released code to be as accessible as possible; and (5) we may be unable or unwilling to follow the instructions because of one or more of the reasons in the previous paragraph. For context, we are unable to run roughly half of the submissions that we receive as-is, so even minor troubleshooting does not scale well near deadlines.
You are welcome (and encouraged) to use the README to help others to use your code. However, in many cases, a team may be able to include instructions as explicit commands that we run when building their Docker image, which makes the instructions unambiguous. In other cases, such as the "--gpus all" argument, a team may not be able to make the change on their own, so please feel free to include it in your README and to email us to request a change.
In short, please ask us directly. Both individual requests and community feedback are welcome.
Best,
Matt
(On behalf of the Challenge team.)
Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.
Reply all
Reply to author
Forward
0 new messages