GPU availability

[justina...@gmail.com]

Hi, we are running experiment with 1 gpu per executor, and while it says 4 gpus available, once we start the exeriment it tries to spin up 2 executors and gpu count reaches 5 queued gpu requests and our experiment cannot run. It should only use 2 GPUs in total, but looks like 9 are being reserved? Could you please look into this?

[Robin Andersson]

Hi!

Can you try running it again now?

Also are you certain that you set the number of GPUs per executor to be 1? And the max number of parallel experiments to be 2?

BR,

Robin

On Thu, Jan 24, 2019 at 7:46 PM <justina...@gmail.com> wrote:

Hi, we are running experiment with 1 gpu per executor, and while it says 4 gpus available, once we start the exeriment it tries to spin up 2 executors and gpu count reaches 5 queued gpu requests and our experiment cannot run. It should only use 2 GPUs in total, but looks like 9 are being reserved? Could you please look into this?

--
You received this message because you are subscribed to the Google Groups "Hops" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hopshadoop+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hopshadoop/843ea697-d4d4-458a-8a2c-f966ab3b5d70%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[justina...@gmail.com]

Hi, I think the problem with GPU queue is back, it worked whatever you did the last time, so maybe you could do that again when you can? Many thanks!

[Robin Andersson]

Hey! Looked into it and it is not actually a problem, although the UI is misleading to some extent. The GPUs are available but there is not enough memory on the machines to give you a container. I see you are running on 8 GPUs right now though. In the future please don't allocate more than 6 at a time to allow for other users to also make use of them. We are working on limiting how many GPUs users may use.

BR,

Robin

To view this discussion on the web visit https://groups.google.com/d/msgid/hopshadoop/e5a4419d-5e8c-429f-a149-25f269b56ed3%40googlegroups.com.

[justina...@gmail.com]

Thanks for the quick reply. We are running a quick grid search experiment, it should take around 1 hour. when I started the job, it said 13 GPUS were available that is why I wondered if it was correct. It's true, we're using many GPUs but only for a short while, promise :)