Does tensorflow serving supports multiple GPUs?

[Jorge Muñoz]

Hi

I am aware you can run tensorflow with multiple GPUs, but I was wondering if tensorflow serving supports multiple GPUs. If with the exported model for one GPU tensorflow serving is able to replicate it in every GPU and then perform the classification.

If tensorflow serving is no able to do what I said, is it possible to export and load a model which supports for multiple GPUs? The same way it does the example of the inception for training in multiple GPUs: https://github.com/tensorflow/models/blob/master/inception/inception/inception_train.py (I am not sure the configuration for each GPU is stored in the graph when exported and then loaded correctly).

Thanks in advance,

Jorge

[Martin Wicke]

Serving uses tensorflow in the backend, so it will make use of multiple GPUs if you have them. 

--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.
To post to this group, send email to dis...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/7930f3d6-56c4-4035-a416-29911c194ecf%40tensorflow.org.

[Jorge Muñoz]

Thanks Martin. I couldn't find in the documentation that tensorflow serving uses more than one GPU when they are available.

[trialcritic]

Here is the page on GPUs

Tensorflow using GPUs

It has a section, Using Multiple GPUs

[Jorge Muñoz]

Those docs are for tensorflow, I can run the training in multiple GPUs already. My question was related to tensorflow serving once you export the inference layer and load it in the server if it makes use of multiple GPUs without any tweak, as it happens in the training were you have to specify the number of GPUs and handle how to split the data.

[Martin Wicke]

serving will just take the graph you have and run it. If your graph works on multiple GPUs, it will work on multiple GPUs when serving. If your serving infrastructure is the same as the one you used in training, that should work without touching the model itself.

Martin

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/f08b25e4-d0e5-4050-9222-3468ab4b9451%40tensorflow.org.

[Toby Chan]

If you're using TensorFlow serving with GPUs, there is one more thing you need to know when running with bazel.

We had the problem to compile and run the script on GPUs and please check out this issue https://github.com/tensorflow/tensorflow/issues/3708 . It seems not a bug and you always need to build with `--config=cuda` parameter.

[黄炎久]

I have run my graph in multiple GPUs， however when I enable the log_devices_placment，it shows that the placer only assign the node to "gpu:0". 

If there is some rules to build graph which would takes advantage of multiple GPUs, please explain it. Thank you. 

在 2016年6月30日星期四 UTC+8下午9:03:07，Jorge Muñoz写道：

[ricky singh]

No tensorflow serving not making use of multiple GPUs 

https://github.com/tensorflow/tensorflow/issues/19640

https://github.com/tensorflow/serving/issues/311

[Jorge]

Please do not bump old threads. The answer to my question is that TF serving uses the graph you exported, so if your graph supports multiple GPUs TF does support them too, as it is calling the normal TF. But this solution requieres a bit more of effort and it is an adhoc solution for a fixed number of GPUs, so no the best way to generalize for different types of machines.

--
You received this message because you are subscribed to a topic in the Google Groups "Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/a/tensorflow.org/d/topic/discuss/B2aSlrWFsuA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss+u...@tensorflow.org.

To post to this group, send email to dis...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/9906ba68-2dd9-4622-b7c7-cf4ad8aeb80b%40tensorflow.org.

[ricky singh]

Thanks for the tip!