sending tensorflow job to remote GPU

99 views
Skip to first unread message

Shreya Bhandare

unread,
Sep 14, 2022, 11:16:44 AMSep 14
to Discuss
Hello,

I understand that tensorflow uses CPU if it doesn not detect GPU or if only tensorflow is being used (instead of tensorflow-gpu). I want to figure out where should I start looking in the source code if I want to instead redirect it to a remote GPU ?

I know of 2 softwares GVirtuS (github), FairGV etc that are capable of executing cuda binaries remotely (over the network) and I was thinking if we can use this principle to execute a tensorflow job remotely ?

Yunhui Zhang

unread,
Sep 22, 2022, 5:22:54 PMSep 22
to Discuss, shreyab...@vt.edu
Tensorflow design with client-server mode. 
You can create graph in client and run it in different server. 
After you create your cluster, you can start your job like 
with tf.device("gpu_worker_1:GPU/0"):
    do this
with tf.device("gpu_workder_1:GPU/1"):
    do that

Khoa Nguyen

unread,
Sep 23, 2022, 3:43:43 AMSep 23
to Yunhui Zhang, Discuss, shreyab...@vt.edu
Don't contact me again!

--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/49e83054-faab-4106-bfda-3f21b7a6ea6bn%40tensorflow.org.


--
facebook
twitter
linkedin
instagram

Khoa Nguyen

Online Marketing Freelancer

Khoa Nguyen Online Marketing Beratung

0176 80220822
dang...@khoa-nguyen.de
https://www.khoa-nguyen.de/
Freibadstraße 30, 81543 München

Shreya Bhandare

unread,
Sep 23, 2022, 1:42:22 PMSep 23
to Discuss, aust...@yelp.com, Shreya Bhandare
Thank you for your response Yunhui, I am research student trying to figure out a "tensorflow as a cloud service" as my research project and and due to that I'm running tensorflow on my VM and want to use a virtualization stack to connect to a VM over a network (TCP/RDMA). which is why I need the source code, where it detect that the current device doesn't have a CPU or nvidia libraries and so I want to add a logic there to use my stack to connect to another GPU. Could you help me out or point to someone who can ? Thank you so much again
the link that you sent is for model level distribution of tensorflow, i think ?

t kevin

unread,
Sep 26, 2022, 10:33:18 PMSep 26
to Shreya Bhandare, Discuss, aust...@yelp.com
Shreya

Tensorflow has a fairly complicated device abstraction layer and it's
definitely much more than redirect a few library calls.
It's important that you understand the current design and
implementation before you'd try some new ideas.

As for "tensorflow as a cloud service", tensorflow colab is a perfect
project you should look into.
https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/beginner.ipynb

Kevin

Shreya Bhandare <shreyab...@vt.edu> 于2022年9月24日周六 01:42写道:
> --
> You received this message because you are subscribed to the Google Groups "Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.
> To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/df9abea9-b18a-46ec-9701-184b9c03f7ccn%40tensorflow.org.

Khoa Nguyen

unread,
Sep 27, 2022, 1:28:03 AMSep 27
to t kevin, Shreya Bhandare, Discuss, aust...@yelp.com
I told you not to spam me again. U want me to Take legal Action? 

Reply all
Reply to author
Forward
0 new messages