Using multiple GPUs

[Swami]

Is it possible to load different nets (and their solvers) in different gpu's, say using pycaffe ?

[sk06]

Hi,

Yes, using Jupyter Notebook, you can load different nets in different gpus and you can simultaneously train the nets. I use six gpus for training six different nets and two gpus for testing two different nets simultaneously.

[swami]

That's interesting. Can you explain (pseudocode is sufficient) how you made that happen using pycaffe ?

--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/BqegxsB0UxE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/5119f005-557f-445b-b403-573171372268%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--Swami

[Sai Kiran]

Please check the below screenshot. You can create multiple notebooks and train different networks on different gpus. Hope this helps you.

--

Saikiran

[swami]

Thanks for the response. I think there is a misunderstanding. Perhaps I should have been more clearer, Sorry!

What I originally meant was: I have two nets which are trained alternatively (N iters for the Net1 and N iters for Net2 and this cycle repeats each iteration).

Net1 and Net2 share a few layers. Now when I start the training in pycaffe, Is it possible to use different gpu's for Net1 and Net2 ?

--

--Swami

[Patrick McNeil]

Swami,

If I am understanding correctly, you are looking to have something like:

• Net1 has layers 1 - 8
• Net2 has layers 1- 8
• The nets share layer 3 - 5 (the other layers are unique).

Is that correct?

I think what you would need to do is something along the lines of the following:

1. Setup the two nets to train like normal on separate GPUs
1. Create Net1 - Use the GPU0 (using the command caffe.set_device(0) to set the GPU)
2. Create Net2 - Use GPU1 (using the command caffe.set_device(1) to set the GPU)
2. For your given number of training iterations:
1. Run a training iteration set (40 iterations or whatever) on each net and then stop the training
2. Take the updated model for the given layers (3-5 in my example) and combine them (since they are trained separately you would have to write this code)
3. Update the models
3. Save the models when finished

I don't think there is a way to coordinate training between different processes using separate GPUs.

An alternative method is to create a single model using multiple inputs and multiple outputs.  So, using the Net1 and Net2 with 8 layers each as above (they wouldn't need to be the same number of layers), you could have the Net1 layers 1 - 2 as one leg of the network, Net2 layers 1 - 2 as a second leg and feed both legs into layer 3 (which is shared).  Layer 5 would feed into to separate output legs for each Net.  If you needed them separate for your application, you could then use net surgery to copy the layer weights to each net separately.  In order to use both GPUs, you could use both GPUs training the same combined network.

Hopefully this helps.

Patrick

On Tuesday, February 14, 2017 at 1:07:52 PM UTC-6, Swami wrote:

Thanks for the response. I think there is a misunderstanding. Perhaps I should have been more clearer, Sorry!

What I originally meant was: I have two nets which are trained alternatively (N iters for the Net1 and N iters for Net2 and this cycle repeats each iteration).

Net1 and Net2 share a few layers. Now when I start the training in pycaffe, Is it possible to use different gpu's for Net1 and Net2 ?

On Tue, Feb 14, 2017 at 1:59 PM, Sai Kiran <saikira...@gmail.com> wrote:

Please check the below screenshot. You can create multiple notebooks and train different networks on different gpus. Hope this helps you.

On Tue, Feb 14, 2017 at 8:48 PM, swami <swam...@gmail.com> wrote:

That's interesting. Can you explain (pseudocode is sufficient) how you made that happen using pycaffe ?

On Tue, Feb 14, 2017 at 8:19 AM, sk06 <saikira...@gmail.com> wrote:

Hi,

Yes, using Jupyter Notebook, you can load different nets in different gpus and you can simultaneously train the nets. I use six gpus for training six different nets and two gpus for testing two different nets simultaneously.

On Tuesday, February 14, 2017 at 11:12:25 AM UTC+5:30, Swami wrote:

Is it possible to load different nets (and their solvers) in different gpu's, say using pycaffe ?

--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/BqegxsB0UxE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.

To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/5119f005-557f-445b-b403-573171372268%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--Swami

--

Saikiran

--

--Swami

[Swami]

H Patrick,

What you have described is what I am trying to do. My approach was going to be similar to what you have outlined.

But strangely, I am struck in Step 1:

1. Setup the two nets to train like normal on separate GPUs
1. Create Net1 - Use the GPU0 (using the command caffe.set_device(0) to set the GPU)
2. Create Net2 - Use GPU1 (using the command caffe.set_device(1) to set the GPU)

Here, it seems like using pycaffe, it is not possible to instantiate two GPUs in the same script.

I get an error saying "invalid resource handle", but of course each gpu device is indeed operational.

Am I missing something trivial here ? Are you able to accomplish something like this ?

[Patrick McNeil]

I am not sure why you would be getting an invalid resource handle error.  One way to do this would be to instantiate two "caffe" modules.

So, something like the following:

# Import and setup the caffe environment (adjust to your actual location)

caffe_root = '/usr/local/src/caffe'
sys.path.insert(0, caffe_root + 'python')
import caffe as caffe1
import caffe as caffe2

# Setup the Caffe Environment
caffe1.set_device(0)
caffe1.set_mode_gpu()

caffe2.set_device(1)
caffe2.set_mode_gpu()

SPROTO1 = sys.argv[1]
SMODEL1 = sys.argv[2]

SPROTO2 = sys.argv[3]
SMODEL2 = sys.argv[4]

# Read in the network
net1 = caffe1.Net(SPROTO1, SMODEL1, caffe1.TRAIN)
net2 = caffe2.Net(SPROTO2, SMODEL2, caffe2.TRAIN)

# Put your training stuff here

Basically, the idea would be to have to of the Caffe modules in the single script.  You would then be able to run the networks simultaneously (you will need to implement some parallel programming in Python to execute the training process at the same time).

Patrick