GPU memory usage does not increase when using multithreading in python

[Yuhua Wei]

Hi, all

   

I am working on deploy some object detection models (yolo v3) on the GPU of our server. However, I found that, when I use multithreading programming in python to deploy multiple models and let them predict in parallel, the GPU memory usage does not increase, the GPU memory usage is almost as much as one single model, which makes me very confused. Could someone help me?

[Yuhua Wei]

load and predict one by one: 

**model 0

gpu usage：total 16280.88 MB， used 859.56 MB， unused 15421.31 MB

model 1

gpu usage：total 16280.88 MB， used 1371.56 MB， unused 14909.31 MB

model 2

gpu usage：total 16280.88 MB， used 1371.56 MB， unused 14909.31 MB

model 3

gpu usage：total 16280.88 MB， used 1371.56 MB， unused 14909.31 MB

2021-07-05 16:45:00.227119: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7

gpu usage：total 16280.88 MB， used 6649.56 MB， unused 9631.31 MB

gpu usage：total 16280.88 MB， used 6649.56 MB， unused 9631.31 MB

gpu usage：total 16280.88 MB， used 6649.56 MB， unused 9631.31 MB

gpu usage：total 16280.88 MB， used 6649.56 MB， unused 9631.31 MB

**

load one by one and predict through multithread: 

**model 0

gpu usage：total 16280.88 MB， used 861.56 MB， unused 15419.31 MB

model 1

gpu usage：total 16280.88 MB， used 1373.56 MB， unused 14907.31 MB

model 2

gpu usage：total 16280.88 MB， used 1373.56 MB， unused 14907.31 MB

model 3

gpu usage：total 16280.88 MB， used 2397.56 MB， unused 13883.31 MB

2021-07-05 17:06:22.285915: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7

thread 0 gpu usage：total 16280.88 MB， used 6651.56 MB， unused 9629.31 MB

thread 1 gpu usage：total 16280.88 MB， used 6651.56 MB， unused 9629.31 MB

thread 2 gpu usage：total 16280.88 MB， used 6651.56 MB， unused 9629.31 MB

thread 3gpu usage：total 16280.88 MB， used 6651.56 MB， unused 9629.31 MB

**

Yuhua Wei 在 2021年7月5日 星期一下午5:34:52 [UTC+8] 的信中寫道：