Tensorflow with GPU has errors

3,217 views
Skip to first unread message

trialcritic

unread,
Jun 21, 2016, 5:55:48 PM6/21/16
to Discuss
I have a GPU machine with 3 GPUs. When I run my program, I get

I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 

name: Tesla K40c

major: 3 minor: 5 memoryClockRate (GHz) 0.745

pciBusID 0000:03:00.0

Total memory: 11.25GiB

Free memory: 11.15GiB

I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 1 with properties: 

name: Tesla K40c

major: 3 minor: 5 memoryClockRate (GHz) 0.745

pciBusID 0000:04:00.0

Total memory: 11.25GiB

Free memory: 11.15GiB

I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 2 with properties: 

name: Quadro K620

major: 5 minor: 0 memoryClockRate (GHz) 1.124

pciBusID 0000:84:00.0

Total memory: 1.99GiB

Free memory: 1.96GiB


When I run the problem in my MBP, it works fine.


When I run the MNIST program in the GPU box, I do not get these messages. What is the problem?

Yaroslav Bulatov

unread,
Jun 21, 2016, 7:24:37 PM6/21/16
to trialcritic, Discuss
ON MBP you probably see this message instead:

I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] OS X does not support NUMA - returning NUMA node zero

Those are not errors. "I" in the beginning means it was generated with LOG(INFO) which is reserved for informational messages. I would ignore it, there's no NUMA support, even on machines with multiple numa nodes it gives the same message

--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.
To post to this group, send email to dis...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/f9dd0718-e431-404d-8fae-32daa9efbd20%40tensorflow.org.

trialcritic

unread,
Jun 21, 2016, 7:31:10 PM6/21/16
to Discuss
I do not get that message in my MBP. If it is not an error message, then no problem. 
Reply all
Reply to author
Forward
0 new messages