CUDA examples don't work after rebuild

[Eric]

I had to have my system re-built and am in the process of re-installing everything.  I'm trying to figure out why the CUDA examples no longer work though the CPU and OpenCL examples do.  The only thing I've intentionally done differently is install Visual Studio 2015 instead of 2013.  Here's what I've done in order:

1. Install Visual Studio 2015 Professional Update 2
2. Install CUDA Toolkit 7.5
3. Install ArrayFire 3.3.2 from pre-built libraries
4. Add C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.5\nvvm\bin to my system path so nvvm64_30_0.dll is available
5. Set my Quadro K2200 to TCC mode (it does not drive any displays; I have an older Quadro 4000 driving my displays).

helloworld_opencl gives:
c:\Program Files\ArrayFire\v3\examples\helloworld>helloworld_opencl
ArrayFire v3.3.2 (OpenCL, 64-bit Windows, build f65dd97)
[0] NVIDIA  : Quadro K2200, 4095 MB
-1- NVIDIA  : Quadro 4000, 2048 MB

nvidia-smi gives:
c:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi
Fri May 13 14:48:20 2016
+------------------------------------------------------+
| NVIDIA-SMI 353.90     Driver Version: 353.90         |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro 4000        WDDM  | 0000:03:00.0      On |                  N/A |
| 41%   85C   P12    N/A /  N/A |    285MiB /  2048MiB |    ERR!      Default |
+-------------------------------+----------------------+----------------------+
|   1  Quadro K2200        TCC  | 0000:04:00.0     Off |                  N/A |
| 42%   45C    P8     1W /  39W |     12MiB /  4095MiB |    ERR!      Default |
+-------------------------------+----------------------+----------------------+

But when I run helloworld_cuda it hangs. It does not even get to the af::info() statement, it just churns away on a single core of the system.  I've tried setting the device to either 0 or 1.

Any thoughts as to what I'm forgetting?  Or do I really need to downgrade back to VS 2013?

Thanks,
Eric

[Shehzan Mohammed]

Hi Eric

We have tested the examples before with VS2015 and they are known to run well.

One possibility is that it might be repopulating the CUDA kernel cache. This is known to take a while if the computes were not included in the original build (We do include compute 20/30/35/37/50/52).

Are you running it in debug or release mode? Also, you could try running using Ctrl+F5 instead of the debugger.

You could also do a break all after you start the debugger and see where exactly its paused.

-Shehzan

[Eric]

Thanks for the info.  I also realized it can't be a Visual Studio issue since I get the same behavior with the pre-built EXE files.

I was running in Release mode but have now tried Debug as well.  It hangs at af::setDevice().  If I download the master branch from GitHub will that allow me to debug into device.cpp (VS 2015 asks for the location of this file when I try to step into it)?  That will be a last resort, though.  First I'll try switching back from TCC to WDDM mode to see if that makes a difference.  Then I may try re-installing CUDA and ArrayFire.

Thanks again,
Eric

[Shehzan Mohammed]

From everything you said, it does look like CUDA might just be caching the kernels. Can you just let it run in release mode. This might be in the order of minutes.

Let me know how it goes.

Shehzan

[Eric]

I'll let it run for a while using the pre-built helloworld_cuda.exe file.

Interestingly, here's what I've done:

1. Switched K2200 back to WDDM mode.
2. helloworld_cuda.exe (prebuilt) works fine.
3. Uninstall AF 3.3.2 and re-install 3.3.1 (which is what I was previously using before my rebuild).
4. helloworld_cuda.exe (prebuilt) works fine.
5. Re-install AF 3.3.2 (which takes care of the 3.3.1 un-install for me).
6. helloworld_cuda.exe (prebuilt) is seemingly hung, but I'll let it run for a while.  It may be doing something as it's using a whole CPU core and the memory usage is fluctuating between about 300MB and 1 GB.

-Eric

[Eric]

Whoops, this should have read

1. Switched K2200 back to WDDM mode.

2. helloworld_cuda.exe (prebuilt) still hangs.

[Eric]

After 10 minutes helloworld_cuda.exe (pre-built) finally did succeed.  I started it again (by just up-arrowing at the command prompt and re-running it) and it looks like it's going to take just as long again.  I can't imagine this is the expected behavior.  Should this cache persist between function calls?  Otherwise it looks like with 3.3.2 I'll have to wait 10 minutes for even a simple application to run.

-Eric

[Pavan Yalamanchili]

Hi Eric

The cache should persist. Looks like something is wrong.. 

On Fri, May 13, 2016, 18:08 Eric <eric.a....@gmail.com> wrote:

After 10 minutes helloworld_cuda.exe (pre-built) finally did succeed.  I started it again (by just up-arrowing at the command prompt and re-running it) and it looks like it's going to take just as long again.  I can't imagine this is the expected behavior.  Should this cache persist between function calls?  Otherwise it looks like with 3.3.2 I'll have to wait 10 minutes for even a simple application to run.

-Eric

--
You received this message because you are subscribed to the Google Groups "ArrayFire Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to arrayfire-use...@googlegroups.com.
To post to this group, send email to arrayfi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/arrayfire-users/5d6e9fa6-8aaa-4201-b34d-f24bcdc6a220%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shehzan Mohammed]

Ok, clearly this is my fault. :-(.

We were testing building installers with static MKL and eventually decided to put it off for later.

In the middle of all that, I made the mistake of copy-pasting which excluded some computes (including 50).

I'm going to generate these installers again and upload them.

I'll post here when it's ready.

Sorry :-(.

-Shehzan

On Friday, May 13, 2016 at 6:10:08 PM UTC-4, Pavan Yalamanchili wrote:

Hi Eric

The cache should persist. Looks like something is wrong.. 

On Fri, May 13, 2016, 18:08 Eric <eric.a....@gmail.com> wrote:

After 10 minutes helloworld_cuda.exe (pre-built) finally did succeed.  I started it again (by just up-arrowing at the command prompt and re-running it) and it looks like it's going to take just as long again.  I can't imagine this is the expected behavior.  Should this cache persist between function calls?  Otherwise it looks like with 3.3.2 I'll have to wait 10 minutes for even a simple application to run.

-Eric

--
You received this message because you are subscribed to the Google Groups "ArrayFire Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to arrayfire-users+unsubscribe@googlegroups.com.
To post to this group, send email to arrayfire-users@googlegroups.com.

[Pavan Yalamanchili]

Hmm, nevertheless it shouldn't keep trying to create the cache over and over. Perhaps a driver bug that we need to file with NVIDIA ?

On Fri, May 13, 2016 at 6:13 PM Shehzan Mohammed <she...@arrayfire.com> wrote:

Ok, clearly this is my fault. :-(.

We were testing building installers with static MKL and eventually decided to put it off for later.

In the middle of all that, I made the mistake of copy-pasting which excluded some computes (including 50).

I'm going to generate these installers again and upload them.

I'll post here when it's ready.

Sorry :-(.

-Shehzan

On Friday, May 13, 2016 at 6:10:08 PM UTC-4, Pavan Yalamanchili wrote:

Hi Eric

The cache should persist. Looks like something is wrong.. 

On Fri, May 13, 2016, 18:08 Eric <eric.a....@gmail.com> wrote:

After 10 minutes helloworld_cuda.exe (pre-built) finally did succeed.  I started it again (by just up-arrowing at the command prompt and re-running it) and it looks like it's going to take just as long again.  I can't imagine this is the expected behavior.  Should this cache persist between function calls?  Otherwise it looks like with 3.3.2 I'll have to wait 10 minutes for even a simple application to run.

-Eric

--
You received this message because you are subscribed to the Google Groups "ArrayFire Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to arrayfire-use...@googlegroups.com.
To post to this group, send email to arrayfi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/arrayfire-users/5d6e9fa6-8aaa-4201-b34d-f24bcdc6a220%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "ArrayFire Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to arrayfire-use...@googlegroups.com.
To post to this group, send email to arrayfi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/arrayfire-users/326ff434-2bd4-4078-ba95-03a96b49cbd5%40googlegroups.com.

[Eric]

I played around with the "CUDA_VISIBLE_DEVICES" environment variable and can confirm that I can't get the Quadro K2200 to load a cache quickly.  If I disable it then the Quadro 4000 (older, compute 2.0 I think), works as expected.

Thanks,
Eric

[Shehzan Mohammed]

The fixed installers are on the website now.

-Shehzan

[Eric]

I got a chance to test the new installer today and everything is working great.  Thanks for your help.

-Eric