GPU training often suspends without completion... And 40.mlp.ipynb tutorial issue

[Ross Andrew Donnachie]

Working through https://github.com/denizyuret/Knet.jl/blob/master/tutorial/40.mlp.ipynb I found that, in order to dismiss my GPU completely, I needed to define mnistdata(,xtype=atype()) where atype() = Array{Float32}. Otherwise the mnistdata is of type KnetArray...

I will make a Pull Request on this, if that is wanted.

The reason I disabled using my GPU is because I would often have training suspend at an arbitrary point... Some other times it would crash because of running out of memory... and sometimes it would succeed, although without really using my GPU, only filling up the dedicated vram...

I have tried to investigate this... And thought that maybe I should attempt to clear the cached memory after the Knet.gc() within the tutorial's trainResults() func..

This is not advised though, as on the page https://juliagpu.gitlab.io/CUDA.jl/usage/memory/.

Are there any thoughts or better suggestions?

I have yet to test calling `CuArrays.reclaim()` and will post the results here when I have them...

Any existing tests or complaints on this?

Windows, Julia 1.4.1, Knet#master

Regards, Ross

[Ross Andrew Donnachie]

Okay. I am getting far more stable performance after an update of the CUDA packages.

So all that remains is the mentioned update of the mlp tutorial (first sentences above)

[Deniz Yuret]

Knet.atype()=Array{Float32} should address the "use the cpu even if my machine has a gpu" issue.

Otherwise Knet prefers to use the GPU if it can find one.

--
You received this message because you are subscribed to the Google Groups "knet-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knet-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knet-users/228bd14d-61ec-46a9-bb74-622d56a46d73%40googlegroups.com.