Bug in gpuarray triu/tril (work-around found), affects slinalg.Cholesky

[Paul Baggenstoss]

Hello,

The slinalg.Cholesky does not work for float64. I traced the problem to
     theano/gpuarray/linalg.py  
where the functions triu,tril  (imported from pygpu.basic) are used. These
functions only support float32. I commented out applying triu/tril to the
Cholesky output result of the cusolver, and the problem went away. Of course,
Cholesky then returns a full matrix, but this is not a problem. One could apply
triu/tril in python.

Are there plans fo add float64 support in pygpu.basic?

[Paul Baggenstoss]

OK, I fixed the bug, I just need someone to commit it to Theano github as I don't have access.


I added support for triu/tril for  float64 in /pygpu/basic.py

First, I added a version of _generate_kernel specially for float64. Here is the function:

 def _generate_kernel64(ctx, cols, upper=True):
    tmpl = Template("""
    #include "cluda.h"
    KERNEL void extract_tri(GLOBAL_MEM ga_double *a, ga_size a_off, ga_uint N) {
        a = (GLOBAL_MEM ga_double *)(((GLOBAL_MEM char *)a) + a_off);
        unsigned int idx = GID_1 * LDIM_0 * GDIM_0 +
                           GID_0 * LDIM_0 + LID_0;
        unsigned int ix = idx/${cols};
        unsigned int iy = idx%${cols};
        if (idx < N) {
            if (ix ${le} iy)
                a[idx] = 0.0;
        }
    }
    """)
    if upper:
        le = '>'
    else:
        le = '<'
    src = tmpl.substitute(cols=cols, le=le)
    spec = [GpuArray, SIZE, 'uint32']
    k = GpuKernel(src, "extract_tri", spec, context=ctx)
    return k


Then I put in the conditions into both triu and tril, replacing

k = _generate_kernel(A.context, cols, upper)

with

if A.dtype=='float64':
    k = _generate_kernel64(A.context, cols, upper)
 elif A.dtype=='float32':
    k = _generate_kernel(A.context, cols, upper)
 else:
    raise ValueError("triu only works for float32,float64")

This fixes both slinalg.Cholesky and its gradient for float64.
Can someone please commit this fix?

Paul

[Wong Hang]

Hi Paul,

What version of theano and pygpu are you using?

I think latest version of pygpu should support float64

https://github.com/Theano/libgpuarray/blob/master/pygpu/basic.py

Best,

wonghang

Paul Baggenstoss <p.m.bag...@ieee.org> 於 2020年1月16日 週四 下午7:18寫道：

--

---
You received this message because you are subscribed to the Google Groups "theano-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/theano-dev/c6d7d62b-8ec7-4a8e-b425-09837394dd90%40googlegroups.com.

[Paul Baggenstoss]

Yes, I see that the new version seems to support 64 bit, but I wonder why I did not get the latest version with pip.

Paul

On Wednesday, January 15, 2020 at 11:02:39 AM UTC+1, Paul Baggenstoss wrote:

[Wong Hang]

theano is no longer maintained.

Would you mind to tell me why do you continue to use theano other than other frameworks like Tensorflow?

Paul Baggenstoss <p.m.bag...@ieee.org> 於 2020年1月16日 週四 下午8:54寫道：

--

---
You received this message because you are subscribed to the Google Groups "theano-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/theano-dev/79c08ebe-4521-409c-8374-3227532565e2%40googlegroups.com.

[Paul Baggenstoss]

Yes,
Theano is much better from my point of view because it is more straight-forward.
Tensorflow is changing to quickly and is way more than I need. I am
creating new types if neural networks, writing new types of activation functions,
and new network structures. For this, I prefer Theano.

Are you still using Theano? Isn't there still a community of users
who "maintain" it?
Paul

On Thursday, January 16, 2020 at 1:05:40 PM UTC+1, Wong Hang wrote:

theano is no longer maintained.

Would you mind to tell me why do you continue to use theano other than other frameworks like Tensorflow?

Paul Baggenstoss <p.m.ba...@ieee.org> 於 2020年1月16日 週四 下午8:54寫道：

Yes, I see that the new version seems to support 64 bit, but I wonder why I did not get the latest version with pip.
Paul

On Wednesday, January 15, 2020 at 11:02:39 AM UTC+1, Paul Baggenstoss wrote:

Hello,

The slinalg.Cholesky does not work for float64. I traced the problem to
     theano/gpuarray/linalg.py  
where the functions triu,tril  (imported from pygpu.basic) are used. These
functions only support float32. I commented out applying triu/tril to the
Cholesky output result of the cusolver, and the problem went away. Of course,
Cholesky then returns a full matrix, but this is not a problem. One could apply
triu/tril in python.

Are there plans fo add float64 support in pygpu.basic?

--

---
You received this message because you are subscribed to the Google Groups "theano-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to thean...@googlegroups.com.

[Wong Hang]

Yes. I am still using Theano. Theano is the only framework that allow me to use my own CUDA code (with pycuda) seamlessly.

Some guys in MILA "maintain" it and continue to merge relevant PR. 

Paul Baggenstoss <p.m.bag...@ieee.org> 於 2020年1月16日 週四 下午11:21寫道：

To unsubscribe from this group and stop receiving emails from it, send an email to theano-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/theano-dev/ee6300de-f789-42d9-91d0-e45b7a964cc9%40googlegroups.com.