Trouble Accessing value of Casted/Shared Tensor

[Mark]

Hi folks,

Let me motivate the following code that demonstrates my current difficulty.  Suppose I am reading in data (xs and ys) that are going to be used for a network model with a logistic output layer.  So, I want the ys to be some integer type.  As in the deep learning tutorial, you can use T.cast to do this.  However, once I do that, I'm losing my ability to access the shared variable inside the cast tensor.  My first try was using myCastedShared.get_value() but that fails because a TensorVariable doesn't have a get_value member function.  

It seems like you should use get_constant_value() to get the wrapped, shared variable, but you can see what happens with my sample program which follows this message.

I need to verify (for debugging purposes) the y values in between the T.cast call and when I pass them to training and testing functions.  Any thoughts?  Am I doing something wrong?

Best,
Mark

*********************************************************************************

start sample code file "debugging-shared.py"

*********************************************************************************

#!/usr/bin/env python

import theano

import theano.tensor as T

import numpy as np

intArray   = np.array([1,2,3], dtype='int64')

fltArray = np.array([1,2,3], dtype='float64')

print "***as expected***"

print intArray

print fltArray

sharedInt = theano.shared(intArray)

sharedFlt = theano.shared(fltArray)

print "***as expected***"

print sharedInt, sharedInt.get_value()

print sharedFlt, sharedFlt.get_value()

fltArray2 = np.array([1,2,3], dtype='float64')

castedFlt = T.cast(theano.shared(fltArray2), 'int32')

print "!!!trouble ahead!!!"

print castedFlt, castedFlt.get_constant_value()

*********************************************************************************

start output:

*********************************************************************************

% python debugging-shared.py 

Using gpu device 0: GeForce 8300

***as expected***

[1 2 3]

[ 1.  2.  3.]

***as expected***

<TensorType(int64, vector)> [1 2 3]

<TensorType(float64, vector)> [ 1.  2.  3.]

!!!trouble ahead!!!

Elemwise{Cast{int32}}.0

Traceback (most recent call last):

  File "debugging-shared.py", line 25, in <module>

    print castedFlt, castedFlt.get_constant_value()

  File "/home/mfenner/created/bytype/code/python/theano/Theano/theano/tensor/basic.py", line 1654, in get_constant_value

    return get_constant_value(self)

  File "/home/mfenner/created/bytype/code/python/theano/Theano/theano/tensor/basic.py", line 501, in get_constant_value

    const = get_constant_value(v.owner.inputs[0])

  File "/home/mfenner/created/bytype/code/python/theano/Theano/theano/tensor/basic.py", line 556, in get_constant_value

    raise TypeError(v)

TypeError: <TensorType(float64, vector)>

[James Bergstra]

Maybe cast the fltArray2 in numpy before creating the shared variable?

[Pascal Lamblin]

On Tue, Jul 31, 2012, Mark wrote:
> I need to verify (for debugging purposes) the y values in between the
> T.cast call and when I pass them to training and testing functions. Any
> thoughts? Am I doing something wrong?

You can try to use test values:
http://deeplearning.net/software/theano/tutorial/debug_faq.html#using-test-values

--
Pascal

[Mark]

James,

That was my initial attempt -- basically using the intArray instead of fltArray.  Unfortunately, later on in the process, a theano function gets very unhappy (throws an exception) if I don't do it in the order I demonstrated.  I don't know why and the exception is buried deeply in the internal parsing logic so it is long and cryptic (to me).

Mark

[Frédéric Bastien]

The function get_contant_value() consider only a few cases. It do not
compile a theano function to get the results.

I think the current way to make this work is by compiling a theano
function and call it to get the value. We are planning doing something
like:

var.eval() to make this easier, but it is not done yet.


I'm wondering, what is the complicated error you have? We try to make
the error message better, so if you can give us more detail or send us
a way to replicate this, we can fix this.

Also, I think it is better to cast before creating the shared variable.

Fred

[Mark]

Fred,

I would far prefer to cast first and then create the shared variable.  If nothing else, it reduces the demand on the symbolic computation; it also puts the data in its "natural" format as early as possible.  However, when I tried using the integer arrays (which then got passed on to a neural network training function), I got into quite a bit of a problem.  

It was one of my first steps with theano, so I don't remember it well and I might have been doing something else that made the error more troublesome.  I'll see if I can recreate it.

Also, I did find a way to look at the data involved in the cast:  basically, use the owner attribute of the tensor to work back to the symbolic graph, then go up the graph and get_value on the proper input.  Fixed code for my original example from the top of this thread follows.

Thanks,

Mark

******************

debugging-shared.py

******************

#!/usr/bin/env python

import theano

import theano.tensor as T

import numpy as np

intArray   = np.array([1,2,3], dtype='int64')

fltArray = np.array([1,2,3], dtype='float64')

print "***as expected***"

print intArray

print fltArray

sharedInt = theano.shared(intArray)

sharedFlt = theano.shared(fltArray)

print "***as expected***"

print sharedInt, sharedInt.get_value()

print sharedFlt, sharedFlt.get_value()

fltArray2 = np.array([1,2,3], dtype='float64')

castedFlt = T.cast(theano.shared(fltArray2), 'int32')

print "***this works!***"

print castedFlt,

print castedFlt.owner.inputs[0].get_value()

***************

output

***************

% python debugging-shared.py

Using gpu device 0: GeForce 8300

***as expected***

[1 2 3]

[ 1.  2.  3.]

***as expected***

<TensorType(int64, vector)> [1 2 3]

<TensorType(float64, vector)> [ 1.  2.  3.]

this works!

Elemwise{Cast{int32}}.0 [ 1.  2.  3.]

[Frédéric Bastien]

Hi,

So all your problem is fixed if I understand you correctly? If you get
into problem when making the cast before making the shared variable,
tell us.

Fred

[Mark]

Fred,

Thanks for checking in on me.  

I reworked much of the system and I paid much closer attention to the numerical data formats (numpy arrays) and the symbolic formats (theano tensors). In my initial implementation, I had "settled" on using float matrices (for the tensors) as a "least common denominator" to make every thing work.  I needed to get my standard network and my rnn playing nicely together so I had to refactor the types so they could take the same input data.  I rewrote both implementations to use the correct vectors/cols/rows etc. with proper types.  

It was a somewhat frustrating process because: 

(1) I was developing an understanding of Theano's typing system (which is mostly straightforward, but I didn't appreciate the vector, row, col distinction), 

(2) trying to understand theano's error messages (which are fairly helpful, but interpreting them requires practice -and- sometimes the path from the input or symbolic mistake to the error message is long), and 

(3) I was figuring out the right set of debugging techniques to work with theano.  My most useful one were those listed here (which includes the test values you mentioned to me): http://deeplearning.net/software/theano/tutorial/debug_faq.html.  Also, I developed a "more explicit" version of test values that looked like this:

x, y = T.matrix(), T.lcol()

sampleX = np.random.rand(20, 4)

sampleY = np.random.randint(0, 2, (20,1))

tricky_result = "some T.function that I don't understand"

out = theano.function([x], tricky_result)(sampleX)

print out.shape # numerical version of tricky_result using sampleX

print out.dtype

print out

I typically dropped into pdb (import pdb; pdb.set_trace()) instead of explicitly printing stuff out.  I'm not sure if test_values have -- or could have -- a setting that would let this be automated (or perhaps that is what you were thinking with  .eval().

(4) typical problems of mis-matched dimensions that always show up in numerical methods (which is generally just user -- that is, my --  carelessness).

Best,
Mark

[李亚]

Hi, Mark,
   I encountered the same problem as you, I try to solve the problem as you said， but it doesn't work. Would you please give me some hlep?

This is the code from the logistic regression:
  
    datasets = load_data(dataset)
    train_set_x, train_set_y = datasets[0]
    valid_set_x, valid_set_y = datasets[1]
    test_set_x, test_set_y = datasets[2]
  
   train_set_x.get_value() really works, but when I use
   train_set_y.owner.inputs[0].get_value() , it does not work and gives the unexpected error
   "AttributeError: 'TensorVariable' object has no attribute 'get_value'"
 
   Do you have any idea what is wrong? I have tried your example code, it really works well.

在 2012年8月1日星期三UTC+8上午2时41分04秒，Mark写道：

[Frédéric Bastien]

Hi,

only shared variable have a get_value attribute. From memory, all the dataset should be shared variable, so from memory this should work:

train_set_y.get_value(). But from the error, we see that train_set_y.owner exist, so it mean train_set_y isn't a shared variable. Did you changed that?

Can you print that is train_set_y like this:

theano.printing.debugprint(train_set_y)

Fred

--
 
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ｙa Li]

Hi Fred,
  theano.printing.debugprint(train_set_y) gives out like this:

 Elemwise{Cast{int32}} [@117570320] ''  
 |HostFromGpu [@118639312] ''  
 | |<CudaNdarrayType(float32, vector)> [@118638544]

Could you figure out what is wrong? I think I did not change the train_set_y.

在 2013年9月14日星期六UTC+8上午3时35分27秒，Frédéric Bastien写道：

[Pascal Lamblin]

On Sun, Sep 15, 2013, Ｙa Li wrote:
> Hi Fred,
> theano.printing.debugprint(train_set_y) gives out like this:
>
> Elemwise{Cast{int32}} [@117570320] ''
> |HostFromGpu [@118639312] ''
> | |<CudaNdarrayType(float32, vector)> [@118638544]
>
> Could you figure out what is wrong? I think I did not change the
> train_set_y.

So, as Fred said, only shared variables actually contain a value, and
have a get_value() method. That's the case for train_set_x, but not
for train_set_y, which is a symbolic expression transforming a shared
variable stored as a float32 vector on the GPU (the one printed as
"<CudaNdarrayType(float32, vector)> [@118638544]" into an int32 vector.

If you want to compute a value from that symbolic expression, starting
from the current value of the shared variable, you can do the following:

train_set_y.eval()

However, this is not an efficient way of getting that value, and you
should only do that for debugging purposes, not during training or
testing.

During training or testing, you should build a symbolic expression, then
call theano.function(...), as explained in the tutorials.

Hope this helps,

> >>> *********************

> >>> start sample code file "debugging-shared.py"
> >>> ****************************************************************

> >>> *********************

> >>> #!/usr/bin/env python
> >>>
> >>> import theano
> >>> import theano.tensor as T
> >>> import numpy as np
> >>>
> >>> intArray = np.array([1,2,3], dtype='int64')
> >>> fltArray = np.array([1,2,3], dtype='float64')
> >>>
> >>> print "***as expected***"
> >>> print intArray
> >>> print fltArray
> >>>
> >>> sharedInt = theano.shared(intArray)
> >>> sharedFlt = theano.shared(fltArray)
> >>>
> >>> print "***as expected***"
> >>> print sharedInt, sharedInt.get_value()
> >>> print sharedFlt, sharedFlt.get_value()
> >>>
> >>> fltArray2 = np.array([1,2,3], dtype='float64')

> >>> castedFlt = T.cast(theano.shared(**fltArray2), 'int32')

> >>>
> >>> print "!!!trouble ahead!!!"
> >>> print castedFlt, castedFlt.get_constant_value()
> >>>
> >>> ****************************************************************

> >>> *********************
> >>> start output:
> >>> ****************************************************************
> >>> *********************

> >>> % python debugging-shared.py
> >>> Using gpu device 0: GeForce 8300
> >>> ***as expected***
> >>> [1 2 3]
> >>> [ 1. 2. 3.]
> >>> ***as expected***
> >>> <TensorType(int64, vector)> [1 2 3]
> >>> <TensorType(float64, vector)> [ 1. 2. 3.]
> >>> !!!trouble ahead!!!
> >>> Elemwise{Cast{int32}}.0
> >>> Traceback (most recent call last):
> >>> File "debugging-shared.py", line 25, in <module>
> >>> print castedFlt, castedFlt.get_constant_value()

> >>> File "/home/mfenner/created/bytype/**code/python/theano/Theano/**theano/tensor/basic.py",

> >>> line 1654, in get_constant_value
> >>> return get_constant_value(self)

> >>> File "/home/mfenner/created/bytype/**code/python/theano/Theano/**theano/tensor/basic.py",
> >>> line 501, in get_constant_value
> >>> const = get_constant_value(v.owner.**inputs[0])
> >>> File "/home/mfenner/created/bytype/**code/python/theano/Theano/**theano/tensor/basic.py",

> >>> line 556, in get_constant_value
> >>> raise TypeError(v)
> >>> TypeError: <TensorType(float64, vector)>
> >>>
> >>> --
> >>
> >> ---
> >> You received this message because you are subscribed to the Google Groups
> >> "theano-users" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to theano-users...@googlegroups.com <javascript:>.

> >> For more options, visit https://groups.google.com/groups/opt_out.
> >>
> >
> >
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to theano-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Pascal