Dealing with None in shape ops

[Dom Luna]

I'm trying to deal with the situation where my computation depends on the shape of the input.

For example:

n = foo.get_shape()[0] # n is used later

The problem is n is always Dimension(None).

TypeError: Expected int32, got Dimension(None) instead.

Is there a way to get around this?

Also, because this would in an indirect way solve the problem. Should I use tf.tile instead of a loop?

In numpy loops are slow so tiling is preferred but it might be different in Tensorflow.

thanks

[Rafał Józefowicz]

Hi Dom,

The reason you get this problem is that the shape isn't know until the graph is execute (as it depends on the input). The way to get around it is to use tf.shape(.) op. In your case:

n = tf.shape(foo)[0]  # n is now a tensor

Now, there a few things to keep in mind. If you want to use n for defining a shape of some other operation, you cannot use it as easily. For example:

tf.reshape(foo, [n, 1])   -- this works if n is an integer but doesn't if n is a tensor.

Many ops take as input a list of ints OR a tensor so to fix this, we have to build a tensor that will contain 2 elements, n and 1. The easiest way to do that is with tf.pack operation:

tf.reshape(foo, tf.pack([n, 1]))

In TensorFlow tile is also preferred to loops whenever feasible (and generates much smaller computational graph). You can also write your own ops if you have a very complicated logic.

-Rafal

[Dominique Luna]

Thanks Rafal. Nice little trick there. Why is it that tf.shape and foo.get_shape return different types, Tensor and Dimension respectively? My intuition would suggest they should give me back the same thing.

--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@tensorflow.org.
To post to this group, send email to dis...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/adcf6867-cd10-4f9b-b34c-80faf6248288%40tensorflow.org.

Best,

Dom

[Rafał Józefowicz]

In many applications (such as yours), the exact shape of the tensor cannot be determined during graph creation. We try to infer as much as possible based on the known information from the ops and how they interact with each other. The shape information that we were able to retrieve during graph creation is available through foo.get_shape(). From there you can most of the time get the rank of the tensor (size of the list) and the value for most of the dimensions (excluding those depending on the batch_size in your case).

If your sizes are statically defined, all the dimensions should generally be known. This is very useful for debugging and quickly hacking some code.

tf.shape always gets the full shape because it's an op that depends on the tensor and all of its dependencies but the result of tf.shape() is only available during the graph execution (session.run(.)). This one should be used in your case, when some of the dimensions are defined dynamically based on the inputs/placeholders.

When you create a new op that you want to share with others it is recommended no to rely on the .get_shape() as much as possible so that people can reuse your ops even when their tensors are not fully defined during graph creation.

Hope that helps,

Rafal

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.

To post to this group, send email to dis...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/adcf6867-cd10-4f9b-b34c-80faf6248288%40tensorflow.org.

Best,

Dom

[Steven Hutt]

It seems that tf.tile somehow loses shape information. If I do c = tf.tile(a, tf.pack([n,1])) where n = tf.shape(b)[0] and a is a variable of shape (1,5), then c.get_shape()[1] is ? rather than 5. 

[Martin Wicke]

If you can come up with a small self-contained test case that fails, you should file an issue on github. It is possible that our shape inference isn't strong enough in this case, but I believe it should be (I may not be seeing something important, I haven't dug into the code).

--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.
To post to this group, send email to dis...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/3e1fce10-2bab-4cfd-b2d4-e55b1d5f11a3%40tensorflow.org.

[Steven Hutt]

Hi Martin,

Thanks for your reply. Here's a self-contained example:

a  = tf.Variable(np.random.uniform(size=(1,5)))

b = tf.placeholder(shape=[None, 12], dtype=tf.float32)

batch_size = tf.shape(b)[0]

c = tf.tile(a, tf.pack([batch_size, 1]))

followed by

print a.get_shape()

print b.get_shape()

print c.get_shape()

returns 

(1, 5)
(?, 12)
(?, ?)

where I would expect c.get_shape() to return (?,5). 

If you think this is a genuine error I will post an issue on github. Thanks.

best,

Steve

On Sun, Aug 7, 2016 at 7:54 AM, Martin Wicke <wi...@google.com> wrote:

If you can come up with a small self-contained test case that fails, you should file an issue on github. It is possible that our shape inference isn't strong enough in this case, but I believe it should be (I may not be seeing something important, I haven't dug into the code).

On Sat, Aug 6, 2016 at 9:32 AM Steven Hutt <steven...@gmail.com> wrote:

It seems that tf.tile somehow loses shape information. If I do c = tf.tile(a, tf.pack([n,1])) where n = tf.shape(b)[0] and a is a variable of shape (1,5), then c.get_shape()[1] is ? rather than 5. 

On Tuesday, November 24, 2015 at 4:29:48 AM UTC, Dom Luna wrote:

I'm trying to deal with the situation where my computation depends on the shape of the input.

For example:

n = foo.get_shape()[0] # n is used later

The problem is n is always Dimension(None).

TypeError: Expected int32, got Dimension(None) instead.

Is there a way to get around this?

Also, because this would in an indirect way solve the problem. Should I use tf.tile instead of a loop?

In numpy loops are slow so tiling is preferred but it might be different in Tensorflow.

thanks

--
You received this message because you are subscribed to the Google Groups "Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@tensorflow.org.

[Martin Wicke]

Ah yes. Constant propagation isn't that good. To TensorFlow, the tensor that you created via pack is just a tensor, it doesn't know that one of its components came from a constant. This is not something we can easily fix.

You can always add back such missing information with set_shape.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.

[Steven Hutt]

So is it that pack loses the knowledge that '1' is a constant and so c.get_shape()[1] gives 5 x ? = ? instead of 5 x 1 = 5. 

Unfortunately c.set_shape(batch_size, 5) throws an error:

int() argument must be a string or a number, not 'Tensor'

since batch_size is unknown at compile time. 

Anyway, I think I understand the problem now, which is a step forward. Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@tensorflow.org.

[Yaroslav Bulatov]

There was a similar problem with `tf.fill` (https://github.com/tensorflow/tensorflow/issues/3102) which @mrry fixed, so perhaps this is worth filing to get it on his radar.

BTW, you could do to do c.set_shape([None, 5])

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/CAF6DV8spTTPBBCExW2Ku7tZnZv7zsP9be938zZfRd%3Don-v%3DpRg%40mail.gmail.com.

[Derek Murray]

I posted an answer on Stack Overflow:

http://stackoverflow.com/a/38820276/3574081

...and the related GitHub issue is:

https://github.com/tensorflow/tensorflow/issues/3692

Derek.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/CALW49E7r%3DDL3noOn%3DDzsN1NrhgmPu5wR2t9QXo-MCh4KAZG4fQ%40mail.gmail.com.

[Steven Hutt]

With c.set_shape([None,5]) the code works as intended. 

Thanks for the quick replies all round!