tf.keras unable to handle sparse inputs

[Cristian Garcia]

This basic piece of code:

x = tf.keras.Input(shape=(32,), sparse=True)
y = tf.keras.layers.Dense(1, activation='sigmoid')(x)

model = tf.keras.Model(x, y)

yield

ValueError: The last dimension of the inputs to `Dense` should be defined. Found `None`.

This issue has been open over a year and was not addressed in the recent 2.1.0 version. I think it deserves some attention as it blocks all applications that use large matrices. 

[Mark Daoust]

I think I have a fix, I'm sending it to review right now.

In the mean time, a workaround is to set the `batch_size` argument to `keras.Input`. It's failing here because the code doesn't handle partially-known shapes.

Caution: Applying a `Dense` layer to a `SparseTensor` still returns a dense-tensor.

[Rishab Pal]

>>> x = tf.keras.Input(shape=(32,), sparse=True, batch_size=1)

>>> y = tf.keras.layers.Dense(1, activation='sigmoid')(x)

I tried to execute the above line but it's still giving conversion error from SparseTensor to Tensor.

TypeError: Failed to convert object of type <class 'tensorflow.python.framework.sparse_tensor.SparseTensor'> to Tensor. Contents: SparseTensor(indices=Tensor("input_7/indices:0", shape=(?, 2), dtype=int64), values=Tensor("input_7/values:0", shape=(?,), dtype=float32), dense_shape=Tensor("input_7/shape:0", shape=(2,), dtype=int64)). Consider casting elements to a supported type.

--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/2b6832f9-d8dd-48b7-a3b6-b005eea1dd57%40tensorflow.org.

--

Regards

Rishab Pal

[Mark Daoust]

What version of tensorflow are you using? This colab clearly shows what does/doesn't work.

https://colab.research.google.com/drive/1Tcxc-zisgVF3cuvArVJJO5xkLwNgjRDp?authuser=1#scrollTo=YwLQdVLmD_ke

[Cristian Garcia]

Hey Mark, 

I can't access your colab notebook, maybe due to permissions.

[Mark Daoust]

Hmmm...  It's set to "anyone with a link can view".
It's just demonstrating, what we already know: it works in 2.1 with batch_size set, and breaks with that error otherwise.

https://colab.research.google.com/drive/1Tcxc-zisgVF3cuvArVJJO5xkLwNgjRDp

%tensorflow_version 2.x

import tensorflow as tf

print(tf.__version__)

x = tf.keras.Input(shape=(3,), batch_size=1, sparse=True)

d = tf.keras.layers.Dense(2)

y = d(x)

model = tf.keras.Model(x, y)

st = tf.SparseTensor(

    indices = [

               (0,0),

               (0,1),

               (0,2),

               (5,0),

               (5,1),

               (5,2)],

    values = [1,1,1,1,1,1],

    dense_shape = (6,3)

)

model(st)

x = tf.keras.Input(shape=(3,), sparse=True)

d = tf.keras.layers.Dense(2)

y = d(x)

model = tf.keras.Model(x, y)

model(st)

[Rishab Pal]

Thanks, I'll check it out. I was using tf 1.14.0 which was causing the error.

[Mark Daoust]

The fix ended up being more complex than I first thought and got tripped up by some integration testing. 

But I think I've got a fully-working solution, and hopefully it'll be through review soon.

[Cristian Garcia]

Thanks a lot Mark! This is very helpful.

[Mark Daoust]

This is fixed, should be in tf-nightly soon.

https://github.com/tensorflow/tensorflow/commit/74195b50fb5e1f22eb95ffd1646b4b0ceca2ea9b