Sparse Variable support

[Sufeng Niu]

Hello,

1. In my model, I need to use sparse variable as trainable weight. It seems that current tensorflow doesn't support sparse variable well. I haven't found any examples online for the sparse variable, I filed a stackoverflow here (https://stackoverflow.com/questions/41156460/tensorflow-doing-gradients-on-sparse-variable). For some people's answer, I need to use tf.IndexedSlices and tf.gather, from the API document, I am losing idea how to plug these APIs into sparse variables. it would be highly appreciate if any one can provide an example

2. Also, is that possible to initialize a variable with tf.tensor. Because tensorflow currently don't have SparseVariable options, I need to manually create a Variable which has many zero elements. However, I found that if I use: w = tf.Variable(a) # a is tf.tensor converted from tf.SparseTensor, it doesn't support it. My point is if I can do a sparse updates by gradients for the first question, how can I create a sparse variable?

Thank you!

S

[Martin Wicke]

There's no sparse variable in TensorFlow. We have a SparseTensor Python utility class, which is basically a i,j,v encoded sparse tensor, backed by three dense tensors (or variables): one storing index tuples, one storing the values at those index tuples, and one storing the shape. This representation should be familiar to people who have worked with sparse matrices.

There are a number of operations defined for sparse tensors, but support is not universal. To do custom things, you may need to use gather and scatter sometimes, but you shouldn't normally have to work directly with IndexedSlices, unless you're defining new ops and their gradients.

Martin

--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@tensorflow.org.
To post to this group, send email to dis...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/c646cac4-d4ca-4918-b4a6-6ecf1ceb9788%40tensorflow.org.

[Sufeng Niu]

Hi Martin,

Thank you. I know the SparseTensor is available, but in my applications, my weight is a sparse matrix which need to do sparse update. This is common for people who perform weight pruning and graph applications. I saw someone also encountered similar problem (ex: http://stackoverflow.com/questions/35803425/update-only-part-of-the-word-embedding-matrix-in-tensorflow). but doesn't have an example code. I found that the tensorflow API doc (https://www.tensorflow.org/api_docs/python/state_ops/sparse_variable_updates) doesn't provide any examples show where to plug these API for gradients update.

Thank you

--

Best Regards,

Sufeng Niu
Department of ECE, Clemson University