RNN

[Federico Simonetta]

Hello,

I'm quite new to Julia and I'm trying to use Knet for a complex network. However, I'm not fully understanding how to use the `RNN` object (the documentation is a little obscure, maybe?).

I have sequences of `L` items, each one represented by `K` features. Each sequence is then represented by a matrix `K x L`. We can add `B` batches by adding one dimension: `K x B x L`.

Thus, we have a 3D matrix.

How should the RNN object be instantiated?

I'm trying something like:

K = 10
B = 120
L = 300

data = [(rand(K, L), Int.(round.(rand(L)*L))) for i in 1:B]

model = RNN(K, 1)

progress!(
    converge(
        Knet.minimize(
            (x, y) -> loss(softmax(model(x)), y),
            ncycle(data, 5)
        )
    )
)

But it fails with error:

AssertionError: vec(value(x)) isa WTYPE

What's wrong?

P.S. Can I join the slack group?

[Deniz Yuret]

There is some dimension mismatch I am afraid:

The RNN constructor takes RNN(X,H) where X is input size, H is hidden size.

The RNN object takes an input of dimension (X,B,T) where B is batch size and T is sequence  length.

The RNN object outputs an array of dimension (H,B,T).

The documentation `@doc RNN` and examples in the tutorial: rnn, imdb, charlm, may be helpful.

Of course you can join the slack group.

--
You received this message because you are subscribed to the Google Groups "knet-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knet-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knet-users/e92626dd-1f8b-49c3-9e13-e608991cc18a%40googlegroups.com.

[Federico Simonetta]

Sorry, it is still not working for me:

using LinearAlgebra, Knet, Statistics
using Base.Iterators
using IterTools: ncycle

K = 10

B = 120
L = 300

data = [(rand(K, B, L), rand(K, B, L)) for i in 1:400]

model = RNN(K, K)


progress!(
    converge(
        Knet.minimize(
            (x, y) -> mean(abs(model(x) - y)),

            ncycle(data, 5)
        )
    )
)

P.S.

To join the slack I need an invitation!

[Deniz Yuret]

Hi Federico,

Julia is pretty picky about types. In particular the array/element types used in your model and your data must match. RNN prefers the KnetArray{Float32} type (if gpu is available), or Array{Float32} (if gpu not available) as most deep learning work is done using low precision on a gpu.  rand() on the other hand defaults to Array{Float64}. The following should fix the mismatch:

model = RNN(K, K, usegpu=false, dataType=Float64)

However note that RNNs are really slow on the CPU, so a better fix would be:

data = [(KnetArray(rand(Float32, K, B, L)), KnetArray(rand(Float32, K, B, L))) for i in 1:400]          

Finally element-wise / broadcasting operations require a dot in the Julia 1.0 syntax, i.e. after abs:

(x, y) -> mean(abs.(model(x) - y))       

This is motivated by the fact that sqrt(x) and sqrt.(x) may mean different things.

About slack: I am not sure how to add/invite people, so if you figure it out let me know what to do. (You may need a slack id and/or be a member of the Julia slack first?)

best,

deniz

                                                               

                                                             

--

You received this message because you are subscribed to the Google Groups "knet-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knet-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/knet-users/1b501fae-c927-4370-96aa-e1520922d1e6%40googlegroups.com.