SimpleRNN with timestep of size 1 - Does this make sense?

[Alberto Simoes]

Hi

I am doing my first steps on RNN, and I was unable to find a suitable graphical representation of the SimpleRNN cell. As far as I understand, I can give a window of events (timesteps), and the output from each timestep processing is sent to the next one.

What is not clear is if the output from the last RNN is sent back to the next training example.

Basically, want to understand if it makes sense to have something like

model = Sequential()
model.add(Input(shape=(1,)))
model.add(Embedding(input_dim=len(vocabulary), input_length=1, output_dim=100))
model.add(SimpleRNN(1))

and if this could be an acceptable approach to mimic an Elman Network

https://web.stanford.edu/~jurafsky/slp3/9.pdf (Fig 9.5)

Thanks

Alberto

[Alberto Simoes]

Looking to the number of parameters for 1 or 2 timesteps, it doesn't look like it is doing what I was expecting:

SimpleRNN looks a lot more like Figure 9.1 in https://web.stanford.edu/~jurafsky/slp3/9.pdf

Is there any "scientific" documentation on Keras Layers, and what weights they are really learning?

Thank you

Alberto

[Matias Valdenegro]

RNNs are standard neural network methods, any book on neural networks will cover RNNs and how they actually work, by sharing weights across timesteps, not across training samples.

[Lance Norskog]

This is a tutorial on using the Keras Simple RNN layer:

https://machinelearningmastery.com/understanding-simple-recurrent-neural-networks-in-keras/

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/3686077.07ICAnoMbH%40vsd-grey-wolf.

--

Lance Norskog
lance....@gmail.com
Redwood City, CA

[Alberto Simões]

Thank you, Lance

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/CAP0ykh8y5FRk3JW4qLSbjOFk8c%3Dcq7hoX_rKkHQvrOJPZ6tv-Q%40mail.gmail.com.

--

Alberto Simões

orcid: 0000-0001-6961-2660

Coordinator of Master's on Applied Artificial Intelligence

2Ai Lab - Applied Artificial Intelligence Laboratory, IPCA

[Lance Norskog]

That site has a lot of good short tutorials.

https://machinelearningmastery.com/

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/CAAkRyrsR687V%3Dbsrw_mzLgwjQZkCrdL7tV2eyEeuG3gKFcgWeA%40mail.gmail.com.