Super simple LSTM prediction example?

[blach...@kujakuja.com]

Hi all,

This has been discussed in another thread... is there a super simple LSTM time-series prediction example anywhere? I am super new to tf and am having a hard time shaping data and trying my best to translate Keras examples I am finding online.

Any help is appreciated!

[Shanqing Cai]

It's probably not to hard putting an example together. Someone on this discussion board can probably do it relatively easily on CodePen. But just to make sure we understand your need: LSTM (an RNNs) in general, can perform different kinds of tasks, including

1. Classifying sequences (example: whether a particular sequence fits a particular "grammar", see Reber grammar)

2. Predicting a number from a sequence (example: assigning a particular sequence a number, like the tensorflow.js sentiment example)

3. Predicting the next item in a sequence.

Which particular one are you especially interested in?

[blach...@kujakuja.com]

Hello Shanqing! Number 3!

[Atanas Stoyanov]

There is the addition RNN sample : https://github.com/tensorflow/tfjs-examples/tree/master/addition-rnn

On Thursday, May 31, 2018 at 3:54:52 PM UTC-4, blach...@kujakuja.com wrote:

Hello Shanqing! Number 3!

[Shanqing Cai]

Sorry for the delay. But I wrote a minimalist LSTM demo at:

https://codepen.io/caisq/pen/vrxOvy

It trains an LSTM to perform a rather simplistic and contrived sequence prediction task:

given a sequence of ten 0-1 items, predict whether there is any stretch of consecutive identical items >= 4 items long.

The code is about 120 lines (with extensive comments) and is much simpler compared to the AdditionRNN example. 

Let me know whether that satisfies your need.

Shanqing

[Tiago Vasconcelos]

Hi, sorry for hijacking the thread, been looking at you Pen and it's great. I'd like to ask something about it though!!

For the sequence [0, 1, 0, 1, 0, 1, 0, 0, 1, 0], you're feeding the LSTM 2 values at a time? I got confused with the one hot encoding!! Each row in the tensor is the one hot encoding of the sequence, is that it? 

also you're passing in 500 samples to the model, i read somewhere that LSTM doesn't like more than 300/400 values, is this true? Also i got about 0.00002 loss with your example, but i can't seem to go bellow 0.05 with a timeseries i'm working on, my guess is my data preparation is screwed.

I'm trying a timeseries classification but i can't seem to get it to work correctly. It trains and all the loss goes about 0.05, accuracy between 0.8/0.9 but then predictions seem all wrong!! Again, i guess my data prep is not right.

[Atanas Stoyanov]

Shanqiq, great sample, but its not working in Safari. Possibly related to https://github.com/tensorflow/tfjs/issues/287

[Shanqing Cai]

On Monday, June 11, 2018 at 12:00:44 PM UTC-4, Tiago Vasconcelos wrote:

Hi, sorry for hijacking the thread, been looking at you Pen and it's great. I'd like to ask something about it though!!

For the sequence [0, 1, 0, 1, 0, 1, 0, 0, 1, 0], you're feeding the LSTM 2 values at a time? I got confused with the one hot encoding!! Each row in the tensor is the one hot encoding of the sequence, is that it? 

The values are fed one at a time. The reason why the input sequence tensor has 2 as its last dimension is that we are doing one-hot encoding of the 0/1 values.

To give a simple example, let's ignore the batch dimension (first dimension). Suppose the sequence has a length of 3 and is [0, 1, 0], it will be encoded as a Tensor of

shape [3, 2], namely:

[[1, 0], [0, 1], [1, 0]]

 

also you're passing in 500 samples to the model, i read somewhere that LSTM doesn't like more than 300/400 values, is this true? Also i got about 0.00002 loss with your example, but i can't seem to go bellow 0.05 with a timeseries i'm working on, my guess is my data preparation is screwed.

The number 500 is just the total number of training examples we give to model.fit(). It'll be split 85-15 for training and validation (see the validationSplit parameter).In this particular toy example, you have a sequence of 10 binary items, so the total number of possible sequence is 1024. 500 is a rough choice to cover half of all the possibilities. You could also try something like 1000. But in that case, you'll end up with more duplicate sequences (because the sequences are randomly generated and bound to have some collisions). To be strict, the sequences should be generated another way to avoid duplication.

I'm not aware of the 300/400 size threshold you mentioned.

 

I'm trying a timeseries classification but i can't seem to get it to work correctly. It trains and all the loss goes about 0.05, accuracy between 0.8/0.9 but then predictions seem all wrong!! Again, i guess my data prep is not right.

yeah, there could be some problem in the way you encode your binary sequences. See my answer about the one-hot encoding above. 

[Tiago Vasconcelos]

My problem is i'm not doing one hot enconding... I'm trying to do crypto classification. Based on input data, mainly indicators, predict a potential move up ou not, Binary Classification!! So i'm feeding the model 4 indicators and 6 timesteps! i have a tensor with shape [6, 4] - 6 timesteps with 4 dimensions, is this right? 

it looks somthing like this : 

[ [ [a, b, c, d], t -5

    [b, a, d, e], t -4

    [a, b, e, f], t -3

    [a, b, c, d], t -2

    [b, a, d, e], t -1

    [a, b, e, f] ], t

    ...

  [ [a, b, d, f],

    [...] ] ]

10k of this... (shape [10000, 6, 4]) so i tried the batch aproach from the mnist example and broke it into batches to feed the model inside a for loop. What i want the model to pick up is, based on the inputs, the factors that lead to a stock/crypto make a move up and predict that move. Maybe i'd be better off doing price prediction and make the classification "by hand" based on the output!! 

I've tried feeding a conv1d with this data but the results are not much different from the lstm... although it's faster to train!!

[ecrtuere tanium]

Hello Shanqing Cai,

Would you mind adding a call to model.predict( #user_input ) to your example please?

Or if you hard code it, use this sequence sequence as an example : [0, 1, 0, 1, 0, 1, 0, 0, 1, 0]

Thank you in advance