Still the same output when predicting sequences using LSTM

[hny...@gmail.com]

Hello,

I try to use LSTM for predicting sequences on toy example below. 

### DATA PREPARATION ###

import pandas as pd

import numpy as np

a = np.array(

    (np.array([0]*10),

    np.array([1]*10),

    np.array([2]*10),

    np.array([1]*10))

    )

data = pd.DataFrame(np.vstack([a]*10000))

def load_data(data, n_prev = 20):

    docX = []

    docY = []

    for i in range(len(data)-n_prev):

        docX.append(data.iloc[i:i+n_prev].as_matrix())

        docY.append(data.iloc[i+n_prev].as_matrix())

    alsX = np.array(docX)

    alsY = np.array(docY)

    

    return(alsX, alsY)

def train_text_split(df, test_size=0.1):

    ntrn = round(len(df) * (1 - test_size))

    

    X_train, y_train = load_data(df.iloc[0:ntrn])

    X_test, y_test = load_data(df.iloc[ntrn:])

    

    return((X_train, y_train), (X_test, y_test))

(X_train, y_train), (X_test, y_test) = train_text_split(data)

print(X_train.shape) # Out: (35980, 20, 10)

### END OF DATA PREPARATION ###

### BUILDING MODEL ###
from keras.models import Sequential

from keras.layers.core import Dense, Dropout, Activation, TimeDistributedDense

from keras.layers.embeddings import Embedding

from keras.layers.recurrent import LSTM

model = Sequential()

model.add(LSTM(10, 200, return_sequences=False))

model.add(Dense(200, 10))

model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

model.fit(X_train, y_train, batch_size=20, nb_epoch=5)

As you can see, it's ridiculously simple case, but the predict results are horrible. It still returns the same pattern (vector of ~0.1):

model.predict(X_test)[:5]

# Out: 
array([[ 0.1001857 ,  0.09980822,  0.10030529,  0.09990708,  0.09957968,
         0.10046917,  0.10008783,  0.09971908,  0.10020511,  0.09973283],
       [ 0.10004262,  0.09994944,  0.09996234,  0.09996925,  0.10004126,
         0.09985186,  0.09995659,  0.10003508,  0.10005915,  0.10013235],
       [ 0.10004639,  0.09995028,  0.09995895,  0.09996869,  0.10004199,
         0.09984995,  0.09995785,  0.10003597,  0.10005963,  0.10013032],
       [ 0.10004825,  0.09995182,  0.09995707,  0.09996101,  0.10004185,
         0.0998318 ,  0.0999568 ,  0.10003491,  0.10006752,  0.10014894],
       [ 0.1001857 ,  0.09980822,  0.10030529,  0.09990708,  0.09957968,
         0.10046917,  0.10008783,  0.09971908,  0.10020511,  0.09973283]])

Using more layers doesnt work at all.

Isn't a problem that the NN should besides the predicting output also take the input and process it in forward mode? How to emulate this in Keras? I've found this reference, but without any difference.

Thank you

[François Chollet]

The network looks reasonable, so I assume your problem lies in the data. You does your loss history looks like (training and validation)? Does it decrease over time? 

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/b1de0236-5338-4e0a-a808-ddde6d071ca4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[hny...@gmail.com]

The problem was that I want to do regression, not classification problem. After changing it to corresponding values, I solved it. More about it is on my blog, if anyone interested.

Dne středa 1. července 2015 19:01:58 UTC+2 François Chollet napsal(a):