LSTM layers suitable for multivariate or only univariate?

[shaifa...@gmail.com]

Hello

Can we use LSTM model of Keras with multidimensional time series or is it used only for univariate time series

[dsad...@gmail.com]

I have also been struggling with multidimensional time series using LSTM's in keras. Specifically, the input_batch_size parameter and invert the normalization properly.

Jason Brownlee's excellent blog post (http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/) on univariate TS is a good start. He also points out in one of the comments that it is possible (go to the link and find the phrase "multivariate"), however my puny mind struggles to understand how.

Did you have any luck with LSTMs for Multivariate TS??

[Deepak Sadulla]

Hi Yosr,

You could try following the steps in the following link on machinelearningmastery.com.

http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

I have added two more columns (with random values) to the dataset hence read_csv has usecols = [1,2,3] to turn it into a toy multivariate dataset.

This link describes the use of univariate time series but can be extended to the multivariate scenario by tweaking the code as below:

You might run into issue based on the versions of tensorflow and keras installed.

# convert an array of values into a dataset matrix

def create_dataset(dataset, look_back=1):

dataX, dataY = [], []

for i in range(len(dataset)-look_back-1):

a = dataset[i:(i+look_back), :]

dataX.append(a)

dataY.append(dataset[i + look_back, 0])

return numpy.array(dataX), numpy.array(dataY)

# fix random seed for reproducibility

numpy.random.seed(7)

# load the dataset

dataframe = read_csv('international-airline-passengers.csv', usecols=[1,2,3], engine='python', skipfooter=3)

dataset = dataframe.values

dataset = dataset.astype('float32')

print(dataset.shape)

# normalize the dataset

scaler = MinMaxScaler(feature_range=(0, 1))

dataset = scaler.fit_transform(dataset)

# split into train and test sets

train_size = int(len(dataset) * 0.67)

test_size = len(dataset) - train_size

train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]

# reshape into X=t and Y=t+1

look_back = 2

trainX, trainY = create_dataset(train, look_back)

testX, testY = create_dataset(test, look_back)

print("Shape of trainX")

print(trainX.shape)

print("Shape of trainY")

print(trainY.shape)

# reshape input to be [samples, time steps, features]

trainX = numpy.reshape(trainX, (trainX.shape[0], look_back, trainX.shape[2]))

testX = numpy.reshape(testX, (testX.shape[0], look_back, testX.shape[2]))

# create and fit the LSTM network

model = Sequential()

model.add(LSTM(4, input_shape=(look_back,trainX.shape[2])))

model.add(Dense(1))

model.compile(loss='mean_squared_error', optimizer='adam')

model.fit(trainX, trainY, epochs=100, batch_size=1, verbose=2)

# make predictions

trainPredict = model.predict(trainX)

testPredict = model.predict(testX)

I have not corrected the code post this point, please invert the minmaxscaler and change how the plotting has to occur.

# invert predictions

trainPredict = scaler.inverse_transform(trainPredict)

trainY = scaler.inverse_transform([trainY])

testPredict = scaler.inverse_transform(testPredict)

testY = scaler.inverse_transform([testY])

# calculate root mean squared error

trainScore = math.sqrt(mean_squared_error(trainY[0], trainPredict[:,0]))

print('Train Score: %.2f RMSE' % (trainScore))

testScore = math.sqrt(mean_squared_error(testY[0], testPredict[:,0]))

print('Test Score: %.2f RMSE' % (testScore))

# shift train predictions for plotting

trainPredictPlot = numpy.empty_like(dataset)

trainPredictPlot[:, :] = numpy.nan

trainPredictPlot[look_back:len(trainPredict)+look_back, :] = trainPredict

# shift test predictions for plotting

testPredictPlot = numpy.empty_like(dataset)

testPredictPlot[:, :] = numpy.nan

testPredictPlot[len(trainPredict)+(look_back*2)+1:len(dataset)-1, :] = testPredict

# plot baseline and predictions

plt.plot(scaler.inverse_transform(dataset))

plt.plot(trainPredictPlot)

plt.plot(testPredictPlot)

plt.show()

I'm fairly new to this and I am not familiar with sklearn and matplotlib, and hence have not touched those part. Would recommend going through all the blogs in the website (machinelearningmastery), it does seem to be a good starting point for LSTM and time forecasting.

HTH.

Deepak

On Tue, Jul 18, 2017 at 7:55 PM Yosr Mzoughi <mzough...@gmail.com> wrote:

Hello ,

I am struggling to find a solution too. I know that it is possible, yet can't find the right documentation to learn how. Have you been luckier than me ?

[caotia...@gmail.com]

Hi, I am also confused about this. It seems the documentation of Keras indicates our input should be 3D Tensor, i.e. (batches, time step, input_shape/features). But if we use an multidimensional time series, the dimension would exceed 3.

I've read through Jason's blog and comments, but I still don't know how to implement his ideas(http://machinelearningmastery.com/time-series-forecasting-supervised-learning/)...Actually now I am thinking about using a conv layer ahead to decrease the dimension. And I am not sure if the convLSTM layer would meet your needs(https://arxiv.org/abs/1506.04214). This has already been implemented in Keras.

Have you got any new ideas?

在 2017年4月3日星期一 UTC-7上午7:24:18，dsad...@gmail.com写道：

[Matias Valdenegro]

No, you are mistaken. the 3D input tensor is not the same as the dimensionality of the features (what you call multidimensional time series). You just need to ser the proper input shape, where the third dimension is what you want.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users+unsubscribe@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/fddf8261-e77f-4be5-9b03-1bc08ecb6da9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[caotia...@gmail.com]

Oh, I got it. You are right. If the input data is still a vector, we just need to use the length of it to specify the argument. Actually I thought you were talking about, say, an input with multiple rows and columns at each time step. 

在 2017年7月26日星期三 UTC-7下午1:42:47，Matias Valdenegro写道：

No, you are mistaken. the 3D input tensor is not the same as the dimensionality of the features (what you call multidimensional time series). You just need to ser the proper input shape, where the third dimension is what you want.

On 26 July 2017 at 20:44, <caotia...@gmail.com> wrote:

Hi, I am also confused about this. It seems the documentation of Keras indicates our input should be 3D Tensor, i.e. (batches, time step, input_shape/features). But if we use an multidimensional time series, the dimension would exceed 3.

I've read through Jason's blog and comments, but I still don't know how to implement his ideas(http://machinelearningmastery.com/time-series-forecasting-supervised-learning/)...Actually now I am thinking about using a conv layer ahead to decrease the dimension. And I am not sure if the convLSTM layer would meet your needs(https://arxiv.org/abs/1506.04214). This has already been implemented in Keras.

Have you got any new ideas?

在 2017年4月3日星期一 UTC-7上午7:24:18，dsad...@gmail.com写道：

I have also been struggling with multidimensional time series using LSTM's in keras. Specifically, the input_batch_size parameter and invert the normalization properly.

Jason Brownlee's excellent blog post (http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/) on univariate TS is a good start. He also points out in one of the comments that it is possible (go to the link and find the phrase "multivariate"), however my puny mind struggles to understand how.

Did you have any luck with LSTMs for Multivariate TS??

On Friday, March 10, 2017 at 4:33:24 PM UTC+5:30, shaifa...@gmail.com wrote:

Hello

Can we use LSTM model of Keras with multidimensional time series or is it used only for univariate time series

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.