LSTM experiment

[andrew morgan]

Hi, I'm learning torch and experimenting with the building of lstm rnn models.

My experiment code, and data that feeds it, is here.

  https://github.com/bytesumo/LSTM-Experiments

The hypothesis is I can lean to predict a slowly changing binomial class called Trend, from a set of time-series features I can create over a data stream.

As the goal is to generate streamed rnn predictions, I've put together the experiment code with that in mind.

While my code certainly isn't going to win any awards, I finally got it to run, but I'm finding the LSTM predicts badly.

I'm sure I messed up something obvious, but seem to have hit the limit of my knowledge.

I wonder if anyone has any tips for how to proceed?

All ideas welcome.

thanks

Andrew

[Bálint Pál Tóth]

Dear Andrew,

I think dp_10_train_DJI_min.csv is truncated (line 102877 is not complete).

However in the first 102876 lines the unit_per_change variable's value look to small. The mean is about 0, but you should scale up to have about 1 for stdev (right now it is about 0.00045). 

Furthermore I think 

                if line.char_output == "U" then                           

                      feed_output[batchcounter] = torch.Tensor({2,1})    

                else   -- then must be "U"                                               

                      feed_output[batchcounter] = torch.Tensor({1,2})    

                end   

should be 

                if line.char_output == "U" then                           

                      feed_output[batchcounter] = torch.FloatTensor({1})    

                else   -- then must be "U"                                               

                      feed_output[batchcounter] = torch.FloatTensor({2})    

                end   

You may also uncomment the dropout layer, probably with higher (0.3-0.5) value.

Sorry that I didn't had time to test your network with these modifications. Maybe later in this week I will.

Best Regards,

Balint Pal Toth

[andrew morgan]

Hi Balint

Thanks for the code review.

I'll fix that data point in the file, and try out your changes.

thoughts:

- Good spotting on the float tensor.

- good idea about the need to scale that unit_per_change value. ( It represents the percent change in the stock market price from the last observation to the current one. )

- thanks for pointing out I can use one output field for the class U/D, good tip.

- I will also try dropout with your settings too

As an aside, that source data file is actually stock market data (1 minute DJI file, on a n=10 trend) pushed through an experimental feature and label generator based broadly on a fork of my other luajit project here: https://bitbucket.org/bytesumo/trendcalculus-public

I'll try the changes tomorrow and post updates. 

thanks again, 

Andrew

[Bálint Pál Tóth]

Dear Andrew,

As a first approach I would suggest to train the LSTMs with the original time series only (with up and down labels as the output), standardized to 0 mean & 1 stdev. 

If it works, then you can include additional features.

Can you upload such a database to github? 

Best Regards,

Balint Pal Toth

[andrew morgan]

Hi Balint,

I've slightly adjusted the code, and uploaded revised data files,  as suggested.

The "change" column now includes an input with better statistical properties, as suggested.

Now it represents the sigmoid of a scaled relative_%_change, which seems to do the job.

Note I also lengthened the timeframe I'm testing to a 30 day trend, as I notice the algorithm is predicting on a timeframe about 2/3 of the target.

This widens it out a bit, to make it clear.

Apart from LSTM, I'm testing with feed forward nets, gradient boosted trees, random forests too, so the gzipped file now includes other features I'm testing too.

In the code it's easy to comment out the undesired columns,to include/exclude features as any test demands.

Andrew

[Balint Toth]

Hi Andrew,

Did you manage to make LSTM work with your data?

If not I am happy (and interrested in) to take a look at your code and training data. 

Best Regards,

Balint TOTH

[andrew morgan]

Hi Balint, in short yes. I'm busy the next two days but will update and get back to you soon.
A