Poorly Predicted Cases are Easily Fit When in Training Set

[TomH488]

DESCRIPTION:

Simple MLP,
Time Series Prediction,
1 hidden layer,
178 inputs (10 day moving average pre-processing),
1 output (10 day delta of the 10 day moving average),
3500 daily input cases,
5 daily output (predicted) cases.

Training/Stepping Method:

1) Train every 5 days,
2) predict next 5 days into future.
3) Do this 60 times for a single back test.
(This is composite output of 60 weeks or 300 days' prediction.

OBSERVATIONS:

1) Some areas predicted splendidly,
2) Some areas predicted horribly,
2.1) Some trends, hills, valleys predicted oppositely,
2.2) Some areas have significant DC offset (average of prediction greatly different than average of actuals)

HYPOTHESIS:

These areas of poor prediction are the result of inadequate inputs to characterize what seems to be special conditions never seen before.

Of True, then these areas should be difficult to train on.

HOWEVER, when training is advanced so these problem areas are in the training set, when the trained values are checked it is found that these "problem" areas are easily learned.

CONTRADICTION?

If there was enough input characterization to learn these areas, why were they so poorly predicted?

Any suggestions, insights, or ridicule (!) would be greatly appreciated? (Help Greg...)

Thanks in advance,
Tom

[Greg Heath]

On Sep 23, 9:10 am, TomH488 <tom...@gmail.com> wrote:
> DESCRIPTION:
>
> Simple MLP,
> Time Series Prediction,
> 1 hidden layer,

Please Clarify:

Dimensionality of input column vector I?
How many hidden nodes, H?
Dimensionality of output column vector O ?
MLP Node topology I-H-O ?
Number of input/output training pairs, Ntrn?
Dimension of input Training matrix [ I Ntrn ] ?
Dimension of output training matrix [ O Ntrn ] ?

> 178 inputs (10 day moving average pre-processing),
> 1 output (10 day delta of the 10 day moving average),
> 3500 daily input cases,
> 5 daily output (predicted) cases

Ntrn = 3500

Input Training data:

I = 178; x(i,t) = mean( z(i, t - 9 : t ), i = 1:178, t = 10: 3490

Output Training Target Data

O = 5, y(j,t) = ??? , j = 1:5, t = 10:3490

.
I'm lost.

Greg

[TomH488]

I'm looking for a qualitative answer.

If someone said, The further you predict into the future with a Nnet,
the better the prediction.
I would say, that is not true all the time and that, in general, the
further into the future the worse the result - just like trying to
predict the weather 2 days -v- 2 weeks from now.
Don't need to know anything about the network.

Keeping this in Discussion Over a Beer Land,
A simple 3-layer forecasting MLP, in particular, does a bad job
predicting the average price of the next 10 future days, YET,
when the training window is advanced such that those poor predictions
become training cases, they are quickly fit during training and are
NOT one of the many areas in the training data that are difficult to
fit.

That sounds like a contradiction or at least an insight into the error
surface.

If asked, I would speculate that that particular case that was poorly
predicted yet easily trained upon:
1) that point was not in a classification cluster, i.e., had no
neighbors, (poor prediction)
2) that point was not a flyer (easily trained)