How to smooth time series prediction, especially the end point

[TomH488]

I have a time series prediction that has noise.

The signal is a very good prediction but I need to get the noise out.

The problem is that I need to get a filtered value for the last point
in the prediction series.

This means I cannot use simple averaging methods and even Fourier
Series since the prediction is of fixed length and aperiodic.

I'm thinking the best way is to do a PCA (principle component
analysis) and use the first n components to achieve a Low Pass
filter. What would be nice about PCA is that I would actually be
using orthogonal terms of the series. I'm also guessing that PCA
should have no end point problems. But to do PCA, I'd have to get the
formulas and write the code.

Of course, something easier would be welcome.

Thanks in advance,
Tom

[Greg Heath]

Aren't you using a commercial NNet package?

[TomH488]

Yes, NeuroShell 2.

Since its forecasting, the incrementing backtest method is:

Train net
Process inputs and get prediction,
Increment inputs
Then every 5th increment, Retrain, otherwise, run inputs through
previously trained net.

For a backtest spanning a couple years, you get a composite curve of
many trainings

There are issues of local minima and each "training" actually consists
of taking the average prediction from a number of nets trained with
varying initial weights.

If I take this predicted curve and filter it using a Fourier Series,
it does very well. The problem is that the backtest curve is a series
of EndPoints.

I could do something like fit a polymonial or a spline (assuming an
elastic beam is smoothly fit to minimum error) but it would be nice to
use actual components of the curve itself.

I realize PCA is not really an option, I don't know what I was
thinking - it is good for reducing the number if inputs and making
them orthogonal, although PCA is closely related to the DFT/FFT.

I did stumble onto Blind Signal Separation using PCA and then
references to ICA, Independent Component Analysis of signals. But
these seem related to the "cocktail party problem" where you try to
separate individual speech which is spatially located using multiple
microphones. Again not quite what I had in mind.

One question I might ask is the Discrete Fourier Transform the only
way to break a finite length data vector into orthogonal components?

I thought of tricks to use with the DFT:

Trick 1

Using all data points:
1) Smooth data using appropriate 1st n Fourier coefficients,
2) Omit the 1st data point and smooth again,
3) Omit the 1st and 2nd data point and smooth again,
...
n) Omit the first n data points and smooth again.

Then take the average of the end point.

Trick 2

Given N data points,
choose a Lambda (period) that is much smaller, say N/5
then calculate the coefficients.

In this case, instead of sweeping from 0 to 360 deg,
I sweep from 0 to N/5 * 360 deg with a few extra "modulo" data points.
This "dilutes" the modulo points which are evaluated once while the
other points are "strengthened" 5 times

But by some method, I need to smooth each predicted point as I add it
to the backtest prediction series.

Hopefully I didn't make this too confusing.

Thanks
Greg

Tom

[TomH488]

Still no solution.

Have tried:

1) more Fourier tricks that are not satisfactory
2) Simplistic Kalman (constants, no matrices) has lag and does no
better that a Simple Moving Average
3) Tukey Median - not sold on this method.

Looking into:

1) 4th Order Butterworth Digital Filter (need to find coefficient
online calculator or code)

Other to do:

1) Fourier Transform, Discrete Time Fourier Transform, Convolution
2) DIY harmonic cross-correlation: find dominant harmonics via FFT,
set phase or each harmonic by corr coef, then weighted sum based on
corr coef. Result should be end point independent and only true
content of data.

QUESTIONS:

1) Would the Discrete Time Fourier Transform provide what I want since
it is aperiodic? My problem is that every example I see of aperiodic
draws down to zero on the ends. I cannot handle that constraint (well
maybe if I tilted my data which in this case might not be so bad an
idea)

2) While PCA is meant to be used to reduce order of multiple inputs,
could it be used to reduce the "order" of curve "harmonics?"

This is my last major hurdle in the project and it is really killing
me that I've spend already about 3 weeks to no avail.

Thanks in advance,
Tom

[Tomasso]

"TomH488" <tom...@gmail.com> wrote in message news:99360401-cc70-4f34...@k4g2000yqn.googlegroups.com...

> Still no solution.
>
> Have tried:
>
> 1) more Fourier tricks that are not satisfactory
> 2) Simplistic Kalman (constants, no matrices) has lag and does no
> better that a Simple Moving Average
> 3) Tukey Median - not sold on this method.

What is your argument here?

It's important to characterise what's going on with your time series.

Eg, Gaussian noise, impulsive noise and other rare events, cross-channel interference, convoluted channel, stochastic
signal components, non-stationarity, ...

If you investigate that, you have the options of various forms of noise filtering, deconvolving, FT or DFT for frequency
filtering, independent component analysis (to manage cross-channel).

OTOH, your TS may be 95%+ noise of unknown characteristics, in which case, you may be stuff...

T.

PS: If your objective is trading for profit, then predictive accuracy isn't the objective. It's about optimising your
detection of trading signals.

[TomH488]

My conclusion is that the lower the cutoff frequency the more the lag
of a LP filter.

Response to a Unit Step will lag. The only way it could not lag is to
be predictive.

So that leaves the Kalman Filter which uses all available information
which is an inaccurate prediction of the next point and an inaccurate
measurement of the next point.

I don't have the math expertise to do any of this in a expeditious
manner but a Kalman filter utilizing the weight matrix of the trained
neural network (for prediction) would be quite effective.

I believe there is no available solution for me using filtering.

What is irritating is that my nnet is predicting quite well except it
is adding a lot of noise to the prediction.

So what kind of things can I do to get the net to predict with less
noise? My first thought is to predict more than a single value.
Perhaps a 3 point forecast where you take the average of the 3 points
to get the final answer.

Oh, one thing regarding Kalman, they say it does well with rather poor
models. So one could use the naive model which is basically the
constant value model which is worked out clearly on the web. It lags
like a simple moving average which is not surprising.

However, if the prediction is a 10 day deta of a 10 day simple moving
average, it turns out that the forecast flips sign about every 10
points on average, so the naive solution would be: the current
forecast increment would be equal to the previous - the derivative is
constant. Hummm.... That would be velocity! The popular example on
the web of Kalman is the storage tank for 3 cases: 1) constant, 2)
linear filling, and 3) sinus sloshing. I should try the linear
filling with my model...

Actually there appears to be a fair amount of stuff on the web about
using neural networks and Kalmans together. It seems like a good
marriage to me.

Thanks
Tom