Recommendations for flexible data smoothing?
Anton81
often I have data points and I want a smooth curve through them. However
gnuplot csplines are too wiggly and bezier misses the points too much. Is
there a possibility to tune the smoothing?
Otherwise I might try to use an external library and program an data
processor. Any suggestions for a shape preserving variable parameter
smoothing method/library?
Anton
Hans-Bernhard Bröker
> often I have data points and I want a smooth curve through them. However
> gnuplot csplines are too wiggly and bezier misses the points too much.
Any particular reason you didn't mention acsplines?
James
A nice looking method is local regression (also known as loess and
lowess). It has user-set parameters defining how smooth or bumpy to
make the fit. It's described in Cleveland's book "Visualizing Data".
Here's a link to a postscript file describing the method...
http://www.stat.purdue.edu/~wsc/papers/localregression.principles.ps
I've never searched for existing source code implementing the method.
James
goo...@catking.net
wrote:
Indeed acsplines is available for "tuning" cspline type smoothing but
what such complex data processing actually does to your data seems
very obscure.
Fiddling with the weighting factors because csplines are fundementally
not suitable for the data set seems a poor solution.
Clearly applying data processing that one does not understand is bad
science.
Doesn't this request , once again, underline the need for some simple,
easily understood smoothing like a running mean?
best regards.
mer...@chauvet.bmsc.washington.edu
<goo...@catking.net> wrote:
>Doesn't this request , once again, underline the need for some simple,
>easily understood smoothing like a running mean?
Running mean demo:
http://gnuplot.sourceforge.net/demo_4.3/data_feedback.html
--
Ethan A Merritt
goo...@catking.net
> In article <5914de92-1223-4d7f-bb23-f71cf1322...@m45g2000hsb.googlegroups.com>,
>
> <goog...@catking.net> wrote:
> >Doesn't this request , once again, underline the need for some simple,
> >easily understood smoothing like a running mean?
>
> Running mean demo:
> http://gnuplot.sourceforge.net/demo_4.3/data_feedback.html
>
> --
> Ethan A Merritt
Thanks for filling in the blanks Ethan. I knew there was a demo but
could not find it to post the link.
BTW where is the data file that goes with the example? "silver.dat"
does not seem to be provided witht the example script.
TIA.
mer...@chauvet.bmsc.washington.edu
<goo...@catking.net> wrote:
>On Aug 16, 5:24=A0pm, merr...@chauvet.bmsc.washington.edu wrote:
>> In article <5914de92-1223-4d7f-bb23-f71cf1322...@m45g2000hsb.googlegroups=
>.com>,
>>
>> =A0<goog...@catking.net> wrote:
>> >Doesn't this request , once again, underline the need for some simple,
>> >easily understood smoothing like a running mean?
>>
>> Running mean demo:
>>
>> --
>> Ethan A Merritt
>
>Thanks for filling in the blanks Ethan. I knew there was a demo but
>could not find it to post the link.
>
>BTW where is the data file that goes with the example? "silver.dat"
>does not seem to be provided witht the example script.
It is one of the data files provided in the gnuplot distribution
package along with the demos. If you have a copy of gnuplot, you
should also have a copy of the demos and the associated data files.
--
Ethan A Merritt
Anton81
>> gnuplot csplines are too wiggly and bezier misses the points too much.
> Any particular reason you didn't mention acsplines?
I tried it some time ago and it wasn't quiet right. This time I forgot to
try (thought its similar to others), but actually it gives good results in
this case.
Yet, maybe someone knows a library with a modern shape preserving technique
with adjustable parameters. It's probably a fundamental requirement for
noisy data with peaks.
goo...@catking.net
It seems you are a bit unclear about what you can expect from data
filtering and processing. You are not asking to preserve the "shape"
of your data you are trying to alter it.
It is up to you to define which aspects you wish to remove or retain
but in all cases you are modifying the data. This is my critisism of
csplines (and hence acsplines). What you are actually doing to your
data set becomes very obscure. Saying the results are "good" because
it does not wiggle too much is clearly of no scientific value. Sadly
this sort of "smoothing" is all that gnuplot offers.
If you have a mathematical model for what your data represent you can
use the fit command . However , if you just want to remove high
frequency "noise" a frequency filter like a running mean may be a good
choise.
In fitting csplines or wieghted csplines you are imposing conditions
on the fit that are very likely inappropriate for whatever your data
respresents. Unless you understand what those conditions are you
should not be using csplines.
Also watch out for data shift in using a running mean , a simple
script like the gnuplot demo shifts the mean by half the sample
window, so the x axis ticks are not valid for the plotted mean. You
could use something like the following awk script and call it from
gnuplot using the system command. This corrects for the shift.
HTH.
#! /usr/bin/awk -f
# calculate running mean over 4,8,12 or 16 pts
# source x data is decimal time
# ppl = pulses per litre in y , set to 1 if not needed
BEGIN { print "# running mean reductions"; ppl60=60/14.4; xm=tm=0.0;
started = 0 ; x1=x2=x3=x4=0.0; t1=t2=t3=t4=0.0;}
debug=0
# skip comment and blank lines
($0 !~ /^#/)&&($0 != "") {
window=16;
if (started == 0) {
count = 1
started = 1
t0 = tl = t1 = $1
x0 = xl = x1 = $2
}
else {
count++;
t4=t3;t3=t2;t2=t1;t1=$1
x4=x3;x3=x2;x2=x1;x1=$2
if (window >= 8){
t8=t7;t7=t6;t6=t5;t5=t4;
x8=x7;x7=x6;x6=x5;x5=x4;
}
if (window >= 12){
t12=t11;t11=t10;t10=t9;t9=t8;
x12=x11;x11=x10;x10=x9;x9=x8;
}
if (window == 16){
t16=t15;t15=t14;t14=t13;t13=t12;
x16=x15;x15=x14;x14=x13;x13=x12;
}
xm=(x1+x2+x3+x4+x5+x6+x7+x8+x9+x10+x11+x12+x13+x14+x15+x16)/
window;
tm=(t1+t2+t3+t4+t5+t6+t7+t8+t9+t10+t11+t12+t13+t14+t15+t16)/
window;
}
if ((count>window-1)&&(count%1==0)) {print tm "\t"(xm-xl)*ppl60/(tm-
tl) }
xl=xm;tl=tm
}
END {
print "\n\n#running mean ",window," complete\n";
}
Anton81
> filtering and processing. You are not asking to preserve the "shape"
> of your data you are trying to alter it.
large, so that the data scatteres a lot. Whenever there is only a
Gaussian peak, I fit the data to the model and it works fine. However
when the bump is a bit more complicating I'm having trouble
reproducing the big bump and yet removing the scatterung noise.
I read something about shape preserving (Fourier?) filters. Thought
there might be a library out there.
> It is up to you to define which aspects you wish to remove or retain
> but in all cases you are modifying the data.
Of course I will show the raw points, but I want to connect them with
a line as a guide to the eye.
> Saying the results are "good" because
> it does not wiggle too much is clearly of no scientific value. Sadly
> this sort of "smoothing" is all that gnuplot offers.
Gnuplot is great. I hoped someknow knows a library to filter out
noise.
> If you have a mathematical model for what your data represent you can
> use the fit command . However , if you just want to remove high
> frequency "noise" a frequency filter like a running mean may be a good
> choise.
The only problem about running mean is that it reduces real peaks.
Thanks for the sample program! I'll go through it. There is something
to learn.
goo...@catking.net
> The only problem about running mean is that it reduces real peaks.
>
> Thanks for the sample program! I'll go through it. There is something
> to learn.
It seems you are looking for a lowpass filter to cut the noise part of
the signal.
You could look into lanczos3 . This is a kind of fourier technique
since it relies on convoluting the signal with a truncated sinc
function which is the fourier transform of a step function in
frequency domain.
lanczos3 uses a precalculated column matrix with 5 elements. You
calculate each filtered point by weighting neighbouring points with
these values. Quite simple to implement really .
There are many sources of code but one example you could look at is
gimp. Since 2.4 it uses this technique in both x and y to scale
images. Look in scale-region.c
Running mean is also a low pass filter but with a less sharp cut-off.
But any filter will reduce your peaks if you don't have enough data
points. You are trying to reduce very heavily the "noise" without
touching the slower peaks. You can only hope to approach this ideal if
you have very, very many more data samples in a peak than in the
glitches you want to remove.
Since the perfect step filter does not exist , you will always get
some attenuation.
HTH.
Hermann Peifer
I have no recommendation but if you are getting to somewhere via some local regression technique, Lanczos algorithm, Fourier transformation or whatever other method: please keep us informed.
Thanks, Hermann
Reginald Beardsley
You might want to look at:
A Practical Guide to Splines
Carl de Boor
Springer-Verlag 2001
ISBN 0-387-95366-3
for which code is available at netlib.org as pppack
and:
Curve and Surface Fitting
Peter Lancaster & Kestutis Salkauskas
Academic Press 1986
ISBN 0-12-436060-2
0-12-436061-0 (paper)