Gnuplot fit giving bad fits
fake...@invalid.domain
it is either bug in my understanding or a bug.
I have exponently decaying data that gnuplot fit generates an
exponently increasing answer. Here is the batch file
# elf 03
set logscale y
set title "elf 03"
set terminal png small color
set output 'elf-03.png'
f(x) = exp(a*x + b)
fit f(x) 'elf-03' using :1 via a, b
plot 'elf-03', exp(a*x+b) with linesp
Here is the data file
31113501
39241025
24378100
34277093
10518000
7979679
7305569
9835725
7084391
398943
289178
275286
256141
228197
125078
75499
The output can be viewed at
http://www.math.fsu.edu/~bellenot/class/f04/cal1/proj/elf-03.png
3 out of 20 files like this are ``bad''
--
Steven Bellenot http://www.math.fsu.edu/~bellenot
Department of Mathematics real....@line.below
Florida State University bellenot at math.fsu.edu
Tallahassee, FL 32306-4510 USA +1 (850) 644-7189 (FAX: 4053)
Juergen Wieferink
> I have exponently decaying data that gnuplot fit generates an
> exponently increasing answer. Here is the batch file
>
> # elf 03
> set logscale y
> set title "elf 03"
> set terminal png small color
> set output 'elf-03.png'
> f(x) = exp(a*x + b)
> fit f(x) 'elf-03' using :1 via a, b
> plot 'elf-03', exp(a*x+b) with linesp
[...]
> 3 out of 20 files like this are ``bad''
It's difficult to say what happened without some output of the fit
command. But I think, the fit just did not converge. This is not a
bug, but some kind of unavoidable if the fit is nonlinear. As a
first step, you could use
f(x) = b * exp(a*x)
to get the fit linear in at least one of the parameters. Using a
good starting point (negative a) should help, too. Possibly you
could use something like
g(x) = b + a*x
fit g(x) 'elf-03' using 0:(log($1)) via a, b
plot 'elf-03', exp(g(x))
to get the fit completely linear. A linear fit should always
converge.
Juergen
PS: You could have guessed this from "help fit" and subnodes.
Petrik Galvosas
with your data and your script I can at least make the fit converge by
normalizing the data. But the error stays large since the data just
doesn't fit.
This suggestion from Jürgen
> fit g(x) 'elf-03' using 0:(log($1)) via a, b
works also fine but keep in mind that the result will be quite different
from the result you get with your model function. Especially with your
data.
Petrik
PS.: The script I used:
set logscale y
N=`head -1 'elf-03'`
f(x) = exp(a*x + b)
f_1(x) = a_1*x + b_1
fit f(x) 'elf-03' using 0:($1/N) via a, b
fit f_1(x) 'elf-03' using 0:(log($1/N)) via a_1, b_1
plot 'elf-03' using 0:($1/N), exp(a*x+b) with linesp, exp(a_1*x+b_1)
with linesp
Hans-Bernhard Broeker
> I really am at a lost to explain why this fit is so bad. Obviously
> it is either bug in my understanding or a bug.
It's a lack of understanding, not exactly a bug. Reading *all* of
'help fit', including all the sub-nodes, should have helped. Lucas
Hart and I went to rather great lengths optimizing all that
documentation. If it still isn't sufficient, would you care
suggesting a modification that would have helped you?
> I have exponently decaying data that gnuplot fit generates an
> exponently increasing answer.
Well, as we say in the docs (under 'help fit tips'), nonlinear fitting
is an art. It's not a point-and-click, fire-and-forget type of tool.
You have to learn how to use it properly, and carefully check results.
It takes experience to use 'fit' correctly.
> Here is the batch file
> # elf 03
> set logscale y
> set title "elf 03"
> set terminal png small color
> set output 'elf-03.png'
> f(x) = exp(a*x + b)
> fit f(x) 'elf-03' using :1 via a, b
> plot 'elf-03', exp(a*x+b) with linesp
The crucial things missing in this batch file are startup values for
the parameters. As it is, both a and b will start out being 1.0,
which are so far away from what the actual solutions are (at least in
terms of function falues) that 'fit' simply doesn't manage to find the
way to the solution point, in (a,b) parameter space.
For a fit to a function as nonlinear as an exponential, it's
_absolutely_crucial_ to have at least somewhat usable startup values.
As a rough measure, make a plot of your data and that target function
*before* you run fit. If the target function doesn't resemble the
data at all, odds are that 'fit' won't be able to find the solution.
One possibility in this particular case is to (ab)use a preliminary,
linear fit to logarithmized data to extract a first shot at the
parameters, before proceeding to the real fit. E.g.:
fit log(f(x)) 'elf-03' using 0:(log($1)) via a, b
fit f(x) 'elf-03' using 0:1 via a, b
--
Hans-Bernhard Broeker (bro...@physik.rwth-aachen.de)
Even if all the snow were burnt, ashes would remain.
fake...@invalid.domain
Hans-Bernhard Broeker <bro...@physik.rwth-aachen.de> wrote:
>fake...@invalid.domain wrote:
>> I really am at a lost to explain why this fit is so bad. Obviously
>> it is either bug in my understanding or a bug.
>
>It's a lack of understanding, not exactly a bug. Reading *all* of
>'help fit', including all the sub-nodes, should have helped. Lucas
>Hart and I went to rather great lengths optimizing all that
>documentation. If it still isn't sufficient, would you care
>suggesting a modification that would have helped you?
Indeed you are right. If I had read more of the sub nodes, I would
have figured out how to give initial values to the parameters and
with intial values of a = -0.4 b = 18, I get results that more
pleasing.
Modifications that would have helped me:
1. Starting the tips with the statement "Nonlinear fitting is an art"
rather that ending with the quote.
2. A statement that says the algorithm can converge to a bad fit. With
the data given. I got the following output.
After 5 iterations the fit converged.
final sum of squares of residuals : 4.65205e+15
rel. change during last iteration : -5.32242e-06
degrees of freedom (ndf) : 14
rms of residuals (stdfit) = sqrt(WSSR/ndf) : 1.82288e+07
variance of residuals (reduced chisquare) = WSSR/ndf : 3.32289e+14
Final set of parameters Asymptotic Standard Error
======================= ==========================
a = 0.65975 +/- 98.13 (1.487e+04%)
b = 2.42015 +/- 1443 (5.963e+04%)
correlation matrix of the fit parameters:
a b
a 1.000
b -0.999 1.000
3. I like this tip below, eventually I concluded that this what I should try.
>The crucial things missing in this batch file are startup values for
>the parameters. As it is, both a and b will start out being 1.0,
>which are so far away from what the actual solutions are (at least in
>terms of function falues) that 'fit' simply doesn't manage to find the
>way to the solution point, in (a,b) parameter space.
Although apparently I found a local min in parameter space.
4. The sub node on starting_values doesn't tell you how to set the
starting values, nor points to the subnode that tells you. I tried
presetting the values of a and b it the script file before the
fit command, in the fit command (at the end). Before it dawned on
me that they had to be in a separate file.
---
While I'm suggesting changes to the help file, another problem that
took me a long time to figure out was how to get the values of the
parameters a and b into the plot. Of course this was done via
set label 'a = %3.5g',a at <xlocation>, <ylocation>
At the main help level I tried several topics. None seemed to help.
Eventually I gave up and tried google instead. Even doing a
grep variable /sw/share/gnuplot/3.8j/gnuplot.gih
didn't seem to help. The glossary doesn't define variable, or parameter.
>
>> I have exponently decaying data that gnuplot fit generates an
>> exponently increasing answer.
>
Thanks for your help.