Canvas precision

Showing 1-5 of 5 messages
Canvas precision tombert 6/14/13 7:13 AM
Before digging deep into gnuplot internals, I would like to know if it's normal that I get a resolution of only 3 digits after the comma when using canvas 2D plots?

I have this data file:

25785.0000998 -68.8343006527
25785.0001028012 -80.5616032527
25785.0001058025 -87.8913075527
25785.0001088037 -87.6995350527
25785.000111805 -87.0571553527
25785.0001148063 -86.7244984527
25785.0001178075 -84.5539128527
...

it has ~18000 entries. For those who are interested, the firs columns are frequencies (MHz), the seconds ones are gains (dB).

Now when zooming deep into the canvas plot the points are drawn only to precision of 0.001 - which means that several points are drawn on the same x-position.
This precision seems to change depending on the number of points I use.

I am using gnuplot 4.6.

Anybody an idea?

thx
Re: Canvas precision Karl 6/14/13 8:08 AM
Have you tried substracting 25785 from the frequencies? Your numbers
differ in the 9-10th position, i悲 say that is 32 bit precision.

- Karl

Re: Canvas precision tombert 6/14/13 12:38 PM
I think I found the problem.

With the following data:
25785.0001 -68.997037
25785.00010027 -69.1994067
25785.00010054 -69.5961962
25785.00010081 -70.1889926
25785.00010108 -70.9806719
25785.00010135 -71.9761248
... till ...
25785.99586 -62.0013247
25785.99668 -61.3820468
25785.9975 -60.851285
25785.99832 -60.4307604
25785.99914 -60.136098
25785.99996 -59.9775515
... 6000 points


gnuplot creates 6000 path points:
M(941,2093);
L(941,2107);
L(941,2133);
L(941,2172);
L(941,2225);
L(941,2292);
... till ...
L(10459,1627);
L(10467,1586);
L(10475,1550);
L(10483,1522);
L(10491,1503);
L(10499,1492);

One sees that the first data points all got translated into 941. This seems to be dependent on the total number of points.

When I delete the last 5000 points, then gnuplot translates into:
M(940,627);
L(944,651);
L(947,697);
L(951,766);
L(955,858);
L(958,974);
...

So I would like to know how to increase the value range for these translations?

thx
Re: Canvas precision tombert 6/14/13 12:48 PM
And probably here is the solution:

from canvas.trm:

#define CANVAS_OVERSAMPLE        10.
#define CANVAS_XMAX                (600 * CANVAS_OVERSAMPLE)

If I increase the sampling would this solve my problem? Would it be somehow nice if this value could be changed via gnuplot command?

thx
Re: Canvas precision tombert 6/15/13 12:28 AM