## Canvas precision

Showing 1-5 of 5 messages
 Canvas precision tombert 6/14/13 7:13 AM Before digging deep into gnuplot internals, I would like to know if it's normal that I get a resolution of only 3 digits after the comma when using canvas 2D plots? I have this data file: 25785.0000998 -68.8343006527 25785.0001028012 -80.5616032527 25785.0001058025 -87.8913075527 25785.0001088037 -87.6995350527 25785.000111805 -87.0571553527 25785.0001148063 -86.7244984527 25785.0001178075 -84.5539128527 ... it has ~18000 entries. For those who are interested, the firs columns are frequencies (MHz), the seconds ones are gains (dB). Now when zooming deep into the canvas plot the points are drawn only to precision of 0.001 - which means that several points are drawn on the same x-position. This precision seems to change depending on the number of points I use. I am using gnuplot 4.6. Anybody an idea? thx Re: Canvas precision Karl 6/14/13 8:08 AM Have you tried substracting 25785 from the frequencies? Your numbers differ in the 9-10th position, i悲 say that is 32 bit precision. - Karl Re: Canvas precision tombert 6/14/13 12:38 PM I think I found the problem. With the following data: 25785.0001 -68.997037 25785.00010027 -69.1994067 25785.00010054 -69.5961962 25785.00010081 -70.1889926 25785.00010108 -70.9806719 25785.00010135 -71.9761248 ... till ... 25785.99586 -62.0013247 25785.99668 -61.3820468 25785.9975 -60.851285 25785.99832 -60.4307604 25785.99914 -60.136098 25785.99996 -59.9775515 ... 6000 points gnuplot creates 6000 path points: M(941,2093); L(941,2107); L(941,2133); L(941,2172); L(941,2225); L(941,2292); ... till ... L(10459,1627); L(10467,1586); L(10475,1550); L(10483,1522); L(10491,1503); L(10499,1492); One sees that the first data points all got translated into 941. This seems to be dependent on the total number of points. When I delete the last 5000 points, then gnuplot translates into: M(940,627); L(944,651); L(947,697); L(951,766); L(955,858); L(958,974); ... So I would like to know how to increase the value range for these translations? thx Re: Canvas precision tombert 6/14/13 12:48 PM And probably here is the solution: from canvas.trm: #define CANVAS_OVERSAMPLE        10. #define CANVAS_XMAX                (600 * CANVAS_OVERSAMPLE) If I increase the sampling would this solve my problem? Would it be somehow nice if this value could be changed via gnuplot command? thx Re: Canvas precision tombert 6/15/13 12:28 AM