Numeric Data type issue

[adil v]

Hi Good day, I have a situation
1 I am using the firebird version 3.0.10  and I cannot save the value  123456789123456789.123456789123456789 in numeric(18,18)  i.e. I cannot save value 18 digits in precision and 18 digits in scale when I tried I getting error "Invalid floating point operation"?
2 also I cannot multiply numeric(15,4) with numeric(15,15) when I tried I getting "The result of an integer operation caused the most significant bit of the result to carry", could you please advise

[Mark Rotteveel]

Your value is not a NUMERIC(18, 18), it is a NUMERIC(36, 18), which
requires Firebird 4.0. The precision is the total number of digits in
the number, before *and* after the decimal point.

When multiplying a NUMERIC(15, 4) with a NUMERIC(15, 15), the result (in
Firebird 3.0 and earlier) would be a NUMERIC(18, 19), which is not
possible, as the scale cannot exceed the precision. You will need to
round the NUMERIC(15, 15) to a lower scale before multiplying, or cast
to DOUBLE PRECISION and after multiplication, cast back to your desired
NUMERIC type.

Mark
--
Mark Rotteveel

[Tim Crawford]

Also note that DOUBLE PRECISION calculations will lose accuracy after I think 15 decimals.

I think FB 3 has a long float, and not sure what up with options in FB 4 , didn't look

  Original Message  

Mark
--
Mark Rotteveel

--
You received this message because you are subscribed to the Google Groups "firebird-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebird-suppo...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/firebird-support/5c7eeb5d-4e3b-22fe-dd66-a19232c0e5cb%40lawinegevaar.nl.

[Mark Rotteveel]

On 21-06-2022 21:29, Tim Crawford wrote:
> Also note that DOUBLE PRECISION calculations will lose accuracy after I think 15 decimals.

Thanks. I hadn't that in an earlier revision of my reply, but somewhere
along the line, I edited that out.

> I think FB 3 has a long float, and not sure what up with options in FB 4 , didn't look

The type LONG FLOAT is a non-standard alias for DOUBLE PRECISION, and
shouldn't be used.

The options in Firebird 4.0 include:

NUMERIC(38, s)/DECIMAL(38, s) or using DECFLOAT (DECFLOAT(34)).

Mark
--
Mark Rotteveel

[Dimitry Sibiryakov]

Mark Rotteveel wrote 22.06.2022 8:24:
> The options in Firebird 4.0 include:
>
> NUMERIC(38, s)/DECIMAL(38, s) or using DECFLOAT (DECFLOAT(34)).

I'm curious what real-life application requires such precision.
It cannot be financial because usual bookkeeping rules require rounding of
intermediate results so... astronomy?..

--
WBR, SD.

[Karol Bieniaszewski]

Even astronomy do not require such precision, but maybe some physicist yes.

 

For example astronomical ppmxl database definition, numeric(10,6) , numeric(7.2)😉.

   1- 19  A19   ---      PPMXL   Identifier (5)

  21- 30  F10.6 deg      RAdeg   Right Ascension J2000.0, epoch 2000.0

  31- 40  F10.6 deg      DEdeg   Declination J2000.0, epoch 2000.0

  42- 49  F8.1  mas/yr   pmRA    Proper Motion in RA*cos(DEdeg)

  51- 58  F8.1  mas/yr   pmDE    Proper Motion in Dec

  60- 66  F7.2  yr       epRA    Mean Epoch (RA)

  68- 74  F7.2  yr       epDE    Mean Epoch (Dec)

  76- 78  I3    mas    e_RAdeg   Mean error in RA*cos(DEdeg) at mean epoch (6)

  80- 82  I3    mas    e_DEdeg   Mean error in Dec at mean epoch (6)

  84- 87  F4.1  mas/yr e_pmRA    Mean error in pmRA*cos(DEdeg)

  89- 92  F4.1  mas/yr e_pmDE    Mean error in pmDec

  94- 99  F6.3  mag      Jmag    ?=- J magnitude from 2MASS (Cat. II/246)

     100  A1    ---      ---     [:]

101-105  F5.3  mag    e_Jmag    ?=- J total magnitude uncertainty

107-112  F6.3  mag      Hmag    ?=- H magnitude from 2MASS (Cat. II/246)

     113  A1    ---      ---     [:]

114-118  F5.3  mag    e_Hmag    ?=- H total magnitude uncertainty

120-125  F6.3  mag      Kmag    ?=- Ks magnitude from 2MASS (Cat. II/246)

     126  A1    ---      ---     [:]

127-131  F5.3  mag    e_Kmag    ?=- Ks total magnitude uncertainty

133-137  F5.2  mag      b1mag   ?=- B mag from USNO-B, first epoch (1)

139-143  F5.2  mag      b2mag   ?=- B mag from USNO-B, second epoch (1)

145-149  F5.2  mag      r1mag   ?=- R mag from USNO-B, first epoch (1)

151-155  F5.2  mag      r2mag   ?=- R mag from USNO-B, second epoch (1)

157-161  F5.2  mag      imag    ?=- I mag from USNO-B (1)

163-167  A5    ---      Smags   [0-8-] Surveys used for USNO-B magnitudes (2)

169-170  I2    ---      No      ?=- Number of observations used (4)

172-173  I2    ---      fl      Flags (3)

 

 

Regards,

Karol Bieniaszewski

--

You received this message because you are subscribed to the Google Groups "firebird-support" group.

To unsubscribe from this group and stop receiving emails from it, send an email to firebird-suppo...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/firebird-support/fc28eb51-f59b-58da-5d13-8a304ff9e92f%40ibphoenix.com.