BigDecimal, big confusion

[Chris Berkhout]

Hi all,

I'm trying to use BigDecimal for accurate calculations. I think I've figured it out (at least enough for now), but wanted to write it up and see if anyone could confirm or correct my thinking...

I want to have 16 significant digits of precision. So the first thing I thought to check was whether I can have 1 / 3 come out as '0.33333 33333 33333 3'.

When I initialise with strings and explicit precisions I get this...

one = BigDecimal.new('1', 16)   # => #<BigDecimal:7f02cdc1ed88,'0.1E1',9(18)>

three = BigDecimal.new('3', 16)   # => #<BigDecimal:7f02cdc14bf8,'0.3E1',9(18)>

one / three   # => #<BigDecimal:7f02cdc0d178,'0.333333333E0',9(27)>

...only 9 significant digits. I get the same when initialising with longer strings, like '1.000000000000000'.

The 9(18) at the end of the objects means current number of significant digits is 9 and maximum number of significant digits is 18.

So the problem seemed to be that I couldn't set the current number of significant digits to higher than 9 if those digits are zero.

According to Wikipedia (http://en.wikipedia.org/wiki/Significant_figures), trailing zeros aren't significant digits (although they could contribute to an "arithmetic precision" defined as a number of decimal places... maybe that would be better).

Looking back at the Ruby documentation I noticed the method BigDecimal#div (aka #quo):

    http://www.ruby-doc.org/stdlib-2.1.0/libdoc/bigdecimal/rdoc/BigDecimal.html#method-i-quo

one = BigDecimal.new('1', 16)

three = BigDecimal.new('3', 16)

one.div(three, 16)   # => #<BigDecimal:7f02cd772528,'0.3333333333 333333E0',18(36)>

So, if I use BigDecimal#div and BigDecimal#mult with significant digits specified, I guess I'll get the answers I'm expecting.

I'm still a bit confused by this:

    BigDecimal.new('1.00000000000000000001', 21)   # => #<BigDecimal:7f02cd1d9ad0,'0.1000000000 0000000000 1E1',36(36)>

    BigDecimal.new('1.00000000000000000001', 21).round(20)   # => #<BigDecimal:7f02cd196d70,'0.1000000000 0000000000 1E1',36(45)>

    BigDecimal.new('1.00000000000000000001', 21).round(19)   # => #<BigDecimal:7f02cd134da0,'0.1E1',9(45)>

The last one suggests it's accurate to 9 places, but it's actually accurate to 19 places.

Any thoughts appreciated!

Cheers,

Chris

[Steve Hoeksema]

It depends on your use-case, but Rational might do what you're after? http://www.ruby-doc.org/core-2.1.0/Rational.html

[Chris Berkhout]

Thanks for the suggestion Steve! Actually its financial numbers, so I do want decimal... Just need to know how precise they will be.

On Feb 24, 2014 5:07 PM, "Steve Hoeksema" <st...@kotiri.com> wrote:

It depends on your use-case, but Rational might do what you're after? http://www.ruby-doc.org/core-2.1.0/Rational.html

--
You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rails-oceani...@googlegroups.com.
To post to this group, send email to rails-...@googlegroups.com.
Visit this group at http://groups.google.com/group/rails-oceania.
For more options, visit https://groups.google.com/groups/opt_out.

[Simon Russell]

Probably not useful input, as I don't know your circumstances, but if you care about precision and it's financial, I'd think seriously about using integers (assuming of course that you haven't already). You get control of exactly where truncation occurs and in what direction.

It sounds like you've chosen a base unit, so you could work from there. At least you won't always be worried that somehow you've accidentally used floating point.

[Steven Ringo]

To expand more on Simon’s suggestion, have you considered using the “money" gem? (or the aptly-named "money-rails” variant). It uses integers under-the-bonnet (by default) but abstracts them away nicely.

The creators of this gem have ostensibly been through similar experiences :-)

Steve

[Chris Rode]

Hi Chris

You probably know this already having looked into the BigDecimal gem however I thought I would post here anyway

 

The money gem is definitely not what you are after as it rounds way too much. It may compute with integer computation (I havnt looked) however it stores 2 decimal places and seems to round between each computation. This unfortunately makes it unusable for any money application where division or multiplication is needed. It is possibly useful to help translate between currencies.. however even then a higher precision would probably be in order

If that doesn't sway you try (19.957).to_money on the console :)

BigDecimal is definitely the way to go (unless you role your own BigDecimal with integers) however I cannot shed much light on your initial question off the top of my head, sorry.

[Steven Ringo]

The money gem is definitely not what you are after as it rounds way too much. It may compute with integer computation (I havnt looked) however it stores 2 decimal places and seems to round between each computation.

Are you sure? You can set currency exponent (https://github.com/RubyMoney/money#currency-exponent) to as many decimal places as you like.

This unfortunately makes it unusable for any money application where division or multiplication is needed. It is possibly useful to help translate between currencies.. however even then a higher precision would probably be in order

If that doesn't sway you try (19.957).to_money on the console :)

No definition of “money” here is provided. What currency is it? How has it been defined? Sure if its USD to 2 decimal places then it will round. But nothing stops you from defining a currency as USD to say 6 decimal places.

Of course the BigDecimal gem may be exactly what you need.

Steve

[Chris Rode]

Hi Steven

I'll definitely look into the currency exponent part of the Money gem, thanks for pointing it out to me

I'll be a very happy chap if I am incorrect :)

--

[Simon Russell]

This is possibly getting quite off track, but the money gem's lack of understanding of the details of different currencies is why I've always ended up rolling my own. Unfortunately I never actually get around to releasing it as a gem...

I also haven't looked at it in a while, but from what I remember the money gem's not afraid of the odd silent floating point calculation as well; which more-or-less rules it out for me.

However, there's nothing wrong with rounding/truncating between operations assuming it has the precision you need. It's quite possibly more correct.

The trick is to know when the rounding is occurring and minimise it. Or apply the appropriate legislated rules, if there are any. If you're relying on excess precision in the middle of an algorithm, you're still rounding, just not as obviously.

Anyway, let's hope Chris's original question gets answered; there are at least a few directions to go in this thread.

--

[Chris Berkhout]

Thanks everyone.

The money gem looks okay, given that the precision can be set per currency, but I'm probably going to stick with BigDecimal.

I want AUD to at least 2 decimal places and units of an equity to at least 8 decimal places. My current preference is to use one kind of number that works for everything. Using an integer is difficult because I'd need to also store what kind of unit it is.

I think I'll wrap BigDecimal, to ensure I'm always using #div and #mult instead of / and *, and always with the correct precision. I could monkey patch BigDecimal to warn if it's been instantiated unexpectedly, but that's probably overkill. I'll store them in PostgreSQL as NUMERIC, which allows for a precision (total significant digits) and scale (decimal places) and will need it to be a little bigger to handle both my cases with one type. PostgreSQL does "round to nearest, round half away from zero". BigDecimal also allows "round to nearest, round half to even", which is less biased, but I'll stick with "round to nearest, round half away from zero" there too, because I think that will be okay and I want to use one kind of rounding everywhere.

So I'll have reasonable and reliable precision and rounding, then the only other thing is to make sure that calculations are done in a way that doesn't amplify the rounding that does take place.

Data is passed around in JSON, and to make that work reasonably I'll use strings rather than ints or floats.

I think that the type of precision provided by https://github.com/jgoizueta/flt might actually be better, but I'm not inclined to use it for now because it's not as fast (currently in pure Ruby) and more importantly, I'm using the Sequel gem to talk to the DB, and Sequel knows how to handle BigDecimal (but not Flt::DecNum).

Cheers,

Chris

[Chris Berkhout]

This is also a good guide: http://floating-point-gui.de/

[Simon Russell]

Sounds like you've thought it through; and certainly wrapping to make
sure you use what you want to use sounds like a good plan.

If it were me, I would choose 8 decimal places for everything, give it
a name, and use it around the place as an integer. (I've dealt with
similar problems relating to the price of electricity on the wholesale
market, which is quoted to thousandths of a cent). When you want to
round to an AUD amount, you can do it with simple integer maths.
There is, clearly, a mental overhead in doing this, but if you're
wrapping stuff anyway, it might not be much.

On Tue, Feb 25, 2014 at 12:12 PM, Chris Berkhout
<chrisb...@gmail.com> wrote:

[Andy Kitchen]

Hi Chris,

Here is most of an answer to your question, but the documentation is
_terrible_. The ruby standard library really has some pretty rough
corners once you get off the beaten track.

So here goes:

1. BigDecimals are mutable, so they have a current and maximum
precision, and this makes a difference. e.g. If you are using a
BigDecimal as an accumulator you care the most about you maximum
precision.

2. Because of the way BigDecimals are implemented: precision both
current and max must be a multiple of Math.log10(BigDecimal::BASE) (on
64-bit machines this is 9, on 32-bit machines it is 4)

3. The "digits" argument of BigDecimal.new only affects maximum
precision and seems to always give you more precision than you ask
for. It will give you the closest multiple of BASE then and BASE one
more time.

e.g.

irb(main):016:0> BigDecimal.new("0", 1)
=> #<BigDecimal:7ff8b9ae9bd8,'0.0',9(18)>

irb(main):017:0> BigDecimal.new("0", 9)
=> #<BigDecimal:7ff8b9af15b8,'0.0',9(18)>

irb(main):024:0> BigDecimal.new("0", 10)
=> #<BigDecimal:7ff8b9ac9018,'0.0',9(27)>

4. BigDecimal.new seems is buggy, you get different results if you use
a string or a fixnum as the initial value. The digits argument doesn't
seem to do anything if the input is a fixnum.

irb(main):027:0> BigDecimal.new(0, 60)
=> #<BigDecimal:7ff8b9aab180,'0.0',9(36)>

irb(main):047:0> BigDecimal.new(0, 100)
=> #<BigDecimal:7ff8b9999e90,'0.0',9(36)>

irb(main):028:0> BigDecimal.new("0", 60)
=> #<BigDecimal:7ff8b9aa3890,'0.0',9(72)>

5. You seem to get somewhat sane results if you use #div and #mul; and
specify your precision. The operator versions #/ and #* seem to try
and pick a decent default output precision, although this is not
documented and doesn't seem to work to well.

6. BigDecimals in ruby are poorly documented and riddled with gotchas
and it seems bugs.

7. Ruby is not for big-boy problems


Hope that helps.

Kind Regards

AK

[Andy Kitchen]

Hope that helps.

Kind Regards

On Mon, Feb 24, 2014 at 4:24 PM, Chris Berkhout <chrisb...@gmail.com> wrote:

[Chris Berkhout]

Oh yeah.... I guess that makes the integer option quite reasonable. Still, I probably need to go via BigDecimal to get into the DB.

After reading more of http://floating-point-gui.de/ and http://speleotrove.com/decimal/decifaq.html, I think I understand it better now.

BigDecimal defines its precision in terms of significant digits instead of decimal places, and doesn't preserve trailing zeros. It allows for "very large or very accurate floating point numbers". That is different from what floating-point-gui.de calls an "exact type" - as defined by IEEE754-2008 and implemented in the Flt gem. It's not surprising the BigDecimal isn't an exact type, since it was written in 2002, and the IEEE standard didn't come until 2008 and still isn't widely implemented.

I believe BigDecimals are often successfully used where an exact type seems better. When using a precision for all instantiations and operations, it's okay to thow away trailing zeros then later assume trailing digits are zero, if you know that that many digits would have been preserved if they were initially non-zero.

Andy's examples of digits of precision being ignored when initialised with a fixnum look bad, but I'm always using strings to initialize (another thing that could be enforced by the wrapper), and BigDecimal giving me more precision than I asked for (this is documented) isn't a problem, but so I guess it'll be okay.

If there is some other issue and I can't depend on BigDecimal for a given size of numbers and number of decimal places, I guess I'll switch my wrapper to use integers internally.

Cheers,

Chris

[Danial Pearce]

On 24 February 2014 23:10, Simon Russell <si...@bellyphant.com> wrote:
> [...] the money gem's lack of

> understanding of the details of different currencies is why I've always
> ended up rolling my own.
>

> I also haven't looked at it in a while, but from what I remember the money
> gem's not afraid of the odd silent floating point calculation as well; which
> more-or-less rules it out for me.

Advising people to not use a gem because it burnt you once, years ago,
on something you aren't sure about and can't even reproduce now, is
not something we should advocate. I'm sure the owners of money gem
would love to know of the "bug" you found so they can fix it for
everyone, if they haven't already.

Solutions. Work out your base precision, convert all numbers to that,
kind of like creating a new "currency" that has 9 decimals and treat
everything as "cents", so you deal purely in integers and deal with
the rounding in your wrapper class.

Alternatively: https://github.com/shanna/big_money

regards,
Dan

[Simon Russell]

I was being polite; the money gem has many reasons I wouldn't use it,
and fixing some of the design problems would change the API too much.

big_money looks okay, I've had a look over that one in the past. It
uses BigDecimal under the hood though, and comes with a fair bit of
baggage.

Your solution is a fine one, which is why I suggested it.

[Shane Hanna]

big_money looks okay, I've had a look over that one in the past.  It
uses BigDecimal under the hood though, and comes with a fair bit of
baggage.

I don't want to derail the thread but I publish the big_money gem. Simon if you'd care to expand on 'a fair bit of baggage' Simon I'll take on board anything specific if you have the time to email me or start a new thread.

For the record I wouldn't use an integer. My solution for currency handling is always big_money, a single internal system currency along with a record of any conversions, postgresql's numeric data type and nested transactions for a double entry accounting table. It's simple but I end up with a single account currency transfer method so it's easy to control rounding and validate integrity.

Shane.

[Simon Russell]

Hi Shane,

Sure, I'll contact you off-list though, and forgive me if it takes a
few days -- currently under a fairly big workload.

Just to clarify for others reading, by "baggage" I didn't mean cruft;
I more meant that it comes with a bunch of functionality
(exchange/parsing), but looking at it better, clearly it's optionally
required, so apologies. I'd still disagree on the BigDecimal thing,
but can take that up off list; and converting currencies into a single
internal one sounds like it might be very application-specific.

Nice work on actually releasing a better Money gem though, it's
definitely needed.

Simon.