Proposal: add native language support for handling Decimal types

[elxi...@gmail.com]

Hi All,

We are evaluating Elixir for a business critical application

Dealing with decimal types is a must.

from https://github.com/ericmj/decimal

iex> alias Decimal, as: D

nil

iex> D.add(D.new(6), D.new(7))

#Decimal<13>

iex> D.div(D.new(1), D.new(3))

#Decimal<0.333333333>

however seeing similar code as above is quite disappointing

being able to write something like below would be more readable and straightforward:

6.75d + 7.293d

6.75d / 3.2d

("d" postfix would differentiate decimal number from float numbers)

we think, having such/similar syntax in-place would be greatly appreciated, 

[Norbert Melzer]

elxi...@gmail.com writes:

> however seeing similar code as above is quite disappointing

Why? It is explicit about its type. And also it makes clear that there
are things not allowed with Decimals.

> 6.75d + 7.293d
>
> 6.75d / 3.2d

In theory, the maintainer of the `decimal` package, could implement some
macros and sigils which allow you to write something like this
(pseudocode):

| D.eval 6.75 + 7.293
| D.eval 6.75 / 3.2

While I am not quite sure right now if the AST of a float were accurate
enough, but a sigil could be used instead:

| D.eval ~d(6.75) + ~d(7.293)

I do not think that it is a necessary addition to core, since the
changes as you suggest them would change parts of the parser to
recognize something new that is just syntactic sugar around a module and
its functions, where most of that is already available in a package on
hex.

Also I do think, that it is easier to enhence `decimal` package like
described above than to integrate the syntax and the functionallity into
elixir itself.

Also regardless how you try it, you will never be able to use them in
guards, your change would create the illusion that they were treatened
by elixir as every other kind of numbers.

[José Valim]

from https://github.com/ericmj/decimal

iex> alias Decimal, as: D

nil

iex> D.add(D.new(6), D.new(7))

#Decimal<13>

iex> D.div(D.new(1), D.new(3))

#Decimal<0.333333333>

I definitely agree it is non-ideal. One option would to at least add a sigil so we can create decimals without going through Decimal.new.

6.75d + 7.293d

6.75d / 3.2d

The issue with this is that we would make the + operator slower for regular operations AND it would still not be allowed in guards, as Norbert mentioned. This has been, as a matter of fact, the biggest blocker for adding Decimals to Elixir itself.

[Peter Hamilton]

This does bring up an alternative sigil, the post numerical type hint.

10d
12f

Etc.

They wouldn't have to be first class citizens in Elixir, but they would have to be lexically supported so macros can use them.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2ByufavV2SKWpBPx0kiqbtS4CD%2BDn0adD8BHi%2BpZc6jxA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Benjamin Scherrey]

My primary target application domain for adoption of Elixir is
financial/logistics apps so being able to deal with currency and apply
certain rounding rules and such is pretty key. Floating point is the
devil when you do these things and most often you build something out
of integers and track significant logical decimal places. Not being
able to use these in guards is probably the most painful limit that I
perceive right now.

From a language perspective, having the kind of numerical support that
Haskell provides for example, would be amazing but perhaps not so
likely. From an implementation perspective, my first inclination that
this is more of a limit of the BEAM environment upon which Elixir
depends, and might need to be made as an extension to Erlang first for
it to become a true first class citizen of Elixir, no?

What actually *is* a viable path for getting such a capability into core Elixir?

-- Ben

[Peter Hamilton]

If you want to roll your own, there are ways to use them in guards. You can build your own guards via macros, provided they decompose to guard compatible functions. It's a little complex and there are effects on performance, etc. You also likely want to use tuples because map support in guards is limited.

Example: suppose we represent decimal as arbitrary precision integers. 10 is {10, 0}, 103.25 is {10325, 100}, etc.

A guard for equivalence could be:

defmacro dec_equals(l, r) do
  quote do
    (l == r) or (elem(l, 1) < elem(r, 1) and elem(r, 0) == elem(l, 0)*elem(r, 1)) or (elem(r, 1) < elem(l, 1) and elem(l, 0) == elem(r, 0)*elem(l, 1))
  end
end

And could be used as:

... when dec_equals(account1, account2) -> ...

Similar approaches could be used for dec_gt, dec_lt and so forth.

Personally, I think we could do a lot more with guards in Elixir if map access was added in Erlang as a guard friendly operator. That may or may not be part of the current discussion though.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CACo3ShhNUVO6JndWwS6LVc%3DC4N0x57hKtT92js94rc_e2Kw4iA%40mail.gmail.com.

[Norbert Melzer]

Peter Hamilton writes:

> You also likely want to use tuples because map support in guards is
> limited.

You probably want to implement them as Maps/Structs, since protocol
support for tuples is limited.

[José Valim]

My primary target application domain for adoption of Elixir is
financial/logistics apps so being able to deal with currency and apply
certain rounding rules and such is pretty key. Floating point is the
devil when you do these things and most often you build something out
of integers and track significant logical decimal places. Not being
able to use these in guards is probably the most painful limit that I
perceive right now.

The interesting thing is that, if you are dealing with money, you could maybe implement your own Money type, possibly inspired by Decimal, and provide sigils such as:

~m(10.00)USD

And money would also guarantee that you don't apply operations to different currencies.

When it comes to operations, we have mentioned multiple options:

1. Use regular functions as in Decimal: number1 |> D.add(number2) |> D.multiply(2)

2. Replace the current + and - and so on by operators that know how to use money or decimal. Such will make regular + and - operators slower but it is likely fine if you are applying it only to parts of your codebase.

3. Provide some sort of decimal evaluator: D.eval number1 + number2 * number3

My long term wish is to actually provide decimal with unit support in the standard library but, because we can't really make it really first class in the standard library (i.e. we can't support it in guards), I still personally feel it is better suited as a separated package. I would be open though to adding support for number sigils like 13.0d, 13.0f although we should also consider how numbers like rationals would play into that.

 

[Ed W]

On 30/06/2016 09:37, José Valim wrote:

My long term wish is to actually provide decimal with unit support in the standard library but, because we can't really make it really first class in the standard library (i.e. we can't support it in guards), I still personally feel it is better suited as a separated package. I would be open though to adding support for number sigils like 13.0d, 13.0f although we should also consider how numbers like rationals would play into that.

What is the appetite likely to be like to include such support in the beam?  It has arbitrary precision integers already, so (without looking) it may be possible to add a decimal place...

The question isn't about doing the work, just about the probability of it being accepted upstream?

Ta

Ed W

[José Valim]

It doesn't need to be accepted upstream. We can support all of those features ourselves. Except guard clauses support which we can still do on our own but it would require us to compile to Core Erlang.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/72d6c82b-959f-af63-5741-f0f23e7e4da3%40wildgooses.com.

[Ed W]

Followup thought:

I ponder how a proposal to leverage the GMP library (or similar) for tighter integration at the beam level?
    https://gmplib.org/

I see there are one or more (rotting?) erlang wrappers, but given that every so often someone wants to compete in the benchmark games and having access to a fast library such as this is vital to get a big boy score...

Probably a bad idea, but would give fast decimals/rationals, in a cross platform way, and good benchmarks scores in silly maths challenges...

Ed W

[Peter Hamilton]

> You probably want to implement them as Maps/Structs, since protocol
support for tuples is limited.

Yep. This is the current conundrum. You can't really do custom guards with maps, and you can't really do protocols with tuples.

As stated, better map support in guards would bridge this gap.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/87ziq3jfry.fsf%40norbert-tuxedo.news.nobbz.de.

[elxi...@gmail.com]

First of all thanks everyone the constructiveness and the great ideas.

I'm new to Elixir, so please forgive me if I've some bad assumptions.

1. Supporting number format like 1.32f or 12.45d, etc would be great
2. Regarding using the regular arithmetic operators like +, -, * and / for decimal type, I fail to see why it would be slower for integers for example. At the end I'd assume only the generated byte code matters. So why not generated the usual byte code when integers or floats are used and generate some advanced logic when the operands are of decimal type?
3. Guards, following Peter's suggestion and using some logic for byte code generation (as mentioned in point #2) again I think it would be possible to handle decimal types transparently.
4. I think simplifying a couple of things, like handling the decimal types (which is a must for financial apps)/removing boiler plates, would have a positive effect on language adoption as well
• I know it's off-topic, but it might worth mention that such simplifications could do wonders, Elixir is great language but why not make even better out of the box, like:
• instead of specifying a method like this:
  @spec add(number, number) :: number 
  def add(x, y), do: ...
  
  it would be far easier to write it a single line like this:
  defs add(number x, number y) :: number, do: ...
• or embrace the familiar lambda syntax for defining in-line functions, so that parameters could be referenced by their name, instead of their position
• etc

[Eric Meadows-Jönsson]

Regarding point 2. We can't generate different bytecode for decimals and integers because at compile we don't what the type of values are.

Regarding point 3. The VM is restrictive in what functions and operations are allowed in guards so we cannot use decimals there.

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/bb8734c9-5d6f-40ad-8bbb-0267786f971e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson

[Dmitry Belyaev]

I'm not sure that decimal type is a must. All the handling can be done on integers representing cents or 0.00001 of a currency or whatever is the required precision, and format the value as appropriate when presenting. That way you don't have all the complications with guards and arithmetic operators.

--
Best wishes,
Dmitry Belyaev

[Onorio Catenacci]

This is quite simple to move forward with (if you still care to).  Make a PR.  Until there's actual code to discuss, we're debating the benefits and drawbacks of theoretical implementations.  

"In theory, theory is no different than practice. In practice though . . . "

It also strikes me as a little bit impractical for someone to say "we should add decimal types to the core of the language" when that someone is not part of the "we" that will build actual code. :)  Please don't misunderstand; I am not saying that you wouldn't help build the code.  I am saying that if you are going to help implement this feature then that's another reason to stop debating theoretical merits and get on with implementing the idea. 

Just my observations.

--

Onorio

[Norbert Melzer]

Dmitry Belyaev writes:

> I'm not sure that decimal type is a must. All the handling can be done
> on integers representing cents or 0.00001 of a currency or whatever is
> the required precision, and format the value as appropriate when
> presenting. That way you don't have all the complications with guards
> and arithmetic operators.

But, I have to admit, even while beeing against the proposal at all, he
looses some safety that you gain by using a module and type which
properly abstracts.

A `1_000_000` that gets passed around is ambigous (are we passing cents
or mills?), while `Decimal<1_000.000>` is obvious.

[Benjamin Scherrey]

Rules regarding rounding and how to distribute the "leftovers" in financial ledgers are critical in financial transactions and belong, properly, in the types and operations of those types. It may be an implementation detail that we utilize thousandths of a unit integer representation behind the scenes but that is only a small part of the design necessary to execute code in a financial domain effectively.

I do agree with Onorio's comment (paraphrased) that working code speaks louder than theory. Unfortunately I don't know enough about the language / compiler implementation especially regarding guard types to provide such code. My limited (and quite possibly incorrect) understanding seems to suggest that the way I would think of doing it likely requires an upstream change in the Erlang VM. This is the question that I am most curious about regarding feasibility of such a type.

  - - Ben

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/D5A1728D-A38F-4D26-A586-46D4924A5463%40gmail.com.

[Robert Virding]

I am getting in this discussion very late but I think getting a change in the Erlang VM to support this would be very difficult. It would require adding a new, very specific data type and I don't see that happening. Sorry.

Robert

[Wiebe-Marten Wijnja]

Regarding this, how likely would it be for Erlang guard support to be updated to allow e.g. fetching an element from a Map for a specific key? I believe that this would be all we need to allow guard-safe checking of components of structs, so operators like `>` and `<` can be overridden.

If Erlang also starts supporting guard-safe short-circuiting boolean expressions that don't crash when the first element evaluates to something else than true/false, then we can implement many beautiful things in Elixir, such as a guard-safe `+` for structs, as well as for instance a guard-safe integer power operation.

[José Valim]

Regarding this, how likely would it be for Erlang guard support to be updated to allow e.g. fetching an element from a Map for a specific key? I believe that this would be all we need to allow guard-safe checking of components of structs, so operators like `>` and `<` can be overridden.

When maps were proposed, they had a syntax for accessing a map key and I believe it was meant to be allowed in guards. However, they could not agree on the syntax so the functionality was never introduced.

However, Core Erlang does support case statements, so maybe you should be able to pattern match on a map and possibly extract a key-value pair? I am uncertain though, I am not sure if Core would allow a case in guard to introduce a new variable. It is something that has been on my list since ever to figure out but I never did.

 

[Wojtek Mach]

(sorry for digging up old topic)

I would be open though to adding support for number sigils like 13.0d, 13.0f although we should also consider how numbers like rationals would play into that. 

Would the idea by that Elixir would allow doing a 13.0d sigil, and Decimal (whether in core or separate library) implements that sigil?

I think another advantage of including Decimal in core is exactly the sigil. Then the inspect implementation could simply return "13.0d" instead of "#Decimal<13.0>" which would make it copy-pasteable in editor/iex (since it'd be a valid Elixir expression) which makes it extremely convenient for actually working with it, debugging code that uses it etc. Another advantage is using it is in match. 

Currently this isn't allowed:

defmodule DecimalSigil do
  def sigil_X(str, _) do
    Decimal.new(str)
  end
end

import DecimalSigil
~X[3] = Decimal.add(~X[1], ~X[2])

cannot invoke remote function DecimalSigil.sigil_X/2 inside match

# see: https://github.com/ericmj/decimal/issues/38

Looks like custom sigils can't be used in matches, but built-in can.

[José Valim]

You should be able to use inside matches if you make it a macro.

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/e5d1fc3e-14d9-489e-8051-2f7a7d2e913e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[Wojtek Mach]

Yes that worked, thanks.

For the record, at first I tried:

  defmacro sigil_X(str, _) do
    quote do
      Decimal.new(unquote(str))
    end
  end

and it failed but with a similar error: cannot invoke remote function Decimal.new/1 inside match

However, when macro expands to struct (here, just hardcoding it to prove the point)

  defmacro sigil_X(_str, _) do
    quote do
      %Decimal{coef: 1, exp: 0, sign: 1}
    end
  end

it works well.

[José Valim]

Yes, you need to parse the decimal at compile time, something like this:

defmacro sigil_X(decimal, _) do

  Macro.escape Decimal.new decimal

end

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/194126ed-1f85-4fd0-aef2-dfae7e17a082%40googlegroups.com.

[Eric Meadows-Jönsson]

Using Decimal in matches is tricky because two equal values can have different precision so that 42.0d != 42.00d. You can even have cases where 420d != 420d, because one has three digits of precision and the other only two digits.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2Bk7sJNPX%3DYH8jF0JdzfwEVbAeGuyqi2kximbt1KWrfOQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Eric Meadows-Jönsson