Rounding in ledger

[Martin Michlmayr]

TL;DR: There are various rounding-related issues in ledger. This
email (a) shows the problems we've encountered, (b) proposes some
ideas for new directives that might be useful, and c) ends with some
open questions for discussion on this list.

We are planning on addressing some of these issues as part of GSoC
work through Conservancy's NPO Accounting project and will be offering
patches upstream, and therefore would be delighted for more input.

The problem
-----------

Bradley Kuhn and I ran into various problems related to rounding with
ledger. This is a proposal based on conversations with Bradley, his
GSoC Tripun Goel and me. This email shows some problems we've run
into and attempts to provide some ideas for a possible solution which
we can use as a base for discussion.

At the moment, ledger keeps all precision internally and only rounds
when presenting the data to you (based on the precision you use to
input data or on the D or commodity format directives). ledger
keeping around all that precision can lead to problems.

Let's look at some examples:

D 10.00 EUR

2014-01-01 Opening balance
Assets:Investments 10 AAA @ 10.1234 EUR
Assets:Investments 10 BBB @ 10.5692 EUR
Equity:Opening balance

I have 10 AAA worth 10.1234 each and 10 BBB worth 10.5692 each.
ledger will just add everything up: 10*10.1234 + 10*10.5692 = 206.926.
Since the display precision is 2 digits, it will round to 206.93. The
calculation is obviously correct. But it doesn't represent what
happens in reality: in real life, your broker will round to cents:
your 10 AAA are worth 101.23 and not 101.234, and 10 BBB is 105.69 and
not 105.692. Now if you add it up, you get 206.92 (101.23+105.69).
So ledger is off by a cent compared to what you'd get in real life.

Now you can argue: why use @ instead of @@? I could solve this
particular problem like this:

2014-01-01 Opening balance - workaround
Assets:Investments2 10 CCC @@ 101.23 EUR
Assets:Investments2 10 DDD @@ 105.69 EUR
Equity:Opening balance

However, there is often value in specifying a detailed exchange rate
with @.

Let's take a look at another example:

D 1000.00 EUR

2012-01-01 * Test
Assets:Investments 1 AAA @@ 10.00 EUR
Assets:Investments 1 BBB @@ 20.00 EUR
Equity:Opening balance

P 2012-07-01 AAA 10.123 EUR
P 2012-07-01 BBB 20.123 EUR

I have 1 AAA which I bought for 10.00 and 1 BBB I bought for 20.00.
Later, the value of the shares go up to 10.123 and 20.123,
respectively. How much do I have now? ledger will do: 10.123 +
20.123 = 30.246 => 30.25, but in reality, it's 30.24. I cannot have
0.003, so it's really worth: 10.12 and 20.12. (Imagine what would
happen if you sold 1 AAA: how much would you get on your account?
10.12 or 10.123?)

However, it makes sense to have a lot of precision in the exchange
rate / value of the shares. If I had 10 AAA instead, the precision of
the pricedb would matter: 10*10.123 = 101.23. You cannot just drop
digits from the pricedb (10.123=>10.12) because this would change the
value if you have a lot.

Another example for this: a currency exchange rate might be something
like 1.6415 (GBP->USD). Of course, if you only exchange 1 GBP to USD,
you wouldn't get 1.6415 -- you would get 1.64 (assuming no fees). But
if you exchanged a 1000, the precision of the pricedb matters: you get
1641.50.

However, for the actual amounts, it doesn't make sense to keep all the
precision around. Many of us think like programmers and want to keep
things as precise and "correct" as possible. But with accounting, you
have to represent reality: and in real life, you don't have 1.6415 USD
when you exchange one GBP to one USD: you have 1.64 USD.

Here's another example showing that the current behaviour of ledger
will lead to strange results:

D $1000.00

2014-06-15 Test 1
Liabilities:Payable €-19.86 @ $1.3869
Expenses

2014-06-16 Test 2
Liabilities:Payable €-19.86 @ $1.3869
Expenses

In this example, you owe 19.86€ to someone, so you charge it as an
expense and accrue it. The books are in USD. You make the same
expense twice. So let's look at it individually: how much is the
first expense? It's $27.54. And you do this twice. So how much did
you spent in total? Well, you'd probably say 2*$27.54 = $55.08. But
if you ask ledger, it will tell you that the total expenses were
$55.09. Why? Because 19.86*1.3869 is actually $27.543834 and not
$27.54. And if you do 2*$27.543834, you get $55.09.

Just imagine a conversation with your accounting. "So you spent
$27.54 and then you spent $27.54, but your total expenses are $55.09?
Why is that?" "Oh, you know, the expenses weren't really $27.54...
they were actually $27.543834". No, they weren't. In practice, each
expense was $27.54.

One final example: you buy 123 shares for 1.23456 each and later sell
them for 2.345 each. So first of all, how much money do you get?
123*2.345 = 288.435. But do you really get that amount? No, you get
288.44 if you're lucky (or 288.43 if you're not).

And how much capital gain was there? Again, the maths is simple:
123*(2.345-1.23456) = 136.58412

But are you really going to tell your tax authority that you made a
gain of 136.58412? No, you're going to tell them 136.58 (or 136.59).

Here's this example in ledger:

D 1000.00 EUR

2014-06-01 * Opening balance
Assets:Investments 123 XXX {1.23456 EUR} @ 1.23456 EUR
Equity:Opening balances -151.85 EUR

2014-06-22 * Sell some shares
Assets:Investments -123 XXX {1.23456 EUR} @ 2.345 EUR
Assets:Cash 288.44 EUR
Income:Capital gains -136.59 EUR

This balances and works fine. But if you change "D 1000.00 EUR" to
"D 1000.000 EUR", suddenly it won't balance:

Unbalanced remainder is:
-0.001 EUR
Amount to balance against:
288.440 EUR

This shouldn't happen. The display precision shouldn't affect the
calculation: as you can see from these examples, the extra ledger
keeps internally is causing lots of problems in real life.

Now it's great that ledger can keep precision if there's a need, but
in most real-life scenarios, we want to round to 2 digits of
precision.

Possible solution
-----------------

We could have a "commodity" directive called "precision" (or something
similar):

commodity EUR
precision 2

This would tell ledger to round to 2 digits of precision. This
rounding is done internally, i.e. ledger wouldn't keep more detail
than that precision.

But is this enough? I think we also need the ability to specify *how*
rounding is done? In most cases, it's probably just rounded properly,
i.e. 0.005+ is rounded up, everything below is rounded down.

But I can imagine that some companies just truncate: so even if you
should get 1.126, they would give you 1.12 instead of 1.23. (I don't
actually know if this is true, but it might be.)

And maybe there should be one option to make rounding "flexible" or
"permissive": in the example above, ledger would accept both 1.12 and
1.13 for the balancing check, so both of these examples would work:

2014-06-23 * Test 1
A 1 AAA @ $1.126
B $1.13

2014-06-23 * Test 2
A 1 AAA @ $1.126
B $1.12

I guess these are actually two separate questions: how should ledger do
rounding if nothing is specified for B:
2014-06-23 * Test 2
A 1 AAA @ $1.126
B
Is B $1.13 or $1.12?

And the second: what should ledger accept for the balancing check if
the user specifies a value as B (i.e. so either of the example above
is accepted).

Anyway, we could have something like (and possibly another directive
to say how "permissive" balancing should be):

commodity EUR
precision 2
rounding truncate

While precision is probably the same for one currency regardless of
the account, you should be able to specify the type of rounding for
each account (one bank may round properly while another may always
truncate):

account Foo
rounding truncate

account Bar
rounding round

In addition to these commodity/account directives, it might also be useful
to specify rounding on the command line to force it once. I'm not sure if
there's a use case for this.

Open questions
--------------

* Do we actually need to specify the precision? After all, ledger
keeps track of the precision you use in your input data (and by
default uses that for the display precision). Maybe we should just
use that precision as the precision for rounding if rounding is
enabled. In other words, we wouldn't need the "precision" directive
-- "rounding" would be enough. I cannot think of any use cases where
this wouldn't work, but maybe I'm missing something. (See below for
rounding to 1000s)

* If we do implement a "precision" directive and someone says
"precision 2", what would happen if someone then wrote a value like
10.123. Should this be rejected or should it override precision?

* What about currencies like CAN and AUD which have phased out the
cent coin? The smallest coin is 5c. I *assume*, bank and investment
accounts still show single cents, but I'm not sure (Martin Blais, can
you comment?).

* Does rounding only happen after the decimal point or are there
currencies where 600 would be rounded to 1000? (I'm thinking of
currencies where one million or billion is worth $1 or so). The
proposal assumes that rounding only happens after the decimal point
but maybe this assumption is wrong.

* Does anyone know how banks/currency exchanges/investment companies
actually do rounding in real life? Do they round correctly or do some
truncate?

* How does Bitcoin work? I know Bitcoin is divisible. So would they
want no rounding?
commodity BTC
rounding off (or precision 0)
Are there use cases where Bitcoin would be rounded?

* How do other accounting packages handle rounding?

* In the proposal above, I've assumed that users can specify rounding
(e.g. "truncate") as a directive. Another way to implement it would
be to allow users to specify a function that does the rounding: this
would be more powerful, but otoh, it makes things more difficult to
use (and probably to implement). And with a function, how do we know
what should balance?

--
Martin Michlmayr
http://www.cyrius.com/

[Michael Norrish]

On 24/06/14 08:06, Martin Michlmayr wrote:
> * What about currencies like CAN and AUD which have phased out the
> cent coin? The smallest coin is 5c. I *assume*, bank and investment
> accounts still show single cents, but I'm not sure (Martin Blais, can
> you comment?).
>

New Zealand has also eliminated the five cent coin. But there, and in
Australia, everything works with all hundred cents to the dollar. If something
costs $9.99 and you pay for it with a credit card, the card account will be
debited $9.99. The elimination of the small value coins only affects cash
transactions. Of course, the cash-rounding is assessed last. If you buy 10
things each costing $9.99, the cost will be $99.90, however you pay.

Michael

[Martin Blais]

Very interesting discussion. Comments inserted below.

On Mon, Jun 23, 2014 at 6:06 PM, Martin Michlmayr <t...@cyrius.com> wrote:

TL;DR: There are various rounding-related issues in ledger.  This
email (a) shows the problems we've encountered, (b) proposes some
ideas for new directives that might be useful, and c) ends with some
open questions for discussion on this list.

[...]

 

Just imagine a conversation with your accounting.  "So you spent
$27.54 and then you spent $27.54, but your total expenses are $55.09?
Why is that?" "Oh, you know, the expenses weren't really $27.54...
they were actually $27.543834".  No, they weren't.  In practice, each
expense was $27.54.

So far all of the examples you brought up point to an obvious implementation choice: 

  "Rounding should occur at the point of calculating the "balance amount", that is, the amount that is used to balance against the rest of the postings of the transaction."

One final example: you buy 123 shares for 1.23456 each and later sell
them for 2.345 each.  So first of all, how much money do you get?
123*2.345 = 288.435.  But do you really get that amount?  No, you get
288.44 if you're lucky (or 288.43 if you're not).

And how much capital gain was there?  Again, the maths is simple:
123*(2.345-1.23456) = 136.58412

But are you really going to tell your tax authority that you made a
gain of 136.58412?  No, you're going to tell them 136.58 (or 136.59).

Actually... in my experience, any reasonable tax authority would accept you making calculations using that more precise reporting. The more intimately I deal with those guys, the more I realize how relative things can be. Nevertheless...

Here's this example in ledger:

D 1000.00 EUR

2014-06-01 * Opening balance
    Assets:Investments            123 XXX {1.23456 EUR} @ 1.23456 EUR
    Equity:Opening balances          -151.85 EUR

2014-06-22 * Sell some shares
    Assets:Investments            -123 XXX {1.23456 EUR} @ 2.345 EUR
    Assets:Cash                        288.44 EUR
    Income:Capital gains              -136.59 EUR

This balances and works fine.  But if you change "D 1000.00 EUR" to
"D 1000.000 EUR", suddenly it won't balance:

Unbalanced remainder is:
          -0.001 EUR
Amount to balance against:
         288.440 EUR

This shouldn't happen.  The display precision shouldn't affect the
calculation: as you can see from these examples, the extra ledger
keeps internally is causing lots of problems in real life.

I've had this problem too, in a currency trading account, and so far my solution has been to admit a 1.5c delta difference in balance checks. This has worked so far, but I don't like its grittiness. At the moment this is not configurable but I will make it be (via an option).

Now it's great that ledger can keep precision if there's a need, but
in most real-life scenarios, we want to round to 2 digits of
precision.

I think the question should be phrased in terms of what precision you want to represent numbers _internally_. Isn't a worthy goal to attempt to replicate the same precision as the real-world accounts they represent?

Possible solution
-----------------

We could have a "commodity" directive called "precision" (or something
similar):

commodity EUR
    precision 2

This would tell ledger to round to 2 digits of precision.  This
rounding is done internally, i.e. ledger wouldn't keep more detail
than that precision.

But is this enough?  I think we also need the ability to specify *how*
rounding is done?  In most cases, it's probably just rounded properly,
i.e.  0.005+ is rounded up, everything below is rounded down.

I think it gets a bit more complicated than this. 

The IEEE standard for decimal numbers has defined various specific methods for rounding.

See the Python implementation doc for some references to it, or mpdecimal:

https://docs.python.org/2/library/decimal.html

http://www.bytereef.org/mpdecimal/doc/cdecimal/index.html

Lots more info here:

http://speleotrove.com/decimal/

(FWIW in Beancount I use the "cdecimal" Python wrapper to mpdecimal, for performance reasons.)

But I can imagine that some companies just truncate: so even if you
should get 1.126, they would give you 1.12 instead of 1.23.  (I don't
actually know if this is true, but it might be.)

I doubt it; this is a common problem. That bank would have to have _really_ bad developers for this to be the case. In your researching previous work, you might want to dig into banking regulation law, there might be something in there that mentions this. I wonder if you call your bank to ask them how they do it if they'd share... you're a customer after all. Fun experiment.

And maybe there should be one option to make rounding "flexible" or
"permissive": in the example above, ledger would accept both 1.12 and
1.13 for the balancing check, so both of these examples would work:

2014-06-23 * Test 1
    A         1 AAA @ $1.126
    B         $1.13

2014-06-23 * Test 2
    A         1 AAA @ $1.126
    B         $1.12

I guess these are actually two separate questions: how should ledger do
rounding if nothing is specified for B:
2014-06-23 * Test 2
    A         1 AAA @ $1.126
    B
Is B $1.13 or $1.12?

If you state that rounding should occur when computing the balance amount, the answer is unambiguous: $-1.13.

And the second: what should ledger accept for the balancing check if
the user specifies a value as B (i.e. so either of the example above
is accepted).

Only -1.13.

IMHO it is reasonable for an input of -1.126 to trigger an error, that is, if you have a clearly specified rounding rule.

Anyway, we could have something like (and possibly another directive
to say how "permissive" balancing should be):

commodity EUR
    precision 2
    rounding truncate

While precision is probably the same for one currency regardless of
the account, you should be able to specify the type of rounding for
each account (one bank may round properly while another may always
truncate):

account Foo
   rounding truncate

account Bar
   rounding round

I have a use case for this: a real-world currency trading account which rounds its precision at 4 digits instead of the customary two. Amounts are tracked in US dollars. Transfers in and out are using two digits, but internally, four are preserved.  (This is an argument against specifying it by commodity.)

In addition to these commodity/account directives, it might also be useful
to specify rounding on the command line to force it once.  I'm not sure if
there's a use case for this.

Open questions
--------------

* Do we actually need to specify the precision?  After all, ledger
keeps track of the precision you use in your input data (and by
default uses that for the display precision).  Maybe we should just
use that precision as the precision for rounding if rounding is
enabled.  In other words, we wouldn't need the "precision" directive
-- "rounding" would be enough.  I cannot think of any use cases where
this wouldn't work, but maybe I'm missing something. (See below for
rounding to 1000s) 

* If we do implement a "precision" directive and someone says
"precision 2", what would happen if someone then wrote a value like
10.123.  Should this be rejected or should it override precision?

Where is rounding defined to occur?

You have to state a rule IMO.

* What about currencies like CAN and AUD which have phased out the
cent coin?  The smallest coin is 5c.  I *assume*, bank and investment
accounts still show single cents, but I'm not sure (Martin Blais, can
you comment?).

Oh have they finally done that? I haven't noticed (I don't live there anymore).

I think this would only apply to cash transactions anyhow.

* Does rounding only happen after the decimal point or are there
currencies where 600 would be rounded to 1000?  (I'm thinking of
currencies where one million or billion is worth $1 or so).  The
proposal assumes that rounding only happens after the decimal point
but maybe this assumption is wrong.

There's a lot of crazy stuff out there... I forget which it is, but I once heard of an exotic currency that rounds at multiples of 5 in its integer amount that gave some friends working in a financial institution an implementation headache. (Personally I feel that you have to support everything, others may disagree.)

* Does anyone know how banks/currency exchanges/investment companies
actually do rounding in real life?  Do they round correctly or do some
truncate?

* How does Bitcoin work?  I know Bitcoin is divisible.  So would they
want no rounding?
  commodity BTC
      rounding off  (or precision 0)
Are there use cases where Bitcoin would be rounded?

* How do other accounting packages handle rounding?

* In the proposal above, I've assumed that users can specify rounding
(e.g. "truncate") as a directive.  Another way to implement it would
be to allow users to specify a function that does the rounding: this
would be more powerful, but otoh, it makes things more difficult to
use (and probably to implement).  And with a function, how do we know
what should balance?

--
Martin Michlmayr
http://www.cyrius.com/

--

---
You received this message because you are subscribed to the Google Groups "Ledger" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ledger-cli+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stefano Zacchiroli]

Thanks Martin for this proposal, as you know I've been bitten by this
and I'm looking forward to a proper solution.

On Mon, Jun 23, 2014 at 06:06:08PM -0400, Martin Michlmayr wrote:
> One final example: you buy 123 shares for 1.23456 each and later sell
> them for 2.345 each. So first of all, how much money do you get?
> 123*2.345 = 288.435. But do you really get that amount? No, you get
> 288.44 if you're lucky (or 288.43 if you're not).
>
> And how much capital gain was there? Again, the maths is simple:
> 123*(2.345-1.23456) = 136.58412

Eh, thanks for glorifying my IRC example :), I'm glad it has been
useful.

> Possible solution
> -----------------
>
> We could have a "commodity" directive called "precision" (or something
> similar):
>
> commodity EUR
> precision 2

I agree with both Martin-s that this is not enough, at the very least
precision should also be customizable at the account level. However, if
we have precision associated to both commodities and accounts, how would
they interact? Account might contain multiple commodities; would
specifying a precision for an account mean that all commodities
contained therein must have the same precision?

Sounds like we might need the ability to specify:

- a default commodity precision
- a default account precision
(with rules deciding who wins among these two)
- a specific precision for commodity/account pairs

> While precision is probably the same for one currency regardless of
> the account

I don't think that's necessarily the case. Either way, this seems to be
an arbitrary restriction to impose, and I'm unsure what it'd give us in
terms of simplicity of the flexible rounding model / design.

> Open questions
> --------------
>
> * Do we actually need to specify the precision? After all, ledger
> keeps track of the precision you use in your input data (and by
> default uses that for the display precision). Maybe we should just
> use that precision as the precision for rounding if rounding is
> enabled.

This is somewhat tangential to your question, but we do need a way to
specify the *display* precision, more flexible than what we have
now. Your initial example using D works in restricting the output
precision for lot prices, but AFAICT D doesn't work in restricting the
display precision for simple postings, e.g.:

zack@timira:~$ cat test-rounding.ledger
D 10.00 EUR

2014-01-01 Foo
Assets:Cash 10.1234 EUR
Income
zack@timira:~$ ledger -f test-rounding.ledger bal
10.1234 EUR Assets:Cash
-10.1234 EUR Income
--------------------
0

> * If we do implement a "precision" directive and someone says
> "precision 2", what would happen if someone then wrote a value like
> 10.123. Should this be rejected or should it override precision?

This is probably me thinking as a programmer :), but I don't see much
value in rejecting the posting. Just let the user write it down and
override the precision.

> * Does rounding only happen after the decimal point or are there
> currencies where 600 would be rounded to 1000? (I'm thinking of
> currencies where one million or billion is worth $1 or so). The
> proposal assumes that rounding only happens after the decimal point
> but maybe this assumption is wrong.

This would be easy to support, e.g. you can say that "precision -N"
rounds to digits *before* the decimal, rather than after it. (But this
won't solve weirder rounding issues.)

Cheers.
--
Stefano Zacchiroli . . . . . . . za...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »

[Simon Michael]

Random notes:

I feel rounding occurs or is used at these times:

1. when calculating a missing posting amount

2. when checking that postings balance

3. when converting from one commodity to another (due to -B/-V/-X or
when balancing a mixed-commodity transaction)

4. when rendering an amount for output

I'm thinking display precision could be set by, in order of precedence:

1. a command-line --precision flag (for simple overriding)

2. the precision declared for the commodity in the account,
if displaying an account balance (not eg the final balance total)

3. the precision declared for the account, if displaying an account balance

4. the precision declared for the commodity

5. the maximum precision encountered for that commodity in posting
amounts explicitly recorded in the journal (but not price amounts)

And display precision can be:

- -N digits (round to tens, hundreds or whatever)
- 0 (no digits)
- N digits
- max (show all available precision)

[Craig Earls]

From reading the code exact arithmetic is used for 1-3 and rounding only occurs on display. 

--

--- You received this message because you are subscribed to the Google Groups "Ledger" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ledger-cli+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Craig, Corona De Tucson, AZ
enderw88.wordpress.com

[tripun goel]

I have prepared a patch which adds precision feature to commodity directive. I have also attached some tests. It works as follows

D 1000.00 EUR
commodity EUR
  precision 2
  default

2012-01-01 *
   A                        2 AAA @ 10.121 EUR
   A                        1 BBB @ 20.121 EUR
   C

Mention precision followed by an integer under commodity directive. This integer is the number of places after decimal to keep. +1 means one place after decimal and -1 means one place before decimal. '0' means round to no decimal places. It does not affect display precision. So if display precision is 3 then amount will be displayed to 3 decimal places however they are rounded to 2 decimal places if you have set precision to 2. 

Example result 
           40.36 EUR  A
          -40.36 EUR  C
--------------------
                   0

Known issues:
1. Cannot set rounding mode or truncate.
2. Display precision is not changed.
3. Range of integer is not checked. 

Please review. Suggest new tests.

[Martin Michlmayr]

* Martin Blais <bl...@furius.ca> [2014-06-23 22:46]:

> So far all of the examples you brought up point to an obvious
> implementation choice:
>
> "Rounding should occur at the point of calculating the "balance amount",
> that is, the amount that is used to balance against the rest of the
> postings of the transaction."

Right, I believe that's correct, as long as it's done for each
transaction and not just once at the end when doing a balance.

> > in most real-life scenarios, we want to round to 2 digits of
> > precision.
> I think the question should be phrased in terms of what precision you want
> to represent numbers _internally_. Isn't a worthy goal to attempt to
> replicate the same precision as the real-world accounts they represent?

Right, I definitely agree. I mentioned 2 digits above since this will
be the precision of most real-world accounts, but I agree that the
precision from the real-world accounts should be reflected internally.
At the moment, ledger internally stores data with all precision -- I
want an option to limit precision stored internally to the desired
precision (according to what the real-world account uses).

> But I can imagine that some companies just truncate: so even if you
> > should get 1.126, they would give you 1.12 instead of 1.23. (I don't
> > actually know if this is true, but it might be.)
>
> I doubt it; this is a common problem. That bank would have to have _really_
> bad developers for this to be the case. In your researching previous work,
> you might want to dig into banking regulation law, there might be something
> in there that mentions this. I wonder if you call your bank to ask them how
> they do it if they'd share... you're a customer after all. Fun experiment.

Yeah, would be interesting to know how this is handled in real life...
does anyone know?

> I have a use case for this: a real-world currency trading account which
> rounds its precision at 4 digits instead of the customary two. Amounts are
> tracked in US dollars. Transfers in and out are using two digits, but
> internally, four are preserved. (This is an argument against specifying it
> by commodity.)

Ok, good to know. Thanks!

I think specifying it by commodity still makes sense, but an account
directive should be able to override it.

> > currencies where 600 would be rounded to 1000? (I'm thinking of
> > currencies where one million or billion is worth $1 or so). The
> > proposal assumes that rounding only happens after the decimal point
> > but maybe this assumption is wrong.
>
> There's a lot of crazy stuff out there... I forget which it is, but I once
> heard of an exotic currency that rounds at multiples of 5 in its integer
> amount that gave some friends working in a financial institution an
> implementation headache. (Personally I feel that you have to support
> everything, others may disagree.)

Interesting. ledger is very flexible so it would be nice to be a
flexible as possible. At the same time, the common cases should be as
easy possible.

[Martin Michlmayr]

* Stefano Zacchiroli <za...@upsilon.cc> [2014-06-24 12:07]:

> I agree with both Martin-s that this is not enough, at the very least
> precision should also be customizable at the account level. However, if
> we have precision associated to both commodities and accounts, how would
> they interact? Account might contain multiple commodities; would
> specifying a precision for an account mean that all commodities
> contained therein must have the same precision?
>
> Sounds like we might need the ability to specify:
>
> - a default commodity precision
> - a default account precision
> (with rules deciding who wins among these two)
> - a specific precision for commodity/account pairs

Maybe something like:

commodity EUR
precision 2

account Foo
commodity EUR
precision 2
commodity GBP
precision 3

I think this would meet all requirements.

> > While precision is probably the same for one currency regardless of
> > the account
>
> I don't think that's necessarily the case. Either way, this seems to be

Yes, Martin already gave an example where it's not true.

> This is somewhat tangential to your question, but we do need a way to
> specify the *display* precision, more flexible than what we have
> now. Your initial example using D works in restricting the output
> precision for lot prices, but AFAICT D doesn't work in restricting the
> display precision for simple postings, e.g.:
>
> zack@timira:~$ cat test-rounding.ledger
> D 10.00 EUR
>
> 2014-01-01 Foo
> Assets:Cash 10.1234 EUR
> Income
> zack@timira:~$ ledger -f test-rounding.ledger bal
> 10.1234 EUR Assets:Cash
> -10.1234 EUR Income
> --------------------
> 0

I think the problem you're trying to raise is that the display
precision can only extend the display precision, not reduce it.

[Martin Blais]

Someone ought to summarize this thread into a well-organized design doc that arranges all the issues nicely, something like Python's PEP documents.

[tripun goel]

Attached a new patch. The link to new patch is
https://gitorious.org/ledger/npo-ledger-cli/raw/c0135638fce02aa3e74e6fe7d180fb65c7c64d0b:rounding/new_precision.patch

I have reimplemented precision feature to solve a minor issue,  posting without any annotated price were not being rounded. So a simple example like this were generating unbalanced amount error.

D 1000.00 EUR
commodity EUR
  precision 2
  default

2012-01-01 *
   A                        3.69 EUR
   C                        -269 Boots @@ 3.69 EUR
 

But this new patch solve that problem too. The patch passes current test suite. Few other tests from this post examples are
https://gitorious.org/ledger/npo-ledger-cli/raw/c0135638fce02aa3e74e6fe7d180fb65c7c64d0b:rounding/bug_reports/dir-commodity-precision.test
https://gitorious.org/ledger/npo-ledger-cli/raw/c0135638fce02aa3e74e6fe7d180fb65c7c64d0b:rounding/bug_reports/dir-commodity-precision-2.test
https://gitorious.org/ledger/npo-ledger-cli/raw/c0135638fce02aa3e74e6fe7d180fb65c7c64d0b:rounding/bug_reports/dir-commodity-precision-3.test
https://gitorious.org/ledger/npo-ledger-cli/raw/c0135638fce02aa3e74e6fe7d180fb65c7c64d0b:rounding/bug_reports/dir-commodity-precision-4.test

 

On Monday, 30 June 2014 16:28:58 UTC+5:30, tripun goel wrote:

I have prepared a patch which adds precision feature to commodity directive. I have also attached some tests. It works as follows

D 1000.00 EUR
commodity EUR
  precision 2
  default

2012-01-01 *
   A                        2 AAA @ 10.122 EUR
   A                        1 BBB @ 20.123 EUR
   C

Mention precision followed by an integer under commodity directive. This integer is the number of places after decimal to keep. +1 means one place after decimal and -1 means one place before decimal. '0' means round to no decimal places. It does not affect display precision. So if display precision is 3 then amount will be displayed to 3 decimal places however they are rounded to 2 decimal places if you have set precision to 2. 

[tripun goel]

Fixed another issue , here is the link to the new patch
https://gitorious.org/ledger/npo-ledger-cli/raw/45b8d981c26a009facdd502504f94dc7af216d93:rounding/final_new_precision.patch
Regards
Tripun Goel

[Nathan Grigg]

Sorry to bring up an old topic. I followed this with some interest back when we discussed it, but my own thoughts hadn’t yet crystalized, so I was silent. Now I have thoughts to share.

Martin Blais:

I think the question should be phrased in terms of what precision you want to represent numbers _internally_. Isn't a worthy goal to attempt to replicate the same precision as the real-world accounts they represent?

I agree. To me, display precision is much less important than the internal representation.

Martin again:

Rounding should occur at the point of calculating the "balance amount", that is, the amount that is used to balance against the rest of the postings of the transaction.

I agree again, but disagree with how this is implemented in both ledger and beancount (I am not familiar with hledger). In particular, I think that the only way to avoid strange situations is for every transaction to balance exactly, as in A+B=0 to the full precision of the internal representation.

Let’s ignore the concept of elided values and start by thinking only about the whether a transaction balances.

There are two cases.

Case 1: Single currency. Postings should balance exactly.

2014-01-01 * transfer

Assets:Bank1    $1.00

        Assets:Bank2   $-1.001

It seems pretty obvious that this should trigger an error. Ledger gets this right because it requires all transactions to balance to the maximum precision specified in the document (not quite, we’ll get to this later). Beancount, with its fixed tolerance, lets this through (Even $-1.01 would pass). Even if the tolerance is exposed as an option, you give the user the opportunity to screw things up.

Case 2: Multiple currencies.

2014-01-01 * buy AAA
Assets:Investments  2 AAA @ 10.1234 EUR
Assets:Bank  -20.25 EUR

The 2 AAA and the -20.25 EUR are actual amounts. The quoted rate may have been 10.1234, but the truth is, you bought at 10.125 EUR (sucker!)

Note that this need to adjust price due to rounding does not arise solely from overprecise pricing, but also when you purchase fractional shares.

2014-01-01 * buy VPMCX

Assets:Investments  9.66 VPMCX @ 103.52 USD

Assets:Bank  1000.00 USD

Again, the $1000 and the 9.66 shares are exact quantities. They do not represent something that has been rounded. You do not have an extra, invisible 0.0032 USD lying around. It is the $103.52 that is wrong.

So one solution is to simply record the share price you paid instead of the share price you were quoted. This is the same as Martin Machlmayr’s suggestion to use ‘@@' instead of ‘@'. But as Martin also pointed out, there is value in using ‘@‘, namely, your cost basis can match the bank’s report, the pricedb gets an accurate market price, and you get an extra check that you haven’t mistyped a number.

A different solution is for the program to replace the cost basis that you specify with the true cost, as long as it is close enough. But this doesn’t solve the problem that you might want to attach the bank-reported cost to the posting, or see it in reports. 

One final solution, and what I have settled on for now, is a rounding account. I insert one manually (with elided amount), but the program could easily do this for you. It sweeps away whatever is left over, and has accumulated 0.12 USD (in expenses) for me so far this year. The alternative is that my double-entry accounting ledger is out of balance by 12 cents per year. (Maybe not a big deal, but it bugs me.)

The point is, no matter how you decide to round the amounts, you have to account for the rounding in some way or accept the fact that your system is out of balance.

Note that either of these automated solutions could probably be implemented using beancount plugins.

Weird things happening

If you don’t require each transaction to balance, you easily drift away from overall balance. In Ledger:

D 1.00 USD

2014-01-01 Test

    assets  1 CAD @ 1.005 USD

    liabilities

2014-01-02 Test

    assets  1 CAD @ 1.005 USD

    liabilities

% ledger bal liabilities

2014-01-01 Test                 liabilities               -1.00 USD    -1.00 USD

2014-01-02 Test                 liabilities               -1.00 USD    -2.01 USD

Here is another:

2014-01-01 Test

    assets  1 CAD @ 1.005 USD

    liabilities  -1.00 USD

2014-01-02 Test

    assets  1 CAD @ 1.005 USD

    liabilities  -1.00 USD

2014-01-03 Test

    assets  1 CAD @ 1.005 USD

    liabilities  -1.005 USD

% ledger bal -B

           3.015 USD  assets

          -3.005 USD  liabilities

--------------------

           0.010 USD

This second example is curious because if you put the third transaction at the beginning, the other two now fail to balance, because the precision becomes 3 decimal points before they are read, whereas before the precision switched from 2 to 3 partway through.

Eliding postings

Once you have what it means to balance figured out, it is easy to complete a transaction with an elided posting. Simply choose the value that minimizes the error. (And then either roll the error into an adjusted price or add a rounding transaction.) Obviously, to do this, you would need rules about precision, per currency and possibly per account. If you really want to, you can have custom rounding rules. 

In my experience, though, you will often still get the number wrong, because at the end of the day, the bank will do what it does for its own reasons. (In my 401k, for example, I contribute a fixed $X to buy Y fractional shares at price Z, but X - Y*Z has been as big as 2.5 cents because strictly speaking the bank is setting Y = round(X/Z, 3), even though I would never think of it that way, or record it that way in my ledger.)

Nathan

[Mike Charlton]

Thank you for posting this.  Your second example explains weirdness that I was seeing but couldn't comprehend.

So to put it succinctly, ledger balances as long as the rounded amounts balance.  Perhaps I don't understand all of the nuances, but I have to say that I don't like that.  If a transaction is off by a fraction of a cent, then that *does not balance* for me!

I am in the same boat as you where my bank reports one thing, but is actually stealing fractions of currency.  I haven't decided whether or not having an elided expense account or simply reporting the exchange rate that was actually used is better.  One of the things that makes me think an elided expense account is better is that the price history seems to be dependent upon transaction order.  I am currently storing the price db in a separate file, so I'm never sure whether ledger will use the price that something traded at, or the price I put in the price db.  For various reasons, it is important to me that it is the latter.

Anyway add me as a vote for balancing at full precision.  I only want rounding to occur in reporting. 

[Martin Blais]

On Mon, Oct 6, 2014 at 5:20 PM, Mike Charlton <mike...@gmail.com> wrote:

Thank you for posting this.  Your second example explains weirdness that I was seeing but couldn't comprehend.

So to put it succinctly, ledger balances as long as the rounded amounts balance.  Perhaps I don't understand all of the nuances, but I have to say that I don't like that.  If a transaction is off by a fraction of a cent, then that *does not balance* for me!

Then you will never be able to balance anything that has a price or a cost. There has to be rounding somewhere.

Or you'll have to add a "catch-the-rest" leg to any transaction that has a price conversion or a cost, which is an ugly solution.

I am in the same boat as you where my bank reports one thing, but is actually stealing fractions of currency.

What makes you think they're stealing? Rounding is not stealing. They can round up or down, and probably. And further, I believe there is a practice whereby rounding can be chosen to be random if tie-breaking is necessary. The purpose is to end up at the EOY with as little rounding error as possible.

 

  I haven't decided whether or not having an elided expense account or simply reporting the exchange rate that was actually used is better. 

Note that an elided expense account effectively disables any balance checks.

This is not a solution.

We need to find a solution to the rounding problem.

(Nathan: nice discussion, BTW. It gave me some ideas, will elaborate later.)

 

One of the things that makes me think an elided expense account is better is that the price history seems to be dependent upon transaction order.  I am currently storing the price db in a separate file, so I'm never sure whether ledger will use the price that something traded at, or the price I put in the price db.  For various reasons, it is important to me that it is the latter.

The price should never matter; you should worry about the cost.

In both Ledger (AFAIK) and Beancount the price is only used to fill in a value in the price database.

Anyway add me as a vote for balancing at full precision.  I only want rounding to occur in reporting. 

It's not a solution.  If it were, this would already be solved.

The best approach is to replicate what the institutions do.

AFAIK what they do is round both the amounts and the prices (your capital gain is probably also calculated from a rounded representation, possibly with 4 or 5 digits instead of 2, or even just 2). This would mean it's impossible to balance precisely.

[Simon Michael]

On 10/6/14 1:31 PM, Nathan Grigg wrote:
> % ledger bal liabilities
> 2014-01-01 Test liabilities -1.00 USD
> -1.00 USD
> 2014-01-02 Test liabilities -1.00 USD
> -2.01 USD

Correction: ledger reg liabilities.

[Nathan Grigg]

On Oct 6, 2014, at 3:58 PM, Martin Blais <bl...@furius.ca> wrote:

On Mon, Oct 6, 2014 at 5:20 PM, Mike Charlton <mike...@gmail.com> wrote:

Thank you for posting this.  Your second example explains weirdness that I was seeing but couldn't comprehend.

So to put it succinctly, ledger balances as long as the rounded amounts balance.  Perhaps I don't understand all of the nuances, but I have to say that I don't like that.  If a transaction is off by a fraction of a cent, then that *does not balance* for me!

Then you will never be able to balance anything that has a price or a cost. There has to be rounding somewhere.

Strictly speaking, the price-adjustment option would be possible in ledger, which uses rational numbers as internal representation, allowing for exact balancing. But ledger is ill-equipped to solve this problem for other reasons, for example, its lack of lot matching.

Beancount could solve this by attaching a total cost to each inventory, in addition to a per-share cost (with the stipulation that | total cost - number * share cost | < tolerance). You still have to face the fact, however, that if you have three shares worth $0.333 each, with a total cost to you of $1.00, and you sell these one at a time at the same price but you receive $0.33 each, even though their price has not changed, there is a penny that has disappeared. Is that an expense due to rounding or a capital loss? Again, it doesn’t really matter, but I feel like I should pick one.

 

  I haven't decided whether or not having an elided expense account or simply reporting the exchange rate that was actually used is better. 

Note that an elided expense account effectively disables any balance checks.

This is not a solution.

If it is built into the system, the elided expense account can also balance check. That is, insert a ‘Expenses:Rounding’ transaction, but only if the difference is less than some tolerance (and only if there is a price/cost)

The question is whether Expenses:Rounding is better than having the ability to run a report listing all transactions which don’t quite balance, so you feel good about your overall balance being out of whack.

Simon Michael: 

Correction: ledger reg liabilities.

Yes, thanks.

Nathan

[Martin Blais]

On Mon, Oct 6, 2014 at 8:57 PM, Nathan Grigg <nat...@nathangrigg.net> wrote:

On Oct 6, 2014, at 3:58 PM, Martin Blais <bl...@furius.ca> wrote:

On Mon, Oct 6, 2014 at 5:20 PM, Mike Charlton <mike...@gmail.com> wrote:

Thank you for posting this.  Your second example explains weirdness that I was seeing but couldn't comprehend.

So to put it succinctly, ledger balances as long as the rounded amounts balance.  Perhaps I don't understand all of the nuances, but I have to say that I don't like that.  If a transaction is off by a fraction of a cent, then that *does not balance* for me!

Then you will never be able to balance anything that has a price or a cost. There has to be rounding somewhere.

Strictly speaking, the price-adjustment option would be possible in ledger, which uses rational numbers as internal representation, allowing for exact balancing. But ledger is ill-equipped to solve this problem for other reasons, for example, its lack of lot matching.

No this still doesn't work, there are many problems with it:

- Precise representation requires that one of the numbers be left for the system to interpolate (you likely cannot even input the precise number by typing it, too many digits may be required). If you specify both cost and price, the mere fact that you're _typing_ decimal numbers implies a limited representation.

- Secondly, precise representation is also not what you want: let's say we decide to make the cost (or price) the number slot we'll choose to represent exactly.... when it's time to calculate the capital gains you will be using that oddball precise cost (price) number to compute the dollar amount for the cost basis of the shares sold (which may not be the full original lot). The resulting number may include more precision than you want to have in order to compute the capital gains. The bank will compute... and the government will expect you to compute... using the recorded rounded price, not this weird "exact" price. Besides, the bank itself surely does not represent the cost with in a rational representation, so your calculation is _guaranteed_ to differ from the bank's own.

This is why I've favored a small tolerance in balancing. Purists will think it's wrong, but if you think about the implications of using a precise representation and the associated complications it creates, it's actually better because it more closely matches what the banks/institutions do and does not lead to any of those problems. 

Finally, in practice, everyone uses the rounded amounts and ignores the rounding errors, and I feel that that's what we should do too: replicate what is used in practice, replicate the real world.

(I do think that we can do better than the method I've implemented in Beancount, but I'm not sure yet what it is. I'm however convinced that using exact numbers is not the solution; inferring better or tighter rounding automatically might be a better direction.)

Beancount could solve this by attaching a total cost to each inventory, in addition to a per-share cost (with the stipulation that | total cost - number * share cost | < tolerance).

This method breaks as soon as you split your lots (e.g. through the sale of a partial lot).

(BTW maintaining the cost basis for the aggregate position is only elegant for an account maintained at average cost booking, where that's what one effectively does on every lot change. That minimizes numerical error.)

 

You still have to face the fact, however, that if you have three shares worth $0.333 each, with a total cost to you of $1.00, and you sell these one at a time at the same price but you receive $0.33 each, even though their price has not changed, there is a penny that has disappeared. Is that an expense due to rounding or a capital loss? Again, it doesn’t really matter, but I feel like I should pick one.

You're supporting my point... it doesn't solve the problem. With my method you get to choose: you put the value you want on the cash side of the sale and as long as the difference is within tolerance (1.5c at the moment) you can replicate what your bank does.  If you want 0.33, 0.33 and 0.34, you can do that and you actually end up with a better result than if you had used a precise representation.

 

 

  I haven't decided whether or not having an elided expense account or simply reporting the exchange rate that was actually used is better. 

Note that an elided expense account effectively disables any balance checks.

This is not a solution.

If it is built into the system, the elided expense account can also balance check. That is, insert a ‘Expenses:Rounding’ transaction, but only if the difference is less than some tolerance (and only if there is a price/cost)

The question is whether Expenses:Rounding is better than having the ability to run a report listing all transactions which don’t quite balance, so you feel good about your overall balance being out of whack.

I'll paraphrase to make sure I understand what you're proposing: you're proposing that some special account be defined and automatically inserted on transactions that don't balance exactly but _only_ in the cases where the difference is very small, say, below some tolerance (e.g., 1.5c like in Beancount), in order to silently swallow the rounding error and now we can force the balancing amounts to sum up exactly, knowing the rounding errors go somewhere. This is supposed to satisfy the user's conviction that transactions balance exactly... by padding them silently when they don't. 

Okay, it's definitely implementable and I'm not against the idea, but what's the point of doing this?  Can we somehow use this rounding account for a particular purpose?  How does it help the user with anything?  Does it make our data entry and calculation work easier?  If we're going to lose something to rounding and we won't be able to use it for anything, why even bother?

 

Simon Michael: 

Correction: ledger reg liabilities.

Yes, thanks.

Nathan

--

[Martin Blais]

Sorry I didn't mean 1.5c in my email, rather 0.5c. 

Beancount's tolerance for balance amounts is 0.005 (not configurable at the moment, I can easily make this configurable).

[Martin Blais]

Very interesting email, thanks for the discussion Nathan! (comments below)

On Mon, Oct 6, 2014 at 3:31 PM, Nathan Grigg <nat...@nathangrigg.net> wrote:

Sorry to bring up an old topic. I followed this with some interest back when we discussed it, but my own thoughts hadn’t yet crystalized, so I was silent. Now I have thoughts to share.

Martin Blais:

I think the question should be phrased in terms of what precision you want to represent numbers _internally_. Isn't a worthy goal to attempt to replicate the same precision as the real-world accounts they represent?

I agree. To me, display precision is much less important than the internal representation.

I beg to differ: to me, internal representation should reflect that is actually in use by the institution whose accounts we're replicating.

Martin again:

Rounding should occur at the point of calculating the "balance amount", that is, the amount that is used to balance against the rest of the postings of the transaction.

I agree again, but disagree with how this is implemented in both ledger and beancount (I am not familiar with hledger). In particular, I think that the only way to avoid strange situations is for every transaction to balance exactly, as in A+B=0 to the full precision of the internal representation.

Let’s ignore the concept of elided values and start by thinking only about the whether a transaction balances.

There are two cases.

Case 1: Single currency. Postings should balance exactly.

2014-01-01 * transfer

Assets:Bank1    $1.00

        Assets:Bank2   $-1.001

It seems pretty obvious that this should trigger an error. Ledger gets this right because it requires all transactions to balance to the maximum precision specified in the document (not quite, we’ll get to this later). Beancount, with its fixed tolerance, lets this through (Even $-1.01 would pass). Even if the tolerance is exposed as an option, you give the user the opportunity to screw things up.

I like the idea that the tolerance can be inferred automatically from the smallest digit in the input instead of using a fixed value. I should implement that in Beancount as well, there's no reason it couldn't do it.This should be a very easy change to make.

At the moment Beancount uses "0.005" regardless of unit... this is a known pending issue (I don't have any Japanese users yet I suppose). I've been thinking that a per-commodity tolerance could be selected instead, i.e., a map of commodity to tolerance value, but while better, this is also unsatisfying.

I wonder if inferring like Ledger works in all cases though; consider this case:

    Assets:Fidelity:Fund        11.27534 FUND {1.2357 USD}

    Assets:Fidelity:Cash       -13.93 USD

The precise cash amount is 13.932937638 USD.

How many digits should this use?

The debited cash is 13.93, this would be the _actual_ amount debited by the fund manager, so it can't be anything else.

In this example, I would argue that you would want the _minimum_ number of digits to be used.

(But certainly not the minimum if you're using an integer number of shares...)

If we can come up with a rule that works well in all cases I'll implement it.

 

Case 2: Multiple currencies.

2014-01-01 * buy AAA
Assets:Investments  2 AAA @ 10.1234 EUR
Assets:Bank  -20.25 EUR

The 2 AAA and the -20.25 EUR are actual amounts. The quoted rate may have been 10.1234, but the truth is, you bought at 10.125 EUR (sucker!)

Note that this need to adjust price due to rounding does not arise solely from overprecise pricing, but also when you purchase fractional shares.

2014-01-01 * buy VPMCX

Assets:Investments  9.66 VPMCX @ 103.52 USD

Assets:Bank  1000.00 USD

Again, the $1000 and the 9.66 shares are exact quantities. They do not represent something that has been rounded. You do not have an extra, invisible 0.0032 USD lying around. It is the $103.52 that is wrong.

But is it wrong, really?  Is the cost basis calculated with 103.52 USD or 103.519668737 USD?  

Correct or not, I think the bank and govt assume a cost basis calculated using 103.52 USD, so that's the correct one.

Also, when you calculate your capital gains later, which of the prices should you use?

Again, I think 103.52 is what you're supposed to use.

So one solution is to simply record the share price you paid instead of the share price you were quoted. This is the same as Martin Machlmayr’s suggestion to use ‘@@' instead of ‘@'. But as Martin also pointed out, there is value in using ‘@‘, namely, your cost basis can match the bank’s report, the pricedb gets an accurate market price, and you get an extra check that you haven’t mistyped a number.

A different solution is for the program to replace the cost basis that you specify with the true cost, as long as it is close enough. But this doesn’t solve the problem that you might want to attach the bank-reported cost to the posting, or see it in reports. 

Both of these worry me, as per my other email, because that propagates cost basis that is unexpectedly exact. I'd rather use the rounded, inexactly calculated cost basis.

One final solution, and what I have settled on for now, is a rounding account. I insert one manually (with elided amount), but the program could easily do this for you. It sweeps away whatever is left over, and has accumulated 0.12 USD (in expenses) for me so far this year. The alternative is that my double-entry accounting ledger is out of balance by 12 cents per year. (Maybe not a big deal, but it bugs me.)

The point is, no matter how you decide to round the amounts, you have to account for the rounding in some way or accept the fact that your system is out of balance.

Note that either of these automated solutions could probably be implemented using beancount plugins.

Yes, you could. 

However, using a plugin you would have to compute each transaction's balance amounts sum twice, which is somewhat expensive, so if this is a great solution, I would consider adding it in the interpolation routine itself to avoid the cost and use the one already computed. I'm not convinced this should always be done though. (See at the end of this message.)

Now that... is something that would really bother me. If anything should be inferred from the precision of the numbers in the input, its effects should always be only _local_ IMO. 

Eliding postings

Once you have what it means to balance figured out, it is easy to complete a transaction with an elided posting. Simply choose the value that minimizes the error. (And then either roll the error into an adjusted price or add a rounding transaction.) Obviously, to do this, you would need rules about precision, per currency and possibly per account. If you really want to, you can have custom rounding rules. 

In my experience, though, you will often still get the number wrong, because at the end of the day, the bank will do what it does for its own reasons. (In my 401k, for example, I contribute a fixed $X to buy Y fractional shares at price Z, but X - Y*Z has been as big as 2.5 cents because strictly speaking the bank is setting Y = round(X/Z, 3), even though I would never think of it that way, or record it that way in my ledger.)

Alright, so here's what I propose:

- I could add an option for the user to insert the name of a rounding account.

- This option would be empty by default, and the current behaviour would not change.

- However, if you set an account for it, all transactions with an inexact balance will receive the balance amount (and perhaps have a new leg inserted on them automatically).

Would that be a reasonable compromise?

With no account, you get 0.005 looseness (or whatever this becomes if inference is implemented).

With an account, you get precise balances throughout, but no manual input is required.

Thoughts?

Thanks for an illuminating discussion! :)

[Nathan Grigg]

On Oct 6, 2014, at 8:30 PM, Martin Blais <bl...@furius.ca> wrote:

I like the idea that the tolerance can be inferred automatically from the smallest digit in the input instead of using a fixed value. I should implement that in Beancount as well, there's no reason it couldn't do it.This should be a very easy change to make.

At the moment Beancount uses "0.005" regardless of unit... this is a known pending issue (I don't have any Japanese users yet I suppose). I've been thinking that a per-commodity tolerance could be selected instead, i.e., a map of commodity to tolerance value, but while better, this is also unsatisfying.

I wonder if inferring like Ledger works in all cases though; consider this case:

    Assets:Fidelity:Fund        11.27534 FUND {1.2357 USD}

    Assets:Fidelity:Cash       -13.93 USD

The precise cash amount is 13.932937638 USD.

How many digits should this use?

Ledger would infer 2 digits of precision for USD and 5 for FUND. The price portion does not affect precision. Ledger also has a D directive to specify the minimum precision per currency.

The debited cash is 13.93, this would be the _actual_ amount debited by the fund manager, so it can't be anything else.

In this example, I would argue that you would want the _minimum_ number of digits to be used.

(But certainly not the minimum if you're using an integer number of shares...)

If we can come up with a rule that works well in all cases I'll implement it.

To be honest, I would prefer to explicitly specify precision, although this makes it harder to work correctly "out of the box". 

Case 2: Multiple currencies.

2014-01-01 * buy AAA
Assets:Investments  2 AAA @ 10.1234 EUR
Assets:Bank  -20.25 EUR

The 2 AAA and the -20.25 EUR are actual amounts. The quoted rate may have been 10.1234, but the truth is, you bought at 10.125 EUR (sucker!)

Note that this need to adjust price due to rounding does not arise solely from overprecise pricing, but also when you purchase fractional shares.

2014-01-01 * buy VPMCX

Assets:Investments  9.66 VPMCX @ 103.52 USD

Assets:Bank  1000.00 USD

Again, the $1000 and the 9.66 shares are exact quantities. They do not represent something that has been rounded. You do not have an extra, invisible 0.0032 USD lying around. It is the $103.52 that is wrong.

But is it wrong, really?  Is the cost basis calculated with 103.52 USD or 103.519668737 USD?  

Correct or not, I think the bank and govt assume a cost basis calculated using 103.52 USD, so that's the correct one.

Also, when you calculate your capital gains later, which of the prices should you use?

Again, I think 103.52 is what you're supposed to use.

I agree that 103.52 is what the bank will use and what you are "supposed" to use. 

So one solution is to simply record the share price you paid instead of the share price you were quoted. This is the same as Martin Machlmayr’s suggestion to use ‘@@' instead of ‘@'. But as Martin also pointed out, there is value in using ‘@‘, namely, your cost basis can match the bank’s report, the pricedb gets an accurate market price, and you get an extra check that you haven’t mistyped a number.

A different solution is for the program to replace the cost basis that you specify with the true cost, as long as it is close enough. But this doesn’t solve the problem that you might want to attach the bank-reported cost to the posting, or see it in reports. 

Both of these worry me, as per my other email, because that propagates cost basis that is unexpectedly exact. I'd rather use the rounded, inexactly calculated cost basis.

I'm pretty sure I agree. 

This is my favorite solution.

Nathan 

[Martin Blais]

On Mon, Oct 6, 2014 at 10:57 PM, Nathan Grigg <nat...@nathangrigg.net> wrote:

On Oct 6, 2014, at 8:30 PM, Martin Blais <bl...@furius.ca> wrote:

I like the idea that the tolerance can be inferred automatically from the smallest digit in the input instead of using a fixed value. I should implement that in Beancount as well, there's no reason it couldn't do it.This should be a very easy change to make.

At the moment Beancount uses "0.005" regardless of unit... this is a known pending issue (I don't have any Japanese users yet I suppose). I've been thinking that a per-commodity tolerance could be selected instead, i.e., a map of commodity to tolerance value, but while better, this is also unsatisfying.

I wonder if inferring like Ledger works in all cases though; consider this case:

    Assets:Fidelity:Fund        11.27534 FUND {1.2357 USD}

    Assets:Fidelity:Cash       -13.93 USD

The precise cash amount is 13.932937638 USD.

How many digits should this use?

Ledger would infer 2 digits of precision for USD and 5 for FUND. The price portion does not affect precision. Ledger also has a D directive to specify the minimum precision per currency.

I'm not sure what that means... there is no need for any precision on units of FUND, the balance amount of that leg is in USD calculated as 11.27534 x 1.2357. In USD. The balance residual is 11.27534 x 1.2357 + -13.93. The only precision that matters is that of USD.

Does it use the precision inferred on units of FUND in any way?

If so, how?

Which brings up an interesting option for inferring the precision: the maximum number of digits from all legs without cost/price conversions. And this should be calculated for each group of postings whose "balance amounts" are in a common commodity. That sounds right to me. 

 

The debited cash is 13.93, this would be the _actual_ amount debited by the fund manager, so it can't be anything else.

In this example, I would argue that you would want the _minimum_ number of digits to be used.

(But certainly not the minimum if you're using an integer number of shares...)

If we can come up with a rule that works well in all cases I'll implement it.

To be honest, I would prefer to explicitly specify precision, although this makes it harder to work correctly "out of the box". 

Why? Because you fear not having a minimum will create pathological cases where a user will insert a balance against an integer number of dollars and make a mistake on the cost that will go undetected?  

2014-09-09 * 

   Assets:Invesments     ...

   Assets:Cash               -1000 USD

Just curious. What cases do you see as problematic?

I've put it in the TODO.

You've convinced me to tackle this before tagging the next release - I'm curious to compute my own residuals now actually - so I'll try to work on this soon and include that change in the next release.

[Nathan Grigg]

On Oct 6, 2014, at 9:46 PM, Martin Blais <bl...@furius.ca> wrote:

On Mon, Oct 6, 2014 at 10:57 PM, Nathan Grigg <nat...@nathangrigg.net> wrote:

On Oct 6, 2014, at 8:30 PM, Martin Blais <bl...@furius.ca> wrote:

I like the idea that the tolerance can be inferred automatically from the smallest digit in the input instead of using a fixed value. I should implement that in Beancount as well, there's no reason it couldn't do it.This should be a very easy change to make.

At the moment Beancount uses "0.005" regardless of unit... this is a known pending issue (I don't have any Japanese users yet I suppose). I've been thinking that a per-commodity tolerance could be selected instead, i.e., a map of commodity to tolerance value, but while better, this is also unsatisfying.

I wonder if inferring like Ledger works in all cases though; consider this case:

    Assets:Fidelity:Fund        11.27534 FUND {1.2357 USD}

    Assets:Fidelity:Cash       -13.93 USD

The precise cash amount is 13.932937638 USD.

How many digits should this use?

Ledger would infer 2 digits of precision for USD and 5 for FUND. The price portion does not affect precision. Ledger also has a D directive to specify the minimum precision per currency.

I'm not sure what that means... there is no need for any precision on units of FUND, the balance amount of that leg is in USD calculated as 11.27534 x 1.2357. In USD. The balance residual is 11.27534 x 1.2357 + -13.93. The only precision that matters is that of USD.

Does it use the precision inferred on units of FUND in any way?

If so, how?

In Ledger, the precision is not per-transaction, but file-wide. (Because Ledger process the file in one pass, this means that the precision may increase, but never decrease, as it processes the file.) So Ledger would infer from this posting that FUND should always be measured and displayed to 5 digits of precision (or at least from now on).

Which brings up an interesting option for inferring the precision: the maximum number of digits from all legs without cost/price conversions. And this should be calculated for each group of postings whose "balance amounts" are in a common commodity. That sounds right to me. 

 

The debited cash is 13.93, this would be the _actual_ amount debited by the fund manager, so it can't be anything else.

In this example, I would argue that you would want the _minimum_ number of digits to be used.

(But certainly not the minimum if you're using an integer number of shares...)

If we can come up with a rule that works well in all cases I'll implement it.

To be honest, I would prefer to explicitly specify precision, although this makes it harder to work correctly "out of the box". 

Why? Because you fear not having a minimum will create pathological cases where a user will insert a balance against an integer number of dollars and make a mistake on the cost that will go undetected?  

2014-09-09 * 

   Assets:Invesments     ...

   Assets:Cash               -1000 USD

Just curious. What cases do you see as problematic?

I don’t have anything particular in mind. I just like having control. But I don’t feel strongly here.

I hadn’t considered per-transaction precision, but if you go that route, one thing you will need to think about in that case is what to do for this:

2014-09-09 * 

   Assets:Invesments     5 FUND @ $2.50233

   Assets:Cash

Nathan

[Martin Blais]

(You're missing the USD, I'll assume it there after 2.50233)

Yes, indeed. It would need to be rounded at some level of precision.

I'm a bit reluctant to define a mapping of {commodity: precision} because for the same commodity, a different precision maybe required per account.

Note that one I could do is delay elided postings interpolation (I have to do that for another feature anyhow) and then run an analysis on each account, and take the dominant mode of the precisions, in this example, that would mean "look at precision of all the explicit amounts on Assets:Cash for USD and choose the most appropriate precision" and then round at that. 

Hmmm this will require a bit more thought.

[Simon Michael]

On 10/7/14 10:03 AM, Nathan Grigg wrote:
>>>> I wonder if inferring like Ledger works in all cases though;
>>>> consider this case:
>>>>
>>>> Assets:Fidelity:Fund 11.27534 FUND {1.2357 USD}
>>>> Assets:Fidelity:Cash -13.93 USD
>>>>
>>>> The precise cash amount is 13.932937638 USD.
>>>> How many digits should this use?
>>>
>>> Ledger would infer 2 digits of precision for USD and 5 for FUND.
>>> The price portion does not affect precision. Ledger also has a D
>>> directive to specify the minimum precision per currency.
>>
>> I'm not sure what that means... there is no need for any precision on
>> units of FUND, the balance amount of that leg is in USD calculated as
>> 11.27534 x 1.2357. In USD. The balance residual is 11.27534 x 1.2357 +
>> -13.93. The only precision that matters is that of USD.
>>
>> Does it use the precision inferred on units of FUND in any way?
>> If so, how?
>
> In Ledger, the precision is not per-transaction, but file-wide. (Because
> Ledger process the file in one pass, this means that the precision may
> increase, but never decrease, as it processes the file.) So Ledger would
> infer from this posting that FUND should always be measured and
> displayed to 5 digits of precision (or at least from now on).

More data points for this intricate topic. hledger's per-commodity
display precisions are also file-wide, but here are some differences
from Ledger:

1 the whole file is processed first, then the precisions are selected.
So a high-precision amount at the end can set the precision for all
(actually, isn't that true for Ledger too ? My testing says yes).

2 the D directive itself does not influence the precision, but
hledger applies the D commodity to any commodity-less amounts,
in which case that adjusted amount will influence the precision
(see example at http://hledger.org/manual#default-commodity).

3 price amounts do not influence the precision, like Ledger... but they
can be used to balance an amountless posting, and that amount does
get displayed with the price's original full precision. This is how
hledger does better in your "Weird things happening" case 1:

D 1.00 USD
2014-01-01 Test
assets 1 CAD @ 1.005 USD
liabilities
2014-01-02 Test
assets 1 CAD @ 1.005 USD
liabilities

$ hledger -f tt.j reg liabilities -w66
2014/01/01 Test liabilities -1.005 USD -1.005 USD
2014/01/02 Test liabilities -1.005 USD -2.010 USD

But I see it still does not influence the canonical precision in
this case, which looks weird and should probably be fixed:

D 1.00 USD
2014-01-01 Test

assets 1 CAD @ 1.00500 USD

liabilities
2014-01-02 Test
assets 1 CAD @ 1.005 USD
liabilities

$ hledger -f tt.j reg liabilities -w66
2014/01/01 Test liabilities -1.00500 USD -1.00500 USD
2014/01/02 Test liabilities -1.005 USD -2.01000 USD


Fun, eh. :)

[Simon Michael]

I'm not following this interesting discussion in every detail, so I'm
not sure if this can help any part of this problem space, but just in
case: Have you considered or already implemented hledger's approach for
balancing transactions, which is based on display precision ? Ie amounts
are considered to balance if their sum appears to be zero when it's
rendered using the canonical display precision. This seems to fit user
expectations well.

[Martin Blais]

That's great. IMO it's a bad idea to change whatever value is derived as the system parses the file. Ordering should never matter, even for precision or formatting.

What do you do about the display precision when you have a mix of transactions with differing precisions?

 

On Tue, Oct 7, 2014 at 4:37 PM, Simon Michael <si...@joyful.com> wrote:

I'm not following this interesting discussion in every detail, so I'm not sure if this can help any part of this problem space, but just in case: Have you considered or already implemented hledger's approach for balancing transactions, which is based on display precision ? Ie amounts are considered to balance if their sum appears to be zero when it's rendered using the canonical display precision. This seems to fit user expectations well.

How do you define the canonical display precision?

You mean the precision derived from the legs that have a simple explicit amount?

[Simon Michael]

Hi Martin,

On 10/7/14 9:43 PM, Martin Blais wrote:
> What do you do about the display precision when you have a mix of
> transactions with differing precisions?

> On Tue, Oct 7, 2014 at 4:37 PM, Simon Michael <si...@joyful.comwrote:

> just in case: Have you considered or already implemented hledger's
> approach for balancing transactions, which is based on display
> precision ? Ie amounts are considered to balance if their sum
> appears to be zero when it's rendered using the canonical display
> precision. This seems to fit user expectations well.
>
>
> How do you define the canonical display precision?
> You mean the precision derived from the legs that have a simple explicit
> amount?

hledger's "canonical display precision" for a commodity is the highest
precision seen among all posting amounts in that commmodity (across all
transactions). (http://hledger.org/manual#amount-display-styles)

Internally, all amounts have the same precision: that of the Double type
used by the current implementation.

Did that answer your questions ?

[Martin Blais]

How timely we've just had this discussion: I completed a script that generates a fake user with a fair amount of detail, and as I'm now writing a script to convert Beancount syntax to Ledger format, the precision issue is biting me. Here's a problematic example with how Ledger does its precision tracking (simplified after removing a lot of other stuff around it).

Consider the following two transactions:

2012-07-09 * Investing 60% of cash in RGAGX

  Assets:US:Vanguard:RGAGX                                               3.15 RGAGX     {114.28 USD}

  Assets:US:Vanguard:Cash                                               -359.98 USD

2012-07-23 * Investing 40% of cash in VBMPX

  Assets:US:Vanguard:VBMPX                                         1.128000000000 VBMPX      {70.95 USD}

  Assets:US:Vanguard:Cash                                                -80.03 USD

Running Ledger on this works fine:

ledger -f /tmp/after.lgr bal --no-color

          3.15 RGAGX

         -440.01 USD

1.128000000000 VBMPX  Assets:US:Vanguard

         -440.01 USD    Cash

          3.15 RGAGX    RGAGX

1.128000000000 VBMPX    VBMPX

--------------------

          3.15 RGAGX

         -440.01 USD

1.128000000000 VBMPX

However, now insert another, unrelated transaction, one that "resets" the precision of USD in-between, like this:

2012-07-09 * Investing 60% of cash in RGAGX

  Assets:US:Vanguard:RGAGX                                               3.15 RGAGX     {114.28 USD}

  Assets:US:Vanguard:Cash                                               -359.98 USD

2012-07-16 * ;; Tag: #trip-new-york-2012 Takahachi |

  Liabilities:US:Chase:Slate                                       -46.833955603895 USD

  Expenses:Food:Restaurant                                         46.833955603895 USD

2012-07-23 * Investing 40% of cash in VBMPX

  Assets:US:Vanguard:VBMPX                                         1.128000000000 VBMPX      {70.95 USD}

  Assets:US:Vanguard:Cash                                                -80.03 USD

Now this does not balance anymore! :

ledger -f /tmp/after.lgr bal --no-color

While parsing file "/tmp/after.lgr", line 11:

While balancing transaction from "/tmp/after.lgr", lines 9-11:

> 2012-07-23 * Investing 40% of cash in VBMPX

>   Assets:US:Vanguard:VBMPX                                         1.128000000000 VBMPX      {70.95 USD}

>   Assets:US:Vanguard:Cash                                                -80.03 USD

Unbalanced remainder is:

  0.001600000000 USD

Amount to balance against:

 80.031600000000 USD

Error: Transaction does not balance

(Yes, the intervening transaction's precision from my incomplete script as I was developing it, but it doesn't matter, you can easily imagine a real scenario where this would happen.)

This is in-line with my suspicion: precision must either be set once and _globally_, or inferred automatically _locally_.

(I'm not sure how I will patch things up.)

--

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> How timely we've just had this discussion: I completed a script that
> generates a fake user with a fair amount of detail, and as I'm now writing a
> script to convert Beancount syntax to Ledger format, the precision issue is
> biting me. Here's a problematic example with how Ledger does its precision
> tracking (simplified after removing a lot of other stuff around it).

Great data, Martin! Can you add this to the bugtracker for one of our
precision related issues, or maybe open a new one?

Thanks,
John

[Martin Blais]

Thanks.

Just to be sure, do you consider this side-effect a bug, or WAI?

I've added a discussion of number representation and precision issues to my comparison document:

https://docs.google.com/document/d/1dW2vIjaXVJAf9hr7GlZVe3fJOkM-MtlVjvCO1ZpNLmg/edit#heading=h.qfxb9inw3w17

Also, I managed to get my example generator output file containing multiple years of data converted from Beancount to Ledger syntax and it parses, and the numbers from the balance reports do match each other. It contains a lto of regular transactions but also funds investments and regular stock trading. The example file should form a nice corpus of realistic transactions to build examples over and make comparisons of reports between our softwares. Simon: I will try to convert to HLedger too.

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> Just to be sure, do you consider this side-effect a bug, or WAI?

I don't like that the behavior changes mid-stream like that. I'm willing to
see that as a bug.

John

[Martin Blais]

FYI, for those reading the thread later in history:

Many of the ideas discussed in this thread have been implemented.

See details here:

http://furius.ca/beancount/doc/tolerances

http://furius.ca/beancount/doc/proposal-rounding