balance assertions semantics

[Simon Michael]

Hi all,

 

In current hledger, a balance assertion like

 

  some:account  $1 = $4

 

asserts that after the $1 posting, some:account's balance is $4.

 

I learned from https://github.com/simonmichael/hledger/issues/195 that in Ledger, it means that some:account's *dollar* balance is $4, and says nothing about any other commodities in the account. This should be clarified at http://ledger-cli.org/3.0/doc/ledger3.html#Balance-assertions.

 

I find this a little harder to understand and explain, and unsatisfying that a balance assertion doesn't definitively nail down (programmatically and visually) what's in an account. You can add more postings, with 0 amount if necessary, to assert the balance of each individual commodity:

 

  some:account  $1 = $4

  some:account   0 = EUR 10

  some:account   0 = FRF 15

 

but you're still not absolutely sure of the account's balance - new commodities can show up there and the assertions won't notice.

 

Also, what does it mean to assert a commodity-less zero balance ? Ledger accepts all the assertions below, which seems wrong:

 

1/1

  a  1 = 1

  a  $1 = $1

  a  $-1 = 0

  a  1 = 0

  a  1 = 0

  b

 

The advantage of Ledger's current way is that you can make at least some assertions about a multi-commodity account. In current hledger this is explicitly not supported, so if you really needed to assert a multi-commodity balance you'd have to separate the commodities into subaccounts.

 

Compatible balance assertions are important for interoperability between ledger-likes, so I'm wondering how to harmonise this. I can make hledger's assertions work like current Ledger, or we could agree to target a new design. Eg I propose we add a one-line syntax for multi-commodity amounts, such as:

 

  some:account  $1 = $4, EUR 10, FRF15    ; comma followed by a space separates amounts

 

and agree that assertions disallow commodities not explicitly mentioned. What do you think ?

 

There's at least one other incompatibility between Ledger and hledger balance assertions:

 

1/1

  a  1 = 1

  a  1 = 1  ; Ledger expects a to be 1 again, ignoring the previous posting; hledger expects 2

  b

 

Can I persuade you that hledger's interpretation is better here ?

 

Either way, I'd also like to have --no-balance-assertions or similar widely supported, so that you can at least ignore them temporarily while running another ledger-like on your data.

 

Thoughts ?

[Martin Blais]

On Mon, Jun 23, 2014 at 12:30 PM, Simon Michael <si...@joyful.com> wrote:

Hi all,

 

In current hledger, a balance assertion like

 

  some:account  $1 = $4

 

asserts that after the $1 posting, some:account's balance is $4. 

 

I learned from https://github.com/simonmichael/hledger/issues/195 that in Ledger, it means that some:account's *dollar* balance is $4, and says nothing about any other commodities in the account. This should be clarified at http://ledger-cli.org/3.0/doc/ledger3.html#Balance-assertions.

Beancount's semantics is the same as Ledger's, that is, an assertion only checks that commodity's value, it looks like this (it's a separate entry):

  2014-06-20 balance  Assets:Some:Account    4 USD

If invoked on a parent account, the balance checks against the sum of all subaccounts as well, but also only for that commodity. 

I find this a little harder to understand and explain, and unsatisfying that a balance assertion doesn't definitively nail down (programmatically and visually) what's in an account. You can add more postings, with 0 amount if necessary, to assert the balance of each individual commodity:

 

  some:account  $1 = $4

  some:account   0 = EUR 10

  some:account   0 = FRF 15

 

but you're still not absolutely sure of the account's balance - new commodities can show up there and the assertions won't notice.

Just to be clear, it also makes sense to do this in Beancount:

  2014-06-20 balance  Assets:Some:Account    413.43 USD

  2014-06-20 balance  Assets:Some:Account    201.24 CAD

I too have been wondering if a check on the totality of an inventory might be useful.

Something like this:

  2014-06-20 balance*  Assets:Some:Account    413.43 USD, 201.24 CAD

whereby if there are any other commodities in this account at that time (e.g., EUR) this would raise an error.

This is a bit of a theoretical question though; in practice, I have few accounts that contain more than one or two commodities, and not having a complete check hasn't been a problem in maintaining my ledger. It's nagging me a bit though, I like to be strict.

Also, what does it mean to assert a commodity-less zero balance ? Ledger accepts all the assertions below, which seems wrong:

 

1/1

  a  1 = 1

  a  $1 = $1

  a  $-1 = 0

  a  1 = 0

  a  1 = 0

  b

 

The advantage of Ledger's current way is that you can make at least some assertions about a multi-commodity account. In current hledger this is explicitly not supported, so if you really needed to assert a multi-commodity balance you'd have to separate the commodities into subaccounts.

 

Compatible balance assertions are important for interoperability between ledger-likes, so I'm wondering how to harmonise this. I can make hledger's assertions work like current Ledger, or we could agree to target a new design. Eg I propose we add a one-line syntax for multi-commodity amounts, such as:

 

  some:account  $1 = $4, EUR 10, FRF15    ; comma followed by a space separates amounts

 

and agree that assertions disallow commodities not explicitly mentioned. What do you think ?

 

There's at least one other incompatibility between Ledger and hledger balance assertions:

 

1/1

  a  1 = 1

  a  1 = 1  ; Ledger expects a to be 1 again, ignoring the previous posting; hledger expects 2

  b

 

Can I persuade you that hledger's interpretation is better here ?

"Better" can be defined by "useful": is it useful if the user always have to repeat all the commodities in a balance assertion?  e.g. let's say I have a lot of cash changes in an investment account. My assertions could have to look like this:

  2014-03-20 balance*  Assets:Some:Account    10 GOOG, 201.24 USD

  2014-04-20 balance*  Assets:Some:Account    10 GOOG, 762.30 USD

  2014-05-20 balance*  Assets:Some:Account    10 GOOG, 784.02 USD

  2014-06-20 balance*  Assets:Some:Account    10 GOOG, 640.40 USD

Do I really want to have to repeat the 10 GOOG every time?

 

 

Either way, I'd also like to have --no-balance-assertions or similar widely supported, so that you can at least ignore them temporarily while running another ledger-like on your data.

 

Thoughts ?

 

I'm unsure which is best.

[Simon Michael]

On 6/23/14 9:42 AM, Martin Blais wrote:
> "Better" can be defined by "useful": is it useful if the user always
> have to repeat all the commodities in a balance assertion? e.g. let's
> say I have a lot of cash changes in an investment account. My assertions
> could have to look like this:
>
> 2014-03-20 balance* Assets:Some:Account 10 GOOG, 201.24 USD
> 2014-04-20 balance* Assets:Some:Account 10 GOOG, 762.30 USD
> 2014-05-20 balance* Assets:Some:Account 10 GOOG, 784.02 USD
> 2014-06-20 balance* Assets:Some:Account 10 GOOG, 640.40 USD
>
> Do I really want to have to repeat the 10 GOOG every time?

Thanks Martin. I agree that the repetition above is more effort, but I
tend to think it's worth the trouble - explicitness and correctness are
why assertions exist after all. And I feel multi-commodity accounts are
rare and needing to write many assertions on them will be rarer, though
perhaps Thierry Daucourt will disagree on this ?

[Simon Michael]

I forgot to add: we could provide (yet another) syntax as you suggest,
eg in h/ledger format, = could assert on a single commodity and == could
assert on the complete multi-commodity balance. If Ledger isn't already
using == for something.

[thierry]

On Monday, June 23, 2014 6:42:19 PM UTC+2, Martin Blais wrote:

Just to be clear, it also makes sense to do this in Beancount:

  2014-06-20 balance  Assets:Some:Account    413.43 USD

  2014-06-20 balance  Assets:Some:Account    201.24 CAD

I too have been wondering if a check on the totality of an inventory might be useful.

Something like this:

  2014-06-20 balance*  Assets:Some:Account    413.43 USD, 201.24 CAD

whereby if there are any other commodities in this account at that time (e.g., EUR) this would raise an error.

Semantically speaking, I would argue that this is two different things:
- balance assertions, are from my point of view, to assert that balance is correct, I mean the *amount*
- checking that EUR is not used, is from my point of view a different thing, and would be better achieved with a directive like "account Assets:Some:Account commodity USD" and "account Assets:Some:Account commodity CAD", those two statements would tell that no other *commodity/currency* is allowed in that account.

Thierry

[thierry]

As you may have expected I do disagree. From "hledger stats" command, applied on my "Assets" accounts only, I have 46 Assets accounts and 48 currencies/commodities. And apart from that, I have 378 balance assertions. That would be a big effort to gather all previous account statements, and to align all of this.

Nevertheless, I support the alignment objective between ledger-likes.

On Monday, June 23, 2014 7:09:54 PM UTC+2, Simon Michael (sm) wrote:

Thanks Martin. I agree that the repetition above is more effort, but I
tend to think it's worth the trouble - explicitness and correctness are
why assertions exist after all.

Yes this is trouble, but I do not get the sentence "worth the trouble".
The current syntax of c++ ledger does offer correctness and explicitness. Again I mean balance assertions are checking the amount. Checking the exhaustiveness of currencies/commodities that are used is another thing, semantically speaking.
 

And I feel multi-commodity accounts are
rare and needing to write many assertions on them will be rarer

In a USD centric world, I believe you that mono-currency is the average.
In rest of the world, I believe that avoiding multiple currencies is more effort. And this is the standard for people living near a border.

And different from currency, as soon as you are trading, then multiple *commodity* is standard. BTW, it is not difficult to start trading, if your company is traded on any stock exchange, you can not avoid that your company is distributing shares.

> 2014-06-20 balance*  Assets:Some:Account    10 GOOG, 640.40 USD

About that syntax:
- I have at least 2 accounts, where I do have more than 6 commodities. With names of commodities that are 20 characters long. This would make things tedious...
- Also, do not forget that some of us are using comma as decimal separator That syntax above may not be the best one.

Thierry

[Martin Blais]

That's a good point. This is already in Beancount BTW, as part of the "open" directive, you can optionally add a list of commodities to restrict to, e.g.

  2010-01-01 open  Assets:Some:Account    USD,CAD

(The "USD,CAD" bit is optional.)

With this restriction, any transaction posting a change to this account in a commodity that is not USD nor CAD will trigger an error.

[Martin Blais]

On Mon, Jun 23, 2014 at 4:39 PM, thierry <th...@free.fr> wrote:

And different from currency, as soon as you are trading, then multiple *commodity* is standard. BTW, it is not difficult to start trading, if your company is traded on any stock exchange, you can not avoid that your company is distributing shares.

> 2014-06-20 balance*  Assets:Some:Account    10 GOOG, 640.40 USD

About that syntax:

Note: I may not have been specific about the context here, but I inferred that the syntax above would be specific to Beancount. I'm not even trying to be backwards-compatible with Ledger syntax at this point, nor am I suggesting that Ledger should change its syntax either.

 

- I have at least 2 accounts, where I do have more than 6 commodities. With names of commodities that are 20 characters long. This would make things tedious...

I don't see it as a problem.

I prefer to have all directives types looking similar, I like the regularity of it:

http://furius.ca/beancount/doc/cheatsheet/beancount-cheatsheet.pdf

 

- Also, do not forget that some of us are using comma as decimal separator That syntax above may not be the best one.

Not supported at the moment in my implementation, but I don't think it should be a problem with the parser.

[Stefano Zacchiroli]

Just a very minor point / syntactic nitpicking (I don't use assertions
much, so I'm not particularly qualified to comment on which design would
be best):

On Mon, Jun 23, 2014 at 09:30:07AM -0700, Simon Michael wrote:
> some:account $1 = $4, EUR 10, FRF15 ; comma followed by a space
> separates amounts

Several countries in the world uses comma as a digital separator.

If not syntactically ambiguous---the subsequent space might save the
day---using comma to separate assertions might get very hard on the eye
for users that do use digital commas, e.g.:

some:account $1 = EUR 10,20, CHF 3,2, $10,27

Please, think of the kitten :)

My 0,02 EUR,
Cheers.
--
Stefano Zacchiroli . . . . . . . za...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »

[Simon Michael]

On 6/24/14 2:13 AM, Stefano Zacchiroli wrote:
> Several countries in the world uses comma as a digital separator.
>
> If not syntactically ambiguous---the subsequent space might save the
> day---using comma to separate assertions might get very hard on the eye
> for users that do use digital commas, e.g.:
>
> some:account $1 = EUR 10,20, CHF 3,2, $10,27

Yes, that's what the space is for; I haven't thought of anything better.
; is for comments, so can't use that. Double space ?

[John Wiegley]

>>>>> Simon Michael <si...@joyful.com> writes:

> I find this a little harder to understand and explain, and unsatisfying that
> a balance assertion doesn't definitively nail down (programmatically and
> visually) what's in an account. You can add more postings, with 0 amount if
> necessary, to assert the balance of each individual commodity:

> some:account $1 = $4
> some:account 0 = EUR 10
> some:account 0 = FRF 15

> but you're still not absolutely sure of the account's balance - new
> commodities can show up there and the assertions won't notice.

You are right, there should be a way to "close" the set of assertions within a
transaction. That is, each individual commodity must balance, but if at the
end we have anything left over, then the group of assertions has failed. This
would then necessity adding 0 = FOO X declarations for all the other
commodities.

John

[Martin Blais]

On a related note: why are balance assertions tied to a transaction? 

In Beancount I treat them as a distinct directive, e.g.:

  2014-06-30 balance  Assets:Some:Account    10 EUR

The advantage is that you can have assertions in between transactions, at any point in time - balance assertions have their own date - they exist without a corresponding transaction. I also prefer the dissociation of these two directives; they represent different actions, the balance assertion corresponds to the "Statement Balance" line of a monthly statement, for instance, and if a statement is empty because of a lack of activity, it still makes sense for a user to be able to insert an assertion. That's often how I know I've checked and updated the balance for a dormant account (e.g. credit-line, which I use only for short-lived cash-flow needs).

The disadvantage is that it cannot insert a balance assertion between transactions within the same day on the same account, because Beancount does not support the notion of "time" (I know that Ledger does, I chose to remove time on purpose, as I did imposing that ordering of directives should not matter). The semantics of the balance assertion is well-defined: it performs the check at the beginning of the day (before any other directives that occur during tha today). In practice, this limitation has not been an issue, it is pretty rare that you would have so many transactions on (1) the same account (2) during the same day and that (3) you would want to assert the balance in-between. But it is a limitation, nonetheless.

Thoughts welcome,

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> On a related note: why are balance assertions tied to a transaction?  In
> Beancount I treat them as a distinct directive, e.g.:

>   2014-06-30 balance  Assets:Some:Account    10 EUR

Because tying them to transactions is more general. I can mimick your
standalone assertion by just having a transaction that does nothing else but
assert a balance.

John

[Martin Blais]

Sure, but then a transaction that does nothing would appear in the register, no?

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> Sure, but then a transaction that does nothing would appear in the register,
> no?

If you are using --empty then yes, it would, otherwise no.

John

[Simon Michael]

On 6/30/14 9:55 PM, John Wiegley wrote:
>>>>>> Martin Blais <bl...@furius.ca> writes:
>> Sure, but then a transaction that does nothing would appear in the register,
>> no?

I feel sympathetic to Martin here - a separate directive seems more
orthogonal. Postings are already expressing quite a lot.

Though, if not taking journal order into account I would have processed
it last, after any transactions on that day.

If taking order into account, which I wouldn't mind doing - h/ledger do
it elsewhere - the separate directive would be just as expressive as the
current scheme, no ? More so, as it would never introduce dummy
transactions and requiring -E to hide (possibly disturbing your report
in other ways).

If the posting-based assertions do remain, then hledger will probably go
ahead and add == to assert against the full multi-commodity amount.

[Martin Blais]

On Tue, Jul 1, 2014 at 9:50 AM, Simon Michael <si...@joyful.com> wrote:

On 6/30/14 9:55 PM, John Wiegley wrote:

Martin Blais <bl...@furius.ca> writes:

Sure, but then a transaction that does nothing would appear in the register,
no?

I feel sympathetic to Martin here - a separate directive seems more orthogonal. Postings are already expressing quite a lot.

I also like the flattening of all directives to have a single meaning. And it keeps transactions simpler, as you say.

Though, if not taking journal order into account I would have processed it last, after any transactions on that day.

I used to feel the same, but I needed both. Then I had a plan for a "balance_end" directive that would apply at the end of the day (otherwise same semantics).  Then I realized I did not need it at all, I just fudge the date manually to the next day. I chose the beginning of the day to be consistent with other date ordering things (can't remember now).

From a technical perspective, this would be trivial to add back, BTW, they would just be converted into "balance" directives with date + 1, it could even be done by the parser itself. I'm not convinced adding one more directive is worth it though. What do you think?

If taking order into account, which I wouldn't mind doing - h/ledger do it elsewhere - the separate directive would be just as expressive as the current scheme, no ? More so, as it would never introduce dummy transactions and requiring -E to hide (possibly disturbing your report in other ways).

I feel that ensuring the property that "order does not matter" simplifies the semantics of a ledger quite a bit, I think it's quite an important guarantee. If you have multiple transactions on the same day, I'm not sure what a balance assertion on a posting means; surely it does not mean "the balance at the end of the day of the posting of this transaction" but rather "the balance right after processing this transaction" which appears to happen in file order (in Ledger).  Now, because of this, if you reorder your directives, e.g., you insert a new directive before one that has a balance assertion on a posting that affects an account with another assertion on the same date, it might break a balance assertion that had previously been working. 

Wow, that was a mouthful. So let's take a concrete example instead, let's say you're organizing transactions by section (in this example: credit card transactions and checking account transactions), the following will not generate an error in Ledger, it works:

;; Credit card account


2014/05/01 opening
  Liabilities:CreditCard    $-1000.00
  Expenses:OpeningBalances

2014/05/12 dinner
  Liabilities:CreditCard    $-74.20
  Expenses:Restaurant


;; Checking account

2014/06/05 salary
  Assets:Checking            $4082.87
  Income:Salary

2014/06/05 cc payment
  Assets:Checking            $-1074.20 = $3008.67
  Liabilities:CreditCard     = $0

If I had instead decided to put the credit card payment in the credit card account section (same set of transactions, I just moved it), this fails (try it):

;; Credit card account


2014/05/01 opening
  Liabilities:CreditCard    $-1000.00
  Expenses:OpeningBalances

2014/05/12 dinner
  Liabilities:CreditCard    $-74.20
  Expenses:Restaurant

2014/06/05 cc payment
  Assets:Checking            $-1074.20 = $3008.67
  Liabilities:CreditCard     = $0


;; Checking account

2014/06/05 salary
  Assets:Checking            $4082.87
  Income:Salary

mandarine [default]:~/p/ledger-experiments$ ledger -f ordering1.lgr reg checking

While parsing file "/home/blais/p/ledger-experiments/ordering1.lgr", line 13: 

While parsing posting:

  Assets:Checking            $-1074.20 = $3008.67

                                         ^^^^^^^^

Error: Balance assertion off by $4082.87

I much prefer if I don't have to worry about where things are. 

And this is not an uncommon pathological case. For example, I place my salary income transactions in their own section, though they get paid via direct deposit to my checking account. It is not uncommon that other entries appear in the checking account on pay day. So with balance assertions on real transactions, whether the balance assertion would work or not would depend on whether it is placed on transaction that will appear last in the ledger file. I don't want to have to think about this.

IMHO, when designing this, the question you have to ask yourself is: where do the balance amounts you report come from? Usually they come from some statement, you copy the value from the line on your statement which says "your balance on YYYY/MM/DD is X". OFX also has a field that tells you the current balance of the account you just downloaded at the current date (you can convert that to a balance assertion automatically, has no transaction associated to it as the date is often several days after the last transaction). My point is: where those amounts come from often has a separate date from transactions.

So with current Ledger semantics, how do you make it order-invariant? 

One way to be certain it will always work would be to 

1) always make balance assertions on dedicated (empty) transactions

2) place all assertions at the end of your file, after all transactions.

This is not practice, but it should always work and should allow you to reorder all transactions at will.

Choosing to define balance assertions on their own removes _just_ enough degrees of freedom that it makes it unambiguous and order-invariant. I think you could easily do the same thing in Ledger, in fact, this _may_ be the only thing that is needed to make Ledger's balance assertions order-invariant. That would a really nice property to have IMO, and relatively easy change.

If the posting-based assertions do remain, then hledger will probably go ahead and add == to assert against the full multi-commodity amount.

How do you plan to disambiguate for the case I point out above?

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> 2) place all assertions at the end of your file, after all transactions.

Doesn't this mean that you couldn't check balances during the course of the
ledger, however? That is my sole use of them; only having assertions at the
very end would be next to useless for me.

John

[Martin Blais]

I agree.

My point with the suggestion was to highlight that a reorder-safe way to use balance assertions that are tied to transactions is not practical; I hope my example highlights this. You can either (1) place them where it makes sense in your Ledger and live with occasional and difficult-to-debug failures due to reordering, or (2) use an impractical method that is reorder-safe.

(1) could be a reasonable choice, "just live with it" - depends on your view - but requires a subtle understanding of what's going on behind the scenes.

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> My point with the suggestion was to highlight that a reorder-safe way to use
> balance assertions that are tied to transactions is not practical; I hope my
> example highlights this. You can either (1) place them where it makes sense in
> your Ledger and live with occasional and difficult-to-debug failures due to
> reordering, or (2) use an impractical method that is reorder-safe.

Ah, OK, I understand now. :)

> (1) could be a reasonable choice, "just live with it" - depends on your view
> - but requires a subtle understanding of what's going on behind the scenes.

I did add a feature recently that allows me to split transactions across
multiple files; I think I mentioned it here on the ML. But yes, the more of
them I add, the more edit-fragile my data becomes.

John

[Simon Michael]

On 7/1/14 7:55 AM, Martin Blais wrote:
> because of this, if you reorder your directives, e.g., you insert a new
> directive before one that has a balance assertion on a posting that
> affects an account with another assertion on the same date, it might
> break a balance assertion that had previously been working.

I'm used to h/ledger working this way (entries with the same date being
processed in journal order). I don't mind it because a. it's intuitive
to users and b. it allows you to assert intra-day balances as well as
inter-day ones.

I do see that being completely independent of journal order is nice too.

> IMHO, when designing this, the question you have to ask yourself is:
> where do the balance amounts you report come from? Usually they come
> from some statement, you copy the value from the line on your statement
> which says "your balance on YYYY/MM/DD is X". OFX also has a field that

I haven't made a balance assertion corresponding to a bank statement yet
(good idea). Mine are more commonly for cash and need not be at the end
of the day. Eg:

2014/6/25 * adjust/reconcile cash
assets:personal:cash:wallet $-1 = $14.50
expenses:personal:misc

> Choosing to define balance assertions on their own removes _just_ enough
> degrees of freedom that it makes it unambiguous and order-invariant. I
> think you could easily do the same thing in Ledger, in fact, this _may_
> be the only thing that is needed to make Ledger's balance assertions
> order-invariant.

Order invariance, and having balance assertions on postings or as
separate directives, are mostly orthogonal issues I think.

> If the posting-based assertions do remain, then hledger will
> probably go ahead and add == to assert against the full
> multi-commodity amount.
>
> How do you plan to disambiguate for the case I point out above?

Do you mean the ordering issue ? The comment you're quoting there
doesn't change anything there, hledger would still process transactions,
postings and assertions in (1) date and (2) journal order.

[Martin Blais]

On Tue, Jul 1, 2014 at 8:01 PM, Simon Michael <si...@joyful.com> wrote:

On 7/1/14 7:55 AM, Martin Blais wrote:

because of this, if you reorder your directives, e.g., you insert a new
directive before one that has a balance assertion on a posting that
affects an account with another assertion on the same date, it might
break a balance assertion that had previously been working.

I'm used to h/ledger working this way (entries with the same date being processed in journal order). I don't mind it because a. it's intuitive to users and b. it allows you to assert intra-day balances as well as inter-day ones.

A dedicated balance directive is also quite intuitive IMO.

I do see that being completely independent of journal order is nice too.

IMHO, when designing this, the question you have to ask yourself is:
where do the balance amounts you report come from? Usually they come
from some statement, you copy the value from the line on your statement
which says "your balance on YYYY/MM/DD is X". OFX also has a field that

I haven't made a balance assertion corresponding to a bank statement yet (good idea). Mine are more commonly for cash and need not be at the end of the day. Eg:

2014/6/25 * adjust/reconcile cash
    assets:personal:cash:wallet       $-1 = $14.50
    expenses:personal:misc

Choosing to define balance assertions on their own removes _just_ enough
degrees of freedom that it makes it unambiguous and order-invariant. I
think you could easily do the same thing in Ledger, in fact, this _may_
be the only thing that is needed to make Ledger's balance assertions
order-invariant.

Order invariance, and having balance assertions on postings or as separate directives, are mostly orthogonal issues I think.

No... that's exactly the problem I'm pointing out! They're not orthogonal. If you place balance assertions on transaction postings, they are assumed to apply after each transaction is added to the balance (or even each posting). OTOH if you have a dedicated directive, it can be constrained to only apply at the beginning or end of a day, in which case there is no ambiguity. There is no ambiguity because doing it that way does not provide the opportunity to the user to set intra-day balance assertions (it's a tiny bit less powerful, but I don't see intra-day balance assertions as necessary, I've been living without them for a while, not an issue in practice).

 

    If the posting-based assertions do remain, then hledger will
    probably go ahead and add == to assert against the full
    multi-commodity amount.

How do you plan to disambiguate for the case I point out above?

Do you mean the ordering issue ? The comment you're quoting there doesn't change anything there, hledger would still process transactions, postings and assertions in (1) date and (2) journal order.

So HLedger is or is not subject to the same problem that I point out in my example?

[Simon Michael]

On 7/1/14 6:23 PM, Martin Blais wrote:
> A dedicated balance directive is also quite intuitive IMO.

Yes, I agree.

> Order invariance, and having balance assertions on postings or as
> separate directives, are mostly orthogonal issues I think.
>
> No... that's exactly the problem I'm pointing out! They're not
> orthogonal.

Ok, not completely orthogonal, there is overlap. Having assertions on
postings does partly determine the order they must be applied. This
depends also on how order-independent transactions (and postings within
transactions) are.

This gets a bit complicated. Eg, currently Ledger respects journal order
more, and transaction dates less, than hledger does when deciding the
order in which to to apply postings and assertions (I don't have an
example to hand). And on the other hand, hledger is more respectful of
journal order than Ledger is when processing postings and assertions
within a single transaction.

> Do you mean the ordering issue ? The comment you're quoting there
> doesn't change anything there, hledger would still process
> transactions, postings and assertions in (1) date and (2) journal order.
>
> So HLedger is or is not subject to the same problem that I point out in
> my example?

You mean the problem that it's possible to break your assertions by
reordering things (postings, transactions, and/or assertions) in the
journal ? Yes hledger is subject to that.

[Martin Blais]

On Tue, Jul 1, 2014 at 9:53 PM, Simon Michael <si...@joyful.com> wrote:

On 7/1/14 6:23 PM, Martin Blais wrote:

A dedicated balance directive is also quite intuitive IMO.

Yes, I agree.

    Order invariance, and having balance assertions on postings or as
    separate directives, are mostly orthogonal issues I think.

No... that's exactly the problem I'm pointing out! They're not
orthogonal.

Ok, not completely orthogonal, there is overlap. Having assertions on postings does partly determine the order they must be applied. This depends also on how order-independent transactions (and postings within transactions) are.

This gets a bit complicated. Eg, currently Ledger respects journal order more, and transaction dates less, than hledger does when deciding the order in which to to apply postings and assertions (I don't have an example to hand). And on the other hand, hledger is more respectful of journal order than Ledger is when processing postings and assertions within a single transaction.

Alright... so here's the punchline: I'm claiming that if Ledger or HLedger gave up on balance assertions attached to postings, if you separated them as I do, just that change, you would both very likely be able to guarantee order-independence w.r.t. your input files! Think about it.  (If you don't think it would be enough, can you share why?)

Personally, I think it would be well worth it, it's a really nice property to have.

It would simplify the semantics of your systems, at relatively little cost.

If you disagree, mighty fine - this is why we have diverse implementations after all - at least the example in this thread should serve as documentation for others who might come across this problem inadvertently. If I were you I'd add the example to your docs too in case someone has this problem.

Cheers,

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> Alright... so here's the punchline: I'm claiming that if Ledger or HLedger
> gave up on balance assertions attached to postings, if you separated them as
> I do, just that change, you would both very likely be able to guarantee
> order-independence w.r.t. your input files! Think about it.  (If you don't
> think it would be enough, can you share why?)

How do you achieve order independence? I often have a statement ending on a
particular date, with transactions before the statement end on that date, and
transactions after the statement end on that same date. I use ordering to
specify when exactly the assertion should occur.

John

[Simon Michael]

On 7/1/14 7:45 PM, Martin Blais wrote:
> Alright... so here's the punchline: I'm claiming that if Ledger or
> HLedger gave up on balance assertions attached to postings, if you
> separated them as I do, just that change, you would both very likely be
> able to guarantee order-independence w.r.t. your input files! Think

Indeed we would, of course, at the cost of giving up intra-day assertions.

I don't feel too strongly about intra-day assertions either way. But I
think hledger needs to support Ledger's style of assertions, at least.
(Though I just added a --ignore-assertions flag as an alternative.)

We could also name and implement all of these three kinds of assertions:

- ledger's single-commodity, ordered, posting-based assertions
- hledger's multi-commodity, ordered, posting-based assertions (using ==)
- beancount's order-independent, directive-based assertions (do these
assert one commodity or all of them ?)

This gives most power and compatibility.. but also more feature bloat.
Anyway I probably shouldn't think about supporting beancount format with
hledger's main parser, due to its other differences.

[Simon Michael]

On 7/2/14 6:35 AM, Simon Michael wrote:
> I don't feel too strongly about intra-day assertions either way. But I
> think hledger needs to support Ledger's style of assertions, at least.
> (Though I just added a --ignore-assertions flag as an alternative.)

To be clear: I mean I'd like hledger to include support for the
idiomatic style of assertions Ledger uses going forward, whatever that
ends up being.

[Martin Blais]

Yes, I'm suggesting that making such an assertion should not be possible anymore. You give up this capability, in exchange for the property of order-independence. That's the compromise I made; for me, order-independence is a much more important property than the capability to make balance assertions between transactions within a day.

You often have this? What kind of account is this? Bank, credit card, trading? And how often do this occur? I have seen very few cases in 8 years where I could not just increment or decrement the balance assertion date, or skip an assertion (very rarely occurs - it's okay to skip one if you have another one later on, they're entirely optional anyway).

[Martin Blais]

On Wed, Jul 2, 2014 at 9:35 AM, Simon Michael <si...@joyful.com> wrote:

On 7/1/14 7:45 PM, Martin Blais wrote:

Alright... so here's the punchline: I'm claiming that if Ledger or
HLedger gave up on balance assertions attached to postings, if you
separated them as I do, just that change, you would both very likely be
able to guarantee order-independence w.r.t. your input files! Think

Indeed we would, of course, at the cost of giving up intra-day assertions.

I don't feel too strongly about intra-day assertions either way. But I think hledger needs to support Ledger's style of assertions, at least. (Though I just added a --ignore-assertions flag as an alternative.)

We could also name and implement all of these three kinds of assertions:

- ledger's single-commodity, ordered, posting-based assertions
- hledger's multi-commodity, ordered, posting-based assertions (using ==)
- beancount's order-independent, directive-based assertions (do these assert one commodity or all of them ?)

For now, a single commodity. I'm attracted to the idea of extending this to cover the entirety of the inventory, I think this is going to be added eventually.

I'm also vaguely interested in ideas for supporting intra-day balance assertions in a way that preserves order-independence. I'm not sure that it's possible, but it's worth throwing around ideas.

This gives most power and compatibility.. but also more feature bloat. Anyway I probably shouldn't think about supporting beancount format with hledger's main parser, due to its other differences.

Oh yeah... I've walked away from even trying to be compatible years ago. I want to push things forward aggressively, especially w.r.t. to the inventory booking semantics, I have a real need for booking at average cost:

https://docs.google.com/document/d/1F8IJ_7fMHZ75XFPocMokLxVZczAhrBRBVN9uMhQFCZ4/

On Wed, Jul 2, 2014 at 9:41 AM, Simon Michael <si...@joyful.com> wrote:

On 7/2/14 6:35 AM, Simon Michael wrote:

I don't feel too strongly about intra-day assertions either way. But I
think hledger needs to support Ledger's style of assertions, at least.
(Though I just added a --ignore-assertions flag as an alternative.)

To be clear: I mean I'd like hledger to include support for the idiomatic style of assertions Ledger uses going forward, whatever that ends up being.

If you're trying to be compatible, that makes sense.

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> Yes, I'm suggesting that making such an assertion should not be possible
> anymore. You give up this capability, in exchange for the property of
> order-independence. That's the compromise I made; for me, order-independence
> is a much more important property than the capability to make balance
> assertions between transactions within a day.

For me, this loses the value of balance assertions for a convenience I don't
presently need.

> You often have this? What kind of account is this? Bank, credit card,
> trading? And how often do this occur? I have seen very few cases in 8 years
> where I could not just increment or decrement the balance assertion date, or
> skip an assertion (very rarely occurs - it's okay to skip one if you have
> another one later on, they're entirely optional anyway).

My credit card statements regularly have transactions within one day, for
which other transactions on that same day occur only on the next statement.
Hence, not having intra-day assertions would mean I couldn't make use of
assertions to help manage reconciliation of statements (my primary use of
them, in fact).

The goal of Ledger has always been flexibility first, allowing users to define
their own semantics. I recognize that Beancount is much more oriented toward
an accounting-centric semantics for the sake of that use case. This is a
healthy division, and I think a cross-pollinating one too.

Intra-day assertions could have other uses too, which I'm unaware of at the
moment. That's reason enough for me to be unwilling to sacrifice them for
convenience. There may even be other solutions here that allow us to have
both that have yet to be explored.

John

[Martin Blais]

On Wed, Jul 2, 2014 at 7:04 PM, John Wiegley <jo...@newartisans.com> wrote:

>>>>> Martin Blais <bl...@furius.ca> writes:

> Yes, I'm suggesting that making such an assertion should not be possible
> anymore. You give up this capability, in exchange for the property of
> order-independence. That's the compromise I made; for me, order-independence
> is a much more important property than the capability to make balance
> assertions between transactions within a day.

For me, this loses the value of balance assertions for a convenience I don't
presently need.

> You often have this? What kind of account is this? Bank, credit card,
> trading?  And how often do this occur? I have seen very few cases in 8 years
> where I could not just increment or decrement the balance assertion date, or
> skip an assertion (very rarely occurs - it's okay to skip one if you have
> another one later on, they're entirely optional anyway).

My credit card statements regularly have transactions within one day, for
which other transactions on that same day occur only on the next statement.
Hence, not having intra-day assertions would mean I couldn't make use of
assertions to help manage reconciliation of statements (my primary use of
them, in fact).

The goal of Ledger has always been flexibility first, allowing users to define
their own semantics.  I recognize that Beancount is much more oriented toward
an accounting-centric semantics for the sake of that use case.  This is a
healthy division, and I think a cross-pollinating one too.

I don't think how that lets users define their own semantics.

Intra-day assertions could have other uses too, which I'm unaware of at the
moment.  That's reason enough for me to be unwilling to sacrifice them for
convenience.  There may even be other solutions here that allow us to have
both that have yet to be explored.

 

John

--

---
You received this message because you are subscribed to the Google Groups "Ledger" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ledger-cli+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Blais]

On Thu, Jul 3, 2014 at 12:17 PM, Martin Blais <bl...@furius.ca> wrote:

On Wed, Jul 2, 2014 at 7:04 PM, John Wiegley <jo...@newartisans.com> wrote:

My credit card statements regularly have transactions within one day, for
which other transactions on that same day occur only on the next statement.
Hence, not having intra-day assertions would mean I couldn't make use of
assertions to help manage reconciliation of statements (my primary use of
them, in fact).

The goal of Ledger has always been flexibility first, allowing users to define
their own semantics.  I recognize that Beancount is much more oriented toward
an accounting-centric semantics for the sake of that use case.  This is a
healthy division, and I think a cross-pollinating one too.

I don't think how that lets users define their own semantics.

Sorry I fat-fingered the send button, I meant: I don't see how this lets users define their own semantics.

[Simon Michael]

On 7/3/14 9:25 AM, Martin Blais wrote:
> Sorry I fat-fingered the send button, I meant: I don't see how this lets
> users define their own semantics.

I think in the sense that they can choose to assert intra-day, or
between days, by where they place the assertions (on a dummy transaction
if necessary).

But note they can't choose (in Ledger) to place dated assertions at one
end of the journal, or in a separate file - because Ledger's assertions
currently follow *only* parse order, ignoring of the date of the posting
they're on. I had forgotten this. It sounds like John would change this
but it's awkward with the current implementation, because it processes
assertions during parsing.

[John Wiegley]

>>>>> Simon Michael <si...@joyful.com> writes:

> But note they can't choose (in Ledger) to place dated assertions at one end
> of the journal, or in a separate file - because Ledger's assertions
> currently follow *only* parse order, ignoring of the date of the posting
> they're on. I had forgotten this. It sounds like John would change this but
> it's awkward with the current implementation, because it processes
> assertions during parsing.

Yes, I've dumped at least a day into this problem, but Ledger's current
processing pipeline won't allow it without back-tracking or caching huge
amounts of data in memory, or recalculating certain figures from scratch each
time they are needed.

I'm convinced Ledger should have adopted an intermediary data representation:
Parse data into "raw" structures, and then "cooking" these structures. At the
moment it parses directly into the cooked form. Having an intermediate "raw"
representation would allow me to do many things much more easily,
order-independence being among them (or so I believe).

My Haskell version of the Ledger parser does exactly this, but changing the
C++ code would be a huge refactoring, and beyond the scope of what I want to
do to that code right now. If I were to spend large amounts of time doing new
development, it would be in Haskell with a hope to merge efforts with Simon
toward more feature parity for hledger. I'm quite happy with C++ ledger's
current feature set and stability, so my goal there is to keep that purring
along, modulo fixing important problems like rounding and currency evaluation.

John

[Martin Blais]

On Thu, Jul 3, 2014 at 1:58 PM, John Wiegley <jo...@newartisans.com> wrote:

>>>>> Simon Michael <si...@joyful.com> writes:

> But note they can't choose (in Ledger) to place dated assertions at one end
> of the journal, or in a separate file - because Ledger's assertions
> currently follow *only* parse order, ignoring of the date of the posting
> they're on. I had forgotten this. It sounds like John would change this but
> it's awkward with the current implementation, because it processes
> assertions during parsing.

Oh wow... this explains it. Now this makes sense. I was under the impression that balancing was still respecting date order (that's how I implemented it in Beancount). So these are really different kinds of operations: I'll call mine, the order-independent ones that have their own directive, a "balance assertion", and a file-order based ones "running assertions." I don't think I would ever really need running assertions, though IFF all the transactions are present in the input and in sorted order, I see how that makes it possible to support intra-day balance checks.

Yes, I've dumped at least a day into this problem, but Ledger's current
processing pipeline won't allow it without back-tracking or caching huge
amounts of data in memory, or recalculating certain figures from scratch each
time they are needed.

I'm convinced Ledger should have adopted an intermediary data representation:
Parse data into "raw" structures, and then "cooking" these structures.  At the
moment it parses directly into the cooked form.  Having an intermediate "raw"
representation would allow me to do many things much more easily,
order-independence being among them (or so I believe).

That's what I do in Beancount. The only "processing" at the parsing stage is ensuring the transactions balance (e.g., filling a missing posting amount).

My Haskell version of the Ledger parser does exactly this, but changing the
C++ code would be a huge refactoring, and beyond the scope of what I want to
do to that code right now.  If I were to spend large amounts of time doing new
development, it would be in Haskell with a hope to merge efforts with Simon
toward more feature parity for hledger.  I'm quite happy with C++ ledger's
current feature set and stability, so my goal there is to keep that purring
along, modulo fixing important problems like rounding and currency evaluation.

On a related note about the C++ implementation: 

What about the booking problem I outlined in this section:

https://docs.google.com/document/d/1F8IJ_7fMHZ75XFPocMokLxVZczAhrBRBVN9uMhQFCZ4/edit#heading=h.e2ug3mq1m700 

The only way I managed to get an inventory to reduce a position was by specifying both the cost and the date.

Is this a bug or intended behavior?

If not, is the user expected to always put both the cost and the date in?

[John Wiegley]

>>>>> Martin Blais <bl...@furius.ca> writes:

> On a related note about the C++ implementation:  What about the booking

> problem I outlined in this section: The only way I managed to get an

> inventory to reduce a position was by specifying both the cost and the date.

> Is this a bug or intended behavior?

To be honest, I'm not sure. Your treatment of the problem is much too
in-depth for me to take in at the moment. Which is great for understanding
the problem, but not so great for me to contribute to the discussion just
now. :)

John

[Martin Blais]

I summarized ideas about balance assertions and make a proposal for extensions here:

https://docs.google.com/document/d/1vyemZFox47IZjuBrT2RjhSHZyTgloYOUeJb73RxMRD0/

Open for commenting in the margins.

Your thoughts are welcome.

Thanks,