Elided amounts and `--infer-costs`
21 views
Skip to first unread message
Claudi Lleyda Moltó
Jan 19, 2023, 1:08:19 PM1/19/23
to hle...@googlegroups.com
Greetings,
Recently I have encountered some behaviour I do not understand when
using transactions with elided amounts and the `--infer-costs` flag.
Since I am not really sure of what is going on, I have written my
findings here in hope for help, as searching around online did not help
me answer make sense of the situation.
The `--infer-costs` flag works as expected with the following
multi-currency transaction
```
2023-01-01 Good
expenses:food 111.00 NOK
equity:conversion -111.00 NOK
equity:conversion 11.41 EUR
assets:bank -11.41 EUR
```
This changes when encountering an elided amount, for example
```
2023-01-02 Bad
expenses:food 111.00 NOK
equity:conversion -111.00 NOK
equity:conversion 11.41 EUR
assets:bank
```
Now running `hledger print --infer-costs` we output the following error
```
hledger: Error: -:1-5:
1 | 2023-01-02 Bad
| expenses:food 111.00 NOK @@ 11.41 EUR
| equity:conversion -111.00 NOK
| equity:conversion 11.41 EUR
| assets:bank -22.82 EUR
| assets:bank 111.00 NOK
This multi-commodity transaction is unbalanced.
The real postings' sum should be 0 but is: -11.41 EUR, 111.00 NOK
Consider adjusting this entry's amounts, adding missing postings,
or recording conversion price(s) with @, @@ or equity postings.
```
It seems `hledger` is adding both amounts found in the conversion
postings to the elided amount, which seems like a weird behaviour at
first, as here it unbalances a valid entry.
From the manual [1] we find
> Postings will be recognised as equity conversion postings if they are
> 1. to account(s) declared with type V (Conversion; or if no such
> accounts are declared, accounts named equity:conversion, equity:trade,
> equity:trading, or subaccounts of these)
> 2. adjacent
> 3. and exactly matching the amounts of two non-conversion postings.
It seems that the condition that is failing in the second entry is the
third one: the equity postings identified involve the amounts 111.00 NOK
and 11.41 EUR, and although the `expenses:food` posting matches the NOK
amount, there is no explicit match for the EUR one.
While the 11.41 EUR amount is implicitly stated, it seems believable
that in general this approach could not use the implicit amount, as when
running `hledger print` with the `--explicit` flag, as for a more
complicated entry the implicit amount might add up to the required
matching amount in an unexpected manner, or something similar.
What confused me is that if we add a commission cost to the examples, we
find similar results:
```
2023-01-01 Good
expenses:food 111.00 NOK
equity:conversion -111.00 NOK
equity:conversion 11.41 EUR
expenses:commission 0.57 EUR
assets:bank -11.98 EUR
2023-01-02 Bad
expenses:food 111.00 NOK
equity:conversion -111.00 NOK
equity:conversion 11.41 EUR
expenses:commission 0.57 EUR
assets:bank
```
Entries of this style are what prompted this message.
Running `hledger print --infer-costs` outputs the following error
```
hledger: Error: -:8-13:
8 | 2023-01-02 Bad
| expenses:food 111.00 NOK @@ 11.41 EUR
| equity:conversion -111.00 NOK
| equity:conversion 11.41 EUR
| expenses:commission 0.57 EUR
| assets:bank -23.39 EUR
| assets:bank 111.00 NOK
This multi-commodity transaction is unbalanced.
The real postings' sum should be 0 but is: -11.41 EUR, 111.00 NOK
Consider adjusting this entry's amounts, adding missing postings,
or recording conversion price(s) with @, @@ or equity postings.
```
In this case, it seems like the first entry should not have satisfied
the third condition from the manual, as there is no posting with the
explicit amount of 11.41 EUR, but is handled correctly regardless.
Meanwhile, the second entry fails, and again we see that `hledger` added
the conversion amounts to the elided amount, again unbalancing it.
Altogether, this leads me to believe that the transaction with the
elided amount could have been handled exactly like the explicit one,
using the amount that can be calculated with `hledger print --explicit`.
In fact, the explicit EUR amount, 11.98 = 11.41 + 0.57, is available to
at some point, as it appears implicitly in the `assets:bank` posting,
since the stated amount in the error is 23.39 = 11.98 + 11.41.
Is there a deeper reason why a transaction with an elided amount is
handled differently? Perhaps I am misunderstanding something, but given
how entries with no elided amounts are handled, it appears that the rest
could be handled in an equivalent manner.
Now, given that the behaviour involving explicit entries does not match
the description from the manual, it might be that the misbehaviour
occurs there, and not in the entries with elided amounts.
Anyway, some help here would be appreciated, as I have been logging my
finances with elided amounts using only one currency since 2019, and
only recently encountered this multi-currency scenario, and do not
really look forward rewriting them to be explicit.
Best,
Claudi
```
$ hledger --version
hledger 1.28, linux-x86_64
```
[1] https://hledger.org/1.28/hledger.html#inferring-cost-from-equity-postings
Recently I have encountered some behaviour I do not understand when
using transactions with elided amounts and the `--infer-costs` flag.
Since I am not really sure of what is going on, I have written my
findings here in hope for help, as searching around online did not help
me answer make sense of the situation.
The `--infer-costs` flag works as expected with the following
multi-currency transaction
```
2023-01-01 Good
expenses:food 111.00 NOK
equity:conversion -111.00 NOK
equity:conversion 11.41 EUR
assets:bank -11.41 EUR
```
This changes when encountering an elided amount, for example
```
2023-01-02 Bad
expenses:food 111.00 NOK
equity:conversion -111.00 NOK
equity:conversion 11.41 EUR
assets:bank
```
Now running `hledger print --infer-costs` we output the following error
```
hledger: Error: -:1-5:
1 | 2023-01-02 Bad
| expenses:food 111.00 NOK @@ 11.41 EUR
| equity:conversion -111.00 NOK
| equity:conversion 11.41 EUR
| assets:bank -22.82 EUR
| assets:bank 111.00 NOK
This multi-commodity transaction is unbalanced.
The real postings' sum should be 0 but is: -11.41 EUR, 111.00 NOK
Consider adjusting this entry's amounts, adding missing postings,
or recording conversion price(s) with @, @@ or equity postings.
```
It seems `hledger` is adding both amounts found in the conversion
postings to the elided amount, which seems like a weird behaviour at
first, as here it unbalances a valid entry.
From the manual [1] we find
> Postings will be recognised as equity conversion postings if they are
> 1. to account(s) declared with type V (Conversion; or if no such
> accounts are declared, accounts named equity:conversion, equity:trade,
> equity:trading, or subaccounts of these)
> 2. adjacent
> 3. and exactly matching the amounts of two non-conversion postings.
It seems that the condition that is failing in the second entry is the
third one: the equity postings identified involve the amounts 111.00 NOK
and 11.41 EUR, and although the `expenses:food` posting matches the NOK
amount, there is no explicit match for the EUR one.
While the 11.41 EUR amount is implicitly stated, it seems believable
that in general this approach could not use the implicit amount, as when
running `hledger print` with the `--explicit` flag, as for a more
complicated entry the implicit amount might add up to the required
matching amount in an unexpected manner, or something similar.
What confused me is that if we add a commission cost to the examples, we
find similar results:
```
2023-01-01 Good
expenses:food 111.00 NOK
equity:conversion -111.00 NOK
equity:conversion 11.41 EUR
expenses:commission 0.57 EUR
assets:bank -11.98 EUR
2023-01-02 Bad
expenses:food 111.00 NOK
equity:conversion -111.00 NOK
equity:conversion 11.41 EUR
expenses:commission 0.57 EUR
assets:bank
```
Entries of this style are what prompted this message.
Running `hledger print --infer-costs` outputs the following error
```
hledger: Error: -:8-13:
8 | 2023-01-02 Bad
| expenses:food 111.00 NOK @@ 11.41 EUR
| equity:conversion -111.00 NOK
| equity:conversion 11.41 EUR
| expenses:commission 0.57 EUR
| assets:bank -23.39 EUR
| assets:bank 111.00 NOK
This multi-commodity transaction is unbalanced.
The real postings' sum should be 0 but is: -11.41 EUR, 111.00 NOK
Consider adjusting this entry's amounts, adding missing postings,
or recording conversion price(s) with @, @@ or equity postings.
```
In this case, it seems like the first entry should not have satisfied
the third condition from the manual, as there is no posting with the
explicit amount of 11.41 EUR, but is handled correctly regardless.
Meanwhile, the second entry fails, and again we see that `hledger` added
the conversion amounts to the elided amount, again unbalancing it.
Altogether, this leads me to believe that the transaction with the
elided amount could have been handled exactly like the explicit one,
using the amount that can be calculated with `hledger print --explicit`.
In fact, the explicit EUR amount, 11.98 = 11.41 + 0.57, is available to
at some point, as it appears implicitly in the `assets:bank` posting,
since the stated amount in the error is 23.39 = 11.98 + 11.41.
Is there a deeper reason why a transaction with an elided amount is
handled differently? Perhaps I am misunderstanding something, but given
how entries with no elided amounts are handled, it appears that the rest
could be handled in an equivalent manner.
Now, given that the behaviour involving explicit entries does not match
the description from the manual, it might be that the misbehaviour
occurs there, and not in the entries with elided amounts.
Anyway, some help here would be appreciated, as I have been logging my
finances with elided amounts using only one currency since 2019, and
only recently encountered this multi-currency scenario, and do not
really look forward rewriting them to be explicit.
Best,
Claudi
```
$ hledger --version
hledger 1.28, linux-x86_64
```
[1] https://hledger.org/1.28/hledger.html#inferring-cost-from-equity-postings
Simon Michael
Jan 19, 2023, 3:40:14 PM1/19/23
to hledger
Hi, thanks for your analysis. I agree with all of it. Indeed it's a little surprising (but good) that cost inference succeeds with your third transaction. In the second and fourth, it does seem like hledger should be able to infer the missing amount first and then successfully infer a cost. Currently it seems to try inferring costs first and inferring a missing balance second. Just now I made a code change which should do them in the other order, but for some reason it doesn't work yet. I'll share it if anyone else wants to look into this.
Simon Michael
Jan 19, 2023, 3:42:03 PM1/19/23
to hledger
On Jan 19, 2023, at 10:40, Simon Michael <si...@joyful.com> wrote:Hi, thanks for your analysis. I agree with all of it. Indeed it's a little surprising (but good) that cost inference succeeds with your third transaction. In the second and fourth, it does seem like hledger should be able to infer the missing amount first and then successfully infer a cost. Currently it seems to try inferring costs first and inferring a missing balance second. Just now I made a code change which should do them in the other order, but for some reason it doesn't work yet. I'll share it if anyone else wants to look into this.
Perhaps laziness is interfering.
diff --git a/hledger-lib/Hledger/Data/Balancing.hs b/hledger-lib/Hledger/Data/Balancing.hs
index 1fe2980d7..375726122 100644
--- a/hledger-lib/Hledger/Data/Balancing.hs
+++ b/hledger-lib/Hledger/Data/Balancing.hs
@@ -158,5 +158,9 @@ balanceTransactionHelper ::
balanceTransactionHelper bopts t = do
- (t', inferredamtsandaccts) <-
+ (t', inferredamtsandaccts) <-
+ dbg0 "cost-inferred t" $
(if infer_transaction_prices_ bopts then fmap (first transactionInferBalancingCosts) else id) $
- transactionInferBalancingAmount (fromMaybe M.empty $ commodity_styles_ bopts) t
+ dbg0 "amount-balanced t" $
+ transactionInferBalancingAmount (fromMaybe M.empty $ commodity_styles_ bopts) $
+ dbg0 "original t"
+ t
case transactionCheckBalanced bopts t' of
@@ -222,3 +226,3 @@ transactionInferBalancingAmount styles t@Transaction{tpostings=ps}
| otherwise
- = let psandinferredamts = map inferamount ps
+ = let psandinferredamts = map inferamount $ trace "inferring balancing amounts" ps
inferredacctsandamts = [(paccount p, amt) | (p, Just amt) <- psandinferredamts]
diff --git a/hledger-lib/Hledger/Data/Transaction.hs b/hledger-lib/Hledger/Data/Transaction.hs
index b57581b65..70093f6d8 100644
--- a/hledger-lib/Hledger/Data/Transaction.hs
+++ b/hledger-lib/Hledger/Data/Transaction.hs
@@ -236,3 +236,3 @@ transactionInferCostsFromEquity :: M.Map AccountName AccountType -> Transaction
transactionInferCostsFromEquity acctTypes t = first (annotateErrorWithTransaction t . T.unpack) $ do
- (conversionPairs, stateps) <- partitionPs npostings
+ (conversionPairs, stateps) <- partitionPs $ trace "inferring costs" npostings
f <- transformIndexedPostingsF addPricesToPostings conversionPairs stateps
Simon Michael
Jan 19, 2023, 3:47:36 PM1/19/23
to hledger
On Jan 19, 2023, at 10:42, Simon Michael <si...@joyful.com> wrote:On Jan 19, 2023, at 10:40, Simon Michael <si...@joyful.com> wrote:Hi, thanks for your analysis. I agree with all of it. Indeed it's a little surprising (but good) that cost inference succeeds with your third transaction. In the second and fourth, it does seem like hledger should be able to infer the missing amount first and then successfully infer a cost. Currently it seems to try inferring costs first and inferring a missing balance second. Just now I made a code change which should do them in the other order, but for some reason it doesn't work yet. I'll share it if anyone else wants to look into this.Perhaps laziness is interfering.
Hmm, the diff I sent looks bad. Ignore it, here's the new code I'm testing:
balanceTransactionHelper bopts t = do
(t', inferredamtsandaccts) <-
dbg0 "cost-inferred t" $
(if infer_transaction_prices_ bopts then fmap (first transactionInferBalancingCosts) else id) $ -- I want this to happen second
dbg0 "amount-balanced t" $
transactionInferBalancingAmount (fromMaybe M.empty $ commodity_styles_ bopts) $ -- I want this to happen first
dbg0 "original t"
t
Simon Michael
Jan 20, 2023, 2:06:22 AM1/20/23
to hledger
On Jan 19, 2023, at 10:47, Simon Michael <si...@joyful.com> wrote:balanceTransactionHelper bopts t = do
Well never mind that, names were confusing and that was the wrong piece of code. Rather, journalFinalise is the place. Currently things are done in this order:
...
1. infer costs from equity conversion postings if requested
2. balance transactions, possibly inferring a missing amount, and/or balancing costs if permitted
3. infer equity conversion postings from costs if requested
...
You would think that 1 could happen after 2. Moving it fixes your examples but breaks other tests. It seems that the current expected behaviour of tolerating excess costs generated with --infer-costs depends on inferring those costs before balancing transactions (somehow I'm not clear on).
Claudi Lleyda Moltó
Jan 20, 2023, 5:20:48 AM1/20/23
to hle...@googlegroups.com
After digging around in the code from your indications I have found this
block of code inside `transactionAddPricesFromEquity`, which is
eventually reached from `journalFinalise`.
```
-- Annotate any errors with the conversion posting pair
first (annotateWithPostings [cp1, cp2]) $
if -- If a single transaction price posting matches the conversion postings,
-- delete it from the list of priced postings in the state, delete the
-- first matching unpriced posting from the list of non-priced postings
-- in the state, and return the transformation function with the new state.
| [(np, (pricep, _))] <- matchingPricePs
, Just newpriceps <- deleteIdx np priceps
-> Right (transformPostingF np pricep, (newpriceps, otherps))
-- If no transaction price postings match the conversion postings, but some
-- of the unpriced postings match, check that the first such posting has a
-- different amount from all the others, and if so add a transaction price to
-- it, then delete it from the list of non-priced postings in the state, and
-- return the transformation function with the new state.
| [] <- matchingPricePs
, (np, (pricep, amt)):nps <- matchingOtherPs
, not $ any (amountMatches amt . snd . snd) nps
, Just newotherps <- deleteIdx np otherps
-> Right (transformPostingF np pricep, (priceps, newotherps))
-- Otherwise it's too ambiguous to make a guess, so return an error.
| otherwise -> Left "There is not a unique posting which matches the conversion posting pair:"
```
Although I have not been able to test so, it seems that the first guard
is being matched; as far as I understand, this branch does not behave as
stated in the comment, as it mentions deleting ``the first matching
unpriced posting from the list of non-priced postings'', but does not
seem to do so.
We can check that the quoted `if` statement is reached by changing the
transaction such that there is no exact match for any of the conversion
costs (here by also adding a commission to the NOK amount).
```
2023-01-02 Bad
expenses:food 110.00 NOK
assets:bank
```
In this scenario, the otherwise block is executed:
```
hledger: Error: -:1-7
There is not a unique posting which matches the conversion posting pair:
equity:conversion 11.41 EUR
equity:conversion -111.00 NOK
2023-01-02 Bad
expenses:food 110.00 NOK
assets:bank
```
This might not have anything to do with the error, but is as far as I
was able to get without compiling and running the code. Could this also
have something to do with what you said here?
> You would think that 1 could happen after 2. Moving it fixes your
> examples but breaks other tests. It seems that the current expected
> behaviour of tolerating excess costs generated with --infer-costs
> depends on inferring those costs before balancing transactions
> (somehow I'm not clear on).
Best,
Claudi
block of code inside `transactionAddPricesFromEquity`, which is
eventually reached from `journalFinalise`.
```
-- Annotate any errors with the conversion posting pair
first (annotateWithPostings [cp1, cp2]) $
if -- If a single transaction price posting matches the conversion postings,
-- delete it from the list of priced postings in the state, delete the
-- first matching unpriced posting from the list of non-priced postings
-- in the state, and return the transformation function with the new state.
| [(np, (pricep, _))] <- matchingPricePs
, Just newpriceps <- deleteIdx np priceps
-> Right (transformPostingF np pricep, (newpriceps, otherps))
-- If no transaction price postings match the conversion postings, but some
-- of the unpriced postings match, check that the first such posting has a
-- different amount from all the others, and if so add a transaction price to
-- it, then delete it from the list of non-priced postings in the state, and
-- return the transformation function with the new state.
| [] <- matchingPricePs
, (np, (pricep, amt)):nps <- matchingOtherPs
, not $ any (amountMatches amt . snd . snd) nps
, Just newotherps <- deleteIdx np otherps
-> Right (transformPostingF np pricep, (priceps, newotherps))
-- Otherwise it's too ambiguous to make a guess, so return an error.
| otherwise -> Left "There is not a unique posting which matches the conversion posting pair:"
```
Although I have not been able to test so, it seems that the first guard
is being matched; as far as I understand, this branch does not behave as
stated in the comment, as it mentions deleting ``the first matching
unpriced posting from the list of non-priced postings'', but does not
seem to do so.
We can check that the quoted `if` statement is reached by changing the
transaction such that there is no exact match for any of the conversion
costs (here by also adding a commission to the NOK amount).
```
2023-01-02 Bad
expenses:food 110.00 NOK
equity:conversion -111.00 NOK
equity:conversion 11.41 EUR
expenses:commission 0.57 EUR
expenses:commission 1.00 NOK
equity:conversion 11.41 EUR
expenses:commission 0.57 EUR
assets:bank
```
In this scenario, the otherwise block is executed:
```
hledger: Error: -:1-7
There is not a unique posting which matches the conversion posting pair:
equity:conversion 11.41 EUR
equity:conversion -111.00 NOK
2023-01-02 Bad
expenses:food 110.00 NOK
equity:conversion -111.00 NOK
equity:conversion 11.41 EUR
expenses:commission 0.57 EUR
expenses:commission 1.00 NOK
equity:conversion 11.41 EUR
expenses:commission 0.57 EUR
assets:bank
```
This might not have anything to do with the error, but is as far as I
was able to get without compiling and running the code. Could this also
have something to do with what you said here?
> You would think that 1 could happen after 2. Moving it fixes your
> examples but breaks other tests. It seems that the current expected
> behaviour of tolerating excess costs generated with --infer-costs
> depends on inferring those costs before balancing transactions
> (somehow I'm not clear on).
Claudi
Simon Michael
Jan 24, 2023, 7:13:26 PM1/24/23
to hledger
On Jan 20, 2023, at 00:20, Claudi Lleyda Moltó <claudi...@gmail.com> wrote:After digging around in the code from your indications
Thanks for that. This is just to let you know I have been working on it, finding it tricky, and it is currently paused but I hope to get it working soon.
It looks like the fix might mean supporting "redundant costs" in general - not just as a special case when --infer-costs is active. (By redundant costs I mean having @ COST notation in addition to equity conversion postings, which hledger's balancing normally does not allow.)
Simon Michael
Jan 25, 2023, 4:13:11 AM1/25/23
to hledger
Reply all
Reply to author
Forward
0 new messages