@@: Beancount vs Ledger

[Matthew Harris]

Okay, here's the situation.

2000-01-01 open Assets:Investments:AA AA

2000-01-01 open Assets:Investments:BB BB

2000-01-01 *

  Assets:Investments:AA  -1 AA @@ 1 USD

  Assets:Investments:BB  1 BB @@ 1 USD

Ledger likes this. Beancount says the first posting should have a negative cost: "@@ -1 USD". Ledger forbids negative costs.

Since Beancount's @@ syntax appears to be undocumented, is this a bug in Beancount, or should I switch to using negative costs? (I'm guessing it's a bug. Otherwise you'll need to correct for it in your Beancount-to-Ledger converter too.)

(Normally I don't use @@, but I've got some really old Fidelity 401(k) statements that are slightly ambiguous.)

Matthew

[Martin Blais]

On Wed, Jan 7, 2015 at 11:21 PM, Matthew Harris <mharr...@gmail.com> wrote:

Okay, here's the situation.

2000-01-01 open Assets:Investments:AA AA

2000-01-01 open Assets:Investments:BB BB

2000-01-01 *

  Assets:Investments:AA  -1 AA @@ 1 USD

  Assets:Investments:BB  1 BB @@ 1 USD

Ledger likes this. Beancount says the first posting should have a negative cost: "@@ -1 USD". Ledger forbids negative costs.

The value after @@ is not a cost. This is a price conversion. This is an important distinction: the units of AA will not have a cost basis...

Anyhow, you have to put the negative amount on the right.

Hmm, maybe I should check for that and disallow sign reversals.

I'll add a check that the signs are the same.

Since Beancount's @@ syntax appears to be undocumented, is this a bug in Beancount, or should I switch to using negative costs? (I'm guessing it's a bug. Otherwise you'll need to correct for it in your Beancount-to-Ledger converter too.)

I'll have to document that feature too.

Here's what I do for the price:

https://bitbucket.org/blais/beancount/src/317b0a91b8d584380b0b750ae438f8af334ef911/src/python/beancount/parser/parser.py?at=default#cl-511

istotal = True

For the cost I just divide the amount by the number of units:

https://bitbucket.org/blais/beancount/src/317b0a91b8d584380b0b750ae438f8af334ef911/src/python/beancount/parser/parser.py?at=default#cl-290

 

(Normally I don't use @@, but I've got some really old Fidelity 401(k) statements that are slightly ambiguous.)

Hmmm, there's really no way you can back out some sort of reasonable cost basis for them?

 

[Matthew S. Harris]

On Wed Jan 07 2015 at 9:25:28 PM Martin Blais <bl...@furius.ca> wrote:

The value after @@ is not a cost. This is a price conversion. This is an important distinction: the units of AA will not have a cost basis...

I think we're saying the same thing with different words. The Ledger manual calls it a total posting cost, and the Ledger error message is "A posting's cost may not be negative."

The lack of a cost basis here is the least of my worries; I'm actually using @ instead of {} throughout my file because lots have caused me nothing but trouble so far. In general, I've got 401(k) contributions every other week and dividends as often as every month and up to eight different buckets in my 401(k) even if they're all a single target-date fund, and if each was recorded as a separate lot, I'd have a big mess every time the 401(k) plan replaced one fund with another, and I don't even know what I'd do with the quarterly account fee, which you've already discussed in your proposal for improving inventory booking. At least it's all in a tax-deferred account where it doesn't matter much. Maybe lots will be more useful with my taxable account.

 

Anyhow, you have to put the negative amount on the right.

Be aware that Ledger is different in that regard then. In your code to export to Ledger format, you'll need to flip the sign.

 

(Normally I don't use @@, but I've got some really old Fidelity 401(k) statements that are slightly ambiguous.)

Hmmm, there's really no way you can back out some sort of reasonable cost basis for them?

No, the old Fidelity 401(k) statements are close to useless. At least with my wife's statements, which led to this example, I have transactions with numbers of shares and total transaction costs. A number of funds are specific to the plan and don't have publicly available prices. With my (older) statements, I don't even have transactions; I have the number of shares and the share price for each month, plus my contributions, employer contributions, change in market value, and dividends, all only in dollar amounts for the month. I haven't bothered to Ledgerize those statements because there doesn't seem to be enough to go on.

Matthew

[Martin Michlmayr]

* Martin Blais <bl...@furius.ca> [2015-01-08 22:48]:

> > 2000-01-01 *
> > Assets:Investments:AA -1 AA @@ 1 USD
> > Assets:Investments:BB 1 BB @@ 1 USD
> >

> The value after @@ is not a cost. This is a price conversion.
>

> Anyhow, you have to put the negative amount on the right.

Does it make sense to have a negative price conversion?

--
Martin Michlmayr
http://www.cyrius.com/

[Matthew Harris]

On Friday, January 9, 2015 at 10:53:49 AM UTC-8, Martin Michlmayr wrote:

* Martin Blais <bl...@furius.ca> [2015-01-08 22:48]:
> > 2000-01-01 *
> >   Assets:Investments:AA  -1 AA @@ 1 USD
> >   Assets:Investments:BB  1 BB @@ 1 USD
> >
> The value after @@ is not a cost. This is a price conversion.
>
> Anyhow, you have to put the negative amount on the right.

Does it make sense to have a negative price conversion? 

It depends on your point of view. Now that I discovered Beancount's view, I think it does (the total price of the posting is a negative dollar amount), but Ledger obviously took the other view and was here first. I can deal with it either way, but I wanted to see whether it was deliberate because it isn't documented and maybe nobody's really using it yet.

[Martin Blais]

On Thu, Jan 8, 2015 at 2:47 AM, Matthew S. Harris <mharr...@gmail.com> wrote:

Just FYI, you sent this directly to me instead of via the group.

Thx for letting me know. That's by accident, I often trip up over gmail's interface because I run my email through a proxy to remove references to the backend and gmail doesn't like 100% of that.

On Wed Jan 07 2015 at 9:25:28 PM Martin Blais <bl...@furius.ca> wrote:

The value after @@ is not a cost. This is a price conversion. This is an important distinction: the units of AA will not have a cost basis...

I think we're saying the same thing with different words. The Ledger manual calls it a total posting cost, and the Ledger error message is "A posting's cost may not be negative."

Do you prefer that semantic? I could switch it around (but I wouldn't call it a cost).  I'm indifferent. Maybe it's better to match Ledger just for the sake of people converting, plus in a sense, it would match the semantics for the cost. Hmmmm I think I should do that...

The lack of a cost basis here is the least of my worries; I'm actually using @ instead of {} throughout my file because lots have caused me nothing but trouble so far. In general, I've got 401(k) contributions every other week and dividends as often as every month and up to eight different buckets in my 401(k) even if they're all a single target-date fund, and if each was recorded as a separate lot, I'd have a big mess every time the 401(k) plan replaced one fund with another, and I don't even know what I'd do with the quarterly account fee, which you've already discussed in your proposal for improving inventory booking. At least it's all in a tax-deferred account where it doesn't matter much. Maybe lots will be more useful with my taxable account.

I have a 401k account with a similar setup (maybe a bit less buckets), and I track lots with cost basis despite the fact they won't be taxed, because that allows me to compute the returns. It also works out fine for the fees. You just have to declare those account's default booking method as "NONE" to disable strict booking.

Despite this, I still recommend that you use a cost, even if it's incorrect (since this is not taxed, this is no worse than using a price conversion... just bring up the EOY price for that year and use that maybe, and make sure to disable strict booking for those accounts so you can book sales without having to worry about matching lots); price conversions are really just for that: conversions of currencies, there should be few of them. Using a cost will allow you to compute market value for those funds, and I think there is some code in holdings reporting that might treat "things held at cost" vs. "non-operating currencies" somewhat differently (though not incorrectly). Your fund units not held at cost will look like "non-operating currencies" to this code, just saying, there's only so much information the holdings reporting has access to when it looks at the contents of inventories and is requested to make aggregations. I think what you're doing now will work anyway though. Ideally I'd have to review the code with an eye towards that to make sure. 

Also, when closing periods, there is an automatic insertion of a correcting entry for foreign currency conversions so that the sum total is precisely an empty inventory... (that's one of the things that drove me crazy with Ledger, I want a precise empty balance when summing all the transactions, so I account for it explicitly, but I think eventually I'll be auto-inserting entries as per the Trading Accounts method).  Your mutual fund units will appear in there, which might be annoying to you (In Equity:Conversions accounts). Again, not incorrect, but you'd want to avoid that ideally.

I really should complete that explanation more clearly, I'll write more about this in the DE intro doc instead.

 

Anyhow, you have to put the negative amount on the right.

Be aware that Ledger is different in that regard then. In your code to export to Ledger format, you'll need to flip the sign.

 

(Normally I don't use @@, but I've got some really old Fidelity 401(k) statements that are slightly ambiguous.)

Hmmm, there's really no way you can back out some sort of reasonable cost basis for them?

No, the old Fidelity 401(k) statements are close to useless. 

At least with my wife's statements, which led to this example, I have transactions with numbers of shares and total transaction costs. A number of funds are specific to the plan and don't have publicly available prices. With my (older) statements, I don't even have transactions; I have the number of shares and the share price for each month, plus my contributions, employer contributions, change in market value, and dividends, all only in dollar amounts for the month. I haven't bothered to Ledgerize those statements because there doesn't seem to be enough to go on.

Note that there is code in LedgerHub that you could leverage to fetch prices at required dates, you could write a little Python script that finds relevant transactions and fetches the prices from G.Finance for all the dates you need and then either automate the repricing of those transactions or change them manually.

One thing I want to do soon is to issue warnings when the price database is looked up and the price point is too far from the requested date, so that the user could go fill in the missing prices. I'd probably issue price entries with the approximated price (approximated with a distant date) and then feed that into another script that would fetch prices for those directives.

[Martin Blais]

No it doesn't.

The signs are intended to match.

I think I'll switch it around so that rate is always positive, so it matches Ledger semantics and disallow a sign on the exchange change.

[Matthew Harris]

On Friday, January 9, 2015 at 7:08:29 PM UTC-8, Martin Blais wrote:

On Thu, Jan 8, 2015 at 2:47 AM, Matthew S. Harris <mharr...@gmail.com> wrote:

On Wed Jan 07 2015 at 9:25:28 PM Martin Blais <bl...@furius.ca> wrote:

The value after @@ is not a cost. This is a price conversion. This is an important distinction: the units of AA will not have a cost basis...

I think we're saying the same thing with different words. The Ledger manual calls it a total posting cost, and the Ledger error message is "A posting's cost may not be negative."

Do you prefer that semantic? I could switch it around (but I wouldn't call it a cost).  I'm indifferent. Maybe it's better to match Ledger just for the sake of people converting, plus in a sense, it would match the semantics for the cost. Hmmmm I think I should do that...

My gut says you should try to match Ledger except where you gain something by being different. The gain isn't clear here, so I assumed this was just a bug.

 

I have a 401k account with a similar setup (maybe a bit less buckets), and I track lots with cost basis despite the fact they won't be taxed, because that allows me to compute the returns. It also works out fine for the fees. You just have to declare those account's default booking method as "NONE" to disable strict booking.

Ah. You should document this feature in the language manual. :-) After you described it, I found it in the CHANGES file.

 

Despite this, I still recommend that you use a cost, even if it's incorrect (since this is not taxed, this is no worse than using a price conversion... just bring up the EOY price for that year and use that maybe, and make sure to disable strict booking for those accounts so you can book sales without having to worry about matching lots); price conversions are really just for that: conversions of currencies, there should be few of them. Using a cost will allow you to compute market value for those funds, and I think there is some code in holdings reporting that might treat "things held at cost" vs. "non-operating currencies" somewhat differently (though not incorrectly). Your fund units not held at cost will look like "non-operating currencies" to this code, just saying, there's only so much information the holdings reporting has access to when it looks at the contents of inventories and is requested to make aggregations. I think what you're doing now will work anyway though. Ideally I'd have to review the code with an eye towards that to make sure.

You say it works for you, so I'll start doing that and ask for help if I get stuck. So I should use {{}} syntax here, where I have the total posting cost but not a fund price?

 

Note that there is code in LedgerHub that you could leverage to fetch prices at required dates, you could write a little Python script that finds relevant transactions and fetches the prices from G.Finance for all the dates you need and then either automate the repricing of those transactions or change them manually.

Yeah, I think I considered fetching prices manually at one point, but some of these funds are specific to the 401(k) plan, so there are no prices available. It didn't seem worth it to have prices for some funds and not others, so I dropped it.

I haven't really looked into LedgerHub yet.

 

One thing I want to do soon is to issue warnings when the price database is looked up and the price point is too far from the requested date, so that the user could go fill in the missing prices. I'd probably issue price entries with the approximated price (approximated with a distant date) and then feed that into another script that would fetch prices for those directives.

Sounds like a good idea.

Matthew

[Martin Blais]

On Sat, Jan 10, 2015 at 12:09 AM, Matthew Harris <mharr...@gmail.com> wrote:

On Friday, January 9, 2015 at 7:08:29 PM UTC-8, Martin Blais wrote:

On Thu, Jan 8, 2015 at 2:47 AM, Matthew S. Harris <mharr...@gmail.com> wrote:

On Wed Jan 07 2015 at 9:25:28 PM Martin Blais <bl...@furius.ca> wrote:

The value after @@ is not a cost. This is a price conversion. This is an important distinction: the units of AA will not have a cost basis...

I think we're saying the same thing with different words. The Ledger manual calls it a total posting cost, and the Ledger error message is "A posting's cost may not be negative."

Do you prefer that semantic? I could switch it around (but I wouldn't call it a cost).  I'm indifferent. Maybe it's better to match Ledger just for the sake of people converting, plus in a sense, it would match the semantics for the cost. Hmmmm I think I should do that...

My gut says you should try to match Ledger except where you gain something by being different. The gain isn't clear here, so I assumed this was just a bug.

I'll make the change soon (added to TODO file, will appear in CHANGES and announcement once it's in).

 

I have a 401k account with a similar setup (maybe a bit less buckets), and I track lots with cost basis despite the fact they won't be taxed, because that allows me to compute the returns. It also works out fine for the fees. You just have to declare those account's default booking method as "NONE" to disable strict booking.

Ah. You should document this feature in the language manual. :-) After you described it, I found it in the CHANGES file.

Done, thanks.

 

 

Despite this, I still recommend that you use a cost, even if it's incorrect (since this is not taxed, this is no worse than using a price conversion... just bring up the EOY price for that year and use that maybe, and make sure to disable strict booking for those accounts so you can book sales without having to worry about matching lots); price conversions are really just for that: conversions of currencies, there should be few of them. Using a cost will allow you to compute market value for those funds, and I think there is some code in holdings reporting that might treat "things held at cost" vs. "non-operating currencies" somewhat differently (though not incorrectly). Your fund units not held at cost will look like "non-operating currencies" to this code, just saying, there's only so much information the holdings reporting has access to when it looks at the contents of inventories and is requested to make aggregations. I think what you're doing now will work anyway though. Ideally I'd have to review the code with an eye towards that to make sure.

You say it works for you, so I'll start doing that and ask for help if I get stuck. So I should use {{}} syntax here, where I have the total posting cost but not a fund price?

Haaa yes, that'll make it easier, this way you won't have to calculate the per-unit price.

I'll admit I'm using per-unit prices everywhere exclusively in my own file.

{{...}} may be a little bit undertested.

Please LMK if you encounter any turbulence.

 

Note that there is code in LedgerHub that you could leverage to fetch prices at required dates, you could write a little Python script that finds relevant transactions and fetches the prices from G.Finance for all the dates you need and then either automate the repricing of those transactions or change them manually.

Yeah, I think I considered fetching prices manually at one point, but some of these funds are specific to the 401(k) plan, so there are no prices available. It didn't seem worth it to have prices for some funds and not others, so I dropped it.

Surprised. I've looked up all my funds from my 401k and a few nights ago helped out my girlfriend selecting hers (from a list on a dusty ADP website) and looking up different options and they were all available as 5-letter tickers, I didn't even have to use the "MUTF:" prefix to get them to match. All were on Google Finance.

I haven't really looked into LedgerHub yet.

 

One thing I want to do soon is to issue warnings when the price database is looked up and the price point is too far from the requested date, so that the user could go fill in the missing prices. I'd probably issue price entries with the approximated price (approximated with a distant date) and then feed that into another script that would fetch prices for those directives.

Sounds like a good idea.

It's a common task, I've found myself often needing to go through this.

There are two parts to it:

1. Generating a list of needed prices. Beancount is suitable for that. The ideal output should be Beancount price directives itself for simplicity.

2. Reading a list of needed prices and fetching them from somewhere. This is beyond Beancount domain (Beancount operates only on the text file, no external interactions, I put that in the "dirty world" category that needs integration scripts). Once again, spitting out the price directives in Beancount format for easy copy-paste or inclusion (when that's supported) is nice.

 

[Martin Blais]

Alright, so looking at this whole thing a little bit closer and writing some unit test I realized Beancount's @ and @@ price conversion syntax is not even consistent with itself. For example, both the following postings result in prices of 200 USD and -200 USD, respectively:

          2013-05-18 * ""                                                                

            Assets:Investments:MSFT      -10 MSFT @ 200.00 USD                           

            Assets:Investments:MSFT      -10 MSFT @@ 2000.00 USD                         

So using a single @ takes a positive number while using the total variant seems to require the same sign. That's wrong. I will fix this right away so they're consistent: both @ and @@ syntaxes should have price or total price amounts that are unsigned. Using a sign on the right side will trigger an error.

Sorry about this. This oversight reflects the fact that I'm not using those total costs very often in my own ledger file and that the unit test did not specifically check for signs.

Thanks again Matthew for pointing it out.

[Martin Blais]

I made the fix in a branch:

https://bitbucket.org/blais/beancount/commits/4ea5c58a16ccc8a06aa0be8eb4e030fbc68ab447

Are you comfortable with me merging this into the mainline stable branch?

This means that you'd have to convert your input files to remove signs off after @@. You could potentially automate it with sed, but I just made the changes on my 8 years of data manually in less than a minute here, iterating using Emacs and I presume yours would be similarly easy. Signed values for @@ will automatically cause an error and you can jump and fix between them using next-error. This won't break anything else. 

(Granted if I had versioning at this point I would bake a new one, but I'm still operating on the basis of "default is stable" thanks to a large battery of unit tests and this simplified process has worked well so far.)

Any feedback from users appreciated,

[Matthew Harris]

Yes, I've been doing lots of Emacs regex replacements to whip my Ledger files into shape for Beancount, so this is no problem. I only have a few transactions affected by this anyway.

Matthew

[Martin Blais]

Alright, I want this fix in right away, as it is a bug, so what I've done is that I've merged the change in default but I've added an environment variable to revert to the old behaviour.  

Option 1. Pull and update to the get the fix by default. You will have to remove the signs on negative prices in your input files. This should be easy: run bean-check and each of these will raise an error.

Option 2. Pull and update, but set the environment variable BEANCOUNT_ALLOW_NEGATIVE_PRICES to anything to keep the old (broken) behaviour. Use this if you don't want to be bothered with making the changes right now (and apologies in advance if this causes anyone friction in trying to get work done). Note that this workaround is a temporary measure, and I'll be removing it in a few weeks.

I added a note about this in the documentation, at the bottom of this section: 

https://docs.google.com/document/d/1wAMVrKIA2qtRGmoVDSUBJGmYZSygUaR0uOMW1GV3YE0/edit#heading=h.mtqrwt24wnzs

Any further question about this, please hit the list.

Thanks again for reporting this Matthew, 

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/29d783a5-d5fe-4697-b41b-982a6695e6c5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Matthew Harris]

  

Despite this, I still recommend that you use a cost, even if it's incorrect (since this is not taxed, this is no worse than using a price conversion... just bring up the EOY price for that year and use that maybe, and make sure to disable strict booking for those accounts so you can book sales without having to worry about matching lots); price conversions are really just for that: conversions of currencies, there should be few of them. Using a cost will allow you to compute market value for those funds, and I think there is some code in holdings reporting that might treat "things held at cost" vs. "non-operating currencies" somewhat differently (though not incorrectly). Your fund units not held at cost will look like "non-operating currencies" to this code, just saying, there's only so much information the holdings reporting has access to when it looks at the contents of inventories and is requested to make aggregations. I think what you're doing now will work anyway though. Ideally I'd have to review the code with an eye towards that to make sure.

You say it works for you, so I'll start doing that and ask for help if I get stuck. So I should use {{}} syntax here, where I have the total posting cost but not a fund price?

Haaa yes, that'll make it easier, this way you won't have to calculate the per-unit price.

I'll admit I'm using per-unit prices everywhere exclusively in my own file.

{{...}} may be a little bit undertested.

Please LMK if you encounter any turbulence.

Okay, here's a stripped-down example that's confusing me. I roll money into a rollover IRA with a cost basis. (I was able to exactly compute the cost basis for the source account because it never had any fees deducted.) Then, using the current price as the fake cost, some fees are deducted, and all of my ABC are converted to DEF at a slightly different price. With this file, I get an error that "A valid price for ABC/USD could not be found", and the bean-web balance sheet shows a negative USD balance in the ABC account. What's going on?

plugin "beancount.plugins.unrealized" "Unrealized"

2000-01-01 open Assets:Investments:IRA:XXX

2000-01-01 open Assets:Investments:RolloverIRA:ABC ABC "NONE"

2000-01-01 open Assets:Investments:RolloverIRA:DEF DEF "NONE"

2000-01-01 open Expenses:Fees

2000-01-01 open Expenses:RoundingError

2012-09-25 *

  Assets:Investments:RolloverIRA:ABC  990.216 ABC {{26494.41725 USD}} @ 32.11 USD

  Assets:Investments:IRA:XXX

2012-12-05 * "Fee"

  Assets:Investments:RolloverIRA:ABC  -0.022 ABC {32.12 USD} @ 32.12 USD

  Expenses:Fees

2013-03-05 * "Fee"

  Assets:Investments:RolloverIRA:ABC  -0.019 ABC {34.77 USD} @ 34.77 USD

  Expenses:Fees

2013-04-01 *

  Assets:Investments:RolloverIRA:ABC  -990.175 ABC {35.06 USD} @ 35.06 USD

  Assets:Investments:RolloverIRA:DEF  926.984 DEF {37.45 USD} @ 37.45 USD

  Expenses:RoundingError  -0.02 USD

2013-04-02 balance Assets:Investments:RolloverIRA:ABC  0.000 ABC

[Martin Blais]

Thanks for this detailed example problem Matthew.

Before we start, here's the final balance report from Beancount:

blais-macbookair:~/p$ bean-report   bug.beancount  balances 

Assets:Investments:IRA:XXX           -26,494.42 USD

Assets:Investments:RolloverIRA:DEF       926.98 DEF

Equity                             

Expenses:Fees                              1.37 USD

Expenses:RoundingError                    -0.02 USD

Income                             

Liabilities                        

Which is the same (minus reporting differences) you'd get out of Ledger:

blais-macbookair:~/p$ ledger -f bug.beancount  bal

         926.984 DEF

       -26494.42 USD  Assets:Investments

       -26494.42 USD    IRA:XXX

         926.984 DEF    RolloverIRA:DEF

            1.35 USD  Expenses

            1.37 USD    Fees

           -0.02 USD    RoundingError

--------------------

         926.984 DEF

       -26493.07 USD

What is happening is that when you're using the "NONE" booking method, Beancount disables the inventory booking verification, so it allows you to deduct any lot at any cost anywhere (like Ledger).  In your case, you're 

As for the 

"A valid price for ABC/USD could not be found"

message: it is emitted from the unrealized gains plugin.

I'll have to hunt it down, sounds like a bug.

[Martin Blais]

(Sorry I fat fingered this, intended to send it tonight, will reply later.)

[Martin Blais]

On Tue, Jan 13, 2015 at 2:07 PM, Martin Blais <bl...@furius.ca> wrote:

Alright, sorry for the delay, I haven't had much time post-work.

There is no bug.

Here's the thing: when you use the default booking method of NONE, you're disabling checks for booking against existing lots. As you can see above, this is the more or less the same thing that Ledger does: it allows unmatched lots and combinations of positive and negative lots. The result is that you're able to "leak" cost basis, and in this case you liquidate all the units in your account and you still have some cost basis in there, even negative cost basis (this is an impossible situation in real life and is why I'm doing lot checking, and I've pointed this out on the Ledger mailing-list a few times but it doesn't seem to raise the hair off anyone's back; personally I don't see the point of the exercise of entering my transactions if my data is going to be all wrong because my software does not verify its correctness, but we'll leave that story for another day). But at the moment, I don't yet support average cost booking which would solve you're problem elegantly, so we go the Ledger way (using the "NONE" booking to disable checks). Such is this precarious situation.

Here's the negative number you should be seeing in bean-web, that leaked cost basis:

bean-query /home/blais/none.beancount "select account, sum(cost(position)) where account ~ ':ABC' group by 1"                                                                                                                                                                                                                 

             account                       sum_cost_position

---------------------------------- ---------------------------------

Assets:Investments:RolloverIRA:ABC -8222.48552000000000000000000 USD

But... the account is empty, in terms of the number of units of ABC:

bean-query /home/blais/none.beancount "select account, sum(units(position)) where account ~ ':ABC' group by 1"

             account

---------------------------------- -

Assets:Investments:RolloverIRA:ABC

Now, here's why: here are all the lots you have in that inventory, here's the detail:

bean-query /home/blais/none.beancount "select account, position where account ~ ':ABC'"

             account                       position

---------------------------------- ------------------------

Assets:Investments:RolloverIRA:ABC  990.216 ABC {26.75 USD}

Assets:Investments:RolloverIRA:ABC   -0.022 ABC {32.12 USD}Assets:Investments:RolloverIRA:ABC   -0.019 ABC {34.77 USD}Assets:Investments:RolloverIRA:ABC -990.175 ABC {35.06 USD}

Sum up the costs and you will obtain -8222.4855 USD...

Such is life under the Ledger model.

BTW, if you want to get the context around any transaction, there's a handy bean-doctor subcommand you can use to find that out where you can provide the filename and line number, e.g. on the last transaction:

bean-doctor context /home/blais/none2.beancount 23

Hash:7f8d0d8f296d81dd11df161a525b3d22

Location: /home/blais/none2.beancount:21

;   Assets:Investments:RolloverIRA:ABC    990.216 ABC {26.75619990991864401302342115 USD}

;   Assets:Investments:RolloverIRA:ABC                             -0.022 ABC {32.12 USD}

;   Assets:Investments:RolloverIRA:ABC                             -0.019 ABC {34.77 USD}

2013-04-01 *

  Assets:Investments:RolloverIRA:ABC  -990.175 ABC {35.06 USD} @ 35.06000 USD  ; -34715.54 USD

  Assets:Investments:RolloverIRA:DEF   926.984 DEF {37.45 USD} @ 37.45000 USD  ;  34715.55 USD

  Expenses:RoundingError                 -0.02 USD                             ;     -0.02 USD

;   Assets:Investments:RolloverIRA:ABC    990.216 ABC {26.75619990991864401302342115 USD}

;   Assets:Investments:RolloverIRA:ABC                             -0.022 ABC {32.12 USD}

;   Assets:Investments:RolloverIRA:ABC                             -0.019 ABC {34.77 USD}

; ! Assets:Investments:RolloverIRA:ABC                           -990.175 ABC {35.06 USD}

This is bound to "C-c x" under beancount-mode and it uses the (point) location. So you put your cursor around the transaction, hit C-c x, and you get the contents of the inventories before and after the transaction is applied, as well as the parsed and completed amount for the transaction. It's useful for debugging.

Now, some observations:

- The Right Thing to do would be to implement average cost booking. I'm working on it. It'll be there eventually, I find this "ignore booking" situation intolerable myself. When I do you'll be able to book at average cost using the {*...} syntax and it'll automatically calculate the per-share price to fulfill all the conditions and not leak any cost basis yet do what you want, and automate the calculations.

- In the meantime, you can just reduce the lot at its remaining cost basis (sum up the cost bases and you will find 26493.04998 USD) and your cost basis will cancel to zero and your DEF position will have the correct cost basis (assuming you did not realize any gain when you made that conversion):

2012-09-25 *

  Assets:Investments:RolloverIRA:ABC  990.216 ABC {{26494.41725 USD}} @ 32.11 USD

  Assets:Investments:IRA:XXX

2012-12-05 * "Fee"

  Assets:Investments:RolloverIRA:ABC  -0.022 ABC {32.12 USD} @ 32.12 USD

  Expenses:Fees

2013-03-05 * "Fee"

  Assets:Investments:RolloverIRA:ABC  -0.019 ABC {34.77 USD} @ 34.77 USD

  Expenses:Fees

2013-04-01 *

  Assets:Investments:RolloverIRA:ABC  -990.175 ABC {{26493.04998 USD}} @ 35.06 USD

  Assets:Investments:RolloverIRA:DEF  926.984 DEF {{26493.04998 USD}} @ 37.45 USD

2013-04-02 balance Assets:Investments:RolloverIRA:ABC  0.000 ABC

It's not ideal from the POV of data entry (it forces you to calculate the cost basis), but it'll work. Solving the problem correctly is not trivial (which is likely why Ledger did not implement it in the first place) but my new inventory booking method is a good general solution for this and hopefully will make such situations easy to handle without having to disable any checks.

- Finally, an observation: on an entry like this where the price equals the cost basis:

  Assets:Investments:RolloverIRA:ABC  -0.022 ABC {32.12 USD} @ 32.12 USD

the price is superfluous. If you simply enter this:

  Assets:Investments:RolloverIRA:ABC  -0.022 ABC {32.12 USD}

You obtain the same result. But don't trust me, print out the generated entries, with bean-report <filename> print and you will see the corresponding price entry appear.

As for the 

"A valid price for ABC/USD could not be found"

message: it is emitted from the unrealized gains plugin.

I'll have to hunt it down, sounds like a bug.

This is indeed, a bug triggered by this situation and I'll have to fix it.

The unrealized gains plugin computes the list of holdings in order to insert the gains, and it ends up with a holding object like this for your leaked cost basis:

Holding(account='Assets:Investments:RolloverIRA:ABC', number=Decimal('0.000'), currency='ABC', cost_number=None, cost_currency='USD', book_value=Decimal('-8222.48552000000000000000000'), market_value=None, price_number=None, price_date=datetime.date(2013, 4, 1))

Now, it expects that if the commodity is not a currency, it will have a price. But price_number is None, so it complains about it.

I'll fix this over the weekend; it should be silent when that occurs, I'll have to find conditions for that.

In any case, using the method I suggest for you above, there is no error.

I hope this helps,

[Matthew Harris]

That does mostly help. I agree with your description of the current situation as "precarious" and "intolerable"; it shouldn't be so much work to make closed accounts worth $0. :-) Your workaround of computing the cost basis has been mostly effective, though my balance report now shows a bunch of accounts with value 0.00 USD and -0.00 USD, presumably due to rounding issues. bean-doctor has been helpful here, though because I'm doing this workaround a lot, I wonder if it should compute the total cost for me.

I recently discovered one interesting case in my records: In my old IRA account, I received dividends on the 25th of every month. On the 24th of one month, I transferred everything out of the account, including the dividends that didn't arrive until the next day. I'm not sure how that's possible. Maybe the 25th was just the settlement date and the statements were unclear, not that that fully explains it. Gotta love old statements.

[Martin Blais]

On Wed, Jan 21, 2015 at 5:23 PM, Matthew Harris <mharr...@gmail.com> wrote:

That does mostly help. I agree with your description of the current situation as "precarious" and "intolerable"; it shouldn't be so much work to make closed accounts worth $0. :-)

So how were you booking these transactions with Ledger?

I can only assume that the errors were going undetected (but I might be wrong).

 

Your workaround of computing the cost basis has been mostly effective, though my balance report now shows a bunch of accounts with value 0.00 USD and -0.00 USD, presumably due to rounding issues. bean-doctor has been helpful here, though because I'm doing this workaround a lot, I wonder if it should compute the total cost for me.

Hmmm interesting.

Do you think you'd be able to build some minimal example that reproduces the 0.00 problem that you could share with me?

I recently discovered one interesting case in my records: In my old IRA account, I received dividends on the 25th of every month. On the 24th of one month, I transferred everything out of the account, including the dividends that didn't arrive until the next day. I'm not sure how that's possible. Maybe the 25th was just the settlement date and the statements were unclear, not that that fully explains it. Gotta love old statements.

Probably settlement.

Note that both Ledger and Beancount will not bitch about an Asset account's balance going under zero, so from their perspective you will not get an error, just a temporary negative balance.

(OH! That could be an interesting plugin idea to add! Yet another constraint. You could configure it with a tolerance.)

 

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/b396ecfa-ba4f-41a9-bfd0-31ac063daf9c%40googlegroups.com.

[Matthew Harris]

On Wednesday, January 21, 2015 at 6:54:06 PM UTC-8, Martin Blais wrote:

On Wed, Jan 21, 2015 at 5:23 PM, Matthew Harris <mharr...@gmail.com> wrote:

That does mostly help. I agree with your description of the current situation as "precarious" and "intolerable"; it shouldn't be so much work to make closed accounts worth $0. :-)

So how were you booking these transactions with Ledger?

I can only assume that the errors were going undetected (but I might be wrong).

I used Ledger to track the numbers and prices of shares, not the basis. I used "20 ABCD @ $12.00" syntax. "ledger bal" showed me the numbers of shares, and "ledger bal -V" showed me dollar amounts. The dollar amount it output was the number of shares times the share price, so when an account was empty, its value was $0. That was the extent of Ledger's abilities, and it was sufficient for tracking expenses and plotting account balances. Beancount is pushing me to be more strict, which will be great at some point. Right now it's mostly more challenging.

 

Your workaround of computing the cost basis has been mostly effective, though my balance report now shows a bunch of accounts with value 0.00 USD and -0.00 USD, presumably due to rounding issues. bean-doctor has been helpful here, though because I'm doing this workaround a lot, I wonder if it should compute the total cost for me.

Hmmm interesting.

Do you think you'd be able to build some minimal example that reproduces the 0.00 problem that you could share with me?

Sure. I stripped it down to a single account, and it does indeed appear to be a rounding issue. I'm not claiming it's a bug. I'll mail it to you separately.

I recently discovered one interesting case in my records: In my old IRA account, I received dividends on the 25th of every month. On the 24th of one month, I transferred everything out of the account, including the dividends that didn't arrive until the next day. I'm not sure how that's possible. Maybe the 25th was just the settlement date and the statements were unclear, not that that fully explains it. Gotta love old statements.

Probably settlement.

So the 25th was the settlement date for the dividends but the 25th was only the trade date for transferring out of the account? That seems inconsistent. The dates in the statements should either be the trade date or the settlement date.

 

Note that both Ledger and Beancount will not bitch about an Asset account's balance going under zero, so from their perspective you will not get an error, just a temporary negative balance.

(OH! That could be an interesting plugin idea to add! Yet another constraint. You could configure it with a tolerance.)

These warning plugins feel like lint warnings. It'd be nice to be able to selectively disable warnings on transactions or postings, with tags or metadata, so I can still get the benefit of these warnings even if one transaction out of the entire file necessarily violates it.

[Matthew Harris]

On Wednesday, January 14, 2015 at 7:48:51 PM UTC-8, Martin Blais wrote:

Now, some observations:

- The Right Thing to do would be to implement average cost booking. I'm working on it. It'll be there eventually, I find this "ignore booking" situation intolerable myself. When I do you'll be able to book at average cost using the {*...} syntax and it'll automatically calculate the per-share price to fulfill all the conditions and not leak any cost basis yet do what you want, and automate the calculations.

- In the meantime, you can just reduce the lot at its remaining cost basis (sum up the cost bases and you will find 26493.04998 USD) and your cost basis will cancel to zero and your DEF position will have the correct cost basis (assuming you did not realize any gain when you made that conversion):

It's not ideal from the POV of data entry (it forces you to calculate the cost basis), but it'll work. Solving the problem correctly is not trivial (which is likely why Ledger did not implement it in the first place) but my new inventory booking method is a good general solution for this and hopefully will make such situations easy to handle without having to disable any checks.

FYI, I gave up on this manual average-cost-booking hack a few weeks ago. When my 401(k) has contributions every couple weeks and a fee every quarter, the manual effort required quickly becomes intolerable. I'm waiting for {*} before I finish correcting the cost basis in my investment accounts.

As for the 

"A valid price for ABC/USD could not be found"

message: it is emitted from the unrealized gains plugin.

I'll have to hunt it down, sounds like a bug.

This is indeed, a bug triggered by this situation and I'll have to fix it.

The unrealized gains plugin computes the list of holdings in order to insert the gains, and it ends up with a holding object like this for your leaked cost basis:

Would you like me to file a bug? A bunch of these pollute my bean-check output, which discourages me from running bean-check as often as I should.

I see you recently updated export_reports.py to use commodity metadata. What should the "export" and "ticker" metadata fields contain? What's the difference between "CASH" and "MONEY"? What metadata should I apply to a commodity that has no ticker symbol?

Matthew

[Martin Blais]

On Wed, Feb 11, 2015 at 1:48 PM, Matthew Harris <mharr...@gmail.com> wrote:

On Wednesday, January 14, 2015 at 7:48:51 PM UTC-8, Martin Blais wrote:

Now, some observations:

- The Right Thing to do would be to implement average cost booking. I'm working on it. It'll be there eventually, I find this "ignore booking" situation intolerable myself. When I do you'll be able to book at average cost using the {*...} syntax and it'll automatically calculate the per-share price to fulfill all the conditions and not leak any cost basis yet do what you want, and automate the calculations.

- In the meantime, you can just reduce the lot at its remaining cost basis (sum up the cost bases and you will find 26493.04998 USD) and your cost basis will cancel to zero and your DEF position will have the correct cost basis (assuming you did not realize any gain when you made that conversion):

It's not ideal from the POV of data entry (it forces you to calculate the cost basis), but it'll work. Solving the problem correctly is not trivial (which is likely why Ledger did not implement it in the first place) but my new inventory booking method is a good general solution for this and hopefully will make such situations easy to handle without having to disable any checks.

FYI, I gave up on this manual average-cost-booking hack a few weeks ago. When my 401(k) has contributions every couple weeks and a fee every quarter, the manual effort required quickly becomes intolerable. I'm waiting for {*} before I finish correcting the cost basis in my investment accounts.

Fair enough.

As for the 

"A valid price for ABC/USD could not be found"

message: it is emitted from the unrealized gains plugin.

I'll have to hunt it down, sounds like a bug.

This is indeed, a bug triggered by this situation and I'll have to fix it.

The unrealized gains plugin computes the list of holdings in order to insert the gains, and it ends up with a holding object like this for your leaked cost basis:

Would you like me to file a bug? A bunch of these pollute my bean-check output, which discourages me from running bean-check as often as I should.

Yes. I forgot to deal with it. I'm overwhelmed... too much to do and the weekends go by really fast. I try to focus on one thing at a time. There are so many things to prioritize. Logging a ticket will help.

I see you recently updated export_reports.py to use commodity metadata. What should the "export" and "ticker" metadata fields contain? What's the difference between "CASH" and "MONEY"? What metadata should I apply to a commodity that has no ticker symbol?

I've been trying to complete a document specifically about this, and I did, and then I realized I should revisit the feature to make it even simpler, and I revised the code this weekend, and now I need to update the document. I'm planning to do this over the weekend. Expect a new "export to OFX" feature that will allow you to keep an eye on changes in your portfolio throughout the day.

 

Matthew

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/41903068-9c5a-46d7-ae45-4763127890d2%40googlegroups.com.

[Matthew Harris]

On Wednesday, February 11, 2015 at 10:57:25 AM UTC-8, Martin Blais wrote:

As for the 

"A valid price for ABC/USD could not be found"

message: it is emitted from the unrealized gains plugin.

I'll have to hunt it down, sounds like a bug.

This is indeed, a bug triggered by this situation and I'll have to fix it.

The unrealized gains plugin computes the list of holdings in order to insert the gains, and it ends up with a holding object like this for your leaked cost basis:

Would you like me to file a bug? A bunch of these pollute my bean-check output, which discourages me from running bean-check as often as I should.

Yes. I forgot to deal with it. I'm overwhelmed... too much to do and the weekends go by really fast. I try to focus on one thing at a time. There are so many things to prioritize. Logging a ticket will help.

Okay, will do. I know you've got TODOs in multiple places, so I didn't want to overwhelm you more by making extra copies of known TODOs.

I've been trying to complete a document specifically about this, and I did, and then I realized I should revisit the feature to make it even simpler, and I revised the code this weekend, and now I need to update the document. I'm planning to do this over the weekend. Expect a new "export to OFX" feature that will allow you to keep an eye on changes in your portfolio throughout the day.

Sounds cool. It isn't one of my immediate priorities, but it will be fun to play with anyway.

Matthew

[Martin Blais]

On Wed, Feb 11, 2015 at 2:09 PM, Matthew Harris <mharr...@gmail.com> wrote:

I've been trying to complete a document specifically about this, and I did, and then I realized I should revisit the feature to make it even simpler, and I revised the code this weekend, and now I need to update the document. I'm planning to do this over the weekend. Expect a new "export to OFX" feature that will allow you to keep an eye on changes in your portfolio throughout the day.

Sounds cool. It isn't one of my immediate priorities, but it will be fun to play with anyway.

Here goes:

https://docs.google.com/document/d/1eZIDRmQZxR6cmDyOJf7U3BnCm4PDMah2twxYFfKPJtM/

Enjoy, (let me know if you run into any problems)