Calculating Portfolio Returns

[Martin Blais]

I took a few weeks off, COVID-style (at home, cooking and coding). So I made some good progress on calculating returns from a Beancount ledger. I'm really excited to share this actually, because it worked so well!

The source code is located here:

https://github.com/beancount/beancount/tree/v2/experiments/returns

and I think it's general enough that you can use it on your own ledger.

I'll need to cover it with unit tests and apply it to the example Beancount file before taking it out of "experiments/" but this should work now.

I've documented the process here:

http://furius.ca/beancount/doc/returns 

https://docs.google.com/document/d/1nPsMIunLnDvdsg6TSsd0PZb7jngojNpFlqnaX36WRp8/

I'm looking forward to feedback, and especially comments from people who manage to make it run on their own ledger and produce useful results.

(The doc is open for comments in suggestion mode.)

[Tuno Tunante]

Wow Martin, what a work! 
I’ve been trying a bit and after some missing/Install packages (protobuf, etc). The configure.py doesn’t work for me.

Output:

Host/returns# python3 configure.py main.bean
Traceback (most recent call last):
 File "configure.py", line 35, in <module>
 options_map: data.Options,
AttributeError: module 'beancount.core.data' has no attribute 'Options'

Cannot find why the error.
The ‘bean-check main.bean' works well.

My beancount is installed from pip3 and my version:
Beancount 2.3.1

Regards.

Tino Tuno

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhNLfJcaD7RFcFsJ%2B-xbk90F3Gr40A8dC0ZQENfHg_QakA%40mail.gmail.com.

[Martin Blais]

Thanks Tuno.

This is living in head of branch v2 at the moment.

I'll release 2.3.2 with it just now.

Keep in mind that you'll probably want to run from a git clone and update regularly, because I'm going to improve this faster than releases for a while.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/e04c3eff-5f00-42b0-a86d-de145e69bad6%40Spark.

[Tuno Tunante]

Hi Martin,

Just to clarify, I’m not too interested in the return project. Just trying to help a bit the projects being a beta-tester ;-)

What I did is: 

1.- Create an environment with `python3 -m venv env` and activating it to don’t mix with my standard config.
2.- Install with pip3 the last ‘git clone’ for the v2 as you indicated. Also all the dependencies needed. 
3.- Run “./configure.py -v bean/main.bean > configuracion" to have the configuration file. There are lines like: 

investment {

    currency: “VTI”

    asset_account: “Assets:Acciones:VTI”

  }

And 

report {

    name: “currency.VTI”

    investment: “Assets:Acciones:VTI”

  }

One for each stock/ETF I have. Until there, no problems.

4. Create a directory “out” and run: “./compute_returns.py bean/main.bean configuracion out”

There is the first error: 

FileNotFoundError: [Errno 2] No such file or directory: ‘out/investments/Assets_Acciones_ABBV.org'

Well.. I imagined I need the directory investments.. Probably would be nice if the directories and sub-directories are done automatically but whatever, I did mkdir for out/investments and still errors about signature sub-sub directories, etc.

I created all the subdirectories that gave me errors…

5. Finally I run: “./compute_returns.py bean/main.bean configuracion out” takes a bit (3 or 4 seconds) Eureka!!! Is working!!!! But…no :-(

I past the results here to avoid the mail too long…: 

https://pastebin.com/LJ7j7B23

Regards.

Tino Tuno

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhN6ndWJoyqPtuNKG77TJMmvwqgQfqb8iQL%2B_jmhxSge8A%40mail.gmail.com.

[Adam Wolenc]

Martin,

Please add 

numpy, scipy, seaborn, and matplotlib to requirements.txt

[Adam Wolenc]

Similarly, fromisoformat is not available until Python 3.7.

[Martin Blais]

Not until the returns code is moved out of experiments. I could do that once I've blanketed it in unit tests.

General speaking I want the v3 repo to be lighter and smaller so maybe adding more officially supported code to v2 isn't wise yet, maybe I should make that project another repo, I'm not sure. Maybe all those experiments don't belong there. 

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/d3084ba9-9272-4281-b377-e9c07c9b5423n%40googlegroups.com.

[Adam Wolenc]

I see. I don't need to build the world in order to get this script to work. Though I do need Python 3.7, and I would like to build the world, near HEAD, successfully at some point.

[Martin Blais]

I'm not sure I understand your comment. Just install google protobuf, numpy, scipy, seaborn, matplotlib, and run from the v2 branch, it should likely just work.

I prototype things in experimental, and eventually promote them to the core when they're got enough of a body and I've got good test coverage; I think this discipline is the only way I've been able to support the core of the project for 12+ years on my off time. But for v3 I want to make the core part I'm supporting directly smaller (bean-price moved out, bean-web/-report deprecated in favor of Fava, and bean-query will move to its own repo) in order to be able to support it actively again (there's too much there now for my bandwidth), so I need to figure out where new things like that will go. It's possible this could just be its own project with its own repo, not sure yet. In the long run I'd like for the Beancount repo to become very lean and move away from also being where "Martin's experiments with Beancount" happen, if you see what I mean. I see more than 300 Beancount-related repos on Github, I'm sure I could define a few more instead of doing everything in one place. Anyhow, I hope this makes sense.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/41d4e5c1-de3a-41c5-aae2-2c7a39ab99b5n%40googlegroups.com.

[Adam Wolenc]

I also get 

    ad.commodity.meta["name"] if ad.commodity else "N/A",

KeyError: 'name'

 as in Tino Tuno's post.

It's because not all of my commodities have names.   After Google Finance's importer stopped working, I didn't have any reason to maintain my commodity directives.

I suggest installing some fall-backs, like so:

        name = 'N/A'

        if ad.commodity:

            name = ad.commodity.currency

        if 'name' in ad.commodity.meta:

            name = ad.commodity.meta["name"]

        rows.append((ad.account, name, status_str))

[Martin Blais]

Absolutely.  Will  do. Thanks for letting me know.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/2a6bf01f-ca97-4bd3-9d61-9c0ef75e4678n%40googlegroups.com.

[Cheng Zhen]

Thanks, Martin.

After lots of account rename/reorganize work, I manage to make it work.

It's really cool.

[redst...@gmail.com]

Terrific. I haven't tested it yet, but will definitely do so when I free up in a few weeks. I started my own initial development on portfolio returns a while ago for fava_investor, and have many tests cases (not yet shareable without anonymization).

[cha...@gmail.com]

Thanks, this looks promising and I'll try it out soon. In the document there is a discussion of comparing your returns to another portfolio. I would suggest using some number of these lazy portfolios for default selections, where of course someone can put in some combination of asset classes to construct a custom portfolio.

[redst...@gmail.com]

This is fantastic, thanks for sharing, Martin. I typed up a review of my experiences with it. See the doc here. I tried to paste it below, but the formatting is lost (this used to work earlier).

Overview

Extracting cashflows needed to compute IRR is challenging because of the flexibility that bookkeeping systems like Beancount need to offer in order to be useful.

This is a review of beancount/experiments/returns that Martin posted recently, including what you need to get up and running, and pitfalls to watch out for.

First, a review:

• Fantastic documentation (as usual) and clarity in code (also as usual), along with clear output that includes the transactions, posting categorizations, templates that were identified, and extracted cashflows in the output files.

• The simplicity of the generalized cashflow method (investments.py: produce_cash_flows_general()) is neat and helpful in building one’s mental model when working with this tool.
• Runtime performance: it takes ~3.5 minutes to run on my transactions of about 10k investment related postings on older hardware (x86, intel core duo 1.2GHz)

 

Per-commodity accounting: not a hard requirement

The documentation may seem to suggest that per-commodity accounts are a requirement or are assumed by the code, but this is not actually true. The code works at the account level, and if you don’t have or use per-commodity accounting, you can still compute returns at an account level. Below are some pitfalls, solutions, and some drawbacks to not using per-commodity accounting.

• The configure.py script may not work for you in this case, but it’s trivial to write your own configuration.
• One problem is, it’s impossible to know which commodity to ascribe a dividend to.  Even worse, reinvested dividends in the “Dividends” computation, if the cash flows into the asset account. Eg:

2020-01-02 * "Buy"

  Assets:Brokerage   1000 HOOL {1 USD}

  Assets:Bank

 

2020-08-01 * "Dividends"

  Assets:Brokerage            500 USD

  Income:Dividends:Brokerage       ; ← not a cash account; not counted

 

2020-08-01 * "Reinvest dividends"

  Assets:Brokerage   500 HOOL {1 USD}

  Assets:Brokerage  -500 USD       ; ← not an external account; not counted

 

However, this is only a problem if you want your total performance broken down into dividends vs. appreciation. If not, the total performance numbers you get will still be correct. Simply ignore the ex-dividends and dividends numbers.

• Several of my accounts correspond to a “real” brokerage account, meaning they mix multiple commodities, cash and/or the money market settlement fund
• returns.py:truncate_cash_flows() fails because my account contains both cash and a settlement fund, variably used to trade commodities. This means the cashflow can’t be reduced to a single currency based on cost, but ends up with two (cash, settlement fund). Note that I use price conversions for the settlement fund (vs. holding them at cost), to avoid generating a ton of lots with no real purpose given that the NAV of these don’t stray much from 1 USD. Fix:

@@ -141,7 +144,10 @@ def truncate_cash_flows(

         balance = compute_balance_at(account_data.transactions, date_start)

         if not balance.is_empty():

             cost_balance = balance.reduce(pricer.get_value, date_start)

+            cost_balance = cost_balance.reduce(convert.convert_position, "USD", pricer.price_map)

             cost_position = cost_balance.get_only_position()

@@ -154,7 +160,13 @@ def truncate_cash_flows(

         balance = compute_balance_at(account_data.transactions, date_end)

         if not balance.is_empty():

             cost_balance = balance.reduce(pricer.get_value, date_end)

-            cost_position = cost_balance.get_only_position()

+            cost_balance = cost_balance.reduce(convert.convert_position, "USD", pricer.price_map)

+            try:

+                cost_position = cost_balance.get_only_position()

 

Verification of results can be challenging:

After building a basic config file and perhaps making a few simple changes to your ledger, compute_returns.py spits out numbers. How can the correctness of these numbers be verified?

 

One way to identify problems is to construct groupings that reflect IRRs provided by brokerages. However, the details and terminology of how IRR calculations are made can vary widely by brokerage. For example, there seems to be no agreement on what the start/end dates should be when computing “3-year performance”. It can thus be a lot of effort to figure these out even if one has just a handful of brokerages.

 

Verifying the output of compute_returns.py may involve some effort. The challenge fundamentally seems to stem from the (fantastic) flexibility Beancount provides in constructing one's ledger, which means that the configuration for compute_returns.py needs to be correct. Short of bugs, all incorrect results are generally because of errors in configuration.

 

So how can one verify that their configuration is correct? Thankfully, this is a place where compute_returns.py shines, with its clear, relevant output. Below, I share a few tips and personal stumbling blocks to help others figure their own:

• Examine the helpful “category map” in the output files. It is the set of all accounts that are involved in transactions with an asset account in question:

cat returns.output/investments/*.org | grep "^.'" | grep None

Also consider whether each of them has been categorized correctly:

cat returns.output/investments/*.org | grep "^.'"

• The root finding solver is initialized to 0.2, so if you see a 20.00% as your CAGR, it’s almost certainly because the solver crashed
• Examining the cashflow graph:
  • Do the inflows and outflows make sense? With many accounts, you have a sense of how frequently you transfer money and how much (eg: every paycheck). Is that being violated?
  • Do the dividends make sense?
  • Transfers should show up as an equal inflow/outflow. If not, investigate
• Spot check at least some of the transactions in the investments/*.org files to ensure their categories make sense. See if there are transactions that may be missing
• Finally, building one’s intuition for what the numbers should be, how the cashflow should look like, and how compute_returns.py works, is generally helpful in spotting errors

 

Modifications I had to make to my ledger:

There were some gotchas that I had to work through that I’ve shared below to give a sense of what one might need to look out for:

• Tighten initial pad postings, so they appear on the day of the first transaction. Else, the period used for computing returns is lengthened, resulting in incorrectly worse performance
• I use the rename plugin I wrote to rename Expenses:Taxes:* to Income:Taxes:*, to avoid it from dominating my expenses for analysis. This meant that the income accounts were categorized as income (investments.py matches against an “Income:” regex) and therefore (incorrectly) not counted as casflow, which happens in transactions that involve tax (eg: sell to cover). I had to add “Income:Taxes:*” as a cash_account. The problems manifested as accounts with obviously incorrect IRR values (eg: -144%), though this may not always happen.
• In transaction entries involving a paycheck (eg: contributions to an employer sponsored retirement account), many postings were uncategorized, and thus eventually had incorrect workflows. My solution was to insulate brokerage transactions from paycheck transactions by having a transfer account in between. I started using transfer accounts in the last few years anyway via the zerosum plugin, but older transactions didn’t
• Eventually, I plan to evaluate switching to per-commodity accounts that Martin has long recommended

 

Configuration  maintenance

As one’s ledger evolves, the reports config needs to be kept up to date. One downside of per-commodity accounts is they are likely to be opened and closed more frequently than physical accounts.

 

Omissions of all types in the returns configuration (asset, dividend, and cash accounts) can be hard to detect. Solutions are likely to be highly specific to the conventions one uses in their ledger. For my ledger for example, I plan to ensure all accounts under “(Assets|Income):Investments:*” appear at least once in the reports config file.

In-kind transfers and returns

One case that is not covered seems to be in-kind transfers. For example:

 

option "operating_currency" "USD"

plugin "beancount.plugins.implicit_prices"

 

2005-01-01 commodity USD

2005-01-01 commodity HOOL

 

2000-01-01 open Assets:Bank

2000-01-01 open Assets:Brokerage:HOOL "STRICT"

2000-01-01 open Assets:Zero-Sum-Accounts:Transfers "STRICT"

 

2010-01-01 * "Buy"

  Assets:Zero-Sum-Accounts:Transfers 1000 HOOL {0.5 USD}

  Assets:Bank

 

2020-01-01 price HOOL 1 USD

 

2020-01-02 * "Transfer"

  Assets:Brokerage:HOOL   1000 HOOL {0.5 USD}

  Assets:Zero-Sum-Accounts:Transfers -1000 HOOL {0.5 USD}

 

2020-12-23 price HOOL 1.1 USD

 

Using the market value on the transfer date for the cashflow value should solve this case.

 

This appears in several cases in me ledger: moving from one brokerage to another, and options exercise. It could be argued that the former could be handled by bundling together the old and new accounts, but that assumes all accounts in question deal with only a single commodity each, which means (for me), redoing a whole bunch of historical transactions.

 

The latter is an interesting case. For context, assume an option to buy stock ABC at 1 USD was exercised when the fair market value of ABC was 5 USD. It was then held for a year, and sold for 10 USD. It is helpful to compute the returns of holding on to the exercised stock (which is an investing decision) separately from the “compensation” portion, which is the 5-1 = 4 USD. To do this, I track them in separate accounts like so:

 

2015-01-01 * "Exercised"

   Assets:Vested             -100 ABC_OPTIONS

   Expenses:Stock-Options     100 ABC_OPTIONS

   Assets:XTrade:Exercised    100 ABC {1 USD}   ; FMV was 4 USD

   Assets:Bank                100 USD           ; 100 * 1 USD

 

2015-01-01 price ABC 4 USD ; FMV

 

2015-01-01 * “Transferred to brokerage”

   Assets:XTrade:Exercised   -100 ABC {1 USD}

   Assets:XTrade:Invested     100 ABC {1 USD}

 

If stock transfers in-kind were computed in the way above, one could simply compute returns separately on Assets:XTrade:Exercised and Assets:XTrade:Invested, and distinguish between the “compensation” and the “investment” portions.

 

Minor feedback

• Legend in graph (in .svg): both amortized value from flows, and market value show up as a black line. In the graph, one is blue + dots, other is black
• How rae trailing, rolling, and total returns period define? Since everyone defines these a bit differently, perhaps balloon help on the UI would clarify it. This script helps, meanwhile:

# Run on 2020-12-23

>>> from reports import *

>>> from tabulate import tabulate

>>> print(tabulate(get_calendar_intervals(TODAY)))

----  ----------  ----------

2005  2005-01-01  2006-01-01

2006  2006-01-01  2007-01-01

2007  2007-01-01  2008-01-01

2008  2008-01-01  2009-01-01

2009  2009-01-01  2010-01-01

2010  2010-01-01  2011-01-01

2011  2011-01-01  2012-01-01

2012  2012-01-01  2013-01-01

2013  2013-01-01  2014-01-01

2014  2014-01-01  2015-01-01

2015  2015-01-01  2016-01-01

2016  2016-01-01  2017-01-01

2017  2017-01-01  2018-01-01

2018  2018-01-01  2019-01-01

2019  2019-01-01  2020-01-01

2020  2020-01-01  2020-12-23

----  ----------  ----------

>>> print(tabulate(get_cumulative_intervals(TODAY)))

--------------------  ----------  ----------

15_years_ago          2005-01-01  2020-12-23

10_years_ago          2010-01-01  2020-12-23

5_years_ago           2015-01-01  2020-12-23

4_years_ago           2016-01-01  2020-12-23

3_years_ago           2017-01-01  2020-12-23

2_years_ago           2018-01-01  2020-12-23

1_year_ago            2019-01-01  2020-12-23

ytd                   2020-01-01  2020-12-23

rolling_6_months_ago  2020-06-25  2020-12-23

rolling_3_months_ago  2020-09-25  2020-12-23

--------------------  ----------  ----------

 

[Martin Blais]

Hey Thanks Red! That's a really nice summary of your user journey. I'm glad to see others picking this up and using it.

--

You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/ebba4de0-c401-4290-a119-2cab59d5b432n%40googlegroups.com.

[redst...@gmail.com]

I have a case that doesn't produce the desired output: a transfer out of an account, in-kind. This gets categorized as an ASSET_OTHERASSET. I'm not sure I understand the reasoning for why this category cashflows out the *cost basis* of the other asset. Shouldn't it be outflowing the market value instead?

More generally, in-kind transfers (in or out) don't seem to work correctly for this reason. They only work if both source and target accounts are considered together in a group.

Source:

--------------------------------------------------------------------------------------------

option "operating_currency" "USD"
plugin "beancount.plugins.implicit_prices"

2005-01-01 commodity USD
2005-01-01 commodity HOOL

2000-01-01 open Assets:Brokerage:USD

2000-01-01 open Assets:Brokerage:HOOL "STRICT"
2000-01-01 open Assets:Zero-Sum-Accounts:Transfers "STRICT"

2020-01-01 * "Buy"
  Assets:Brokerage:HOOL 1000 HOOL {1 USD}
  Assets:Brokerage:USD

2020-10-01 price HOOL 1.1 USD

2020-10-02 * "Transfer out, in kind"
  Assets:Brokerage:HOOL   -1000 HOOL {1 USD}
  Assets:Zero-Sum-Accounts:Transfers  1000 HOOL {1 USD}

Config:

--------------------------------------------------------------------------------------------

investments {
  investment {
    currency: "HOOL"
    asset_account: "Assets:Brokerage:HOOL"
    cash_accounts: "Assets:Brokerage:USD"
    dividend_accounts: "Income:Dividends:Brokerage:HOOL"
  }
}
groups {
  group {
    name: "currency.HOOL"
    investment: "Assets:Brokerage:HOOL"
  }
}

Results:

--------------------------------------------------------------------------------------------

Desired (meaningful) output: 12% CAGR

Actual output: 0.00% (since the cashflow is -1000 for the buy, and +1000 for the transfer).

[Martin Blais]

Move to https://github.com/beancount/beangrow/issues/6

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/fdb4dad4-1d92-4f7b-92e0-26e3732e7603n%40googlegroups.com.

[Bman Q]

Guys hi

First of all i want to thank Martin for superb work! I have been using beancount for 2 years and flexibility / functionality beancount provides are simply amazing. I always recommend beancount on plain text accounting sub and now have 1 extra reason to suggest it :)

Secondly i have several questions / issues with installation and will appreciate your help:

1. should i use "v2 master + beangrow repo" OR "beancount 2.3.3" ?

2. when i run configure.py, it generates empty file, what can be the reason for this?

i run:

"python scripts/configure.py -v b.beancount > conf-returns"

screen output:

INFO    : Reading ledger: b.beancount
INFO    : Inferring configuration.
INFO    : Done.

but file is empty

My accounts structure seems fine:

Assets:Business:...

Assets:Cash

Assets:Investing:CSchwab:Cash                      USD
Income:Investing:CSchwab:Interest                 USD
Income:Investing:CSchwab:Gains                    USD
Income:Investing:CSchwab:MiscCredits         USD
Expenses:Investing:CSchwab:Fees                  USD

Assets:Investing:CSchwab:RDFN                     RDFN

Assets:Investing:CSchwab:SQ                          SQ

...

Assets:Personal:...

(i never received dividends so don't have accounts for them yet)

3. Do i understand it right that the only change we need to do in main beancount file is add signature handlers?

Basically

2019-09-03 * "Buy; REDFIN CORP 00500"
  Assets:Investing:CSchwab:RDFN                     326 RDFN {15.55 USD, 2019-09-03}
  Expenses:Investing:CSchwab:Fees                  4.95 USD
  Assets:Investing:CSchwab:Cash                -5074.25 USD

will become

2019-09-03 * "Buy; REDFIN CORP 00500"
  Assets:Investing:CSchwab:RDFN                     326 RDFN {15.55 USD, 2019-09-03}

     flow: ASSET

  Expenses:Investing:CSchwab:Fees                  4.95 USD

     flow: EXPENSES

  Assets:Investing:CSchwab:Cash                -5074.25 USD

      flow: CASH                                       

4. I tried creating simple config file myself

investments {
  # Accounts at Schwab.                                           
  investment {
    currency: "RDFN"
    asset_account: "Assets:Investing:CSchwab:RDFN"
    cash_accounts: "Assets:Investing:CSchwab:Cash"
  }
  investment {
    currency: "SQ"
    asset_account: "Assets:Investing:CSchwab:SQ"
    cash_accounts: "Assets:Investing:CSchwab:Cash"
  }
}

groups {
  group {
    name: "stocks"
    investment: "Assets:Investing:CSchwab:RDFN"
    investment: "Assets:Investing:CSchwab:SQ"
  }
  group {
    name: "all"
    investment: "Assets:Investing:*"
    currency: "USD"
  }
}

but it resulted in error:

(using beancount master and beangrow repo)

python beangrow/compute_returns.py b.beancount conf-returns out

R:\b\b-office>python beangrow/compute_returns.py b.beancount conf-returns out

Traceback (most recent call last):

  File "beangrow/compute_returns.py", line 105, in <module>
    main()
  File "beangrow/compute_returns.py", line 80, in main
    account_data_map = investments.extract(
  File "R:\b\b-office\beangrow\investments.py", line 553, in extract
    account_data = [process_account_entries(pruned_entries, config.investments, aconfig,
  File "R:\b\b-office\beangrow\investments.py", line 553, in <listcomp>
    account_data = [process_account_entries(pruned_entries, config.investments, aconfig,
  File "R:\b\b-office\beangrow\investments.py", line 399, in process_account_entries
    comm = commodity_map[currency] if currency else None
KeyError: 'RDFN'

(using beancount 2.3.2)

R:\b\b-office>python returns/compute_returns.py b.beancount conf-returns out

Traceback (most recent call last):

  File "returns/compute_returns.py", line 97, in <module>
    main()
  File "returns/compute_returns.py", line 73, in main
    config = configlib.read_config(args.config, args.filter_reports, accounts)
  File "R:\b\b-office\returns\config.py", line 44, in read_config
    text_format.Merge(infile.read(), config)
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\google\protobuf\text_format.py", line 696, in Merge
    return MergeLines(
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\google\protobuf\text_format.py", line 770, in MergeLines
    return parser.MergeLines(lines, message)
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\google\protobuf\text_format.py", line 795, in MergeLines
    self._ParseOrMerge(lines, message)
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\google\protobuf\text_format.py", line 817, in _ParseOrMerge
    self._MergeField(tokenizer, message)
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\google\protobuf\text_format.py", line 907, in _MergeField
    raise tokenizer.ParseErrorPreviousToken(
google.protobuf.text_format.ParseError: 15:1 : Message type "beancount.returns.Config" has no field named "groups".

5. Could someone please share sample beancount and returns config files?

[William Bean]

You need to define commodities for everything you're using beangrow for, in your case I imagine this is RDFN and SQ. You don't need to define the metadata in your bullet item 3.

You received this message because you are subscribed to a topic in the Google Groups "Beancount" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beancount/OJBIGFvbBC8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/50e30e64-b537-4fc3-8654-75a2fa3945c2n%40googlegroups.com.

[Bman Q]

By define commodities do you mean

"2019-09-03 open Assets:Investing:CSchwab:RDFN                      RDFN"

etc ? Sure, they are defined, otherwise i won't be able to use them in beancount/fava.

UPDATE: i see, i think you mean i need to do

"2019-09-03 commodity RDFN". Strange why beancount/fava worked just fine without this declaration.

[Martin Blais]

On Sun, Sep 12, 2021 at 8:40 PM Bman Q <mpl...@gmail.com> wrote:

Guys hi

First of all i want to thank Martin for superb work! I have been using beancount for 2 years and flexibility / functionality beancount provides are simply amazing. I always recommend beancount on plain text accounting sub and now have 1 extra reason to suggest it :)

Thanks!

 

Secondly i have several questions / issues with installation and will appreciate your help:

1. should i use "v2 master + beangrow repo" OR "beancount 2.3.3" ?

v2 if you like stability

beangrow should work with it

master if you want the bleeding edge, but frankly right now it's ~ the same. New code is being written in there but it's not being run through the main pass (there are two parsers in there, the new one is still undergoing development).

 

2. when i run configure.py, it generates empty file, what can be the reason for this?

[...]

beangrow has been written a good long time ago and it did work great at the time.

I haven't run it in about a year myself, since I've switch a much more active trading style, so there may have been drift and breakage.

(FWIW I've been developing a completely different trade reporting system to accommodate active trading (https://github.com/beancount/johnny) but it's much more complicated and doesn't integrate well into beancount yet.)

Unlike beancount, beangrow is a bit more experimental and much less mature than beancount, you'll have to insert debugging code and figure out why somehow the inferred configuration isn't working.  It's not something I have ongoing tests for, and I have more than 10 active projects + a full time job, not enough cycles to keep all of those in tip-top shape.

 

3. Do i understand it right that the only change we need to do in main beancount file is add signature handlers?

No you shouldn't have to do that.

You should be able to create an input configuration that works on your regular Beancount file.

Do you have postings with the currency RDFN in your Beancount file?

 

 

(using beancount 2.3.2)

R:\b\b-office>python returns/compute_returns.py b.beancount conf-returns out
Traceback (most recent call last):
  File "returns/compute_returns.py", line 97, in <module>
    main()
  File "returns/compute_returns.py", line 73, in main
    config = configlib.read_config(args.config, args.filter_reports, accounts)
  File "R:\b\b-office\returns\config.py", line 44, in read_config
    text_format.Merge(infile.read(), config)
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\google\protobuf\text_format.py", line 696, in Merge
    return MergeLines(
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\google\protobuf\text_format.py", line 770, in MergeLines
    return parser.MergeLines(lines, message)
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\google\protobuf\text_format.py", line 795, in MergeLines
    self._ParseOrMerge(lines, message)
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\google\protobuf\text_format.py", line 817, in _ParseOrMerge
    self._MergeField(tokenizer, message)
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\google\protobuf\text_format.py", line 907, in _MergeField
    raise tokenizer.ParseErrorPreviousToken(
google.protobuf.text_format.ParseError: 15:1 : Message type "beancount.returns.Config" has no field named "groups".

Not sure what's up.

 

5. Could someone please share sample beancount and returns config files?

Yeah it would be nice if the config worked on the big example.beancount file from the beancount repo.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/50e30e64-b537-4fc3-8654-75a2fa3945c2n%40googlegroups.com.

[Bman Q]

thank you, it worked.

On Sunday, September 12, 2021 at 8:46:06 PM UTC-4 wbe...@gmail.com wrote:

[Bman Q]

thank you for your reply,

i understand that beangrow is experimental, and i really appreciate what it can do even in this form, hopefully we (i see some users on github) will contribute to work too. Your work is really helpful to lots of people, one more time thank you.

I managed to make bean grow work, i needed to add "commodity" declarations. (surprisingly beancount / fava worked without them just fine) Plus couple windows related bugs for which i opened issues on beangrow's github providing temp fixes.

I have couple of additional questions on bean grow set-up:

1. configure.py seems to ignore closed accounts, is it how it supposed to work?

(usually for strictness i was closing accounts after i exited/sold position, example: "2020-10-06 close Assets:Investing:CSchwab:RDFN")

2. "Including Uninvested Cash"

Do i understand it right that the easiest way to account for uninvested cash is to create

-commodity "IDDLEUSD"

-account Assets:Investing:IDDLEUSD

buy IDDLEUSD for all cash (USD) that i can invest at rate 1:1,

and then at each interval i do reports "price IDDLEUSD 1 USD",

and when i want to buy smng sell IDDLEUSD for USD, and buy what i want.

?

OR is there a way to make my investment cash accounts "Assets:Investing:CSchwab:Cash", etc to be counted as investment (with 0 return) ?

3. Is there recommended time frame for "price" quotes?

Usually i open positions for 3-5 years, so not interested in daily or even monthly fluctuations.

Do i understand it right that if i use

-"plugin "beancount.plugins.implicit_prices"" (to get price from buy/sell transactions)

-price my investments on 1st of each Quarter

-run compute_returns.py with -e DATE = 1st of each Quarter (basically same cut off as date of last price quote)

then i will be fine and won't need any more price quotes

?

[Martin Blais]

On Mon, Sep 13, 2021 at 8:12 PM Bman Q <mpl...@gmail.com> wrote:

thank you for your reply,

i understand that beangrow is experimental, and i really appreciate what it can do even in this form, hopefully we (i see some users on github) will contribute to work too. Your work is really helpful to lots of people, one more time thank you.

I managed to make bean grow work, i needed to add "commodity" declarations. (surprisingly beancount / fava worked without them just fine) Plus couple windows related bugs for which i opened issues on beangrow's github providing temp fixes.

Ah I see. I may have assumed you use the "check_commodity" plugin, which ensures all commodities have a corresponding declarations.

(This will be default in v3.)

 

I have couple of additional questions on bean grow set-up:

1. configure.py seems to ignore closed accounts, is it how it supposed to work?

(usually for strictness i was closing accounts after i exited/sold position, example: "2020-10-06 close Assets:Investing:CSchwab:RDFN")

I don't remember, but I'm pretty sure I calculated returns on closed accounts.

 

2. "Including Uninvested Cash"

Do i understand it right that the easiest way to account for uninvested cash is to create

-commodity "IDDLEUSD"

-account Assets:Investing:IDDLEUSD

buy IDDLEUSD for all cash (USD) that i can invest at rate 1:1,

and then at each interval i do reports "price IDDLEUSD 1 USD",

and when i want to buy smng sell IDDLEUSD for USD, and buy what i want.

?

OR is there a way to make my investment cash accounts "Assets:Investing:CSchwab:Cash", etc to be counted as investment (with 0 return) ?

I don't remember, honestly, but this is a really important aspect. Uninvested cash has to count. I doubt I ignored it.

 

3. Is there recommended time frame for "price" quotes?

Usually i open positions for 3-5 years, so not interested in daily or even monthly fluctuations.

Do i understand it right that if i use

-"plugin "beancount.plugins.implicit_prices"" (to get price from buy/sell transactions)

-price my investments on 1st of each Quarter

-run compute_returns.py with -e DATE = 1st of each Quarter (basically same cut off as date of last price quote)

then i will be fine and won't need any more price quotes

?

Yes, that's good enough.

IIRC I was using a weekly or monthly interval.

Just depends how smooth you want the curve to be.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/702f73e7-70b1-4dfe-b119-706ab320243bn%40googlegroups.com.

[redst...@gmail.com]

On Monday, September 13, 2021 at 5:12:30 PM UTC-7 Bman Q wrote:

I have couple of additional questions on bean grow set-up:

1. configure.py seems to ignore closed accounts, is it how it supposed to work?

(usually for strictness i was closing accounts after i exited/sold position, example: "2020-10-06 close Assets:Investing:CSchwab:RDFN")

Hmm, it shouldn't: see this line and comment and its code:

"""Return a list of account names from the balance sheet which either aren't
closed or are closed now but were still open at the given start date. """

If you're seeing a problem case, would you mind opening a bug in the repo with a demo example?

2. "Including Uninvested Cash"

Do i understand it right that the easiest way to account for uninvested cash is to create

-commodity "IDDLEUSD"

-account Assets:Investing:IDDLEUSD

buy IDDLEUSD for all cash (USD) that i can invest at rate 1:1,

and then at each interval i do reports "price IDDLEUSD 1 USD",

and when i want to buy smng sell IDDLEUSD for USD, and buy what i want.

?

OR is there a way to make my investment cash accounts "Assets:Investing:CSchwab:Cash", etc to be counted as investment (with 0 return) ?

The latter is exactly what I do for now: simply include "Assets:Investing:CSchwab:Cash" as a commodity, and it'll show up with very low returns.

However, for me, this means that cash in other parts of the hierarchy (example: banks, paypal, etc.) are not counted. AFAIK, beangrow does not offer a global solution for this. See "Including Uninvested Cash" in Martin's doc.

Not really an analysis solution, but as a strategy, I personally use fava_investor to monitor uninvested cash.

 

3. Is there recommended time frame for "price" quotes?

Usually i open positions for 3-5 years, so not interested in daily or even monthly fluctuations.

Do i understand it right that if i use

-"plugin "beancount.plugins.implicit_prices"" (to get price from buy/sell transactions)

-price my investments on 1st of each Quarter

-run compute_returns.py with -e DATE = 1st of each Quarter (basically same cut off as date of last price quote)

then i will be fine and won't need any more price quotes

?

After including the implicit_prices plugin, and running beangrow once, you will want to run:

beangrow/beangrow/download_prices_from_file.py <your_returns_output_dir>/prices/prices.beancount

and include the resulting price database in your source, and re-reun beangrow. Without that, beangrow uses the price values on the closest dates it can find, which in your case could be weeks or months off, which means your returns numbers will be off.

The file generated above provides the only dates on which the prices matter to beangrow for your configuration. These are the start and end dates of your cashflow. Meaning, period opening and closing dates, and dates on which you bought/sold, IIRC. More detail here.

Enjoy!

 

[Bman Q]

"assumed you use the "check_commodity" plugin"

- thanks for recommending the plugin, It's indeed useful and good idea making it default in v3.

[Bman Q]

thank you for reply,

"If you're seeing a problem case, would you mind opening a bug in the repo with a demo example?"

- opened the issue https://github.com/beancount/beangrow/issues/12

"The latter is exactly what I do for now: simply include "Assets:Investing:CSchwab:Cash" as a commodity, and it'll show up with very low returns."

- could you please share your config file for this cash account?

I can't seem to make it work with currency same as my main currency (USD). Or do you use separate currency/commodity just for that account?

I tried

"investments {
  investment {
    currency: "USD"
    asset_account: "Assets:Investing:CSchwab:Cash"
    cash_accounts: "Assets:Cash"
  }
}
groups {
  group {
    name: "CSchwabCash"
    investment: "Assets:Investing:CSchwab:Cash"
  }
}"

and

"investments {
  investment {
    currency: "USD"
    asset_account: "Assets:Investing:CSchwab:Cash"
    cash_accounts: "Assets:Cash"
  }
}
groups {
  group {
    name: "CSchwabCash"
    investment: "Assets:Investing:CSchwab:Cash"
    currency: "USD"
  }
}"

But i am getting error

"R:\b\b-office>python beangrow/compute_returns.py -e 2021-09-01 b.beancount conf-returns returns
R:\b\b-office\beangrow\returns.py:63: RuntimeWarning: invalid value encountered in power
  return np.sum(cash_flows / (1. + irr) ** years)
R:\b\b-office\beangrow\returns.py:63: RuntimeWarning: invalid value encountered in power
  return np.sum(cash_flows / (1. + irr) ** years)

Traceback (most recent call last):
  File "beangrow/compute_returns.py", line 105, in <module>
    main()

  File "beangrow/compute_returns.py", line 94, in main
    reports.generate_price_pages(account_data_map,
  File "R:\b\b-office\beangrow\reports.py", line 528, in generate_price_pages
    all_prices = prices.get_all_prices(price_map, base_quote)
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\beancount\core\prices.py", line 303, in get_all_prices
    return _lookup_price_and_inverse(price_map, base_quote)
  File "C:\Users\Btycoon\AppData\Local\Programs\Python\Python38\lib\site-packages\beancount\core\prices.py", line 278, in _lookup_price_and_inverse
    return price_map[base_quote]
KeyError: ('USD', 'USD')"

[Bman Q]

I managed to bypass the "KeyError: ('USD', 'USD')"" by removing "currency: "USD"" from investment, basically making it:

"investments {
  investment {

    asset_account: "Assets:Investing:CSchwab:Cash"

    cash_accounts: "Assets:Personal:Checking"

  }
}
groups {
  group {
    name: "CSchwabCash"
    investment: "Assets:Investing:CSchwab:Cash"
  }
}"

But then when i try to add other cash accounts (as i understand it for CSchwab:Cash my investments (RDFN, SQ, etc) become cash_accounts (accounts external to the investment)):

  investment {

    asset_account: "Assets:Investing:CSchwab:Cash"

    cash_accounts: "Assets:Personal:Checking"

    cash_accounts: "Assets:Investing:CSchwab:RDFN"

    cash_accounts: "Assets:Investing:CSchwab:SQ

    .....

  }

i get error

R:\b\b-office>python beangrow/compute_returns.py -e 2021-09-01 b.beancount conf-returns returns

Traceback (most recent call last):
  File "beangrow/compute_returns.py", line 105, in <module>
    main()
  File "beangrow/compute_returns.py", line 80, in main
    account_data_map = investments.extract(
  File "R:\b\b-office\beangrow\investments.py", line 553, in extract
    account_data = [process_account_entries(pruned_entries, config.investments, aconfig,
  File "R:\b\b-office\beangrow\investments.py", line 553, in <listcomp>
    account_data = [process_account_entries(pruned_entries, config.investments, aconfig,

  File "R:\b\b-office\beangrow\investments.py", line 376, in process_account_entries
    flows_general = produce_cash_flows_general(entry, account)
  File "R:\b\b-office\beangrow\investments.py", line 178, in produce_cash_flows_general
    assert not posting.cost
AssertionError

(same as the one you reported on github for In-kind transfers https://github.com/beancount/beangrow/issues/3 )

On Monday, September 13, 2021 at 11:15:15 PM UTC-4 redst...@gmail.com wrote:

[redst...@gmail.com]

On Tuesday, September 14, 2021 at 8:32:20 AM UTC-7 Bman Q wrote:

thank you for reply,

"If you're seeing a problem case, would you mind opening a bug in the repo with a demo example?"

- opened the issue https://github.com/beancount/beangrow/issues/12

 

Using the `--start-date` option to `configure.py` makes it work as expected. However, I agree this could be better: the default behavior, when that option is not used is unintuitive IMHO. More in that bug.

 

"The latter is exactly what I do for now: simply include "Assets:Investing:CSchwab:Cash" as a commodity, and it'll show up with very low returns."

- could you please share your config file for this cash account?

I can't seem to make it work with currency same as my main currency (USD). Or do you use separate currency/commodity just for that account?

I looked up what I do, and it's not clear to me that it works correctly since it failed a simple example. Let me get back on this.

[Bman Q]

"Yeah it would be nice if the config worked on the big example.beancount file from the beancount repo."

- i tried to create sample project from example.beantcount file, but there were too many moving pieces, and some things were not clear (like "Dividends on portfolio" was not clear to which investment it belongs to)

so i created another one:

https://github.com/gety9/beangrow/tree/examples/examples

it has capital gains returns, dividend returns, grouping growth / value, investments separated in difference accounts and i explain a bit how math and cashflows work showing that computed IRR is accurate.

could you have a look and tell what you think? if it's fine i will create PR

"I don't remember, honestly, but this is a really important aspect. Uninvested cash has to count. I doubt I ignored it."

- it's in the section of "Future Work" of google doc file :)

but we can achieve "uninvested cash account" now with following setting in config

  investment {
    asset_account: "Assets:Investing:CSchwab:Cash"
    cash_accounts: "Assets:Personal:Checking"

    cash_accounts: "Income:Investing:CSchwab:KO:Dividends"

    cash_accounts: "Income:Investing:CSchwab:Gains"
    cash_accounts: "Expenses:Investing:CSchwab:Fees"
  }

it works exactly as it should, with no interest return is 0, with interest return equals to interest,

the downsides:

- "--check-explicit-flows" fails (cause of ASSET_OTHERASSET when assets are purchased / sold)

but general rules produce list of cash flows perfectly fine.

- if one have lots of Dividend accounts he have to enter all of them or combine them in one account

if you think above solution is appropriate then i can add it to aove example files

i also create brunch for https://github.com/beancount/beangrow/issues/4

https://github.com/gety9/beangrow/commit/5a0368a3637fbb1bf21cc370767d8b1c6c49535c

and added minimal instructions to readme (things i notice asked in github issues and here)

https://github.com/gety9/beangrow/tree/read-me

if you think they are ok i will create PR.

[Chary Chary]

Martin,

I was reading through the Beancount - Calculating Portfolio Returns document

Very thorough (as always)! 

May be this is a naïve question, but what is your definition of the portfolio return in general, without diving in beancount?

E.g. 

if on 1 January I invested 100 USD which grew by 1 February to 150 USD and then I withdrew all of this

and then on 1 June I unvested 1000 USD, which by 1 December grew to 1010 USD, which I withdrew

Then how would you calculate the portfolio return?

[Martin Blais]

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/87548033-01bd-463c-b096-aceafc15fc7fn%40googlegroups.com.

[Chary Chary]

Martin,

thanks! I learned something today.

There now a question as to why brokers do not calculate this correctly, provided there is even an Excel formula, but this is another story.

P.S. may be it makes sense to update the Calculating Portfolio Returns document with this information to prevent questions.